目标:爬取“湖南大学”百科词条并处理数据

 

需要获取的数据:

python简单爬虫 用beautifulsoup爬取百度百科词条

 

源代码:

<div class="basic-info cmn-clearfix">
<dl class="basicInfo-block basicInfo-left">
<dt class="basicInfo-item name">中文名</dt>
<dd class="basicInfo-item value">
湖南大学
</dd>
<dt class="basicInfo-item name">外文名</dt>
<dd class="basicInfo-item value">
Hunan University
</dd>
<dt class="basicInfo-item name">&nbsp;&nbsp;&nbsp;&nbsp;</dt>
<dd class="basicInfo-item value">
<a target="_blank" href="/item/%E6%B9%96%E5%A4%A7/406102" data-lemmaid="406102">湖大</a>·<a target="_blank" href="/item/HNU">HNU</a>
</dd>
<dt class="basicInfo-item name">创办时间</dt>
<dd class="basicInfo-item value">
1903年
</dd>
<dt class="basicInfo-item name">&nbsp;&nbsp;&nbsp;&nbsp;</dt>
<dd class="basicInfo-item value">
公立大学
</dd>
<dt class="basicInfo-item name">&nbsp;&nbsp;&nbsp;&nbsp;</dt>
<dd class="basicInfo-item value">
综合类<a target="_blank" href="/item/%E7%A0%94%E7%A9%B6%E5%9E%8B%E5%A4%A7%E5%AD%A6">研究型大学</a>
</dd>
.......
<dd class="basicInfo-item value"> <a target="_blank" href="/item/%E9%92%9F%E5%BF%97%E5%8D%8E">钟志华</a><a target="_blank" href="/item/%E8%B0%AD%E8%94%9A%E6%B3%93">谭蔚泓</a><a target="_blank" href="/item/%E4%BD%95%E4%BA%BA%E5%8F%AF">何人可</a><a target="_blank" href="/item/%E7%86%8A%E6%99%93%E9%B8%BD">熊晓鸽</a><a target="_blank" href="/item/%E8%92%8B%E8%B6%85%E8%89%AF">蒋超良</a><a target="_blank" href="/item/%E9%82%93%E9%A3%9E/1525913" data-lemmaid="1525913">邓飞</a><a target="_blank" href="/item/%E8%B0%AD%E8%80%95">谭耕</a></dd> </dl></div>

 

库:由于百度有反爬机制,所以使用urllib.request库获取网页;BeautifulSoup

 

代码:

from bs4 import BeautifulSoup
import urllib.request


url = "https://baike.baidu.com/item/%E6%B9%96%E5%8D%97%E5%A4%A7%E5%AD%A6/179157?fr=aladdin" response = urllib.request.urlopen(url) # 访问并打开url html = response.read() # 创建html对象读取页面源代码 soup = BeautifulSoup(html, 'html.parser') # 创建soup对象,获取html代码 title = soup.find_all('dt', class_="basicInfo-item name") # 找到所有dt标签,返回一个列表 node = soup.find_all('dd', class_="basicInfo-item value") # 找到所有dd标签,返回一个列表 allunivinfo = [] titlelist = [] infolist = [] for i in title: # 将所有dt标签内容存入列表 title = i.get_text() titlelist.append(title) for i in node: # 将所有dd标签内容存入列表 info = i.get_text() infolist.append(info) for i, j in zip(titlelist, infolist): # 多遍历循环,zip()接受一系列可迭代对象作为参数,将对象中对应的元素打包成一个个tuple(元组),然后返回由这些tuples组成的list(列表)。 info = ''.join((str(i)+':'+str(j)).split()) allunivinfo.append(info) print(allunivinfo)

 

运行结果:

python简单爬虫 用beautifulsoup爬取百度百科词条