爬虫之Requests&beautifulsoup

　　网络爬虫（又被称为网页蜘蛛，网络机器人，在FOAF社区中间，更经常的称为网页追逐者），是一种按照一定的规则，自动地抓取万维网信息的程序或者脚本。另外一些不常使用的名字还有蚂蚁、自动索引、模拟程序或者蠕虫。

目录

一、Requests

二、BeautifulSoup

三、自动登陆抽屉并点赞

四、“破解”微信公众号

五、自动登陆示例

一、Requests

Python标准库中提供了：urllib、urllib2、httplib等模块以供Http请求，但是，它的 API 太渣了。它是为另一个时代、另一个互联网所创建的。它需要巨量的工作，甚至包括各种方法覆盖，来完成最简单的任务。

• 封装urllib请求

import urllib2
import json
import cookielib


def urllib2_request(url, method="GET", cookie="", headers={}, data=None):
    """
    :param url: 要请求的url
    :param cookie: 请求方式，GET、POST、DELETE、PUT..
    :param cookie: 要传入的cookie，cookie= 'k1=v1;k1=v2'
    :param headers: 发送数据时携带的请求头，headers = {'ContentType':'application/json; charset=UTF-8'}
    :param data: 要发送的数据GET方式需要传入参数，data={'d1': 'v1'}
    :return: 返回元祖，响应的字符串内容 和 cookiejar对象
    对于cookiejar对象，可以使用for循环访问：
        for item in cookiejar:
            print item.name,item.value
    """
    if data:
        data = json.dumps(data)

    cookie_jar = cookielib.CookieJar()
    handler = urllib2.HTTPCookieProcessor(cookie_jar)
    opener = urllib2.build_opener(handler)
    opener.addheaders.append(['Cookie', 'k1=v1;k1=v2'])
    request = urllib2.Request(url=url, data=data, headers=headers)
    request.get_method = lambda: method

    response = opener.open(request)
    origin = response.read()

    return origin, cookie_jar


# GET
result = urllib2_request('http://127.0.0.1:8001/index/', method="GET")

# POST
result = urllib2_request('http://127.0.0.1:8001/index/',  method="POST", data= {'k1': 'v1'})

# PUT
result = urllib2_request('http://127.0.0.1:8001/index/',  method="PUT", data= {'k1': 'v1'})

Requests 是使用 Apache2 Licensed 许可证的 基于Python开发的HTTP 库，其在Python内置模块的基础上进行了高度的封装，从而使得Pythoner进行网络请求时，变得美好了许多，使用Requests可以轻而易举的完成浏览器可有的任何操作。

1、GET请求

# 1、无参数实例
 
import requests
 
ret = requests.get('https://github.com/timeline.json')
 
print ret.url
print ret.text
 
# 2、有参数实例
 
import requests
 
payload = {'key1': 'value1', 'key2': 'value2'}
ret = requests.get("http://httpbin.org/get", params=payload)
 
print ret.url
print ret.text

向 https://github.com/timeline.json 发送一个GET请求，将请求和响应相关均封装在 ret 对象中。

实例：爬取汽车之家新闻标题、链接和图片

import requests
import scrapy
from bs4 import BeautifulSoup
import os
import uuid

response = requests.get(url='https://www.autohome.com.cn/news/',)
# 用下载自带的编码规则解析
# 也可以指定response.encoding = 'utf-8'
response.encoding = response.apparent_encoding
# response.status_code 返回的状态码
# lxml性能更好，但需要安装，html.parser是Python内置的
soup = BeautifulSoup(response.text,features='html.parser')
tag1 = soup.find(id='auto-channel-lazyload-article')
# tag2 = tag1.find('li') # 只找到第一个
tag_list = tag1.find_all('li')  # 找到全部，以列表形式输出
for tag in tag_list:
    tag_a = tag.find('a')
    if tag_a:  # 有a标签的才拿属性
        # print(tag_a.attrs.get('href')) # 新闻链接
        h3_text = tag_a.find('h3')  # 其实是一个对象，只是打印的时候显示文本
        # text和string都是获取标签对象的文本
        # print(h3_text.string,'*****') # 新闻标题
        # print(h3_text.text,'----')   # 新闻标题
        img_url = tag_a.find('img').attrs.get('src')
        # print(img_url) # 新闻图片
        # 下载图片,.text返回的是文本，.content返回的是字节
        # 如果直接写url=img_url，最后会出现http:////p9.pstatp.com/list/pgc-image/153838020
        img_response = requests.get("http:" + img_url).content
        file_name = 'imgs/' + str(uuid.uuid4()) + '.jpg'
        # with open(file_name,'wb') as f:
        #     f.write(img_response)

2、POST请求

# 1、基本POST实例
 
import requests
 
payload = {'key1': 'value1', 'key2': 'value2'}
ret = requests.post("http://httpbin.org/post", data=payload)
 
print ret.text
 
# 2、发送请求头和数据实例
 
import requests
import json
 
url = 'https://api.github.com/some/endpoint'
payload = {'some': 'data'}
headers = {'content-type': 'application/json'}
 
ret = requests.post(url, data=json.dumps(payload), headers=headers)
 
print ret.text
print ret.cookies

向https://api.github.com/some/endpoint发送一个POST请求，将请求和相应相关的内容封装在 ret 对象中。

3、其他请求

requests.get(url, params=None, **kwargs)
requests.post(url, data=None, json=None, **kwargs)
requests.put(url, data=None, **kwargs)
requests.head(url, **kwargs)
requests.delete(url, **kwargs)
requests.patch(url, data=None, **kwargs)
requests.options(url, **kwargs)
 
# 以上方法均是在此方法的基础上构建
requests.request(method, url, **kwargs)
params={'k1':'v1'} 以GET形式放入URL传到后台的参数

requests模块已经将常用的Http请求方法为用户封装完成，用户直接调用其提供的相应方法即可，其中方法的所有参数有：

def request(method, url, **kwargs):
    """Constructs and sends a :class:`Request <Request>`.

    :param method: method for the new :class:`Request` object.
    :param url: URL for the new :class:`Request` object.
    :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
    :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
    :param json: (optional) json data to send in the body of the :class:`Request`.
    :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
    :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
    :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': ('filename', fileobj)}``) for multipart encoding upload.
    :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
    :param timeout: (optional) How long to wait for the server to send data
        before giving up, as a float, or a :ref:`(connect timeout, read
        timeout) <timeouts>` tuple.
    :type timeout: float or tuple
    :param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed.
    :type allow_redirects: bool
    :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
    :param verify: (optional) whether the SSL cert will be verified. A CA_BUNDLE path can also be provided. Defaults to ``True``.
    :param stream: (optional) if ``False``, the response content will be immediately downloaded.
    :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
    :return: :class:`Response <Response>` object
    :rtype: requests.Response

    Usage::

      >>> import requests
      >>> req = requests.request('GET', 'http://httpbin.org/get')
      <Response [200]>
    """

    # By using the 'with' statement we are sure the session is closed, thus we
    # avoid leaving sockets open which can trigger a ResourceWarning in some
    # cases, and look like a memory leak in others.
    with sessions.Session() as session:
        return session.request(method=method, url=url, **kwargs)

View Code