Requests高级用法
1.文件上传
我们知道requests可以模拟提交一些数据。假如有的网站需要上传文件,我们也可以用requests来实现。
import requests files = {'file': open('favicon.ico', 'rb')} res = requests.post('http://httpbin.org/post', files=files) print(res.text)
上一篇博客中,我们保存了一个favicon.ico文件,这次用它来模拟文件上传的过程。favicon.ico文件需要和当前脚本在同一目录下,运行结果如下:
{ "args": {}, "data": "", "files": { "file": "data:application/octet-stream;base64,AAABAAIAEBAAAAEAIAAoBQAAJgAAACAgAAABACAAKBQAAE4FAAA......AAAAAAA=" }, "form": {}, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Content-Length": "6665", "Content-Type": "multipart/form-data; boundary=f03eff231fa14cfbbd7ef2cd51bae3b6", "Host": "httpbin.org", "User-Agent": "python-requests/2.18.4" }, "json": null, "origin": "183.129.61.15, 183.129.61.15", "url": "https://httpbin.org/post" }
以上在files字段中省略了部分内容,这个网站会返回响应,里面包含files字段,而form字段为空,这说明文件上传会使用单独的字段来标识。
2.Cookies
先用一个示例查看获取Cookies的过程:
# 获取Cookies res = requests.get('http://www.baidu.com') print(res.cookies) for k, v in res.cookies.items(): print(k + ":" + v)
结果如下:
<RequestsCookieJar[<Cookie BDORZ=27315 for .baidu.com/>]> BDORZ:27315
这里调用cookies属性即可获取到Cookies,可以发现它是一个RequestsCookieJar类型,使用items()方法将其转化为元祖组成的列表,遍历输出每一个Cookies的名称和值,实现Cookies的遍历解析。
当然,我们也可以使用Cookies来维持登陆状态,下面以百度个人中心为例。首先登录百度个人中心,将Headers中的Cookies内容复制下来:
可以替换成 你自己的Cookie,将其设置到Headers里面,然后在发送请求:
# 设置cookie,发送请求 headers = { 'Cookie': 'BIDUPSID=EDBD62A1D608B5CDC72B06CA0C59680A; PSTM=1546959880; BDUSS=BCSHh3OVMtTW5XUnB6SEF4UFExUFhPRUc5Ty1sV' 'FRkRERvRWZ3bE9sWlN2Y0ZjQVFBQJCQAAAAAAAAAAAEAAABW7900TGltaXTSwcrW1drM7AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAFIwmlxSMJpcV3; MCITY=-340%3A; BAIDUID=5C140375092D783C088CEA7282B89772' ':FG=1; delPer=0; PSINO=5; BDRCVFR[bLbo9QmdyQn]=aeXf-1x8UdYcs; BDORZ=FFFB88E999055A3F8A630C64834BD6D0; ' 'H_PS_PSSID=1427_21108_29064_28519_29099_28831_28585_22157; PHPSESSID=63vpocfpukn41u0h1nolsluq95; ' 'Hm_lvt_4010fd5075fcfe46a16ec4cb65e02f04=1558949858; Hm_lpvt_4010fd5075fcfe46a16ec4cb65e02f04=1558949858', 'Host':'i.baidu.com' } res = requests.get('http://i.baidu.com/', headers=headers) print(res.text)
结果如下:
<!doctype html> <html> <head> <meta charset="UTF-8"> <!-- <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0, user-scalable=no" /> --> <title>百度个人中心——您在百度的家</title> <meta name="description" content="百度个人中心是百度用户管理个性化信息的平台,在这里,您可以体验贴吧的交流、知道的解答、百科的全面、空间的个性、云盘的强大……感受百度为您带来的方便、快捷与乐趣。"> <meta name="keywords" content="个人中心,个人主页,个人帐号,个人空间"> <!--[if lt IE 7]> <script> window.location.href = 'http://i.baidu.com/static/ieup/index.html'; </script> <![endif]--> <link rel="shortcut icon" type="image/ico" href="http://www.baidu.com/favicon.ico" /> <link rel="stylesheet" href="http://uc-vs.bdimg.com/v4/20181107x2/dep/font-awesome/4.0.3/less/font-awesome.css"> <!--[if IE 7]> <link rel="stylesheet" href="http://uc-vs.bdimg.com/v4/20181107x2/dep/font-awesome/4.0.3/less/font-awesome-ie7.css"> <![endif]--> <link rel="stylesheet" href="http://uc-vs.bdimg.com/v4/20181107x2/asset/home/css/main.css"> <!--[if lt IE 9]> <script src="http://uc-vs.bdimg.com/v4/20181107x2/dep/htmlshiv/3.7.0/html5shiv.min.js"></script> <![endif]--> <script src="http://uc-vs.bdimg.com/v4/20181107x2/dep/respond/1.3.0/respond.min.js"></script> <link href="http://uc-vs.bdimg.com/v4/20181107x2/dep/respond/1.3.0/respond-proxy.html" id="respond-proxy" rel="respond-proxy" /> <link href="http://i.baidu.com/static/respond/respond.proxy.gif" id="respond-redirect" rel="respond-redirect" /> <script src="http://i.baidu.com/static/respond/respond.proxy.js"></script> <script> var _hmt = _hmt || []; (function() { var hm = document.createElement("script"); hm.src = "//hm.baidu.com/hm.js?4010fd5075fcfe46a16ec4cb65e02f04"; var s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(hm, s); })(); window.GLOBAL = {"token":"5307adbc3a64a28464950c46be298366"}; GLOBAL['apps'] = { open: [ 'video', 'nuomi', 'lvyou', 'zhidao', 'baifubao', 'koubei' ] }; GLOBAL['feRoot'] = 'http://uc-vs.bdimg.com/v4/20181107x2'; </script> </head> <body class="result-op"> <div class="main-wrap"> <header data-click='{"mod":"header"}'> <div class="header-stackup" data-scroll-reveal="enter from the top over 0.66s"> <div class="ibx-container row clr"> <a href="/" target="_self" class="header-logo header-logo-index"></a> <div class="header-tool header-tool-user"> <a class="header-tu-img header-tool-user-nick" href="javascript:;" /> Limit伊手遮天 </a> <div class="header-tc-content"> <span class="header-tc-ct-bg"></span> <ul class="header-user-menu-list"> <li class="header-user-menu-item"> <a data-click='{"act":"h_history"}' href="/my/history"> 搜索记录 </a> </li> <li class="header-user-menu-item"> <a data-click='{"act":"h_cal"}' href="/my/cal"> 我的日历 </a> </li> <li class="header-user-menu-item"> <a data-click='{"act":"h_favorite"}' href="/my/collect"> 我的收藏 </a> </li> <li class="header-user-menu-item"> <a data-click='{"act":"h_jubao"}' href="/my/jubao"> 我的举报 </a> </li> <li class="header-user-menu-item"> <a data-click='{"act":"h_hb"}' target="_blank" href="https://www.baifubao.com/user/0/my_bfb/0"> 我的钱包 </a> </li> </ul> <a class="header-tuc-logout" href="http://passport.baidu.com/?logout&u=http://i.baidu.com/welcome/">退出帐号</a> </div> </div> <div class="header-tool header-tool-config"> <a href="/my/allconf/#newsconf" class="header-tool-conf-link">设置</a> <div class="header-tc-content"> <span class="header-tc-ct-bg"></span> <a class="header-tcc-account" target="_blank" href="http://passport.baidu.com/center">帐号设置</a> <a class="header-tcc-search" href="/my/allconf/#newsconf">消息设置</a> </div> </div> <div class="header-tool header-tool-cal"> <a data-click='{"act":"cal_click"}' href="/my/cal" class="header-tool-cal-link"> 日历 </a> <span class="header-tool-cal-bg"></span> <div class="header-tc-content"> <span class="header-tc-ct-bg"></span> </div> </div> <div class="header-tool header-tool-news"> <a href="/msg/messages/list/" class="header-tool-news-num">消息</a> <div class="header-tc-content"> <span class="header-tc-ct-bg"></span> <div class="news-header">数据加载中。。。</div> </div> </div> <div class="header-tool header-tool-ps"> <a href="https://www.baidu.com" target="_blank"/>百度首页</a> </div> </div> </div> </header> <section data-scroll-reveal="enter from the top over 0.66s" class="header-app ibx-container" data-click='{"mod":"header"}'> <ul class="ibx-header-app clr"> <li class="ibx-header-app-item"> <div class="ibx-hai-title ibx-hai-tool">工具</div> </li> <li class="ibx-header-app-item clr"> <span class="ibx-hai-link"><a data-click='{"act":"sousuo"}' target="_blank" href="http://www.baidu.com?fr=ibaidu">搜索</a></span> <span class="ibx-hai-link"><a href="/my/collect">收藏</a></span> </li> <li class="ibx-header-app-item"> <span class="ibx-hai-link"><a data-click='{"act":"wangpan"}' target="_blank" href="http://pan.baidu.com/disk/home?fr=ibaidu">云盘</a></span> <span class="ibx-hai-link"><a href="/my/cal">日历</a></span> </li> <li class="ibx-header-app-item"> <span class="ibx-hai-link"><a data-click='{"act":"yingyin"}' target="_blank" href="http://player.baidu.com/yingyin.html">影音</a></span> <span class="ibx-hai-link"><a data-click='{"act":"fanyi"}' target="_blank" href="http://fanyi.baidu.com?fr=ibaidu">翻译</a></span> </li> <li class="ibx-header-app-item"> <div class="ibx-hai-space"></div> </li> <li class="ibx-header-app-item"> <div class="ibx-hai-title ibx-hai-sns">社区</div> </li> <li class="ibx-header-app-item"> <span class="ibx-hai-link"><a data-click='{"act":"tieba"}' target="_blank" href="http://tieba.baidu.com/home/main?un=Limit伊手遮天&fr=ibaidu&ie=utf-8">贴吧</a></span> <span class="ibx-hai-link"><a data-click='{"act":"baike"}' target="_blank" href="http://baike.baidu.com/usercenter?fr=ibaidu">百科</a></span> </li> <li class="ibx-header-app-item"> <span class="ibx-hai-link"><a data-click='{"act":"zhidao"}' target="_blank" href="http://zhidao.baidu.com/uhome?fr=ibaidu">知道</a></span> <span class="ibx-hai-link"><a data-click='{"act":"wenku"}' target="_blank" href="http://wenku.baidu.com/user/index?fr=ibaidu">文库</a></span> </li> <li class="ibx-header-app-item"> <span class="ibx-hai-link"><a data-click='{"act":"jingyan"}' target="_blank" href="http://jingyan.baidu.com/user/nuc">经验</a></span> </li> <li class="ibx-header-app-item"> <div class="ibx-hai-space"></div> </li> <li class="ibx-header-app-item"> <div class="ibx-hai-title ibx-hai-life">生活</div> </li> <li class="ibx-header-app-item"> <span class="ibx-hai-link"><a data-click='{"act":"ditu"}' target="_blank" href="http://map.baidu.com?fr=ibaidu">地图</a></span> <span class="ibx-hai-link"><a data-click='{"act":"licai"}' target="_blank" href="https://licai.baidu.com/user/0/center/0?fr=ibaidu">理财</a></span> </li> <li class="ibx-header-app-item"> <span class="ibx-hai-link"><a data-click='{"act":"tuangou"}' target="_blank" href="http://www.nuomi.com/?cid=002544&fr=ibaidu">团购</a></span> <span class="ibx-hai-link"><a data-click='{"act":"baifubao"}' target="_blank" href="http://baifubao.baidu.com/jump?uri=/user/0/my_bfb/0&fr=ibaidu">钱包</a></span> </li> <li class="ibx-header-app-item"> <span class="ibx-hai-link"><a data-click='{"act":"lvyou"}' target="_blank" href="http://lvyou.baidu.com/user/?fr=ibaidu">旅游</a></span> <span class="ibx-hai-link"><a data-click='{"act":"education"}' target="_blank" href="http://www.chuanke.com/?fr=ibaidu">教育</a></span> </li> <li class="ibx-header-app-item"> <span class="ibx-hai-link"><a data-click='{"act":"waimai"}' target="_blank" href="http://waimai.baidu.com/waimai?qt=find&fr=ibaidu">外卖</a></span> </li> <li class="ibx-header-app-item"> <div class="ibx-hai-space"></div> </li> <li class="ibx-header-app-item"> <div class="ibx-hai-title ibx-hai-fun">娱乐</div> </li> <li class="ibx-header-app-item"> <span class="ibx-hai-link"><a data-click='{"act":"news"}' target="_blank" href="http://news.baidu.com?fr=ibaidu">新闻</a></span> <span class="ibx-hai-link"><a data-click='{"act":"games"}' target="_blank" href="http://youxi.baidu.com/my_game_center.xhtml?fr=ibaidu">游戏</a></span> </li> <li class="ibx-header-app-item"> <span class="ibx-hai-link"><a data-click='{"act":"video"}' target="_blank" href="http://v.baidu.com/user/?fr=ibaidu">视频</a></span> <span class="ibx-hai-link"><a data-click='{"act":"music"}' target="_blank" href="http://play.baidu.com/?fr=ibaidu">音乐</a></span> </li> <li class="ibx-header-app-item"> <div class="ibx-hai-space"></div> </li> <li class="ibx-header-app-item"> <div class="ibx-hai-title ibx-hai-v">诚信</div> </li> <li class="ibx-header-app-item"> <span class="ibx-hai-link"><a data-click='{"act":"baozhang"}' target="_blank" href="http://baozhang.baidu.com/guarantee?fr=ibaidu">网民权益保障</a></span> <span class="ibx-hai-link"><a data-click='{"act":"koubei"}' target="_blank" href="http://koubei.baidu.com/truth/wall">口碑</a></span> </li> <li class="ibx-header-app-item"> <span class="ibx-hai-link"><a href="javascript:;"> </a></span> <span class="ibx-hai-link"><a data-click='{"act":"jubao"}' href="/my/jubao">举报</a></span> </li> </ul> </section> <main class="ibx-container row"> <div data-scroll-reveal class="row user-panel"> <div class="col span_5_1"> <div id="ibx-uc" data-click='{"mod":"uc"}'> <div class="ibx-inner"> <div class="ibx-uc-uimg"> <div class="ibx-uc-uimg-mask"> <a data-click='{"act":"uc_set"}' class="ibx-uc-ulink" target="_blank" href="http://www.baidu.com/p/setting/profile/portrait">更换头像</a> </div> <img class="ibx-uc-img" src="http://himg.bdimg.com/sys/portrait/item/56ef4c696d6974e4bc8ae6898be981aee5a4a9dd34.jpg" /> </div> <div class="ibx-uc-unick"> <a data-click='{"act":"uc_name"}' target="_blank" href="http://www.baidu.com/p/setting/profile/basic" class="ibx-uc-nick">Limit伊手遮天</a> </div> <div class="ibx-uc-utool"> <span class="ibx-uc-utool-cover"></span> <span class="ibx-uc-utool-cover ibx-uc-utool-cover-mask"></span> <div class="ibx-uc-utool-title">认证 :</div> <div class="ibx-uc-utool-content clr"> <a data-click='{"act":"icon_mobile"}' class="ibx-uc-utool-mobile current" target="_blank" href="http://passport.baidu.com/center" data-title="绑定手机"> </a> <a data-click='{"act":"icon_email"}' class="ibx-uc-utool-envelope current" target="_blank" href="http://passport.baidu.com/center" data-title="绑定邮箱"> </a> </div> </div> <div class="ibx-uc-uop"> <div class="ibx-uc-uop-item first"> <a data-click='{"act":"uc_dingdan"}' target="_blank" href="http://dingdan.baidu.com/my/order/" title="订单中心"> 订单中心 </a> </div> <div class="ibx-uc-uop-item"> <a data-click='{"act":"uc_setting"}' target="_blank" href="http://passport.baidu.com" title="帐户设置"> 帐户设置 </a> </div> </div> </div> </div> </div> <div class="col span_5_4"> <div id="ibx-cal" data-click='{"mod":"cal"}'> <div class="ibx-inner" id="ibx-cal-content"> </div> </div> </div> </div> <div data-scroll-reveal class="row card-panel" data-click='{"mod": "card_video"}'> <span class="rule-wrap"><a id="video" name="video" class="rule"></a></span> <div class="col span_4_4"> <div class="ibx-even"> <div class="ibx-inner" id="ibx-mod-video"> </div> </div> </div> </div> <div data-scroll-reveal class="row card-panel" data-click='{"mod": "card_nuomi"}'> <span class="rule-wrap"><a id="nuomi" name="nuomi" class="rule"></a></span> <div class="col span_4_4"> <div class="ibx-even"> <div class="ibx-inner" id="ibx-mod-nuomi"> </div> </div> </div> </div> <div data-scroll-reveal class="row card-panel" data-click='{"mod": "card_lvyou"}'> <span class="rule-wrap"><a id="lvyou" name="lvyou" class="rule"></a></span> <div class="col span_4_4"> <div class="ibx-even"> <div class="ibx-inner" id="ibx-mod-lvyou"> </div> </div> </div> </div> <div class="row card-panel"> <div data-scroll-reveal class="col span_4_2" data-click='{"mod": "card_zhidao"}'> <span class="rule-wrap"><a id="zhidao" name="zhidao" class="rule"></a></span> <div class="ibx-odd"> <div id="ibx-mod-zhidao" class="ibx-inner ibx-mod-zhidao"> </div> </div> </div> <div data-scroll-reveal class="col span_4_2" data-click='{"mod": "card_baifubao"}'> <span class="rule-wrap"><a id="baifubao" name="baifubao" class="rule"></a></span> <div class="ibx-odd"> <div id="ibx-mod-baifubao" class="ibx-inner ibx-mod-baifubao"> </div> </div> </div> <div data-scroll-reveal class="col span_4_2" data-click='{"mod": "card_koubei"}'> <span class="rule-wrap"><a id="koubei" name="koubei" class="rule"></a></span> <div class="ibx-odd"> <div id="ibx-mod-koubei" class="ibx-inner ibx-mod-koubei"> </div> </div> </div> <div data-scroll-reveal class="col span_4_2 OP_LOG_LINK" data-click='{"mod":"card_mgr","act":"msg_click"}'> <div class="ibx-even editCard"> <div class="ibx-inner editCard-inner"> <div class="editCard-inner-add"></div> <div class="editCard-inner-tip">添加卡片</div> </div> </div> </div> </div> </main> <footer class="row" data-click='{"mod":"footer"}'> ©2019 Baidu <a class="beforebd" href="http://www.baidu.com/duty/">使用百度前必读</a> 京ICP证030173号 </footer> <div class="ibx-advice"> <a class="ibx-advice-logo" href="javascript:;"></a> <div class="ibx-advice-ctx">意见反馈</div> </div> <div data-click='{"mod":"card_mgr","act":"msg_click"}' class="ibx-manage"> <a class="ibx-manage-logo" href="javascript:;"></a> <div class="ibx-manage-ctx OP_LOG_OTHERS">卡片管理</div> </div> <div class="back-to-top-wrap" data-click='{"mod":"back-to-top","act":"back-to-top-link"' > <a class="back-to-top" href="javascript:;"></a> </div> <div class="ibx-card-manage" data-click='{"mod":"card_mgr"}'> <div class="ibx-cm-mask"></div> <div class="ibx-cm-main"> <div class="ibx-cm-main-title clr"> <div class="ibx-cm-main-title-ctx">卡片管理</div> <a href="javascript:;" class="ibx-cm-main-close" title="关闭"></a> </div> <div class="ibx-cm-main-add"> <div class="ibx-cm-main-add-ctx"> <span class="ibx-cm-main-adding-ctx">可添加</span> </div> <ul class="ibx-cmm-add-list"> <li class="ibx-cmm-add-item default"> 无可添加卡片 </li> </ul> </div> <div class="ibx-cm-main-added"> <div class="ibx-cm-main-add-ctx"> <span class="ibx-cm-main-added-ctx">已添加</span> </div> <ul class="ibx-cmm-add-list"> <li data-click='{"act":"set_video"}' class="ibx-cmm-add-item ibx-card-video" data-type="video"> </li> <li data-click='{"act":"set_nuomi"}' class="ibx-cmm-add-item ibx-card-nuomi" data-type="nuomi"> </li> <li data-click='{"act":"set_lvyou"}' class="ibx-cmm-add-item ibx-card-lvyou" data-type="lvyou"> </li> <li data-click='{"act":"set_zhidao"}' class="ibx-cmm-add-item ibx-card-zhidao" data-type="zhidao"> </li> <li data-click='{"act":"set_baifubao"}' class="ibx-cmm-add-item ibx-card-baifubao" data-type="baifubao"> </li> <li data-click='{"act":"set_koubei"}' class="ibx-cmm-add-item ibx-card-koubei" data-type="koubei"> </li> <li class="ibx-cmm-add-item default"> 无已添加卡片 </li> </ul> </div> </div> </div> </div> <script src="http://s1.bdstatic.com/r/www/cache/ecom/esl/1-6-6/esl.min.js"></script> <script> require.config({ 'baseUrl': 'http://uc-vs.bdimg.com/v4/20181107x2/asset', 'paths': { 'css': 'common/js/esl/css', 'js': 'common/js/esl/js', 'text': 'common/js/esl/text' }, 'packages': [ { 'name': 'jquery', 'location': '../dep/jquery/1.9.1/asset', 'main': 'jquery.min.js' }, { 'name': 'saber-emitter', 'location': '../dep/saber-emitter/0.3.0/asset', 'main': 'emitter' }, { 'name': 'hogan', 'location': '../dep/hogan/2.0.0', 'main': 'hogan' }, { 'name': 'zxui', 'location': '../dep/zxui/0.4.8/asset/ui', 'main': 'Control' }, { 'name': 'zrender', 'location': '../dep/zrender/1.0.6/asset', 'main': 'zrender' }, { 'name': 'echarts', 'location': '../dep/echarts/1.2.1/asset', 'main': 'echarts' }, { "name": "scrollReveal", "location": "../dep/scrollReveal/0.1.1/dist", "main": "scrollReveal" }, { "name": "json2", "location": "../dep/json2/1.0.0/asset", "main": "json2" }, { 'name': 'muplayer', 'location': '../dep/muplayer/0.9.1/dist', 'main': 'player' }, { 'name': 'moment', 'location': '../dep/moment/2.0.1/asset', 'main': 'moment' }, { 'name': 'video', 'location': './home/video', 'main': 'main' }, { 'name': 'picture', 'location': './home/picture', 'main': 'main' }, { 'name': 'music', 'location': './home/music', 'main': 'main' }, { 'name': 'tieba', 'location': './home/tieba', 'main': 'main' }, { 'name': 'history', 'location': './home/history', 'main': 'main' }, { 'name': 'yun', 'location': './home/yun', 'main': 'main' }, { 'name': 'zhidao', 'location': './home/zhidao', 'main': 'main' }, { 'name': 'baifubao', 'location': './home/baifubao', 'main': 'main' }, { 'name': 'koubei', 'location': './home/koubei', 'main': 'main' }, { 'name': 'lvyou', 'location': './home/lvyou', 'main': 'main' }, { 'name': 'chuanke', 'location': './home/chuanke', 'main': 'main' }, { 'name': 'nuomi', 'location': './home/nuomi', 'main': 'main' }, { 'name': 'xinwen', 'location': './home/xinwen', 'main': 'main' } ] }); require(['jquery', 'home/main', 'zxui/log'], function ($, home, log) { log.config({ action: 'http://i.baidu.com/static/v.gif?', data: { ppid: '886959958' } }); $(document).ready(function () { home.init(); $('.ibx-advice').click(function() { if (window.bds && bds.qa && bds.qa.ShortCut) { bds.qa.ShortCut.initRightBar(); } else { $.getScript('http://f3.baidu.com/index.php/feedback/zx/loadfeedback?pid=21', function() { bds.qa.ShortCut.initRightBar(); }); }; }); log.start(); }); }); </script> </body> </html> Process finished with exit code 0
View Code
本站文章如无特殊说明,均为本站原创,如若转载,请注明出处:爬虫—Requests高级用法 - Python技术站