python爬虫知识点总结（四）Requests库的基本使用

官方文档：http://docs.python-requests.org/en/master

安装方法

　　命令行下输入：pip3 install requests。详见：https://www.cnblogs.com/cthon/p/9388304.html

一、什么是Requets?

requets

实例引入

import requests

response = requests.get('https://www.baidu.com')

print(type(response))

print(response.status_code)

print(type(response.text))

print(response.text)

print(response.cookies)

各种请求方式

import requests

requests.post('http://httpbin.org/post')

requests.put('http://httpbin.org/put')

requests.delete('http://httpbin.org/delete')

requests.get('http://httpbin.org/get')

requests.options('http://httpbin.org/get')

请求

基本GET请求

基本写法

import requests

response = requests.get('http://httpbin.org/get')

print(response.text)

带参数GET请求

import requests

response = requests.get('http://httpbin.org/get?name=jack&age=22')

print(response.text)

import requests

data = {

    'name':'jack',

    'age':22

}

response = requests.get('http://httpbin.org/get',params=data)

print(response.text)

解析json

import requests

import json

response = requests.get('https://github.com/get')

print(type(response.text))

print(response.json())

print(json.loads(response.text))

print(type(response.json()))

获取二进制数据

import requests

response = requests.get('https://github.com/favicon.ico')

print(type(response.text),type(response.content))

print(response.text)

print(response.content)

import requests

response = requests.get('https://www.bilibili.com/video/av24028845/?p=9')

with open('q.avi','wb') as f:

    f.write(response.content)

    f.close()

添加headers

import requests

response = requests.get('https://zhihu.com/explore')

print(response.text)

import requests

headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36'}

response = requests.get('https://www.zhihu.com/explore',headers=headers)

print(response.text)

基本POST请求

import requests

data = {'name':'jack','age':'22'}

response = requests.post('https://httpbin.org/post',data=data)

print(response.text)

print(response.json())

import requests

data = {'name':'jack','age':'22'}

headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36'}

response = requests.post('https://httpbin.org/post',data=data,headers=headers)

print(response.text)

print(response.json())

响应

response属性

import requests

response = requests.get('http://www.jianshu.com')

print(type(response.status_code),response.status_code)

print(type(response.headers),response.headers)

print(type(response.cookies),response.cookies)

print(type(response.url),response.url)

print(type(response.history),response.history)

状态码判断

import requests

response = requests.get('http://www.cnblogs.com/cthon/p/9383778.html')

exit() if not response.status_code == requests.codes.not_found else print('404 Not Found')

import requests

response = requests.get('http://www.cnblogs.com/cthon/p/9383778.html')

exit() if not response.status_code == 200 else print('Request Successfully')

状态码

100:('continue',),

101:('switching_protocols',),

102:('processing',),

103:('checkpoint',),

122:('url_too_long','request_url_too_long'),

200:('ok','okay','all_ok','all_okay','all_good','\\o/','√',),

201:('created',),

202:('accepted',),

203:('non_authoritative_info','non_authoritative_information'),

204:('no_content',),

205:('reset_content','reset',),

206:('partial_content','partial'),

207:('multi_status','multiple_status','multi_stati','multiple_status'),

208:('already_reported',),

226:('im_used',),

#Redirection

300:('multiple_choices',),

301:('moved_permanently','moved','\\o-'),

302:('found',),

303:('see_other','other'),

304:('not_modified',),

305:('use_proxy',),

306:('switch_proxy',),

307:('temporary_redirect','temporary_moved','temporary'),

308:('permanent_redirect','temporary_moved','temporary',),#There 2 to be removed in 3.0

#Client Error

400:('bad_request','bad'),

401:('unauthorized',),

402:('payment_required','payment'),

403:('forbidden',),

404:('not_found','-o-'),

405:('method_not_allowed','not_allowed'),

406:('not_acceptable',),

407:('proxy_authentication_required','proxy_auth','proxy_authentication'),

408:('request_timeout','timeout'),

409:('confict',),

410('gone',),

411:('length_required',),

412:('precondition_failed','precondition'),

413:('request_entity_too_large',),

414:('request_url_too_large',),

415:('unsupported_media_type','unsupported_media','media_type'),

416:('requested_range_not_satisfiable','requestd_range','range_not_satisfiable'),

417:('expectation_request',),

418:('im_a_teapot','teapot','i_am_a_teapot'),

421:('misdirected_request',),

422:('unprocessable_entity','unprocessable'),

423:('locked',),

424:('failed_dependency','dependency'),

425:('unordered_collection','unordered'),

426:('upgrade_required','upgrade'),

428:('precondition_required','precondition'),

429:('too_many_requests','too_many'),

431:('header_fields_too_large','fields_too_large'),

444:('no_response','none'),

449:('retry_with','retry'),

450:('blocked_by_windows_parental_controls','parental_controls'),

451:('unavailable_for_legal_reasons','legal_reasons'),

499:('client_closed_request',),

#Server Error

500:('internal_server_error','server_error','/o\\','×'),

501:('not_implemented',),

502:('bad_gateway',),

503:('service_unavailable','unavailable'),

504:('gateway_timeout',),

505:('http_version_not_supported','http_version'),

506:('variant_also_negotiaes',),

507:('insufficient_storage',),

509:('bandwidth_limit_exceeded','bandwidth'),

510:('not_extended',),

511:('network_aurhentication_required','network_auth','network_authentication'),

高级文件操作

import requests

files= {'file':open('favicon.ico','rb')}

response = requests.post('http://httpbin.org/post',files=files)

print(response.text)

获取Cookie

import requests

response = requests.get('http://www.baidu.com')

print(response.cookies)

for key,value in response.cookies.items():

    print(key+'='+value)

会话维持

import requests

requests.get('http://httpbin.org/cookies/set/number/123456789')

response=requests.get('http://httpbin.org/cookies')

print(response.text)

import requests

s = requests.Session()

s.get('http://httpbin.org/cookies/set/number/123456789')

response=s.get('http://httpbin.org/cookies')

print(response.text)

证书验证

#12306错误证书，请求失败

import requests

response = requests.get('https://www.12306.cn/')

print(response.status_code)

import requests

from requests.packages import urllib3

urllib3.disable_warnings()

response = requests.get('https://www.12306.cn',verify = False)

print(response.status_code)

import requests

reeponse = requests.get('https://www.12306.cn',cer=('/path/server.crt','/path/key'))

print(response.status_code)

代理设置

　　http代理

import requests

proxies = {

    'http':'http://127.0.0.1:9743',

    'https':'https://127.0.0.1:9743'

}

response = requests.get('https://www.taobao.com',proxies=proxies)

print(response.status_code)

import requests

proxies = {

    'http':'http:/user:password@/127.0.0.1:9743'

}

response = requests.get('https://www.taobao.com',proxies=proxies)

print(response.status_code)

　　socket代理

pip3 install 'requests[socks]'

import requests

proxies = {

    'http':'socks5://127.0..0.1.9742',

    'https':'socks5://127.0.0.1:9742'

}

response = requests.get('https://www.taobao.com',proxies=proxies)

print(response.status_code)

超时设置

import requests

from requests.exceptions import ReadTimeout

try:

    response = requests.get('http://www.baidu.com',timeout = 0.01)

    print(response.status_code)

except ReadTimeout:

    print('Timeout')

认证设置

import requests

from requests.auth import HTTPBasicAuth 

r = requests.get('http://120.27.34.24:9001',auth=HTTPBasicAuth('user','123'))

print(r.status_code)

import requests

r = requests.get('http://120.27.34.24:9001',auth=('user','123'))

print(r.status_code)

异常处理

import requests

from requests.exceptions import ReadTimeout,HTTPError,RequestException

try:

    response = requests.get('http://www.baidu.com',timeout=0.1)

    print(response.status_code)

except ReadTimeout:

    print('Timeout')

except HTTPError:

    print('Http error')

except ConnectionError:

    print('Connection Error')

except RequestException:

    print('Error')

python爬虫知识点总结（四）Requests库的基本使用的更多相关文章

(转)Python爬虫利器一之Requests库的用法
官方文档以下内容大多来自于官方文档,本文进行了一些修改和总结.要了解更多可以参考官方文档安装利用 pip 安装 $ pip install requests 或者利用 easy_install ...
Python爬虫利器一之Requests库的用法
前言之前我们用了 urllib 库,这个作为入门的工具还是不错的,对了解一些爬虫的基本理念,掌握爬虫爬取的流程有所帮助.入门之后,我们就需要学习一些更加高级的内容和工具来方便我们的爬取.那么这一节来 ...
python爬虫入门三：requests库
urllib库在很多时候都比较繁琐,比如处理Cookies.因此,我们选择学习另一个更为简单易用的HTTP库:Requests. requests官方文档 1. 什么是Requests Request ...
python爬虫（八） requests库之 get请求
requests库比urllib库更加方便,包含了很多功能. 1.在使用之前需要先安装pip,在pycharm中打开: 写入pip install requests命令,即可下载在github中有关 ...
python爬虫（6）--Requests库的用法
1.安装利用pip来安装reques库,进入pip的下载位置,打开cmd,默认地址为 C:\Python27\Scripts 可以看到文件中有pip.exe,直接在上面输入cmd回车,进入命令行界面 ...
9.Python爬虫利器一之Requests库的用法（一）
requests 官方文档: http://cn.python-requests.org/zh_CN/latest/user/quickstart.html request 是一个第三方的HTTP库 ...
Python爬虫学习笔记-2.Requests库
Requests是Python的一个优雅而简单的HTTP库,它比Pyhton内置的urllib库,更加强大. 0X01 基本使用安装 Requests,只要在你的终端中运行这个简单命令即可: pip ...
python爬虫（九） requests库之post请求
1.方法: response=requests.post("https://www.baidu.com/s",data=data) 2.拉勾网职位信息获取因为拉勾网设置了反爬虫机 ...
python爬虫学习，使用requests库来实现模拟登录4399小游戏网站。
1.首先分析请求,打开4399网站. 右键检查元素或者F12打开开发者工具.然后找到network选项, 这里最好勾选perserve log 选项,用来保存请求日志.这时我们来先用我们的账号密码登陆 ...
python爬虫学习(6) —— 神器 Requests
Requests 是使用 Apache2 Licensed 许可证的 HTTP 库.用 Python 编写,真正的为人类着想. Python 标准库中的 urllib2 模块提供了你所需要的大多数 H ...

随机推荐

Xenomai 3 migration
Xenomai 3 的rtdm驱动更像一般的Linux驱动,named device会在/dev/rtdm/xxx创建一个设备文件.而用户空间使用时,写得来也和Linux的一般char设备相似,ope ...
Dubbo（一）Dubbo资料
这个资料绝对权威了:http://dubbo.io/user-guide/
Mac修改默认python版本
研究python爬虫,需要用到Beautiful Soup 但是Mac默认的python版本为2.7 自己安装了3.6的版本 import 报错查找资料: Mac在启动,会先加载系统配置文件(包括~ ...
【selenium+Python unittest】之发送带中文附件的邮箱
完整原码如下: import smtplib from email.mime.text import MIMEText #from email.header import Header from em ...
【PyCharm编辑器】之报：Spellchecker inspection helps locate typos and misspelling in your code, comments and literals, and fix them in one click.问题
如上图,输入一个单词时会出现波浪线,报:Spellchecker inspection helps locate typos and misspelling in your code, comment ...
pycharm 5 注册码
BIG3CLIK6F-eyJsaWNlbnNlSWQiOiJCSUczQ0xJSzZGIiwibGljZW5zZWVOYW1lIjoibGFuIHl1IiwiYXNzaWduZWVOYW1lIjoiI ...
多媒体开发之rtp打包---打包中的FU-A分包方式说明
继上篇rtp中的时间戳和负载类型之后,升入到了nalu的分片打包问题,这里做下笔记 (1)fu-a的打包格式 1.基于RTP协议的打包及解包 (1)单个NAL打包 H.264NALU单元常由[star ...
Vue中表单校验
1.安装校验插件vee-validate npm install vee-validate --save 2.在main.js中引用插件 // 表单校验 import VeeValidate, { V ...
怎么使用Aspose.Cells读取excel 转化为Datatable
说明:vs2012 asp.net mvc4 c# 使用Aspose.Cells 读取Excel 转化为Datatable 1.HTML前端代码 <%@ Page Language=" ...
如何在windows上创建文件名以“.”开头的文件
比如要创建.env文件,正常会提示必须输入文件名才能创建的,但是可以在后面再加一个点就能创建了,.env.这样就可以了

python爬虫知识点总结（四）Requests库的基本使用

requets

实例引入

各种请求方式

请求

基本GET请求

基本写法

带参数GET请求

解析json

获取二进制数据

添加headers

基本POST请求

响应

response属性

状态码判断

状态码

高级文件操作

获取Cookie

会话维持

证书验证

代理设置

http代理

socket代理

超时设置

认证设置

异常处理

python爬虫知识点总结（四）Requests库的基本使用的更多相关文章

随机推荐

热门专题

　　http代理

　　socket代理