爬虫多次访问同一个网站一段时间后会出现错误 HTTPConnectionPool(host:XX)Max retries exceeded with url '<requests.packages.urllib3.connection.HTTPConnection object at XXXX>: Failed to establish a new connection: [Errno 99] Cannot assign requested address'

是因为在每次数据传输前客户端要和服务器建立TCP连接,为节省传输消耗,默认为keep-alive,即连接一次,传输多次,然而在多次访问后不能结束并回到连接池中,导致不能产生新的连接

headers中的Connection默认为keep-alive,

将header中的Connection一项置为close

headers = {

    'Connection': 'close',

}

r = requests.get(url, headers=headers)

此时问题解决

HTTPConnectionPool(host:XX)Max retries exceeded with url 解决方法的更多相关文章

  1. HTTPConnectionPool(host='xx.xx.xx.xx', port=xx): Max retries exceeded with url:(Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x0000015A25025EB8>...))

    HTTPConnectionPool(host='xx.xx.xx.xx', port=xx): Max retries exceeded with url:(Caused by ConnectTim ...

  2. scrapy::Max retries exceeded with url

    运行scrapy时出错这个错误:Max retries exceeded with url解决方法: img1=requests.get(url=aa,headers=header1,timeout= ...

  3. HTTPConnectionPool(host:XX)Max retries exceeded with url

    爬虫多次访问同一个网站一段时间后会出现错误 HTTPConnectionPool(host:XX)Max retries exceeded with url '<requests.package ...

  4. 关于python3.6上传文件时报错:HTTPSConnectionPool(host='***.org', port=443): Max retries exceeded with url: /post (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAIL解决办法

    第一个报错: 最近在练习post请求中上传文件时遇到了一个奇葩事情,两台电脑上写了一模一样的代码,一个运行正常,另一个一片红. 最后了解了一下原因以及解决办法.先记录下关键代码: files = {& ...

  5. windows环境pip安装时一直报错Could not fetch URL https://pypi.org/simple/xrld/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url:

    最近项目不忙了~~有开始专研的python大业,上来想用pip安装一个第三方的库,就一直报错: Could not fetch URL https://pypi.org/simple/xrld/: T ...

  6. HTTPSConnectionPool(host='xxxxx', port=443): Max retries exceeded with url:xxxxxxxx (Caused by NewConnectionError('<urllib3.connect,Max retries exceeded with ,(Caused by NewConnectionError

    HTTPSConnectionPool(host='f6ws-sha8re-o88k.s3.ama66zaws.com', port=443): Max retries exceeded with u ...

  7. Max retries exceeded with url

    78 Traceback (most recent call last):   File "thread072413.py", line 163, in <module> ...

  8. 解决Max retries exceeded with url的问题

    requests.exceptions.ConnectionError: HTTPSConnectionPool(host='itunes.apple.com', port=443): Max ret ...

  9. requests.exceptions.SSLError……Max retries exceeded with url错误求助!!!

    import requests head = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl ...

随机推荐

  1. hdu3294 Girls' research manacher

    One day, sailormoon girls are so delighted that they intend to research about palindromic strings. O ...

  2. CentOS下如何根据Dump文件分析线上问题

    https://blog.csdn.net/lixin03080/article/details/79711296 一.保存现场 1.记录系统整体资源使用情况,进程信息.线程信息 top -b -n ...

  3. js 三目运算

    语法: 一个条件,二选一: 条件?当条件满足时返回的值:当条件不满足时返回的值 多个条件,多选一: 条件1?条件1满足时的值: 条件2?条件2满足时的值: 条件n ?条件n 满足时的值: 默认值: 以 ...

  4. WCF 学习总结5 -- 消息拦截实现用户名验证(转)

    WCF建立在基于消息的通信这一概念基础上.通过方法调用(Method Call)形式体现的服务访问需要转化成具体的消息,并通过相应的编码(Encoding)才能通过传输通道发送到服务端:服务操作执行的 ...

  5. golang xml parent node add attribute without struct

    question: golang  encoding/xml: foo>bar,attr - foo ignored solution: you can replace output resul ...

  6. Centos7安装teamviewer 32/64位

    centos7安装teamviewer teamviewer安装包,自己去官网下吧,不用找了,估摸着就没有Centos7 64位的 所以直接下centos teamviewer 32位那个rpm包吧 ...

  7. zabbix自定义触发器进行监控

    给某一主机创建触发器 触发器属性,其中centos是主机名,也就是你监控的那台主机的名字,可以点击bp2,查看该主机的hostname 检测该触发器 在该主机下可以看到刚创建的触发器 最后我们给该主机 ...

  8. execve函数的介绍与使用

    #include<stdio.h> #include<unistd.h> int main() { char *filename[]={"./BP",NUL ...

  9. Understanding how uid and gid work in Docker containers

    转自:https://medium.com/@mccode/understanding-how-uid-and-gid-work-in-docker-containers-c37a01d01cf Un ...

  10. skipper backend 负载均衡配置

    skipper 对于后端是支持负载均衡处理的,支持官方文档并没有提供,实际使用中,这个还是比较重要的 同时支持健康检查. 格式 hello_lb_group: Path("/foo" ...