爬虫多次访问同一个网站一段时间后会出现错误 HTTPConnectionPool（host:XX）Max retries exceeded with url '<requests.packages.urllib3.connection.HTTPConnection object at XXXX>: Failed to establish a new connection: [Errno 99] Cannot assign requested address'
是因为在每次数据传输前客户端要和服务器建立TCP连接，为节省传输消耗，默认为keep-alive，即连接一次，传输多次，然而在多次访问后不能结束并回到连接池中，导致不能产生新的连接
headers中的Connection默认为keep-alive，
将header中的Connection一项置为close

headers = {
    'Connection': 'close',
}
r = requests.get(url, headers=headers)

此时问题解决

HTTPConnectionPool（host:XX）Max retries exceeded with url的更多相关文章

  1. HTTPConnectionPool（host:XX）Max retries exceeded with url 解决方法

    爬虫多次访问同一个网站一段时间后会出现错误 HTTPConnectionPool(host:XX)Max retries exceeded with url '<requests.package ...

  2. HTTPConnectionPool(host='xx.xx.xx.xx', port=xx): Max retries exceeded with url:(Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x0000015A25025EB8>...))

    HTTPConnectionPool(host='xx.xx.xx.xx', port=xx): Max retries exceeded with url:(Caused by ConnectTim ...

  3. 关于python3.6上传文件时报错：HTTPSConnectionPool(host='***.org', port=443): Max retries exceeded with url: /post (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAIL解决办法

    第一个报错: 最近在练习post请求中上传文件时遇到了一个奇葩事情,两台电脑上写了一模一样的代码,一个运行正常,另一个一片红. 最后了解了一下原因以及解决办法.先记录下关键代码: files = {& ...

  4. windows环境pip安装时一直报错Could not fetch URL https://pypi.org/simple/xrld/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url:

    最近项目不忙了~~有开始专研的python大业,上来想用pip安装一个第三方的库,就一直报错: Could not fetch URL https://pypi.org/simple/xrld/: T ...

  5. HTTPSConnectionPool(host='xxxxx', port=443): Max retries exceeded with url:xxxxxxxx (Caused by NewConnectionError('<urllib3.connect，Max retries exceeded with ，(Caused by NewConnectionError

    HTTPSConnectionPool(host='f6ws-sha8re-o88k.s3.ama66zaws.com', port=443): Max retries exceeded with u ...

  6. centos7中python3.6报错ModuleNotFoundError: No module named '_ssl' 或者 Max retries exceeded with url: / (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.",))

    如果在运行爬虫时报此错:requests.exceptions.SSLError: HTTPSConnectionPool(host='www.baidu.com', port=443): Max r ...

  7. Max retries exceeded with url

    78 Traceback (most recent call last):   File "thread072413.py", line 163, in <module> ...

  8. 解决Max retries exceeded with url的问题

    requests.exceptions.ConnectionError: HTTPSConnectionPool(host='itunes.apple.com', port=443): Max ret ...

  9. requests.exceptions.SSLError……Max retries exceeded with url错误求助！！！

    import requests head = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl ...

随机推荐

  1. HDFS的操作SHELL和API

    HDFS的shell操作和JavaAPI的使用: WEB WEB端口50090查看SecondaryNameNode信息.可以查看Hadoop的版本,NameNode的IP,Checkpoint等信息 ...

  2. for循环循环时间

    )) { Console.WriteLine(dt); } ("2011-5-5")       按需求定义 AddDays函数,   一天一天的增长

  3. DIV左、中、右三列布局的各类情况说明

    一.中间定宽.左.右侧百分比自适应: 1.HTML代码: <div id="left"> <div id="innerLeft"> &l ...

  4. redis永久化存储

    redis持久化存储 原因:redis是存放在内存中的,断电会导致数据丢失解决方法:把redis数据进行持久性存储,将其存储在磁盘中. 存储方式:1.RDBRDB中文名为快照/内存快照,Redis按照 ...

  5. 02-spark sql

    1.概念 Spark SQL是一个用来处理结构化数据的Spark组件. 优点:  ①SparkSQL是一个SQL解析引擎,将SQL解析成特殊的RDD(DataFrame),然后在Spark集群中运行 ...

  6. CentOS网络设置 couldn't resolve host 'mirrorlist.centos.org问题解决

    在VBox上安装完CentOS6.5之后,首次使用时yum命令安装软件时,经常遇到"couldn't resolve host 'mirrorlist.centos.org"这个问 ...

  7. WPF简单实用方法（持续更新）

    1:点击退出提示框 MessageBoxResult result = MessageBox.Show("你真的要退出吗?", "", MessageBoxBu ...

  8. ES6学习笔记<三> 生成器函数与yield

    为什么要把这个内容拿出来单独做一篇学习笔记? 生成器函数比较重要,相对不是很容易理解,单独做一篇笔记详细聊一聊生成器函数. 标题为什么是生成器函数与yield? 生成器函数类似其他服务器端语音中的接口 ...

  9. leetCode 557. Reverse Words in a String I

    Input: "Let's take LeetCode contest" Output: "s'teL ekat edoCteeL tsetnoc" 解:输入一 ...

  10. 遇到的问题&思考

    MYSQL需要批量修改表前缀: http://jingyan.baidu.com/article/11c17a2c742561f446e39d2f.html Select CONCAT( 'ALTER ...