爬虫多次访问同一个网站一段时间后会出现错误 HTTPConnectionPool(host:XX)Max retries exceeded with url '<requests.packages.urllib3.connection.HTTPConnection object at XXXX>: Failed to establish a new connection: [Errno 99] Cannot assign requested address'
是因为在每次数据传输前客户端要和服务器建立TCP连接,为节省传输消耗,默认为keep-alive,即连接一次,传输多次,然而在多次访问后不能结束并回到连接池中,导致不能产生新的连接
headers中的Connection默认为keep-alive,
将header中的Connection一项置为close

headers = {
    'Connection': 'close',
}
r = requests.get(url, headers=headers)

此时问题解决

HTTPConnectionPool(host:XX)Max retries exceeded with url的更多相关文章

  1. HTTPConnectionPool(host:XX)Max retries exceeded with url 解决方法

    爬虫多次访问同一个网站一段时间后会出现错误 HTTPConnectionPool(host:XX)Max retries exceeded with url '<requests.package ...

  2. HTTPConnectionPool(host='xx.xx.xx.xx', port=xx): Max retries exceeded with url:(Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x0000015A25025EB8>...))

    HTTPConnectionPool(host='xx.xx.xx.xx', port=xx): Max retries exceeded with url:(Caused by ConnectTim ...

  3. 关于python3.6上传文件时报错:HTTPSConnectionPool(host='***.org', port=443): Max retries exceeded with url: /post (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAIL解决办法

    第一个报错: 最近在练习post请求中上传文件时遇到了一个奇葩事情,两台电脑上写了一模一样的代码,一个运行正常,另一个一片红. 最后了解了一下原因以及解决办法.先记录下关键代码: files = {& ...

  4. windows环境pip安装时一直报错Could not fetch URL https://pypi.org/simple/xrld/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url:

    最近项目不忙了~~有开始专研的python大业,上来想用pip安装一个第三方的库,就一直报错: Could not fetch URL https://pypi.org/simple/xrld/: T ...

  5. HTTPSConnectionPool(host='xxxxx', port=443): Max retries exceeded with url:xxxxxxxx (Caused by NewConnectionError('<urllib3.connect,Max retries exceeded with ,(Caused by NewConnectionError

    HTTPSConnectionPool(host='f6ws-sha8re-o88k.s3.ama66zaws.com', port=443): Max retries exceeded with u ...

  6. centos7中python3.6报错ModuleNotFoundError: No module named '_ssl' 或者 Max retries exceeded with url: / (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.",))

    如果在运行爬虫时报此错:requests.exceptions.SSLError: HTTPSConnectionPool(host='www.baidu.com', port=443): Max r ...

  7. Max retries exceeded with url

    78 Traceback (most recent call last):   File "thread072413.py", line 163, in <module> ...

  8. 解决Max retries exceeded with url的问题

    requests.exceptions.ConnectionError: HTTPSConnectionPool(host='itunes.apple.com', port=443): Max ret ...

  9. requests.exceptions.SSLError……Max retries exceeded with url错误求助!!!

    import requests head = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl ...

随机推荐

  1. Distributed traceability with Spring Cloud: Sleuth and Zipkin

    I. Sleuth 0. Concept Trace A set of spans that form a call tree structure, forms the trace of the re ...

  2. Unable to find the wrapper ”https” - did youforget to enable it when you configured PHP?

    Unable to find the wrapper ”https” - did youforget to enable it when you configured PHP? 这是在Windows的 ...

  3. shell脚本选择LOG里面特定的行,生成新文件并rsync上传

    rsync.sh #!/bin/bash tool_path=$(cd `dirname $`; pwd) eval `cat ${tool_path}/conf.properties` rsync_ ...

  4. Guava实现 过滤文本,排序,转换内容,分组计数转换map 等等

    重要点 :看注释 从access.log中统计数据 对healthcheck.html的请求不计入统计 输出请求总量,以及GET和POST分别的总量 输出请求最频繁的10个接口及其次数,按次数降序 输 ...

  5. WPF Stake

    WPF中的StackPanel.WrapPanel.DockPanel 转:http://blog.sina.com.cn/s/blog_6c81891701017a34.html StackPane ...

  6. webkitAnimationEnd动画事件

    春节终于过完了,自己春节似乎过的有点大,过完春节之后,态度一直没有调整好,总有一股过节的情绪,没有完全进入学习和工作的状态来.继续调整当中…… 这两天项目中遇到一个小需求,十分类似于支付宝蚂蚁森林给小 ...

  7. django交互vue遇到的问题

    接受列表(数组): request.POST.get('array', '') # 结果得到数组的最后一个元素 request.POST.getlist('array', '') # 获取整个列表

  8. linux socket使用情况 ss -s ss -t -a | cat /proc/net/socketstat

    Linux系统中,查看SOCKET使用情况可以使用ss命令. 1.命令格式:ss [参数]ss [参数] [过滤]2.命令功能:ss(Socket Statistics的缩写)命令可以用来获取 soc ...

  9. java判断字符串中是否包含中文 过滤中文

    package com.test; import java.util.regex.Matcher; import java.util.regex.Pattern; public class Test ...

  10. 《算法》第六章部分程序 part 7

    ▶ 书中第六章部分程序,加上自己补充的代码,包括全局最小切分 Stoer-Wagner 算法,最小权值二分图匹配 ● 全局最小切分 Stoer-Wagner 算法 package package01; ...