INFO: Ignoring response <403 https://movie.douban.com/top250>: HTTP status code is not handled or not allowed
爬取豆瓣电影top250,出现以下报错:
2018-08-11 22:02:16 [scrapy.core.engine] INFO: Spider opened
2018-08-11 22:02:16 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-08-11 22:02:16 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-08-11 22:02:17 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://movie.douban.com/robots.txt> (referer: None)
2018-08-11 22:02:17 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://movie.douban.com/top250> (referer: None)
2018-08-11 22:02:17 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://movie.douban.com/top250>: HTTP status code is not handled or not
allowed
2018-08-11 22:02:17 [scrapy.core.engine] INFO: Closing spider (finished)
防止反爬机制,伪装user_agent
【1】打开豆瓣top250 : https://movie.douban.com/top250
【2】F12 打开控制台->刷新页面 ->Network->请求头部找到 User-Agent
在scrapy项目中找到settings.py的 USER_AGENT = ' ' (把注释去掉,加以下内容)
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3493.3 Safari/537.36'
重新执行即可
scrapy crawl douban_spider
INFO: Ignoring response <403 https://movie.douban.com/top250>: HTTP status code is not handled or not allowed的更多相关文章
- python scrapy 报错 DEBUG: Ignoring response 403
DEBUG: Ignoring response <403 http://movie.douban.com/top250>: HTTP status code is not handled ...
- docker 1.12.3版本搭建私有仓库,上传镜像报错:server gave HTTP response to HTTPS client”
系统环境:centos7 docker版本: 1.12.3(注意版本,可能存在不同版本设置不同的情况) docker registry版本:2.4.1 问题: 成功安装docker registry, ...
- http: server gave HTTP response to HTTPS client & Get https://192.168.2.119/v2/: dial tcp 192.168.2.119:443: getsockopt: connection refused
http: server gave HTTP response to HTTPS client 出现这问题的原因是:Docker自从1.3.X之后docker registry交互默认使用的是HTTP ...
- [原]Docker-issue(2) http: server gave HTTP response to HTTPS client
系统环境 查看 文章末尾 附录 问题点:新建local registry后,push新的image到local registry 未能成功,并报错误: The push refers to repo ...
- docker local registry server gave HTTP response to HTTPS client
server gave HTTP response to HTTPS client报错是在insecure_registry中加入了http前缀,如果本地registry不是https的 就不要加任何 ...
- docker registry push错误“server gave HTTP response to HTTPS client”
系统环境:centos7 docker版本: 1.12.3(注意版本,可能存在不同版本设置不同的情况) docker registry版本:2.4.1 问题: 成功安装docker registry, ...
- (七)VMware Harbor 问题:Get https://192.168.3.135:8088/v2/: http:server gave HTTP response to HTTPS client
(一)问题描述 登陆时,报错 docker Get https://192.168.3.135:8088/v2/: http:server gave HTTP response to HTTPS cl ...
- 【解决】http: server gave HTTP response to HTTPS client
[问题]上传镜像到私有仓库时报错 $ docker push xxx.xxx.xxx.xxx:5000/java-8 The push refers to repository [xxx.xxx.xx ...
- springMVC or response redirect https
<bean class="org.springframework.web.servlet.view.InternalResourceViewResolver"> < ...
随机推荐
- 陌上花开(bzoj 3262)
Description 有n朵花,每朵花有三个属性:花形(s).颜色(c).气味(m),又三个整数表示.现要对每朵花评级,一朵花的级别是它拥有的美丽能超过的花的数量.定义一朵花A比另一朵花B要美丽,当 ...
- 洛谷 [P3205] 合唱队
区间DP 手动模拟一下,我们发现本题就是一个左右加数的区间DP #include <iostream> #include <cstdio> #include <cstri ...
- 标准C程序设计七---20
Linux应用 编程深入 语言编程 标准C程序设计七---经典C11程序设计 以下内容为阅读: <标准C程序设计>(第7版) 作者 ...
- Yii 安装学习
(1)打开yii官方网站: http://www.yiichina.com (2)点击下载,跳转到下载页面: (3)找到从归档文件安装,新手学习,使用[ Yii2的基本应用程序模板]: (4)下载解压 ...
- 快速掌握RabbitMQ(四)——两种消费模式和QOS的C#实现
本篇介绍一下RabbitMQ中的消费模式,在前边的所有栗子中我们采用的消费者都是EventingBasicConsumer,其实RabbitMQ中还有其他两种消费模式:BasicGet和QueueBa ...
- android 环境变量配置,以及sdcard配置
第一步,打开环境变量配置窗口.右击计算机,属性-高级系统设置-环境变量. 第二步,添加android系统环境变量.在系统变量下,选择path,点击编辑,然后在最前面输入android sdk开发工具, ...
- 高级算法设计讲义 Lecture Notes for Advanced Algorithm Design
(Last modification: 2012-12-17) Textbooks: (1) David Williamson, David Shmoys. The Design of Approxi ...
- Cocos2d-x JSB 自己主动绑定bindings
Javascript Binding (简称JSB) 自己主动绑定教程. Cocos2d-x JSB 自己主动绑定bindings-generator (以下简称B-G) 使用心得 假设想弄清深入原理 ...
- Java中的反射机制,利用反射访问私有
利用反射,首先是Class对象的获取,之后是Method和Field对象的获取. 以Method为例,从文档中可以看到: getMethod()方法返回的是public的Method对象, 而getD ...
- 转: 在CentOS 6.X 上面安装 Python 2.7.X
转:https://ruiaylin.github.io/2014/12/12/python%20update/ 评注: yum -y update //这个更新太坑了,1120更新包...想死的心都 ...