按照官方的文档写的demo,只是多了个init函数,最终执行时提示没有_rules这个属性的错误日志如下:

 ......
File "C:\ProgramData\Anaconda3\lib\site-packages\scrapy\spiders\crawl.py", line 82, in _parse_response
for request_or_item in self._requests_to_follow(response):
File "C:\ProgramData\Anaconda3\lib\site-packages\scrapy\spiders\crawl.py", line 60, in _requests_to_follow
for n, rule in enumerate(self._rules):
AttributeError: 'TestSpider' object has no attribute '_rules'

出问题的spider代码如下:

# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from newtest.items import NewtestItem class TestSpider(CrawlSpider): def __init__(self,*args, **kwargs):
self.headers = {
'Content-Type':'application/x-www-form-urlencoded; charset=UTF-8',
'Accept-Encoding':'gzip, deflate',
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
} name = 'test'
allowed_domains = ['example.com']
start_urls = ['http://www.example.com'] rules = (
# Extract links matching 'category.php' (but not matching 'subsection.php')
# and follow links from them (since no callback means follow=True by default).
Rule(LinkExtractor(allow=('category\.php', ), deny=('subsection\.php', ))), # Extract links matching 'item.php' and parse them with the spider's method parse_item
Rule(LinkExtractor(allow=('item\.php', )), callback='parse_item'),
) def parse_item(self, response):
self.logger.info('Hi, this is an item page! %s', response.url)
item = scrapy.Item()
item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)')
item['name'] = response.xpath('//td[@id="item_name"]/text()').extract()
item['description'] = response.xpath('//td[@id="item_description"]/text()').extract()
return item

后来仔细看了下,跟官方不一样的就是自己重写了init初始化方法,而根据这个提示的日志,应该是覆盖了CrawlSpider的init方法但是没有调用父类的init导致_rules这个属性没有声明导致的。我们来看下CrawlSpider的源码:





所以如果我们的Spider是从CrawlSpider继承过来的,并且自己需要实现__init__ 方法的话,记住要调用父类的__init__方法保障能正常初始化crawlspider的属性。

修改后的代码如下:

第11行的super(TestSpider, self).__init__(*args, **kwargs) 是关键:

# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from newtest.items import NewtestItem class TestSpider(CrawlSpider): def __init__(self, *args, **kwargs):
super(TestSpider, self).__init__(*args, **kwargs) # 这里是关键
self.headers = {
'Content-Type':'application/x-www-form-urlencoded; charset=UTF-8',
'Accept-Encoding':'gzip, deflate',
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
} name = 'test'
allowed_domains = ['example.com']
start_urls = ['http://www.example.com'] rules = (
# Extract links matching 'category.php' (but not matching 'subsection.php')
# and follow links from them (since no callback means follow=True by default).
Rule(LinkExtractor(allow=('category\.php', ), deny=('subsection\.php', ))), # Extract links matching 'item.php' and parse them with the spider's method parse_item
Rule(LinkExtractor(allow=('item\.php', )), callback='parse_item'),
) def parse_item(self, response):
self.logger.info('Hi, this is an item page! %s', response.url)
item = scrapy.Item()
item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)')
item['name'] = response.xpath('//td[@id="item_name"]/text()').extract()
item['description'] = response.xpath('//td[@id="item_description"]/text()').extract()
return item

Scrapy Crawl 运行出错 AttributeError: 'xxxSpider' object has no attribute '_rules' 的问题解决的更多相关文章

  1. Python3.7 Scrapy crawl 运行出错解决方法

    安装的是Python3.7,装上依赖包和scrapy后运行爬虫命令出错 File "D:\Python37\lib\site-packages\scrapy\extensions\telne ...

  2. 提示AttributeError: 'module' object has no attribute 'HTTPSHandler'解决方法

    今天在新机器上安装sqlmap,运行提示AttributeError: 'module' object has no attribute 'HTTPSHandler' 网上找了找资料,发现一篇文章ht ...

  3. python 脚本运行时报错: AttributeError: 'module' object has no attribute ***

    最近在编写Python脚本过程中遇到一个问题比较奇怪:Python脚本完全正常没问题,但执行总报错"AttributeError: 'module' object has no attrib ...

  4. Django 运行报异常:AttributeError: 'str' object has no attribute 'get'

    Technorati Tags: Python,Django,Web 在使用django.contrib.auth用户机制进行用户的验证.登录.注销操作时,遇到这个异常. 首先是写了一个登录的视图,要 ...

  5. 在运行create_list.sh时候报错:AttributeError: 'module' object has no attribute 'LabelMap'

    Traceback (most recent call last):File "/opt/xuben-project/caffe/data/VOC0712/../../scripts/cre ...

  6. python3.x运行的坑:AttributeError: 'str' object has no attribute 'decode'

    1.Python3.x和Python2.X版本有一些区别,我遇到了两个问题如下: a.第一个报:mysqlclient 1.3版本不对: 解决办法:注释掉这行即可: b.第二个报:字符集的问题: 报错 ...

  7. AttributeError: 'dict' object has no attribute 'has_key'

    运行下面的代码: if (locals().has_key('data')): del data gc.collect() 出错: if (locals().has_key('data')): Att ...

  8. Python AttributeError: 'Module' object has no attribute 'STARTF_USESHOWINDOW'

    夫学须志也,才须学也,非学无以广才,非志无以成学.--诸葛亮 生活有度,自得慈铭 --杜锦阳 今天新来的同事安装环境遇到个莫名其妙的问题: AttributeError: 'Module' objec ...

  9. 解决opencv:AttributeError: 'NoneType' object has no attribute 'copy'

    情况一: 路径中有中文,更改即可 情况二:可以运行代码,在运行结束时显示 AttributeError: 'NoneType' object has no attribute 'copy' 因为如果是 ...

随机推荐

  1. spring cloud 路由网关zuul的高可用

    Zuul的高可用非常关键,因为外部请求到后端微服务的流量都会经过Zuul.故而在生产环境中,我们一般都需要部署高可用的Zuul以避免单点故障. 笔者分两种场景讨论Zuul的高可用. Zuul客户端也注 ...

  2. JS实现简单的观察者模式

    <!DOCTYPE html> <html> <head> <meta charset="UTF-8"> <title> ...

  3. JavaScript 高级程序设计 第二版

    function outputNumbers(count) {     (function () {         for (var i =0;i < count; i++) {        ...

  4. pycharm安装mysql驱动包

    新的环境配置pycharm的项目时,发现pycharm不能连接到mysql数据库.由于安了java环境但是还没配置相关的库,并且jetbrains家的IDE一般都是java写的,于是猜想可能是java ...

  5. 史上最简单的SpringCloud教程 | 第五篇: 路由网关(zuul)

    在微服务架构中,需要几个基础的服务治理组件,包括服务注册与发现.服务消费.负载均衡.断路器.智能路由.配置管理等,由这几个基础组件相互协作,共同组建了一个简单的微服务系统.一个简答的微服务系统如下图: ...

  6. 深入理解JSCore

    https://blog.csdn.net/MeituanTech/article/details/82108667

  7. NHibernate:no persister for 异常

    几种原因: 1.配置文件后缀名写错 mapping file 必须是.hbm.xml结尾 2.Web.config配置里面引用实体 <session-factory> .......... ...

  8. C#学习-类和结构

    类和结构体,对两者进行比较 语法上的区别在于,定义类要使用关键字class,而定义结构体则使用关键字struct; 结构体中不可对声明字段进行初始化,但类可以: 如果没有为类显式地定义构造函数,C#编 ...

  9. CentOS6.9安装Filebeat监控Nginx的访问日志发送到Kafka

    一.下载地址: 官方:https://www.elastic.co/cn/downloads/beats/filebeat 百度云盘:https://pan.baidu.com/s/1dvhqb0 二 ...

  10. Windows Server 2012 R2 设置 NTP 服务

    其实和以前的server版本配置没啥不一样 都是先改注册表: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config\ ...