Scrapy 源代码分析系列-4 scrapy.commands 子包

子包scrapy.commands定义了在命令scrapy中使用的子命令(subcommand): bench, check, crawl, deploy, edit, fetch,

genspider, list, parse, runspider, settings, shell, startproject, version, view。 所有的子命令模块都定义了一个继承自

类ScrapyCommand的子类Command。

首先来看一下子命令crawl, 该子命令用来启动spider。

1. crawl.py

关注的重点在方法run(self, args, opts):

 def run(self, args, opts):
if len(args) < 1:
raise UsageError()
elif len(args) > 1:
raise UsageError("running 'scrapy crawl' with more than one spider is no longer supported")
spname = args[0] crawler = self.crawler_process.create_crawler() # A
spider = crawler.spiders.create(spname, **opts.spargs) # B
crawler.crawl(spider) # C
self.crawler_process.start() # D

那么问题来啦,run接口方法是从哪里调用的呢? 让我们回到 Python.Scrapy.11-scrapy-source-code-analysis-part-1

中 "1.2 cmdline.py command.py" 关于"_run_print_help() "的说明。

A: 创建类Crawler对象crawler。在创建Crawler对象时, 同时将创建Crawler对象的实例属性spiders(SpiderManager)。如下所示:

 class Crawler(object):

     def __init__(self, settings):
self.configured = False
self.settings = settings
self.signals = SignalManager(self)
self.stats = load_object(settings['STATS_CLASS'])(self)
self._start_requests = lambda: ()
self._spider = None
# TODO: move SpiderManager to CrawlerProcess
spman_cls = load_object(self.settings['SPIDER_MANAGER_CLASS'])
self.spiders = spman_cls.from_crawler(self) # spiders 的类型是: SpiderManager

Crawler对象对应一个SpiderManager对象,而SpiderManager对象管理多个Spider。

B: 获取Sipder对象。

C: 为Spider对象安装Crawler对象。(为蜘蛛安装爬行器)

D: 类CrawlerProcess的start()方法如下:

     def start(self):
if self.start_crawling():
self.start_reactor() def start_crawling(self):
log.scrapy_info(self.settings)
return self._start_crawler() is not None def start_reactor(self):
if self.settings.getbool('DNSCACHE_ENABLED'):
reactor.installResolver(CachingThreadedResolver(reactor))
reactor.addSystemEventTrigger('before', 'shutdown', self.stop)
reactor.run(installSignalHandlers=False) # blocking call def _start_crawler(self):
if not self.crawlers or self.stopping:
return name, crawler = self.crawlers.popitem()
self._active_crawler = crawler
sflo = log.start_from_crawler(crawler)
crawler.configure()
crawler.install()
crawler.signals.connect(crawler.uninstall, signals.engine_stopped)
if sflo:
crawler.signals.connect(sflo.stop, signals.engine_stopped)
crawler.signals.connect(self._check_done, signals.engine_stopped)
crawler.start() # 调用类Crawler的start()方法
return name, crawler

类Crawler的start()方法如下:

     def start(self):
yield defer.maybeDeferred(self.configure)
if self._spider:
yield self.engine.open_spider(self._spider, self._start_requests()) # 和Engine建立了联系 (ExecutionEngine)
yield defer.maybeDeferred(self.engine.start)

关于类ExecutionEngine将在子包scrapy.core分析涉及。

2. startproject.py

3. subcommand是如何加载的

在cmdline.py的方法execute()中有如下几行代码:

     inproject = inside_project()
cmds = _get_commands_dict(settings, inproject)
cmdname = _pop_command_name(argv)

_get_commands_dict():

 def _get_commands_dict(settings, inproject):
cmds = _get_commands_from_module('scrapy.commands', inproject)
cmds.update(_get_commands_from_entry_points(inproject))
cmds_module = settings['COMMANDS_MODULE']
if cmds_module:
cmds.update(_get_commands_from_module(cmds_module, inproject))
return cmds

_get_commands_from_module():

 def _get_commands_from_module(module, inproject):
d = {}
for cmd in _iter_command_classes(module):
if inproject or not cmd.requires_project:
cmdname = cmd.__module__.split('.')[-1]
d[cmdname] = cmd()
return d

To Be Continued

接下来解析settings相关的逻辑。Python.Scrapy.15-scrapy-source-code-analysis-part-5

Python.Scrapy.14-scrapy-source-code-analysis-part-4的更多相关文章

  1. Memcached source code analysis (threading model)--reference

    Look under the start memcahced threading process memcached multi-threaded mainly by instantiating mu ...

  2. Golang Template source code analysis(Parse)

    This blog was written at go 1.3.1 version. We know that we use template thought by followed way: fun ...

  3. Memcached source code analysis -- Analysis of change of state--reference

    This article mainly introduces the process of Memcached, libevent structure of the main thread and w ...

  4. Apache Commons Pool2 源码分析 | Apache Commons Pool2 Source Code Analysis

    Apache Commons Pool实现了对象池的功能.定义了对象的生成.销毁.激活.钝化等操作及其状态转换,并提供几个默认的对象池实现.在讲述其实现原理前,先提一下其中有几个重要的对象: Pool ...

  5. Redis source code analysis

    http://zhangtielei.com/posts/blog-redis-dict.html http://zhangtielei.com/assets/photos_redis/redis_d ...

  6. linux kernel & source code analysis& hacking

    https://kernelnewbies.org/ http://www.tldp.org/LDP/lki/index.html https://kernelnewbies.org/ML https ...

  7. 2018.6.21 HOLTEK HT49R70A-1 Source Code analysis

    Cange note: “Reading TMR1H will latch the contents of TMR1H and TMR1L counter to the destination”? F ...

  8. The Ultimate List of Open Source Static Code Analysis Security Tools

    https://www.checkmarx.com/2014/11/13/the-ultimate-list-of-open-source-static-code-analysis-security- ...

  9. Top 40 Static Code Analysis Tools

    https://www.softwaretestinghelp.com/tools/top-40-static-code-analysis-tools/ In this article, I have ...

  10. Source Code Reading for Vue 3: How does `hasChanged` work?

    Hey, guys! The next generation of Vue has released already. There are not only the brand new composi ...

随机推荐

  1. SpringMVC处理静态资源

    若将DispatcheServlet请求映射设置为/,则SpringMvc将捕获WEB容器的所有请求,包括静态资源的请求,SpringMvc会将它们当成一个普通的请求处理,那么将会出现因找不到对应的处 ...

  2. python---hashlib

    简介 用于加密相关的操作,代替了md5模块和sha模块,主要提供SHA1,SHA224,SHA256,SHA384,SHA512,MD5算法. 在python3中已经废弃了md5和sha模块,简单说明 ...

  3. Linxu IO测试软件

    fio 安装 apt-get install fio fdisk -l Device Boot Start End Blocks Id System/dev/sda1 * 2048 968390655 ...

  4. SparkContext源码阅读

    SparkContext是spark的入口,通过它来连接集群.创建RDD.广播变量等等. class SparkContext(config: SparkConf) extends Logging w ...

  5. Corel Painter 15在Surface Pro 4下开启笔触压力感应

    之前一直是用Wacom的板子,所以只需要下载Wacom板子相应的驱动安装即可就能在PS和Corel Painter中开启压力感应来调节笔触出线的粗细.Surface Pro 4的笔是支持压力感应的,但 ...

  6. java基础要点

    Java语言是现在比较常用的编程语言,因为Java语言可以编写桌用程序,也可以编写web程序,还能编写嵌入式程序.这是由于相比脚本语言,Java 的运行速度要快.而对于底层语言,Java与平台无关,可 ...

  7. error BK1506 : cannot open file '.\Debug\????????.sbr': No such file or dire

    http://blog.csdn.net/shuilan0066/article/details/8738035 分类:            调试错误信息2013-03-29 19:08492人阅读 ...

  8. DataTable排序(转)

    DataTable 排序   DataRow[] rows = dataTable1.Select("", "ord asc");   DataTable t ...

  9. PHP文件读写操作之文件写入代码

    在PHP网站开发中,存储数据通常有两种方式,一种以文本文件方式存储,比如txt文件,一种是以数据库方式存储,比如Mysql,相对于数据库存储,文件存储并没有什么优势,但是文件读写操作在基本的PHP开发 ...

  10. 安卓:assets目录下的文本文件(不受R文件节制)

    try { InputStream in = getAssets().open("testAsset.txt"); byte[] buffer = new byte[1024]; ...