流程框架

安装Scrapy:

　　(1)在pycharm里直接就可以进行安装Scrapy

　　  

　　  (2)若在conda里安装scrapy，需要进入cmd里输入指令conda install scrapy

 Scrapy框架的搭建

1、先创建scrapy工程

scrapy startproject quotetutorial

在pycharm里的Terminal里输入上面的指令代码创建一个名为quotetutorial的工程，创建后会出现下面的代码，就代表scrapy工程创建成功。

New Scrapy project 'quotetutorial', using template directory 'G:\\Anaconda3-5.0.1\\install\\lib\\site-packages\\scrapy\\templates\\project', created in:

    C:\Users\Administrator\Desktop\爬虫程序\Scrapy\quotetutorial

2、进入工程文件夹里

cd quotetutorial

3、在工程文件夹里创建scrapy.cfg文件

scrapy.cfg

4、利用genspider创建一个Spider

scrapy genspider quotes quotes.toscrape.com

若出现下面的提示说明创建成功

Created spider 'quotes' using template 'basic' in module:

  quotetutorial.spiders.quotes

5、利用Pycharm打开scrapy工程，若与下面图片一样，则说明工程创建成功

此时quotes.py的代码是：

# -*- coding: utf-8 -*-

import scrapy

class QuotesSpider(scrapy.Spider):

    name = 'quotes'

    allowed_domains = ['quotes.toscrape.com']

    start_urls = ['http://quotes.toscrape.com/']

    def parse(self, response):

        pass

6、要运行Spider，则需要在pycharm里的Terminal输入下面代码指令

scrapy crawl quotes

此时如出现下面的代码，则代表Spider运行成功

2018-10-09 17:32:49 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: quotetutorial)

2018-10-09 17:32:49 [scrapy.utils.log] INFO: Versions: lxml 4.1.0.0, libxml2 2.9.4, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python 3.6.3 |A

naconda custom (64-bit)| (default, Oct 15 2017, 03:27:45) [MSC v.1900 64 bit (AMD64)], pyOpenSSL 17.2.0 (OpenSSL 1.0.2p  14 Aug 2018), cryptography 2.0.3, Platf

orm Windows-7-6.1.7601-SP1

2018-10-09 17:32:49 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'quotetutorial', 'NEWSPIDER_MODULE': 'quotetutorial.spiders', 'ROBOTSTXT_OBEY': Tru

e, 'SPIDER_MODULES': ['quotetutorial.spiders']}

2018-10-09 17:32:50 [scrapy.middleware] INFO: Enabled extensions:

['scrapy.extensions.corestats.CoreStats',

 'scrapy.extensions.telnet.TelnetConsole',

 'scrapy.extensions.logstats.LogStats']

2018-10-09 17:32:50 [scrapy.middleware] INFO: Enabled downloader middlewares:

['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',

 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',

 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',

 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',

 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',

 'scrapy.downloadermiddlewares.retry.RetryMiddleware',

 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',

 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',

 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',

 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',

 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',

 'scrapy.downloadermiddlewares.stats.DownloaderStats']

2018-10-09 17:32:50 [scrapy.middleware] INFO: Enabled spider middlewares:

['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',

 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',

 'scrapy.spidermiddlewares.referer.RefererMiddleware',

 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',

 'scrapy.spidermiddlewares.depth.DepthMiddleware']

2018-10-09 17:32:50 [scrapy.middleware] INFO: Enabled item pipelines:

[]

2018-10-09 17:32:50 [scrapy.core.engine] INFO: Spider opened

2018-10-09 17:32:51 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

2018-10-09 17:32:51 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023

2018-10-09 17:32:52 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://quotes.toscrape.com/robots.txt> (referer: None)

2018-10-09 17:32:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/> (referer: None)

2018-10-09 17:32:53 [scrapy.core.engine] INFO: Closing spider (finished)

2018-10-09 17:32:53 [scrapy.statscollectors] INFO: Dumping Scrapy stats:

{'downloader/request_bytes': 446,

 'downloader/request_count': 2,

 'downloader/request_method_count/GET': 2,

 'downloader/response_bytes': 2701,

 'downloader/response_count': 2,

 'downloader/response_status_count/200': 1,

 'downloader/response_status_count/404': 1,

 'finish_reason': 'finished',

 'finish_time': datetime.datetime(2018, 10, 9, 9, 32, 53, 365239),

 'log_count/DEBUG': 3,

 'log_count/INFO': 7,

 'response_received_count': 2,

 'scheduler/dequeued': 1,

 'scheduler/dequeued/memory': 1,

 'scheduler/enqueued': 1,

 'scheduler/enqueued/memory': 1,

 'start_time': datetime.datetime(2018, 10, 9, 9, 32, 51, 43106)}

2018-10-09 17:32:53 [scrapy.core.engine] INFO: Spider closed (finished)

7、Scrapy自带的shell工具，其指令为

scrapy shell quotes.toscrape.com

会出现下面的提示

2018-10-09 19:48:46 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: quotetutorial)

2018-10-09 19:48:46 [scrapy.utils.log] INFO: Versions: lxml 4.1.0.0, libxml2 2.9.4, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python 3.6.3 |A

naconda custom (64-bit)| (default, Oct 15 2017, 03:27:45) [MSC v.1900 64 bit (AMD64)], pyOpenSSL 17.2.0 (OpenSSL 1.0.2p  14 Aug 2018), cryptography 2.0.3, Platf

orm Windows-7-6.1.7601-SP1

2018-10-09 19:48:46 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'quotetutorial', 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter', 'LOGSTATS

_INTERVAL': 0, 'NEWSPIDER_MODULE': 'quotetutorial.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['quotetutorial.spiders']}

2018-10-09 19:48:46 [scrapy.middleware] INFO: Enabled extensions:

['scrapy.extensions.corestats.CoreStats',

 'scrapy.extensions.telnet.TelnetConsole']

2018-10-09 19:48:46 [scrapy.middleware] INFO: Enabled downloader middlewares:

['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',

 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',

 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',

 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',

 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',

 'scrapy.downloadermiddlewares.retry.RetryMiddleware',

 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',

 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',

 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',

 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',

 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',

 'scrapy.downloadermiddlewares.stats.DownloaderStats']

2018-10-09 19:48:46 [scrapy.middleware] INFO: Enabled spider middlewares:

['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',

 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',

 'scrapy.spidermiddlewares.referer.RefererMiddleware',

 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',

 'scrapy.spidermiddlewares.depth.DepthMiddleware']

2018-10-09 19:48:46 [scrapy.middleware] INFO: Enabled item pipelines:

[]

2018-10-09 19:48:46 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023

2018-10-09 19:48:46 [scrapy.core.engine] INFO: Spider opened

2018-10-09 19:48:51 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://quotes.toscrape.com/robots.txt> (referer: None)

2018-10-09 19:48:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com> (referer: None)

2018-10-09 19:49:02 [traitlets] DEBUG: Using default logger

2018-10-09 19:49:02 [traitlets] DEBUG: Using default logger

[s] Available Scrapy objects:

[s]   scrapy     scrapy module (contains scrapy.Request, scrapy.Selector, etc)

[s]   crawler    <scrapy.crawler.Crawler object at 0x00000000022BBB38>

[s]   item       {}

[s]   request    <GET http://quotes.toscrape.com>

[s]   response   <200 http://quotes.toscrape.com>

[s]   settings   <scrapy.settings.Settings object at 0x0000000004F43C50>

[s]   spider     <QuotesSpider 'quotes' at 0x51c98d0>

[s] Useful shortcuts:

[s]   fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)

[s]   fetch(req)                  Fetch a scrapy.Request and update local objects

[s]   shelp()           Shell help (print this help)

[s]   view(response)    View response in a browser

In [1]:

在shell里输入

In [2]: response

系统会输出

Out[2]: <200 http://quotes.toscrape.com>

在输入

In [3]: quotes = response.css(".quote")

In [4]: quotes

系统会输出

Out[4]:

[<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype=

"h'>,

 <Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype=

"h'>,

 <Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype=

"h'>,

 <Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype=

"h'>,

 <Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype=

"h'>,

 <Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype=

"h'>,

 <Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype=

"h'>,

 <Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype=

"h'>,

 <Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype=

"h'>,

 <Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype=

"h'>]

再输入

In [5]: quotes[0]

Out[5]: <Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope it

emtype="h'>

In [6]: quotes[0].css(".text::text")

Out[6]: [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/text()" data='“The world as we have c

reated it is a pr'>]

In [7]: quotes[0].css(".text")

Out[7]: [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]" data='<span class="text" itemprop="te

xt">“The '>]

In [8]: quotes[0].css(".text").extract()

Out[8]: ['<span class="text" itemprop="text">“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.

”</span>']

In [9]: quotes[0].css(".text::text").extract()

Out[9]: ['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”']

In [10]: quotes[0].css(".text").extract_first()

Out[10]: '<span class="text" itemprop="text">“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.

”</span>'

In [11]: quotes[0].css(".text::text").extract_first()

Out[11]: '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'

Scrapy实战

　　--利用Scrapy爬取quotes.toscrape.com中的名言警句

在Scrapy里的quotes.py代码如下

# -*- coding: utf-8 -*-

import scrapy

class QuoteItem(scrapy.Item):

    # define the fields for your item here like:

    # name = scrapy.Field()

    text = scrapy.Field()

    author = scrapy.Field()

    tags = scrapy.Field()

class QuotesSpider(scrapy.Spider):

    name = 'quotes'

    allowed_domains = ['quotes.toscrape.com']

    start_urls = ['http://quotes.toscrape.com/']

    def parse(self, response):

        quotes = response.css('.quote') # 筛选class = "quote" 的div

        for quote in quotes:

            item = QuoteItem()

            text = quote.css('.text::text').extract_first # 进一步筛选，'.text::text'输出text文本的内容,extract_first找第一个

            author = quote.css('.author::text').extract_first  # 进一步筛选，'.text::text'输出text文本的内容

            tags = quote.css(".tags .tag::text").extract # extract把所有结果全部找出来

            item['text'] = text

            item['author'] = author

            item['tags'] = tags

            yield item

            next = response.css('.pager .next a::attr(href)').extract_first() # 查找下一页

            url = response.urljoin(next)

            yield scrapy.Request(url = url,callback=self.parse)

在Terminal里敲入指令

C:\Users\Administrator\Desktop\爬虫程序\Scrapy\quotetutorial>scrapy crawl quotes

会出现下面的代码

2018-10-09 20:26:30 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: quotetutorial)

2018-10-09 20:26:30 [scrapy.utils.log] INFO: Versions: lxml 4.1.0.0, libxml2 2.9.4, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python 3.6.3 |A

naconda custom (64-bit)| (default, Oct 15 2017, 03:27:45) [MSC v.1900 64 bit (AMD64)], pyOpenSSL 17.2.0 (OpenSSL 1.0.2p  14 Aug 2018), cryptography 2.0.3, Platf

orm Windows-7-6.1.7601-SP1

2018-10-09 20:26:30 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'quotetutorial', 'NEWSPIDER_MODULE': 'quotetutorial.spiders', 'ROBOTSTXT_OBEY': Tru

e, 'SPIDER_MODULES': ['quotetutorial.spiders']}

2018-10-09 20:26:30 [scrapy.middleware] INFO: Enabled extensions:

['scrapy.extensions.corestats.CoreStats',

 'scrapy.extensions.telnet.TelnetConsole',

 'scrapy.extensions.logstats.LogStats']

2018-10-09 20:26:30 [scrapy.middleware] INFO: Enabled downloader middlewares:

['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',

 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',

 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',

 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',

 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',

 'scrapy.downloadermiddlewares.retry.RetryMiddleware',

 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',

 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',

 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',

 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',

 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',

 'scrapy.downloadermiddlewares.stats.DownloaderStats']

2018-10-09 20:26:30 [scrapy.middleware] INFO: Enabled spider middlewares:

['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',

 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',

 'scrapy.spidermiddlewares.referer.RefererMiddleware',

 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',

 'scrapy.spidermiddlewares.depth.DepthMiddleware']

2018-10-09 20:26:30 [scrapy.middleware] INFO: Enabled item pipelines:

[]

2018-10-09 20:26:30 [scrapy.core.engine] INFO: Spider opened

2018-10-09 20:26:30 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

2018-10-09 20:26:30 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023

2018-10-09 20:26:33 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://quotes.toscrape.com/robots.txt> (referer: None)

2018-10-09 20:26:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/> (referer: None)

2018-10-09 20:26:34 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data='Albert Einstein'>]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='change'>, <Selector xpath="descendant-or-self::*[

@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '),

 ' tag ')]/text()" data='deep-thoughts'>, <Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/desce

ndant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='thinking'>, <Selector xpath="descendant-or-self::*[@cl

ass and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), '

tag ')]/text()" data='world'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data='“The world as we have created it is a pr'>]>}

2018-10-09 20:26:34 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data='J.K. Rowling'>]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='abilities'>, <Selector xpath="descendant-or-self:

:*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), '

'), ' tag ')]/text()" data='choices'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data='“It is our choices, Harry, that show wha'>]>}

2018-10-09 20:26:34 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET http://quotes.toscrape.com/page/2/> - no more duplicates will be shown (see DUP

EFILTER_DEBUG to show all duplicates)

2018-10-09 20:26:34 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data='Albert Einstein'>]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='inspirational'>, <Selector xpath="descendant-or-s

elf::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-or-self::*/*<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-o

r-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='reading'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data='“I declare after all there is no enjoyme'>]>}

2018-10-09 20:26:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/9/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data='Jane Austen'>]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='elizabeth-bennet'>, <Selector xpath="descendant-o

r-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@cla

ss), ' '), ' tag ')]/text()" data='jane-austen'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data='“There are few people whom I really love'>]>}

2018-10-09 20:26:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/9/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data='C.S. Lewis'>]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='age'>, <Selector xpath="descendant-or-self::*[@cl

ass and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), '

tag ')]/text()" data='fairytales'>, <Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-

or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='growing-up'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data='“Some day you will be old enough to star'>]>}

2018-10-09 20:26:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/9/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data='C.S. Lewis'>]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='god'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data='“We are not necessarily doubting that Go'>]>}

2018-10-09 20:26:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/9/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data='Mark Twain'>]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='death'>, <Selector xpath="descendant-or-self::*[@

class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '),

' tag ')]/text()" data='life'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data='“The fear of death follows from the fear'>]>}

2018-10-09 20:26:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/9/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data='Mark Twain'>]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='misattributed-mark-twain'>, <Selector xpath="desc

endant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-or-self::*/*[@class and contains(concat(' ', normalize-sp

ace(@class), ' '), ' tag ')]/text()" data='truth'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data='“A lie can travel half way around the wo'>]>}

2018-10-09 20:26:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/9/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data='C.S. Lewis'>]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='christianity'>, <Selector xpath="descendant-or-se

lf::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class),

 ' '), ' tag ')]/text()" data='faith'>, <Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descend

ant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='religion'>, <Selector xpath="descendant-or-self::*[@clas

s and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' ta

g ')]/text()" data='sun'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data='“I believe in Christianity as I believe '>]>}

2018-10-09 20:26:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/10/> (referer: http://quotes.toscrape.com/page/9/)

2018-10-09 20:26:39 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data='J.K. Rowling'>]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='truth'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data='“The truth." Dumbledore sighed. "It is a'>]>}

2018-10-09 20:26:39 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data='Jimi Hendrix'>]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='death'>, <Selector xpath="descendant-or-self::*[@

class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '),

' tag ')]/text()" data='life'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data="“I'm the one that's got to die when it's">]>}

2018-10-09 20:26:39 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data='J.M. Barrie'>]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='adventure'>, <Selector xpath="descendant-or-self:

:*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), '

'), ' tag ')]/text()" data='love'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data='“To die will be an awfully big adventure'>]>}

2018-10-09 20:26:39 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data='E.E. Cummings'>]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='courage'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data='“It takes courage to grow up and become '>]>}

2018-10-09 20:26:39 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data='Khaled Hosseini'>]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='life'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data='“But better to get hurt by the truth tha'>]>}

2018-10-09 20:26:39 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data='Harper Lee'>]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='better-life-empathy'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data='“You never really understand a person un'>]>}

2018-10-09 20:26:39 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data="Madeleine L'Engle">]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='books'>, <Selector xpath="descendant-or-self::*[@

class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '),

' tag ')]/text()" data='children'>, <Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-

or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='difficult'>, <Selector xpath="descendant-or-self::*[@class a

nd contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag '

)]/text()" data='grown-ups'>, <Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-or-sel

f::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='write'>, <Selector xpath="descendant-or-self::*[@class and contain

s(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()"

 data='writers'>, <Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-or-self::*/*[@clas

s and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='writing'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data='“You have to write the book that wants t'>]>}

2018-10-09 20:26:39 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data='Mark Twain'>]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='truth'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data='“Never tell the truth to people who are '>]>}

2018-10-09 20:26:39 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data='Dr. Seuss'>]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='inspirational'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data="“A person's a person, no matter how smal">]>}

2018-10-09 20:26:39 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>

{'author': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' author ')]

/text()" data='George R.R. Martin'>]>,

 'tags': <bound method SelectorList.getall of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/

descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '), ' tag ')]/text()" data='books'>, <Selector xpath="descendant-or-self::*[@

class and contains(concat(' ', normalize-space(@class), ' '), ' tags ')]/descendant-or-self::*/*[@class and contains(concat(' ', normalize-space(@class), ' '),

' tag ')]/text()" data='mind'>]>,

 'text': <bound method SelectorList.get of [<Selector xpath="descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' text ')]/tex

t()" data='“... a mind needs books as a sword needs'>]>}

2018-10-09 20:26:39 [scrapy.core.engine] INFO: Closing spider (finished)

2018-10-09 20:26:39 [scrapy.statscollectors] INFO: Dumping Scrapy stats:

{'downloader/request_bytes': 2870,

 'downloader/request_count': 11,

 'downloader/request_method_count/GET': 11,

 'downloader/response_bytes': 24812,

 'downloader/response_count': 11,

 'downloader/response_status_count/200': 10,

 'downloader/response_status_count/404': 1,

 'dupefilter/filtered': 91,

 'finish_reason': 'finished',

 'finish_time': datetime.datetime(2018, 10, 9, 12, 26, 39, 346571),

 'item_scraped_count': 100,

 'log_count/DEBUG': 113,

 'log_count/INFO': 7,

 'request_depth_max': 10,

 'response_received_count': 11,

 'scheduler/dequeued': 10,

 'scheduler/dequeued/memory': 10,

 'scheduler/enqueued': 10,

 'scheduler/enqueued/memory': 10,

 'start_time': datetime.datetime(2018, 10, 9, 12, 26, 30, 499065)}

2018-10-09 20:26:39 [scrapy.core.engine] INFO: Spider closed (finished)

若想对上面输出的信息进行保存的话，则需要输入指令

C:\Users\Administrator\Desktop\爬虫程序\Scrapy\quotetutorial>scrapy crawl quotes -o quotes.json

执行成功后，在工程里会出现quotes.json的文件，且quotes.json的内容为

[

{"text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d", "author": "Albert Einstein", "tags": ["change", "deep-thoughts", "thinking", "world"]},

{"text": "\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d", "author": "J.K. Rowling", "tags": ["abilities", "choices"]},

{"text": "\u201cThere are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.\u201d", "author": "Albert Einstein", "tags": ["inspirational", "life", "live", "miracle", "miracles"]},

{"text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d", "author": "Jane Austen", "tags": ["aliteracy", "books", "classic", "humor"]},

{"text": "\u201cImperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.\u201d", "author": "Marilyn Monroe", "tags": ["be-yourself", "inspirational"]},

{"text": "\u201cTry not to become a man of success. Rather become a man of value.\u201d", "author": "Albert Einstein", "tags": ["adulthood", "success", "value"]},

{"text": "\u201cIt is better to be hated for what you are than to be loved for what you are not.\u201d", "author": "Andr\u00e9 Gide", "tags": ["life", "love"]},

{"text": "\u201cI have not failed. I've just found 10,000 ways that won't work.\u201d", "author": "Thomas A. Edison", "tags": ["edison", "failure", "inspirational", "paraphrased"]},

{"text": "\u201cA woman is like a tea bag; you never know how strong it is until it's in hot water.\u201d", "author": "Eleanor Roosevelt", "tags": ["misattributed-eleanor-roosevelt"]},

{"text": "\u201cA day without sunshine is like, you know, night.\u201d", "author": "Steve Martin", "tags": ["humor", "obvious", "simile"]},

{"text": "\u201cThis life is what you make it. No matter what, you're going to mess up sometimes, it's a universal truth. But the good part is you get to decide how you're going to mess it up. Girls will be your friends - they'll act like it anyway. But just remember, some come, some go. The ones that stay with you through everything - they're your true best friends. Don't let go of them. Also remember, sisters make the best friends in the world. As for lovers, well, they'll come and go too. And baby, I hate to say it, most of them - actually pretty much all of them are going to break your heart, but you can't give up because if you give up, you'll never find your soulmate. You'll never find that half who makes you whole and that goes for everything. Just because you fail once, doesn't mean you're gonna fail at everything. Keep trying, hold on, and always, always, always believe in yourself, because if you don't, then who will, sweetie? So keep your head high, keep your chin up, and most importantly, keep smiling, because life's a beautiful thing and there's so much to smile about.\u201d", "author": "Marilyn Monroe", "tags": ["friends", "heartbreak", "inspirational", "life", "love", "sisters"]},

{"text": "\u201cIt takes a great deal of bravery to stand up to our enemies, but just as much to stand up to our friends.\u201d", "author": "J.K. Rowling", "tags": ["courage", "friends"]},

{"text": "\u201cIf you can't explain it to a six year old, you don't understand it yourself.\u201d", "author": "Albert Einstein", "tags": ["simplicity", "understand"]},

{"text": "\u201cYou may not be her first, her last, or her only. She loved before she may love again. But if she loves you now, what else matters? She's not perfect\u2014you aren't either, and the two of you may never be perfect together but if she can make you laugh, cause you to think twice, and admit to being human and making mistakes, hold onto her and give her the most you can. She may not be thinking about you every second of the day, but she will give you a part of her that she knows you can break\u2014her heart. So don't hurt her, don't change her, don't analyze and don't expect more than she can give. Smile when she makes you happy, let her know when she makes you mad, and miss her when she's not there.\u201d", "author": "Bob Marley", "tags": ["love"]},

{"text": "\u201cI like nonsense, it wakes up the brain cells. Fantasy is a necessary ingredient in living.\u201d", "author": "Dr. Seuss", "tags": ["fantasy"]},

{"text": "\u201cI may not have gone where I intended to go, but I think I have ended up where I needed to be.\u201d", "author": "Douglas Adams", "tags": ["life", "navigation"]},

{"text": "\u201cThe opposite of love is not hate, it's indifference. The opposite of art is not ugliness, it's indifference. The opposite of faith is not heresy, it's indifference. And the opposite of life is not death, it's indifference.\u201d", "author": "Elie Wiesel", "tags": ["activism", "apathy", "hate", "indifference", "inspirational", "love", "opposite", "philosophy"]},

{"text": "\u201cIt is not a lack of love, but a lack of friendship that makes unhappy marriages.\u201d", "author": "Friedrich Nietzsche", "tags": ["friendship", "lack-of-friendship", "lack-of-love", "love", "marriage", "unhappy-marriage"]},

{"text": "\u201cGood friends, good books, and a sleepy conscience: this is the ideal life.\u201d", "author": "Mark Twain", "tags": ["books", "contentment", "friends", "friendship", "life"]},

{"text": "\u201cLife is what happens to us while we are making other plans.\u201d", "author": "Allen Saunders", "tags": ["fate", "life", "misattributed-john-lennon", "planning", "plans"]},

{"text": "\u201cI love you without knowing how, or when, or from where. I love you simply, without problems or pride: I love you in this way because I do not know any other way of loving but this, in which there is no I or you, so intimate that your hand upon my chest is my hand, so intimate that when I fall asleep your eyes close.\u201d", "author": "Pablo Neruda", "tags": ["love", "poetry"]},

{"text": "\u201cFor every minute you are angry you lose sixty seconds of happiness.\u201d", "author": "Ralph Waldo Emerson", "tags": ["happiness"]},

{"text": "\u201cIf you judge people, you have no time to love them.\u201d", "author": "Mother Teresa", "tags": ["attributed-no-source"]},

{"text": "\u201cAnyone who thinks sitting in church can make you a Christian must also think that sitting in a garage can make you a car.\u201d", "author": "Garrison Keillor", "tags": ["humor", "religion"]},

{"text": "\u201cBeauty is in the eye of the beholder and it may be necessary from time to time to give a stupid or misinformed beholder a black eye.\u201d", "author": "Jim Henson", "tags": ["humor"]},

{"text": "\u201cToday you are You, that is truer than true. There is no one alive who is Youer than You.\u201d", "author": "Dr. Seuss", "tags": ["comedy", "life", "yourself"]},

{"text": "\u201cIf you want your children to be intelligent, read them fairy tales. If you want them to be more intelligent, read them more fairy tales.\u201d", "author": "Albert Einstein", "tags": ["children", "fairy-tales"]},

{"text": "\u201cIt is impossible to live without failing at something, unless you live so cautiously that you might as well not have lived at all - in which case, you fail by default.\u201d", "author": "J.K. Rowling", "tags": []},

{"text": "\u201cLogic will get you from A to Z; imagination will get you everywhere.\u201d", "author": "Albert Einstein", "tags": ["imagination"]},

{"text": "\u201cOne good thing about music, when it hits you, you feel no pain.\u201d", "author": "Bob Marley", "tags": ["music"]},

{"text": "\u201cThe more that you read, the more things you will know. The more that you learn, the more places you'll go.\u201d", "author": "Dr. Seuss", "tags": ["learning", "reading", "seuss"]},

{"text": "\u201cOf course it is happening inside your head, Harry, but why on earth should that mean that it is not real?\u201d", "author": "J.K. Rowling", "tags": ["dumbledore"]},

{"text": "\u201cThe truth is, everyone is going to hurt you. You just got to find the ones worth suffering for.\u201d", "author": "Bob Marley", "tags": ["friendship"]},

{"text": "\u201cNot all of us can do great things. But we can do small things with great love.\u201d", "author": "Mother Teresa", "tags": ["misattributed-to-mother-teresa", "paraphrased"]},

{"text": "\u201cTo the well-organized mind, death is but the next great adventure.\u201d", "author": "J.K. Rowling", "tags": ["death", "inspirational"]},

{"text": "\u201cAll you need is love. But a little chocolate now and then doesn't hurt.\u201d", "author": "Charles M. Schulz", "tags": ["chocolate", "food", "humor"]},

{"text": "\u201cWe read to know we're not alone.\u201d", "author": "William Nicholson", "tags": ["misattributed-to-c-s-lewis", "reading"]},

{"text": "\u201cAny fool can know. The point is to understand.\u201d", "author": "Albert Einstein", "tags": ["knowledge", "learning", "understanding", "wisdom"]},

{"text": "\u201cI have always imagined that Paradise will be a kind of library.\u201d", "author": "Jorge Luis Borges", "tags": ["books", "library"]},

{"text": "\u201cIt is never too late to be what you might have been.\u201d", "author": "George Eliot", "tags": ["inspirational"]},

{"text": "\u201cA reader lives a thousand lives before he dies, said Jojen. The man who never reads lives only one.\u201d", "author": "George R.R. Martin", "tags": ["read", "readers", "reading", "reading-books"]},

{"text": "\u201cYou can never get a cup of tea large enough or a book long enough to suit me.\u201d", "author": "C.S. Lewis", "tags": ["books", "inspirational", "reading", "tea"]},

{"text": "\u201cYou believe lies so you eventually learn to trust no one but yourself.\u201d", "author": "Marilyn Monroe", "tags": []},

{"text": "\u201cIf you can make a woman laugh, you can make her do anything.\u201d", "author": "Marilyn Monroe", "tags": ["girls", "love"]},

{"text": "\u201cLife is like riding a bicycle. To keep your balance, you must keep moving.\u201d", "author": "Albert Einstein", "tags": ["life", "simile"]},

{"text": "\u201cThe real lover is the man who can thrill you by kissing your forehead or smiling into your eyes or just staring into space.\u201d", "author": "Marilyn Monroe", "tags": ["love"]},

{"text": "\u201cA wise girl kisses but doesn't love, listens but doesn't believe, and leaves before she is left.\u201d", "author": "Marilyn Monroe", "tags": ["attributed-no-source"]},

{"text": "\u201cOnly in the darkness can you see the stars.\u201d", "author": "Martin Luther King Jr.", "tags": ["hope", "inspirational"]},

{"text": "\u201cIt matters not what someone is born, but what they grow to be.\u201d", "author": "J.K. Rowling", "tags": ["dumbledore"]},

{"text": "\u201cLove does not begin and end the way we seem to think it does. Love is a battle, love is a war; love is a growing up.\u201d", "author": "James Baldwin", "tags": ["love"]},

{"text": "\u201cThere is nothing I would not do for those who are really my friends. I have no notion of loving people by halves, it is not my nature.\u201d", "author": "Jane Austen", "tags": ["friendship", "love"]},

{"text": "\u201cDo one thing every day that scares you.\u201d", "author": "Eleanor Roosevelt", "tags": ["attributed", "fear", "inspiration"]},

{"text": "\u201cI am good, but not an angel. I do sin, but I am not the devil. I am just a small girl in a big world trying to find someone to love.\u201d", "author": "Marilyn Monroe", "tags": ["attributed-no-source"]},

{"text": "\u201cIf I were not a physicist, I would probably be a musician. I often think in music. I live my daydreams in music. I see my life in terms of music.\u201d", "author": "Albert Einstein", "tags": ["music"]},

{"text": "\u201cIf you only read the books that everyone else is reading, you can only think what everyone else is thinking.\u201d", "author": "Haruki Murakami", "tags": ["books", "thought"]},

{"text": "\u201cThe difference between genius and stupidity is: genius has its limits.\u201d", "author": "Alexandre Dumas fils", "tags": ["misattributed-to-einstein"]},

{"text": "\u201cHe's like a drug for you, Bella.\u201d", "author": "Stephenie Meyer", "tags": ["drug", "romance", "simile"]},

{"text": "\u201cThere is no friend as loyal as a book.\u201d", "author": "Ernest Hemingway", "tags": ["books", "friends", "novelist-quotes"]},

{"text": "\u201cWhen one door of happiness closes, another opens; but often we look so long at the closed door that we do not see the one which has been opened for us.\u201d", "author": "Helen Keller", "tags": ["inspirational"]},

{"text": "\u201cLife isn't about finding yourself. Life is about creating yourself.\u201d", "author": "George Bernard Shaw", "tags": ["inspirational", "life", "yourself"]},

{"text": "\u201cThat's the problem with drinking, I thought, as I poured myself a drink. If something bad happens you drink in an attempt to forget; if something good happens you drink in order to celebrate; and if nothing happens you drink to make something happen.\u201d", "author": "Charles Bukowski", "tags": ["alcohol"]},

{"text": "\u201cYou don\u2019t forget the face of the person who was your last hope.\u201d", "author": "Suzanne Collins", "tags": ["the-hunger-games"]},

{"text": "\u201cRemember, we're madly in love, so it's all right to kiss me anytime you feel like it.\u201d", "author": "Suzanne Collins", "tags": ["humor"]},

{"text": "\u201cTo love at all is to be vulnerable. Love anything and your heart will be wrung and possibly broken. If you want to make sure of keeping it intact you must give it to no one, not even an animal. Wrap it carefully round with hobbies and little luxuries; avoid all entanglements. Lock it up safe in the casket or coffin of your selfishness. But in that casket, safe, dark, motionless, airless, it will change. It will not be broken; it will become unbreakable, impenetrable, irredeemable. To love is to be vulnerable.\u201d", "author": "C.S. Lewis", "tags": ["love"]},

{"text": "\u201cNot all those who wander are lost.\u201d", "author": "J.R.R. Tolkien", "tags": ["bilbo", "journey", "lost", "quest", "travel", "wander"]},

{"text": "\u201cDo not pity the dead, Harry. Pity the living, and, above all those who live without love.\u201d", "author": "J.K. Rowling", "tags": ["live-death-love"]},

{"text": "\u201cThere is nothing to writing. All you do is sit down at a typewriter and bleed.\u201d", "author": "Ernest Hemingway", "tags": ["good", "writing"]},

{"text": "\u201cFinish each day and be done with it. You have done what you could. Some blunders and absurdities no doubt crept in; forget them as soon as you can. Tomorrow is a new day. You shall begin it serenely and with too high a spirit to be encumbered with your old nonsense.\u201d", "author": "Ralph Waldo Emerson", "tags": ["life", "regrets"]},

{"text": "\u201cI have never let my schooling interfere with my education.\u201d", "author": "Mark Twain", "tags": ["education"]},

{"text": "\u201cI have heard there are troubles of more than one kind. Some come from ahead and some come from behind. But I've bought a big bat. I'm all ready you see. Now my troubles are going to have troubles with me!\u201d", "author": "Dr. Seuss", "tags": ["troubles"]},

{"text": "\u201cIf I had a flower for every time I thought of you...I could walk through my garden forever.\u201d", "author": "Alfred Tennyson", "tags": ["friendship", "love"]},

{"text": "\u201cSome people never go crazy. What truly horrible lives they must lead.\u201d", "author": "Charles Bukowski", "tags": ["humor"]},

{"text": "\u201cThe trouble with having an open mind, of course, is that people will insist on coming along and trying to put things in it.\u201d", "author": "Terry Pratchett", "tags": ["humor", "open-mind", "thinking"]},

{"text": "\u201cThink left and think right and think low and think high. Oh, the thinks you can think up if only you try!\u201d", "author": "Dr. Seuss", "tags": ["humor", "philosophy"]},

{"text": "\u201cWhat really knocks me out is a book that, when you're all done reading it, you wish the author that wrote it was a terrific friend of yours and you could call him up on the phone whenever you felt like it. That doesn't happen much, though.\u201d", "author": "J.D. Salinger", "tags": ["authors", "books", "literature", "reading", "writing"]},

{"text": "\u201cThe reason I talk to myself is because I\u2019m the only one whose answers I accept.\u201d", "author": "George Carlin", "tags": ["humor", "insanity", "lies", "lying", "self-indulgence", "truth"]},

{"text": "\u201cYou may say I'm a dreamer, but I'm not the only one. I hope someday you'll join us. And the world will live as one.\u201d", "author": "John Lennon", "tags": ["beatles", "connection", "dreamers", "dreaming", "dreams", "hope", "inspirational", "peace"]},

{"text": "\u201cI am free of all prejudice. I hate everyone equally. \u201d", "author": "W.C. Fields", "tags": ["humor", "sinister"]},

{"text": "\u201cThe question isn't who is going to let me; it's who is going to stop me.\u201d", "author": "Ayn Rand", "tags": []},

{"text": "\u201c\u2032Classic\u2032 - a book which people praise and don't read.\u201d", "author": "Mark Twain", "tags": ["books", "classic", "reading"]},

{"text": "\u201cAnyone who has never made a mistake has never tried anything new.\u201d", "author": "Albert Einstein", "tags": ["mistakes"]},

{"text": "\u201cA lady's imagination is very rapid; it jumps from admiration to love, from love to matrimony in a moment.\u201d", "author": "Jane Austen", "tags": ["humor", "love", "romantic", "women"]},

{"text": "\u201cRemember, if the time should come when you have to make a choice between what is right and what is easy, remember what happened to a boy who was good, and kind, and brave, because he strayed across the path of Lord Voldemort. Remember Cedric Diggory.\u201d", "author": "J.K. Rowling", "tags": ["integrity"]},

{"text": "\u201cI declare after all there is no enjoyment like reading! How much sooner one tires of any thing than of a book! -- When I have a house of my own, I shall be miserable if I have not an excellent library.\u201d", "author": "Jane Austen", "tags": ["books", "library", "reading"]},

{"text": "\u201cThere are few people whom I really love, and still fewer of whom I think well. The more I see of the world, the more am I dissatisfied with it; and every day confirms my belief of the inconsistency of all human characters, and of the little dependence that can be placed on the appearance of merit or sense.\u201d", "author": "Jane Austen", "tags": ["elizabeth-bennet", "jane-austen"]},

{"text": "\u201cSome day you will be old enough to start reading fairy tales again.\u201d", "author": "C.S. Lewis", "tags": ["age", "fairytales", "growing-up"]},

{"text": "\u201cWe are not necessarily doubting that God will do the best for us; we are wondering how painful the best will turn out to be.\u201d", "author": "C.S. Lewis", "tags": ["god"]},

{"text": "\u201cThe fear of death follows from the fear of life. A man who lives fully is prepared to die at any time.\u201d", "author": "Mark Twain", "tags": ["death", "life"]},

{"text": "\u201cA lie can travel half way around the world while the truth is putting on its shoes.\u201d", "author": "Mark Twain", "tags": ["misattributed-mark-twain", "truth"]},

{"text": "\u201cI believe in Christianity as I believe that the sun has risen: not only because I see it, but because by it I see everything else.\u201d", "author": "C.S. Lewis", "tags": ["christianity", "faith", "religion", "sun"]},

{"text": "\u201cThe truth.\" Dumbledore sighed. \"It is a beautiful and terrible thing, and should therefore be treated with great caution.\u201d", "author": "J.K. Rowling", "tags": ["truth"]},

{"text": "\u201cI'm the one that's got to die when it's time for me to die, so let me live my life the way I want to.\u201d", "author": "Jimi Hendrix", "tags": ["death", "life"]},

{"text": "\u201cTo die will be an awfully big adventure.\u201d", "author": "J.M. Barrie", "tags": ["adventure", "love"]},

{"text": "\u201cIt takes courage to grow up and become who you really are.\u201d", "author": "E.E. Cummings", "tags": ["courage"]},

{"text": "\u201cBut better to get hurt by the truth than comforted with a lie.\u201d", "author": "Khaled Hosseini", "tags": ["life"]},

{"text": "\u201cYou never really understand a person until you consider things from his point of view... Until you climb inside of his skin and walk around in it.\u201d", "author": "Harper Lee", "tags": ["better-life-empathy"]},

{"text": "\u201cYou have to write the book that wants to be written. And if the book will be too difficult for grown-ups, then you write it for children.\u201d", "author": "Madeleine L'Engle", "tags": ["books", "children", "difficult", "grown-ups", "write", "writers", "writing"]},

{"text": "\u201cNever tell the truth to people who are not worthy of it.\u201d", "author": "Mark Twain", "tags": ["truth"]},

{"text": "\u201cA person's a person, no matter how small.\u201d", "author": "Dr. Seuss", "tags": ["inspirational"]},

{"text": "\u201c... a mind needs books as a sword needs a whetstone, if it is to keep its edge.\u201d", "author": "George R.R. Martin", "tags": ["books", "mind"]}

]

爬虫--Scrapy框架的基本使用的更多相关文章

  1. python爬虫scrapy框架——人工识别登录知乎倒立文字验证码和数字英文验证码(2)

    操作环境:python3 在上一文中python爬虫scrapy框架--人工识别知乎登录知乎倒立文字验证码和数字英文验证码(1)我们已经介绍了用Requests库来登录知乎,本文如果看不懂可以先看之前 ...

  2. 爬虫scrapy框架之CrawlSpider

    爬虫scrapy框架之CrawlSpider   引入 提问:如果想要通过爬虫程序去爬取全站数据的话,有几种实现方法? 方法一:基于Scrapy框架中的Spider的递归爬取进行实现(Request模 ...

  3. 安装爬虫 scrapy 框架前提条件

    安装爬虫 scrapy 框架前提条件 (不然 会 报错) pip install pypiwin32

  4. 爬虫Ⅱ:scrapy框架

    爬虫Ⅱ:scrapy框架 step5: Scrapy框架初识 Scrapy框架的使用 pySpider 什么是框架: 就是一个具有很强通用性且集成了很多功能的项目模板(可以被应用在各种需求中) scr ...

  5. Python爬虫Scrapy框架入门（2）

    本文是跟着大神博客,尝试从网站上爬一堆东西,一堆你懂得的东西 附上原创链接: http://www.cnblogs.com/qiyeboy/p/5428240.html 基本思路是,查看网页元素,填写 ...

  6. 爬虫Scrapy框架运用----房天下二手房数据采集

    在许多电商和互联网金融的公司为了更好地服务用户,他们需要爬虫工程师对用户的行为数据进行搜集.分析和整合,为人们的行为选择提供更多的参考依据,去服务于人们的行为方式,甚至影响人们的生活方式.我们的scr ...

  7. 自己动手实现爬虫scrapy框架思路汇总

    这里先简要温习下爬虫实际操作: cd ~/Desktop/spider scrapy startproject lastspider # 创建爬虫工程 cd lastspider/ # 进入工程 sc ...

  8. 爬虫--Scrapy框架课程介绍

    Scrapy框架课程介绍: 框架的简介和基础使用 持久化存储 代理和cookie 日志等级和请求传参 CrawlSpider 基于redis的分布式爬虫 一scrapy框架的简介和基础使用 a)    ...

  9. Python网咯爬虫 — Scrapy框架应用

    Scrapy框架       Scrapy是一个高级的爬虫框架,它不仅包括了爬虫的特征,还可以方便地将爬虫数据保存到CSV.Json等文件中.       Scrapy用途广泛,可以用于数据挖掘.监测 ...

随机推荐

  1. 第四周PSP &进度条

    团队项目PSP 一:表格     C类型 C内容 S开始时间 E结束时间 I时间间隔 T净时间(mins) 预计花费时间(mins) 讨论 讨论开发环境.工具以及技术 8:37 10:42 25 10 ...

  2. Jarvis OJ平台basic部分wirteup

    Base64? 题目描述: GUYDIMZVGQ2DMN3CGRQTONJXGM3TINLGG42DGMZXGM3TINLGGY4DGNBXGYZTGNLGGY3DGNBWMU3WI=== Base3 ...

  3. Hibernate 中一级缓存和快照区的理解

    刚刚开始的时候觉得这个快照区很难理解,在网上看了很多博客之后,开始明白了.我是结合 ADO.NET 理解的,在ADO.NET 中有一个类, 叫 SqlCommandBuilder,在我看来,他就是 A ...

  4. js & 快捷键

    js & 快捷键 demo js-keyboard-shortcuts.html https://codepen.io/webgeeker/pen/MLYrNq let isCtrl = fa ...

  5. angularjs 指令间相互调用

    <div ng-app="app"> <div ng-controller="myctl"> <button superman s ...

  6. list的迭代器能解决并发问题,collection 的迭代器不能解决并发问题，for可以解决并发问题

    list的迭代器能解决并发问题,collection 的迭代器不能解决并发问题 为什么list支持add,collection不支持 例如有两个人同时添加第三个元素 list的迭代器能锁定线程 只有等 ...

  7. HDFS集中式的缓存管理原理与代码剖析--转载

    原文地址:http://yanbohappy.sinaapp.com/?p=468 Hadoop 2.3.0已经发布了,其中最大的亮点就是集中式的缓存管理(HDFS centralized cache ...

  8. Python中int()函数的用法浅析

      int()是Python的一个内部函数 Python系统帮助里面是这么说的 >>> help(int)  Help on class int in module __builti ...

  9. 洛谷 P2574 XOR的艺术

    刚刚学了,线段树,一道线段树入门题试试水 下面是题面 题目描述 AKN觉得第一题太水了,不屑于写第一题,所以他又玩起了新的游戏.在游戏中,他发现,这个游戏的伤害计算有一个规律,规律如下 1. 拥有一个 ...

  10. 洛谷 P3391 【模板】文艺平衡树（Splay）

    题目背景 这是一道经典的Splay模板题——文艺平衡树. 题目描述 您需要写一种数据结构(可参考题目标题),来维护一个有序数列,其中需要提供以下操作:翻转一个区间,例如原有序序列是5 4 3 2 1, ...