爬虫之牛掰的scrapy框架

一. Scrapy简介及安装

http://python.jobbole.com/86405/ Scrapy的详细介绍

1.简介

2.安装

1.window上安装:

先安装依赖包:pip3 install wheel

https://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted 下载以后安装pip3 install 安装包

pip3 install pywin32

pip3 install scrapy

2mac上: pip3 install scrapy

二. Scrapy常见命令

1.创建项目: scrapy startproject project-name

2.创建爬虫文件: scrapy genspider filename 指定网站

3.运行项目: scrapy crawl filename

scrapy crawl filename --nolog 不打印日志

三.Scrapy的基本使用

1.创建项目.

通过命令进行创建: scrapy startproject 项目名

2.自动创建目录的结果.

文件说明:

1.boss.py:爬虫文件.一般创建爬虫文件时,以网站域名命名.

通过命令创建: scrapy genspider boss 指定网站

2. items.py:设置数据储存模块,用于结构化数据

3. middlewares

4.pipelines

5.settings:配置文件.

6.spiders 爬虫目录.如:创建文件编写爬虫规则

3,编写文件

(1)爬虫文件中

# -*- coding: utf-8 -*-

import scrapy

from bossDemo.items import BossdemoItem

# 爬虫文件的作用

# 1.url的指定

# 2.请求的发送

# 3.数据的解析

# 4.将item对象通过yield传给管道文件

class BossSpider(scrapy.Spider):

name = 'boss'

# allowed_domains = ['www.xxx.com']

start_urls = ['https://www.zhipin.com/job_detail/?query=python&scity=101180100&industry=&position=']

def parse(self, response):

li_list = response.xpath('//*[@id="main"]/div/div[3]/ul/li')

# print(li_list)

for li in li_list:

title = li.xpath('./div/div[1]/h3/a/div[1]/text()').extract_first()

salary = li.xpath('./div/div[1]/h3/a/span/text()').extract_first()

company = li.xpath('./div/div[2]/div/h3/a/text()').extract_first()

print(title +salary + company)

# 实例化一个item对象

item = BossdemoItem()

# 将解析后的数据储存到item对象中

item['title'] = title

item['salary'] = salary

item['company'] = company

# 将item对象传给管道文件进行持久化储存

yield item

(2)items.py

import scrapy

class BossdemoItem(scrapy.Item):

# define the fields for your item here like:

title = scrapy.Field()

salary = scrapy.Field()

company = scrapy.Field()

(3)Pipelines.py

import pymysql

from redis import Redis

import json

class BossdemoPipeline(object):

# 这个函数只会在开始爬取的时候执行一次

def open_spider(self, spider):

print('爬虫开始')

self.fp = open('./job.txt', 'w', encoding='utf-8' )

# 每提交一次item,这个文件就执行一次

def process_item(self, item, spider):

self.fp.write(item['title'] + '\t' + item['salary'] + '\t' + item['company'] + '\n')

print('爬取中')

return item

# 存储结束后执行这个函数

def close_spider(self,spider):

print('爬虫结束')

self.fp.close()

class MysqlPipeline(object):

cursor = None

conn = None

def open_spider(self, spider):

print('mysql爬虫开始')

self.conn = pymysql.Connect(host='127.0.0.1', port=3306, user='root', password='', db='db1' )

def process_item(self, item, spider):

self.cursor = self.conn.cursor()

sql = 'insert into boss (title, salary, company) values ("%s","%s","%s")'%(item["title"], item["salary"],item["company"])

try:

print('mysql爬虫中')

self.cursor.execute(sql)

self.conn.commit()

except Exception as e:

print(e)

self.conn.rollback()

return item

def close_spider(self, spider):

print('mysql爬虫结束')

self.cursor.close()

self.conn.close()

class RedisPipeline(object):

def open_spider(self, spider):

print('redis储存')

self.conn = Redis(host='127.0.0.1', port='6379')

def process_item(self, item, spider):

dic = {

'title': item['title'],

'salary': item['salary'],

'company': item['company'],

}

print('redis存储中...')

self.conn.lpush('Jobinfo', json.dumps(dic))

def close_spider(self, spider):

print('redis结束')

备注:

1.爬虫文件需要定义一个类,并继承(scrapy.Spider)

2.必须定义name, 即爬虫名,如果没有那么,会报错,因为源码中这样规定的:

3.编写函数parse,这里需要注意的是,该函数不能改变,是因为Scrapy中默认callback函数的函数名就是parse.

4.scrapy发送post请求

# 1.scrapy中post请求的发送:重写源码中的start_requests方法

# 因为源码中这样写的:for url in self.start_urls:

# yield self.make_requests_from_url(url) (make_requests_from_url方法的返回结果是Request对象)

# 2.在scrapy 框架中,会自动对cookie进行处理,可以在settings中设置不处理 COOKIES_ENABLED = False

示例代码:

import scrapy

class LoginSpider(scrapy.Spider):

name = 'login'

# allowed_domains = ['www.xxx.com']

start_urls = ['http://www.renren.com/ajaxLogin/login?1=1&uniqueTimestamp=201903160368']

def start_requests(self):

data = {

"email": "15516092050",

"icode": '',

"origURL": "http://www.renren.com/home",

"domain": "renren.com",

"key_id": '1',

"captcha_type": "web_login",

"password": "5e088a2ee22d34dd081aac25578e67bd3a2d851cdfbcf1f0c9ab7056bd1bad62",

"rkey": "3f4696f6fa1b89e9061868300bf11484",

"f": "http%3A%2F%2Fwww.renren.com%2F969395731",

}

for url in self.start_urls:

yield scrapy.FormRequest(url=url, formdata=data, callback=self.parse)

def parse(self, response):

detail_url = 'http://www.renren.com/969395731'

yield scrapy.Request(url = detail_url, callback=self.GetDetail)

def GetDetail(self, response):

ret = response.text

print(ret)

5.scrapy请求传参的方式

1.用scrapy爬取数据时,如果发现需要爬取的数据不在同一页面内,则必须使用请求传参的方式进行持久化储存

2.示例代码

class MovieSpider(scrapy.Spider):

name = 'movie'

# allowed_domains = ['www.xxx.com']

start_urls = ['http://www.55xia.com/']

def parse(self, response):

div_list = response.xpath('/html/body/div[1]/div[2]/div[1]/div/div | /html/body/div[1]/div[2]/div[3]/div/div')

for div in div_list:

item = MoviedemoItem()

detail_url = div.xpath('./div/div/h1/a/@href')

if not detail_url:

continue

else:

detail_url = 'http:' + detail_url.extract_first()

name = div.xpath('./div/div/h1/a/text()').extract_first()

score = div.xpath('./div/div/h1/em/text()')

if not score:

score = '暂无评价'

else:

score = score.extract_first()

item['name'] = name

item['score'] = score

print(name)

print(score)

yield scrapy.Request(url=detail_url, callback=self.GetDetail, meta={'item':item})

def GetDetail(self,response):

item = response.meta['item']

direct = response.xpath('/html/body/div[1]/div/div/div[1]/div[1]/div[2]/table/tbody/tr[1]/td[2]/a/text()').\

extract_first()

detail = response.xpath('/html/body/div[1]/div/div/div[1]/div[2]/div[2]//text()'). extract_first()

item['direct'] = direct

item['detail'] = detail

print(direct)

print(detail)

yield item

6.提高scrapy框架效率

# 增加并发：

# 默认scrapy开启的并发线程为32个，可以适当进行增加。在settings配置文件中修改CONCURRENT_REQUESTS = 100值为100,并发设置成了为100。

# 降低日志级别：

# 在运行scrapy时，会有大量日志信息的输出，为了减少CPU的使用率。可以设置log输出信息为INFO或者ERROR即可。在配置文件中编写：LOG_LEVEL = ‘INFO’

# 禁止cookie：

# 如果不是真的需要cookie，则在scrapy爬取数据时可以禁止cookie从而减少CPU的使用率，提升爬取效率。在配置文件中编写：COOKIES_ENABLED = False

# 禁止重试：

# 对失败的HTTP进行重新请求（重试）会减慢爬取速度，因此可以禁止重试。在配置文件中编写：RETRY_ENABLED = False

# 减少下载超时：

# 如果对一个非常慢的链接进行爬取，减少下载超时可以能让卡住的链接快速被放弃，从而提升效率。在配置文件中进行编写：DOWNLOAD_TIMEOUT = 10 超时时间为10s

CONCURRENT_REQUESTS = 10

LOG_LEVEL = 'ERROR'

COOKIES_ENABLED = False

RETRY_ENABLED = False

DOWNLOAD_TIMEOUT = 5

7.设置代理池和UA池

1.UA池：User-Agent池

- 作用：尽可能多的将scrapy工程中的请求伪装成不同类型的浏览器身份。

- 操作流程：

1.在下载中间件中拦截请求

2.将拦截到的请求的请求头信息中的UA进行篡改伪装

3.在配置文件中开启下载中间件

user_agent_list = [

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 "

"(KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",

"Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 "

"(KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 "

"(KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",

"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 "

"(KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",

"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 "

"(KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",

"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 "

"(KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",

"Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 "

"(KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "

"(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",

"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 "

"(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",

"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 "

"(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",

"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "

"(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "

"(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",

"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "

"(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "

"(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",

"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 "

"(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",

"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "

"(KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",

"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 "

"(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",

"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 "

"(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"

]

2.代理池

- 作用：尽可能多的将scrapy工程中的请求的IP设置成不同的。

- 操作流程：

1.在下载中间件中拦截请求

2.将拦截到的请求的IP修改成某一代理IP

3.在配置文件中开启下载中间件

示例代码

http_proxy = ['http://91.226.35.93:53281', 'http://110.52.235.73:9999', 'http://151.3.53.246:53281']

https_proxy = ['https://106.104.168.15:8080', 'https://93.190.143.59:1080', 'https://223.27.212.41:8080' ]

if request.url.split(':')[0] == 'http':

request.meta['proxy'] = random.choice(http_proxy)

else:

request.meta['proxy'] = random.choice(https_proxy)

8 五大组件