Day 22 22.1.2:增量式爬虫 - 场景2的实现
场景2的实现:
数据指纹
- 使用详情页的url充当数据指纹即可。
创建爬虫爬虫文件:
- cd project_name(进入项目目录)
- scrapy genspider 爬虫文件的名称(自定义一个名字即可) 起始url
- (例如:scrapy genspider first www.xxx.com)
- 创建成功后,会在爬虫文件夹下生成一个py的爬虫文件
进入爬虫文件:
- cd 爬虫文件的名称(即自定义的名字)
爬虫文件
import scrapy
import redis
import random
from ..items import Zlsdemo2ProItem
class JianliSpider(scrapy.Spider):
name = "jianli"
# allowed_domains = ["www.xxx.com"]
start_urls = ["https://sc.chinaz.com/jianli/free.html"]
# 连接数据库
conn = redis.Redis(
host='127.0.0.1',
port=6379,
)
def parse(self, response):
div_list = response.xpath('//*[@id="container"]/div')
for div in div_list:
title = div.xpath('./p/a/text()').extract_first()
detail_url = div.xpath('./p/a/@href').extract_first()
# print(title, detail_url)
ex = self.conn.sadd('data_id', detail_url)
#建立item对象
item = Zlsdemo2ProItem()
item['title'] = title#将title传给item
if ex == 1:#数据库中没有存储爬取到的数据
print('有最新数据更新,正在采集中......')
#做请求传参,将title通过meta传递给parse_detail函数
yield scrapy.Request(url=detail_url,callback=self.parse_detail,meta={'item':item})
else:#数据库中已存储爬取到的数据
print('暂无数据更新!')
def parse_detail(self,response):
item = response.meta['item']
download_li = response.xpath('//*[@id="down"]/div[2]/ul/li')
download_urls = []
for li in download_li:
download_url_e = li.xpath('./a/@href').extract_first()
download_urls.append(download_url_e)
download_url = random.choice(download_urls)
item['download_url'] = download_url
yield item
items.py
# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html
import scrapy
class Zlsdemo2ProItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
title = scrapy.Field()
download_url = scrapy.Field()
pipelines.py
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
# useful for handling different item types with a single interface
from itemadapter import ItemAdapter
class Zlsdemo2ProPipeline:
def process_item(self, item, spider):
conn = spider.conn
conn.lpush('data1', item)
return item
settings
# Scrapy settings for zlsDemo2Pro project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# https://docs.scrapy.org/en/latest/topics/settings.html
# https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
BOT_NAME = "zlsDemo2Pro"
SPIDER_MODULES = ["zlsDemo2Pro.spiders"]
NEWSPIDER_MODULE = "zlsDemo2Pro.spiders"
# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
# Obey robots.txt rules
ROBOTSTXT_OBEY = False
LOG_LEVEL = 'ERROR'
LOG_LEVEL = 'WARNING'
# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32
# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16
# Disable cookies (enabled by default)
#COOKIES_ENABLED = False
# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False
# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
# "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
# "Accept-Language": "en",
#}
# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
# "zlsDemo2Pro.middlewares.Zlsdemo2ProSpiderMiddleware": 543,
#}
# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
# "zlsDemo2Pro.middlewares.Zlsdemo2ProDownloaderMiddleware": 543,
#}
# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
# "scrapy.extensions.telnet.TelnetConsole": None,
#}
# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
"zlsDemo2Pro.pipelines.Zlsdemo2ProPipeline": 300,
}
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = "httpcache"
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = "scrapy.extensions.httpcache.FilesystemCacheStorage"
# Set settings whose default value is deprecated to a future-proof value
REQUEST_FINGERPRINTER_IMPLEMENTATION = "2.7"
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
FEED_EXPORT_ENCODING = "utf-8"
Day 22 22.1.2:增量式爬虫 - 场景2的实现的更多相关文章
- Scrapy 增量式爬虫
Scrapy 增量式爬虫 https://blog.csdn.net/mygodit/article/details/83931009 https://blog.csdn.net/mygodit/ar ...
- 基于Scrapy框架的增量式爬虫
概述 概念:监测 核心技术:去重 基于 redis 的一个去重 适合使用增量式的网站: 基于深度爬取的 对爬取过的页面url进行一个记录(记录表) 基于非深度爬取的 记录表:爬取过的数据对应的数据指纹 ...
- 增量式爬虫 Scrapy-Rredis 详解及案例
1.创建scrapy项目命令 scrapy startproject myproject 2.在项目中创建一个新的spider文件命令: scrapy genspider mydomain mydom ...
- 爬虫 crawlSpider 分布式 增量式 提高效率
crawlSpider 作用:为了方便提取页面整个链接url,不必使用创参寻找url,通过拉链提取器,将start_urls的全部符合规则的URL地址全部取出 使用:创建文件scrapy startp ...
- python爬虫---CrawlSpider实现的全站数据的爬取,分布式,增量式,所有的反爬机制
CrawlSpider实现的全站数据的爬取 新建一个工程 cd 工程 创建爬虫文件:scrapy genspider -t crawl spiderName www.xxx.com 连接提取器Link ...
- 爬虫07 /scrapy图片爬取、中间件、selenium在scrapy中的应用、CrawlSpider、分布式、增量式
爬虫07 /scrapy图片爬取.中间件.selenium在scrapy中的应用.CrawlSpider.分布式.增量式 目录 爬虫07 /scrapy图片爬取.中间件.selenium在scrapy ...
- 爬虫---scrapy分布式和增量式
分布式 概念: 需要搭建一个分布式的机群, 然后在每一台电脑中执行同一组程序, 让其对某一网站的数据进行联合分布爬取. 原生的scrapy框架不能实现分布式的原因 调度器不能被共享, 管道也不能被共享 ...
- [开源 .NET 跨平台 数据采集 爬虫框架: DotnetSpider] [三] 配置式爬虫
[DotnetSpider 系列目录] 一.初衷与架构设计 二.基本使用 三.配置式爬虫 四.JSON数据解析与配置系统 上一篇介绍的基本的使用方式,虽然自由度很高,但是编写的代码相对还是挺多.于是框 ...
- 增量式PID计算公式4个疑问与理解
一开始见到PID计算公式时总是疑问为什么是那样子?为了理解那几道公式,当时将其未简化前的公式“活生生”地算了一遍,现在想来,这样的演算过程固然有助于理解,但假如一开始就带着对疑问的答案已有一定看法后再 ...
- 增量式PID简单翻板角度控制
1.研究背景 随着电子技术.信息技术和自动控制理论技术的完善与发展,近来微型处理器在控制方面的应用也越来越多.随之逐渐渗透到我们生活的各个领域.如导弹导航装置,飞机上仪表的控制,网络通讯与数据传输,工 ...
随机推荐
- js获取的 后端的列表。
1.引号被转码 处理办法 {% autoescape off %} var tmp = '{{ data1 }}'; var tmp = '{{ data2 }}'; {% endautoescape ...
- git---全局设置用户名、密码、邮箱
# git config命令的–global参数,用了这个参数,表示你这台机器上所有的Git仓库都会使用这个配置,当然也可以对某个仓库指定不同的用户名和Email地址. # 1.查看git配置信息 $ ...
- JavaSE——金额转换
package com.zhao.stringtest; import java.util.Scanner; public class Test3 { //金额转换 //查表法 public stat ...
- PLSQL DEVELOPER使用
一.下载plsql developer.(oracle官网下载地址:https://www.oracle.com/technetwork/topics/winsoft-085727.html) 二.下 ...
- 076_Master Detail 与Lookup
- web基础(5): CSS3介绍
chapter5 CSS3 新性能 (一)圆角边框与阴影 1.border-radius属性 例1 border-top-left-radius:40px 20px ; 两个值分别表示水平方向.垂直方 ...
- Linuxt通过命令lsof或者extundelete工具恢复误删除的文件或者目录
Linux不像windows有那么显眼的回收站,不是简单的还原就可以了.linux删除文件还原可以分为两种情况,一种是删除以后在进程存在删除信息,一种是删除以后进程都找不到,只有借助于工具还原.这里分 ...
- 如果同时存在application.properties 和application.yml 文件的话,springboot会优先使用application.properties吗
如果同时存在application.properties 和application.yml 文件的话,springboot会优先使用application.properties吗 总结:如果appli ...
- 链接mysql数据库报错:2003-cant connect to Mysql server on ‘localhost’(10038)
今天用navicat连mysql时候突然报错了 我百度了一下,知道了是mysql服务没开,但是我打开了服务,发现服务里面没有mysql,于是就去cmd以管理员身份打开命令提示符,切换到安装mysql的 ...
- Apache和Nginx设置伪静态(URL Rewrite)的方法
在Apache中有两种方法配置,一是通过httpd.conf文件,二是通过.htaccess文件. 使用 httpd.conf 来配置 rewrite 策略. 要使用 httpd.conf 文件来设置 ...