如何运行简单的scrapy
1.建scrapy工程
scrapy startproject python123demo
2.在工程中写一个爬虫文件
cd python123demo
scrapy genspider demo python123.io
3.写爬虫的配置文件
4.运行爬虫
scrapy crawl demo
运行的时候出了一些小问题,这些问题是在安装scrapy时没有把关联的包安装上导致的。
ModuleNotFoundError: No module named 'win32api'
上述问题需要
pywin32-221-cp36-cp36m-win_amd64.whl这个包
ImportError: DLL load failed: 找不到指定的模块。
上述问题是由于没有成功安装pywin32-221-cp36-cp36m-win_amd64.whl这个包
重新运行生成的pywin32_postinstall.py文件即可
python.exe Scripts\pywin32_postinstall.py -install
但是可能还会出现错误,
F:\Python36>python.exe Scripts\pywin32_postinstall.py -install
Copied pythoncom36.dll to F:\Python36\pythoncom36.dll
Copied pywintypes36.dll to F:\Python36\pywintypes36.dll
You do not have the permissions to install COM objects.
The sample COM objects were not registered.
-> Software\Python\PythonCore\3.6\Help[None]=None
-> Software\Python\PythonCore\3.6\Help\Pythonwin Reference[None]='F:\\Python36\\Lib\\site-packages\\PyWin32.chm'
Pythonwin has been registered in context menu
Creating directory F:\Python36\Lib\site-packages\win32com\gen_py
Can't install shortcuts - 'C:\\Users\\asus\\AppData\\Roaming\\Microsoft\\Windows\\Start Menu\\Programs\\Python 3.6' is not a folder
The pywin32 extensions were successfully installed.
很明显,需要使用管理员权限使用上述命令
成功则显示如下信息
PS C:\WINDOWS\system32> f:
PS F:\> cd .\Python36\
PS F:\Python36> python.exe Scripts\pywin32_postinstal
Copied pythoncom36.dll to C:\WINDOWS\system32\pythonc
Copied pywintypes36.dll to C:\WINDOWS\system32\pywint
Registered: Python.Interpreter
Registered: Python.Dictionary
Registered: Python
-> Software\Python\PythonCore\3.6\Help[None]=None
-> Software\Python\PythonCore\3.6\Help\Pythonwin Refe
Pythonwin has been registered in context menu
Shortcut for Pythonwin created
Shortcut to documentation created
The pywin32 extensions were successfully installed.
再次运行爬虫,终于成功了
F:\pyProject\python123demo>scrapy crawl demo
2017-10-29 09:16:43 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: python123demo)
2017-10-29 09:16:43 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'python123demo', 'NEWSPIDER_MODULE': 'python123demo.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['python123demo.spiders']}
2017-10-29 09:16:43 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2017-10-29 09:16:43 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-10-29 09:16:43 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-10-29 09:16:43 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-10-29 09:16:43 [scrapy.core.engine] INFO: Spider opened
2017-10-29 09:16:44 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-10-29 09:16:44 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-10-29 09:16:44 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://python123.io/robots.txt> from <GET http://python123.io/robots.txt>
2017-10-29 09:16:44 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://python123.io/robots.txt> (referer: None)
2017-10-29 09:16:44 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://python123.io/ws/demo.html> from <GET http://python123.io/ws/demo.html>
2017-10-29 09:16:44 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://python123.io/ws/demo.html> (referer: None)
2017-10-29 09:16:44 [scrapy.core.scraper] ERROR: Spider error processing <GET https://python123.io/ws/demo.html> (referer: None)
Traceback (most recent call last):
File "f:\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\pyProject\python123demo\python123demo\spiders\demo.py", line 14, in parse
self.log('Save file %s.' % name)
NameError: name 'name' is not defined
2017-10-29 09:16:44 [scrapy.core.engine] INFO: Closing spider (finished)
2017-10-29 09:16:44 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 884,
'downloader/request_count': 4,
'downloader/request_method_count/GET': 4,
'downloader/response_bytes': 1595,
'downloader/response_count': 4,
'downloader/response_status_count/200': 1,
'downloader/response_status_count/301': 2,
'downloader/response_status_count/404': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2017, 10, 29, 1, 16, 44, 929393),
'log_count/DEBUG': 5,
'log_count/ERROR': 1,
'log_count/INFO': 7,
'response_received_count': 2,
'scheduler/dequeued': 2,
'scheduler/dequeued/memory': 2,
'scheduler/enqueued': 2,
'scheduler/enqueued/memory': 2,
'spider_exceptions/NameError': 1,
'start_time': datetime.datetime(2017, 10, 29, 1, 16, 44, 121136)}
2017-10-29 09:16:44 [scrapy.core.engine] INFO: Spider closed (finished)
总结:为防止出现scrapy相关依赖安装失败,可以自己逐个下载依赖
https://www.lfd.uci.edu/~gohlke/pythonlibs/
lxml
pywin32
Twisted
OpenSSL
依赖放在scripts下
通过pip install 对应whl文件即可
最后使用import对应模块判断是否安装成功
最后可通过命令升级scrapy
如何运行简单的scrapy的更多相关文章
- 简单的scrapy实战:爬取腾讯招聘北京地区的相关招聘信息
简单的scrapy实战:爬取腾讯招聘北京地区的相关招聘信息 简单的scrapy实战:爬取腾讯招聘北京地区的相关招聘信息 系统环境:Fedora22(昨天已安装scrapy环境) 爬取的开始URL:ht ...
- 简单的scrapy实例
前天实验室的学长要求写一个简单的scrapy工程出来,之前也多少看了点scrapy的知识,但始终没有太明白,刚好趁着这个机会,加深一下对scrapy工作流程的理解.由于临近期末,很多作业要做(其实.. ...
- 做一个简单的scrapy爬虫
前言: 做一个简单的scrapy爬虫,带大家认识一下创建scrapy的大致流程.我们就抓取扇贝上的单词书,python的高频词汇. 步骤: 一,新建一个工程scrapy_shanbay 二,在工程中中 ...
- JAVA 基础开发环境 vscode 搭建 Windows下VSCode编译运行简单java
JAVA 基础开发环境 vscode 搭建 来源 https://www.cnblogs.com/freewsf/p/7744728.html 对于使用 Visual Studio Code 的 Ja ...
- 《从0到1学习Flink》—— Mac 上搭建 Flink 1.6.0 环境并构建运行简单程序入门
准备工作 1.安装查看 Java 的版本号,推荐使用 Java 8. 安装 Flink 2.在 Mac OS X 上安装 Flink 是非常方便的.推荐通过 homebrew 来安装. brew in ...
- 简单的scrapy命令和中间件
创建爬虫 scrapy genspider 名字 xxx.com 运行爬虫 运行名为usnews的爬虫scrapy crawl usnews运行爬虫文件scrapy runspider quote_ ...
- 亲测——pycharm下运行第一个scrapy项目 ©seven_clear
最近在学习scrapy,就想着用pycharm调试,但不知道怎么弄,从网上搜了很多方法,这里总结一个我试成功了的. 首先当然是安装scrapy,安装教程什么的网上一大堆,这里推荐一个详细的:http: ...
- 一个简单的scrapy爬虫抓取豆瓣刘亦菲的图片地址
一.第一步是创建一个scrapy项目 sh-3.2# scrapy startproject liuyifeiImage sh-3.2# chmod -R 777 liuyifeiImage/ 二.分 ...
- 同时运行多个scrapy爬虫的几种方法(自定义scrapy项目命令)
试想一下,前面做的实验和例子都只有一个spider.然而,现实的开发的爬虫肯定不止一个.既然这样,那么就会有如下几个问题:1.在同一个项目中怎么创建多个爬虫的呢?2.多个爬虫的时候是怎么将他们运行起来 ...
随机推荐
- vs2015 编译时项目出现NuGet程序包还原失败,找不到xxx.xxx.xxx版本的程序包,怎么解决这个问题?
vs2015 编译时出现这个NuGet程序包还原失败问题,项目还是运行得了,就是每次看到错误列表中有很多个错误,就感觉不舒服. 总算被我找到解决方法了 问题截图:
- react router @4 和 vue路由 详解(三)react如何在路由里面定义一个子路由
完整版:https://www.cnblogs.com/yangyangxxb/p/10066650.html 5.react如何在路由里面定义一个子路由? a.引入在需要子路由的页面引入Rout ...
- 【用例管理】用testng的groups管理用例
一.需求: 测试时经常有两种场景,第一种是冒烟测试的小部分用例:一类是全部用例. 二.针对第一种运行部分的用例,可以用groups来管理 package com.testcases; import o ...
- UVa LA 3882 - And Then There Was One 递推,动态规划 难度: 2
题目 https://icpcarchive.ecs.baylor.edu/index.php?option=com_onlinejudge&Itemid=8&page=show_pr ...
- css3 前端开发
一.前缀: -moz(例如 -moz-border-radius)用于Firefox -webkit(例如:-webkit-border-radius)用于Safari和Chrome. 二.CSS3圆 ...
- memset详解 设置无穷大INF
memest原型 (please type "man memset" in your shell) void *memset(void *s, int c, size_t n); ...
- 完全卸载vs2013 2015
/uninstall /force 解压你的vs2013的安装包(iso格式). cd到解压后的文件夹 vs_ultimate.exe /uninstall /force 或者创建一个快捷方式到桌面 ...
- Android开发---基本UI组件1:自动拨电话,自动上网,输入框不换行、只输数字、只输文本、只输密码
1.activity_main.xml 描述:构建一个按钮 <?xml version="1.0" encoding="utf-8"?> <L ...
- Androi开发 ---Fragment片段布局案例
效果图: 1.MainActivity.java package com.example.android_activity; import android.app.Activity; import a ...
- Oracle备份
今天被吊,特来学习备份. https://blog.csdn.net/zhaiqi618/article/details/5616215 https://www.cnblogs.com/yingpp/ ...