scrapy初试水 day01

1.安装
pip install Scrapy
#一定要以管理员身份运行dos窗口
conda install scrapy
2.创建项目
scrapy startproject hello
3.在hello/spiders下创建dmoz_spider.py
import scrapy

class DmozSpider(scrapy.Spider):
name = "dmoz"
allowed_domains = ["dmoz.org"]
start_urls = [
"http://www.runoob.com/python/python-tutorial.html",
"http://python.jobbole.com/"
]
#每爬完一个网页会回调parse方法
def parse(self, response):
filename = response.url.split("/")[-2]
print('-------------------------------')
print(filename)
with open(filename, 'wb') as f:
f.write(response.body)

4.运行,在项目根目录下dos执行:
scrapy crawl dmoz
5.在根目录下生成两个文件
python.jobbole.com和python，分别是爬下的两个网页