一、依赖

virtualenv -p python3.6 xx

pip install scrapy

pip install pymysql

二、

1、创建项目和spider1

scrapy startproject scraw_swagger

scrapy genspider spider1  xxx.com  (执行之后在项目的spiders目录下会生成一个spider1.py的文件)

以下代码主要实现了将swagger的第一级目录爬下来存在一个叫:interfaces_path的文件下

# -*- coding: utf-8 -*-
import scrapy
import json
from scraw_swagger import settings class Spider1Spider(scrapy.Spider): name = 'spider1'
allowed_domains = ['xxx.com'']
scrawl_domain = settings.interface_domain+'/api-docs'
start_urls = [scrawl_domain] def parse(self, response):
# 调试代码
# filename = 'mid_link'
# open(filename, 'wb').write(response.body)
# /////////
response = response.body
response_dict = json.loads(response)
apis = response_dict['apis']
n = len(apis)
temppath = []
i = 0 domain = settings.interface_domain+'/api-docs'
filename = 'interfaces_path'
file = open(filename, 'w')
for i in range(0, n):
subapi = apis[i]
path = subapi['path']
path = ','+domain + path
temppath.append(path)
file.write(path)

2、创建spider2

scrapy genspider spider2 xxx.com  (执行之后在项目的spiders目录下会生成一个spider2.py的文件)

以下代码主要实现了获取interfaces_path的文件下的地址对应的内容

# # -*- coding: utf-8 -*-
import scrapy
from scraw_swagger.items import ScrawSwaggerItem
import json
from scraw_swagger import settings class Spider2Spider(scrapy.Spider):
name = 'spider2'
allowed_domains = ['xxx.com']
file = open('interfaces_path', 'r')
file = file.read()
list_files = []
files = file.split(',')
n = len(files)
for i in range(1, n):
file = files[i]
list_files.append(file)
start_urls = list_files def parse(self, response):
outitem = ScrawSwaggerItem()
out_interface = []
out_domain = []
out_method = []
out_param_name = []
out_data_type = []
out_param_required = []
# 调试代码
# filename = response.url.split("/")[-1]
# open('temp/'+filename, 'wb').write(response.body)
# ///////
response = response.body
response_dict = json.loads(response)
items = response_dict['apis']
items_len = len(items)
for j in range(0, items_len):
path = items[j]['path']
# interface组成list
operations = items[j]['operations'][0]
method = operations['method']
parameters = operations['parameters']
parameters_len = len(parameters)
param_name = []
param_required = []
data_type = []
for i in range(0, parameters_len):
name = parameters[i]['name']
param_name.append(name)
required = parameters[i]['required']
param_required.append(required)
type = parameters[i]['type']
data_type.append(type) out_interface.append(path)
interface_domain = settings.interface_domain
out_domain.append(interface_domain)
out_method.append(method)
out_data_type.append(data_type)
out_param_name.append(param_name)
out_param_required.append(param_required) outitem['interface'] = out_interface
outitem['domain'] = out_domain
outitem['method'] = out_method
outitem['param_name'] = out_param_name
outitem['param_required'] = out_param_required
outitem['data_type'] = out_data_type
yield outitem

3、settings.py文件

# -*- coding: utf-8 -*-

# Scrapy settings for scraw_swagger project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# https://doc.scrapy.org/en/latest/topics/settings.html
# https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
# https://doc.scrapy.org/en/latest/topics/spider-middleware.html interface_domain = 'test'
token = 'test' # 调试代码
# interface_domain = 'http://xxxx..net'
# token = 'xxxxxx'
# /////////////// BOT_NAME = 'scraw_swagger' SPIDER_MODULES = ['scraw_swagger.spiders']
NEWSPIDER_MODULE = 'scraw_swagger.spiders' FEED_EXPORT_ENCODING = 'utf-8' # Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'scraw_swagger (+http://www.yourdomain.com)' # Obey robots.txt rules
ROBOTSTXT_OBEY = True # Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32 # Configure a delay for requests for the same website (default: 0)
# See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16 # Disable cookies (enabled by default)
#COOKIES_ENABLED = False # Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False # Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
# 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
# 'Accept-Language': 'en',
#} # Enable or disable spider middlewares
# See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
# 'scraw_swagger.middlewares.ScrawSwaggerSpiderMiddleware': 543,
#} # Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
# 'scraw_swagger.middlewares.ScrawSwaggerDownloaderMiddleware': 543,
#} # Enable or disable extensions
# See https://doc.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
# 'scrapy.extensions.telnet.TelnetConsole': None,
#} # Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
'scraw_swagger.pipelines.ScrawSwaggerPipeline': 300,
# 'scraw_swagger.pipelines.MysqlTwistedPipline': 200,
} # Enable and configure the AutoThrottle extension (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False # Enable and configure HTTP caching (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage' MYSQL_HOST = ''
MYSQL_DBNAME = ''
MYSQL_USER = ''
MYSQL_PASSWD = ''
MYSQL_PORT = 3306

4、存入数据库。编写pipelines.py文件。提取返回的item,并将对应的字段存入数据库

# -*- coding: utf-8 -*-
import pymysql
from scraw_swagger import settings
from twisted.enterprise import adbapi
import pymysql.cursors
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html class ScrawSwaggerPipeline(object):
def process_item(self, item, spider):
try:
# 插入数据sql
sql = """
insert into interfaces (domain, interface, method, param_name ,data_type, param_required)
VALUES (%s, %s, %s, %s, %s, %s)
"""
domain = item['domain']
n = len(domain)
for i in range(0, n):
domain = str(item['domain'][i])
interface = str(item["interface"][i])
method = str(item["method"][i])
param_name = str(item["param_name"][i])
data_type = str(item["data_type"][i])
param_required = str(item["param_required"][i])
a = (domain, interface, method, param_name, data_type, param_required)
self.cursor.execute(sql, a)
self.connect.commit()
except Exception as error:
# 出现错误时打印错误日志
print(error)
# self.connect.close()
return item def __init__(self):
# 连接数据库
self.connect = pymysql.connect(
host=settings.MYSQL_HOST,
db=settings.MYSQL_DBNAME,
user=settings.MYSQL_USER,
passwd=settings.MYSQL_PASSWD,
port=settings.MYSQL_PORT,
charset='utf8',
use_unicode=True) # 通过cursor执行增删查改
self.cursor = self.connect.cursor()

swagger上的接口写入数据库的更多相关文章

  1. 开机后将sim/uim卡上的联系人写入数据库

    tyle="margin:20px 0px 0px; font-size:14px; line-height:26px; font-family:Arial; color:rgb(51,51 ...

  2. 通过POI实现上传EXCEL的批量读取数据写入数据库

    最近公司新增功能要求导入excel,并读取其中数据批量写入数据库.于是就开始了这个事情,之前的文章,记录了上传文件,本篇记录如何通过POI读取excel数据并封装为对象上传. 上代码: 1.首先这是一 ...

  3. c#上传文件并将word pdf转化成txt存储并将内容写入数据库

    c#上传文件并将word pdf转化成txt存储并将内容写入数据库 using System; using System.Data; using System.Configuration; using ...

  4. Django上传excel表格并将数据写入数据库

    前言: 最近公司领导要统计技术部门在各个业务条线花费的工时百分比,而 jira 当前的 Tempo 插件只能统计个人工时.于是就写了个报表工具,将 jira 中导出的个人工时excel表格 导入数据库 ...

  5. c#WebApi使用form表单提交excel,实现批量写入数据库

    思路:用户点击下载模板按钮,获取到excel模板,然后向里面填写数据保存.from表单提交的时候选择保存好的excel,实现数据的批量导入过程 先把模板放在服务器的项目目录下面:如 模板我一般放在:F ...

  6. 使用log4j让日志写入数据库

    之前做的一个项目有这么个要求,在日志管理系统里,需要将某些日志信息存储到数据库里,供用户.管理员查看分析.因此我就花了点时间搞了一下这一功能,各位请看. 摘要:我们知道log4j能提供强大的可配置的记 ...

  7. Excel 导入到Datatable 中,再使用常规方法写入数据库

    首先呢?要看你的电脑的office版本,我的是office 2013 .为了使用oledb程序,需要安装一个引擎.名字为AccessDatabaseEngine.exe.这里不过多介绍了哦.它的数据库 ...

  8. SQLBulkCopy使用实例--读取Excel写入数据库/将 Excel 文件转成 DataTable

    MS SQL Server 提供一个称为 bcp 的流行的命令提示符实用工具,用于将数据从一个表移动到另一个表(表可以在不同服务器上). SqlBulkCopy 类允许编写提供类似功能的托管代码解决方 ...

  9. thinkphp + 美图秀秀api 实现图片裁切上传,带数据库

    思路: 1.数据库 创建test2 创建表img,字段id,url,addtime 2.前台页: 1>我用的是bootstrap 引入必要的js,css 2>引入美图秀秀的js 3.后台: ...

随机推荐

  1. Linux命令的应用

    目录 Linux命令 Linux文件管理命令 用户管理 权限管理 vi文本编辑器 find查找命令 磁盘管理命令 压缩及解压 Linux 进程 Linux运行tomcat Linux安装mysql 卸 ...

  2. 【Android】修改Android Studio的SDK位置

    解决SDK占用C盘空间问题 由于Android Studio默认会将环境下载到C盘,会导致C盘空间被大量占用. 对于C盘窘迫的童鞋非常不友好. 可以通过修改SDK位置的方式缓解C盘空间焦虑. 打开&q ...

  3. PMP考位抢夺攻略(二)

    为什么会有第二篇文章呢,因为北京周边的考点太难抢了,都不是页面样式能不能展示的问题了!!! 如何在网页完全打不开的情况下报考PMP? 首先,自动登录. 打开浏览器,输入网址http://exam.ch ...

  4. 【CTF】CTFHub 技能树 文件头检查 writeup

    PHP一句话木马 <?php @eval($_POST["pass"]);?> <?php eval($_REQUEST["pass"]);? ...

  5. 网络编程Netty入门:Netty的启动过程分析

    目录 Netty的启动过程 Bootstrap 服务端的启动 客户端的启动 TCP粘包.拆包 图示 简单的例子 Netty编解码框架 Netty解码器 ByteToMessageDecoder实现类 ...

  6. 【全网首发】鸿蒙开源三方组件--强大的弹窗库XPopup组件

    目录: 1.介绍 2.效果一览 3.依赖 4.如何使用 5.下载链接 6.<鸿蒙开源三方组件>文章合集 1. 介绍 ​ XPopup是一个弹窗库,可能是Harmony平台最好的弹窗库.它从 ...

  7. F - Lakes in Berland(BFS)

    The map of Berland is a rectangle of the size n × m, which consists of cells of size 1 × 1. Each cel ...

  8. 一文抽丝剥茧带你掌握复杂Gremlin查询的调试方法

    摘要:Gremlin是图数据库查询使用最普遍的基础查询语言.Gremlin的图灵完备性,使其能够编写非常复杂的查询语句.对于复杂的问题,我们该如何编写一个复杂的查询?以及我们该如何理解已有的复杂查询? ...

  9. hdu1316 水大数

    题意:      给你一个区间,问这个区间有多少个斐波那契数. 思路:      水的大数,可以直接模拟,要是懒可以用JAVA,我模拟的,打表打到1000个就足够用了... #include<s ...

  10. Python中的Pandas模块

    目录 Pandas Series 序列的创建 序列的读取 DataFrame DataFrame的创建 DataFrame数据的读取 Panel Panel的创建 Pandas Pandas ( Py ...