python3爬虫环境搭建
安装python3
sudo apt-get install python3-dev build-essential libssl-dev libffi-dev libxml2 libxml2-dev libxslt1-dev zlib1g-dev
sudo apt-get install python3
sudo apt-get install ptyhon3-pip
ubuntu安装mongo
sudo apt-get install mongodb
mongod
mongo
> show dbs
admin (empty)
local 0.078GB
> use local
switched to db local
> db.test.insert({'a':'b'})
WriteResult({ "nInserted" : 1 })
mac安装mongo
brew install mongodb
brew services start mongodb
ubuntu安装redis
sudo apt-get install redis-server
redis-cli
127.0.0.1:6379> set 'a' 'b'
OK
127.0.0.1:6379> get 'a'
"b"
127.0.0.1:6379>
root@iZwz91u9ywulp6n78yyed6Z:~# sudo vi /etc/redis/redis.conf
requirepass myredisserver
sudo service redis restart
root@iZwz91u9ywulp6n78yyed6Z:~# redis-cli -a myredisserver
127.0.0.1:6379> get 'a'
"b"
127.0.0.1:6379>
mac 安装redis
brew install redis
qunzhudeMacBook-Air:/ qunzhupu$ brew services start redis
==> Successfully started `redis` (label: homebrew.mxcl.redis)
qunzhudeMacBook-Air:/ qunzhupu$ redis-cli
127.0.0.1:6379> set 'a' 'b'
OK
127.0.0.1:6379> get 'a'
"b"
cd /usr/local/etc
sudo vim redis.conf
requirepass foobared
ubuntu配置mysql
root@iZwz91u9ywulp6n78yyed6Z:~# cd /etc/mysql/
root@iZwz91u9ywulp6n78yyed6Z:/etc/mysql# ls
conf.d debian.cnf debian-start my.cnf my.cnf.fallback mysql.cnf mysql.conf.d
root@iZwz91u9ywulp6n78yyed6Z:/etc/mysql# cd mysql.conf.d/
root@iZwz91u9ywulp6n78yyed6Z:/etc/mysql/mysql.conf.d# ls
mysqld.cnf mysqld_safe_syslog.cnf
root@iZwz91u9ywulp6n78yyed6Z:/etc/mysql/mysql.conf.d# vi mysqld.cnf
root@iZwz91u9ywulp6n78yyed6Z:/etc/mysql/mysql.conf.d# sudo service mysql restart
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
FLUSH PRIVILEGES;
selenium库使用
>>> import selenium
>>> from selenium import webdriver
>>> driver - webdriver.Chrome()
>>> driver.get('http://www.baidu.com')
安装beautifulsoup4和pyquery
pip3 install beautifulsoup4
pip3 install pyquery
qunzhudeMacBook-Air:bin qunzhupu$ python3
Python 3.7.0 (default, Jun 29 2018, 20:13:13)
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyquery import PyQuery as pq
>>> dec =pq('<html></html>')
>>> dec =pq('<html>Hello</html>')
>>> result = dec('html').text()
>>> result
'Hello'
>>>
pymysql连接Mysql
qunzhudeMacBook-Air:bin qunzhupu$ python3
Python 3.7.0 (default, Jun 29 2018, 20:13:13)
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymysql
>>> conn = pymysql.connect(host='123.206.204.80',user='puqunzhu',password='123456',port=3306,db='puqunzhu')
>>> cursor = conn.cursor()
>>> cursor.execute('select * from article')
1
pymongo连接mongoDB
pip3 install pymongo
qunzhudeMacBook-Air:bin qunzhupu$ python3
Python 3.7.0 (default, Jun 29 2018, 20:13:13)
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymongo
>>> client = pymongo.MongoClient('localhost')
>>> db = client['newtestdb']
>>> db['table'].insert({'name':'Bob'})
__main__:1: DeprecationWarning: insert is deprecated. Use insert_one or insert_many instead.
ObjectId('5bc5b059e18a3620cc46bdb4')
>>> db['table'].find_one({'name':'Bob'})
{'_id': ObjectId('5bc5b059e18a3620cc46bdb4'), 'name': 'Bob'}
redis连接redis
pip3 install redis
qunzhudeMacBook-Air:bin qunzhupu$ python3
Python 3.7.0 (default, Jun 29 2018, 20:13:13)
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import redis
>>> r = redis.Redis('localhost',6379)
>>> r.set('name','puqunzhu')
True
>>> r.get('name')
b'puqunzhu'
flask代理获取
pip3 install flask
django
pip3 install django
jupyter
pip3 install jupyter
qunzhudeMacBook-Air:bin qunzhupu$ jupyter notebook
python3爬虫环境搭建的更多相关文章
- selenium+python爬虫环境搭建
前言: 准备使用selenium爬取网站数据,先搭建selenium+python爬虫环境搭建 系统环境: 64位win10系统,同时装python2.7和python3.6两个版本,IDE为pych ...
- Python基础之Windows下Python3.x环境搭建
本文介绍使用Python3.x搭建Python环境,文章最后将使用Windows自带的cmd写一个简单的Python程序. 下载Python环境安装包 在https:/www.python.org/下 ...
- python3之环境搭建以及基础语法
早些时候,有一段时间吧,为了想免费下载网易云的收费音乐,也是翻了下python的教程的,语法方面没细致地去看,跟java.php.javascript这些都大同小异,也是面向对象的.那么,近期准备快速 ...
- 云效（阿里云）流水线 + nginx + uWsgi + flask + python3 基础环境搭建 --备忘
一.开发环境搭建 1.安装python3 yum -y groupinstall "Development tools" yum -y install zlib-devel bzi ...
- Python3+selenium3环境搭建笔记
系统:win7 64位浏览器:ie9 64位 chrome70 32位 firefox63 64位python版本:3.6.5 Windows x86 executable installersele ...
- 2 Selenium3.0+Python3.6环境搭建
[说明] 再次搭建一次环境,是因为遇到怎么都打不开IE的问题了,环境信息为:Selenium3.0+Python3.6+win7+ie10 [搭建步骤] 1.下载Python3.6,并点击安装和配置环 ...
- 金玉良缘易配而木石前盟难得|M1 Mac os(Apple Silicon)天生一对Python3开发环境搭建(集成深度学习框架Tensorflow/Pytorch)
原文转载自「刘悦的技术博客」https://v3u.cn/a_id_189 笔者投入M1的怀抱已经有一段时间了,俗话说得好,但闻新人笑,不见旧人哭,Intel mac早已被束之高阁,而M1 mac已经 ...
- python——python3.6环境搭建（Windows10,64位）
1.python软件资源下载 1.1 打开python官网地址:https://www.python.org 1.2 根据自己电脑的设置选择下载合适的python3.6.2 1.3 此处选择windo ...
- vs2015 + Python3.5 环境搭建
1. vs2015只支持Python3.5及以前的版本,对应Anaconda3.4.2之前的版本. 2. 卸载掉所有安装过的Python 3. 建议重装VS2015, 因为增量升级Python Too ...
随机推荐
- scrapy-redis分布式爬取猫眼电影
能够利用redis缓存数据库的优点去重来避免数据的大面积冗余 1.首先就是要创建猫眼爬虫项目 2.进入项目内部创建一个爬虫文件 创建完文件之后就是要爬取的内容,我这边以爬取猫眼电影的title和lin ...
- Docker-Compose简介与Ubuntu Server 上安装Compose
Compose简介 Compose是Docker官方的开源项目,负责对Docker容器集群的快速编排. Compose是定义和运行多个Docker容器的应用. 举例来说: 一个项目除了Tomcat容器 ...
- CentOS7 安装 Pure-ftpd
博客地址:http://www.moonxy.com 一.摘要 FTP 是 File Transfer Protocol(文件传输协议)的英文简称,而中文简称为"文传协议”.用于Intern ...
- Linux 笔记 - 第四章 Linux 文件和目录管理
博客地址:http://www.moonxy.com 1. 绝对路径和相对路径 绝对路径:由根目录 "/" 写起的.如:/usr/local/mysql 相对路径:不是由根目录 & ...
- charles 启用/禁用断点
本文参考:charles 启用/禁用断点 1.3. enable/disable breakpoints 和 2.3 breakpoints settings 断点设置是常用的了,没啥好说的了,可以设 ...
- 6.Sentinel源码分析—Sentinel是如何动态加载配置限流的？
Sentinel源码解析系列: 1.Sentinel源码分析-FlowRuleManager加载规则做了什么? 2. Sentinel源码分析-Sentinel是如何进行流量统计的? 3. Senti ...
- 41 (OC)* OC的理解与动态特性
1:动态类型(Dynamic typing),动态绑定(Dynamic binding)和动态加载(Dynamic loading). 动态类型:即运行时再决定对象的类型.简单说就是id类型.id类型 ...
- git远程仓库常用命令
1. git add . 将工作区的文件推到暂存区: 2. git commit -m " 备注信息" 将暂存区内容提交 ...
- [VB.NET Tips]程序的启动和终止
当执行一个VB.NET应用程序时,CLR会把IL翻译成x86指令,并且寻找一个名为Main的方法. 并从该方法开始执行程序.Main方法也称为程序的"入口"(entry point ...
- Vue学习之不同组件之间的消息传递
<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8&quo ...