爬虫--PyQuery
什么是PyQuery？
PyQuery
初始化
字符串初始化
from pyquery import PyQuery as pq html="""
<div>
<ul>
<li class="item-0">first item</li>
<li class="item-1">
<a href="link2.html">second item</a>
</li>
<li class="item-0 active">
<a href="link3.html>
<span class="bold">third item</span>
</a>
</li>
<li class="item-1 active">
<a href="link4.html">f
ourth item
</a>
</li>
<li class="item-0">
<a href="link5.html">
fifth item
</a>
</li>
</ul>
</div>
""" doc=pq(html)
print(doc("li"))
<li class="item-0">first item</li>
<li class="item-1">
<a href="link2.html">second item</a>
</li>
<li class="item-0 active">
<a href="link3.html> <span class=" bold="">third item
</a>
</li>
<li class="item-1 active">
<a href="link4.html">f
ourth item
</a>
</li>
<li class="item-0">
<a href="link5.html">
fifth item
</a>
</li>
打印后的结果为：
URL初始化
from pyquery import PyQuery as pq doc = pq(url="http://www.baidu.com")
print(doc("head"))
<head><meta http-equiv="content-type" content="text/html;charset=utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=Edge"/><meta content="always" name="referrer"/><link rel="stylesheet" type="text/css" href="http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css"/><title>ç¾åº¦ä¸ä¸ï¼ä½ å°±ç¥é</title></head>
打印后的结果为：
文件初始化
from pyquery import PyQuery as pq doc = pq(filename="demo.html")
print(doc("li"))
基本CSS选择器
from pyquery import PyQuery as pq html= """
<div id = "container">
<ul>
<li class="item-0">
first item
</li>
<li class="item-1">
<a href="link2.html">second item</a>
</li>
<li class="item-2 active">
<a href="link3.html>
<span class="bold">third item</span>
</a>
</li>
<li class="item-3 active">
<a href="link4.html">f
ourth item
</a>
</li>
<li class="item-4">
<a href="link5.html">
fifth item
</a>
</li>
</ul>
</div>
"""
doc = pq(html)
print(doc("#container .item-0"))
<li class="item-0">
first item
</li>
打印的结果为：
查找元素
子元素
from pyquery import PyQuery as pq html= """
<div id = "container">
<ul class="list>
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
"""
doc = pq(html)
items = doc(".list")
print(type(items))
docs = doc.find("li")
print(type(docs))
print(docs)
<class 'pyquery.pyquery.PyQuery'>
<class 'pyquery.pyquery.PyQuery'>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
打印后的结果为：
from pyquery import PyQuery as pq html= """
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
"""
doc = pq(html)
items = doc(".list")
docs = items.children() # 查找所有的直接子元素
docs1 = items.children(".active")
print(docs)
print(docs1)
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
打印后的结果为：
父元素
from pyquery import PyQuery as pq html= """
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
"""
doc = pq(html)
items = doc(".list")
docs = items.parent() # 查找所有的直接父元素
print(docs)
print(type(docs))
<div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div> <class 'pyquery.pyquery.PyQuery'>
打印后的结果：
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".list")
docs = items.parents() # 查找所有的直接子元素
print(type(docs))
print(docs)
<class 'pyquery.pyquery.PyQuery'>
<html><body><div class="wrap">
<div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
</div></div></body></html><body><div class="wrap">
<div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
</div></div></body><div class="wrap">
<div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
</div></div><div id="container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
打印后的结果为：
兄弟元素
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-0")
print(type(items.siblings()))
print(items.siblings())
<class 'pyquery.pyquery.PyQuery'>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
打印后的结果为：
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-0active") # 同时匹配.item-0和active，若都包含item-0 + active则打印，否则不打印
item = doc(".item-0")
print(item.siblings())
print(type(items.siblings()))
print(items.siblings())
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li> <class 'pyquery.pyquery.PyQuery'>
打印后的结果为：
遍历
单个元素
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-0.active") # 同时匹配.item-0和active，若都包含item-0 + active则打印，否则不打印
print(type(items))
print(items)
<class 'pyquery.pyquery.PyQuery'>
<li class="item-0 active">first item</li>
打印后的结果为：
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc("li").items() # 遍历items ,此时的items是一个迭代器
print(type(items))
for li in items:
print(li)
<class 'generator'>
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class=" bold="">third item</a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
打印后的结果为：
获取信息
获取属性
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1 a")
print(items.attr("href"))
print(items.attr.href)
link2.html
link2.html
打印后的结果为：
获取文本
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items.text())
second item
打印后的结果为：
获取html
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items)
print(items.html())
<li class="item-1"><a href="link2.html">second item</a></li> <a href="link2.html">second item</a>
打印后的结果为：
DOM操作
addClass , removeClass
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1 active"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items.remove_class("active"))
print(items.add_class("actives"))
<li class="item-1"><a href="link2.html">second item</a></li> <li class="item-1 actives"><a href="link2.html">second item</a></li>
打印后的结果为：
attr , css
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1 active"><a href="link2.html">second item</a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items.attr("name","names"))
print(items.css("font-size","14px"))
<li class="item-1 active" name="names"><a href="link2.html">second item</a></li> <li class="item-1 active" name="names" style="font: 14px"><a href="link2.html">second item</a></li>
打印后的结果为：
remove
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
items = doc(".item-1")
print(items.text())
print("---------------")
items.find('p').remove()
print(items.text())
second item
Third times
---------------
second item
打印后的结果为：
其他DOM方法
https://pythonhosted.org/pyquery/api.html
伪类选择器
from pyquery import PyQuery as pq html= """
<div class = "wrap">
<div id = "container">
<ul class="list">
<li class="item-0 active">first item</li>
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
<li class="item-2 active"><a href="link3.html><span class="bold">third item</span></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
</ul>
</div>
<div>
""" doc = pq(html)
li = doc("li:first-child")
print(li)
print("------------------------------------------------------------")
li = doc("li:last-child")
print(li)
print("------------------------------------------------------------")
li = doc("li:gt(2)")
print(li)
print("------------------------------------------------------------")
li = doc("li:nth-child(2)") # 指定一个索引顺序，获取第二个li标签
print(li)
print("------------------------------------------------------------")
li = doc("li:nth-child(2n)") # 指定一个索引顺序，获取偶数的li标签
print(li)
print("------------------------------------------------------------")
li = doc("li:contains(second)") # 查找包含second的文本标签
print(li)
<li class="item-0 active">first item</li>
------------------------------------------------------------
<li class="item-4"><a href="link5.html">fifth item</a></li>
------------------------------------------------------------
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
<li class="item-4"><a href="link5.html">fifth item</a></li>
------------------------------------------------------------
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
------------------------------------------------------------
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
<li class="item-3 active"><a href="link4.html">fourth item</a></li>
------------------------------------------------------------
<li class="item-1 active"><a href="link2.html">second item<p>Third times</p></a></li>
打印后的结果为：
更多的CSS选择器可以查看：http://www.w3school.com.cn/css/index.asp
Pyquery的官方文档可以查看：https://pythonhosted.org/pyquery/api.html
爬虫--PyQuery的更多相关文章
- 一起学爬虫——PyQuery常用用法总结
什么是PyQuery PyQuery是一个类似于jQuery的解析网页工具,使用lxml操作xml和html文档,它的语法和jQuery很像.和XPATH,Beautiful Soup比起来,PyQu ...
- Python爬虫-- PyQuery库
PyQuery库 PyQuery库也是一个非常强大又灵活的网页解析库,PyQuery 是 Python 仿照 jQuery 的严格实现.语法与 jQuery 几乎完全相同,所以不用再去费心去记一些奇怪 ...
- 爬虫--pyquery使用
强大又灵活的网页解析库. 初始化 字符串初始化 html = ''' <div> <ul> <li class="item-0">first ...
- Python新手需要掌握的知识点
一.基础语法 1 变量 2 逻辑判断 3 循环 4 函数 二.数据结构 1 数字(加减乘除) 2 字符串(一串字符) 3 布尔 (真假) 4 元组 (不能修改的列表) 5 列表(Python的苦力,最 ...
- 一起学爬虫——使用selenium和pyquery爬取京东商品列表
layout: article title: 一起学爬虫--使用selenium和pyquery爬取京东商品列表 mathjax: true --- 今天一起学起使用selenium和pyquery爬 ...
- Python爬虫之PyQuery使用（六）
Python爬虫之PyQuery使用 PyQuery简介 pyquery能够通过选择器精确定位 DOM 树中的目标并进行操作.pyquery相当于jQuery的python实现,可以用于解析HTML网 ...
- Python爬虫利器六之PyQuery的用法
前言 你是否觉得 XPath 的用法多少有点晦涩难记呢? 你是否觉得 BeautifulSoup 的语法多少有些悭吝难懂呢? 你是否甚至还在苦苦研究正则表达式却因为少些了一个点而抓狂呢? 你是否已经有 ...
- PYTHON 爬虫笔记十:利用selenium+PyQuery实现淘宝美食数据搜集并保存至MongeDB（实战项目三）
利用selenium+PyQuery实现淘宝美食数据搜集并保存至MongeDB 目标站点分析 淘宝页面信息很复杂的,含有各种请求参数和加密参数,如果直接请求或者分析Ajax请求的话会很繁琐.所以我们可 ...
- 小白学 Python 爬虫（23）：解析库 pyquery 入门
人生苦短,我用 Python 前文传送门: 小白学 Python 爬虫(1):开篇 小白学 Python 爬虫(2):前置准备(一)基本类库的安装 小白学 Python 爬虫(3):前置准备(二)Li ...
随机推荐
- CodeForces Round #527 (Div3) D2. Great Vova Wall (Version 2)
http://codeforces.com/contest/1092/problem/D2 Vova's family is building the Great Vova Wall (named b ...
- windows下面安装redis
一.下载windows版本的Redis 链接:https://pan.baidu.com/s/1i6X2klv 密码:j4pi 二.安装Redis 这里下载的是Redis-x64-3.2.100版本, ...
- SQL优化套路
直奔主题: (1) SELECT(2) DISTINCT <SELECT_LIST>(3) FROM <LEFT_TABLE> <JOIN_TYPE> (4) JO ...
- 动画中的id与class使用css3的优先级问题
今天在做一个项目,用zepto给元素增加一个class,class里面有transform的效果.开始的时候,元素的样式是用id选择器写的,但是增加class之后,发现动画效果出不来,当时头好晕没想出 ...
- 【前端学习笔记】利用iframe实现表单的无刷新提交案例
<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8&quo ...
- 第207天：HTTP协议头字段详解大全
本篇重点介绍一下HTTP常用的Header HTTP Header非常之多,很少有人能完全分清这些Header到底是干什么的.鉴于RFC文件规范艰深晦涩难懂,本文对协议规范中列出的HTTP Heade ...
- UVA12546_LCM Pair Sum
题目的意思是求 [西伽马(p+q)]其中lcm(p,q)=n. 又见数论呀. 其实这个题目很简单,考虑清楚了可以很简单的方法飘过. 我一开始是这样来考虑的. 对于每一个单独的质因子,如果为p,它的次数 ...
- 程序猿必备技能：数据库管理——关于MySQL
一.初识MySQL 1.什么是数据库? 数据库(Database,DB)简而言之就是存放数据的仓库,是为了实现一定目的,按照某种规则组织起来的数据的集合. 2.使用数据库的必要性 (1)结构化存储大量 ...
- ORZ hzwer——OI省选算法汇总
简单列了一点 1.1 基本数据结构 1. 数组 2. 链表,双向链表 3. 队列,单调队列,双端队列 4. 栈,单调栈 1.2 中级数据结构 1. 堆 2. 并查集与带权并查集 3. hash 表 自 ...
- 【hdu6093】Rikka with Number
多校第五場的題. 首先是一個好數只在某個進制下,不會是在兩個進制下都爲好數. 另外每個進制好數的個數爲d!-(d-1)!,因爲要保證第一位不爲0. 然後就是在臨界進制下有多少個好數的問題,可以變成兩個 ...