爬虫之pyquery库

官方文档：https://pyquery.readthedocs.io/en/latest/

PyQuery是一个强大又灵活的网页解析库。如果你觉得正则写起来太麻烦、BeautifulSoup语法太难记，而你熟悉jQury的语法，那么PyQuery就是你的绝佳选择。

一、开始

字符串初始化：

from pyquery import PyQuery as pq

d = pq("<html>哈哈哈</html>")  # 现在d就相当于jQuery的$

print(d("html"))

URL初始化：

from pyquery import PyQuery as pq

d = pq(url="https://www.baidu.com")

print(d("head"))

文件初始化：

from pyquery import PyQuery as pq

d = pq(filename='demo.html')  # filename指定文件路径

print(d("head"))

二、基本CSS选择器

html = """

<div id="container">

    <ul class="list">

         <li class="item-0">first item</li>

         <li class="item-1"><a href="link2.html">second item</a></li>

         <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

         <li class="item-1 active"><a href="link4.html">fourth item</a></li>

         <li class="item-0"><a href="link5.html">fifth item</a></li>

     </ul>

 </div>

"""

from pyquery import PyQuery as pq

d = pq(html)

print(d("#container .list li"))

三、查找元素

子元素

d("css选择器").find("li")

html = """

<div id="container">

    <ul class="list">

         <li class="item-0">first item</li>

         <li class="item-1"><a href="link2.html">second item</a></li>

         <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

         <li class="item-1 active"><a href="link4.html">fourth item</a></li>

         <li class="item-0"><a href="link5.html">fifth item</a></li>

     </ul>

 </div>

"""

from pyquery import PyQuery as pq

d = pq(html)

items = d(".list")

print(type(items))  # <class 'pyquery.pyquery.PyQuery'>

li = items.find("li")

print(type(li))    # <class 'pyquery.pyquery.PyQuery'>

print(li)

"""

 <li class="item-0">first item</li>

 <li class="item-1"><a href="link2.html">second item</a></li>

 <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

 <li class="item-1 active"><a href="link4.html">fourth item</a></li>

 <li class="item-0"><a href="link5.html">fifth item</a></li>

"""

父元素

d("css选择器").parent(<css选择器(可无)>)

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

items = d(".list")

parents = items.parents()

print(parents)

"""

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

        </ul>

    </div>

</div>

<div id="container">

    <ul class="list">

         <li class="item-0">first item</li>

         <li class="item-1"><a href="link2.html">second item</a></li>

         <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

         <li class="item-1 active"><a href="link4.html">fourth item</a></li>

         <li class="item-0"><a href="link5.html">fifth item</a></li>

    </ul>

</div>

"""

d(".list").parents()

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

items = d(".list")

parents = items.parents(".wrap")

print(parents)

"""

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

        </ul>

    </div>

</div>

"""

d(".list").parents(".wrap")

兄弟元素

d("css选择器").siblings(<css选择器(可无)>)

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

li = d(".list .item-0.active")

print(li.siblings())

"""

<li class="item-1"><a href="link2.html">second item</a></li>

<li class="item-0">first item</li>

<li class="item-1 active"><a href="link4.html">fourth item</a></li>

<li class="item-0"><a href="link5.html">fifth item</a></li>

"""

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

li = d(".list .item-0.active")

print(li.siblings(".active"))

"""

<li class="item-1 active"><a href="link4.html">fourth item</a></li>

"""

四、遍历

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

li = d("li").items()

print(type(li))  # <class 'generator'>

for i in li:

    print(i)

"""

<li class="item-0">first item</li>

<li class="item-1"><a href="link2.html">second item</a></li>

<li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

<li class="item-1 active"><a href="link4.html">fourth item</a></li>

<li class="item-0"><a href="link5.html">fifth item</a></li>

"""

五、获取信息

获取属性

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

a = d(".item-0.active a")

print(a.attr("href"))

print(a.attr.href)

获取文本

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

a = d(".item-0.active a")

print(a.text())

"""

third item

"""

获取html

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

li = d(".item-0.active")

print(li)

print(li.html())

"""

<li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

<a href="link3.html"><span class="bold">third item</span></a>

"""

六、DOM操作

addClass()、removeClass()

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

li = d(".item-0.active")

print(li)

li.removeClass("active")

print(li)

li.addClass("active")

print(li)

"""

<li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

<li class="item-0"><a href="link3.html"><span class="bold">third item</span></a></li>

<li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

"""

attr()、css()

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

         </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

li = d(".item-0.active")

print(li)

li.attr("name", "link")

print(li)

li.css("font-size", "14px")

print(li)

"""

<li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

<li class="item-0 active" name="link"><a href="link3.html"><span class="bold">third item</span></a></li>

<li class="item-0 active" name="link" style="font-size: 14px"><a href="link3.html"><span class="bold">third item</span></a></li>

"""

remove()

html = """

<div class="wrap">

    Hello, World.

    <p>This is a paragraph.</p>

 </div>

"""

from pyquery import PyQuery as pq

d = pq(html)

wrap = d(".wrap")

print(wrap.text())

"""

Hello, World.

This is a paragraph.

"""

wrap.find("p").remove()

print(wrap.text())  # Hello, World.

其他DOM方法

https://pyquery.readthedocs.io/en/latest/api.html

七、伪类选择器

html = """

<div class="wrap">

    <div id="container">

        <ul class="list">

             <li class="item-0">first item</li>

             <li class="item-1"><a href="link2.html">second item</a></li>

             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>

             <li class="item-1 active"><a href="link4.html">fourth item</a></li>

             <li class="item-0"><a href="link5.html">fifth item</a></li>

        </ul>

    </div>

</div>

"""

from pyquery import PyQuery as pq

d = pq(html)

li = d("li:first-child")

print(li)  # <li class="item-0">first item</li>

li = d("li:last-child")

print(li)  # <li class="item-0"><a href="link5.html">fifth item</a></li>

li = d("li:nth-child(2)")

print(li)  # <li class="item-1"><a href="link2.html">second item</a></li>

li = d("li:gt(2)")  # 从0开始计数，索引大于2

print(li)

"""

<li class="item-1 active"><a href="link4.html">fourth item</a></li>

 <li class="item-0"><a href="link5.html">fifth item</a></li>

"""

li = d("li:nth-child(2n)")  # 获取偶数顺序的元素（从1开始）

print(li)

"""

<li class="item-1"><a href="link2.html">second item</a></li>

<li class="item-1 active"><a href="link4.html">fourth item</a></li>

"""

li = d("li:contains(second)")  # 根据文本匹配，匹配文本包含second的标签

print(li)  # <li class="item-1"><a href="link2.html">second item</a></li>

爬虫之pyquery库的更多相关文章

Python爬虫之pyquery库的基本使用
# 字符串初始化 html = ''' <div> <ul> <li class = "item-0">first item</li> ...
python爬虫从入门到放弃（七）之 PyQuery库的使用
PyQuery库也是一个非常强大又灵活的网页解析库,如果你有前端开发经验的,都应该接触过jQuery,那么PyQuery就是你非常绝佳的选择,PyQuery 是 Python 仿照 jQuery 的严 ...
爬虫常用库之pyquery 库
pyquery库是jQuery的Python实现,可以用于解析HTML网页内容,我个人写过的一些抓取网页数据的脚本就是用它来解析html获取数据的.他的官方文档地址是:http://packages. ...
Python爬虫-- PyQuery库
PyQuery库 PyQuery库也是一个非常强大又灵活的网页解析库,PyQuery 是 Python 仿照 jQuery 的严格实现.语法与 jQuery 几乎完全相同,所以不用再去费心去记一些奇怪 ...
PYTHON 爬虫笔记六:PyQuery库基础用法
知识点一:PyQuery库详解及其基本使用初始化字符串初始化 html = ''' <div> <ul> <li class="item-0"&g ...
第四节：Web爬虫之pyquery解析库
PyQuery库也是一个非常强大又灵活的网页解析库,如果你有前端开发经验的,都应该接触过jQuery,那么PyQuery就是你非常绝佳的选择,PyQuery 是 Python 仿照 jQuery 的严 ...
python之爬虫（九）PyQuery库的使用
PyQuery库也是一个非常强大又灵活的网页解析库,如果你有前端开发经验的,都应该接触过jQuery,那么PyQuery就是你非常绝佳的选择,PyQuery 是 Python 仿照 jQuery 的严 ...
Python3 网络爬虫（请求库的安装）
Python3 网络爬虫(请求库的安装) 爬虫可以简单分为几步:抓取页面,分析页面和存储数据在页面爬取的过程中我们需要模拟浏览器向服务器发送请求,所以需要用到一些python库来实现HTTP的请求操 ...
爬虫之PyQuery的base了解
爬虫之PyQuery的base了解 pyquery库是jQuery的Python实现,能够以jQuery的语法来操作解析 HTML 文档,易用性和解析速度都很好,和它差不多的还有BeautifulSo ...

随机推荐

龙尚3G、4G模块嵌入式Linux系统使用说明【转】
本文转载自;http://blog.csdn.net/zqixiao_09/article/details/52506812 驱动部分: 1.kernle/drivers/usb/serial/opt ...
HDU 1379:DNA Sorting
DNA Sorting Time Limit: 2000/1000 MS (Java/Others) Memory Limit: 65536/32768 K (Java/Others) Tota ...
【POJ 3070】 Fibonacci
[题目链接] 点击打开链接 [算法] 矩阵乘法快速幂 [代码] #include <algorithm> #include <bitset& ...
gitlab https
https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/installation.md#using-https https:// ...
安装Git（转载）
转自:http://www.liaoxuefeng.com/wiki/0013739516305929606dd18361248578c67b8067c8c017b000/00137396287703 ...
MySQL索引使用以及优化
优化后台业主评价服务人员运行缓慢. 案发现场:后台业主评价服务人员列表页以及搜索页运行缓慢.运行时间为24074ms. 排查过程: 1.代码开头加时间,结束加时间.看运行了多少秒. 2.给评价 ...
Linux 搭建Discuz论坛
title: Linux 搭建Discuz论坛 Welcome to Fofade's Blog! 这里是Linux 搭建论坛的一些命令记录命令摘记: 下载文件:Discuz 安装环境:PHP Ap ...
Linux之线程相关命令及常用命令
查进程 top命令:查看系统的资源状况.#top top -d 10 //指定系统更新进程的时间为10秒 ps:查看当前用户的活动进程.#ps -A ps命令查找与进程相关的PID号: ps ...
ASP.NET Core&EF 笔记
首先创建Asp.net Core项目,然后通过 NuGet 安装 EntityFrameworkCore: Microsoft.EntityFrameworkCore.SqlServer Micros ...
生成清除某个数据库下的所有表的SQL语句
方法1:重建库和表用mysqldump --no-data把建表SQL导出来,然后drop database再create database,执行一下导出的SQL文件: 方法2:生成清空所有表的SQ ...

爬虫之pyquery库

一、开始

二、基本CSS选择器

三、查找元素

子元素

父元素

兄弟元素

四、遍历

五、获取信息

获取属性

获取文本

获取html

六、DOM操作

addClass()、removeClass()

attr()、css()

remove()

其他DOM方法

七、伪类选择器

爬虫之pyquery库的更多相关文章

随机推荐

热门专题