Python爬虫之PyQuery使用

PyQuery简介

pyquery能够通过选择器精确定位 DOM 树中的目标并进行操作。pyquery相当于jQuery的python实现,可以用于解析HTML网页等。它的语法与jQuery几乎完全相同,对于使用过jQuery的人来说很熟悉,也很好上手。

初始化

有 4 种方法可以进行初始化:
可以通过传入 字符串、lxml、文件 或者 url 来使用PyQuery

from pyquery import PyQuery as pq
from lxml import etree d = pq("<html></html>")#传入字符串
d = pq(etree.fromstring("<html></html>"))#传入lxml
d = pq(url='http://baidu.com/') #传入url
d = pq(filename=path_to_html_file) #传入文件

  

基本CSS选择器

html='''
<html>
<body>
<ul class="mh-col">
<li class="g-ellipsis">
<a class="g-a-noline" data-md='{"b":"list","p":"1-1"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E6%AD%8C%E6%89%8B%E9%AB%98%E7%A9%BA%E6%8B%8DMV%E5%9D%A0%E4%BA%A1&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
歌手高空拍MV坠亡
</a>
<span class="mh-ico-up">
</span>
</li>
<li class="g-ellipsis">
<a class="g-a-noline" data-md='{"b":"list","p":"1-2"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E4%BD%9B%E7%A5%96%E6%9C%B1%E9%BE%99%E5%B9%BF%E9%87%91%E5%A9%9A&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D&fr=hao_360so_history_b" target="_blank">
佛祖朱龙广金婚
</a>
<span class="mh-ico-down">
</span>
</li>
<li class="g-ellipsis">
<a class="g-a-noline" data-md='{"b":"list","p":"1-3"}' href="http://www.so.com/link?url=http%3A%2F%2Fbaike.so.com%2Fzt%2Fzufangfangpian.html%3Fsrc%3Dreci&q=%E5%91%A8%E6%9D%B0%E4%BC%A6%E7%9A%84%E6%AD%8C&ts=1540365398&t=c6890eee0e669832ca96d5223582d6e" target="_blank">
常见租房陷阱
</a>
<span class="mh-ico-up">
</span>
</li>
<li class="g-ellipsis">
<a class="g-a-noline" data-md='{"b":"list","p":"1-4"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E9%9D%B3%E4%B8%9C%E5%9B%9E%E5%BA%94%E5%8F%91%E9%94%99%E8%AF%97%E8%AF%8D&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
靳东回应发错诗词
</a>
</li>
<li class="g-ellipsis">
<a class="g-a-noline" data-md='{"b":"list","p":"1-5"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E7%85%A4%E8%80%81%E6%9D%BF%E4%BB%AC%E7%9A%84%E5%BD%B1%E8%A7%86%E6%B1%9F%E6%B9%96&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
煤老板们的影视江湖
</a>
</li>
<li class="g-ellipsis">
<a class="g-a-noline" data-md='{"b":"list","p":"1-6"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=1024%E7%A8%8B%E5%BA%8F%E5%91%98%E8%8A%82&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
1024程序员节
</a>
</li>
<li class="g-ellipsis">
<a class="g-a-noline" data-md='{"b":"list","p":"1-7"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E7%BE%8E%E7%9A%84%E5%90%88%E5%B9%B6%E5%B0%8F%E5%A4%A9%E9%B9%85&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
美的合并小天鹅
</a>
</li>
<li class="g-ellipsis">
<a class="g-a-noline" data-md='{"b":"list","p":"1-8"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E4%BA%AC%E6%98%86%E9%AB%98%E9%80%9F4%E8%BD%A6%E7%9B%B8%E6%92%9E&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
京昆高速4车相撞
</a>
</li> </ul>
</body>
</html>
'''

from pyquery import PyQuery as pq
doc = pq(html)

# 获取所有a标签
print(doc('body .mh-col li a'))


注意:
类名用.
id用#
标签用标签名

另外选择的是具有层级关系,从左到右,不是直接的父子的关系。

运行结果如下:

<a class="g-a-noline" data-md="{"b":"list","p":"1-1"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E6%AD%8C%E6%89%8B%E9%AB%98%E7%A9%BA%E6%8B%8DMV%E5%9D%A0%E4%BA%A1&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
歌手高空拍MV坠亡
</a>
<a class="g-a-noline" data-md="{"b":"list","p":"1-2"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E4%BD%9B%E7%A5%96%E6%9C%B1%E9%BE%99%E5%B9%BF%E9%87%91%E5%A9%9A&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D&fr=hao_360so_history_b" target="_blank">
佛祖朱龙广金婚
</a>
<a class="g-a-noline" data-md="{"b":"list","p":"1-3"}" href="http://www.so.com/link?url=http%3A%2F%2Fbaike.so.com%2Fzt%2Fzufangfangpian.html%3Fsrc%3Dreci&q=%E5%91%A8%E6%9D%B0%E4%BC%A6%E7%9A%84%E6%AD%8C&ts=1540365398&t=c6890eee0e669832ca96d5223582d6e" target="_blank">
常见租房陷阱
</a>
<a class="g-a-noline" data-md="{"b":"list","p":"1-4"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E9%9D%B3%E4%B8%9C%E5%9B%9E%E5%BA%94%E5%8F%91%E9%94%99%E8%AF%97%E8%AF%8D&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
靳东回应发错诗词
</a>
<a class="g-a-noline" data-md="{"b":"list","p":"1-5"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E7%85%A4%E8%80%81%E6%9D%BF%E4%BB%AC%E7%9A%84%E5%BD%B1%E8%A7%86%E6%B1%9F%E6%B9%96&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
煤老板们的影视江湖
</a>
<a class="g-a-noline" data-md="{"b":"list","p":"1-6"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=1024%E7%A8%8B%E5%BA%8F%E5%91%98%E8%8A%82&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
1024程序员节
</a>
<a class="g-a-noline" data-md="{"b":"list","p":"1-7"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E7%BE%8E%E7%9A%84%E5%90%88%E5%B9%B6%E5%B0%8F%E5%A4%A9%E9%B9%85&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
美的合并小天鹅
</a>
<a class="g-a-noline" data-md="{"b":"list","p":"1-8"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E4%BA%AC%E6%98%86%E9%AB%98%E9%80%9F4%E8%BD%A6%E7%9B%B8%E6%92%9E&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
京昆高速4车相撞
</a>

  

操作

html='''
<html>
<body>
<ul class="mh-col">
<li class="g-ellipsis1">
<a class="g-a-noline1" data-md='{"b":"list","p":"1-1"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E6%AD%8C%E6%89%8B%E9%AB%98%E7%A9%BA%E6%8B%8DMV%E5%9D%A0%E4%BA%A1&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
歌手高空拍MV坠亡
</a>
<span class="mh-ico-up">
</span>
</li>
<li class="g-ellipsis2">
<a class="g-a-noline2" data-md='{"b":"list","p":"1-2"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E4%BD%9B%E7%A5%96%E6%9C%B1%E9%BE%99%E5%B9%BF%E9%87%91%E5%A9%9A&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D&fr=hao_360so_history_b" target="_blank">
佛祖朱龙广金婚
</a>
<span class="mh-ico-down">
</span>
</li>
<li class="g-ellipsis3">
<a class="g-a-noline3" data-md='{"b":"list","p":"1-3"}' href="http://www.so.com/link?url=http%3A%2F%2Fbaike.so.com%2Fzt%2Fzufangfangpian.html%3Fsrc%3Dreci&q=%E5%91%A8%E6%9D%B0%E4%BC%A6%E7%9A%84%E6%AD%8C&ts=1540365398&t=c6890eee0e669832ca96d5223582d6e" target="_blank">
常见租房陷阱
</a>
<span class="mh-ico-up">
</span>
</li>
<li class="g-ellipsis4">
<a class="g-a-noline4" data-md='{"b":"list","p":"1-4"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E9%9D%B3%E4%B8%9C%E5%9B%9E%E5%BA%94%E5%8F%91%E9%94%99%E8%AF%97%E8%AF%8D&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
靳东回应发错诗词
</a>
</li>
<li class="g-ellipsis5">
<a class="g-a-noline5" data-md='{"b":"list","p":"1-5"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E7%85%A4%E8%80%81%E6%9D%BF%E4%BB%AC%E7%9A%84%E5%BD%B1%E8%A7%86%E6%B1%9F%E6%B9%96&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
煤老板们的影视江湖
</a>
</li>
<li class="g-ellipsis6">
<a class="g-a-noline6" data-md='{"b":"list","p":"1-6"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=1024%E7%A8%8B%E5%BA%8F%E5%91%98%E8%8A%82&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
1024程序员节
</a>
</li>
<li class="g-ellipsis7">
<a class="g-a-noline7" data-md='{"b":"list","p":"1-7"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E7%BE%8E%E7%9A%84%E5%90%88%E5%B9%B6%E5%B0%8F%E5%A4%A9%E9%B9%85&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
美的合并小天鹅
</a>
</li>
<li class="g-ellipsis8">
<a class="g-a-noline8" data-md='{"b":"list","p":"1-8"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E4%BA%AC%E6%98%86%E9%AB%98%E9%80%9F4%E8%BD%A6%E7%9B%B8%E6%92%9E&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
<p>新闻</p>
京昆高速4车相撞
</a>
</li> </ul>
</body>
</html>
'''
from pyquery import PyQuery as pq
doc = pq(html)
items = doc('.mh-col') #.find():查找嵌套元素
alist = items.find('li a')
print(alist)

#查找所有子元素
alist2 = items.children()
print(alist2) #查找指定的子元素
alist3 = items.children('.g-ellipsis1')
print(alist2)
#查找父元素 
#注意:一个元素只有一个父元素
body = items.parent()
print(body) #查找祖先元素
content = items.parents()
print(content) #查找兄弟元素
li = doc('.mh-col .g-ellipsis1')
print(li.siblings()) #遍历 单个元素
#遍历所有的a标签
alist =doc('.mh-col li a').items()
  for a in alist: print(a)

  

获取信息

获取属性
a =doc('.mh-col li .g-a-noline8')
print(a.attr['href'])
print(a.attr.href) 获取文本
a =doc('.mh-col li .g-a-noline8')
print(a.text()) 获取HTML
a =doc('.mh-col li .g-a-noline8')
print(a.html())

 

简单的DOM操作

#addClass、removerClass
#修改类名
a =doc('.mh-col li .g-a-noline8')
print(a)
a.removeClass('g-a-noline8')
print(a)
a.addClass('g-a-noline8')
print(a) #attr、css
#修改属性和样式
a =doc('.mh-col li .g-a-noline8')
print(a)
a.attr('name','link')
print(a)
a.css('font-size','14px')
print(a)
#remove
#删除标签
li = doc('.mh-col .g-ellipsis8')
print(li)
li.find('a').remove()
print(li)

  

更多的DOM操作:https://pyquery.readthedocs.io/en/latest/api.html

Python爬虫之PyQuery使用(六)的更多相关文章

  1. [Python爬虫] 之二十六:Selenium +phantomjs 利用 pyquery抓取智能电视网站图片信息

    一.介绍 本例子用Selenium +phantomjs爬取智能电视网站(http://www.tvhome.com/news/)的资讯信息,输入给定关键字抓取图片信息. 给定关键字:数字:融合:电视 ...

  2. python爬虫神器PyQuery的使用方法

    你是否觉得 XPath 的用法多少有点晦涩难记呢? 你是否觉得 BeautifulSoup 的语法多少有些悭吝难懂呢? 你是否甚至还在苦苦研究正则表达式却因为少些了一个点而抓狂呢? 你是否已经有了一些 ...

  3. Python爬虫小白入门(六)爬取披头士乐队历年专辑封面-网易云音乐

    一.前言 前文说过我的设计师小伙伴的设计需求,他想做一个披头士乐队历年专辑的瀑布图. 通过搜索,发现网易云音乐上有比较全的历年专辑信息加配图,图片质量还可以,虽然有大有小. 我的例子怎么都是爬取图片? ...

  4. python爬虫之PyQuery的基本使用

    PyQuery库也是一个非常强大又灵活的网页解析库,如果你有前端开发经验的,都应该接触过jQuery,那么PyQuery就是你非常绝佳的选择,PyQuery 是 Python 仿照 jQuery 的严 ...

  5. python爬虫之pyquery学习

    相关内容: pyquery的介绍 pyquery的使用 安装模块 导入模块 解析对象初始化 css选择器 在选定元素之后的元素再选取 元素的文本.属性等内容的获取 pyquery执行DOM操作.css ...

  6. 【Python爬虫】PyQuery解析库

    PyQuery解析库 阅读目录 初始化 基本CSS选择器 查找元素 遍历 获取信息 DOM操作 伪类选择器 PyQuery 是 Python 仿照 jQuery 的严格实现.语法与 jQuery 几乎 ...

  7. 爬取网易云音乐评论!python 爬虫入门实战(六)selenium 入门!

    说到爬虫,第一时间可能就会想到网易云音乐的评论.网易云音乐评论里藏了许多宝藏,那么让我们一起学习如何用 python 挖宝藏吧! 既然是宝藏,肯定是用要用钥匙加密的.打开 Chrome 分析 Head ...

  8. Python爬虫之pyquery库的基本使用

    # 字符串初始化 html = ''' <div> <ul> <li class = "item-0">first item</li> ...

  9. python爬虫知识点总结(六)BeautifulSoup库详解

    官方学习文档:https://www.crummy.com/software/BeautifulSoup/bs4/doc.zh/ 一.什么时BeautifulSoup? 答:灵活又方便的网页解析库,处 ...

随机推荐

  1. 阿里云服务器 ECS Linux 禁止IP 通过 SSH 登录

    这几天买的服务器老是受到黑客攻击被破解登录密码,今天修改了登录规则发现只有固定ip可以访问,其他ip即使有密码也无法登录我的服务器,但是能通过ip访问我的网站,哈哈. 限制 IP SSH 登录解决步骤 ...

  2. hashMap之jdk1.7和jdk1.8

    参考链接: http://allenwu.itscoder.com/hashmap-analyse https://tech.meituan.com/java-hashmap.html

  3. 2018-2019-2 网络对抗技术 20165227 Exp2 后门原理与实践

    2018-2019-2 网络对抗技术 20165227 Exp2 后门原理与实践 (1)例举你能想到的一个后门进入到你系统中的可能方式? 接收邮件的方式 (2)例举你知道的后门如何启动起来(win及l ...

  4. ARMV8 datasheet学习笔记3:AArch64应用级体系结构

    1.前言 本文主要从应用的角度介绍ARMV8的编程模型和存储模型 2. AArch64应用级编程模型 从应用的角度看到的ARM处理器元素: 可见的元素(寄存器/指令) 说明 可见的寄存器 R0-R30 ...

  5. V4L2学习记录【转】

    转自:http://blog.chinaunix.net/uid-30254565-id-5637600.html V4L2学习记录 这个还没有分析完,先在这放着,防止电脑坏掉丢了,以后再完善 V4L ...

  6. 如果你的ie内核浏览器总是缓冲数据的话

    如果你的ie内核浏览器总是缓冲数据的话 运行cmd,输入netsh winsock reset wincock是支持多种协议的网络编程接口 因为ie内核的浏览器的一些设置和插件可能会被其他软件篡改,所 ...

  7. MySQL占用IO过高解决方案【转】

    1.日志产生的性能影响: 由于日志的记录带来的直接性能损耗就是数据库系统中最为昂贵的IO资源.MySQL的日志包括错误日志(ErrorLog),更新日志(UpdateLog),二进制日志(Binlog ...

  8. wget安装pip和pip3

    pip的安装 1.1 pip下载 wget "https://pypi.python.org/packages/source/p/pip/pip-1.5.4.tar.gz#md5=834b2 ...

  9. Node.jsでMySQLを使うメモ

    インストール npm install mysql コネクション var mysql = require('mysql'); var connection = mysql.createConnectio ...

  10. C:详解C中volatile关键字

    原文地址:http://www.cnblogs.com/yc_sunniwell/archive/2010/06/24/1764231.html volatile提醒编译器它后面所定义的变量随时都有可 ...