爬虫——BeautifulSoup4解析器
BeautifulSoup用来解析HTML比较简单，API非常人性化，支持CSS选择器、Python标准库中的HTML解析器，也支持lxml的XML解析器。
其相较与正则而言，使用更加简单。
示例：
首先必须要导入bs4库
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") # 格式化输出 soup 对象的内容
print(soup.prettify())
运行结果
<html>
<head>
<title>
The Dormouse's story
</title>
</head>
<body>
<p class="title" name="dromouse">
<b>
The Dormouse's story
</b>
</p>
<p class="story">
Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">
<!-- Elsie -->
</a>
,
<a class="sister" href="http://example.com/lacie" id="link2">
Lacie
</a>
and
<a class="sister" href="http://example.com/tillie" id="link3">
Tillie
</a>
;
and they lived at the bottom of a well.
</p>
<p class="story">
...
</p>
</body>
</html>
四大对象种类
BeautifulSoup将复杂的HTML文档转换成一个复杂的树形结构，每个节点都是Python对象，所有对象可以归纳为4种：
- Tag
- NavigableString
- BeautifulSoup
- Comment
1.Tag
Tag 通俗点讲就是HTML中的一个个标签，例如：
<head><title>The Dormouse's story</title></head>
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
上面title head a p 等等HTML标签加上里面包括的内容就是Tag，那么试着使用BeautifulSoup来获取Tags：
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") # # 打印title标签
print(soup.title) # 打印head标签
print(soup.head) # 打印a标签
print(soup.a) # 打印p标签
print(soup.p) # 打印soup.p的类型
print(type(soup.p))
运行结果
<title>The Dormouse's story</title>
<head><title>The Dormouse's story</title></head>
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<class 'bs4.element.Tag'>
我们可以利用soup加标签名轻松地获取这些标签内容，这些对象的类型是bs4.element.Tag。但是注意，它查找的是在所有内容中的第一个符合要求的标签。如果需要查询所有的标签，后面会进行介绍。
对于Tag，它有两个重要的属性，就是name和attrs
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") # soup对象比较特殊，它的name为[document]
print(soup.name) # 对于其他内部标签，输出的值便为标签本身的名称
print(soup.head.name) # 打印p标签的所有属性，其类型是一个字典
print(soup.p.attrs) # 打印p标签的class属性
print(soup.p['class'])
# 还可以利用get方法获取属性，传入属性的名称，与上面的方法等价
print(soup.p.get('class')) print(soup.p) # 修改属性
soup.p['class'] = "newClass"
print(soup.p) # 删除属性
del soup.p['class']
print(soup.p)
运行结果
[document]
head
{'class': ['title'], 'name': 'dromouse'}
['title']
['title']
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="newClass" name="dromouse"><b>The Dormouse's story</b></p>
<p name="dromouse"><b>The Dormouse's story</b></p>
2.NavigableString
既然我们已经得到了标签的内容，那么问题来了，我们想要获取标签内部的文字怎么办呢？很简单，用.string即可，例如：
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") # 打印p标签的内容
print(soup.p.string) # 打印soup.p.string的类型
print(type(soup.p.string))
运行结果
The Dormouse's story
<class 'bs4.element.NavigableString'>
3.BeautifulSoup
BeautifulSoup对象表示的是一个文档的内容。大部分时候，可以把它当作Tag对象，是一个特殊的Tag，我们可以分别获取它的类型，名称，以及属性
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") # 类型
print(type(soup.name)) # 名称
print(soup.name) # 属性
print(soup.attrs)
运行结果
<class 'str'>
[document]
{}
4.Comment
Comment对象是一个特殊类型的NavigableString对象，其输出的内容不包括注释符号。
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") print(soup.a) print(soup.a.string) print(type(soup.a.string))
运行结果
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>
Elsie
<class 'bs4.element.Comment'>
a标签里的内容实际上是注释，但是如果我们利用.string来输出它的内容时，注释符号已经去掉了。
遍历文档树
1.直接子节点：.contents .children属性
.content
Tag的.content属性可以将Tag的子节点以列表的方式输出
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") # 输出方式为列表
print(soup.head.contents) print(soup.head.contents[0])
运行结果
[<title>The Dormouse's story</title>]
<title>The Dormouse's story</title>
.children
它返回的不是一个列表，不过我们可以通过遍历获取所有的子节点。
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") # 输出方式为列表生成器对象
print(soup.head.children) # 通过遍历获取所有子节点
for child in soup.head.children:
print(child)
运行结果
<list_iterator object at 0x008FF950>
<title>The Dormouse's story</title>
2.所有子孙节点：.descendants属性
上面讲的.contents和.children属性仅包含Tag的直接子节点，.descendants属性可以对所有Tag的子孙节点进行递归循环，和children类似，我们也需要通过遍历的方式获取其中的内容。
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") # 输出方式为列表生成器对象
print(soup.head.descendants) # 通过遍历获取所有子孙节点
for child in soup.head.descendants:
print(child)
运行结果
<generator object descendants at 0x00519AB0>
<title>The Dormouse's story</title>
The Dormouse's story
3.节点内容：.string属性
如果Tag只有一个NavigableString类型子节点，那么这个Tag可以使用.string得到子节点。如果一个Tag仅有一个子节点，那么这个Tab也可以使用.string方法，输出结果与当前唯一子节点的.string结果相同。
通俗点来讲就是：如果一个标签里面没有标签了，那么.string就会返回标签里面的内容。如果标签里面只有唯一的一个标签了，那么.string也会返回里面的内容。例如：
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") print(soup.head.string) print(soup.head.title.string)
运行结果
The Dormouse's story
The Dormouse's story
搜索文档树
1.find_all(name, attrs, recursive, text, **kwargs)
1)name参数
name参数可以查找所有名字为name的Tag，字符串对象会被自动忽略掉
a.传字符串
最简单的过滤器就是字符串，在搜索方法中传入一个字符串参数，Beautiful Soup会查找与字符串完整匹配所有的内容，返回一个列表。
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") print(soup.find_all("b")) print(soup.find_all("a"))
运行结果
[<b>The Dormouse's story</b>]
[<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
B.传正则表达式
如果传入正则表达式作为参数，Beautiful Soup会通过正则表达式match()来匹配内容
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup
import re html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") for tag in soup.find_all(re.compile("^b")):
print(tag.name)
运行结果
body
b
C.传列表
如果传入列表参数，Beautiful Soup会将与列表中任一元素匹配的内容以列表方式返回
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") print(soup.find_all(['a', 'b']))
2)keyword参数
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") print(soup.find_all(id="link1"))
运行结果
[<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>]
3)text参数
通过text参数可以搜索文档中的字符串内容，与name参数的可选值一样，text参数接受字符串，正则表达式，列表
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup
import re html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") # 字符串
print(soup.find_all(text = " Elsie ")) # 列表
print(soup.find_all(text = ["Tillie", " Elsie ", "Lacie"])) # 正则表达式
print(soup.find_all(text = re.compile("Dormouse")))
运行结果
[' Elsie ']
[' Elsie ', 'Lacie', 'Tillie']
["The Dormouse's story", "The Dormouse's story"]
CSS选择器
这是另一种与find_all()方法有异曲同工的查找方法
- 写CSS时，标签名不加任何修饰，类名前加.，id名前加#
- 在这里我们也可以利用类似的方法来筛选元素，用到的方法是soup.select()，返回的类型是list
（1）通过标签名查找
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") print(soup.select("title")) print(soup.select("b")) print(soup.select("a"))
运行结果
[<title>The Dormouse's story</title>]
[<b>The Dormouse's story</b>]
[<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
（2）通过类名查找
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") print(soup.select(".title"))
运行结果
[<p class="title" name="dromouse"><b>The Dormouse's story</b></p>]
（3）通过id名查找
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") print(soup.select("#link1"))
运行结果
[<p class="title" name="dromouse"><b>The Dormouse's story</b></p>]
（4）组合查找
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") print(soup.select("p #link1"))
运行结果
[<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>]
（5）属性查找
查找时还可以加入属性元素，属性需要用中括号括起来，注意属性和标签属于同一节点，所以中间不能加空格，否则会无法匹配到。
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") print(soup.select("a[class='sister']"))
运行结果
[<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
同样，属性仍然可以与上述查找方式组合，不在同一节点的空格隔开，同一节点的不加空格
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") print(soup.select("p a[class='sister']"))
运行结果
[<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
（6）获取内容
以上的select()方法返回的结果都是列表形式，可以遍历形式输出，然后用get_text()方法来获取它的内容
#!/usr/bin/python3
# -*- coding:utf-8 -*-
__author__ = 'mayi' from bs4 import BeautifulSoup html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" # 创建 Beautiful Soup 对象，指定lxml解析器
soup = BeautifulSoup(html, "lxml") print(soup.select("p a[class='sister']")) for item in soup.select("p a[class='sister']"):
print(item.get_text())
运行结果
[<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>] Lacie
Tillie
注意：<!-- Elsie -->为注释内容，未输出
爬虫——BeautifulSoup4解析器的更多相关文章
- 爬虫笔记(四)------关于BeautifulSoup4解析器与编码
前言:本机环境配置:ubuntu 14.10,python 2.7,BeautifulSoup4 一.解析器概述 如同前几章笔记,当我们输入: soup=BeautifulSoup(response. ...
- Python爬虫开发【第1篇】【beautifulSoup4解析器】
CSS 选择器:BeautifulSoup4 Beautiful Soup 也是一个HTML/XML的解析器,主要的功能也是如何解析和提取 HTML/XML 数据. pip 安装:pip instal ...
- 爬虫中BeautifulSoup4解析器
CSS 选择器:BeautifulSoup4 和 lxml 一样,Beautiful Soup 也是一个HTML/XML的解析器,主要的功能也是如何解析和提取 HTML/XML 数据. lxml 只会 ...
- 关于BeautifulSoup4 解析器的说明
一.解析器概述 如同前几章笔记,当我们输入: soup=BeautifulSoup(response.body) 对网页进行析取时,并未规定解析器,此时使用的是python内部默认的解析器“html. ...
- Python爬虫beautifulsoup4常用的解析方法总结
摘要 如何用beautifulsoup4解析各种情况的网页 beautifulsoup4的使用 关于beautifulsoup4,官网已经讲的很详细了,我这里就把一些常用的解析方法做个总结,方便查阅. ...
- Python爬虫beautifulsoup4常用的解析方法总结（新手必看）
今天小编就为大家分享一篇关于Python爬虫beautifulsoup4常用的解析方法总结,小编觉得内容挺不错的,现在分享给大家,具有很好的参考价值,需要的朋友一起跟随小编来看看吧摘要 如何用beau ...
- python爬虫主要就是五个模块：爬虫启动入口模块，URL管理器存放已经爬虫的URL和待爬虫URL列表，html下载器，html解析器，html输出器 同时可以掌握到urllib2的使用、bs4（BeautifulSoup）页面解析器、re正则表达式、urlparse、python基础知识回顾（set集合操作）等相关内容。
本次python爬虫百步百科,里面详细分析了爬虫的步骤,对每一步代码都有详细的注释说明,可通过本案例掌握python爬虫的特点: 1.爬虫调度入口(crawler_main.py) # coding: ...
- Python爬虫(十四)_BeautifulSoup4 解析器
CSS选择器:BeautifulSoup4 和lxml一样,Beautiful Soup也是一个HTML/XML的解析器,主要的功能也是如何解析和提取HTML/XML数据. lxml只会局部遍历,而B ...
- Python HTML解析器BeautifulSoup(爬虫解析器)
BeautifulSoup简介 我们知道,Python拥有出色的内置HTML解析器模块——HTMLParser,然而还有一个功能更为强大的HTML或XML解析工具——BeautifulSoup(美味的 ...
随机推荐
- CSS动画效果
CSS变形效果 Transform translate:平移 translate(x,y) translateX(x) translateY(y)相对于元素原始位置平移. scale:缩放 大于1放大 ...
- axios 发 post 请求，后端接收不到参数的解决方案
问题场景 场景很简单,就是一个正常 axios post 请求: axios({ headers: { 'deviceCode': 'A95ZEF1-47B5-AC90BF3' }, method: ...
- Angular选项卡
前几天我发的东西,可能对于没有基础的人很难理解,那么今天,咱们就发点简单点的东西吧! Angular显示隐藏,选项卡! 还是那句话,话不多说,上代码: <!DOCTYPE html> &l ...
- STROME --realtime & online parallel computing
Data Collections ---> Stream to Channel (as source input) ----> Parallel Computing---> Resu ...
- Eclipse Common API
Platform runtime Defines the extension point and plug-in model. It dynamically discovers plug-ins an ...
- freebsd mount linprocfs
mount用来做什么? to prepare and graft a special device or the remote node(rhost:path) on to the file syst ...
- [转]查找问题的利器 - Git Bisect
转自:http://gitbook.liuhui998.com/5_4.html 假设你在项目的'2.6.18'版上面工作, 但是你当前的代码(master)崩溃(crash)了. 有时解决这种问题的 ...
- MySQL联合索引最左匹配范例
MySQL联合索引最左匹配范例 参考文章:http://blog.jobbole.com/24006/ 创建示例表. 示例表来自MySQL官方文档: https://dev.mysql.com/doc ...
- pthread使用
https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/Multithreading/CreatingTh ...
- HDU 3625 第一类斯特林数
题目链接:http://acm.hdu.edu.cn/showproblem.php?pid=3625 题意: n个房间,房间里面放着钥匙,允许破门而入k个,拿到房间里面的钥匙后可以打开对应的门,但是 ...