Web页面解析 / Web page parsing

1 HTMLParser解析

下面介绍一种基本的Web页面HTML解析的方式，主要是利用Python自带的html.parser模块进行解析。其主要步骤为:

  1. 创建一个新的Parser类，继承HTMLParser类;
  2. 重载handler_starttag等方法，实现指定功能;
  3. 实例化新的Parser并将HTML文本feed给类实例。

完整代码

 from html.parser import HTMLParser

 # An HTMLParser instance is fed HTML data and calls handler methods when start tags, end tags, text, comments, and other markup elements are encountered

 # Subclass HTMLParser and override its methods to implement the desired behavior

 class MyHTMLParser(HTMLParser):

     # attrs is the attributes set in HTML start tag

     def handle_starttag(self, tag, attrs):

         print('Encountered a start tag:', tag)

         for attr in attrs:

             print('     attr:', attr)

     def handle_endtag(self, tag):

         print('Encountered an end tag :', tag)

     def handle_data(self, data):

         print('Encountered some data  :', data)

 parser = MyHTMLParser()

 parser.feed('<html><head><title>Test</title></head>'

             '<body><h1>Parse me!</h1></body></html>'

             '<img src="python-logo.png" alt="The Python logo">')

代码中首先对模块进行导入，派生一个新的 Parser 类，随后重载方法，当遇到起始tag时，输出并判断是否有定义属性，有则输出，遇到终止tag与数据时同样输出。

Note: handle_starttag()函数的attrs为由该起始tag属性组成的元组元素列表，即列表中包含元组，元组中第一个参数为属性名，第二个参数为属性值。

输出结果

Encountered a start tag: html

Encountered a start tag: head

Encountered a start tag: title

Encountered some data  : Test

Encountered an end tag : title

Encountered an end tag : head

Encountered a start tag: body

Encountered a start tag: h1

Encountered some data  : Parse me!

Encountered an end tag : h1

Encountered an end tag : body

Encountered an end tag : html

Encountered a start tag: img

     attr: ('src', 'python-logo.png')

     attr: ('alt', 'The Python logo')

从输出中可以看到，解析器将HTML文本进行了解析，并且输出了tag中包含的属性。

2 BeautifulSoup解析

接下来介绍一种第三方的HTML页面解析包BeautifulSoup，同时与HTMLParser进行对比。

首先需要进行BeautifulSoup的安装，安装方式如下，

pip install beautifulsoup4

完整代码

 from html.parser import HTMLParser

 from io import StringIO

 from urllib import request

 from bs4 import BeautifulSoup, SoupStrainer

 from html5lib import parse, treebuilders

 URLs = ('http://python.org',

         'http://www.baidu.com')

 def output(x):

     print('\n'.join(sorted(set(x))))

 def simple_beau_soup(url, f):

     'simple_beau_soup() - use BeautifulSoup to parse all tags to get anchors'

     # BeautifulSoup returns a BeautifulSoup instance

     # find_all function returns a bs4.element.ResultSet instance,

     # which contains bs4.element.Tag instances,

     # use tag['attr'] to get attribute of tag

     output(request.urljoin(url, x['href']) for x in BeautifulSoup(markup=f, features='html5lib').find_all('a'))

 def faster_beau_soup(url, f):

     'faster_beau_soup() - use BeautifulSoup to parse only anchor tags'

     # Add find_all('a') function

     output(request.urljoin(url, x['href']) for x in BeautifulSoup(markup=f, features='html5lib', parse_only=SoupStrainer('a')).find_all('a'))

 def htmlparser(url, f):

     'htmlparser() - use HTMLParser to parse anchor tags'

     class AnchorParser(HTMLParser):

         def handle_starttag(self, tag, attrs):

             if tag != 'a':

                 return

             if not hasattr(self, 'data'):

                 self.data = []

             for attr in attrs:

                 if attr[0] == 'href':

                     self.data.append(attr[1])

     parser = AnchorParser()

     parser.feed(f.read())

     output(request.urljoin(url, x) for x in parser.data)

     print('DONE')

 def html5libparse(url, f):

     'html5libparse() - use html5lib to parser anchor tags'

     #output(request.urljoin(url, x.attributes['href']) for x in parse(f) if isinstance(x, treebuilders.etree.Element) and x.name == 'a')

 def process(url, data):

     print('\n*** simple BeauSoupParser')

     simple_beau_soup(url, data)

     data.seek(0)

     print('\n*** faster BeauSoupParser')

     faster_beau_soup(url, data)

     data.seek(0)

     print('\n*** HTMLParser')

     htmlparser(url, data)

     data.seek(0)

     print('\n*** HTML5lib')

     html5libparse(url, data)

     data.seek(0)

 if __name__=='__main__':

     for url in URLs:

         f = request.urlopen(url)

         data = StringIO(f.read().decode())

         f.close()

         process(url, data)

分段解释

首先将所需模块进行导入，其中StringIO模块用来实现字符串缓存容器，

 from html.parser import HTMLParser

 from io import StringIO

 from urllib import request

 from bs4 import BeautifulSoup, SoupStrainer

 from html5lib import parse, treebuilders

 URLs = ('http://python.org',

         'http://www.baidu.com')

接着定义一个输出函数，利用集合消除重复参数同时进行换行分离。

 def output(x):

     print('\n'.join(sorted(set(x))))

此处定义一个简单的bs解析函数，首先利用BeautifulSoup类传入HTML文本以及features(新版提示使用‘html5lib’)，生成一个BeautifulSoup实例，再利用find_all()函数返回所有tag为‘a’的链接锚集合类(bs4.element.Tag)，通过Tag获取href属性，最后利用urljoin函数生成链接并输出。

 def simple_beau_soup(url, f):

     'simple_beau_soup() - use BeautifulSoup to parse all tags to get anchors'

     # BeautifulSoup returns a BeautifulSoup instance

     # find_all function returns a bs4.element.ResultSet instance,

     # which contains bs4.element.Tag instances,

     # use tag['attr'] to get attribute of tag

     output(request.urljoin(url, x['href']) for x in BeautifulSoup(markup=f, features='html5lib').find_all('a'))

接着定义一个新的解析函数，这个函数可以通过参数传入parse_only来设置需要解析的锚标签，从而加快解析的速度。

Note: 这部分存在一个问题，当使用‘html5lib’特性时，是不支持parse_only参数的，因此会对整个标签进行搜索。有待解决。

 def faster_beau_soup(url, f):

     'faster_beau_soup() - use BeautifulSoup to parse only anchor tags'

     # Add find_all('a') function

     output(request.urljoin(url, x['href']) for x in BeautifulSoup(markup=f, features='html5lib', parse_only=SoupStrainer('a')).find_all('a'))

再定义一个用html方式进行解析的函数，可参见前节使用方式，首先建立一个锚解析的类，在遇到起始标签时，判断是否为‘a’锚，在进入时判断是否有data属性，没有的话初始化属性为空，随后对attrs参数遍历，获取href参数。最后生成实例并feed数据。

 def htmlparser(url, f):

     'htmlparser() - use HTMLParser to parse anchor tags'

     class AnchorParser(HTMLParser):

         def handle_starttag(self, tag, attrs):

             if tag != 'a':

                 return

             if not hasattr(self, 'data'):

                 self.data = []

             for attr in attrs:

                 if attr[0] == 'href':

                     self.data.append(attr[1])

     parser = AnchorParser()

     parser.feed(f.read())

     output(request.urljoin(url, x) for x in parser.data)

     print('DONE')

最后定义一个process函数，对于传入的data，每次使用完后都需要seek(0)将光标移回初始。

 def process(url, data):

     print('\n*** simple BeauSoupParser')

     simple_beau_soup(url, data)

     data.seek(0)

     print('\n*** faster BeauSoupParser')

     faster_beau_soup(url, data)

     data.seek(0)

     print('\n*** HTMLParser')

     htmlparser(url, data)

     data.seek(0)

     print('\n*** HTML5lib')

     html5libparse(url, data)

     data.seek(0)

最终解析的结果为网页内所有的链接。

 if __name__=='__main__':

     for url in URLs:

         f = request.urlopen(url)

         data = StringIO(f.read().decode())

         f.close()

         process(url, data)

运行输出结果

*** simple BeauSoupParser

http://blog.python.org

http://bottlepy.org

http://brochure.getpython.info/

http://buildbot.net/

http://docs.python.org/3/tutorial/

http://docs.python.org/3/tutorial/controlflow.html

http://docs.python.org/3/tutorial/controlflow.html#defining-functions

http://docs.python.org/3/tutorial/introduction.html#lists

http://docs.python.org/3/tutorial/introduction.html#using-python-as-a-calculator

http://feedproxy.google.com/~r/PythonInsider/~3/TmC0nYZBrz4/python-364rc1-and-370a3-now-available.html

http://feedproxy.google.com/~r/PythonInsider/~3/rMFQQbvrekU/python-364-is-now-available.html

http://feedproxy.google.com/~r/PythonInsider/~3/ubEu3XCqoFM/python-370a2-now-available-for-testing.html

http://feedproxy.google.com/~r/PythonInsider/~3/xUpvN2wKt2s/python-364rc1-and-370a3-now-available.html

http://feedproxy.google.com/~r/PythonInsider/~3/zVC80sq9s00/python-364-is-now-available.html

http://flask.pocoo.org/

http://ipython.org

http://jobs.python.org

http://pandas.pydata.org/

http://planetpython.org/

http://plus.google.com/+Python

http://pycon.blogspot.com/

http://pyfound.blogspot.com/

http://python.org

http://python.org#content

http://python.org#python-network

http://python.org#site-map

http://python.org#top

http://python.org/

http://python.org/about/

http://python.org/about/apps

http://python.org/about/apps/

http://python.org/about/gettingstarted/

http://python.org/about/help/

http://python.org/about/legal/

http://python.org/about/quotes/

http://python.org/about/success/

http://python.org/about/success/#arts

http://python.org/about/success/#business

http://python.org/about/success/#education

http://python.org/about/success/#engineering

http://python.org/about/success/#government

http://python.org/about/success/#scientific

http://python.org/about/success/#software-development

http://python.org/accounts/login/

http://python.org/accounts/signup/

http://python.org/blogs/

http://python.org/community/

http://python.org/community/awards

http://python.org/community/diversity/

http://python.org/community/forums/

http://python.org/community/irc/

http://python.org/community/lists/

http://python.org/community/logos/

http://python.org/community/merchandise/

http://python.org/community/sigs/

http://python.org/community/workshops/

http://python.org/dev/

http://python.org/dev/core-mentorship/

http://python.org/dev/peps/

http://python.org/dev/peps/peps.rss

http://python.org/doc/

http://python.org/doc/av

http://python.org/doc/essays/

http://python.org/download/alternatives

http://python.org/download/other/

http://python.org/downloads/

http://python.org/downloads/mac-osx/

http://python.org/downloads/release/python-2714/

http://python.org/downloads/release/python-364/

http://python.org/downloads/source/

http://python.org/downloads/windows/

http://python.org/events/

http://python.org/events/calendars/

http://python.org/events/python-events

http://python.org/events/python-events/543/

http://python.org/events/python-events/611/

http://python.org/events/python-events/past/

http://python.org/events/python-user-group/

http://python.org/events/python-user-group/605/

http://python.org/events/python-user-group/619/

http://python.org/events/python-user-group/620/

http://python.org/events/python-user-group/past/

http://python.org/jobs/

http://python.org/privacy/

http://python.org/psf-landing/

http://python.org/psf/

http://python.org/psf/donations/

http://python.org/psf/sponsorship/sponsors/

http://python.org/shell/

http://python.org/success-stories/

http://python.org/success-stories/industrial-light-magic-runs-python/

http://python.org/users/membership/

http://roundup.sourceforge.net/

http://tornadoweb.org

http://trac.edgewall.org/

http://twitter.com/ThePSF

http://wiki.python.org/moin/Languages

http://wiki.python.org/moin/TkInter

http://www.ansible.com

http://www.djangoproject.com/

http://www.facebook.com/pythonlang?fref=ts

http://www.pylonsproject.org/

http://www.riverbankcomputing.co.uk/software/pyqt/intro

http://www.saltstack.com

http://www.scipy.org

http://www.web2py.com/

http://www.wxpython.org/

https://bugs.python.org/

https://devguide.python.org/

https://docs.python.org

https://docs.python.org/3/license.html

https://docs.python.org/faq/

https://github.com/python/pythondotorg/issues

https://kivy.org/

https://mail.python.org/mailman/listinfo/python-dev

https://pypi.python.org/

https://status.python.org/

https://wiki.gnome.org/Projects/PyGObject

https://wiki.python.org/moin/

https://wiki.python.org/moin/BeginnersGuide

https://wiki.python.org/moin/Python2orPython3

https://wiki.python.org/moin/PythonBooks

https://wiki.python.org/moin/PythonEventsCalendar#Submitting_an_Event

https://wiki.qt.io/PySide

https://www.openstack.org

https://www.python.org/psf/codeofconduct/

javascript:;

*** faster BeauSoupParser

Warning (from warnings module):

  File "C:\Python35\lib\site-packages\bs4\builder\_html5lib.py", line 63

    warnings.warn("You provided a value for parse_only, but the html5lib tree builder doesn't support parse_only. The entire document will be parsed.")

UserWarning: You provided a value for parse_only, but the html5lib tree builder doesn't support parse_only. The entire document will be parsed.

http://blog.python.org

http://bottlepy.org

http://brochure.getpython.info/

http://buildbot.net/

http://docs.python.org/3/tutorial/

http://docs.python.org/3/tutorial/controlflow.html

http://docs.python.org/3/tutorial/controlflow.html#defining-functions

http://docs.python.org/3/tutorial/introduction.html#lists

http://docs.python.org/3/tutorial/introduction.html#using-python-as-a-calculator

http://feedproxy.google.com/~r/PythonInsider/~3/TmC0nYZBrz4/python-364rc1-and-370a3-now-available.html

http://feedproxy.google.com/~r/PythonInsider/~3/rMFQQbvrekU/python-364-is-now-available.html

http://feedproxy.google.com/~r/PythonInsider/~3/ubEu3XCqoFM/python-370a2-now-available-for-testing.html

http://feedproxy.google.com/~r/PythonInsider/~3/xUpvN2wKt2s/python-364rc1-and-370a3-now-available.html

http://feedproxy.google.com/~r/PythonInsider/~3/zVC80sq9s00/python-364-is-now-available.html

http://flask.pocoo.org/

http://ipython.org

http://jobs.python.org

http://pandas.pydata.org/

http://planetpython.org/

http://plus.google.com/+Python

http://pycon.blogspot.com/

http://pyfound.blogspot.com/

http://python.org

http://python.org#content

http://python.org#python-network

http://python.org#site-map

http://python.org#top

http://python.org/

http://python.org/about/

http://python.org/about/apps

http://python.org/about/apps/

http://python.org/about/gettingstarted/

http://python.org/about/help/

http://python.org/about/legal/

http://python.org/about/quotes/

http://python.org/about/success/

http://python.org/about/success/#arts

http://python.org/about/success/#business

http://python.org/about/success/#education

http://python.org/about/success/#engineering

http://python.org/about/success/#government

http://python.org/about/success/#scientific

http://python.org/about/success/#software-development

http://python.org/accounts/login/

http://python.org/accounts/signup/

http://python.org/blogs/

http://python.org/community/

http://python.org/community/awards

http://python.org/community/diversity/

http://python.org/community/forums/

http://python.org/community/irc/

http://python.org/community/lists/

http://python.org/community/logos/

http://python.org/community/merchandise/

http://python.org/community/sigs/

http://python.org/community/workshops/

http://python.org/dev/

http://python.org/dev/core-mentorship/

http://python.org/dev/peps/

http://python.org/dev/peps/peps.rss

http://python.org/doc/

http://python.org/doc/av

http://python.org/doc/essays/

http://python.org/download/alternatives

http://python.org/download/other/

http://python.org/downloads/

http://python.org/downloads/mac-osx/

http://python.org/downloads/release/python-2714/

http://python.org/downloads/release/python-364/

http://python.org/downloads/source/

http://python.org/downloads/windows/

http://python.org/events/

http://python.org/events/calendars/

http://python.org/events/python-events

http://python.org/events/python-events/543/

http://python.org/events/python-events/611/

http://python.org/events/python-events/past/

http://python.org/events/python-user-group/

http://python.org/events/python-user-group/605/

http://python.org/events/python-user-group/619/

http://python.org/events/python-user-group/620/

http://python.org/events/python-user-group/past/

http://python.org/jobs/

http://python.org/privacy/

http://python.org/psf-landing/

http://python.org/psf/

http://python.org/psf/donations/

http://python.org/psf/sponsorship/sponsors/

http://python.org/shell/

http://python.org/success-stories/

http://python.org/success-stories/industrial-light-magic-runs-python/

http://python.org/users/membership/

http://roundup.sourceforge.net/

http://tornadoweb.org

http://trac.edgewall.org/

http://twitter.com/ThePSF

http://wiki.python.org/moin/Languages

http://wiki.python.org/moin/TkInter

http://www.ansible.com

http://www.djangoproject.com/

http://www.facebook.com/pythonlang?fref=ts

http://www.pylonsproject.org/

http://www.riverbankcomputing.co.uk/software/pyqt/intro

http://www.saltstack.com

http://www.scipy.org

http://www.web2py.com/

http://www.wxpython.org/

https://bugs.python.org/

https://devguide.python.org/

https://docs.python.org

https://docs.python.org/3/license.html

https://docs.python.org/faq/

https://github.com/python/pythondotorg/issues

https://kivy.org/

https://mail.python.org/mailman/listinfo/python-dev

https://pypi.python.org/

https://status.python.org/

https://wiki.gnome.org/Projects/PyGObject

https://wiki.python.org/moin/

https://wiki.python.org/moin/BeginnersGuide

https://wiki.python.org/moin/Python2orPython3

https://wiki.python.org/moin/PythonBooks

https://wiki.python.org/moin/PythonEventsCalendar#Submitting_an_Event

https://wiki.qt.io/PySide

https://www.openstack.org

https://www.python.org/psf/codeofconduct/

javascript:;

*** HTMLParser

http://blog.python.org

http://bottlepy.org

http://brochure.getpython.info/

http://buildbot.net/

http://docs.python.org/3/tutorial/

http://docs.python.org/3/tutorial/controlflow.html

http://docs.python.org/3/tutorial/controlflow.html#defining-functions

http://docs.python.org/3/tutorial/introduction.html#lists

http://docs.python.org/3/tutorial/introduction.html#using-python-as-a-calculator

http://feedproxy.google.com/~r/PythonInsider/~3/TmC0nYZBrz4/python-364rc1-and-370a3-now-available.html

http://feedproxy.google.com/~r/PythonInsider/~3/rMFQQbvrekU/python-364-is-now-available.html

http://feedproxy.google.com/~r/PythonInsider/~3/ubEu3XCqoFM/python-370a2-now-available-for-testing.html

http://feedproxy.google.com/~r/PythonInsider/~3/xUpvN2wKt2s/python-364rc1-and-370a3-now-available.html

http://feedproxy.google.com/~r/PythonInsider/~3/zVC80sq9s00/python-364-is-now-available.html

http://flask.pocoo.org/

http://ipython.org

http://jobs.python.org

http://pandas.pydata.org/

http://planetpython.org/

http://plus.google.com/+Python

http://pycon.blogspot.com/

http://pyfound.blogspot.com/

http://python.org

http://python.org#content

http://python.org#python-network

http://python.org#site-map

http://python.org#top

http://python.org/

http://python.org/about/

http://python.org/about/apps

http://python.org/about/apps/

http://python.org/about/gettingstarted/

http://python.org/about/help/

http://python.org/about/legal/

http://python.org/about/quotes/

http://python.org/about/success/

http://python.org/about/success/#arts

http://python.org/about/success/#business

http://python.org/about/success/#education

http://python.org/about/success/#engineering

http://python.org/about/success/#government

http://python.org/about/success/#scientific

http://python.org/about/success/#software-development

http://python.org/accounts/login/

http://python.org/accounts/signup/

http://python.org/blogs/

http://python.org/community/

http://python.org/community/awards

http://python.org/community/diversity/

http://python.org/community/forums/

http://python.org/community/irc/

http://python.org/community/lists/

http://python.org/community/logos/

http://python.org/community/merchandise/

http://python.org/community/sigs/

http://python.org/community/workshops/

http://python.org/dev/

http://python.org/dev/core-mentorship/

http://python.org/dev/peps/

http://python.org/dev/peps/peps.rss

http://python.org/doc/

http://python.org/doc/av

http://python.org/doc/essays/

http://python.org/download/alternatives

http://python.org/download/other/

http://python.org/downloads/

http://python.org/downloads/mac-osx/

http://python.org/downloads/release/python-2714/

http://python.org/downloads/release/python-364/

http://python.org/downloads/source/

http://python.org/downloads/windows/

http://python.org/events/

http://python.org/events/calendars/

http://python.org/events/python-events

http://python.org/events/python-events/543/

http://python.org/events/python-events/611/

http://python.org/events/python-events/past/

http://python.org/events/python-user-group/

http://python.org/events/python-user-group/605/

http://python.org/events/python-user-group/619/

http://python.org/events/python-user-group/620/

http://python.org/events/python-user-group/past/

http://python.org/jobs/

http://python.org/privacy/

http://python.org/psf-landing/

http://python.org/psf/

http://python.org/psf/donations/

http://python.org/psf/sponsorship/sponsors/

http://python.org/shell/

http://python.org/success-stories/

http://python.org/success-stories/industrial-light-magic-runs-python/

http://python.org/users/membership/

http://roundup.sourceforge.net/

http://tornadoweb.org

http://trac.edgewall.org/

http://twitter.com/ThePSF

http://wiki.python.org/moin/Languages

http://wiki.python.org/moin/TkInter

http://www.ansible.com

http://www.djangoproject.com/

http://www.facebook.com/pythonlang?fref=ts

http://www.pylonsproject.org/

http://www.riverbankcomputing.co.uk/software/pyqt/intro

http://www.saltstack.com

http://www.scipy.org

http://www.web2py.com/

http://www.wxpython.org/

https://bugs.python.org/

https://devguide.python.org/

https://docs.python.org

https://docs.python.org/3/license.html

https://docs.python.org/faq/

https://github.com/python/pythondotorg/issues

https://kivy.org/

https://mail.python.org/mailman/listinfo/python-dev

https://pypi.python.org/

https://status.python.org/

https://wiki.gnome.org/Projects/PyGObject

https://wiki.python.org/moin/

https://wiki.python.org/moin/BeginnersGuide

https://wiki.python.org/moin/Python2orPython3

https://wiki.python.org/moin/PythonBooks

https://wiki.python.org/moin/PythonEventsCalendar#Submitting_an_Event

https://wiki.qt.io/PySide

https://www.openstack.org

https://www.python.org/psf/codeofconduct/

javascript:;

DONE

*** HTML5lib

*** simple BeauSoupParser

http://e.baidu.com/?refer=888

http://home.baidu.com

http://image.baidu.com/search/index?tn=baiduimage&ps=1&ct=201326592&lm=-1&cl=2&nc=1&ie=utf-8&word=

http://ir.baidu.com

http://jianyi.baidu.com/

http://map.baidu.com

http://map.baidu.com/m?word=&fr=ps01000

http://music.baidu.com/search?fr=ps&ie=utf-8&key=

http://news.baidu.com

http://news.baidu.com/ns?cl=2&rn=20&tn=news&word=

http://tieba.baidu.com

http://tieba.baidu.com/f?kw=&fr=wwwt

http://v.baidu.com

http://v.baidu.com/v?ct=301989888&rn=20&pn=0&db=0&s=25&ie=utf-8&word=

http://wenku.baidu.com/search?word=&lm=0&od=0&ie=utf-8

http://www.baidu.com/

http://www.baidu.com/cache/sethelp/help.html

http://www.baidu.com/duty/

http://www.baidu.com/gaoji/preferences.html

http://www.baidu.com/more/

http://www.beian.gov.cn/portal/registerSystemInfo?recordcode=11000002000001

http://www.hao123.com

http://xueshu.baidu.com

http://zhidao.baidu.com/q?ct=17&pn=0&tn=ikaslist&rn=10&word=&fr=wwwt

https://passport.baidu.com/v2/?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2F

javascript:;

*** faster BeauSoupParser

http://e.baidu.com/?refer=888

http://home.baidu.com

http://image.baidu.com/search/index?tn=baiduimage&ps=1&ct=201326592&lm=-1&cl=2&nc=1&ie=utf-8&word=

http://ir.baidu.com

http://jianyi.baidu.com/

http://map.baidu.com

http://map.baidu.com/m?word=&fr=ps01000

http://music.baidu.com/search?fr=ps&ie=utf-8&key=

http://news.baidu.com

http://news.baidu.com/ns?cl=2&rn=20&tn=news&word=

http://tieba.baidu.com

http://tieba.baidu.com/f?kw=&fr=wwwt

http://v.baidu.com

http://v.baidu.com/v?ct=301989888&rn=20&pn=0&db=0&s=25&ie=utf-8&word=

http://wenku.baidu.com/search?word=&lm=0&od=0&ie=utf-8

http://www.baidu.com/

http://www.baidu.com/cache/sethelp/help.html

http://www.baidu.com/duty/

http://www.baidu.com/gaoji/preferences.html

http://www.baidu.com/more/

http://www.beian.gov.cn/portal/registerSystemInfo?recordcode=11000002000001

http://www.hao123.com

http://xueshu.baidu.com

http://zhidao.baidu.com/q?ct=17&pn=0&tn=ikaslist&rn=10&word=&fr=wwwt

https://passport.baidu.com/v2/?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2F

javascript:;

*** HTMLParser

http://e.baidu.com/?refer=888

http://home.baidu.com

http://image.baidu.com/search/index?tn=baiduimage&ps=1&ct=201326592&lm=-1&cl=2&nc=1&ie=utf-8&word=

http://ir.baidu.com

http://jianyi.baidu.com/

http://map.baidu.com

http://map.baidu.com/m?word=&fr=ps01000

http://music.baidu.com/search?fr=ps&ie=utf-8&key=

http://news.baidu.com

http://news.baidu.com/ns?cl=2&rn=20&tn=news&word=

http://tieba.baidu.com

http://tieba.baidu.com/f?kw=&fr=wwwt

http://v.baidu.com

http://v.baidu.com/v?ct=301989888&rn=20&pn=0&db=0&s=25&ie=utf-8&word=

http://wenku.baidu.com/search?word=&lm=0&od=0&ie=utf-8

http://www.baidu.com/

http://www.baidu.com/cache/sethelp/help.html

http://www.baidu.com/duty/

http://www.baidu.com/gaoji/preferences.html

http://www.baidu.com/more/

http://www.beian.gov.cn/portal/registerSystemInfo?recordcode=11000002000001

http://www.hao123.com

http://xueshu.baidu.com

http://zhidao.baidu.com/q?ct=17&pn=0&tn=ikaslist&rn=10&word=&fr=wwwt

https://passport.baidu.com/v2/?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2F

javascript:;

DONE

*** HTML5lib

参考链接

《Python 核心编程 第3版》

Python的Web编程[0] -> Web客户端[1] -> Web 页面解析的更多相关文章

  1. Web 2.0应用客户端性能问题十大根源《转载》

    前言 Web 2.0应用的推广为用户带来了全新的体验,同时也让开发人员更加关注客户端性能问题.最近,资深Web性能诊断专家.知名工具dynatrace的创始人之一Andreas Grabner根据自己 ...

  2. Web编程前端之7：web.config详解 【转】

    http://www.cnblogs.com/alvinyue/archive/2013/05/06/3063008.html 声明:这篇文章是摘抄周公(周金桥)的<asp.net夜话> ...

  3. Python的Web编程[0] -> Web客户端[0] -> 统一资源定位符 URL

    统一资源定位符 / URL 目录 URL 构成 URL 解析 URL 处理 1 URL构成 统一资源定位符(Uniform Resource Locator) 是对可以从互联网上得到的资源的位置和访问 ...

  4. Python的异步编程[0] -> 协程[0] -> 协程和 async / await

    协程 / Coroutine 目录 生产者消费者模型 从生成器到异步协程– async/await 协程是在一个线程执行过程中可以在一个子程序的预定或者随机位置中断,然后转而执行别的子程序,在适当的时 ...

  5. Python的网络编程[0] -> socket[0] -> socket 与 TCP / UDP

    Socket socket 简述 / socket Abstract 网络进程通信与 socket 网络中进程之间如何通信,首要解决的问题是如何唯一标识一个进程,否则通信无从谈起.在本地可以通过进程 ...

  6. Python的网络编程[0] -> socket[1] -> socket 模块

    socket 1. 常量 / Constants AF_* 和 SOCK_* 分别属于 AddressFamily 和 SocketType 1.1 AF_*类常量 socket.AF_UNIX:  ...

  7. 遇到奇怪的问题：web.py 0.40中使用web.input()，出现一堆奇怪的错误

    有的请求很正常,有的请求就出现了500错误. 这里使用POST请求,然后在web.input()中出现了很长很长的错误. 猜测是这个机器上安装了python2.7 / python 3.6 / pyt ...

  8. 【python】网络编程-SocketServer 实现客户端与服务器间非阻塞通信

    利用SocketServer模块来实现网络客户端与服务器并发连接非阻塞通信.首先,先了解下SocketServer模块中可供使用的类:BaseServer:包含服务器的核心功能与混合(mix-in)类 ...

  9. Python的异步编程[0] -> 协程[1] -> 使用协程建立自己的异步非阻塞模型

    使用协程建立自己的异步非阻塞模型 接下来例子中,将使用纯粹的Python编码搭建一个异步模型,相当于自己构建的一个asyncio模块,这也许能对asyncio模块底层实现的理解有更大的帮助.主要参考为 ...

随机推荐

  1. 对setTimeout函数的理解

    之前去面试一家公司时,面试官出了一道关于js的setTimeout函数的题目: /* *面试官给的原题目如下: *执行mytest()后,控制台输出内容是_____ *function mytest( ...

  2. linux统计分析流量-wireshark

    wireshark是一款带界面的开源抓包工具,可以用来对系统流量进行统计分析. 安装 由于wireshark是带界面的,所以一般在界面环境下运行,可以通过yum安装: $ yum install -y ...

  3. centos6安装openfire(mysql)

    一.安装JDK 我的系统是centos6.8x64,首先安装jdk1.7.0 二.安装openfire 我装的包是:openfire-3.9.3-1.i386.rpm,复制到/opt目录 #rpm - ...

  4. 牛客网暑期ACM多校训练营（第一场）：J-Different Integers（分开区间不同数+树状数组）

    链接:J-Different Integers 题意:给出序列a1, a2, ..., an和区间(l1, r1), (l2, r2), ..., (lq, rq),对每个区间求集合{a1, a2, ...

  5. How to solve SyntaxError on autogenerated manage.py?

    错误原因: 今天尝试使用pycharm中的Django. 使用的是基于python3.5的Django2.0.5. 按照官网教程https://docs.djangoproject.com/en/2. ...

  6. 重复造轮子系列--内存池（C语言）

    这个代码是我上个公司工作项目的里面内存管理(基于伙伴算法)的一个简化又简化的版本. 因为没有内存边界检查: 因为没有内存使用统计: 因为没有考虑线程安全: 因为没有内存分配操作的具体文件位置信息: 因 ...

  7. vs 2012 未能找到与约束contractName Microsoft.VisualStudio.Utilities...匹配的导出

    系统自动更新后,打开项目进行维护时,居然出错了,报的错误信息为“未能找到与约束contractName Microsoft.VisualStudio.Utilities...匹配的导出” 上网查了下, ...

  8. jQuery中Ajax的属性设置

    1.全局设置为同步 $.ajaxSetup({ async: false });

  9. C#类和类成员初始化顺序

    1.不带静态成员的普通类,首先通过构造函数初始化. 2.带静态属性的类,无论是普通类还是静态类,都会先初始化静态字段,再执行构造函数. 3.类初始化时,不会执行类中方法,无论是否是静态.若想执行方法, ...

  10. Bsd内核选项总结

    Bsd内核选项总结 一: 下面这个选项在每个内核中都要有: machine i386 它指明了机器的硬件体系结构.它必须是i386, pc98, sparc64, alpha, ia64, amd64 ...