python之https爬虫出现 SSL: CERTIFICATE_VERIFY_FAILED (同时打开fiddler就会出现)

1.参考

Py 坑之 CERTIFICATE_VERIFY_FAILED

Python 升级到 2.7.9 之后引入了一个新特性，当你urllib.urlopen一个 https 的时候，会验证一次 SSL 证书，当目标网站使用的是自签名的证书时就会爆出一个 urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)> 的错误消息

Python Requests throwing up SSLError

The problem you are having is caused by an untrusted SSL certificate.

Like @dirk mentioned in a previous comment, the quickest fix is setting verify=False.

Please note that this will cause the certificate not to be verified. This will expose your application to security risks, such as man-in-the-middle attacks.

Of course, apply judgment. As mentioned in the comments, this may be acceptable for quick/throwaway applications/scripts, but really should not go to production software.

If just skipping the certificate check is not acceptable in your particular context, consider the following options, your best option is to set the verify parameter to a string that is the path of the .pem file of the certificate (which you should obtain by some sort of secure means).

So, as of version 2.0, the verify parameter accepts the following values, with their respective semantics:

• True: causes the certificate to validated against the library's own trusted certificate authorities (Note: you can see which Root Certificates Requests uses via the Certifi library, a trust database of RCs extracted from Requests: Certifi - Trust Database for Humans).
• False: bypasses certificate validation completely.
• Path to a CA_BUNDLE file for Requests to use to validate the certificates.

Source: Requests - SSL Cert Verification

Also take a look at the cert parameter on the same link.

2.解决办法

(1)全局设置，对requests不生效

ssl._create_default_https_context = ssl._create_unverified_context

(2)urllib2.urlopen 传参

context = ssl._create_unverified_context()

urllib2.urlopen(context=context)

(3)参考 urllib2.urlopen 为opener叠加handler

# C:\Program Files\Anaconda2\Lib\urllib2.py
    # def urlopen
        # elif context:
            # https_handler = HTTPSHandler(context=context)
            # opener = build_opener(https_handler)   

# urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>
try:
    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))
    resp = opener.open(req)
except urllib2.URLError as err:
    print err
    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar), urllib2.HTTPSHandler(context=context))  #叠加多个 handler
    resp = opener.open(req)    

(4)request.get 传参

# requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
try:
    r = requests.get(url, headers=headers, cookies=cookie)
except requests.exceptions.SSLError as err:
    print err
    r = requests.get(url, headers=headers, cookies=cookie, verify=False)