上一篇我写了如何爬取百度网盘的爬虫,在这里还是重温一下,把链接附上:

http://www.cnblogs.com/huangxie/p/5473273.html

这一篇我想写写如何爬取百度图片的爬虫,这个爬虫也是:搜搜gif在线制作功能点我) 的爬虫代码,其实爬虫整体框架还是差不多的,但就是会涉及到图片的的一些处理,还是花费了我不少时间的,所以我请阅读的本爬虫的孩子还是认真一些,毕竟程序猿都不容易啊。好的,我也不想多说,爬虫的代码我会分享到去转盘网,想下载本爬虫代码的孩子请点我下载,如果没有下载到,请点击这个链接

附代码:

PS:不会python的孩子赶快去补补吧,先把基础搞清楚再说

#coding:utf-8

"""

Created on 2015-9-17

@author: huangxie

"""

import time,math,os,re,urllib,urllib2,cookielib 

from bs4 import BeautifulSoup

import time  

import re

import uuid

import json

from threading import Thread

from Queue import Queue 

import MySQLdb as mdb

import sys

import threading

import utils

import imitate_browser

from MySQLdb.constants.REFRESH import STATUS

reload(sys)

sys.setdefaultencoding('utf-8')

DB_HOST = '127.0.0.1'

DB_USER = 'root'

DB_PASS = 'root'

proxy = {u'http':u'222.39.64.13:8118'}

TOP_URL="http://image.baidu.com/i?tn=resultjsonavatarnew&ie=utf-8&word={word}&pn={pn}&rn={rn}"

KEYWORD_URL="https://www.baidu.com/s?ie=utf-8&f=8&tn=baidu&wd={wd}"

"""

i_headers = {'User-Agent':'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',

              'Accept':'json;q=0.9,*/*;q=0.8',

              'Accept-Charset':'utf-8;q=0.7,*;q=0.3',

              'Accept-Encoding':'gzip',

              'Connection':'close',

              'Referer':None #注意如果依然不能抓取的话,这里可以设置抓取网站的host

            }

"""

i_headers = {'User-Agent':'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.48'}

def GetDateString():

    x = time.localtime(time.time())

    foldername = str(x.__getattribute__("tm_year"))+"-"+str(x.__getattribute__("tm_mon"))+"-"+str(x.__getattribute__("tm_mday"))

    return foldername 

class BaiduImage(threading.Thread):     

    def __init__(self):

        Thread.__init__(self)

        self.browser=imitate_browser.BrowserBase()

        self.chance=0

        self.chance1=0

        self.request_queue=Queue()

        self.wait_ana_queue=Queue()

        #self.key_word_queue.put((("动态图", 0, 24)))

        self.count=0

        self.mutex = threading.RLock() #可重入锁,使单线程可以再次获得已经获得的锁

        self.commit_count=0

        self.ID=500

        self.next_proxy_set = set()

        self.dbconn = mdb.connect(DB_HOST, DB_USER, DB_PASS, 'sosogif', charset='utf8')

        self.dbconn.autocommit(False)

        self.dbcurr = self.dbconn.cursor()

        self.dbcurr.execute('SET NAMES utf8')

    """

    def run(self):

        while True:

            self.get_pic()

    """

    def work(self,item):

        print "start thread",item

        while True: #MAX_REQUEST条以上则等待

            self.get_pic()

            self.prepare_request()

    def format_keyword_url(self,keyword):

        return KEYWORD_URL.format(wd=keyword).encode('utf-8') 

    def generateSeed(self,url):

        html = self.browser.openurl(url).read()

        if html:

            try:

                soup = BeautifulSoup(html)

                trs = soup.find('div', id='rs').find('table').find_all('tr') #获得所有行

                for tr in trs:

                    ths=tr.find_all('th')

                    for th in ths:

                        a=th.find_all('a')[0]

                        keyword=a.text.strip()

                        if "动态图" in keyword or "gif" in keyword:

                            print "keyword",keyword

                            self.dbcurr.execute('select id from info where word=%s',(keyword))

                            y = self.dbcurr.fetchone()

                            if not y:

                                self.dbcurr.execute('INSERT INTO info(word,status,page_num,left_num,how_many) VALUES(%s,0,0,0,0)',(keyword))

                    self.dbconn.commit()

            except:

                pass

    def prepare_request(self):

        self.lock()

        self.dbcurr.execute('select * from info where status=0')

        result = self.dbcurr.fetchone()

        if result:

            id,word,status,page_num,left_num,how_many=result

            self.request_queue.put((id,word,page_num)) 

            if page_num==0 and left_num==0 and how_many==0:

                url=self.format_keyword_url(word)

                self.generateSeed(url)

                html=""

                try:

                    url=self.format_top_url(word, page_num, 24)

                    html = self.browser.openurl(url).read()

                except Exception as err:

                    print "err",err

                    #pass

                if html!="":

                    how_many=self.how_many(html)

                    print "how_many",how_many

                    if how_many==None:

                        how_many=0

                    t=math.ceil(how_many/24*100) #只要前1/100即可

                    num = int(t)

                    for i  in xrange(0,num-1):

                        self.dbcurr.execute('INSERT INTO info(word,status,page_num,left_num,how_many) VALUES(%s,%s,%s,%s,%s)',(word,0,i*24,num-i,how_many))

                    self.dbcurr.execute('update info SET status=1 WHERE id=%s',(id)) #置为已经访问

                    self.dbconn.commit()

        self.unlock()

    def start_work(self,req_max):

        for item in xrange(req_max):

            t = threading.Thread(target=self.work, args=(item,))

            t.setDaemon(True)

            t.start()

    def lock(self): #加锁

        self.mutex.acquire()

    def unlock(self): #解锁

        self.mutex.release()

    def get_para(self,url,key):

        values = url.split('?')[-1]

        for key_value in values.split('&'):

            value=key_value.split('=')

            if value[0]==key:

                return value[1]

        return None  

    def makeDateFolder( self,par,child):

        #self.lock()

        if os.path.isdir( par ):

            path=par + '//' + GetDateString()

            newFolderName = path+'//'+child

            if not os.path.isdir(path):

                os.mkdir(path)

            if not os.path.isdir( newFolderName ):

                os.mkdir( newFolderName )

            return newFolderName

        else:

            return par 

        #self.unlock()

    def parse_json(self,data):

        ipdata = json.loads(data)

        try:

            if ipdata['imgs']:  

                for n in ipdata['imgs']: #data子项 

                    if n['objURL']:  

                        try:

                            proxy_support = urllib2.ProxyHandler(proxy)

                            opener = urllib2.build_opener(proxy_support)

                            urllib2.install_opener(opener)

                            #print "proxy",proxy

                            self.lock()

                            self.dbcurr.execute('select ID from pic_info where objURL=%s', (n['objURL']))

                            y = self.dbcurr.fetchone()

                            #print "y=",y

                            if y:

                                print "database exist"

                                self.unlock() #continue 前解锁

                                continue

                            else:

                                real_extension=utils.get_extension(n['objURL'])

                                req = urllib2.Request(n['objURL'],headers=i_headers)

                                resp = urllib2.urlopen(req,None,5)

                                dataimg=resp.read()

                                name=str(uuid.uuid1())

                                filename=""

                                if len(real_extension)>4:

                                    real_extension=".gif"

                                real_extension=real_extension.lower()

                                if real_extension==".gif":

                                    filename  =self.makeDateFolder("E://sosogif", "d"+str(self.count % 60))+"//"+name+"-www.sosogif.com-搜搜gif贡献"+real_extension

                                    self.count+=1

                                else:

                                    filename  =self.makeDateFolder("E://sosogif", "o"+str(self.count % 20))+"//"+name+"-www.sosogif.com-搜搜gif贡献"+real_extension

                                    self.count+=1

                                """

                                name=str(uuid.uuid1())

                                filename=""

                                if len(real_extension)>4:

                                    real_extension=".gif"

                                filename  =self.makeDateFolder("E://sosogif", "d"+str(self.count % 60))+"//"+name+"-www.sosogif.com-搜搜gif贡献"+real_extension

                                self.count+=1 

                                """

                                try: 

                                    if not os.path.exists(filename): 

                                        file_object = open(filename,'w+b')  

                                        file_object.write(dataimg)  

                                        file_object.close()

                                        self.anaylis_info(n,filename,real_extension) #入库操作

                                    else:

                                        print "file exist" 

                                except IOError,e1:  

                                    print "e1=",e1

                                    pass

                            self.unlock()

                        except IOError,e2:  

                            #print "e2=",e2 

                            pass  

                            self.chance1+=1

        except Exception as parse_error:

            print "parse_error",parse_error

            pass

    def title_dealwith(self,title):

        #print "title",title

        a=title.find("<strong>")

        temp1=title[0:a]

        b=title.find("</strong>")

        temp2=title[a+8:b]

        temp3=title[b+9:len(title)]

        return (temp1+temp2+temp3).strip()

    def anaylis_info(self,n,filename,real_extension):

        print "success."

        #if self.wait_ana_queue.qsize()!=0:

            #n,filename,real_extension=self.wait.ana_queue.get()

        #self.lock()

        objURL=n['objURL'] #图片地址

        fromURLHost=n['fromURLHost'] #来源网站

        width=n['width']  #宽度

        height=n['height'] #高度

        di=n['di'] #用来唯一标识

        type=n['type'] #格式

        fromPageTitle=n['fromPageTitle'] #来自网站

        keyword=self.title_dealwith(fromPageTitle)

        cs=n['cs'] #未知

        os=n['os'] #未知

        temp = time.time()

        x = time.localtime(float(temp))

        acTime = time.strftime("%Y-%m-%d %H:%M:%S",x) #爬取时间

        self.dbcurr.execute('select ID from pic_info where cs=%s', (cs))

        y = self.dbcurr.fetchone()

        if not y:

            print 'add pic',filename

            self.commit_count+=1

            self.dbcurr.execute('INSERT INTO pic_info(objURL,fromURLHost,width,height,di,type,keyword,cs,os,acTime,filename,real_extension) VALUES(%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)',(objURL,fromURLHost,width,height,di,type,keyword,cs,os,acTime,filename,real_extension))

            if self.commit_count==10:

                self.dbconn.commit()

                self.commit_count=0

        #self.unlock()

    def format_top_url(self,word,pn,rn):

        url = TOP_URL.format(word=word, pn=pn,rn=rn).encode('utf-8') 

        return url

    def how_many(self,data):

        try:

            ipdata = json.loads(data)

            if ipdata['displayNum']>0:

                how_many=ipdata['displayNum']

                return int(how_many)

            else:

                return 0

        except Exception as e:

            pass

    def get_pic(self):

        """

        word="gif"

        pn=0

        rn=24

        if self.key_word_queue.qsize()!=0:

            word,pn,rn=self.key_word_queue.get()

        url=self.format_top_url(word,pn,rn)

        global proxy

        if url:

            try:

                html=""

                try:

                    req = urllib2.Request(url,headers=i_headers)

                    response = urllib2.urlopen(req, None,5)

                    #print "url",url

                    html = self.browser.openurl(url).read()

                except Exception as err:

                    print "err",err

                    #pass

                if html:

                    how_many=self.how_many(html)

                    #how_many=10000

                    print "how_many",how_many

                    word=self.get_para(url,"word")

                    rn=int(self.get_para(url,"rn"))

                    t=math.ceil(how_many/rn)

                    num = int(t)

                    for item  in xrange(0,num-1):

        """

        try:

            global proxy

            print "size of queue",self.request_queue.qsize()

            if self.request_queue.qsize()!=0:

                id,word,page_num = self.request_queue.get()            

                u=self.format_top_url(word,page_num,24)

                self.lock()

                self.dbcurr.execute('update info SET status=1 WHERE id=%s',(id))

                self.dbconn.commit()

                if self.chance >0 or self.chance1>1: #任何一个出问题都给换代理

                    if self.ID % 100==0:

                        self.dbcurr.execute("select count(*) from proxy")

                        for r in self.dbcurr:

                            count=r[0]

                        if self.ID>count:

                            self.ID=50

                    self.dbcurr.execute("select * from proxy where ID=%s",(self.ID))

                    results = self.dbcurr.fetchall()

                    for r in results:

                        protocol=r[1]

                        ip=r[2]

                        port=r[3]

                        pro=(protocol,ip+":"+port)

                        if pro not in self.next_proxy_set:

                            self.next_proxy_set.add(pro)

                    self.chance=0

                    self.chance1=0

                    self.ID+=1

                self.unlock() 

                proxy_support = urllib2.ProxyHandler(proxy)

                opener = urllib2.build_opener(proxy_support)

                urllib2.install_opener(opener)

                html=""

                try:

                    req = urllib2.Request(u,headers=i_headers)

                    #print "u=",u

                    response = urllib2.urlopen(req, None,5)

                    html = response.read()

                    if html:

                        #print "html",type(html)

                        self.parse_json(html)

                except Exception as ex1:

                    #print "error=",ex1

                    pass

                    self.chance+=1

                    if self.chance>0 or self.chance1>1:

                        if len(self.next_proxy_set)>0:

                            protocol,socket=self.next_proxy_set.pop()

                            proxy= {protocol:socket}

                            print "change proxy finished<<",proxy,self.ID

        except Exception as e:

            print "error1",e

            pass

if __name__ == '__main__':

    app = BaiduImage() 

    app.start_work(80)

    #app.generateSeed()

    while 1:

        pass

  

百度图片爬虫-python版-如何爬取百度图片?的更多相关文章

  1. Python爬虫实战二之爬取百度贴吧帖子

    大家好,上次我们实验了爬取了糗事百科的段子,那么这次我们来尝试一下爬取百度贴吧的帖子.与上一篇不同的是,这次我们需要用到文件的相关操作. 前言 亲爱的们,教程比较旧了,百度贴吧页面可能改版,可能代码不 ...

  2. 转 Python爬虫实战二之爬取百度贴吧帖子

    静觅 » Python爬虫实战二之爬取百度贴吧帖子 大家好,上次我们实验了爬取了糗事百科的段子,那么这次我们来尝试一下爬取百度贴吧的帖子.与上一篇不同的是,这次我们需要用到文件的相关操作. 本篇目标 ...

  3. Python开发简单爬虫(二)---爬取百度百科页面数据

    一.开发爬虫的步骤 1.确定目标抓取策略: 打开目标页面,通过右键审查元素确定网页的url格式.数据格式.和网页编码形式. ①先看url的格式, F12观察一下链接的形式;② 再看目标文本信息的标签格 ...

  4. python学习(十七) 爬取MM图片

    这一篇巩固前几篇文章的学到的技术,利用urllib库爬取美女图片,其中采用了多线程,文件读写,目录匹配,正则表达式解析,字符串拼接等知识,这些都是前文提到的,综合运用一下,写个爬虫示例爬取美女图片.先 ...

  5. python+selenium+bs4爬取百度文库内文字 && selenium 元素可以定位到,但是无法点击问题 && pycharm多行缩进、左移

    先说一下可能用到的一些python知识 一.python中使用的是unicode编码, 而日常文本使用各类编码如:gbk utf-8 等等所以使用python进行文字读写操作时候经常会出现各种错误, ...

  6. python 利用selenium爬取百度文库的word文章

    今天学习如何使用selenium库来爬取百度文库里面的收费的word文档 from selenium import webdriver from selenium.webdriver.common.k ...

  7. Python爬虫:通过关键字爬取百度图片

    使用工具:Python2.7 点我下载 scrapy框架 sublime text3 一.搭建python(Windows版本) 1.安装python2.7 ---然后在cmd当中输入python,界 ...

  8. Python爬虫之简单的爬取百度贴吧数据

    首先要使用的第类库有 urllib下的request  以及urllib下的parse  以及 time包  random包 之后我们定义一个名叫BaiduSpider类用来爬取信息 属性有 url: ...

  9. 百度图片爬虫-python版

               self.browser=imitate_browser.BrowserBase()            self.chance=0            self.chanc ...

随机推荐

  1. [Aaronyang] 写给自己的WPF4.5 失传的第一本古老秘籍[多扩展显示器编程]

    世界上只有想不通的人,没有走不通的路 --Aaronyang的博客(www.ayjs.net)-www.8mi.me =============时隔两年后再看WPF,有些秘籍不太适合公开,公开了就不值 ...

  2. 完美实现开机启动虚拟WIFI,顺便实现目前的WP8系统使用VPN(7.1修)

    众所周知,windows7系统的机器若带有无线网卡(台式机可以买一个USB无线网卡,京东目前39元,TP-Link的),可以虚拟出wifi,供手机等移动设备使用. 虚拟的WIFI的命了和软件在网上都找 ...

  3. nginx root && alias 文件路径配置

    文章摘自:http://www.ttlsa.com/nginx/nginx-root_alias-file-path-configuration/ nginx指定文件路径有两种方式root和alias ...

  4. [转]Java中的对象和对象引用实例浅析

    在Java中,有一组名词经常一起出现,它们就是“对象和对象引用”,很多朋友在初学Java的时候可能经常会混淆这2个概念,觉得它们是一回事,事实上则不然.今天我们就来一起了解一下对象和对象引用之间的区别 ...

  5. 在CentOS上装 ElasticSearch

    参考官方文档:Install Elasticsearch with RPM ElasticSearch依赖Java,所以需要先安装Java: 到Oracle官网找到下载链接 http://www.or ...

  6. BZOJ-1228 E&D 博弈SG+找啊找啊找规律

    讨厌博弈,找规律找半天还是错的.... 1228: [SDOI2009]E&D Time Limit: 10 Sec Memory Limit: 162 MB Submit: 666 Solv ...

  7. 【bzoj1179】 Apio2009—Atm

    www.lydsy.com/JudgeOnline/problem.php?id=1179 (题目链接) 题意 给出一张有向图,每个节点有点权.标记一些点,找出一条路径,可以重复经过一条边,使得总点权 ...

  8. Threat Risk Modeling Learning

    相关学习资料 http://msdn.microsoft.com/en-us/library/aa302419(d=printer).aspx http://msdn.microsoft.com/li ...

  9. JAVA中的数组是对象吗?

    public class Main{ public static void main(String[] args) { int a[]={1,9}; //Object obj=new int[10]; ...

  10. C++ STL之stack

    因为总用vector,却忘记了有stack,今天用到栈顶的值才想起来,说起来stack很方便,也很容易用,看下边例子: #include<stack> #include<iostre ...