Twitter数据抓取的方法(三)
Scraping Tweets Directly from Twitters Search – Update
Published August 1, 2015
Sorry for my delayed response to this as I’ve seen several comments on this topic, but I’ve been pretty busy with some other stuff recently, and this is the first chance I’ve had to address this!
As with most web scraping, at some point a provider will change their source code and scrapers will break. This is something that Twitter has done with their recent site redesign. Having gone over the changes, there are two that effect this scraping script.
The first change is tiny. Originally, to get all tweets rather than “top tweet”, we used the type_param “f” to denote “realtime”. However, the value for this has changed to just “tweets”.
Second change is a bit trickier to counter, as the scroll_cursor parameter no longer exists. Instead, if we look at the AJAX call that Twitter makes on its infinite scroll, we get a different parameter:
max_position:TWEET-399159003478908931-606844263347945472-BD1UO2FFu9QAAAAAAAAETAAAAAcAAAASAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
The highlighted parameter there, “max_position”, looks very similar to the original scroll_cursor parameter. However, unlike the scroll_cursor which existed in the response to be extracted, we have to create this one ourself.
As can be seen from the example, we have “TWEET” followed by two sets of numbers, and what appears to be “BD1UO2FFu9” screaming and falling off a cliff. The good news is, we actually only need the first three components.
“TWEET” will always stay the same, but the two sets of numbers are actually tweet ID’s, representing the oldest to most recently created tweets you’ve extracted.
For our newest tweet (2nd number set), we only need to extract this once as we can keep it the same for all calls, as Twitter does.
The oldest tweet (1st number set), we need to extract the last tweet id in our results each time to change our max_position value.
So, lets take a look at some of the code I’ve changed:
String minTweet = null; |
Rather than our original scroll_cursor value, we now have “minTweet”. Initially this is set to null, as we don’t have one to begin with. On our first call though, we get the first tweet in our response, and set the ID to minTweet, if minTweet is still null.
Next, we need to get the maxTweet. As previously said before, we get this by getting the last tweet in our results, and returning that ID. So we don’t repeat results, we need to make sure that the minTweet does not equal the maxTweet ID, and if not, we construct our “max_position” query with the format “TWEET-{maxTweetId}-{minTweetId}”.
You’ll also notice I changed the SCROLL_CURSOR_PARAM to “max_position” from “scroll_cursor”. Normally I’d change the variable name as well, but for visual reference, I’ve kept it the same for now, so you know where to change it.
Also, in constructUrl, the TYPE_PARAM value has also been set to “tweets”.
Finally, make sure you modify your TwitterResponse class so that it mirrors the parameters that are returned by the JSON file.
All you need to do is replace the original class variables with these, and update the constructor and getter/setter fields:
private boolean has_more_items; |
Twitter数据抓取的方法(三)的更多相关文章
- Twitter数据抓取的方法(一)
Scraping Tweets Directly from Twitters Search Page – Part 1 Published January 8, 2015 EDIT – Since I ...
- Twitter数据抓取的方法(二)
Scraping Tweets Directly from Twitters Search Page – Part 2 Published January 11, 2015 In the previo ...
- Twitter数据抓取
说明:这里分三个系列介绍Twitter数据的非API抓取方法.有兴趣的QQ群交流: BitCrawler网络爬虫QQ群 322937592 1.Twitter数据抓取(一) 2.Twitter数据抓取 ...
- Twitter数据非API采集方法
说明:这里分三个系列介绍Twitter数据的非API抓取方法. 在一个老外的博看上看到的,想详细了解的可以自己去看原文. 这种方法可以采集基于关键字在twitter上搜索的结果推文,已经实现自动翻页功 ...
- python爬虫数据抓取方法汇总
概要:利用python进行web数据抓取方法和实现. 1.python进行网页数据抓取有两种方式:一种是直接依据url链接来拼接使用get方法得到内容,一种是构建post请求改变对应参数来获得web返 ...
- 数据抓取的艺术(三):抓取Google数据之心得
本来是想把这部分内容放到前一篇<数据抓取的艺术(二):数据抓取程序优化>之中.但是随着任务的完成,我越来越感觉到其中深深的趣味,现总结如下: (1)时间 时间是一个与抓取规模相形而 ...
- 数据抓取的艺术(一):Selenium+Phantomjs数据抓取环境配置
数据抓取的艺术(一):Selenium+Phantomjs数据抓取环境配置 2013-05-15 15:08:14 分类: Python/Ruby 数据抓取是一门艺术,和其他软件不同,世界上 ...
- 【Python入门只需20分钟】从安装到数据抓取、存储原来这么简单
基于大众对Python的大肆吹捧和赞赏,作为一名Java从业人员,我本着批判与好奇的心态买了本python方面的书<毫无障碍学Python>.仅仅看了书前面一小部分的我......决定做一 ...
- Python爬虫工程师必学——App数据抓取实战 ✌✌
Python爬虫工程师必学——App数据抓取实战 (一个人学习或许会很枯燥,但是寻找更多志同道合的朋友一起,学习将会变得更加有意义✌✌) 爬虫分为几大方向,WEB网页数据抓取.APP数据抓取.软件系统 ...
随机推荐
- CSS中@import与link的具体区别
我们知道在网页中引用外部CSS有两种方式:@import和link 我们也经常听到有人说要使用link来引入CSS更好,但是你知道为什么吗? 继续往下看 link:link就是把外部CSS与网页连接起 ...
- Swap Nodes in Pairs leetcode
Given a linked list, swap every two adjacent nodes and return its head. For example,Given 1->2-&g ...
- 使用promise手动封装ajax函数
最近在做一个单页应用,node和浏览器仅通过json传输数据,因为是只有自己用等于是锻炼一下自己,所以也不用考虑seo的问题,node端我已经写好了,但是浏览器端想要用ajax原生太麻烦,用封装的函数 ...
- android学习——环境的搭建
1.安装JDK(java开发工具箱) 下载地址:http://www.oracle.com/technetwork/java/javase/downloads/index.html(根据自己需要下载) ...
- OpenCV使用FindContours进行二维码定位
我使用过FindContours,而且知道有能够直接寻找联通区域的函数.但是我使用的大多只是"最大轮廓"或者"轮廓数目"这些数据.其实轮廓还有另一个很重要的性质 ...
- Cordova原理一
我们知道cordova中js要和native通信都是通过 cordova plugin来实现的.如果我们自己创建一个cordova plugin,它其实就是几个独立的文件按照一定的目录结构放在一起,然 ...
- php+apache+mysql的安装
1.LAMP的安装顺序问题,现在是默认安装好了Linux系统,我的版本是Ubuntu 12.04.一般来说比较建议的顺序是Mysql Apache 最后安装PHP,在我实践下来 Apache和Mysq ...
- webpack学习笔记(二)-- 初学者常见问题及解决方法
这篇文章是webpack学习第二篇,主要罗列了本人在实际操作中遇到的一些问题及其解决方法,仅供参考,欢迎提出不同意见. 注:本文假设读者已有webpack方面相关知识,故文中涉及到的专有名词不做另外解 ...
- iOS开发之核心动画(Core Animation)
1.概述 Core Animation是一组非常强大的动画处理API,使用它能做出非常炫丽的动画效果,而且往往是事半功倍,使用它需要先添加QuartzCore.framework和引入对应的框架< ...
- iOSiOS开发之数据存储之NSKeyedArchiver
1.概述 NSKeyedArchiver归档和plist文件存储不同的是NSKeyedArchiver可以直接保存对象.如果对象是NSString.NSDictionary.NSArray.NSDat ...