Twitter数据抓取的方法(三)
Scraping Tweets Directly from Twitters Search – Update
Published August 1, 2015
Sorry for my delayed response to this as I’ve seen several comments on this topic, but I’ve been pretty busy with some other stuff recently, and this is the first chance I’ve had to address this!
As with most web scraping, at some point a provider will change their source code and scrapers will break. This is something that Twitter has done with their recent site redesign. Having gone over the changes, there are two that effect this scraping script.
The first change is tiny. Originally, to get all tweets rather than “top tweet”, we used the type_param “f” to denote “realtime”. However, the value for this has changed to just “tweets”.
Second change is a bit trickier to counter, as the scroll_cursor parameter no longer exists. Instead, if we look at the AJAX call that Twitter makes on its infinite scroll, we get a different parameter:
max_position:TWEET-399159003478908931-606844263347945472-BD1UO2FFu9QAAAAAAAAETAAAAAcAAAASAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
The highlighted parameter there, “max_position”, looks very similar to the original scroll_cursor parameter. However, unlike the scroll_cursor which existed in the response to be extracted, we have to create this one ourself.
As can be seen from the example, we have “TWEET” followed by two sets of numbers, and what appears to be “BD1UO2FFu9” screaming and falling off a cliff. The good news is, we actually only need the first three components.
“TWEET” will always stay the same, but the two sets of numbers are actually tweet ID’s, representing the oldest to most recently created tweets you’ve extracted.
For our newest tweet (2nd number set), we only need to extract this once as we can keep it the same for all calls, as Twitter does.
The oldest tweet (1st number set), we need to extract the last tweet id in our results each time to change our max_position value.
So, lets take a look at some of the code I’ve changed:
String minTweet = null; |
Rather than our original scroll_cursor value, we now have “minTweet”. Initially this is set to null, as we don’t have one to begin with. On our first call though, we get the first tweet in our response, and set the ID to minTweet, if minTweet is still null.
Next, we need to get the maxTweet. As previously said before, we get this by getting the last tweet in our results, and returning that ID. So we don’t repeat results, we need to make sure that the minTweet does not equal the maxTweet ID, and if not, we construct our “max_position” query with the format “TWEET-{maxTweetId}-{minTweetId}”.
You’ll also notice I changed the SCROLL_CURSOR_PARAM to “max_position” from “scroll_cursor”. Normally I’d change the variable name as well, but for visual reference, I’ve kept it the same for now, so you know where to change it.
Also, in constructUrl, the TYPE_PARAM value has also been set to “tweets”.
Finally, make sure you modify your TwitterResponse class so that it mirrors the parameters that are returned by the JSON file.
All you need to do is replace the original class variables with these, and update the constructor and getter/setter fields:
private boolean has_more_items; |
Twitter数据抓取的方法(三)的更多相关文章
- Twitter数据抓取的方法(一)
Scraping Tweets Directly from Twitters Search Page – Part 1 Published January 8, 2015 EDIT – Since I ...
- Twitter数据抓取的方法(二)
Scraping Tweets Directly from Twitters Search Page – Part 2 Published January 11, 2015 In the previo ...
- Twitter数据抓取
说明:这里分三个系列介绍Twitter数据的非API抓取方法.有兴趣的QQ群交流: BitCrawler网络爬虫QQ群 322937592 1.Twitter数据抓取(一) 2.Twitter数据抓取 ...
- Twitter数据非API采集方法
说明:这里分三个系列介绍Twitter数据的非API抓取方法. 在一个老外的博看上看到的,想详细了解的可以自己去看原文. 这种方法可以采集基于关键字在twitter上搜索的结果推文,已经实现自动翻页功 ...
- python爬虫数据抓取方法汇总
概要:利用python进行web数据抓取方法和实现. 1.python进行网页数据抓取有两种方式:一种是直接依据url链接来拼接使用get方法得到内容,一种是构建post请求改变对应参数来获得web返 ...
- 数据抓取的艺术(三):抓取Google数据之心得
本来是想把这部分内容放到前一篇<数据抓取的艺术(二):数据抓取程序优化>之中.但是随着任务的完成,我越来越感觉到其中深深的趣味,现总结如下: (1)时间 时间是一个与抓取规模相形而 ...
- 数据抓取的艺术(一):Selenium+Phantomjs数据抓取环境配置
数据抓取的艺术(一):Selenium+Phantomjs数据抓取环境配置 2013-05-15 15:08:14 分类: Python/Ruby 数据抓取是一门艺术,和其他软件不同,世界上 ...
- 【Python入门只需20分钟】从安装到数据抓取、存储原来这么简单
基于大众对Python的大肆吹捧和赞赏,作为一名Java从业人员,我本着批判与好奇的心态买了本python方面的书<毫无障碍学Python>.仅仅看了书前面一小部分的我......决定做一 ...
- Python爬虫工程师必学——App数据抓取实战 ✌✌
Python爬虫工程师必学——App数据抓取实战 (一个人学习或许会很枯燥,但是寻找更多志同道合的朋友一起,学习将会变得更加有意义✌✌) 爬虫分为几大方向,WEB网页数据抓取.APP数据抓取.软件系统 ...
随机推荐
- rhel 6.7 离线安装docker
本机系统信息 [test@rhel67temp ~]$ uname -a Linux rhel67temp 2.6.32-573.el6.x86_64 #1 SMP Wed Jul 1 18:23:3 ...
- Laravel使用Seeder自动填充数据
要查看代码,可以点击 或者转到链接:https://github.com/laravel/framework Laravel自动填充数据使用的是Seeder类 <?php use Illumin ...
- 剑指offer编程题Java实现——面试题12打印1到最大的n位数
题目:打印1到最大的n位数 输入数字n,按顺序打印输出从1到最大的n位十进制数,比如输入3,打印从1到999. 这道题考察的地方是如何表示大数问题.由于n是任意大的数组,如果n太大的话n位数就超过了l ...
- apicloud中dialog使用方法
var dialog = new auiDialog(); function openDialog(type){ switch (type) { case "text": dial ...
- 从数组中每次取一个不同的数组成员 getRandomItem(arr)
积累些常用的方法, 都是随写的, 不好之处, 望指出. getRandomItem(arr)函数如下: var getRandomItem = function () { var preItem = ...
- JAVA: List用法
1.List中可以添加任何对象,包括自己定义的新的类. class Person{.....}上面定义了一个Person类,下面看好如何使用ListPerson p1=new Person();Per ...
- Docker存储驱动之Device Mapper简介
Device Mapper是一个基于kernel的框架,它增强了很多Linux上的高级卷管理技术.Docker的devicemapper驱动在镜像和容器管理上,利用了该框架的超配和快照功能.为了区别, ...
- ST 单元测试之maven引入junit包
按照上篇博客,已经完成了mavne以及eclipse的安装配置,新建好了一个maven项目. 接下来打开项目,双击打开pom.xml,可以看到如下所示, 点击下方的pom.xml,然后添加如下代码,即 ...
- canvas的使用
1.概念 canvas一般就是用来绘制图像的 2.基本知识 上下文对象 var canvas = doucment.getElementById("canvas") ...
- Web 页面测试总结—控件类
web端页面测试,最常见的是基本控件的测试,只有了解常见的控件和其测试方法,才能掌握测试要点,避免漏测情况发生.根据日常工作总结,将控件和常见逻辑集合在一起,总结了几个控件类测试查场景如下. 导航条 ...