Extend API for academic purposes

felipeeltermann · 2013-08-26T18:13:02.000Z

Hello,

My university colleagues and I started developing a Twitter based project for the Machine Learning course.

We are using the REST API to gather retweets, but we are stuck with a limitation from “GET statuses/retweets/:id” request.
According to the documentation, it returns up to 100 retweets of a given tweet. However, we need to have access to all of the retweets of a given tweet.
So, we’d like to be able to navigate through the whole retweets collection (for instance using a cursor parameter - https://dev.twitter.com/docs/misc/cursoring - or something like that).

Is it possible for us to have this kind of access through the REST API? How could we extend the limit?

We will provide further information about the project, if necessary.

episod · 2013-08-26T21:36:58.000Z

No, it’s unfortunately a hard limit. The best way to collect more retweets for a specific tweet is to have collected them in real-time as they were tweeted by using one of our streaming APIs.

felipeeltermann · 2013-08-27T00:03:56.000Z

Thank you.
We are going to work with the Streaming API then.

Since we need access to as much accurate information as possible, we’d like to use the “GET statuses/firehose” endpoint, but according to its documentation, it requires “special permission”.
How can one obtain it?

episod · 2013-08-27T15:21:33.000Z

The firehose is not generally available. You should just use the already very “thick” statuses/sample endpoint instead.

felipeeltermann · 2013-08-27T15:49:31.000Z

OK, I’ll give it a try.
Thanks for the quick reply.

felipeeltermann · 2013-09-03T21:20:00.000Z

Hello episod,

In my quest to obtain the Retweets tree of a given Tweet, I tried both REST and Streaming API’s.
As you can see on this discussion (20922) and here (https://dev.twitter.com/discussions/21152), I got limited by the REST API.

Streaming API “statuses/sample” would be useful to process information in a purely statistical manner.
However, our research is strongly based on analyzing the spread of a particular tweet over the network, and a sample of 1% is not accurate enough (the actual tree will be easily “broken”).

We need:
REST API “GET statuses/retweets/:id” request not to be limited
or
access to Streaming API “GET statuses/firehose” endpoint

Could you please advise?

episod · 2013-09-03T21:29:44.000Z

You’re not going to be able to get that kind of access. You would need to rescope your project to what you can obtain via the APIs available to you or consider commercial providers of Twitter data.

felipeeltermann · 2013-09-04T11:10:00.000Z

OK. Thank you.