Count a User's mentions over a year?


#1

Hi Tweeters,

I wanted to code up small personal project that would require use of the REST API, but it looks like my idea would hit the rate limits pretty quickly. I’ll explain one of the things I’d hoped to do that looks like it’d be problematic, and maybe you all can tell me if I’m missing something, or if there’s a good alternate approach. Here goes:

Use case:
I want to count the number of times a User has been mentioned, by others, over the past year.

Approach:
Looks to me like I’d need to use the API’s search to get mentions of the User, then cursor through the results til I get to my end date. Once I get to the end date I can just count all the results I got.

The search URL for this would, at its most basic and before encoding, be something like this:
…/search/tweets.json?q=@TwitterHandle±TwitterHandle&count=100&max_id=[12345]

This is straightforward, but if the User I’m interested in has thousands of mentions, I’d need to call the API tens of times. And if I want to do this on small set of Users (say, a few hundred), that’ll mean hundreds, probably thousands, of searches. And aside from this count of mentions, I’d have one or two other similar queries I’d want to run for each user. The point being, I’m way past the rate limit, with this approach.

So I guess what I’m asking (at great length–sorry) is…

  1. Is there any way to run thousands of searches without hitting the rate limit? Is there any mechanism by which Twitter lifts rate limits for individuals?

  2. Alternatively, am I missing some way to use the API to get metrics like this? Twitter Analytics looks like it has this sort of info… is there any plan to offer a public API for it?

Thank you!

ps. I realize could code around the limit by proactively idling my process when necessary, but that feels inelegant, and also it’d be a hassle, and anyway I suspect it’d fuzz up the accuracy of the results I’m looking for. Alternatively, I could register a handful of Twitter accounts for this purpose, and when one hits the limit, switch to the other… but that seems like an even bigger hassle and, to boot, like an abuse of the API.


#2

Hey, welcome.
The rate limit is by far not the only problem here. The Twitter REST API can not be used to retrieve this old data.
Especially the search endpoint will only provide you with data approximately one week old. Even the mentions endpoint for a user will only provide you with a maximum of 800 mentions.
The only thing I can think of is that you use Gnip, a partner of Twitter that provides access to the data you need. See this page for more information about Gnip historical Tweet access.


#3

Ha! Thank you, ePirat! Well I’m glad I didn’t start coding, now. :smile:

I’ll check out Gnip, and maybe set my idea aside until whenever Twitter exposes its analytics data with an API.

Thank you again.