#1

Hi,

I just registered for full archive access for 500 requests to collect up to 500 tweets each from 458 accounts. Before I ran my code on the premium account I tested it with in the sandbox. I encountered a problem with my paid premium search that did not happen in the sandbox. I have collected tweets from 208 accounts but the dashboard reports 500 requests and the API shuts me down. Only in a few instances did a request actually reach 500 tweets (the average number of tweets downloaded per account is 73.85).

I don’t understand why I am exceeding my request limits right now. Under what circumstances can request up to 500 tweets from an account end up as multiple requests in the dashboard?

1 Like
#2

Each request can fetch up to 500 tweets and up to 30 days. So if that one user didn’t tweet 500 tweets per month, you need to page to the prior months to get the rest of tweets.
I am guessing you are using some sort of library that’s doing the paging automatically for you?

#3

Thanks for your reply @Connexinet!

I am using rtweet and have included my R code below. I read through the Twitter and rtweet documentation and was not able to find an explanation for the issue.

Since you mentioned the switch to the prior month: I forgot to add that I have limited the time frame of my requests to 30 days. In a given one month period (before an election) I am trying to collect up to 500 tweets for each user in a list of users that I am interested in. That is why I am ending up with an average of 73.85 tweets per user. While some tweet up to and exceeding 500 tweets in that period most of them don’t.

for (i in 1:length(usernames)){
  tweets <- lapply(usernames[i], function(x) search_fullarchive(x, 
                                                                   n = 500, 
                                                                   env_name = "election15",
                                                                   fromDate = "201504080000", 
                                                                   toDate   = "201505082359"))
  csvfile <- paste0("tweets_",i,".csv")
  tweets <- do.call(rbind.data.frame, tweets)
  tweets <- data.frame(lapply(tweets, as.character), stringsAsFactors=FALSE)
  write.csv(tweets, csvfile, row.names=FALSE)
  print(paste("I collected tweets from", usernames[i]))
  Sys.sleep(10)
}
#4

OK yeah that’s strange.
When you ran your test, did you have all 500 requests available? When you did your testing with sandbox, was that in prior month?
You can see on the dashboard if your requests were all made on the same day or not:
https://developer.twitter.com/en/dashboard
If that helps?

#5

What’s the exact query that gets searched? the @username to find all mentions as well as tweets? or is it just from:username ?

I know it’s from 2015 but the 3200 tweet limit in Standard Timeline calls https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html is usually enough to cover a significant timeframe for a lot of accounts - so with some extra work in preparing users & appropriate queries you could significantly cut down on the required premium calls - no reason to use premium to retrieve a user’s tweet when they are still retrievable in Standard REST.

Also, remember that it’s more efficient to make bigger calls in premium, to hit the 500 tweet limit more often: Try to fit in as many of the 458 user ids into one call like from:one OR from:two OR from: three (Queries are limited to 1,024 characters per request) that way you are much more likely to get better use of the limits.

#6

Hi @IgorBrigadir,

thanks, this was very helpful.

I have already collected as many tweets as possible with the free tier. Now I have some accounts outstanding that tweeted so much that the one month period in 2015 is out of reach. For those I tried to use the premium account.

I have 458 accounts that I still need to collect for my research project. In theory I should be able to collect those with one additional purchase of 500 tweets. However, my issue is that for a reason that I do not understand requesting exactly 500 tweets from an account for a 30 day period is counted as multiple requests in the dashboard. I’m concerned that packing multiple accounts into one request will still have the discrepancy in the number.

1 Like
#7

This is due to how the requests are counted - for every call to the endpoint, it’s 500 tweets or 30 days of tweets, whichever comes first will create a new “page”. A new page requires a new call, and you get charged per call - So if you pack in as many ids as it will take, you’re much more likely to get 500 tweets in a 30 day window and use up just 1 request, instead of potentially many.

Also forgot to mention earlier - having a paid premium tier (any level, as long as it’s not sandbox) will give you access to the counts endpoint: https://developer.twitter.com/en/docs/tweets/search/api-reference/premium-search#CountsEndpoint which you can use to see how many tweets are in your query over time

It’s a bit convoluted - hope i got all that right!

1 Like
#8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.