Increase maximum data per page


#1

I’m using the REST API to collect tweets with a certain hashtag every two days. There’s only about 30 tweets with that hashtag, yet because they’re spread out over multiple months, I can’t get them all in one GET request (“while the count endpoint will include 31-days worth of data per page.”, https://developer.twitter.com/en/docs/tweets/search/overview/premium), which means I have to loop the request until there’s no more next token. Because of this, I can very quickly exceed the maximum amount of requests per month (50), even though I only request around 450 tweets per month. Is there a way to increase this maximum data per page to more than just 31 days? I don’t have the money to increase the maximum requests per month, and I really shouldn’t need to.

Thank you


#3

Hi @DevMaarten - Can you share your full request here so I can see what we can adjust (ensure your keys/tokens are redacted)?


opened #4

#5

Hi @Hamza, this is the request:
endpoint: https://api.twitter.com/1.1/tweets/search/fullarchive/prod.json
parameters: query: ‘#allestransactie’,
fromDate: “201512220000”,


#6

If you’re not already doing so, use the Filter Stream instead to collect tweets - it’s way more efficient to just dump tweets into a queue or file as they arrive. https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter.html

And if it’s every 2 days, you don’t even need premium - use Standard REST API Search, that covers at least 5-7 days: https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets (check your parameters for setting since_id, max_id, counts etc.)

If you’re only interested in archives from the last 30 days, the 30day Premium endpoint has more requests per month you can use (250 calls with 100 tweets max per call).

Only worth using the fullarchive for much older tweets, if those are the ones you need most.

I can see you have fromDate: “201512220000” do you also have a toDate set? You don’t want to unintentionally request the same period of time over and over, it might be useful to define a list of 30 day time windows and process them one by one, keeping track of progress.

You have 50 Calls in every “billing cycle”, and every call is limited to returning 100 tweets, and from a max time window of 31 days, so if you make a request for a 30 day window, and get less than 100 tweets, you have retrieved all tweets and can move on to the next time window. If in the next time window you have more than 100 tweets, you’ll need to spend another call to page through results, eg: 450 tweets in a given 30 day time window will use up 5 calls. etc.

If i’m counting that right, and that hashtag frequency is typical, you can get data roughly from about 2015-12-22 to about 2016-10-01 in one “billing cycle” using 50 calls (10 30 day time windows with 5 calls used per time window), and so on… so to get the full archive should roughly take you the next 4 months.