What to do if you've exceeded requests

[marcorighetto]

Hi!

I am using the Premium Search API Full Archive Sandbox to download all tweets with certain hashtags. The problem I’m incurring is that I’m not a trained engineer or developer and I did some trial and error code. I just realized that I’ve unknowingly exceeded my rate limits (I got a 429 error). What can I do now? I don’t have the budget for the paid plan — it’s more of a research project. Do I just have to wait? How long? How can I avoid exceeding the limit again? Here is my code:

from TwitterAPI import TwitterAPI
import csv

SEARCH_TERM = 'my-search-term'
PRODUCT = 'fullarchive'
LABEL = 'my-label'
NEXT = ''
FROM_DATE = '200603220000'
TO_DATE = '201806020000'

api = TwitterAPI("consumer-key", 
                 "consumer-secret", 
				 "token", 
			     "token-secret")

while True:
	r = api.request('tweets/search/%s/:%s' % (PRODUCT, LABEL), 
	                {'query':SEARCH_TERM,
	                'fromDate': FROM_DATE,
	                'toDate':TO_DATE,
	                'next': NEXT
	                }
	                )
	print(r.status_code)
	csvFile = open('2006-2018.csv', 'a')
	csvWriter = csv.writer(csvFile)
	if r.status_code != 200:
	    break
	for item in r:
	    csvWriter.writerow([item['created_at'],item['user']['screen_name'], item['text'] if 'text' in item else item])
	json = r.json()
	if 'next' not in json:
	    break
	NEXT = json['next']

[Connexinet]

There are two limits:
Rate limit: which is the rate at which you make the calls, you need to slow down your code. The rate depends on the API, for example the 30 days search you can only request 30 requests/min (for sandbox), and 10 requests/sec. So if you exceed your rate limits just wait till the next time slot (next minute, second), but ideally slow your code down so you don’t get rate limit errors.

Quota limits: each plan is allotted a quota (e.g. sandbox has 250 per month for the 30 days search API). If you exhaust it, you need to wait till the next billing period (your account should tell you the start date), or upgrade to a higher plan.