I have created an application in Python that searches Twitter for specific topics and collects said tweets for further analysis. Pretty basic stuff indeed.
For this purposes I am using the TwitterSearch 0.75 module. For those who are familiar with this module, I am using the iterative method (searchTweetsIterable). This application runs on a server in our data centre and, as far as I am aware, it should be the only device accessing twitter.com through our network.
I believe myself to be familiar with the concepts of the requests windows and as I am running this under the standard user authentication, I should have 180 requests per window (currently 15 minutes).
In my code I carefully monitor the number of remaining requests - every time I collect a new page of tweets, I also retrieve the number of remaining requests. Along with the search results, I also store the latest remaining requests number. Before each search, my application checks to see what the last number of available requests was, when that period was going to reset - when there are too few requests remaining (counting also the requests required for handshake and other overhead), the application will wait until the last known request reset window has expired, so that we have a fresh 180 requests to use within the window of 15 minutes.
However, I frequently experience very strange things - whereby my application ends a search and I still have 160 requests available.
My application retrieves the next search item to look for, including geo coordinates, and upon connecting to Twitter, I immediately get an error 429 - indicating that I have exhausted my requests for that time period.
My application then automatically waits 15+1 minutes before trying again - allowing me to collect a handful of tweets before I once again get slapped with a 429 error.
When I look at the logging in my application, there is no way that I have exhausted or even achieved 180 requests per 15 minute window.
Is this a flaw in the TwitterSearch 0.75 module or is the Twitter Search API passing the wrong amount of remaining requests??
I may very well be doing something wrong here - so please do point out what I am not seeing here! Any help would be much be appreciated because saying that this is getting a little frustrating is a little bit of an understatement