I have a list of users (say 10 users). I also have a list of keywords (say 2 keywords: happy and sad).
Now, I want to search for all the tweets in the history of these users mentioning either “happy” or “sad”.
I know of one way of doing this:
- For every user using the user_timeline get their last 3200 tweets ( rate is limited to 180 requests per window. So around 18 windows)
- Then from that collection search for the tweets which mention “happy” or “sad”.
Now, this looks like a really really bad way of solving this problem to me. To make it worse, assume that for a particular user, in the end, we found only 5 relevant tweets (that mentioned “happy” or “sad”). So, fetched 3200 tweets, wasted lot of time waiting on windows, just to fetch 5 relevant tweets. Now multiply this for 2000 users and 100 keywords.
Other Restriction: I want to use only twitter api and no third party solutions (reliability issues).
Any Suggestions?? I will greatly appreciate your help.