Hello. As I said, you can’t do that from 140dev. It connects to the Stream. You have to use the REST API, but last I read it only goes back 7 days. There are other services you can use (I’ve never tried them), or if you’ve got the money you can try Gnip.
Regarding the length of time it takes to retrieve tweets, what’s your consume rate from get_tweets_keyword.php? How many tweets are coming into json_cache?
Parse_tweets can only process what’s available. Chances are you’re not consuming a lot of tweets. It all depends on what’s in your collection_words table.
Your php error log should display your consume rate. The records will show something like this:
[29-Sep-2016 13:08:39 America/New_York] Phirehose: Consume rate: 52 status/sec (3118 total), avg enqueueStatus(): 0.92ms, avg checkFilterPredicates(): 1.28ms (12 total) over 60 seconds, max stream idle period: 1 seconds.
In this, you can see I’m currently consuming 52 tweets/second or 3118/minute.
Also, depending how how many tweets you’ve already collected and inserted into your database, your MySQL will start to slow down. For the cheap VPS that I use, the magic number seems to be 9-10 million tweets before you start to notice the database can’t keep up.
