Retrieve latest tweets from twitter and insert in database using phirehose

php

#1

hi all
i am able to retrieve the tweets based on keyword and inserting in the database using the sample code in
http://140dev.com/twitter-api-programming-blog/streaming-api-enhancements-part-4-parsing-tweets-for-keywords/

but i want to get only limited no. of tweets(count=100) , so how to mention that in the code and also i want to retrieve only last 6 months tweets.
please tell me the where i have to mention the count and date in above sample code.


#2

Hi. The 140dev tool connects to the Stream to get current twitter activity - live data that’s going on right now. It’s not able to retrieve past data.


#3

Thanks for ur reply.
If i want to get tweets between particular dates (ie from 29/9/2015 to 29/102016) and what about the count. ie if i want to get only 100 tweets, then where i have to mention that in the code.
Also it is taking very long time to retrieve tweets from twitter and inserting the tweet in the database.
Iam running it as http://localhost/threats/parse_tweets_keyword.php


#4

Hello. As I said, you can’t do that from 140dev. It connects to the Stream. You have to use the REST API, but last I read it only goes back 7 days. There are other services you can use (I’ve never tried them), or if you’ve got the money you can try Gnip.

Regarding the length of time it takes to retrieve tweets, what’s your consume rate from get_tweets_keyword.php? How many tweets are coming into json_cache?
Parse_tweets can only process what’s available. Chances are you’re not consuming a lot of tweets. It all depends on what’s in your collection_words table.

Your php error log should display your consume rate. The records will show something like this:

[29-Sep-2016 13:08:39 America/New_York] Phirehose: Consume rate: 52 status/sec (3118 total), avg enqueueStatus(): 0.92ms, avg checkFilterPredicates(): 1.28ms (12 total) over 60 seconds, max stream idle period: 1 seconds.

In this, you can see I’m currently consuming 52 tweets/second or 3118/minute.

Also, depending how how many tweets you’ve already collected and inserted into your database, your MySQL will start to slow down. For the cheap VPS that I use, the magic number seems to be 9-10 million tweets before you start to notice the database can’t keep up.