Get all tweets having a specific word

[ujwal8]

I am fairly new to twitter API. I am trying to get all tweets from start date to end date containing a specific word. I am writing a java program to do that and give me an output in a json format. I have a simple code in place where I tested using both Query class and also HttpGet method. However I am not getting what I am looking for. Also the number of status is very limited. I am more interested in 10-20 gb of data.
Any ideas,tips,guidance would be appreciated.

Thanks

[andypiper]

There’s a limit to the amount of data that you can retrieve via the public API - specifically historical data only goes back about a week. If you need a larger volume of data, across arbitrary dates, you should be looking into our commercial Gnip data products.

[SyedYasinSherif]

Hi & Greetings!

We had used the streaming API, it gives just about 1% of the tweets & now we have been trying to use the Twitter search API…

As documented the search API gives only about 1500 results (max) for a given query and goes back to extract results only from the past 7 days.

However, we would like to know if these search results offered by this API are coming from the index of the complete set of tweets or is it just hitting 1% of the actual Twitter index (just like streaming API).

This information would be very helpful for going forward.

[andypiper]

The 1% volume limitation refers to the streaming API only, however, be aware that the Search API is based on an incomplete index optimised for recency so not all Tweets may be included - this is per the documentation.

[SyedYasinSherif]

Thanks!, I appreciate your speedy response.

I understand from documentation and blogs that country/city specific tweets are possible to extract basis geo-location (lat/long paramenter with radius) and this depends entirely on the tweets, which are geo-tagged by the users, which by the way statistically results in only about 1%~2% of the tweets…so many tweets are lost.

Is their an alternative, by which we can get all the tweets of a specific country/city without having to use this geo-tag parameter.

Can one crawl externally to fetch country specific tweets/data; is this really possible?; their are some 3rd party vendors (i do not want to disclose the name of one such company I had interaction with) who claim to sell country specific tweets by way of Indexing them using crawlers;

Your guidance on this would be appreciated - Thanks!

[andypiper]

If the Tweets are not specifically geotagged by the user, then there’s no location data associated with them, and no way to know whether a user is sending them from a specific location or not.

It doesn’t matter whether someone is “crawling” the Twitter website (which by the way is against our terms of service and may result in permanent blocks of IP addresses, app IDs, etc) - if there is no geo-data to match against, there’s no way to identify where Tweets were sent from.

The “sort-of” exception to this is the geo-enrichment that is performed by Gnip, which can include inferred location information from a user’s profile - this is not the same as knowing that a specific Tweet was definitely sent from a particular country/city, but it does increase the volumes on a geoscoped-level. If this level of data is important to you then I would encourage you to consider the Gnip products from Twitter.

[SyedYasinSherif]

Thanks a ton for this information…very informative and helpful!

Does GNIP give data/tweets on trail for 2 to 3 days, for us to evaluate our business case.

[andypiper]

If you contact Gnip then their team can evaluate your use case and discuss with you - I’m not able to advise you further on commercial questions, but can answer API and platform issues.

[SyedYasinSherif]

Thanks!, you have been very helpful. Keep up the good work.