Why data gathering from API is very slow


taking more than a day


You need to be more specific. What data are you gathering? How many records? Is it because of the rate limits that it’s taking so long?


i want to scrap tweets and related info against different hashtags. for some it works quite fast, but for others its taking so much time…is it old data issue or else


Are you using the search endpoint or a stream (or something else)?




The Streaming API provides access to up to 1% of the Twitter firehose. If you’re tracking terms that are rarely used then this may mean it takes a longer time to gather a lot of data. It really depends on your parameters and methodology.


so… alternate could be?


Track a term with a higher volume (although it will still be limited to 1% of the full stream), or look into the commercial APIs (PowerTrack etc). I expect the latter may not be an option for you, and it is really difficult to suggest without understanding the scope and purpose of your data gathering.


i want tweets for hybrid cars specifically… i found few tags as twitter is not much loaded with it


This does not sound like a technical API issue, but just something to examine in your data requirements. Please let us know if the API is not behaving as expected. Thanks.