Why data gathering from API is very slow


#1

taking more than a day


#2

You need to be more specific. What data are you gathering? How many records? Is it because of the rate limits that it’s taking so long?


#3

i want to scrap tweets and related info against different hashtags. for some it works quite fast, but for others its taking so much time…is it old data issue or else


#4

Are you using the search endpoint or a stream (or something else)?


#5

streaming


#6

The Streaming API provides access to up to 1% of the Twitter firehose. If you’re tracking terms that are rarely used then this may mean it takes a longer time to gather a lot of data. It really depends on your parameters and methodology.


#7

so… alternate could be?


#8

Track a term with a higher volume (although it will still be limited to 1% of the full stream), or look into the commercial APIs (PowerTrack etc). I expect the latter may not be an option for you, and it is really difficult to suggest without understanding the scope and purpose of your data gathering.


#9

i want tweets for hybrid cars specifically… i found few tags as twitter is not much loaded with it


#10

This does not sound like a technical API issue, but just something to examine in your data requirements. Please let us know if the API is not behaving as expected. Thanks.