Why data gathering from API is very slow

[Mswr90]

taking more than a day

[DanielCHood]

You need to be more specific. What data are you gathering? How many records? Is it because of the rate limits that it’s taking so long?

[Mswr90]

i want to scrap tweets and related info against different hashtags. for some it works quite fast, but for others its taking so much time…is it old data issue or else

[DanielCHood]

Are you using the search endpoint or a stream (or something else)?

[Mswr90]

streaming

[andypiper]

The Streaming API provides access to up to 1% of the Twitter firehose. If you’re tracking terms that are rarely used then this may mean it takes a longer time to gather a lot of data. It really depends on your parameters and methodology.

[Mswr90]

so… alternate could be?

[andypiper]

Track a term with a higher volume (although it will still be limited to 1% of the full stream), or look into the commercial APIs (PowerTrack etc). I expect the latter may not be an option for you, and it is really difficult to suggest without understanding the scope and purpose of your data gathering.

[Mswr90]

i want tweets for hybrid cars specifically… i found few tags as twitter is not much loaded with it

[andypiper]

This does not sound like a technical API issue, but just something to examine in your data requirements. Please let us know if the API is not behaving as expected. Thanks.