Volume drop in streaming API

litcandlewick · 2015-05-29T19:30:12.873Z

I’ve been collecting geotagged tweets through the streaming API for some time now. As of February, there has been a significant drop in the number of tweets received. This looks to be related to:

Potential adjustments to Streaming API sample volumes Announcements

The public Streaming API sample endpoints (aka POST statuses/filter and GET statuses/sample) are intended to be levelled at approximately 1% of the public Tweet volumes at any time. Due to some past inconsistencies in configuration, there have been periods of time where the volumes of Tweets delivered via the Streaming API may have exceeded these parameters. This notice is to indicate that over the next couple of weeks, we will be making changes to the public Streaming API to rebalance the v…

When I plot the number of hashtags tweeted per day as a function of time does, this is what I get:

which shows a massive drop off, both in mean and standard deviation. But is this behavior expected? Daily volume varies and so I would expect that my 1% share should also vary—at this scale, it appears mostly flat and makes me wonder if I’m being capped at a fixed total rather than a percentage. Could someone explain the massive drop, and also whether what I am seeing is still consistent with 1%? Is the “volume” that Twitter uses to determine 1% averaged over days, and thus, relatively constant?

boxnumber03 · 2015-05-29T22:08:38.795Z

I wonder if this is related to this issue. Twitter has begun “emphasizing” place instead of coordinates.

andypiper · 2015-05-30T13:47:51.341Z

I would suggest that this is exactly the reason - users who want to share their location on Tweets are now more likely to select a place from the UI than to share coordinates.

dartstar7 · 2015-06-02T15:24:18.403Z

Don’t you think that was a bit of a mistake to emphasize places? A user can say that they are at any place, that isn’t that accurate…

The graph really makes it look like twitter is filtering/cleaning the data. No up and downs based on the day/night cycle… just all level. However, I don’t think that twitter is filtering. I inquired with a friend who uses gnip and they are seeing the same drop.

Kinda disappointing.

andypiper · 2015-06-03T14:15:47.094Z

Thanks for the feedback. The decision to make this geo change was complex, and I realise that it may be disappointing to you. You will indeed see the same results in the Gnip API as the data from the firehose is just the same (but on a larger scale) that you will see through the public API.