Streaming API rate limit response feature request


#1

Twitter,

I’ve been using the streaming API for a while now and conducting some analytics on various terms over various time intervals. I have a feature request, and it has nothing to do with providing more tweet data and less rate limiting. Actually, I’m relatively content with the 50 tweets per second that you’ll feed my application. I’d just like more detail in the rate limit response itself.

Two things that I’d love to see in such responses are listed here, with a discussion below. My main request is (2).
(1) A UNIX timestamp (or some potentially fake snowflake ID that bitshifts similarly, or whatever)
(2) The number of tweets that the response represents regardless of previous rate limit notifications, in addition to the ‘track’ record that is currently returned.

Ideally, the data saved from the streaming API should be capable of analysis after storage. Rate limit responses are a bit vague because they contain information about the connection at a point in time … but I have to add timestamps to them to record that time, or associate them with nearby data. It’s crossed my mind that the rate limit notifications may intentionally be timeless for various time window accuracy reasons - how long it takes to funnel and filter a massive amount of data. I suppose that’s a slightly more complicated issue if true.

My main request is (2). Having the ‘track’ component of the ‘limit’ notification is nice, but it makes analysis of both live and stored data challenging. Since Twitter knows how many tweets I previously missed (‘track’), and also know how much to add to that value when sending a rate limit response, it seems like a relatively simple request to ask for that data. Here’s are a few sample reasons why I want this:

(a) At 1393739497 I get the response ‘{“limit”:{“track”:328}}’. At 1393739563 I get a response ‘{“limit”:{“track”:338}}’. One of two things happened here. Either my stream was continued between the two responses and the second response represents 10 new tweets, or my stream disconnected and reconnected between the two responses and the second response represents 338 new missed tweets.
(b) When the time interval is greater than what is given in (a), analyzing data from today’s rate limit response may be dependent on data in an entirely different day … so we either store it in memory or go looking for it during the analysis.

Thanks for your time.

Cheers,
Jason