Why are track values in LIMIT notices out of order and how to interpret them?

[vmknta]

I am using the streaming API in python via the tweepy client. https://dev.twitter.com/streaming/overview/messages-types#limit_notices describes LIMIT notices as:

Limit notices contain a total count of the number of undelivered Tweets since the connection was opened …

However, they appear to arrive out of order, with the counts going up and down. Based on documentation cited above, I expected the total counts to be increasing monotonically within the scope of a given connection. This makes me wonder whether the documentation is incorrect.

Below is an example of the LIMIT notices that my app received recently, listed in the order that they were received within a single connection cycle. Should I interpret these LIMIT notices by ignoring the order and extracting the max value to determine the total undelivered tweet count since the stream connection was opened as the twitter documentation would seem to suggest - 112 in this case - or should I be adding them up (1 + 4 + 8 + 6 + 10 + 11 + 4 + etc.)?

2015-04-14 17:04:07,578 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 1}}
2015-04-14 17:04:07,589 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 4}}
2015-04-14 17:04:15,379 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 8}}
2015-04-14 17:04:19,289 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 6}}
2015-04-14 17:04:19,358 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 10}}
2015-04-14 17:04:19,359 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 11}}
2015-04-14 17:04:19,360 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 4}}
2015-04-14 17:04:23,166 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 15}}
2015-04-14 17:04:24,239 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 44}}
2015-04-14 17:04:25,249 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 46}}
2015-04-14 17:04:27,140 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 39}}
2015-04-14 17:04:27,140 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 42}}
2015-04-14 17:04:27,164 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 51}}
2015-04-14 17:04:27,164 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 56}}
2015-04-14 17:04:27,165 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 59}}
2015-04-14 17:04:27,176 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 53}}
2015-04-14 17:04:28,681 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 63}}
2015-04-14 17:04:28,682 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 64}}
2015-04-14 17:04:28,684 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 71}}
2015-04-14 17:04:28,685 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 63}}
2015-04-14 17:04:29,720 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 67}}
2015-04-14 17:04:29,721 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 81}}
2015-04-14 17:04:29,722 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 69}}
2015-04-14 17:04:29,724 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 70}}
2015-04-14 17:04:31,552 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 68}}
2015-04-14 17:04:31,553 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 72}}
2015-04-14 17:04:32,979 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 79}}
2015-04-14 17:04:32,979 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 97}}
2015-04-14 17:04:32,981 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 87}}
2015-04-14 17:04:32,982 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 81}}
2015-04-14 17:04:33,003 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 88}}
2015-04-14 17:04:33,003 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 91}}
2015-04-14 17:04:33,003 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 102}}
2015-04-14 17:04:34,503 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 82}}
2015-04-14 17:04:34,533 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 110}}
2015-04-14 17:04:34,534 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 98}}
2015-04-14 17:04:34,534 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 87}}
2015-04-14 17:04:34,536 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 101}}
2015-04-14 17:04:36,831 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 108}}
2015-04-14 17:04:36,831 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 112}}
2015-04-14 17:04:36,831 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 112}}
2015-04-14 17:04:41,989 - twitter_direct_agent(23230) - WARNING - Received Twitter LIMIT message={u’limit’: {u’track’: 90}}

[vmknta]

Any twitter-heads out there that could shed some light on this problem? That would be much appreciated. Thank you.

[boxnumber03]

I get the same results. Not sure why this is. If the limit values are simply arriving out-of-order then I suppose the solution is to throw out values that are smaller than the last received value.

[vmknta]

Thanks @boxnumber03 for your suggestion. This is what I have been doing. However, if this is not a simple “out of order” issue, then my calculations could be way off.

It would be great to get someone from Twitter support/engineering to chime in to clarify this issue.

Many thanks in advance!

[litcandlewick]

This has been an issue for a while now. Without a proper answer, it is hard to estimate the number of tweets missed. See also:

Non-monotonic Rate Limit Message Counts

With the streaming API, I am receiving non-monotonic rate limit counts. I have only a single stream open. The link provides a plot of the rate limits against the number of rate-limit messages received. It appears as if there are two distinct linear (and noisy) trend lines. Below is a sample of extracted rate limits. How can I make sense of this to get a more reliable estimate of the number of tweets missed? # http://imgur.com/SU0QZly,aq86i81#1 # Stream restarted: Tue Aug 12 10:31:00 2014 Tu…

{"limit":{"track":x}} value unordered - negative deltas

Hi, we've recently noticed that the values returned by the Streaming API when we're being rate limited have become unordered. Historically the value x in {"limit":{"track":x}} was always increased in value from one payload to the next. Now it seems the values are unordered, resulting in negative deltas between pairs of payloads. Is this a deliberate change in the way the track messages are designed to work? A side effect of another change, which is expected to remain this way? Or an unintent…

[boxnumber03]

@litcandlewick Thank you for the links. I missed these discussions. Looking at the graph you posted, I take back my earlier guess of the limit values arriving out of sequence. It appears the values fluctuate between an upper and lower value (both monotonic) and, occasionally, do some jazz in between the upper and lower values. I made my own graph for fun. Below is a small section. Also, the difference between the upper and lower values is not constant but fluctuates quite a bit.

http://imgur.com/xyxi7VF

[litcandlewick]

That looks similar to what I was experiencing. Though, I had a second set of upper and lower values. It would be nice to get an official answer on this. As is, I don’t think the documentation is accurate, since a “cumulative count” should not fluctuate like this.

[vmknta]

This is still happening. IT’S FRUSTRATING THAT THIS BUG HAS PERSISTED SO LONG AND THERE IS STILL NO OFFICIAL SOLUTION. TWITTER, PLEASE STEP UP TO THE TASK AND HELP THE DEVELOPER COMMUNITY. THANK YOU

[boxnumber03]

…or, simply an acknowledgement of the situation would be nice.

Maybe someone at Twitter could advise which limit values to keep (the lows, the middles, or the highs in the above graph).

[edsu]

Yes, after watching a data collection job from the streaming API for a few days I noticed what seemed to be multiple counters at work. In my case there appeared to be 4.

[vivienanglesio]

Yeah, it’s a very annoying behavior. Please Twitter give us a sign