Non-monotonic Rate Limit Message Counts


With the streaming API, I am receiving non-monotonic rate limit counts. I have only a single stream open.
The link provides a plot of the rate limits against the number of rate-limit messages received. It appears as if there are two distinct linear (and noisy) trend lines. Below is a sample of extracted rate limits. How can I make sense of this to get a more reliable estimate of the number of tweets missed?

# Stream restarted: Tue Aug 12 10:31:00 2014

Tue Aug 12 10:31:14 2014        {"limit":{"track":1}}
Tue Aug 12 10:32:13 2014        {"limit":{"track":4}}
Tue Aug 12 10:32:56 2014        {"limit":{"track":4}}
Tue Aug 12 10:33:29 2014        {"limit":{"track":4}}
Tue Aug 12 10:34:26 2014        {"limit":{"track":7}}
Tue Aug 12 10:34:51 2014        {"limit":{"track":6}}
Tue Aug 12 10:34:56 2014        {"limit":{"track":8}}
Tue Aug 12 10:35:13 2014        {"limit":{"track":10}}
Tue Aug 12 10:35:13 2014        {"limit":{"track":6}}
Tue Aug 12 10:35:16 2014        {"limit":{"track":9}}
Tue Aug 12 10:35:24 2014        {"limit":{"track":8}}
Tue Aug 12 10:35:28 2014        {"limit":{"track":13}}
Tue Aug 12 10:35:36 2014        {"limit":{"track":17}}

Why are track values in LIMIT notices out of order and how to interpret them?

Related to {"limit":{"track":x}} value unordered - negative deltas

This is still happening. Does anyone have an explanation?

And just to clarify, since I didn’t label the axes in the linked images, the linked images plot the rate-limit “track” value for each rate-limit message received (in order). The “track” value, as I understand it, is supposed to be the cumulative number of missed tweets from the time that the stream started. Image #0 shows an example of restarting the stream midway through. Image #1 zooms in on a segment of the figure, and is reproduced here:

Not only does the cumulative number of missed tweets seem to vary non-monotonically, it also appears as if there are two distinct signals in that plot, each with different variances. I think most people would expect to see staircase plots.


It looks like there are multiple counters working at the same time, which I thik are visible in your graph. I logged them as I was doing a long data collection job, and then color coded what appear to be the different counters: