Low TPS rate and very high limit numbers

[Alasdair]

Hello there

I’ve been running a connection to the public streaming API with a small filter / track set (7 accounts, 10 hashtags - the UK General Election). I am recieving ~ 10 TPS right now but every second am seeing limit messages showing up to ~500 tweets being limited.

Why are so few tweets making it through? Is their a procedure for determining what is at fault?

I’m not seeing any stall warnings, and my code is storing tweets in 10ms on average so can’t see how this is an I/O problem at my end.

As a charity we are unable to afford GNIP’s pricing and was hoping to take advantage of the public APIs to their limits.

[Alasdair]

And yet running a connection filtering on the word ‘the’ is returning me 52 TPS and the sample stream yields 60 TPS on the same hardware.

[boxnumber03]

That is strange. If it were me, I would try each of the 7 accounts and 10 hashtags separately to see if any one of them is causing this. I’m not sure how that would help you, but it might give you a clue.

Also, the limit messages are not trustworthy. The values jump around a lot, so that may be misleading you.