Not getting expected amount of TPM from public stream


#1

We have been having issues with the amount of traffic we’re getting from Twitter streams. We first noticed last Friday, that the TPM for our stream was low given the terms it was tracking. We noticed that userstreams has been having issues, via the status page. But we connect to the public stream, which aren’t listed, so we tried to debug the issue to make sure we weren’t doing anything unspecified by the API docs.

The particular phrases that we were concentrating on were about the olympics. But we also have several other phrases and user ids attached to the same stream. Removing everything except the sochi phrases, listed below, made the stream output the regular amount of Tweets we expected. But since we are tracking more than that, it’s not a viable solution.

Removing user IDs from the request, and the stall_warnings=true param did not help. Reducing the amount of phrases we were tracking to 50 seemed to have fixed it the first few times it was tried. Which made me think the list of phrases were not getting properly encoded for the POST, or there was something not supported. But filtering for phrases that met a certain pattern, such as /^[a-zA-Z0-9]+$/, did not help.

Using a GET request instead of a POST request did not help. Using user streams instead of public streams, without the follow param, did not help. Not urlencoding the commas, along with any other needed characters, in the requests didn’t help. Sending the filter_level=none param (even though docs say it won’t be deployed for some time but worth a try right?) didn’t help.

Here’s an example of the data we’re sending with a post request

{ url: 'https://stream.twitter.com/1.1/statuses/filter.json',
  body: 'track=warby%2Cwhatawhatawhata%2Cwinter%20olympics%2Cworld%20record%2Czoya%20nail%20polish%2Czupasdf&stall_warnings=true&filter_level=none&follow=432895323%2C435943551%2C43910797%2C476256944%2C47747074%2C47975734%2C486694111%2C50515656%2C516068866%2C538185166%2C551373363%2C555474658%2C5558312%2C59889953%2C600463589%2C60828944%2C64574086%2C64915003%2C72198806%2C7270292%2C72950235%2C73303753%2C7334402%2C74171837%2C74286565%2C7429102%2C75014376%2C75364211%2C76456274%2C76649729%2C7712452%2C77527412%2C78403308%2C78648493%2C8128442%2C83901492%2C87222643%2C87510313%2C88784440%2C90368802%2C92186819%2C93761782%2C94154021',
  headers: 
   { 'user-agent': 'stweam/0.2.0',
     'content-type': 'application/x-www-form-urlencoded; charset=utf-8' } }

We are also getting stall warning messages, which is odd when our app is running at ~3000 tpm, when it has sustained ~12k tpm consistently before.

{ code: 'FALLING_BEHIND',
  message: 'Your connection is falling behind and messages are being queued for delivery to you. Your queue is now over 30% full. You will be disconnected when the queue is full.',
  percentage: 36 }

Are we connecting incorrectly? Is there something going on with the streaming API?