Site Streams disconnect messages


#1

Hi.

We are seeing some strange behaviour in our app site streams, and hope you can give us some insights.
Our app currently uses site streams to handle streams for nearly 7k users. We do not have to track all users all the time, some of them come in as new users in the stream and some of them come out of the stream depending on some conditions. For this, we use control streams and the proper control_uris for adding and removing users.

We have implemented a solution to handle the cases where a stream is disconnected, may it be for an error (sometimes we receive random 503-service unavailable errors) or for a disconnection (sometimes we just see the stream has been closed). When this happens, we retrieve the users we have been monitoring in the affected stream and try to reconnect or send them to other non-closed streams (or just open new streams if there is not enough room).

This is the typical disconnection we receive:

status code: 200
headers: { ‘content-type’: ‘application/json’,
‘x-preamble-size’: ‘957988’,
connection: ‘close’ }

… and this typically triggers our reconnect logic.

We are also handling stall warnings and disconnect messages, trying to get some more info on those random stream closings we receive (just trying to know if we are stalling and that causes Twitter to close our stream or if there is any other problem).

Today, we have received this disconnect message:

{ disconnect:
{ code: 3,
stream_name: ‘SocialBro-sitestream98419-sunaomunakata’,
reason: ‘control request for SocialBro-sitestream98419 ipaddresshere /1.1/site.json sitestream’ } }

Which is really cryptic for me. This is what the streams guide says about code 3:

“3 Control request Control streams was used to close a stream (applies to sitestreams).”

A control stream was used to close a stream? All control stream request we perform are the ones covered in the guides: info requests, remove user requests and add user requests… as far as I know, there is no close stream request, am I right? All the requests we perform on the control uri fall beneath one of those 3 categories… so what does this mean? In addition, we received this disconnect message for a stream which we were removing users from, but after receiving this message, the stream did not close at all. It continues connected right now and is receiving events correctly (we did not receive any error/close event after this disconnect message and the users we were monitoring are still being tracked).

So, what does this disconnect message mean? Does it tell me that from Twitter’s side, a close request has been performed on the control stream (a ‘hypothetical’ close request that is not detailed here: https://dev.twitter.com/docs/streaming-apis/streams/site/control)??

Also, It would be great if you could provide us with some insights on what are those 503-service unavailable we receive randomly, as sometimes (and this is the reason why we are tracking all this down) we end up losing tweets, maybe because our reconnection process fails.

Thanks for your help, if you need any more details, please ask.