User Stream stopped sending tweets

AndreyGrunyov · 2014-04-10T07:23:29.000Z

Hi!
I have implemented User Streams service, everything working fine, but sometimes the tweets/events stop coming!
I can see that stream is still a live - there is no errors in log, it is just running
That’s very strange! And I had have to restart the connection manually
I’m using linqtotwitter component, but I had used my own implementation of Users Stream, and I had exactly the same problem before
and that’s why I’ve implemented User Streams by well known 3rd party library linqtotwitter

I have investigated possible cases from this resourse https://dev.twitter.com/docs/streaming-apis/connecting#Disconnections

A client establishes too many connections with the same credentials.
At the moment we have only one connection at the same time, and we do not creating no any other connections - we are processing only one Twitter account by User Streams at this time.

A client stops reading data suddenly.
Our service just read all new tweets/events and put them to the queue - it works very fast and we had not any problem with stopping reading data.

A client reads data too slowly.
That’s also not possible, our service just reading the data and move them to the queue for processing, and at the moment we have not so many tweets (about 150-200 per minute). So, it is not a problem

A streaming server is restarted. This is usually related to a code deploy and is not very frequent.
Twitter’s network configuration changes.
But in this case we should get TCP or HTTP error - but we haven’t got any errors

Stalls
if that happened, I would see it in the log, because I have implemented that 90sec timer which described here https://dev.twitter.com/docs/streaming-apis/connecting#Stalls

What could it be?

AndreyGrunyov · 2014-04-13T08:57:35.000Z

I have investigated this issue and found that it is always happen in case when there is no activity on current account (no events and no tweets/mentions/direct messages/etc) for a long time on nights
So, nothing comes for a long time, but stream still alive, I can not see any errors. And when new event should comes - it doesn’t come to the stream.

I have implemented a workaround solution by sending “keep alive” tweet - and seems everything now works fine.
But I don’t think that it is good idea.

Is there any proper solution?

AndreyGrunyov · 2014-04-21T22:19:04.000Z

Just got similar issue, but only for Direct Messages
I’m sending “keep alive” direct message every 6 min (to prevent Rate Limit)
but during 12 min Stream doesn’t get any direct messages!
How it can possible?
At the same time I can see mentions in stream and we are process them successfully
Looks like Users Stream loses tweets!

compston · 2014-06-30T20:35:15.000Z

Hi Andrey,

On the first portion – Are you setting a socket read timeout, or something different?

Second, for the behavior you described in your last message, it may be due to a buffering behavior that is common in many clients (e.g. Java) when decompressing Gzip streams. This would occur when there is very low volume on the stream, and would result in your http connection receiving the direct message, but bufferring it until a larger quantity of data has accumulated, making it look like it has disappeared or not arrived. It would match what you’re seeing here.

The behavior is described in our docs below, for reference. You should be able to override the default behavior fairly easily, as is shown in the java example there: https://dev.twitter.com/docs/streaming-apis/processing#gzip-compression