Issues reported with Streams since 10/21


#1

Hey everyone. We’ve been aware of a number of reports related to connections to the Twitter streaming over the past few days.

Thank you for the reports. First of all, there is no “everything on lock down” going on here, as was mentioned as a possibility in one thread - the Dyn issues last week clearly affected us as you’ll see from Tweets from @Support and may be related to these reports, but there’s no deliberate change that I’m aware of.

I am trying to work through 4 threads open on the forums (and some Tweets) to see whether I can identify a common and specific pattern between the reports.

As far as I understand it, the 420 error should only happen on connection - not during a connection, since the stream is already open. Several of the reports here seem to refer to very short-lived and then broken connections across public sample, user streams, and site streams. Presumably you mean that when you then try to reconnect you see the 420 error.

Creating new account and application keys are very unlikely to help in situations like this as the system will just start to log multiple connection attempts from different apps on single IPs, which is problematic (unless you are authorised for site streams).

What I believe I’m seeing in common here is:

  • all of the streaming endpoints seem affected
  • connections that have previously seen no issues now seem to disconnect 1-5 minutes after opening, and subsequent connection attempts produce 420 errors
  • in the case of Site Streams, multiple connections are not being allowed

I’d like to keep the investigation as focussed as possible so let’s please try to limit this thread to related discussion rather than everyone saying “me too” or “oh and I have this additional issue”. Equally please do not repeat what you posted in the other 4 threads, we’ve seen them.

Please bear with us - and where possible, please help us to narrow down the focus across these reports!


User stream is closing short after start
Twitter Stream Stall and Disconnections!
Site Streams limited to single connection now?
Streaming api
Single connection started getting 420
#2

Hi Andy -

I restart my statuses/sample collector every ten minutes, and since Friday only about 1 out of 10 of these sessions completes normally, processing 25K or so tweets. The other 9 out of 10 get dropped connections, which shows up in Tweepy as bad JSON, presumably from the payload ending early (“unterminated string”, etc.). The 420s were presumably from reconnecting too rapidly, but I’ve made my backoff much more conservative and those seem to have gone away. Still, if the connection gets dropped in the session, it pretty consistently gets dropped repeatedly - that is, all of my 10-minute sessions either run to completion, or else get lots of errors and collect only a few dozen tweets.

I think I’m seeing greater than usual connection problems on the REST API as well (e.g. “the read operation timed out” in search, statuses/user_timeline, trends/place), but the statuses/sample endpoint is particularly bad.

I’m happy to poke around my code or logs some more, or help diagnose this however I can.


#3

Thanks. I saw another report of slowness on REST which could be residual slowness on URL lookups. We have an active ticket trying to track this (streaming) so appreciate the new info.


#4

Yeah, the REST issues seem qualitatively different, and also much less frequent (although of course the volume of data moving around is much less than for streams).


#5

Hey Andy,
I noticed that the IPv4s of all streaming endpoints have changed to different /24s this afternoon. Is this change related to fixing the issues previously reported or just normal load balancing? I am asking because it appears that the user streams are not crashing anymore.


#6

Site streams are still preventing each app from making more than one connection, according to our more recent test here at Meshfire.


#7

Andy

If you examine the Meshfire account site stream logs you should see that when multiple servers try to setup connections the connection is allowed briefly and then shut down. Sometimes the shutdown will hit the current connection server (which is limited to 1,000 streams) and shut it down. It all should be in the logs.

Mike


#8

I don’t know if this is helpful, but here’s a log of recent connections and reconnections after the connection was closed. The pattern seems to be that the stream connection is established and then a few seconds later it is closed. This is using this code:

[I 161025 01:54:46 tweetstream:139] open_twitter_stream
[I 161025 01:54:46 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:54:46 tweetstream:231] stream connection established
[E 161025 01:54:50 tweetstream:355] close callback
[I 161025 01:54:50 tweetstream:139] open_twitter_stream
[I 161025 01:54:55 tweetstream:139] open_twitter_stream
[I 161025 01:54:55 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:54:55 tweetstream:231] stream connection established
[E 161025 01:55:13 tweetstream:355] close callback
[I 161025 01:55:13 tweetstream:139] open_twitter_stream
[I 161025 01:55:13 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:55:13 tweetstream:231] stream connection established
[E 161025 01:55:34 tweetstream:355] close callback
[I 161025 01:55:34 tweetstream:139] open_twitter_stream
[I 161025 01:55:34 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:55:34 tweetstream:231] stream connection established
[E 161025 01:55:49 tweetstream:355] close callback
[I 161025 01:55:49 tweetstream:139] open_twitter_stream
[I 161025 01:55:49 tweetstream:159] socket address:(‘199.16.156.217’, 443)
[I 161025 01:55:49 tweetstream:231] stream connection established
[E 161025 01:56:20 tweetstream:355] close callback
[I 161025 01:56:20 tweetstream:139] open_twitter_stream
[I 161025 01:56:20 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:56:20 tweetstream:231] stream connection established
[E 161025 01:56:50 tweetstream:355] close callback
[I 161025 01:56:50 tweetstream:139] open_twitter_stream
[I 161025 01:56:50 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:56:50 tweetstream:231] stream connection established
[E 161025 01:57:21 tweetstream:355] close callback
[I 161025 01:57:21 tweetstream:139] open_twitter_stream
[I 161025 01:57:21 tweetstream:159] socket address:(‘199.16.156.217’, 443)
[I 161025 01:57:21 tweetstream:231] stream connection established
[E 161025 01:57:51 tweetstream:355] close callback
[I 161025 01:57:51 tweetstream:139] open_twitter_stream
[I 161025 01:57:51 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:57:51 tweetstream:231] stream connection established
[E 161025 01:58:22 tweetstream:355] close callback
[I 161025 01:58:22 tweetstream:139] open_twitter_stream
[I 161025 01:58:22 tweetstream:159] socket address:(‘199.16.156.217’, 443)
[I 161025 01:58:22 tweetstream:231] stream connection established
[E 161025 01:58:53 tweetstream:355] close callback
[I 161025 01:58:53 tweetstream:139] open_twitter_stream
[I 161025 01:58:53 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:58:53 tweetstream:231] stream connection established
[E 161025 01:58:57 tweetstream:355] close callback
[I 161025 01:58:57 tweetstream:139] open_twitter_stream
[I 161025 01:59:02 tweetstream:139] open_twitter_stream
[I 161025 01:59:02 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:59:03 tweetstream:231] stream connection established
[E 161025 01:59:11 tweetstream:355] close callback
[I 161025 01:59:11 tweetstream:139] open_twitter_stream
[I 161025 01:59:11 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:59:11 tweetstream:231] stream connection established
[E 161025 01:59:15 tweetstream:355] close callback
[I 161025 01:59:15 tweetstream:139] open_twitter_stream
[I 161025 01:59:20 tweetstream:139] open_twitter_stream
[I 161025 01:59:20 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:59:20 tweetstream:231] stream connection established
[E 161025 01:59:33 tweetstream:355] close callback
[I 161025 01:59:33 tweetstream:139] open_twitter_stream
[I 161025 01:59:33 tweetstream:159] socket address:(‘199.16.156.217’, 443)
[I 161025 01:59:33 tweetstream:231] stream connection established
[E 161025 01:59:36 tweetstream:355] close callback
[I 161025 01:59:36 tweetstream:139] open_twitter_stream
[I 161025 01:59:41 tweetstream:139] open_twitter_stream
[I 161025 01:59:41 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:59:41 tweetstream:231] stream connection established


#9

Question for those who are experiencing issues with Site Streams: Are you using Twitter4j or some other SDK? I may have found an issue in Twitter4j site streams handling when error 420 is thrown. Still examining it, so we don’t know for sure.


#10

I am using Python - Tweepy


#11

OK, thanks. Not sure if the issue I found is real or not and won’t know until more testing is done. I do not presently think that this is the cause of the reported issues (though it seems like it could cause repeated attempts to open a connection) and I’m sure Andy has a better read on things.

But if I find a useful (though unrelated) fix, we’ll contributed it back to the Twitter4j project, as we do.


#12

I have what looks like a related issue. I was connected to the stream for the last couple of days. I had to restart my connection after updating my processing algorithm, and now I can’t connect anymore. Here is what I see in the logs:

[main] INFO com.twitter.hbc.httpclient.BasicClient - New connection executed: Hosebird-Client-01, endpoint: /1.1/statuses/filter.json?delimited=length&stall_warnings=true
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - Hosebird-Client-01 Establishing a connection
[hosebird-client-io-thread-0] WARN com.twitter.hbc.httpclient.ClientBase - Hosebird-Client-01 IOException caught when establishing connection to https://stream.twitter.com/1.1/statuses/filter.json?delimited=length&stall_warnings=true
[hosebird-client-io-thread-0] WARN com.twitter.hbc.httpclient.ClientBase - Hosebird-Client-01 failed to establish connection properly
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - Hosebird-Client-01 Done processing, preparing to close connection

When I looked inside com.twitter.hbc.httpclient.Connection.connect(), it looks like the response I’m getting from Twitter API has 200 status code, but no content.

Is there anything I can do to get the connection back. Or should I sit and wait for Twitter to resolve their issues?


#13

Hi guys,

i’m also experiencing a lot of stream disconnections.

I inspected the http headers of the twitter response. Is it normal that the connection header is set to ‘close’ and not ‘keep-alive’ ?

Don’t you think some network hardware may close the socket whith that kind of header ?

Thx


#14

Update - we believe that a failover has mitigated at least some of the issues, but there may still be a number of 420 errors. We are working to resolve and reduce these.


#15

I don’t get any 420 error… because I follow your reconnection process recommendations


#16

So to clarify, you’re posting on this thread because you’re seeing a large number of short-term disconnections?


#17

So after some investigation, the disconnections are due to the timer that disconnect the stream when there is no activity on the stream for more than 90sec like mentionned in your documentation : https://dev.twitter.com/streaming/overview/connecting

My timer is based on http level activity (the timer is reseted each time an http chunk is received from twitter’s server)


#18

Yes i’m posting on this thread because i have a lot of disconnect/reconnect on my streams. My application is running since several years and i’m experiencing this kind of issue only since the 10/21. That’s why i’m posting here.

I hope it’s the good place :slight_smile:


#19

Thanks for clarifying. We’ll keep this thread open for the overarching issue.


#20

Hi Andy

We are now getting 413 errors on our site streams connection attempts:

“Error 413: Parameter user_id has too many items specified”

We connect a single stream at a time to the site stream connection so it isn’t a matter of trying to initiate too many streams.