Issues reported with Streams since 10/21

[talkingfrog1950]

Andy

If you examine the Meshfire account site stream logs you should see that when multiple servers try to setup connections the connection is allowed briefly and then shut down. Sometimes the shutdown will hit the current connection server (which is limited to 1,000 streams) and shut it down. It all should be in the logs.

Mike

[mulka]

I don’t know if this is helpful, but here’s a log of recent connections and reconnections after the connection was closed. The pattern seems to be that the stream connection is established and then a few seconds later it is closed. This is using this code:

mulka/TweetStream

[I 161025 01:54:46 tweetstream:139] open_twitter_stream
[I 161025 01:54:46 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:54:46 tweetstream:231] stream connection established
[E 161025 01:54:50 tweetstream:355] close callback
[I 161025 01:54:50 tweetstream:139] open_twitter_stream
[I 161025 01:54:55 tweetstream:139] open_twitter_stream
[I 161025 01:54:55 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:54:55 tweetstream:231] stream connection established
[E 161025 01:55:13 tweetstream:355] close callback
[I 161025 01:55:13 tweetstream:139] open_twitter_stream
[I 161025 01:55:13 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:55:13 tweetstream:231] stream connection established
[E 161025 01:55:34 tweetstream:355] close callback
[I 161025 01:55:34 tweetstream:139] open_twitter_stream
[I 161025 01:55:34 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:55:34 tweetstream:231] stream connection established
[E 161025 01:55:49 tweetstream:355] close callback
[I 161025 01:55:49 tweetstream:139] open_twitter_stream
[I 161025 01:55:49 tweetstream:159] socket address:(‘199.16.156.217’, 443)
[I 161025 01:55:49 tweetstream:231] stream connection established
[E 161025 01:56:20 tweetstream:355] close callback
[I 161025 01:56:20 tweetstream:139] open_twitter_stream
[I 161025 01:56:20 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:56:20 tweetstream:231] stream connection established
[E 161025 01:56:50 tweetstream:355] close callback
[I 161025 01:56:50 tweetstream:139] open_twitter_stream
[I 161025 01:56:50 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:56:50 tweetstream:231] stream connection established
[E 161025 01:57:21 tweetstream:355] close callback
[I 161025 01:57:21 tweetstream:139] open_twitter_stream
[I 161025 01:57:21 tweetstream:159] socket address:(‘199.16.156.217’, 443)
[I 161025 01:57:21 tweetstream:231] stream connection established
[E 161025 01:57:51 tweetstream:355] close callback
[I 161025 01:57:51 tweetstream:139] open_twitter_stream
[I 161025 01:57:51 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:57:51 tweetstream:231] stream connection established
[E 161025 01:58:22 tweetstream:355] close callback
[I 161025 01:58:22 tweetstream:139] open_twitter_stream
[I 161025 01:58:22 tweetstream:159] socket address:(‘199.16.156.217’, 443)
[I 161025 01:58:22 tweetstream:231] stream connection established
[E 161025 01:58:53 tweetstream:355] close callback
[I 161025 01:58:53 tweetstream:139] open_twitter_stream
[I 161025 01:58:53 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:58:53 tweetstream:231] stream connection established
[E 161025 01:58:57 tweetstream:355] close callback
[I 161025 01:58:57 tweetstream:139] open_twitter_stream
[I 161025 01:59:02 tweetstream:139] open_twitter_stream
[I 161025 01:59:02 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:59:03 tweetstream:231] stream connection established
[E 161025 01:59:11 tweetstream:355] close callback
[I 161025 01:59:11 tweetstream:139] open_twitter_stream
[I 161025 01:59:11 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:59:11 tweetstream:231] stream connection established
[E 161025 01:59:15 tweetstream:355] close callback
[I 161025 01:59:15 tweetstream:139] open_twitter_stream
[I 161025 01:59:20 tweetstream:139] open_twitter_stream
[I 161025 01:59:20 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:59:20 tweetstream:231] stream connection established
[E 161025 01:59:33 tweetstream:355] close callback
[I 161025 01:59:33 tweetstream:139] open_twitter_stream
[I 161025 01:59:33 tweetstream:159] socket address:(‘199.16.156.217’, 443)
[I 161025 01:59:33 tweetstream:231] stream connection established
[E 161025 01:59:36 tweetstream:355] close callback
[I 161025 01:59:36 tweetstream:139] open_twitter_stream
[I 161025 01:59:41 tweetstream:139] open_twitter_stream
[I 161025 01:59:41 tweetstream:159] socket address:(‘199.16.156.20’, 443)
[I 161025 01:59:41 tweetstream:231] stream connection established

[eliasisrael]

Question for those who are experiencing issues with Site Streams: Are you using Twitter4j or some other SDK? I may have found an issue in Twitter4j site streams handling when error 420 is thrown. Still examining it, so we don’t know for sure.

[Eva_ZHOU00]

I am using Python - Tweepy

[eliasisrael]

OK, thanks. Not sure if the issue I found is real or not and won’t know until more testing is done. I do not presently think that this is the cause of the reported issues (though it seems like it could cause repeated attempts to open a connection) and I’m sure Andy has a better read on things.

But if I find a useful (though unrelated) fix, we’ll contributed it back to the Twitter4j project, as we do.

[skruk]

I have what looks like a related issue. I was connected to the stream for the last couple of days. I had to restart my connection after updating my processing algorithm, and now I can’t connect anymore. Here is what I see in the logs:

[main] INFO com.twitter.hbc.httpclient.BasicClient - New connection executed: Hosebird-Client-01, endpoint: /1.1/statuses/filter.json?delimited=length&stall_warnings=true
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - Hosebird-Client-01 Establishing a connection
[hosebird-client-io-thread-0] WARN com.twitter.hbc.httpclient.ClientBase - Hosebird-Client-01 IOException caught when establishing connection to https://stream.twitter.com/1.1/statuses/filter.json?delimited=length&stall_warnings=true
[hosebird-client-io-thread-0] WARN com.twitter.hbc.httpclient.ClientBase - Hosebird-Client-01 failed to establish connection properly
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - Hosebird-Client-01 Done processing, preparing to close connection

When I looked inside com.twitter.hbc.httpclient.Connection.connect(), it looks like the response I’m getting from Twitter API has 200 status code, but no content.

Is there anything I can do to get the connection back. Or should I sit and wait for Twitter to resolve their issues?

[vivienanglesio]

Hi guys,

i’m also experiencing a lot of stream disconnections.

I inspected the http headers of the twitter response. Is it normal that the connection header is set to ‘close’ and not ‘keep-alive’ ?

Don’t you think some network hardware may close the socket whith that kind of header ?

Thx

[andypiper]

Update - we believe that a failover has mitigated at least some of the issues, but there may still be a number of 420 errors. We are working to resolve and reduce these.

[vivienanglesio]

I don’t get any 420 error… because I follow your reconnection process recommendations

[andypiper]

So to clarify, you’re posting on this thread because you’re seeing a large number of short-term disconnections?

[vivienanglesio]

So after some investigation, the disconnections are due to the timer that disconnect the stream when there is no activity on the stream for more than 90sec like mentionned in your documentation : https://dev.twitter.com/streaming/overview/connecting

My timer is based on http level activity (the timer is reseted each time an http chunk is received from twitter’s server)

[vivienanglesio]

Yes i’m posting on this thread because i have a lot of disconnect/reconnect on my streams. My application is running since several years and i’m experiencing this kind of issue only since the 10/21. That’s why i’m posting here.

I hope it’s the good place

[andypiper]

Thanks for clarifying. We’ll keep this thread open for the overarching issue.

[talkingfrog1950]

Hi Andy

We are now getting 413 errors on our site streams connection attempts:

“Error 413: Parameter user_id has too many items specified”

We connect a single stream at a time to the site stream connection so it isn’t a matter of trying to initiate too many streams.

[phuc77]

Hi,

I’m also wondering about the “connection: close” header in the response when connecting to stream.twitter.com.
Shouldn’t this be keep-alive?

Using Apache httpclient 4.5.2, the request goes like this:
“http-outgoing-0 >> “POST /1.1/statuses/filter.json HTTP/1.1[\r][\n]””
“http-outgoing-0 >> “Authorization: OAuth oauth_consumer_key=”…”, oauth_nonce="…", oauth_signature="…", oauth_signature_method=“HMAC-SHA1”, oauth_timestamp=“1477402380”, oauth_token="…", oauth_version=“1.0”[\r][\n]""
“http-outgoing-0 >> “Content-Length: 31[\r][\n]””
“http-outgoing-0 >> “Content-Type: application/x-www-form-urlencoded; charset=UTF-8[\r][\n]””
“http-outgoing-0 >> “Host: stream.twitter.com[\r][\n]””
“http-outgoing-0 >> “Connection: Keep-Alive[\r][\n]””
“http-outgoing-0 >> “[\r][\n]””
“http-outgoing-0 >> “track=trump&stall_warnings=true””

And in the response, I get:
“http-outgoing-0 << “HTTP/1.1 200 OK[\r][\n]””
“http-outgoing-0 << “connection: close[\r][\n]””
“http-outgoing-0 << “content-type: application/json[\r][\n]””
“http-outgoing-0 << “date: Tue, 25 Oct 2016 13:33:02 GMT[\r][\n]””
“http-outgoing-0 << “server: tsa[\r][\n]””
“http-outgoing-0 << “transfer-encoding: chunked[\r][\n]””
“http-outgoing-0 << “x-connection-hash: a9aab11f353f7410a61066ae9b4d0b82[\r][\n]””
“http-outgoing-0 << “[\r][\n]””

If I try to do the same using https://dev.twitter.com/rest/tools/console
the response is:
HTTP/1.1 200 OK
x-connection-hash: 981b94c1fb752a65e152faff3aeedb77
Date: Tue, 25 Oct 2016 13:41:00 GMT
Content-Length: 0
Connection: keep-alive
Content-Type: application/json
Server: tsa

So, why am I getting “connection: close” header?

[skruk]

I’m getting the very same responses.

[andypiper]

I’m getting the very same responses.

Can you be specific - the same HTP headers, or the 413 error reported? Need to be very clear to understand how this relates to what is being reported. Thank you!

[skruk]

I’m getting HTTP/1.1 200 OK followed by Connection: close, Server: tsa

[SidecastrPicks]

We are also are getting “close” in our response header :

New Connection Status: 200, Reason: OK Headers: {“CONNECTION”=>“close”, “CONTENT_TYPE”=>“application/json”, “DATE”=>“Tue, 25 Oct 2016 17:32:36 GMT”, “SERVER”=>“tsa”, “TRANSFER_ENCODING”=>“chunked”, “X_CONNECTION_HASH”=>“281d7650dd1eeb0b287a6e956b0553ab”}

[eliasisrael]

Andy –

Please, please tell me you are onto some fix. We’re 5 days into trying to resolve this and it’s holding up our ability to sell, costing us users, and preventing us doing a new feature release as well.

We’re in kind of a lot of pain here. Please tell me we’re going to see the light at the end of the tunnel?
Thanks.

Best,
Eli