Search API v1.1 with max_id parameter failing at very low level (Short gzip encoded response corruption)

johnbarratt · 2013-03-20T03:42:15.000Z

From about 6 hours ago we have noticed many low level errors coming through when doing a search with Twitter API v1.1 with a max_id parameter via the Twitter v4.6.1 gem (also v4.2 & v4.6.2). This doesn’t crash for all cases, but we can reproduce it reliably for many queries. The following fails reliably on multiple hosts, in multiple locations using multiple app & user keys:

Works just fine:

Twitter::Client.new.search(“melbourne”)
This one fails every time:

Twitter::Client.new.search(“melbourne”, :max_id => 311565637459775489)

As you can see from the backtrace, this is a very low level error :

/opt/ruby19/lib/ruby/1.9.1/openssl/buffering.rb:174:in `sysread_nonblock': end of file reached (Twitter::Error::ClientError)
        from /opt/ruby19/lib/ruby/1.9.1/openssl/buffering.rb:174:in `read_nonblock'
        from /opt/ruby19/lib/ruby/1.9.1/net/protocol.rb:141:in `rbuf_fill'
        from /opt/ruby19/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
        from /opt/ruby19/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
        from /opt/ruby19/lib/ruby/1.9.1/net/http.rb:2770:in `read_chunked'
        from /opt/ruby19/lib/ruby/1.9.1/net/http.rb:2750:in `read_body_0'
        from /opt/ruby19/lib/ruby/1.9.1/net/http.rb:2710:in `read_body'
        from /opt/ruby19/lib/ruby/1.9.1/net/http.rb:1029:in `block in get'
        from /opt/ruby19/lib/ruby/1.9.1/net/http.rb:1322:in `block (2 levels) in transport_request'
        from /opt/ruby19/lib/ruby/1.9.1/net/http.rb:2671:in `reading_body'
        from /opt/ruby19/lib/ruby/1.9.1/net/http.rb:1321:in `block in transport_request'
        from /opt/ruby19/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
        from /opt/ruby19/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request'
        from /opt/ruby19/lib/ruby/1.9.1/net/http.rb:1293:in `request'
        from /opt/ruby19/lib/ruby/1.9.1/net/http.rb:1286:in `block in request'
        from /opt/ruby19/lib/ruby/1.9.1/net/http.rb:745:in `start'
        from /opt/ruby19/lib/ruby/1.9.1/net/http.rb:1284:in `request'
        from /opt/ruby19/lib/ruby/1.9.1/net/http.rb:1026:in `get'
        from /opt/ruby19/lib/ruby/gems/1.9.1/gems/faraday-0.8.4/lib/faraday/adapter/net_http.rb:73:in `perform_request'
        from /opt/ruby19/lib/ruby/gems/1.9.1/gems/faraday-0.8.4/lib/faraday/adapter/net_http.rb:37:in `call'
        from /opt/ruby19/lib/ruby/gems/1.9.1/gems/faraday-0.8.4/lib/faraday/response.rb:8:in `call'
        from /opt/ruby19/lib/ruby/gems/1.9.1/gems/faraday-0.8.4/lib/faraday/response.rb:8:in `call'
        from /opt/ruby19/lib/ruby/gems/1.9.1/gems/faraday-0.8.4/lib/faraday/response.rb:8:in `call'
        from /opt/ruby19/lib/ruby/gems/1.9.1/gems/faraday-0.8.4/lib/faraday/request/url_encoded.rb:14:in `call'
        from /opt/ruby19/lib/ruby/gems/1.9.1/gems/faraday-0.8.4/lib/faraday/request/multipart.rb:13:in `call'
        from /opt/ruby19/lib/ruby/gems/1.9.1/gems/twitter-4.6.1/lib/twitter/request/multipart_with_file.rb:14:in `call'
        from /opt/ruby19/lib/ruby/gems/1.9.1/gems/faraday-0.8.4/lib/faraday/connection.rb:226:in `run_request'
        from /opt/ruby19/lib/ruby/gems/1.9.1/gems/faraday-0.8.4/lib/faraday/connection.rb:87:in `get'
        from /opt/ruby19/lib/ruby/gems/1.9.1/gems/twitter-4.6.1/lib/twitter/client.rb:81:in `request'
        from /opt/ruby19/lib/ruby/gems/1.9.1/gems/twitter-4.6.1/lib/twitter/client.rb:64:in `get'
        from /opt/ruby19/lib/ruby/gems/1.9.1/gems/twitter-4.6.1/lib/twitter/api/utils.rb:82:in `object_from_response'
        from /opt/ruby19/lib/ruby/gems/1.9.1/gems/twitter-4.6.1/lib/twitter/api/search.rb:32:in `search'

johnbarratt · 2013-03-20T04:19:22.000Z

Further to this, it seems it is more generally occurring when there are no, or few results, eg this geocoded search in the high Arctic, and a search for gibberish give the same error as above:

Twitter::Client.new.search("", :geocode => “67.6761,-35.1562,10km”)
Twitter::Client.new.search(“134ya89jhakjndkasyd07yasdnadjaod9asd”)

gggodhwani · 2013-03-20T05:00:50.000Z

In-fact, all the queries with max_id > 1 are failing since almost 11 hours. Only queries with max_id = 1 are working fine for now. Seem to be issue in Twitter Search Index.

johnbarratt · 2013-03-20T09:13:15.000Z

This seems to be more widespread than just search in fact. We are now seeing errors with posting a new tweet, and accessing non-existent tweets. It appears to be a bad interaction with this particular library, which has been working fine up until this point. It can also be reproduced by using this simple call :

Twitter::Client.new.status(313986097430540288)

milou74 · 2013-03-20T09:41:15.000Z

I am also getting no results using Twitter search with parameter max_id

johnbarratt · 2013-03-20T12:05:01.000Z

Upon further investigation, it appears most generically to be a problem with short responses when using gzip encoding. Removing the ‘accept-encoding’ header from the request fixes the problem in all cases. So seems there has been a bug introduced in the webservers for these small responses with Gzip.

To work around the problem until it is resolved (and to also transfer a lot more data ) you can patch the request method in the Twitter gem like this by add in an empty accept-encoding header. This prevents the default ruby http lib from sending its own header with gzip allowed :

module Twitter class Client def request(method, path, params={}, signature_params=params) connection.send(method.to_sym, path, params) do |request| request.headers[:authorization] = auth_header(method.to_sym, path, signature_params).to_s request.headers['accept-encoding'] = "" ## Disable gzip encoding in responses end.env rescue Faraday::Error::ClientError raise Twitter::Error::ClientError rescue MultiJson::DecodeError raise Twitter::Error::DecodeError end end end

daretorant · 2013-03-20T16:56:25.000Z

We’re seeing this same behavior, except we are using since_id rather than max_id. I imagine based on your comment re: repro’ing it with the simple new status call, this is a pretty widespread issue.

Does anyone know if Twitter has acknowledged this issue and/or is working on it? We will look into the suggested workaround (thanks @johnbarratt), but would like to avoid patching the Twitter gem if possible.

TapastreetDev · 2013-03-20T17:09:56.000Z

Hello,
I have the same problem for some cities.
I hope the problem will be resolved soon.
However when there is a lot of data, the queries Twitter back correctly.

kurrik · 2013-03-20T17:10:54.000Z

Is it possible that the Ruby gem just isn’t flushing its gzip buffer correctly? I can see that the responses are rather small, but curl is happy to parse the response:


$ curl --get 'https://api.twitter.com/1.1/search/tweets.json' --data 'q=134ya89jhakjndkasyd07yasdnadjaod9asd' --header 'Authorization: OAuth oauth_consumer_key="xxx", oauth_nonce="dcc8f7b029639f183047bdef614544cd", oauth_signature="xxx", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1363799150", oauth_token="7588892-xxx", oauth_version="1.0"' --compressed --verbose
< HTTP/1.1 200 OK

< cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0

< content-type: application/json;charset=utf-8

< date: Wed, 20 Mar 2013 17:08:48 GMT

< expires: Tue, 31 Mar 1981 05:00:00 GMT

< last-modified: Wed, 20 Mar 2013 17:08:48 GMT

< status: 200 OK

< x-access-level: read-write

< x-frame-options: SAMEORIGIN

< x-rate-limit-limit: 180

< x-rate-limit-remaining: 174

< x-rate-limit-reset: 1363800081

< Transfer-Encoding: chunked

< Content-Encoding: gzip

<

Connection #0 to host api.twitter.com left intact
Closing connection #0

SSLv3, TLS alert, Client hello (1):

{“statuses”:[],“search_metadata”:{“completed_in”:0.004,“max_id”:314423228615499776,“max_id_str”:“314423228615499776”,“query”:“134ya89jhakjndkasyd07yasdnadjaod9asd”,“refresh_url”:"?since_id=314423228615499776&q=134ya89jhakjndkasyd07yasdnadjaod9asd&include_entities=1",“count”:15,“since_id”:0,“since_id_str”:“0”}}

gurrinderk · 2013-03-20T18:25:00.000Z

The suggested fix by @johnbarratt does work. I have forked the gem for now with these changes. Feel free to use it, until we get a more permanent fix https://github.com/gurrinder/twitter

daretorant · 2013-03-20T18:27:05.000Z

@kurrik any ETA on a fix for this on the API? haven’t seen any updates or acknowledgements from Twitter.

P.S. Is there a bug system that Twitter uses similar to Facebook, or is this discussion forum our only option to escalate / raise awareness regarding issues?

gggodhwani · 2013-03-20T18:51:29.000Z

I agree with @johnbarratt, the problem is temporarily resolved by changing the response header, as response files received for these queries are not gzipped files. Proposed workaround by @johnbarratt is working for my case.

daretorant · 2013-03-20T19:11:33.000Z

The workaround is working for us too… sucks that we all have to fork the gem to do this though

johnbarratt · 2013-03-20T22:08:04.000Z

Interesting, could well be something specific to ruby, however it wasn’t present prior to yesterday, and it hasn’t been an issue with other services that I am aware of, and this code has been static for a long time.

Also it could be that curl is handling this error condition gracefully (eg if a buffer is an incorrect size, it ignores it), while the ruby libs don’t.

It could be that there is a subtle difference in the way SSL/GZIP/short content is served from Twitter that is causing this bug. Was there a change in the server configs yesterday that might help point to which side has the bug? I can find the exact minute this problem started if you need it for reference.

EDIT: About 21:10 UTC on the 19th of March was when we first saw it start happening, and there appeared to have been a secondary, though smaller increase in the problem around 21:22 UTC.

Thanks,

JB.

johnbarratt · 2013-03-20T22:10:42.000Z

FYI. You can just include the above code once somewhere in your code, after the Twitter gem has loaded, before you make calls to Twitter, and it will override the function. This at least saves having to do a full fork of the gem, though this form of patching is still far from ideal.

waynep · 2013-03-21T00:57:27.000Z

Some acknowledgement from the Twitter team would be nice here. Either it’s a known issue and won’t be fixing it on their side OR they’re working on it so we know how to proceed.

daixque · 2013-03-21T04:24:00.000Z

I have same problem. I’ve found some issues about this problem on github.

https://github.com/sferik/twitter/issues/354
https://github.com/sferik/twitter/issues/369
https://github.com/sferik/twitter/issues/370
Now I’m trying to access against HTTP endpoint (see #354).
client = Twitter::Client.new(:endpoint => “http://api.twitter.com”)
Indeed this should be temporary solution, but it currently works fine.

kurrik · 2013-03-21T16:39:40.000Z

I don’t think this is a good solution. I was able to connect from Ruby using an explicit SSL setup, see https://gist.github.com/sferik/5208022 for more information.

Ultimately, I don’t know why the behavior of net/http changed, but it’s flat-out sending the wrong type of request (a http request to a https endpoint) and I don’t consider this a bug on Twitter’s end.

kurrik · 2013-03-21T17:05:03.000Z

We have been seeing some “end of file” issues with Ruby which appear to be related to SSL, but not really related to gzip (they’re issues if you connect with or without gzip). My feeling is that maybe some server on Twitter’s end has started being more strict about SSL connections and the gzip switch may be routing your request slightly differently.

If you do identify a gzip issue, please start a new thread - I’m going to treat this one primarily as dealing with SSL.

kurrik · 2013-03-21T17:09:00.000Z

From what I’m seeing, request from Ruby are malformed. It may be possible that we’ve started being more strict about what we accept, but ultimately changes need to occur in Ruby code to send the request correctly, specifically setting up SSL explicitly:

http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_PEER

You can see the tests I’ve been running in this gist:

gist.github.com

https://gist.github.com/kurrik/5214671

twitter_ssl.rb

require 'net/https'

# Returns {"errors":[{"message":"Bad Authentication data","code":215}]}
# Issues a HTTPS request to: GET https://api.twitter.com/1.1/users/show.json?user_id=33978
# Prints: Explicit SSL: #<Net::HTTPBadRequest:0x10b65f188>
uri = URI.parse("https://api.twitter.com/1.1/users/show.json?user_id=33978")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
request = Net::HTTP::Get.new(uri.request_uri)

This file has been truncated. show original

Regarding bugs and escalations, this is the correct place to surface API issues. We have an issue tracker as well, but in cases like this it’s nice to have a discussion thread so that we can collect more information.