400 errors and x-rate-limit and x-rate-limit-remaining undefined

[JoshfireFactory]

Hello,
We are using authenticated calls to the Twitter search API.

The server that is doing the calls to the Twitter API is a shared server: It contains many applications that are configured with a different Twitter App and that are each calling the API. All of these apps have a cache mechanism and everything set up to not excess the rate limit.
This means that the calls are coming from the same IP but not from the same apps (they are apps created by our users).

The problem is that we very often receive errors 400 (with the exact 400 code).
The body of the response is empty and the x-rate-limit and x-rate-limit-remaining are “undefined”.

This is very strange. I would expect the x-rate-limit-remaining header to be 0 and the body to contain an “errors” attribute, telling us that we reached our rate limit. But it is not the case.
I suspect that the limit is applied to our IP adress instead of the user apps, but it shouldn’t since every call is authenticated.

Can someone help us ? Does anyone have the same issue ? Should I report it in the issue list ?

[episod]

Are your calls to api.twitter.com/1.1/search/tweets? (Authentication isn’t supported on the retiring search.twitter.com/search.json API)

When API v1.1 is rate limited, you should get a HTTP 429, not a HTTP 400 – it sounds like you’re being rate limited in the deprecated version of the API that doesn’t support auth.

[JoshfireFactory]

Hi, thanks a lot for your reply.
I can confirm that we are sending our authenticated calls to http://api.twitter.com/1.1/search/tweets.json And are receiving a “400 error”.

We are on Heroku, we noticed that by re-launching a new dyno (a new machine), the problem is fixed. I really suspect some sort of rate limit on the IP.

[episod]

Have you examined the response body of the request? What kind of queries are you sending? HTTP 400 gets sent by our APIs for a variety of reasons – including when a query is deemed too complex by the Search API.

[JoshfireFactory]

Hi Taylor,

The body is unfortunately empty. I would expect at least an error message of the form {“errors”:[{“message”:“something”,“code”:somecode}]}

We are sending very simple search queries with these parameters: { q: ‘joshfire’, include_entities: ‘1’}
We are also getting these 400 errors when calling the favorite endpoint: api.twitter.com/1.1/favorites/list.json with a single screen_name as parameter.
If we were over our limit, I think it would send an error message and an 429 error.

I read that you may fallback to API 1 when authentication on API 1.1 fails, is this correct ? As far as I can tell, authentication is done correctly, but how could we make sure authentication is correct?

Thanks for your support

[episod]

Do you proxy your requests at all? (Do you have control of the most significant parts of your request path?) Anything that might be serving that HTTP 400 that isn’t Twitter?

I’m pretty sure this isn’t rate limiting.

Are you able to capture the response headers you’re getting back?

[JoshfireFactory]

Hi,
I had a look to the full list of headers.
The status is 400 and there is only one header “connection close”

I’m not aware of a particular proxy, we are directly performing our call to the Twitter API from the Heroku server

Thanks for your help

[episod]

If at all possible, could you capture the full request-and-response cycle, complete with HTTP headers both ways? I’d like to track this down.

[JoshfireFactory]

Unfortunately, that’s not possible for the moment on our production server. We will try to reproduce the issue locally to be able to do so.

Here is the log I have:

send API request http://api.twitter.com/1.1/favorites/list.json
count=100 user=SubstanceActive
authorization OAuth realm="",oauth_timestamp=“1362697061”,oauth_nonce=“LgeT0e”,oauth_consumer_key=“BNmwKUKG7KdSVXovxGnFtg”,oauth_token=“207009500-LMeRNFskOwAhuHUGzgPNxRieWz8ZVjIVPn83dTw2”,oauth_version=“1.0”,oauth_signature_method=“HMAC-SHA1”,oauth_signature=“dwFudIZAI6uuanx%2BhFAYOYA58f4%3D”

json response received status=400
headers: connection close

I hope it can help.

[JoshfireFactory]

Hi,
We still have the issue. There are 2 strange things:

1. The request that is returning a 400 error is coming right after a request that had x-rate-limit-remaining 13.
2. After the response with the 400 error, any request to Twitter from this server gets a 400 error (independently from the twitter app used). If we restart this server, the Twitter API works as it should.

For example, this morning, the first request that lead to the 400 error was :
send API request http://api.twitter.com/1.1/favorites/list.json
authorization OAuth realm="",oauth_timestamp=“1363003410”,oauth_nonce=“7UCTtR”,oauth_consumer_key=“BNmwKUKG7KdSVXovxGnFtg”,oauth_token=“207009500-LMeRNFskOwAhuHUGzgPNxRieWz8ZVjIVPn83dTw2”,oauth_version=“1.0”,oauth_signature_method=“HMAC-SHA1”,oauth_signature=“RfrFnB%2BZq2cYtIB4Uitj0SfTk%2FA%3D”

[episod]

If at all possible, it would be great to see a sample of the HTTP headers being set, whether implicit or explicitly in your HTTP client. Another developer facing a similar issue with POST requests was making slightly malformed HTTP 1.1 requests. That you’re performing a GET here makes it a different situation, and I’m curious if there’s a particular aspect of your request that is unique.

[JoshfireFactory]

Hi,

We identified the issue. It was due to our HTTP request library which was keeping cookies between its requests. So after some use, one of our request was then sent to Twitter with a malformed cookie header (something like machin=blabla ; =true ;hello=there").

We now clean the cookie header for every request and it works fine.
Thanks a lot for your support.

[codesallday]

This might not be useful, but thought I should post anyway in case it helps someone randomly. I was losing my mind with the 401 Unauthorized errors after I thought I had everything working fine and then I realized I had checked the option that says:

Enable Callback Locking (It is recommended to enable callback locking to ensure apps cannot overwrite the callback url)

Unchecking this got it working again.