REST API timeout


#1

Two weeks ago, I found that some of my rest api request had to wait up to 15min. After that, I set the request timeout to 2min. Then lots of requests failed, and sometimes it seems that Twitter stops giving me response for tens of min.
Does anyone know why? Does twitter let client wait if there are too many requests in a very short time?


General API slowness
A lot Timeouts recently. Am I being rate limited?
#2

I am seeing the same thing. After a certain number of requests the next request hangs for precisely 15 minutes, then completes. I am checking the rate limit and I still have requests remaining. But this is not really a rate limit as it is not returning a status code 88, it is simply holding the request. Is this some kind of frequency oriented rate limit-esque mechanism possibly?

This appeared at the same time for me, 2 weeks ago. Has anyone seen the same thing or know what could be going on? Our app that has been working for a long time is suddenly experiencing drastic delays.


#3

@chunfengd Did you ever resolve this? What language/library are you using to make the requests? Still seeing this on our end.


#4

No. still the same. I’m using clojure.


#5

Interesting, so are we. Have you verified the problem outside of the clojure lib? I assume you are using this one: https://github.com/adamwynne/twitter-api


#6

Just FYI, I have isolated our problem to Ning (twitter-api depends on http.async.client depends on Ning, an asynchronous http java lib). I swapped out the twitter lib with direct calls to the twitter api with clj-http and it no longer hangs!

Not sure what exactly Ning is doing to cause the problem, but at least things are working for now.


#7

We started seeing the same problem between 08/11/2014 and 08/19/2014. We are using Apache HttpClient 4.3.1 to make calls to Twitter API. At some point Twitter responses block long enough to cause HttpClient to time out. We don’t make enough API calls to violate throttle limits. I bumped up HttpClient timeout and was able to receive a response after 20 mins of blocking!

I wonder if Twitter API server cleans up all these open HTTP requests that time out on the clients or it ends up processing them eventually waisting server resources. I think the API should be changed not to block but return a 429 or some other error code.

Any thoughts on this?


#8

Hi Ryan, I swapped out the twitter lib and create requests with http-kit. The good news is that it won’t block other requests, but I still see lots of timeouts (I set timeout of each request to 60s). Do u have many timeouts?


#9

Experiencing the same issues for the last several days too. We’re using twitter4j (which uses the Java JDK classes for HTTP).

Has anyone created an issue yet or know of one for this problem?


#10

I am having the same issue. I am using .NET 4.5 and HttpWebRequest/HttpWebResponse to issue request/get response.
I am making multiple requests in a row by changing each time the max_id but after 1 or 2 requests (no more than that) it just times out.
I tried to issue manually (i.e. via Postman) the very same request that fails when issued from code and I do get a proper response with tweets.


#11

You’re getting timeouts, not HTTP response codes?


#12

Yes, that’s correct!


#13

OK cool (and weird).

Can you share any example code - and which HTTP endpoints are you calling?


#14

We’re seeing frequent read timeouts on the following endpoints.

account/verify_credentials
users/lookup
friendships/create
statuses/retweets/id

However, I don’t think the issue is necessarily specific to these endpoints. That is, these are just the first endpoints called as part of some discrete tasks in our app and the timeout causes the task to fail (i.e. other endpoints that would have been called, and likely timeout, are not called).

I can include some sample code when I have access to source but it’s very vanilla (we use twitter4j) and was working fine for 2+ years until a couple weeks ago.


#15

Thanks for this - useful background info. I’ll see if I’m able to reproduce.


#16

I am only using the GET search/tweets endpoint.

As @james_jory it was working fine for a couple of months until a couple of weeks ago.

The main piece of code (with the task of getting a number of recent tweets inside a given area) is structured as follows:

//1. Build the url with all the parameters we are interested in

for (var i = 0; i < numberOfRequests ; i++)
{
	//2. If max_id is defined append it to the url
	
	// - _oAuthCreationService has all the information needed to authenticate the request
	// - _searchApiUrl is https://api.twitter.com/1.1/search/tweets.json
	// - searchParameters is a Dictionary with all the variables and corresponding values used in the request, for example:
	//		[q, ]
	//		[geocode, 33.9835968657262,-117.87334312989,85km]
	//		[result_type, recent]
	//		[include_entities, true]
	//		[count, 100]
	// - httpMethod is GET
	var authorizationHeader = HttpRequestResponse.GetTwitterHttpWebRequestHeader(_oAuthCreationService,
                    _searchApiUrl, searchParameters, httpMethod);
					
	// - urlBuilder at this point is something like https://api.twitter.com/1.1/search/tweets.json?q=&geocode=33.9835968657262,-117.87334312989,85km&result_type=recent&include_entities=True&count=100
	// - authorizationHeader we trust is correct since the first or second request is issued correctly
	var request = HttpRequestResponse.GetRequest(urlBuilder.ToString(), httpMethod, authorizationHeader);
	var responseJson = HttpRequestResponse.GetResponse(request);
	
	//3. Do something with responseJson
	//4. Get max_id from returned tweets and add it to the searchParameters
}

The other methods used for creating a request and getting a response are:

public static HttpWebRequest GetRequest(string url, string httpMethod, string authorizationHeader)
{
	var request = (HttpWebRequest)WebRequest.Create(url);
	if (!string.IsNullOrEmpty(authorizationHeader))
        {
		request.Headers.Add("Authorization", authorizationHeader);   
        }
    
	request.Method = httpMethod;
        request.Timeout = 30 * 1000;
	
	return request;
}

public static string GetResponse(HttpWebRequest request)
{
	var responseText = string.Empty;
    //Here is where we get stuck and get a WebException for the request timing out
    var response = request.GetResponse() as HttpWebResponse;

    if (response != null && response.StatusCode != HttpStatusCode.OK)
    {
		//Deal with error
    }
	
	if (response != null)
        {
		var stream = response.GetResponseStream();
		if (stream != null)
		{
			using (var reader = new StreamReader(stream))
			{
				responseText = reader.ReadToEnd();
                        }	
		}
	}

	return responseText;
}

I hope this helps!


#17

The timeouts appear to have been resolved sometime yesterday. Any changes on your side, @andypiper?


#18

Nothing that I can definitively report as a recent change, but I’m glad to hear that this is working better for you again! Sorry for the inconvenience.


#19

I can confirm, timeouts are not happening for me anymore on the search/tweets endpoint.


#20

We noticed Read timeout errors after 30 seconds, Twitter not responding. In our case, we made 4 calls in rapid succession and the first one or two woul succeed, the rest would for a period of time show the timeout. Due to the behavior, we began to suspect that Twitter may have lately changed how they defend against DDoS attacks. We spaced out our calls by 5 seconds and managed to circumvent the issue.