Slow responses from REST API endpoints

FollowingGuru1 · 2017-11-19T00:00:08.287Z

For the last 3 days our site (the.following.guru) has been experiencing very slow responses from various endpoints. Up until the 15th, the API responses have been fast. We are experiencing intermittent delays and timeouts. This is occurring most noticeably on users/lookup, users/search, friendships/create, friendships/destroy.

It has been suggested that automated actions such as follows may have gotten our service “limited via the automation rules” but our service does not provide any sort of bulk automation, everything is per click, passed through immediately to the endpoint, and results returned in realtime. Perhaps the twitter service’s algorithm has mistakenly identified our service, in which case what should I do about that? Twitter could obviously actually use our service and profile it, and see both sides and actually know what is going on instead of making assumptions based on a description of the service and algorithmic responses.

It seems coincidental that these issues come up right as the premium API is introduced. Our code involved in communicating with the REST endpoints has not changed in many months, has worked flawlessly till now. We have been in “alpha” for quite some time, and now preparing to move into beta with updated payment options and some graphical updates. Now this problem comes up.

Any suggestions? Is anyone else having similar issues?

FollowingGuru1 · 2017-11-19T05:14:17.548Z

from the logs:

2940962304 - Sat Nov 18 22:45:17 CST 2017 - - Requesting rate limit data…
2940962304 - Sat Nov 18 22:45:23 CST 2017 - - => Rate Limit returned in 6079ms
2940962304 - Sat Nov 18 22:45:23 CST 2017 - - Checking rate limits…
2940962304 - Sat Nov 18 22:45:23 CST 2017 - - rate_limit_context = 2940962304-UMjs8yZpA12yNpdpl7ak1eEtPWhq8MVmG4s91Hp
2940962304 - Sat Nov 18 22:45:23 CST 2017 - - limit_friends_list = 900
2940962304 - Sat Nov 18 22:45:23 CST 2017 - - remaining_friends_list = 899
2940962304 - Sat Nov 18 22:45:23 CST 2017 - - rate_limit_reset_friends_list = 1511067584
2940962304 - Sat Nov 18 22:45:23 CST 2017 - - Rate limit has not been exceeded. Carry on…
2940962304 - Sat Nov 18 22:45:23 CST 2017 - - Requesting a page of data from twitter…
2940962304 - Sat Nov 18 22:45:23 CST 2017 - - => API response returned in 320ms

2940962304 - Sat Nov 18 22:45:29 CST 2017 - - Requesting rate limit data…
2940962304 - Sat Nov 18 22:45:33 CST 2017 - - => Rate Limit returned in 3932ms
2940962304 - Sat Nov 18 22:45:33 CST 2017 - - Checking rate limits…
2940962304 - Sat Nov 18 22:45:33 CST 2017 - - rate_limit_context = 2940962304-UMjs8yZpA12yNpdpl7ak1eEtPWhq8MVmG4s91Hp
2940962304 - Sat Nov 18 22:45:33 CST 2017 - - limit_friends_list = 900
2940962304 - Sat Nov 18 22:45:33 CST 2017 - - remaining_friends_list = 898
2940962304 - Sat Nov 18 22:45:33 CST 2017 - - rate_limit_reset_friends_list = 1511067584
2940962304 - Sat Nov 18 22:45:33 CST 2017 - - Rate limit has not been exceeded. Carry on…
2940962304 - Sat Nov 18 22:45:33 CST 2017 - - Requesting a page of data from twitter…
2940962304 - Sat Nov 18 22:46:03 CST 2017 - - => API response returned in 30000ms

2940962304 - Sat Nov 18 22:50:56 CST 2017 - - => Rate Limit returned in 121ms
2940962304 - Sat Nov 18 22:50:56 CST 2017 - - Checking rate limits…
2940962304 - Sat Nov 18 22:50:56 CST 2017 - - rate_limit_context = 2940962304-UMjs8yZpA12yNpdpl7ak1eEtPWhq8MVmG4s91Hp
2940962304 - Sat Nov 18 22:50:56 CST 2017 - - limit_friends_list = 900
2940962304 - Sat Nov 18 22:50:56 CST 2017 - - remaining_friends_list = 897
2940962304 - Sat Nov 18 22:50:56 CST 2017 - - rate_limit_reset_friends_list = 1511067584
2940962304 - Sat Nov 18 22:50:56 CST 2017 - - Rate limit has not been exceeded. Carry on…
2940962304 - Sat Nov 18 22:50:56 CST 2017 - - Requesting a page of data from twitter…
2940962304 - Sat Nov 18 22:51:26 CST 2017 - - => API response returned in 30000ms

2940962304 - Sat Nov 18 22:51:46 CST 2017 - - Requesting rate limit data…
2940962304 - Sat Nov 18 22:51:56 CST 2017 - - => Rate Limit returned in 9762ms
2940962304 - Sat Nov 18 22:51:56 CST 2017 - - Checking rate limits…
2940962304 - Sat Nov 18 22:51:56 CST 2017 - - rate_limit_context = 2940962304-UMjs8yZpA12yNpdpl7ak1eEtPWhq8MVmG4s91Hp
2940962304 - Sat Nov 18 22:51:56 CST 2017 - - limit_friends_list = 900
2940962304 - Sat Nov 18 22:51:56 CST 2017 - - remaining_friends_list = 896
2940962304 - Sat Nov 18 22:51:56 CST 2017 - - rate_limit_reset_friends_list = 1511067584
2940962304 - Sat Nov 18 22:51:56 CST 2017 - - Rate limit has not been exceeded. Carry on…
2940962304 - Sat Nov 18 22:51:56 CST 2017 - - Requesting a page of data from twitter…
2940962304 - Sat Nov 18 22:52:26 CST 2017 - - => API response returned in 30001ms

30 seconds latency on the user/lookup endpoint.
6 to 9 seconds latency on rate limit endpoint.

Relevant code:
if($logging){LogEntry(“Requesting a page of data from twitter…”);}

$g_params = array(
“user_id” => $userlist,
“include_entities” => “false”
);

// -------------------------- record API roundtrip time
$_api_start_time = round(microtime(true) * 1000); // ms
// -------------------------- start timestamp

$user_info = $twitteroauth->post(‘users/lookup’, $g_params);

if (isset($user_info->errors)) {
// deal with the error

// ...... error handling stuff .....

} else {
// deal with the data

// -------------------------- record API roundtrip time
$_api_end_time = round(microtime(true) * 1000); // ms
$_api_round_trip = ($_api_end_time - $_api_start_time);
if($logging){LogEntry("=> API response returned in ".$_api_round_trip."ms ");}	
// -------------------------- end timestamp

// ...... response handling stuff .....

}

There is nothing going on between the start timestamp and the api call, nor between the api call and the end timestamp. The 30+ second api delays effectively timeout the page load.

APIuser342 · 2018-06-12T10:33:36.823Z

No one had any idea of why these slow responses did occur ?