Intermittent 404 responses from REST API

api

#1

Hello,

We’re seing some intermittent 404 failures on some API calls, for instance user timeline, mentions, DMS, just like if only one of many servers was sending 404 errors.

Anyone having this issue ?

Cheers


GET statuses/lookup returning empty string
#2

Same problem here.
I’m encountering about 10 404 per hour while calling regularily routes such as mentions, dms, search and user_timeline.
Had the same a dozen days ago after the leap second, and it just started again about 24 hours ago


#3

Same issue here, we have started to get intermittent 404 (FileNotFoundException from Twitter4J).
It started suddenly on july 16th, around 1am (UTC).
Do you still have this issue?


#4

Yes, getting this issue on users/show endpoint. Difficult to reproduce as it happens on approximately 2-3% of requests.


#5

Still having this issue as of today …


#6

Also continuing here. This is clearly an issue coming from Twitter’s side. Could anyone check what’s happening?


#7

I can confirm we are still being affected.

@Gazou_medialab2 @Intrepidd I’d expect more complaints if this issue was widespread. Anything we have in common? We are getting this issue on 2 separate apps, using separate libraries (one PHP, one NodeJS). We are using Rackspace Cloud in the London region for hosting. We are getting this issue on two separate endpoints - statuses/timeline and users/show . It happens on approximately 1-3% of requests.


#8

Good idea to investigate @jamiel :smile:
I’m having the issue on all my projects relying on Twitter, which all use the same python library to communicate with the API https://github.com/sixohsix/twitter
I know about other people using my projects on other servers and having the same problem.
I cannot measure the exact percentage of requests concerned but an order of a few percents seems right.
As said earlier, it concerns in my case the following routes : statuses/mentions_timeline, statuses/user_timeline, search/tweets, direct_messages


#9

Indeed good idea @jamiel .
We are using Twitter4J java lib and we are hosted on AWS EU-West region (Ireland).

Example of API endpoints with intermittent 404:

I don’t know what happened yesterday, but the problem suddenly stopped during 3 hours (around 4PM UTC) as you can seen on this graph:


You can also see on the graph that the problem started on July 16th (without any change on our side), as reported by most of us.


#10

Regarding frequency and volume, I can also point out I encountered 959 “404 error” on these cumulated routes since the 16


#11

Just wanted to chime in and say that I also experience the issue on a wide range of endpoints:

friendships/create
friendships/destroy
followers/list
users/show
search/tweet

Problem started around the 16th as well. I’m don’t see any sign of it having decreased during the 3 hours @benorama mentioned, but my logs are a mess so it’s hard to tell for sure.


#12

Too early to party, today the unmanaged 404 is back


#13

Really can’t find any reproducible way to report this issue. We are unable to uncover a pattern to recreate it. The same query works immediately afterwards, so I guess we just need to start building 404 retries into all our Twitter libraries if Twitter don’t feel this is an issue their end? :fearful:


#14

Although it looks like we are all having the same issue while we are all calling the API with our own ways or different libraries…
This really sounds to me like a problem to fix on twitter’s side…


#15

Hi all,

We have managed to put a temporary hack in to clear the errors this morning by forcing api.twitter.com to known working IPs. It seems one of the servers being returned in their DNS resolution might be having issues.

To resolve you can try adding this into your /etc/hosts on any boxes making API calls:

199.16.156.231 api.twitter.com

Some libs cache DNS results so you may need to restart your application for the change to take effect.

This is of course temporary and will stop working the day 199.16.156.231 is no longer a Twitter API server, so will need some alerting on this for production services and worth adding multiple IP’s (use “dig +short api.twitter.com A” to list possible options to test with) whilst Twitter resolve the issue.

Don’t forget to set yourself a reminder to remove it in the future!

Jamie


#16

Since CET morning it works fine also without any DNS trick. We are monitoring and catching any 404 since the beginning but I hope Twitter keep us informed about when their intermittent 404 is solved.
We related our fixes to this Twitter issue,
Cheers
Renato


#17

We’re still seeing lots of errors


#18

Probably is now in some way “zone dependent” within official Twitter balancers. Or only lucky.
Btw our servers located in NY1&NY2 plus EU1 didn’t get any 404 since this morning (CET tmz). Where usually we got since the beginning at any time.


#19

Indeed, we don’t get any 404s since this morning, but we’ve got plenty of 500s instead :frowning: (much more than 404s).

500:Something is broken. Please post to the group (https://dev.twitter.com/docs/support) so the Twitter team can investigate.
message - Internal error
code - 131

Here is the increase in number of exceptions by our error tracking system:

And nothing official on Twitter status page https://dev.twitter.com/overview/status.

Don’t you get thousands of 500s instead of 404s?


#20

Here in France, still getting (a bit less) 404s, haven’t seen any 500.
Did not try and use a dedicated host, I’d rather know it’s happening than hide it and break another day.