Timeouts to api.twitter.com

[SimonJouet]

Hi,

I’ve been using the twitter API for a while now without any problem, however in the last few days I’ve been having quite a few problem with connection to api.twitter.com timing out.

The error i’m getting is a fairly straightforward Error: connect ETIMEDOUT 104.244.42.66:443 in node. Doing some checks, the DNS can be resolved properly and the latency is okay (I guess depending if I hit a EU or a US server)

# ping api.twitter.com
PING api.twitter.com (104.244.42.130): 56 data bytes
64 bytes from 104.244.42.130: icmp_seq=0 ttl=57 time=8.336 ms
64 bytes from 104.244.42.130: icmp_seq=1 ttl=57 time=8.378 ms

# ping 104.244.42.66
PING 104.244.42.66 (104.244.42.66): 56 data bytes
64 bytes from 104.244.42.66: icmp_seq=0 ttl=53 time=96.461 ms
64 bytes from 104.244.42.66: icmp_seq=1 ttl=53 time=96.505 ms

Anyway I was wondering if anybody else is having a similar issue? This starting showing up in the last few days, it still works most of the time it’s just not very reliable. If it’s of any interest the server is running in OVH.

Thanks

[reynaldborer]

Hi,

We @ https://paper.li are experiencing the same issue. In fact, api.twitter.com is resolving to 4 different IPs and 2 of them are showing an increased latency since the short outage of 18.01.2018 20:15:00 (GMT+1).

Here are the mtr report for the 4 IPs behind api.twitter.com. It clearly show that latency & loss are higher on 104.244.42.2 and 104.244.42.66 (~ 100ms) than 104.244.42.194 et 104.244.42.130 (~ 15ms).

reynald@prod-processor1:~$ for ip in 104.244.42.2 104.244.42.194 104.244.42.66 104.244.42.130; do sudo mtr -r $ip; done
Start: Mon Jan 22 08:41:56 2018
HOST: prod-processor1             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.-- 37.187.165.252             0.0%    10    1.3   1.1   0.6   2.1   0.0
  2.-- be115.sbg-d1-a75.fr.eu     0.0%    10    0.2   0.2   0.2   0.3   0.0
  3.-- 10.95.48.8                 0.0%    10    1.1   1.4   1.1   1.9   0.0
  4.-- be100-1254.th2-1-a9.fr.eu 20.0%    10    6.5   6.5   6.2   6.7   0.0
  5.-- twitter1.par.franceix.net  0.0%    10    6.4   7.0   6.3   8.5   0.7
  6.-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  7.-- 104.244.42.2               0.0%    10   94.7  94.7  94.7  94.9   0.0
Start: Mon Jan 22 08:42:08 2018
HOST: prod-processor1             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.-- 37.187.165.252             0.0%    10    1.8   1.2   0.6   2.3   0.3
  2.-- be115.sbg-d2-a75.fr.eu     0.0%    10    0.2   0.2   0.1   0.3   0.0
  3.-- 10.95.48.10                0.0%    10    1.6   3.3   1.2  20.0   5.8
  4.-- be100-1258.gsw-1-a9.fr.eu  0.0%    10    6.6   6.5   6.2   6.7   0.0
  5.-- be100-2.th2-1-a9.fr.eu    20.0%    10    6.5   6.7   6.5   6.9   0.0
  6.-- twitter2.par.franceix.net  0.0%    10    6.5   7.5   6.4  16.2   3.1
  7.-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  8.-- 104.244.42.194             0.0%    10   14.1  14.1  14.0  14.1   0.0
Start: Mon Jan 22 08:42:20 2018
HOST: prod-processor1             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.-- 37.187.165.252             0.0%    10    0.8   1.0   0.6   2.0   0.0
  2.-- be115.sbg-d2-a75.fr.eu     0.0%    10    0.2   0.2   0.2   0.3   0.0
  3.-- 10.95.48.10                0.0%    10    1.2   5.8   1.1  44.5  13.6
  4.-- be100-1258.gsw-1-a9.fr.eu  0.0%    10    6.6   6.4   6.2   6.6   0.0
  5.-- be100-2.th2-1-a9.fr.eu    10.0%    10    6.7   6.6   6.5   6.9   0.0
  6.-- twitter1.par.franceix.net  0.0%    10   15.7   7.4   6.4  15.7   2.9
  7.-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  8.-- 104.244.42.66              0.0%    10   94.9  94.9  94.8  95.0   0.0
Start: Mon Jan 22 08:42:31 2018
HOST: prod-processor1             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.-- 37.187.165.252             0.0%    10    0.6   1.4   0.6   3.2   0.5
  2.-- be115.sbg-d1-a75.fr.eu     0.0%    10    0.2   0.2   0.2   0.3   0.0
  3.-- 10.95.48.8                 0.0%    10    1.6  13.0   1.2 111.6  34.7
  4.-- be100-1254.th2-1-a9.fr.eu 40.0%    10    6.2  11.0   6.2  21.1   6.2
  5.-- twitter2.par.franceix.net  0.0%    10    6.3   6.6   6.3   8.9   0.7
  6.-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  7.-- 104.244.42.130             0.0%    10   13.9  14.0  13.9  14.0   0.0

I have the same mtr output from different locations in France (OVH too) & Switzerland. It looks like it affects only Europe since I don’t have the same latency differences from a US-based server.

@andypiper are you already aware of the issue?

Cheers,
Reynald

[andypiper]

Thanks for these reports. I’ll have to check in with some of the network folks on this one. Your patience is appreciated.

[SimonJouet]

I’ve noticed so far three different behaviour for api.twitter.com

1. Works fine but depending on the server the latency is an order of magnitude different (<10ms to ~100ms)
2. The request results in a connection timeout
3. The SSL certificate is misconfigured resulting in a unable to verify the first certificate error in our node.js application

@reynaldborer have you noticed any issue with the SSL certificates too?

[reynaldborer]

@SimonJouet I haven’t noticed any SSL certificate issues. Only timeouts.

[andypiper]

Thanks both for the details here.

Due to some network issues, we anticipate that slower responses may be seen for a few days in the EU region and we are working to restore full performance again as soon as possible. We apologise for the disruption here - this is a complex issue that the team is working on at the moment.

I’m unable to comment on the SSL issue - I think that’s likely to be separate from the latency issue.

[reynaldborer]

Thanks for your confirmation @andypiper . Would it be possible for you to post a message here as soon as the situation is fixed? This way I can confirm that everything looks fine from my side too.

[andypiper]

I’ve got a watch on the relevant tickets internally so I’ll do that - again, this may take a number of days unfortunately, and we do appreciate your patience while we work to resolve.

[whattheshot]

This is a great thread. I was about to post the same. Since mid-december, I’m getting tons of timeout errors because of api.twitter.com (I’m on OVH as well). Before that, I was having maybe one cURL timeout every month, and my app does hundreds of API calls every day, if not thousands.

I extended the retry function which I implemented in the TwitterOauth PHP library earlier to retry when I was getting 500+ errors from Twitter API. Now it handles cURL timeouts as well.

Even with this retry function (up to 5 times), a lot of API calls didn’t make it. Various endpoints, not just one (status/update, mentions_timeline etc). After debugging cURL, I found out that most of the timeouts come from the IP 104.244.42.2.

Therefore, I changed my /etc/hosts to this:

104.244.42.194 api.twitter.com

I picked this IP because out of the 4 IPs of api.twitter.com, it was the less present in my curl Timeouts log through several days. And it worked. Since I did that, I almost get no timeout at all.

If that can help.

[andypiper]

Are you able to share any source IPs or netblocks that this is affecting?

[reynaldborer]

Hi @andypiper,

I can exhibit the problem from the following ranges (slightly obfuscated):

• 149.202.221.XX (OVH Strasbourg)
• 176.31.228.XX (OVH Roubaix)
• 185.19.29.XX (Exoscale)

Interestingly, it now seems that only 104.244.42.2 has an important latency (104.244.42.66 is now showing the same latency as the 2 remaining IPs).

Cheers,
Reynald

[andypiper]

Thanks - will aid in the work we’re undertaking.

[SimonJouet]

@andypiper Thanks for looking into that, did you get any news by any chance?

The problems looks to be mostly coming from 104.244.42.2, the source IP is 91.121.11.xx (OVH Roubaix)

PING 104.244.42.2 (104.244.42.2) 56(84) bytes of data.
64 bytes from 104.244.42.2: icmp_seq=1 ttl=54 time=99.2 ms
64 bytes from 104.244.42.2: icmp_seq=2 ttl=54 time=99.1 ms

PING 104.244.42.66 (104.244.42.66) 56(84) bytes of data.
64 bytes from 104.244.42.66: icmp_seq=1 ttl=58 time=18.0 ms
64 bytes from 104.244.42.66: icmp_seq=2 ttl=58 time=18.1 ms

PING 104.244.42.194 (104.244.42.194) 56(84) bytes of data.
64 bytes from 104.244.42.194: icmp_seq=1 ttl=58 time=11.3 ms
64 bytes from 104.244.42.194: icmp_seq=2 ttl=58 time=11.3 ms

PING 104.244.42.130 (104.244.42.130) 56(84) bytes of data.
64 bytes from 104.244.42.130: icmp_seq=1 ttl=58 time=11.3 ms
64 bytes from 104.244.42.130: icmp_seq=2 ttl=58 time=11.3 ms

Regarding the SSL error I posted previously the error looks to be irrelevant to this issue.

[andypiper]

The work to optimise and resolve this is ongoing on our side.

[iFonny]

Is the problem still there?
I’m still getting random ETIMEDOUT

From OVH Strasbourg (SBG1) - France

[whattheshot]

Same here. I still force the use of 104.244.42.194 to reduce the number of timeouts. (OVH Gravelines).

[maithree]

Same here . Exception details: System.Net.WebException: Unable to connect to the remote server — System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 104.244.42.194:443

[andypiper]

[andypiper]

The original issue here was resolved, and I am going to close this topic. If you have a specific network routing error please open a new topic. However, note that individual / one-off network issues are difficult to prioritize. Our infrastructure teams may dig in to cases where there is a common / broader issue inbound to our network, but we are unable to address every individual concern.