Twitterbots shutting down server, seem to be seeking robots.txt info


#1

I just set up a new dedicated server the last week, and it ran fine for a few days. Then requests from Twitterbots overburdened the server enough to shut it down. They restarted it, but it just happened again. Around 10 times in the last 24 or so hours (basically every time).

HostGator, my server host, has been trying a dozen things to try to figure the problem out (and honestly they seem like they do not know what they are doing, just guessing at this and that, because they’ve tried so many things now).

The issue seems now to be requests from Twitterbot (“because the account is being overwhelmed with requests from twitterbot that use up all the available resources” Before the server is available to respond. mysql.)

11 199.59.148.209 twitter
11 199.59.148.210 twitter.

  3 199.59.148.211
  1 62.210.84.178
  1 66.249.65.42
  2 68.180.228.154
  5 70.66.236.119

NetRange: 199.59.148.0 - 199.59.151.255
CIDR: 199.59.148.0/22
NetName: TWITTER-NETWORK
NetHandle: NET-199-59-148-0-1
Parent: NET199 (NET-199-0-0-0-0)
NetType: Direct Assignment
OriginAS: AS13414
Organization: Twitter Inc. (TWITT)
RegDate: 2010-11-23
Updated: 2013-05-16
Ref: http://whois.arin.net/rest/net/NET-199-59-148-0-1

OrgName: Twitter Inc.
OrgId: TWITT
Address: 1355 Market Street
Address: Suite 900
City: San Francisco
StateProv: CA

1 /
1 /cambodia-becoming-popular-choice-expatriates-living-abroad/
1 /catalonias-bid-independence-uncovering-deep-rift-spains-political-social-fabric/
1 /contagion-happen/
1 /favicon.ico
1 /feed/
2 /headlines/secret-exchanges-and-informal-interactions-new-report-shines-light-on-lobbying-in-the-eu/
1 /health/how-to-do-pedicure-at-home-in-7-simple-steps/
20 /robots.txt
1 /south-sudan-parliament-re-deliberate-controversial-security-bill/
2 /tag/politics/
1 /the-actual-invention-of-tie-dye-by-hippies-on-acid-was-actually-caught-on-film-in-1964-video-2/
1 /violence-justify-violence/
1 /wp-admin/admin-ajax.php
1 /wp-cron.php?doing_wp_cron
1 /wp-cron.php?doing_wp_cron=1434785397.5876801013946533203125

00:29 0.17 0.18 2668 421 1548 1163 3932 139 0 0 0 46 1 0 1 10 2 dovenull mysql -
00:30 11.72 3.00 3748 1 28 82 3024 167 7 23 0 76 32 0 5 10 26 thespeak dovenull mysql
00:31 31.89 10.56 3732 3 28 97 1742 197 1 45 0 85 38 0 9 10 26 thespeak dovenull bloguni
00:32 39.41 16.76 3730 11 34 99 1568 176 0 35 0 71 27 0 5 10 29 thespeak dovenull bloguni
00:33 39.36 20.75 3716 5 29 113 1553 182 1 42 0 78 33 0 8 10 ? thespeak dovenull speakoff
00:34 39.52 24.38 3720 3 34 110 1500 180 0 39 0 70 26 0 4 10 32 thespeak dovenull bloguni
00:35 34.37 25.79 3673 10 76 157 1836 179 2 25 0 69 25 0 6 10 34 thespeak dovenull atsail
00:36 35.50 27.79 2894 9 62 936 1756 174 0 32 0 65 21 0 3 10 27 thespeak dovenull techupwi
00:37 32.06 28.26 3195 8 64 635 1825 167 1 23 0 61 14 0 5 10 25 thespeak dovenull mysql
00:38 27.47 27.67 2156 13 80 1673 2255 157 0 17 0 56 12 0 2 10 16 thespeak dovenull mysql
00:39 17.22 24.75 1616 17 87 2214 2802 150 0 9 0 56 11 0 2 10 11 thespeak dovenull mysql
00:40 11.96 21.93 532 20 101 3298 3719 139 0 0 0 45 1 0 0 10 2 dovenull mysql -
00:41 4.51 17.97 606 21 112 3223 3720 140 1 0 0 48 4 0 3 10 3 dovenull mysql thespeak
00:42 1.85 14.75 607 25 161 3223 3723 140 0 0 0 45 1 0 0 10 2 dovenull mysql -
00:43 0.72 12.08 609 25 161 3221 3724 140 0 0 0 47 3 0 2 10 2 dovenull mysql -
00:44 0.39 9.91 618 27 166 3212 3725 139 0 0 0 46 2 0 1 10 2 dovenull mysql -
00:45 0.21 8.12 661 40 189 3169 3726 139 0 0 0 46 1 0 1 10 2 dovenull mysql -
00:46 0.07 6.64 662 40 190 3168 3726 139 0 0 0 45 1 0 0 10 2 dovenull mysql -
00:47 0.10 5.45 665 41 190 3165 3726 140 0 0 0 45 0 0 0 10 2 dovenull mysql -
00:48 0.28 4.53 970 43 225 2860 3728 142 2 0 0 48 4 0 3 10 4 dovenull thespeak mysql
00:49 0.17 3.73 783 44 239 3046 3728 141 1 0 0 46 1 0 1 10 3 dovenull mysql thespeak
00:50 0.26 3.11 725 45 240 3105 3729 139 0 0 0 45 1 0 0 10 2 dovenull mysql -
00:51 0.18 2.57 726 46 240 3104 3730 139 0 0 0 45 1 0 0 10 2 dovenull mysql -
00:52 0.14 2.11 735 48 241 3095 3730 145 0 0 1 48 1 0 2 11 2 dovenull mysql -

This has shut us down a few times now (the last several).

Anyone have any idea what the problem is here? HostGator support does not seem to be able to answer this. They merely suggested “we could ban the IP”. As if a website can ban Twitter’s IP.