Twitter bots overwhelming website

[emojichat]

This is happening again. Except now it is 600 page requests within a minute.

Twitter is requesting hundreds of links on my site at random times in very short 1 minute intervals.

It comes in like a DDOS attack and I’ve never seen any webcrawler crawling bot do this. I’m seeing 300+ page requests in a single minute interval from a group of twitter owned IPs:

199.59.148.211
199.59.148.209
199.59.148.210
199.16.156.124
199.16.156.126

Is there a way to throttle these requests on my end without blocking your ips?

[emojichat]

@niall What is happening? I have started blocking some of these requests. 600 in a second is too many. If the server’s response is empty, just try agin later?

There are certain courtesies that people should know when writing a bot:

http://blog.mischel.com/2011/12/20/writing-a-web-crawler-politeness/