Twitter bots overwhelming website


This is happening again. Except now it is 600 page requests within a minute.

Twitter is requesting hundreds of links on my site at random times in very short 1 minute intervals.

It comes in like a DDOS attack and I’ve never seen any webcrawler crawling bot do this. I’m seeing 300+ page requests in a single minute interval from a group of twitter owned IPs:

Is there a way to throttle these requests on my end without blocking your ips?


@niall What is happening? I have started blocking some of these requests. 600 in a second is too many. If the server’s response is empty, just try agin later?

There are certain courtesies that people should know when writing a bot: