Twitter bots overwhelming website


I’m trying to figure out Twitter is requesting hundreds of links on my site at random times in very short 1 minute intervals.

It comes in like a DDOS attack and I’ve never seen any webcrawler crawling bot do this. I’m seeing 300+ page requests in a single minute interval from a group of twitter owned IPs:

Why would Twitter do this?


What are the User-Agents of the requests? Twitter may request your web site’s pages to generate a Twitter Card using the User-Agent such as Twitterbot/1.0


The user agents of the requests are “Twitterbot/1.0” as you predicted.

Is there any way to configure how often these connections are made or a way I can throttle them?


TwitterBot seems to be playing nice now. Thank you.

closed #5