Twitter bots overwhelming website


#1

I’m trying to figure out Twitter is requesting hundreds of links on my site at random times in very short 1 minute intervals.

It comes in like a DDOS attack and I’ve never seen any webcrawler crawling bot do this. I’m seeing 300+ page requests in a single minute interval from a group of twitter owned IPs:
199.59.148.211
199.59.148.209
199.59.148.210
199.16.156.124
199.16.156.126

Why would Twitter do this?


#2

What are the User-Agents of the requests? Twitter may request your web site’s pages to generate a Twitter Card using the User-Agent such as Twitterbot/1.0


#3

The user agents of the requests are “Twitterbot/1.0” as you predicted.

Is there any way to configure how often these connections are made or a way I can throttle them?


#4

TwitterBot seems to be playing nice now. Thank you.


#5