Twitter bots overwhelming website

emojichat · 2014-11-17T09:59:15.443Z

I’m trying to figure out Twitter is requesting hundreds of links on my site at random times in very short 1 minute intervals.

It comes in like a DDOS attack and I’ve never seen any webcrawler crawling bot do this. I’m seeing 300+ page requests in a single minute interval from a group of twitter owned IPs:
199.59.148.211
199.59.148.209
199.59.148.210
199.16.156.124
199.16.156.126

Why would Twitter do this?

niall · 2014-11-17T16:33:19.333Z

What are the User-Agents of the requests? Twitter may request your web site’s pages to generate a Twitter Card using the User-Agent such as Twitterbot/1.0

emojichat · 2014-11-17T17:18:08.015Z

The user agents of the requests are “Twitterbot/1.0” as you predicted.

Is there any way to configure how often these connections are made or a way I can throttle them?

emojichat · 2014-12-07T03:17:02.381Z

TwitterBot seems to be playing nice now. Thank you.

andypiper · 2014-12-07T22:05:55.048Z