Twitterbot making huge number of requests to my server



Twitterbot is periodically killing my server. I tweet a lot (thousands in the course of a month) of unique links that resolve to my server. Periodically Twitterbot requests many old links in a short period of time. This puts a very high load on my server. I have my robots.txt entry set to:

User-agent: Twitterbot
Disallow: *

But I’m still getting Twitterbot requests.

I have tried returning status code 410 (Gone). Yet Twitterbot still comes back later making the same requests.

This Twitterbot behavior just started about a half ago after a break of several weeks.

I’m looking for ideas about how to make this stop.




This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.