Twitterbot making huge number of requests to my server

[subspacelog]

Hello,

Twitterbot is periodically killing my server. I tweet a lot (thousands in the course of a month) of unique links that resolve to my server. Periodically Twitterbot requests many old links in a short period of time. This puts a very high load on my server. I have my robots.txt entry set to:

User-agent: Twitterbot
Disallow: *

But I’m still getting Twitterbot requests.

I have tried returning status code 410 (Gone). Yet Twitterbot still comes back later making the same requests.

This Twitterbot behavior just started about a half ago after a break of several weeks.

I’m looking for ideas about how to make this stop.

Thanks,

Eric

[system]

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.