Twitterbot is periodically killing my server. I tweet a lot (thousands in the course of a month) of unique links that resolve to my server. Periodically Twitterbot requests many old links in a short period of time. This puts a very high load on my server. I have my robots.txt entry set to:
But I’m still getting Twitterbot requests.
I have tried returning status code 410 (Gone). Yet Twitterbot still comes back later making the same requests.
This Twitterbot behavior just started about a half ago after a break of several weeks.
I’m looking for ideas about how to make this stop.