DDOS-levels of website-traffic generated byTwitterBot


#1

We are periodically experiencing high traffic to our website, generated by user-agents identifying themselves as ‘Twitterbot/1.0’. These requests are numerous enough that they effectively act as a DDOS, occasionally slow down webserver-response-time to tens of seconds (even when using multiple levels of caching). Sometimes the request-groups contain many repetitions of the same URL, sometimes they contain many different URLs.

The IP-addresses involved are
199.16.156.124
199.16.156.125
199.16.156.126
199.59.148.209
199.59.148.210
199.59.148.211

As I understand it TwitterBot fetches the page-contents in order to compose Twitter Cards, and we would like to allow that, but this is a bit much. Is there a way to get the Twitter servers to slow down their requests?

Note: the robots.txt-file contains a line
Crawl-delay: 10
but this appears to be ignored.

I will try to post a log of two minutes of Twitter requests as a reply to this message as an illustration.


#2

Extract of 2 minutes of server-log:

199.16.156.124 - - [23/Nov/2015:12:19:04 +0100] "GET /wereld/nigeriaans-voetbalteam-geweigerd-nederland HTTP/1.1" 200 18537 "-" "Twitterbot/1.0"
199.16.156.125 - - [23/Nov/2015:12:19:19 +0100] "GET /blog/dedeisme/ga-ik-me-ooit-vrij-voelen-dit-land-van-vrijheid?utm_content=buffer7205c&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 200 19496 "-" "Twitterbot/1.0"
199.16.156.126 - - [23/Nov/2015:12:19:43 +0100] "GET /robots.txt HTTP/1.1" 200 964 "-" "Twitterbot/1.0"
199.16.156.124 - - [23/Nov/2015:12:19:36 +0100] "GET /water/klimaat-toekomst/de-zweetklieren-van-de-aarde-drogen-op?utm_content=buffer8aa75&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 200 19317 "-" "Twitterbot/1.0"
199.16.156.126 - - [23/Nov/2015:12:19:50 +0100] "GET /robots.txt HTTP/1.1" 200 964 "-" "Twitterbot/1.0"
199.16.156.124 - - [23/Nov/2015:12:19:50 +0100] "GET /robots.txt HTTP/1.1" 200 964 "-" "Twitterbot/1.0"
199.16.156.126 - - [23/Nov/2015:12:19:50 +0100] "GET /robots.txt HTTP/1.1" 200 964 "-" "Twitterbot/1.0"
199.59.148.211 - - [23/Nov/2015:12:19:50 +0100] "GET /robots.txt HTTP/1.1" 200 964 "-" "Twitterbot/1.0"
199.16.156.126 - - [23/Nov/2015:12:19:43 +0100] "GET /water/klimaat-toekomst/de-zweetklieren-van-de-aarde-drogen-op?utm_content=buffer8aa75&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 200 19117 "-" "Twitterbot/1.0"
199.16.156.126 - - [23/Nov/2015:12:19:43 +0100] "GET /toekomstdenkers/mens/afrikaanse-jongeren-moeten-onrecht-persoonlijk-opvatten?utm_content=buffer7ac8e&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 200 18458 "-" "Twitterbot/1.0"
199.16.156.124 - - [23/Nov/2015:12:19:57 +0100] "GET /robots.txt HTTP/1.1" 200 964 "-" "Twitterbot/1.0"
199.16.156.124 - - [23/Nov/2015:12:19:57 +0100] "GET /robots.txt HTTP/1.1" 200 964 "-" "Twitterbot/1.0"
199.59.148.209 - - [23/Nov/2015:12:19:57 +0100] "GET /robots.txt HTTP/1.1" 200 964 "-" "Twitterbot/1.0"
199.59.148.209 - - [23/Nov/2015:12:20:06 +0100] "GET /robots.txt HTTP/1.1" 200 964 "-" "Twitterbot/1.0"
199.59.148.209 - - [23/Nov/2015:12:20:06 +0100] "GET /robots.txt HTTP/1.1" 200 964 "-" "Twitterbot/1.0"
199.59.148.211 - - [23/Nov/2015:12:20:13 +0100] "GET /robots.txt HTTP/1.1" 200 964 "-" "Twitterbot/1.0"
199.59.148.211 - - [23/Nov/2015:12:20:13 +0100] "GET /robots.txt HTTP/1.1" 200 964 "-" "Twitterbot/1.0"
199.16.156.126 - - [23/Nov/2015:12:19:50 +0100] "GET /toekomstdenkers/mens/afrikaanse-jongeren-moeten-onrecht-persoonlijk-opvatten?utm_content=buffer7ac8e&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 103 17167 "-" "Twitterbot/1.0"
199.59.148.211 - - [23/Nov/2015:12:20:13 +0100] "GET /werken/vacatures/journalistieke-stage-oneworld-research-0?vacature HTTP/1.1" 403 15818 "-" "Twitterbot/1.0"
199.16.156.126 - - [23/Nov/2015:12:19:50 +0100] "GET /werken/vacatures/journalistieke-stage-oneworld-research-0?vacature HTTP/1.1" 103 8280 "-" "Twitterbot/1.0"
199.16.156.126 - - [23/Nov/2015:12:19:50 +0100] "GET /research/eeuwige-roem-voor-jouw-masterscriptie-over-duurzame-ontwikkeling HTTP/1.1" 103 17040 "-" "Twitterbot/1.0"
199.16.156.125 - - [23/Nov/2015:12:19:50 +0100] "GET /research/wereldwijde-vraagstukken/meerderheid-nederlanders-tegen-opgelegd-quotum-asielzoekers?utm_content=bufferccb49&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 103 17172 "-" "Twitterbot/1.0"
199.16.156.126 - - [23/Nov/2015:12:19:50 +0100] "GET /research/wereldwijde-vraagstukken/zorgplicht-voor-multinationals-dan-het-hek-van-de-dam HTTP/1.1" 103 17170 "-" "Twitterbot/1.0"
199.16.156.124 - - [23/Nov/2015:12:19:50 +0100] "GET /research/wereldwijde-vraagstukken/we-mogen-weer-praten-over-slavernij HTTP/1.1" 103 17131 "-" "Twitterbot/1.0"
199.16.156.126 - - [23/Nov/2015:12:19:50 +0100] "GET /research/wereldwijde-vraagstukken/drie-stappen-malaria-uitroeien?utm_content=bufferd132e&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 103 17140 "-" "Twitterbot/1.0"
199.59.148.210 - - [23/Nov/2015:12:20:21 +0100] "GET /robots.txt HTTP/1.1" 200 964 "-" "Twitterbot/1.0"
199.59.148.210 - - [23/Nov/2015:12:20:21 +0100] "GET /robots.txt HTTP/1.1" 200 964 "-" "Twitterbot/1.0"
199.59.148.210 - - [23/Nov/2015:12:20:21 +0100] "GET /research/wereldwijde-vraagstukken/vluchteling-en-staatloos?utm_content=buffer2c57b&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 200 18158 "-" "Twitterbot/1.0"
199.59.148.211 - - [23/Nov/2015:12:20:21 +0100] "GET /research/wereldwijde-vraagstukken/de-buurttuin-vooral-een-ontmoetingsplek HTTP/1.1" 200 18681 "-" "Twitterbot/1.0"
199.59.148.209 - - [23/Nov/2015:12:20:21 +0100] "GET /robots.txt HTTP/1.1" 200 964 "-" "Twitterbot/1.0"
199.16.156.124 - - [23/Nov/2015:12:19:50 +0100] "GET /research/wereldwijde-vraagstukken/vluchteling-en-staatloos?utm_content=buffer2c57b&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 103 17135 "-" "Twitterbot/1.0"
199.59.148.210 - - [23/Nov/2015:12:20:21 +0100] "GET /research/wereldwijde-vraagstukken/zorgplicht-voor-multinationals-dan-het-hek-van-de-dam HTTP/1.1" 200 19296 "-" "Twitterbot/1.0"
199.59.148.210 - - [23/Nov/2015:12:20:21 +0100] "GET /research/wereldwijde-vraagstukken/drie-stappen-malaria-uitroeien?utm_content=bufferd132e&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 200 17313 "-" "Twitterbot/1.0"
199.59.148.210 - - [23/Nov/2015:12:20:21 +0100] "GET /research/wereldwijde-vraagstukken/meerderheid-nederlanders-tegen-opgelegd-quotum-asielzoekers?utm_content=bufferccb49&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 200 17659 "-" "Twitterbot/1.0"
199.59.148.211 - - [23/Nov/2015:12:20:21 +0100] "GET /robots.txt HTTP/1.1" 200 964 "-" "Twitterbot/1.0"
199.59.148.209 - - [23/Nov/2015:12:20:21 +0100] "GET /research/eeuwige-roem-voor-jouw-masterscriptie-over-duurzame-ontwikkeling HTTP/1.1" 200 17746 "-" "Twitterbot/1.0"
199.59.148.211 - - [23/Nov/2015:12:20:21 +0100] "GET /research/wereldwijde-vraagstukken/we-mogen-weer-praten-over-slavernij HTTP/1.1" 200 18097 "-" "Twitterbot/1.0"
199.59.148.211 - - [23/Nov/2015:12:19:51 +0100] "GET /research/wereldwijde-vraagstukken/wantrouwen-afghaanse-migranten-staat-hulp-thuisland-de-weg HTTP/1.1" 103 17177 "-" "Twitterbot/1.0"
199.16.156.126 - - [23/Nov/2015:12:19:50 +0100] "GET /research/wereldwijde-vraagstukken/de-buurttuin-vooral-een-ontmoetingsplek HTTP/1.1" 103 17149 "-" "Twitterbot/1.0"
199.16.156.124 - - [23/Nov/2015:12:19:50 +0100] "GET /research/wereldwijde-vraagstukken/de-buurttuin-vooral-een-ontmoetingsplek?utm_content=buffer46e9a&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 103 17149 "-" "Twitterbot/1.0"
199.16.156.125 - - [23/Nov/2015:12:20:26 +0100] "GET /robots.txt HTTP/1.1" 200 964 "-" "Twitterbot/1.0"
199.16.156.125 - - [23/Nov/2015:12:19:57 +0100] "GET /research/wereldwijde-vraagstukken/drie-stappen-malaria-uitroeien?utm_content=bufferd132e&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 103 17140 "-" "Twitterbot/1.0"
199.16.156.126 - - [23/Nov/2015:12:19:57 +0100] "GET /research/wereldwijde-vraagstukken/vluchteling-en-staatloos?utm_content=buffer2c57b&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 103 17135 "-" "Twitterbot/1.0"
199.16.156.126 - - [23/Nov/2015:12:19:57 +0100] "GET /research/wereldwijde-vraagstukken/zorgplicht-voor-multinationals-dan-het-hek-van-de-dam HTTP/1.1" 103 17170 "-" "Twitterbot/1.0"
199.16.156.126 - - [23/Nov/2015:12:19:57 +0100] "GET /research/wereldwijde-vraagstukken/we-mogen-weer-praten-over-slavernij HTTP/1.1" 103 17131 "-" "Twitterbot/1.0"
199.16.156.124 - - [23/Nov/2015:12:19:57 +0100] "GET /research/wereldwijde-vraagstukken/de-buurttuin-vooral-een-ontmoetingsplek HTTP/1.1" 103 17149 "-" "Twitterbot/1.0"
199.16.156.124 - - [23/Nov/2015:12:19:57 +0100] "GET /research/wereldwijde-vraagstukken/de-buurttuin-vooral-een-ontmoetingsplek?utm_content=buffer46e9a&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 103 17149 "-" "Twitterbot/1.0"
199.16.156.124 - - [23/Nov/2015:12:19:57 +0100] "GET /research/wereldwijde-vraagstukken/meerderheid-nederlanders-tegen-opgelegd-quotum-asielzoekers?utm_content=bufferccb49&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 103 17172 "-" "Twitterbot/1.0"
199.16.156.125 - - [23/Nov/2015:12:19:57 +0100] "GET /werken/vacatures/journalistieke-stage-oneworld-research-0?vacature HTTP/1.1" 103 8280 "-" "Twitterbot/1.0"
199.59.148.209 - - [23/Nov/2015:12:19:58 +0100] "GET /research/wereldwijde-vraagstukken/wantrouwen-afghaanse-migranten-staat-hulp-thuisland-de-weg HTTP/1.1" 103 17177 "-" "Twitterbot/1.0"
199.16.156.124 - - [23/Nov/2015:12:19:57 +0100] "GET /research/eeuwige-roem-voor-jouw-masterscriptie-over-duurzame-ontwikkeling HTTP/1.1" 103 17040 "-" "Twitterbot/1.0"
199.59.148.209 - - [23/Nov/2015:12:20:06 +0100] "GET /research/wereldwijde-vraagstukken/de-buurttuin-vooral-een-ontmoetingsplek?utm_content=buffer46e9a&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 103 17149 "-" "Twitterbot/1.0"
199.59.148.209 - - [23/Nov/2015:12:20:06 +0100] "GET /werken/vacatures/journalistieke-stage-oneworld-research-0?vacature HTTP/1.1" 103 8280 "-" "Twitterbot/1.0"
199.16.156.125 - - [23/Nov/2015:12:20:03 +0100] "GET /groen/energie/amsterdamse-verwarming-draait-op-afval-uit-engeland HTTP/1.1" 103 17126 "-" "Twitterbot/1.0"
199.16.156.126 - - [23/Nov/2015:12:20:10 +0100] "GET /groen/energie/amsterdamse-verwarming-draait-op-afval-uit-engeland HTTP/1.1" 103 17145 "-" "Twitterbot/1.0"
199.59.148.211 - - [23/Nov/2015:12:20:13 +0100] "GET /research/wereldwijde-vraagstukken/de-buurttuin-vooral-een-ontmoetingsplek?utm_content=buffer46e9a&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 103 17168 "-" "Twitterbot/1.0"
199.16.156.125 - - [23/Nov/2015:12:20:15 +0100] "GET /food/wereldvoedseldag/pluk-je-wild HTTP/1.1" 103 17110 "-" "Twitterbot/1.0"
199.16.156.125 - - [23/Nov/2015:12:20:26 +0100] "GET /water/klimaat-toekomst/de-waarde-van-water-land-en-de-bevolking?utm_content=buffer51ff3&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 103 17192 "-" "Twitterbot/1.0"
199.16.156.124 - - [23/Nov/2015:12:20:33 +0100] "GET /water/klimaat-toekomst/de-waarde-van-water-land-en-de-bevolking?utm_content=buffer51ff3&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer HTTP/1.1" 200 19281 "-" "Twitterbot/1.0"
199.16.156.125 - - [23/Nov/2015:12:20:22 +0100] "GET /food/wereldvoedseldag/pluk-je-wild HTTP/1.1" 103 17110 "-" "Twitterbot/1.0"

#5

We’re experiencing the exact same issue here. Is there any more information on this?


#6

We have had no responses from anybody regarding this issue.
We’ve had to block all TwitterBot requests to prevent it causing problems with the site for normal visitors.

I noticed a similar issue posted here.


Twitterbot bring down down site