Twitterbot high web traffic

divy_niit · 2017-07-26T13:53:22.157Z

We are getting lots of request(10,000+ in a day/approx 30 request per sec) from twitter bot which is unusual traffic. Due to this reason our website performance getting down.
Wondering, why we are getting such kind of huge traffic from twitter bot.
Let us know how we can get rid of this issue?

andypiper · 2017-07-26T15:01:35.971Z

The user-agent “Twitterbot/1.0” is used by our cards crawler service. Are links from your website being shared on Twitter? if so, then Twitterbot will visit your URLs to look for card metadata so that a card can be displayed in the Tweet.

If you don’t want it to do that, you could always either block the Twitterbot user-agent, or Twitter’s IP range (which is documented on our Cards troubleshooting page). You may also want to confirm that the calls are in fact coming from Twitter’s crawler by matching user-agent and IP address range with your logs - if not, it could be from another unrelated source.

divy_niit · 2017-07-27T12:58:42.280Z

Yes, links from our website being shared on Twitter. So we can’t block the Twitterbot user-agent.
We have checked the event logs most of the calls are coming from Twitter’s crawler.
Is there any alternate solution for resolving this issue?

IrenasBooks · 2017-11-15T10:08:14.991Z

I am experiencing the same problem as the original OP. IP matches up as per your publicly listed IPs. How do I stop this. It is a strain on our resources. It is creating 404s, which is very strange, ie, going to the same non existent page every few seconds. Almost like a Neg Seo Attack. Bizarre.

We are not running twitter ads etc, so am at a loss, to comprehend, twitterbot hammering my small business wordpress site every few seconds

Is there an elegant solution apart from the obvious blunt force measure.