Twitterbot/1.0

website-card
cards

#1

We are seeing that there are many requests from twitterbot in a short span of time and its bringing our production site slow. Please let us know if you have any recommendation on handling this situation better ?. Please find the below sample extract from web server logs.

msstate.bncollege.com 10.55.52.10 [16/Jun/2016:15:01:39 -0400] “GET /webapp/wcs/stores/servlet/CollectionsView?categoryId=40857&catalogId=10001&langId=-1&storeId=10058&topCatId=40510 HTTP/1.1” 200 203406 “-” “Twitterbot/1.0” 339069289 - - - “-/- (-%)” “199.16.156.124”

Some of the other urls which are requested by twitterbot are given below.

http://loyola.bncollege.com/webapp/wcs/stores/servlet/CollectionsView?categoryId=40857&catalogId=10001&langId=-1&storeId=65053&topCatId=40510
http://ubalt.bncollege.com/webapp/wcs/stores/servlet/CollectionsView?categoryId=40857&catalogId=10001&langId=-1&storeId=46551&topCatId=40510
http://tulane.bncollege.com/webapp/wcs/stores/servlet/CollectionsView?categoryId=40857&catalogId=10001&langId=-1&storeId=13559&topCatId=40510
http://slu.bncollege.com/webapp/wcs/stores/servlet/CollectionsView?categoryId=40857&catalogId=10001&langId=-1&storeId=19068&topCatId=40510
http://yale.bncollege.com/webapp/wcs/stores/servlet/CollectionsView?categoryId=40857&catalogId=10001&langId=-1&storeId=16556&topCatId=40510


#2

Hello, Twitterbot/1.0 is constantly hitting up my site, causing errors. The site is www.conferencecalltranscripts.org/Q/summary/?id=3242097

Can you please stop the out-of-control Bot please?

Thanks!