Denied by robots.txt


#1

The card validator appears to now obey robots.txt guidelines.This is an unhelpful change while working on a development environment to verify and validate changes. Is it possible to turn this off for card previews being accessed through the developer tools?

Screenshot of what I see: https://www.evernote.com/shard/s221/sh/c385ffed-0f60-4810-9d62-90ddda3d8e24/f62c21d1237eb125731f40420b20b26d/res/2744ab5b-bf55-4886-a0c9-2684117f0c08/skitch.png


#2

Our crawler respects robots.txt, as outlined in the “URL Crawling” section of [node:15977]. The Validator Tool validates that your content is configured correctly and that it is accessible by the crawler.


#3

What needs to be added to robots.txt to permit Twitter’s crawler?


#4

Never mind; I just realized it’s in the file you linked. Sorry!


#5

Glad it helped!


#6

what kind of help you got , please share with others , i see many post , it helped , it helped , i never saw how ? please share


#7

Did you check the troubleshooting faq? Did you check your robots.txt file?