Twitter caching your robots.txt file?

creativeskills · 2012-11-17T22:54:57.000Z

It seems like Twitter caches robots.txt settings, or something like that. Whatever I do, previewing my cards always shows the following error:

“Pre-fetching image http://creativeskills.be/cache/flickrjobs/visual-designer-xaop-13-11-12_25.png failed because it’s denied by robots.txt”

While my robots.txt file is exactly this:

User-agent: Twitterbot
Disallow:/cache/
Allow: /cache/images/

User-agent: *
Disallow:/cache/
Allow: /cache/images/

You can check the robots.txt file here: http://creativeskills.be/robots.txt

Even when I empty the robots.txt file I get this error. But the weird thing is, sometimes (let’s say in 5% of the cases) the preview successfully shows the card, but when trying again (with the same URL and without making any changes to robots.xt) I get the above error.

So it looks like the Twitter Preview tool is unreliable or just moody.

zack2zack · 2012-11-26T09:45:00.000Z

Facing the same issue. Verified robots.txt using http://tools.seobook.com/robots-txt/analyzer/

Multiple robot rules found
Robots allowed: Twitterbot
Robots disallowed: All robots

froginthevalley · 2012-11-27T19:13:54.000Z

We cache files (including robots.txt and images) for 24 hours. Try again after that delay, retweeting a link to another page with cards markup (or the same page using a link shortener) is a good trigger for recrawl.

gpstracklog · 2013-01-05T15:46:35.000Z

I’m having the same issue.

lauzis · 2013-03-08T14:23:00.000Z

allowed everything for twitter bot… for rest disaloved what was neccassary to dissalow and that worked just fine… thanks
User-agent: Twitterbot
Disallow:
User-Agent: *
Disallow: /something/
Disallow: /something-else/

cimsek_ · 2013-05-04T09:13:10.000Z

absurd situation for test tool

ssev · 2013-05-04T09:14:43.000Z

24 hours cache for test tools?
so weird!

ssev · 2013-05-04T09:15:23.000Z

24 hours cache for test tools?
so weird!

guptasamiksha08 · 2013-06-18T06:07:08.000Z

Hi,
I am also getting
Denied by robots.txt when using validator tool. Why this error is coming? And how to fix it please guide.

theSeanCook · 2013-06-18T17:44:58.000Z

At the root of your site, you have a robots.txt file that is explicitly preventing our crawler from accessing your content. Review the instructions for robots.txt at https://dev.twitter.com/docs/cards/getting-started to resolve the issue and try to re-submit.

-Sean

KCNAWatch · 2013-07-02T12:08:48.000Z

This is totally unacceptable for a validator tool. All you need to add is a way to flush the cache. Epic fail for twitter

oivlas1 · 2013-07-09T08:24:33.000Z

Hi,
I’m trying to validate a link already validated successfully in the past. Now I get the error Denied by robots.txt. There is no robots.txt on the site. How can I do to solve?

tattygram · 2013-07-19T21:52:47.000Z

epic fail is right.

eduardomartines · 2013-08-08T02:10:19.000Z

So weird! Would be interesting to know why twitter needs cache it for this dev tool…#fail

Agent47DarkSoul · 2013-08-19T09:10:09.000Z

What kind of validation tool caches for 24 hrs. One mistake and your whole day is wasted?

JCutrell · 2013-09-04T17:33:04.000Z

I agree with the negative sentiment here. When I develop Wordpress sites, I disallow search engines to index them until I am done. This creates the robots.txt response that is rejecting the card validator, and now I can’t do anything but wait for 24 hours?

Putting an entire task list on halt because of these little annoyances (like not being able to control the cache of my own robots.txt file) makes even light Twitter integrations very frustrating to deal with.

Ealtbay · 2013-11-23T07:54:02.000Z

Along with the user agent, what http request does the twitter bot use - I have everything but a “get” blocked, do I need to open another up?

EricJHoonLee · 2014-10-11T22:30:54.115Z

Has Twitter fixed this yet? I tried mine Thursday evening, so it’s been more than 24 hours now but still says deniedbyrobots.txt. Any updates on this???