I have updated the robots.txt in our web app ( to get twitter card working. It is working for one path but not other.


User-agent: Twitterbot
Disallow: *
Allow: /records/* (Working)
Allow: /user_sets/* (Not Working)

User-agent: *
Allow: /
Disallow: /records
Allow: /records/*
Allow: /user_sets/*
Crawl-delay: 1

In card validator i am getting this message for user_sets

INFO: Page fetched successfully
INFO: 5 metatags were found
WARN: Not whitelisted


What is the exact URL you are validating?


I tried this urls in the validator.

And these are the meta we have on this page.




When I retrieve that page using curl -A Twitterbot I don’t see the meta tags. Are you injecting them via Javascript after the page loads? That will not work.


Thanks @andypiper. We will have a look at this and get back to you if needed.


This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.