Twitter Cards Not Displaying

cassiechism · 2016-09-01T23:42:11.095Z

Hi Andy,

We have checked into our robots.txt and can confirm that everything is working as it is supposed to. Can you confirm that Twitter supports the Allow directive with your crawlers that hit robots.txt?

Thanks!
Cassie

andypiper · 2016-09-02T12:46:48.854Z

Yes, it does - see our troubleshooting page for our suggested robots.txt directive.

cassiechism · 2016-09-02T18:47:00.552Z

Hi Andy,

We have been looking over our robots.txt and below are two directives that are part of our image file paths.
User-agent: *
Allow: /sites/default/files/styles/*
Disallow: /sites/

So, given the above this URL: https://www.commonsensemedia.org/sites/default/files/styles/blog_article/public/blog/csm-blog/2016-08-10-10tvkids-blog-1138x658.jpg why would this URL not pass, since it fits into the “allow pattern”? Do you see that passing or disallowing?

cassiechism · 2016-09-07T21:05:44.953Z

Hi Andy,

Following up with this question, do you have any insight on my above question?

Thanks!
Cassie

andypiper · 2016-09-07T21:21:24.426Z

Hi Cassie, I have not had a chance to dig in with the developers on whatever change has happened, so I don’t have anything to share - so sorry about that. It’s very odd from what I can debug. Apologies, I’ll continue to see if I can get time for anyone to take a look.

cassiechism · 2016-09-08T17:18:57.052Z

Thanks Andy, we’re anxiously awaiting your reply. We have turned off all of our Twitter share icons for the time being, because we do not like the user experience of sharing a “blank” Twitter card.

abrotman · 2016-09-12T17:05:14.090Z

HI Andy, I just wanted to check if there was any update on this issue. We have all our Twitter buttons removed, but we want to turn them back on as soon as possible. We’re just waiting to resolve this image issues.

andypiper · 2016-09-12T17:12:31.757Z

What links are not working for you?

abrotman · 2016-09-12T19:20:53.544Z

Thanks Andy!

I work with Cassie (other user on this thread) so they’re the same links she has noted above.

jaydub2099 · 2016-09-13T18:23:02.376Z

I am one of the developers working on this with Amy and Cassie and the main issue with the images not showing is that we have NOT changed anything with our robots.txt prior to the issue starting to happen. Twitter previously happily read the share images and displayed with the same robots.txt as we have now. And we do not have similar issues with any other share service. Robots.txt validators online such as Googles show that our rules are fine for the images that we are trying to share.

So the issue is why does Twitter not like the robots.txt rules we have? We have not gotten an answer to that here so wondering how we can escalate this issue.

To summarize:

As we have been told that Twitter supports the Allow directive for robots.txt, our current set of directives that would be in context here should work:

User-agent: *
Allow: /sites/default/files/styles/*
Disallow: /sites/

For an example URL such as https://www.commonsensemedia.org/sites/default/files/styles/share_link_image_large/public/blog/csm-blog/2016-08-10-10tvkids-blog-1138x658.jpg

the above rules should work.

In addition last friday I also made a change to try and test the same above rules but this time specifically for Twitterbot:

User-agent: Twitterbot
Allow: /sites/default/files/styles/*
Disallow: /sites/
User-agent: *
Disallow: /

Using the Twitter dev test tool for cards, the image still does not show (using dev.commonsensemedia.org domain).

So we are still were we started with this ticket which is waiting on Twitter to tell us why the rules we have are blocking Twitter from fetching images even though by scan of the rules themselves, the images should NOT be blocked from fetching by Twitterbot?

andypiper · 2016-09-13T20:42:45.271Z

Thanks. Yes I understand your confusion and frustration, I’m right there too as I’d like to see this resolved for you. We made some changes to our fetcher a couple of weeks ago and I’m still working with the teams involved to understand some of the edge cases that seem to have caused changed behaviour. When we have news we will be sure to share it.

abrotman · 2016-09-14T01:32:43.187Z

This is Amy (works with Jeff and Cassie above). Let me know if there is anyone else we should reach out to for support as well. We’re concerned since this issue has been going on for a few weeks. We have removed the Twitter buttons on our site for now until this is resolved, and we’re unable to launch Twitter Ads until this is resolved.

andypiper · 2016-09-14T15:21:41.588Z

Thanks, I’ll continue to raise this with the cards team.

jaydub2099 · 2016-09-15T19:51:08.727Z

I wanted to see if you can pass along to your developers this question:

This appears NOT to work despite it being a standard set of robots.txt directives:

User-agent: Twitterbot
Allow: /sites/default/files/styles/*
Disallow: /sites/
User-agent: *
Disallow: /

This DOES appear to work despite it allowing Twitterbot access to everything on our site which we clearly do not want:

User-agent: Twitterbot
Disallow:
User-agent: *
Disallow: /

The above are testing robots.txt and not on our production where our production robots.txt has many more Disallow directives to tag a number of paths as disallowed. We would like to be able to keep our current Allow path followed by Disallow path as per the first entry above since it keeps the same block for all robots AND since it is what we’ve had all along and that USED to work just fine with Twitter and still works fine with every other popular share tool (Facebook, G+, Pinterest, etc).

andypiper · 2016-09-19T15:54:55.273Z

We’ve been continuing to investigate this, and our crawler attempts to follow Google’s robots.txt specification closely.

There are potentially some issues here between the order of precedence between wildcard and non-wildcard paths between group members.

The reference for this is https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt#order-of-precedence-for-group-member-records

I don’t have a resolution from our side at this stage.

cassiechism · 2016-09-19T23:13:02.873Z

Hi Andy,

Please let us know when you all have a resolution for this. We’re anxious to get our Twitter share back on our site and we would like to have our images displaying ASAP.

cassiechism · 2016-09-20T21:21:18.441Z

Hi Andy,

Just wanted to check in on this – is there anything we can do to help test out a solution? We would really prefer not to allow the Twitterbot to access everything on our site, but we need to figure out this issue ASAP.

We have a big Twitter presence and host Twitter chats often, and we’d really like to bring the Twitter share icons back to our site.

Thanks!
Cassie

andypiper · 2016-09-20T23:08:14.033Z

Hey Cassie - the team are re-checking how we handle the robots.txt rules right now. I really appreciate your frustration here! I’m doing what I can to move this along positively. I’ll share what I can when I have news

cassiechism · 2016-09-23T16:50:42.869Z

Hi Andy,

I just wanted to check in again on this issue. We’re about to launch a series of Twitter chats, ones that receive a lot of tweets and shares, and we’d really like our images to start displaying. Is there any temporary fix we can do, that does not open all of our site to Twitter via robots?

cassiechism · 2016-09-26T22:26:27.979Z

Hi Andy,

We’re thinking of removing our so we can bring our Twitter social icons back and allow our users to share from our site again. Do you have any updates? We really do not want to do this, but our social numbers dropped tremendously this month and we’re very concerned about getting this problem fixed.