Twitter Validator does not find metatags, but every other web scraper does

cards

#1

Description of issue:
Twitter does not show a card for the specified URL even though all of the OG and Twitter metatags are present.

The card validator states:
INFO: Page fetched successfully
WARN: No metatags found

URL affected (must be public):


view-source:https://shha.re/thXw5

Troubleshooting steps attempted:

  1. Twitter Card metatag is present
  1. Getting the webpage as Twitterbot (using curl) still shows metatags

  2. Metatags are static – there is no manipulation client side

  3. The robots.txt file allows Twitterbot
    curl http://shha.re/robots.txt

User-agent: Twitterbot
Disallow:
Allow: *

The image hoster/domain also allows Twitterbot

curl https://pcore-customer-media.s3.amazonaws.com/robots.txt

User-agent: Twitterbot
Disallow:
Allow: *

  1. Yes, we are using a current, valid twitter:card – ‘summary_large_image’

  2. Image URL is public and absolute

The image falls within necessary dimension requirements.
2092x1396
388KB

  1. SSL is valid and page was fetched successfully.

  2. Yes, the URL is a fully qualified DNS.

Thank you in advance for your help!


#2

I am currently hitting an instant SSL error when attempting to curl that URL.


#3

Hey @andypiper,

Thanks for the quick response!

What part of the certificate is failing for you (CA? Hostname Verifier? etc…)? For me, the whole chain checks out.

  • Connected to shha.re (107.23.14.142) port 443 (#0)
  • TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  • Server certificate: shha.re
  • Server certificate: Amazon
  • Server certificate: Amazon Root CA 1
  • Server certificate: Starfield Services Root Certificate Authority - G2

#4

I’m running iCurlHTTP on an iPad and if I disable the SSLv3 check it works. if that is enabled, it fails. I am not in the office right now so I cannot comment on whta the craweler might be having issues with.


#5

If it was an SSL issue, would the crawler/validator still be able to fetch the page?

The card validator states:
INFO: Page fetched successfully
WARN: No metatags found

In terms of SSL and architecture, it’s the same configuration we’re using for other services (e.g. cloudcampaign.io) that populate Twitter cards fine.


#6

There is no Content-Type header on the server response either, that would be a high likelihood of why the tags are not being pulled in.


#7

Ah, that could be it. Thanks so much, I’ll add that now!

Is this the correct content-type? “text/html; charset=UTF-8”


#8

that would work I think.


#9

@andypiper – you are a saint! Honestly, thank you so much.

I have been pulling my hair out over this all afternoon. It works now.

Thanks again!