Twitter Card Error "ERROR: Fetching the page failed because other errors."

[ralphsnart]

link:

Marc Hansen Stuff!

My web host is godaddy - they’re not blocking Twitter.

I was able to curl using Twitterbot successfully.

Robot.txt:

User-agent: Twitterbot
Disallow: /
Allow: /caches/
Allow: /scart/
Allow: /scart/product_images/

User-agent: *
Disallow: /cgi/
Disallow: /_db_backups/
Disallow: /caches/
Disallow: /comictouch/
Disallow: /counter/
Disallow: /fansite/images/
Disallow: /fansite/DLOADS/
Disallow: /fansite/games/
Disallow: /fansite/flv_player/
Disallow: /comics/images/
Disallow: /comics/backupads/
Disallow: /comics/bads/
Disallow: /comics/books/
Disallow: /comics/cover_thumbs/
Disallow: /comics/retailerads/
Disallow: /html5/
Disallow: /kfribley/
Disallow: /mmontgomery/
Disallow: /no_crawl/
Disallow: /rmiller/
Disallow: /rrdonnelly/
Disallow: /scart/css/
Disallow: /scart/images/
Disallow: /scart/js/
Disallow: /scart/product_images/
Disallow: /scart/VsKRDGVs9sNCcd/
Disallow: /stats/
Disallow: /timetracker/
Disallow: /wbfhfm/

htaccess:

#file caching:
# DEFAULT SETTINGS
Options +ExecCGI -Indexes
DirectoryIndex index.php index.html index.htm

### DEFAULTS ###
ServerSignature Off
AddType video/x-flv .flv
AddType application/x-shockwave-flash .swf
AddType image/x-icon .ico
AddDefaultCharset UTF-8
DefaultLanguage en-US
SetEnv TZ America/Detroit
SetEnv SERVER_ADMIN marchansenstuff@gmail.com

<IfModule mod_deflate.c>
	<FilesMatch "\.(html|htm|css|js)$">
		SetOutputFilter DEFLATE
	</FilesMatch>
</IfModule>

Header unset ETag
FileETag None
<FilesMatch "(?i)^.*\.(html|htm|css|js|ico|pdf|flv|jpg|jpeg|png|gif|swf)$">
	#Header unset Last-Modified
	Header set Expires "Tue, 12 Jul 2011 10:00:00 GMT"
	Header set Cache-Control "public, no-transform"
</FilesMatch>

# DONT CACHE
<FilesMatch "\.(php)$">
   Header unset Cache-Control
</FilesMatch>

<FilesMatch "\.(js|css|xml|gz)$">
    Header append Vary Accept-Encoding
</FilesMatch>

#prevent hotlinking:
#don't forget the blog!
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} .*jpg$|.*gif$|.*png$|.*jpeg$ [NC]
RewriteCond %{HTTP_REFERER} !^$ 
RewriteCond %{HTTP_REFERER} !marchansenstuff\.com [NC]
RewriteCond %{HTTP_REFERER} !ralphsnart.blogspot\.com [NC]
RewriteCond %{HTTP_REFERER} !twitter\.com [NC]
RewriteRule (.*) /http://www.marchansenstuff.com/cache/hotlink.png

#https
#RewriteEngine On
#RewriteCond %{SERVER_PORT} !=443
#RewriteCond %{HTTP_HOST} ^(www\.)?marchansenstuff\.com$ [NC]
#RewriteRule ^$ https://www.marchansenstuff.com/%{REQUEST_URI} [R,L]

#301 non-www to www redirect
#RewriteEngine On
#RewriteCond %{HTTP_HOST} ^marchansenstuff.com [NC]
#RewriteRule ^(.*)$ https://www.marchansenstuff.com/$1 [L,R=301]

#redirect http to https
RewriteEngine On
RewriteCond %{HTTPS} !^on$
RewriteRule (.*) https://www.marchansenstuff.com/$1 [R,L]

ErrorDocument 404 http://www.marchansenstuff.com/notfound.html

[andypiper]

It looks like a problem on our side with our fetcher currently unable to access your site. Sorry about that - hopefully that will resolve itself in a little while.

[ralphsnart]

Don’t mean to be a pain, but what’s a little while? Still not working after 3 days, or are we talking weeks, months to resolve. Thanks -

[andypiper]

Hopefully only about a week or so as the cards crawler usually has a 7 day cache, but on rare occasions it can take a little bit longer.

[BillTCCC]

I am having the same problem on one of our sites: https://intofarms.org
I have been setting Twitter Cards up on all our sites and have been successful on every one except this one. I can’t find what is causing this. The site is identical in terms of theme and plugins to our main site: https://intorog.org which works perfectly. Both are on GoDaddy hosts (but different servers). Have been trying for about a week, but that might be the problem because I did find some insecure link references on the intofarms homepage which have now fixed. I guess we wait a week and see?

[andypiper]

I’ve looked into this a bit more and in both cases (intofarms.org and marchansenstuff.com) I’m seeing an SSL Handshake error with an “unrecognised name at the remote address”, so I think there could be a configuration issue - I can’t tell what that is, though, I’m not familiar with GoDaddy setup. Additionally, I assume there’s a typo in https://intorog.org/ as that doesn’t resolve for me.

[BillTCCC]

Hi Andy. Thanks the SSL Handshake is a very helpful clue. I will check into it and then test again.
I will test another SSL site on the same server and see if it has the same problem.
You are correct. It was a typo the working example is https://intoorg.org

[BillTCCC]

Hi Andy. Worked with GoDaddy to fix this and made several changes to the https://intofarms.org website. But that did not seem to fix the problem. Would you be able to try it once more and see if it is giving the same handshake error? Thanks

[andypiper]

Yes, same thing as far as i can tell, unless something is cached on our side.

[ralphsnart]

Andy, Here’s a link to a SSL server test of marchansenstuff.com. Don’t know if this helps or not. Also (server variables at bottom).
Does Twitterbot have SNI support? Or are godaddy’s servers misconfigured?

[ralphsnart]

Bill, what changes did you make?

[BillTCCC]

Hi. GoDaddy suggested that we add a plugin "really simple SSL’. I did that but as far as I can tell it did not make any difference. Just checked Twitter Card Validation on both of the problem sites and still same error. I had been waiting to see if it was a Twitter cach issue but suspect it is something to do with GoDaddy. The GoDaddy server where the two effected sites we have live is a Linix server. The SSL is: Encryption Strength - GoDaddy SHA-2
What is the host type and Encryption of yours?

[ralphsnart]

I have Godaddy’s Standard SSL 256-bit encryption on Linux. I get a worse grade than your site though when I test at https://www.ssllabs.com/ssltest/index.html ( you get an A and I get a C grade).

Looks like because my server uses RC4 and doesn’t have Forward Secrecy. Yours doesn’t support RC4 and has Forward Secrecy. This would just affect clients though. Depends on what Twitterbot demands perhaps. I’m not sure.

Do you have the Premium SSL plan? Don’t know if it matters since so far neither work.

Looks like All Really Simple SSL does is to redirect http to https, which can already be done in your htacess file.

[ralphsnart]

I can connect using cURL (however this is using tsl1).

curl -A “Twitterbot” -v -1 https://www.marchansenstuff.com

When I use openssl to connect, it works using tsl1 but I get a handshake failure when using ssl2 and ssl3. Note, when using tsl1, it’s using the less secure RC4.

openssl s_client -connect marchansenstuff.com:443 -tls1

[BillTCCC]

Wow. Thanks Ralph, good detecting. Have you talked to GoDaddy about this? I am thinking that we need Andy’s help on Monday to figure out what is going on from the Twitter perspective.
You are absolutely right the Really Simple SSL plugin, while useful, did not do anything I had not already done manually when we set the site up about a year ago.
I did use it on a new site and was pleased with it.

[andypiper]

Currently I’m not sure what else I can add beyond what I’ve mentioned, which is what the internal log reports from the crawler.

[ralphsnart]

Bill,
This worked for me. I added this to my .htaccess file. It redirects http to https except for Twitterbot. The card URL should use http instead of https.

RewriteEngine On
RewriteCond %{HTTPS} !^on$
RewriteCond %{HTTP_USER_AGENT} !Twitterbot [NC]
RewriteRule (.*) https://www.yourdomain.com/$1 [R,L]

[BillTCCC]

Thanks Ralph. Glad you got yours working. I will try this on our site.

[HaythamElMogazy]

Im having the same problem, and the log only show minimal info “ERROR: Fetching the page failed because other errors.”. I tried @ralphsnart .htaccess solution, but no success. The Url: https://riskinnovation.asu.edu/ @andypiper can you take a look at the log and give me some more info?

Thanks
Haytham

[ralphsnart]

Make sure you’re not blocking Twitterbot. Add this to the end of robot.txt…

User-agent: Twitterbot
Disallow:

Also, your validating url and image meta url needs to be http and not https.