Failed to Crawl URL

Maracerella · 2014-04-15T03:57:51.000Z

I added Twitter Cards meta tags to my forum site. The site is developed in PHP using vbulletin. However, whenever I run the URL in the Card Validator, it showing Failed to Crawl URL, debugId: 242571936811…

Please help out.

Thanks

Maracerella · 2014-04-15T06:16:26.000Z

Can somebody help please?

I checked and there is no robots.txt in the server.
One of the URLs is http://www.pinoyexchange.com/forums/showthread.php?t=335436

Thanks.

romainhuet · 2014-04-15T08:48:28.000Z

Icel, could you please try adding a robots.txt file with the contents below and try again?


User-agent: Twitterbot
Allow: *

Maracerella · 2014-04-15T09:30:52.000Z

I added a robots.txt file as instructed but Im still getting the same error.

Thanks for helping out.

ifkya1957gh · 2014-04-15T09:39:37.000Z

Try and edit your source in note book and then reconvert into php or html as it will automatically convert into php.

romainhuet · 2014-04-15T10:42:52.000Z

Icel, I think I found the origin of your issue. I tried to retrieve the page you mentioned via cURL using the command below:


$ curl -v -A Twitterbot http://www.pinoyexchange.com/forums/showthread.php?t=335436

However, this is returning a HTTP 500 error:


...
< HTTP/1.0 500 Internal Server Error
...

Actually, even in a browser, the page http://www.pinoyexchange.com/forums/showthread.php?t=335436 returns HTML as expected but a 500 response code.

Maracerella · 2014-04-28T07:48:03.000Z

Hi Romain,

Our server has been fixed. However, I am still showing ‘Failed to crawl URL’ error.

What should I do? Is it because I’m using vbulletin?

romainhuet · 2014-04-28T10:50:25.000Z

Hi Icel,

I’m still seeing a 500 error on the URL above. Could you please check with cURL before retrying on the validator?

Thanks,
Romain

itsamansworlduk · 2014-05-01T10:30:31.000Z

same here with me

jbulava · 2014-05-09T19:20:10.000Z

Hi Icel, I wanted to follow up on this thread. I see two possible issues here.

(1) The robots.txt file is not in the right place or it cannot be accessed at http://www.pinoyexchange.com/robots.txt

(2) You have no value for the twitter:creator tag. It’s unfortunate, but our crawler throws an error when tags have empty values instead of ignoring them and rendering without them. I would suggest leaving out creator if you are not using it, or not include it on pages where it would be empty. I’m bringing this to engineering team as the error handling should be better.

Maracerella · 2014-05-12T02:32:21.000Z

Hi,

It just suddenly work! Just hoping that this is stable.

Thank you for helping out.

Icel