Failed to Crawl URL


#1

I added Twitter Cards meta tags to my forum site. The site is developed in PHP using vbulletin. However, whenever I run the URL in the Card Validator, it showing Failed to Crawl URL, debugId: 242571936811…

Please help out.

Thanks


#2

Can somebody help please?

I checked and there is no robots.txt in the server.
One of the URLs is http://www.pinoyexchange.com/forums/showthread.php?t=335436

Thanks.


#3

Icel, could you please try adding a robots.txt file with the contents below and try again?

User-agent: Twitterbot Allow: *

#4

I added a robots.txt file as instructed but Im still getting the same error.

Thanks for helping out.


#5

Try and edit your source in note book and then reconvert into php or html as it will automatically convert into php.


#6

Icel, I think I found the origin of your issue. I tried to retrieve the page you mentioned via cURL using the command below:

$ curl -v -A Twitterbot http://www.pinoyexchange.com/forums/showthread.php?t=335436

However, this is returning a HTTP 500 error:

... < HTTP/1.0 500 Internal Server Error ...

Actually, even in a browser, the page http://www.pinoyexchange.com/forums/showthread.php?t=335436 returns HTML as expected but a 500 response code.


#7

Hi Romain,

Our server has been fixed. However, I am still showing ‘Failed to crawl URL’ error.

What should I do? Is it because I’m using vbulletin?


#8

Hi Icel,

I’m still seeing a 500 error on the URL above. Could you please check with cURL before retrying on the validator?

Thanks,
Romain


#9

same here with me


#10

Hi Icel, I wanted to follow up on this thread. I see two possible issues here.

(1) The robots.txt file is not in the right place or it cannot be accessed at http://www.pinoyexchange.com/robots.txt

(2) You have no value for the twitter:creator tag. It’s unfortunate, but our crawler throws an error when tags have empty values instead of ignoring them and rendering without them. I would suggest leaving out creator if you are not using it, or not include it on pages where it would be empty. I’m bringing this to engineering team as the error handling should be better.


#11

Hi,

It just suddenly work! Just hoping that this is stable.

Thank you for helping out.

Icel