Getting Intermittent "Fetching the page failed because the request timed out" error

[kmrsujeet]

Hi,
I am getting ‘ERROR: Fetching the page failed because the request timed out.’ intermittently for article pages of our site, http://gulfnews.com and http://m.gulfnews.com (mobile version).
When validating on card validator, some times it works while on other occasions it gives this error, even for same page.
I have checked and all the required meta tags are there on page.
Twitterbots are allowed in our robots.txt as we are using ‘User-agent: *’ for allowing bots.

Suggestion and help is appreciated.

[andypiper]

If this is intermittent it could just be an occasional performance issue with your site responding to requests from the crawler.

[kmrsujeet]

Thanks Andy,
What is the optimal range for response time for web pages for Twitter crawler?
We have caching mechanism in place and you can find response time for cached pages between 300-500 ms.

[kmrsujeet]

We are checking within seconds for same page (i.e. same site performance) but still getting this error randomly.
Please suggest.

[kmrsujeet]

@andypiper This is really impacting our site, appreciate your thoughts/ suggestions on this.

[andypiper]

Can you be specific about what the impact is on your site?

I’ve just Tweeted a link to one of your articles and the card renders fine, so I’m not clear what the issue is here.

[kmrsujeet]

thanks @andypiper for looking into this.

Impact is on sharing articles on twitter as sometimes images from article do not appear instantaneously on twitter and take some time before it appears on twitter, which is not a good user experience.
In order to check the issue when we try to preview card on twitter card validator, some times it works and some times it fails and give request timeout error.
Please note that we are using same article web page and trying to preview multiple times, repeatedly within same minute.
Even though it was cached and response time and TTFB are in range of 300-600 milliseconds.
Can you please suggest on causes and how this can be fixed.

[kmrsujeet]

hi @andypiper - Did you get a chance to look into this, appreciate your help.

[andypiper]

I’m unable to reproduce the issue and when I Tweet a link to an article on your site, a card renders just fine.

[kmrsujeet]

Thanks @andypiper - Please try same article multiple times on card validator.

[andypiper]

There’s no need to validate an article or link multiple times. I’ve just Tweeted a number of links from your site and they are rendering cards fine. Your domain is whitelisted already, you should not be needing to validate each article individually.

[kmrsujeet]

Thanks @andypiper for your time and inputs.

[ravijoshi0790]

hello i am getting same issue in my site can you help me on this.

[andypiper]

We cannot help if we do not have an example link and details of your issue. I assume you’ve already thoroughly checked the troubleshooting FAQs.

[ravijoshi0790]

hello @andypiper,

thanks for your response

i am testing this url http://inpicto.com/exclusive-the-mentalist-simon-baker-back-in-sydney-after-race-week/65492 using card validator and its giving time out error .Please help me on this

[andypiper]

It looks like your site has performance issues of some kind. I tried your link using the iframely debugger and I see exactly the same issue there, so this is very unlikely to be an issue with our cards crawler.

[ravijoshi0790]

hello ,
I check the time taken by page to load it taken only 2 sec. so i am not getting what type of performance is there can you please tell what type of issue is there and its working fine in previous month

[andypiper]

Well, unfortunately that will be dependent on geography, so if your site is hosted locally and you only have a few hops, it won’t take long. All I can tell you is that both our crawler, and another one from iframely, both timeout when trying to connect.

[ravijoshi0790]

Hello ,
Thanks @andypiper
but its working fine in my test site which one also getting data from same data base then why this happen on this site . and i also check that the when request is come on that function it take only 1 sec to response on live site(inpicto.com) .
my test site link is http://54.255.134.211/inpicto/queen-letizia-attends-meeting-of-instituto-cervantes-in-madrid-305754/62555

[ravijoshi0790]

@andypiper Please provide update on this