I’m getting an error from Twitter OAuth when Google (and other indexers) try to follow the link to log in via Twitter (at least I think that’s what is happening).
Several times a week, we get 10K+ errors through our homegrown OAuth Twitter login system saying that there was a 403 error because the callback url is not approved. However, when I go to the login page on our site and click the “login with Twitter button”, it works just fine. Which says to me that our callback url is properly set.
The large number of failures seems to say to me that this is a bot (like Google bot). Looking at the web server log files, I can see a large number of requests from Google bot (and a few others) that are hitting our callback page in the error window.
Anyone have any ideas of how I can get Google (and other bots) to NOT “click” the button to “Login with Twitter”? I really do want the page the button is on to be indexed. I put a nofollow on the link, but of course reading the documentation for that on Google shows that they will still “follow” the link, but not include it in our results, so…
I’m researching this in my “spare cycles”, so I don’t know when I’ll get to put a bit more instrumentation in the error message that is being sent… Here’s a snippet of what we’re getting now (with the identifying info clipped out):
ml: S, pragma: no-cache, Content-Length: 203, Cache-Control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0, Content-Type: application/xml;charset=utf-8,
Date: Thu, 05 Jul 2018 15:41:14 GMT, Expires: Tue, 31 Mar 1981 05:00:00 GMT, Last-Modified: Thu, 05 Jul 2018 15:41:14 GMT,
Set-Cookie: personalization_id=“redacted hex”; Expires=Sat, 04 Jul 2020 15:41:14 GMT; Path=/;
Domain=.twitter.com,guest_id=redacted x64; Expires=Sat, 04 Jul 2020 15:41:14 GMT; Path=/;
Domain=.twitter.com, Server: tsa_a, status: 403 Forbidden,
strict-transport-security: max-age=631138519, x-connection-hash: redacted hex,
x-content-type-options: nosniff, x-frame-options: SAMEORIGIN, x-response-time: 12,
x-transaction: redacted hex, x-twitter-response-tags]: BouncerCompliant,
x-ua-compatible: IE=edge,chrome=1, x-xss-protection: 1; mode=block;
report=https://twitter.com/i/xss_report, Error requesting email: The remote server returned an error:
(403) Forbidden., Forbidden, Forbidden, System.Net.SyncMemoryStream,
Response Stream: <?xml version="1.0" encoding="UTF-8"?>
Callback URL not approved for this client application.
Approved callback URLs can be adjusted in your application settings
TIA,
Owen
