Webhooks invalidated by CRC - best practice

[JiriUlip]

Hey,
I have a question regarding the regular CRC of webhooks. More often than not my webhook gets the “valid” attribute set to false. As recommended, a simple PUT to the webhook endpoint fixes that as it forces a CRC refresh.

What I’d very much interested in is if there’s a way of finding out why the webhook got “valid” set to false. Also, what’s the best practice of ensuring that my webhook is up and running? I can imagine a web service checking the validity of the webhook, say, every 5 minutes, and enforce the new CRC check if its “valid” is set to false. I’m wondering if it’s the right thing to do though as it kinda goes against the idea of webhooks.

Cheers,

Jiri

[Connexinet]

Hi Jiri,
As far as I understand, the CRC check happens once per hour, and your service needs to respond in less than 3 seconds.
The PUT basically triggers this manually.

A CRC will be sent when you register your webhook URL, so implementing your CRC response code is a fundamental first step. Once your webhook is established, Twitter will trigger a CRC on an hourly basis

https://developer.twitter.com/en/docs/accounts-and-users/subscribe-account-activity/guides/securing-webhooks.html

I don’t know how many CRC failures disable the webhook (it sounds like if it fails once, game over)

In the event that the response is not posted within 3 seconds or becomes invalid, events will cease to be sent to the registered webhook.

[JiriUlip]

Yup, our understanding of the problem is the same. My issue is that with a check every hour and one failed reply equaling game over it’s very likely that the webhook will get disabled sooner or later. That’s why I’m wondering if we could get more info on why the CRC failed (perhaps the status code it got from the service). The bottom line is that I need to be sure the webhook is up and running (CRC PUT would basically restart it if my another web service finds out it’s out). Not sure this is the most efficient way of handling that, hence the post :).

[LeBraat]

Hey @JiriUlip,

It is a best practice to build logic into your system to recognize when we deliver an error message during one of the hourly CRC, and to automatically trigger a manual CRC to revalidate the webhook. If you are using the paid version of the API, then you will be able to take advantage of our retry logic.

A single CRC failure will disable the webhook. This is in place to both protect you and Twitter from malicious actors attempting to inject themselves into the system. We want to be sure that we are sending the data to the proper webhook URL, and you want to be sure that you are receiving data from Twitter and not someone else.

We do have a couple of different error messages that deliver according to the specific scenario. Both of which are Code 214:

High latency on CRC GET request. Your webhook should respond in less than 3 seconds.

Non-200 response code during CRC GET request (i.e. 404, 500, etc).

Please let me know if you have any other questions!

[JiriUlip]

Thanks! That’s a great approach, much more elegant than building a service pinging the webhook! Should have thought of that :).

[JiriUlip]

Hey @LeBraat,

Thanks again for the suggestion. When implementing it, I realized I’m missing an important piece of info regarding “…to recognize when we deliver an error message during one of the hourly CRC…”. How exactly do you deliver that message? I know that when I use PUT to do the refresh then the message is in the response. However if you initiate the CRC via GET at my endpoint then how do I get to know you were satisfied with my response to your GET?

Just on the side, I actually wanted to test it out myself and so I was watching for all requests sent at my endpoint in hope to see the hourly CRC, but over the past ~4 hours there has been no hourly CRC check from you (the exception was when I initiated a forced refresh via PUT, then you sent the GET request as expected).

Cheers,

Jiri

[LeBraat]

Hey @JiriUlip,

How exactly do you deliver that message?

We discuss this in our documentation:
In order to verify that you are both the owner of the app and the webhook URL, Twitter will perform a Challenge-Response Check (CRC), which is not to be confused with a cyclic redundancy check. When a CRC is sent, Twitter will make a GET request of your web app with a ; crc_token parameter. When that request is received, your web app needs to build an encrypted response_token based on the crc_token parameter and your app’s Consumer Secret (details below). The response_token must be encoded in JSON (see example below) and returned within three seconds. When successful, a webhook id will be returned.

[JiriUlip]

Hey @LeBraat,
Yes, I am aware of that piece of documentation and I actually have that implemented. It doesn’t however solve my problem :(. The bit “When successful, a webhook id will be returned.” applies to me creating a new trigger via POST and that works fine. I think that the logic is as follows:

1. I run the initial webhook registration via POST request at your endpoint
2. Your reaction to my POST is a GET request at my endpoint.
3. You process my GET response
4. You generate the response to my original POST which is either webhook id or an error.

Now, this process works great. But what I’m struggling with is that sometimes your hourly CRC makes my existing trigger invalid. Since you suggested above “…to recognize when we deliver an error message during one of the hourly CRC…” I suppose there needs to be a way for me to detect such situation. Since your hourly check is just a GET at my endpoint (to which I respond with the encrypted response) I don’t see how I’m supposed to get info on whether you were happy with the GET result or not (unlike the flow with POST I described above). So just to be sure, did I understand it correctly that it’s supposed to be possible to somehow receive the result of the hourly CRC check you initiate via GET?

[KyleW]

Hey @JiriUlip - These are great questions. I want to clarify a few things.

Our docs are inaccurate (we are working to update those), and we actually only kick off a CRC once every ~24 hours (~24 hours from the last time it was validated) - it is not done hourly. Once we initiate a CRC, your server has 3 seconds to respond to the first request before we timeout. If it doesn’t respond within the first ~3 seconds, we retry the request and leave that open for ~1.75 seconds. Meaning, you have roughly ~5 seconds over 2 attempts to respond. In the event your server doesn’t respond (or returns a transient error), we will retry for ~4 hours, sending a new request every ~30 minutes. This means you have 8 attempts (with the same ~5-second retry logic) to confirm validation. Those 8 attempts happen when we receive a transient error (a 4XX, 5XX, no response, etc.)

There are a couple of things that we could encounter where we immediately mark the webhook invalid:

• Getting the wrong response back, would mark it invalid (no retry logic).
• Having an incorrect cert configured would also mark it as failed (no retry logic).
• Your server returning a non-2XX, non-4XXX, non-5XXX response code would immediately mark it as invalid
• You specifying the use of gzip, without actually sending it (or not specifying gzip, but actually sending it in the response) would also invalidate the webhook immediately.

If you are seeing something different in your logs than I mentioned, let me know. Also, if you want support understanding why this continues to happen, post your App ID and I can look into it a bit more.

Hope this help!

[JiriUlip]

Hey @KyleW,

These are perfect insights! Thanks so much for them - I think they’re pointing me in the right direction :). Before I get into heavy testing, I would like to focus on my prime suspect. What IP ranges are those 24-hour GET CRCs sent from?

The reason I’m asking is that I white-listed the ranges suggested in your Securing Webhooks article, namely 199.59.148.0/22 and 199.16.156.0/22. However when testing my service I found out (and forgot) that the CRC GET requests you send after I force CRC by PUT are coming from an IP out of the white-listed ranges, specifically 199.16.157.172 (so I white-listed the whole 199.16.157.0/255 and forgot about that). So my suspicion is that the 24-hour CRC GET may be coming yet from another IP range and since it’s not whitelisted the request gets killed and hence there’re no retries or anything. Also this would explain why all people aren’t affected as lots of them probably don’t use the IP restrictions.

Cheers,

Jiri

[KyleW]

Hey @JiriUlip - that range is accurate. I would check out a site like this one which breaks out the IP ranges accordingly.

[av3ar]

Hi,
I have doubt in creating web hook events(i.e) how to create web hook events using python.
could you please help me?

[JiriUlip]

Hey @KyleW, thx for the link! I have added all ranges in it and will let you know tomorrow if the CRC check made it through .

[JiriUlip]

Hey @av3ar,

I’m sorry, I got no Python knowledge, I did my web service in Apex (a language Salesforce uses). However https://developer.twitter.com/en/docs/accounts-and-users/subscribe-account-activity/guides/getting-started-with-webhooks seems to have examples in Python so you can probably try that.

Cheers,

Jiri

[LeBraat]

Hey @av3ar - I don’t believe we have any Python libraries for the Account Activity API at the moment. @andypiper might know of one though.

[JiriUlip]

Hey @KyleW,

I have the root cause! As suspected, the CRC GET request you send comes from an IP address which I didn’t have white-listed (199.59.150.171). The issue is that besides the IP not being in webhook documentation it also isn’t in the IP info you suggested.

I can certainly white-list an extra IP address, but I’d like to know if the IP address is something you’d be changing at some point, because it pretty much breaks my app.

Cheers,

Jiri

[KyleW]

Thanks for the reply -

The docs are correct, there are two ranges:

Twitter aggregate network blocks

For added security you may wish to whitelist the following aggregate network blocks:

• 199.59.148.0/22
• 199.16.156.0/22

I believe I only expanded one of the two ranges. If you expand the second range using a tool similiar to what I posted, you should see this IP pop up too

[JiriUlip]

@KyleW, Ah, that’s it! Thanks for extending my knowledge of how the notation of the network blocks really works, I had it completely wrong. This also explains why it wasn’t working only for me . Thanks again!

[LeBraat]