Root problem found! And it’s actually kind of hilarious. So, story…
I contracted a former co-worker, excellent social media manager, to manage the Twitter account for my app. I gave her an application account, of course. She asked for credentials for the Twitter account so she could sign up for a couple of services that require authenticating with Twitter.
One of the services she authenticated the account for is my own!
Some months ago, I changed some options for the Twitter application which, unbeknownst to me, resulted in new access tokens whenever users reauthorized the app.
So, when she authorized the account for it’s own application it got a new access token and secret which no longer matched the tokens the Streaming API daemon used.
Two seconds after she authorized the account, the Streaming API daemon failed and has been in a tight loop trying to reconnect ever since.
Lessons learned:
- store the access tokens in one place—the place they are updated on reauthorization
- alerting when an unexpected condition occurs repeatedly
- progressive backoff on reconnect
Another fun day working with the Twitter API.
