SiteStreams bad request

episod · 2014-01-08T21:01:52.000Z

To which methods are you seeing this problem arise? The site stream itself or control streams for site streams?

covati · 2014-01-08T21:07:45.000Z

This is on initial connect to the sitestream.

Slightly revamped log messages:
Connecting to ssl://199.16.156.49, port=443, connectTimeout=5
Connection established for 16788, about to read
HTTP Error: HTTP/1.1 404 Not Found
Connection: close
Date: Wed, 08 Jan 2014 20:14:07 GMT
Not Found

richardhyland · 2014-01-08T21:08:06.000Z

Initial connection to sitestreams

GET /1.1/site.json?delimited=length&follow=xxxxxx HTTP/1.1
Host: sitestream.twitter.com:443

The oauth headers are correct as if I modify them to make them ‘wrong’ I get a 401 error instead.

If I add the correct headers I get HTTP ERROR 400: Bad Request (Bad Request)

dirkbonhomme · 2014-01-08T21:08:11.000Z

I can confirm these sudden issues. We can’t connect to sitestreams.


Error 401: Unauthorized, API message: Insufficient privileges

JvrBaena · 2014-01-08T21:12:45.000Z

Same here, no changes in our codebases, and, oddly enough, one of our backup server has not suffered this issue, only our main server :S

covati · 2014-01-08T21:30:57.000Z

It’s odd, we’re getting 404s…

I’m wondering if there was a significant code change that caught a couple of us. I just checked and we were actually using an old endpoint “/2b/site.json” which I updated to “/1.1/site.json”, but that didn’t help at all.

episod · 2014-01-08T21:33:15.000Z

I’m asking engineering to look into this; hopefully it’s a temporary error condition. Stay tuned.

richardhyland · 2014-01-08T21:34:11.000Z

Yeah I’m wondering if there was a code deployment that is much stricter. Are you getting 404 on /1.1/site.json?

I’ve tried on two different servers, same issue.

richardhyland · 2014-01-08T21:35:23.000Z

I can email you the full headers including oauth if it’s any help or needed

episod · 2014-01-08T21:45:04.000Z

Please do – including the screen name you’re using to connect. episod at twitter dot com – if I can’t help look into it, I’ll forward it to someone who can.

episod · 2014-01-08T21:49:47.000Z

When you’re getting a 404 – can you verify that all the users you’re specifying to connect with using the follow parameter are both existent users and users you have valid access tokens for?

covati · 2014-01-08T21:51:00.000Z

Yea, I had hoped that maybe our use of the old endpoint was the problem. But our switch to to “https://sitestream.twitter.com/1.1/site.json” still results in immediate disconnects with a 404 response.

covati · 2014-01-08T22:01:10.000Z

I’ll take a look - unless recently deleted, they are existent.

However, I don’t think the api provides a great way to let us know which users may have revoked their access. Unless I’m missing something.

We do remove them when we discover they are bad, but proactively searching them out is problematic. We can probably come up with a test script to weed out any bad tokens if that is now required.

episod · 2014-01-08T22:03:53.000Z

Though it might be unrelated, I really recommend having a regular process that goes through the access tokens you have and runs account/verify_credentials with them; it’s probably the best way to check in on current status of each of your tokens.

covati · 2014-01-08T22:09:11.000Z

Ok, that makes sense - thanks for the tip.

LogicalArthur · 2014-01-08T22:22:59.000Z

When you connect to sitestream.twitter.com, are you also sending that as the host header?

LogicalArthur · 2014-01-08T22:57:40.000Z

Can you please email me the full request and response (headers and body) that’s resulting in these 400s, please? My first name at twitter.com is my email address.

Something off a raw cURL would be best.

LogicalArthur · 2014-01-08T23:16:49.000Z

Something else that might help us to uniquely identify your connection attempts in our systems is for you to use a unique User Agent (just make something up and let me know).

richardhyland · 2014-01-08T23:17:25.000Z

I emailed the information to Taylor. It appears to all be working fine now, do you still need the information to your email address?

LogicalArthur · 2014-01-09T21:58:00.000Z

After quite a bit of in-depth investigation by our engineering teams, it looks like the vast majority of issues with 4xx errors were the result of client-side problems.

As you may remember, we [node:17585, title=“announced deprecation of HTTP 1.0”] back in May 2013 and have posted several reminders since then. A recent backend update more strictly enforces the HTTP 1.1 standard (http://www.w3.org/Protocols/rfc2616/rfc2616.html), which is partially responsible for some of the 4xx problems posted about here.

For those developers using Phirehose (https://github.com/fennb/phirehose), we strongly recommend updating to the lastest version, which is reported to be HTTP 1.1 compliant and fully compatible with Site Streams. (For those not married to PHP, you may consider HBC (https://github.com/twitter/hbc), an open-source client that Twitter maintains.)