SiteStreams bad request

covati · 2014-01-08T21:30:57.000Z

It’s odd, we’re getting 404s…

I’m wondering if there was a significant code change that caught a couple of us. I just checked and we were actually using an old endpoint “/2b/site.json” which I updated to “/1.1/site.json”, but that didn’t help at all.

episod · 2014-01-08T21:33:15.000Z

I’m asking engineering to look into this; hopefully it’s a temporary error condition. Stay tuned.

richardhyland · 2014-01-08T21:34:11.000Z

Yeah I’m wondering if there was a code deployment that is much stricter. Are you getting 404 on /1.1/site.json?

I’ve tried on two different servers, same issue.

richardhyland · 2014-01-08T21:35:23.000Z

I can email you the full headers including oauth if it’s any help or needed

episod · 2014-01-08T21:45:04.000Z

Please do – including the screen name you’re using to connect. episod at twitter dot com – if I can’t help look into it, I’ll forward it to someone who can.

episod · 2014-01-08T21:49:47.000Z

When you’re getting a 404 – can you verify that all the users you’re specifying to connect with using the follow parameter are both existent users and users you have valid access tokens for?

covati · 2014-01-08T21:51:00.000Z

Yea, I had hoped that maybe our use of the old endpoint was the problem. But our switch to to “https://sitestream.twitter.com/1.1/site.json” still results in immediate disconnects with a 404 response.

covati · 2014-01-08T22:01:10.000Z

I’ll take a look - unless recently deleted, they are existent.

However, I don’t think the api provides a great way to let us know which users may have revoked their access. Unless I’m missing something.

We do remove them when we discover they are bad, but proactively searching them out is problematic. We can probably come up with a test script to weed out any bad tokens if that is now required.

episod · 2014-01-08T22:03:53.000Z

Though it might be unrelated, I really recommend having a regular process that goes through the access tokens you have and runs account/verify_credentials with them; it’s probably the best way to check in on current status of each of your tokens.

covati · 2014-01-08T22:09:11.000Z

Ok, that makes sense - thanks for the tip.

LogicalArthur · 2014-01-08T22:22:59.000Z

When you connect to sitestream.twitter.com, are you also sending that as the host header?

LogicalArthur · 2014-01-08T22:57:40.000Z

Can you please email me the full request and response (headers and body) that’s resulting in these 400s, please? My first name at twitter.com is my email address.

Something off a raw cURL would be best.

LogicalArthur · 2014-01-08T23:16:49.000Z

Something else that might help us to uniquely identify your connection attempts in our systems is for you to use a unique User Agent (just make something up and let me know).

richardhyland · 2014-01-08T23:17:25.000Z

I emailed the information to Taylor. It appears to all be working fine now, do you still need the information to your email address?

LogicalArthur · 2014-01-09T21:58:00.000Z

After quite a bit of in-depth investigation by our engineering teams, it looks like the vast majority of issues with 4xx errors were the result of client-side problems.

As you may remember, we [node:17585, title=“announced deprecation of HTTP 1.0”] back in May 2013 and have posted several reminders since then. A recent backend update more strictly enforces the HTTP 1.1 standard (http://www.w3.org/Protocols/rfc2616/rfc2616.html), which is partially responsible for some of the 4xx problems posted about here.

For those developers using Phirehose (https://github.com/fennb/phirehose), we strongly recommend updating to the lastest version, which is reported to be HTTP 1.1 compliant and fully compatible with Site Streams. (For those not married to PHP, you may consider HBC (https://github.com/twitter/hbc), an open-source client that Twitter maintains.)

covati · 2014-01-13T18:00:26.000Z

We are getting 404s again, starting on Friday, it appears.

It looks like we are putting sitestream.twitter.com into the header, but connecting to the IP, do you think that is causing the problem? The 404 error isn’t very helpful in determining the problem - it’s not explicitly an auth error, and we haven’t made much auth changes recently. We did make some changes to way we connected back in Nov to support the deprecation of HTTP 1.0.

EDIT: We are actually using the IP everywhere, I was wrong. We are making a GET rather than a POST though. I’m not sure if that is an issue…

I just did a check and at least in my test case it’s not because we don’t have access to the user. Any thoughts?

Thanks,
Adam

LogicalArthur · 2014-01-13T17:59:45.000Z

I would recommend always connecting to sitestream.twitter.com and explicitly hard-coding an IP address. We make no guarantees about how long any particular IP address will be used for any particular purpose, and we often use DNS to more effectively load-balance traffic.

covati · 2014-01-13T18:41:38.000Z

Sorry I wasn’t more clear. Previously the request was for us to resolve sitestream.twitter.com to an IP and to use that. So we resolve the IP and use that IP - we do this dynamically using gethostbynamel() in php.

I’ll go ahead and try pulling that out to see if connecting by hostname helps.

BTW, is it best to continue this conversation here or via email?

covati · 2014-01-13T22:29:55.000Z

It looks like the issue was related to us using gethostbynamel, when just using the sitestream.twitter.com host the connection worked fine again.

We’re assuming this is due to internal routing/server changes made with relation to the other recent api changes.

LogicalArthur · 2014-01-15T02:30:31.000Z

Great! Glad that this issue was resolved for you!