401 and 503 errors while using Control Streams for Site Streams

[chockenberry]

Today, I’ve noticed that 401 responses are being generated from the stream control endpoints (add_user.json and info.json mainly.) I was following the requirement of 25 requests per second for add_user.json, but when I backed off the request rate (one every 300 ms, or about 3 per second) the 401 errors occurred much less frequently.

The error code description “other login rate limiting” on https://dev.twitter.com/docs/streaming-api-response-codes makes me think that there’s some new rate limiting is in place. If so, can we get an idea of what the new parameters are? Also, it’s not clear if the rate limits are per endpoint, per control_uri, or per account.

Also, in addition to the 401 errors on info.json, I’m seeing an occasional 503 error. Since “You shouldn’t see this error on the streaming API”, consider this a bug report

-ch

[kurrik]

Hi Craig, could you give me the body of the response? Any additional information about ways to reproduce would be helpful.

Thanks,
~Arne

[chockenberry]

Here’s the curl output from your OAuth tool:

% curl --get 'https://sitestream.twitter.com/1.1/site/c//info.json' --header 'Authorization: OAuth , oauth_version="1.0"' --verbose * About to connect() to sitestream.twitter.com port 443 (#0) * Trying 199.59.148.228... connected * Connected to sitestream.twitter.com (199.59.148.228) port 443 (#0) * successfully set certificate verify locations: * CAfile: /opt/local/share/curl/curl-ca-bundle.crt CApath: none * SSLv3, TLS handshake, Client hello (1): * SSLv3, TLS handshake, Server hello (2): * SSLv3, TLS handshake, CERT (11): * SSLv3, TLS handshake, Server finished (14): * SSLv3, TLS handshake, Client key exchange (16): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSL connection using RC4-SHA * Server certificate: * subject: C=US; ST=CA; L=San Francisco; O=Twitter, Inc.; OU=Twitter Security; CN=sitestream.twitter.com * start date: 2012-10-23 00:00:00 GMT * expire date: 2015-10-28 12:00:00 GMT * subjectAltName: sitestream.twitter.com matched * issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert High Assurance CA-3 * SSL certificate verify ok. > GET /1.1/site/c//info.json HTTP/1.1 > User-Agent: curl/7.19.6 (i386-apple-darwin10.4.0) libcurl/7.19.6 OpenSSL/0.9.8x zlib/1.2.5 > Host: sitestream.twitter.com > Accept: */* > Authorization: > < HTTP/1.1 401 Unauthorized < Content-Type: text/html < Transfer-Encoding: chunked < Unknown URL. See Twitter Streaming API documentation at http://dev.twitter.com/pages/streaming_api

• Connection #0 to host sitestream.twitter.com left intact
• Closing connection #0
• SSLv3, TLS alert, Client hello (1):
  

If you need to control_uri or OAuth header, please get in contact with Gmail’s craig.hockenberry account.

[chockenberry]

Hi Arne,

An update: what were periodic 401 failures have become permanent now. The stream is still running fine, but any attempts to use the control_uri with add_user or info ends up with the “Unknown URL” error above.

It appears that once the stream gets in this state, the error is persistent.

Is it advisable to force the stream to close and then reopen it?