Streaming: "Transfer-Encoding" header sometimes lower-case ("transfer-encoding: chunked")

hootphil · 2014-04-22T19:02:49.000Z

Relatively rarely, connecting to the Streaming API at https://stream.twitter.com/1.1/statuses/filter returns the “Transfer-Encoding” header in lower-case: “transfer-encoding: chunked”. My understanding is that it should always be capitalized, and while I can adjust my client, some libraries may choke on this. Ref: http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-26#section-3.3.1

Here’s a log of all headers that were sent in such a case:

connection: close
Content-Type: application/json
transfer-encoding: chunked
date: Tue, 22 Apr 2014 17:14:08 UTC
x-connection-hash: 17b9b9d305a66496fc4e703e8f3d8ee9

Thanks,
Phil

step3profit · 2014-04-23T00:06:53.000Z

We are also experiencing this.

domingopa · 2014-04-23T00:34:07.000Z

actually we got lowercase header: transfer-encoding: chunked

hootphil · 2014-04-23T00:37:42.000Z

It appears as if the lower-case version is now returned on every connect. I would have to speculate that there was a release today that introduced this behaviour.

SEHRdeveloper · 2014-04-23T00:54:56.000Z

So that would explain why the phirehose library is getting an error:

PHP Fatal error: Uncaught exception ‘Exception’ with message ‘Twitter did not send a chunking header. Is this really HTTP/1.1? Here are headers:
connection: close
Content-Type: application/json
transfer-encoding: chunked
date: Wed, 23 Apr 2014 00:52:11 UTC
x-connection-hash: e429d181bd6b89f18d185c81fe83d3a4
’ in /var/www/html/SE/140dev/libraries/phirehose/Phirehose.php:739
Stack trace:
#0 /var/www/html/SE/140dev/libraries/phirehose/Phirehose.php(830): Phirehose->connect()
#1 /var/www/html/SE/140dev/libraries/phirehose/Phirehose.php(409): Phirehose->reconnect()
#2 /var/www/html/SE/140dev/db/get_tweets.php(263): Phirehose->consume()
#3 {main}
thrown in /var/www/html/SE/140dev/libraries/phirehose/Phirehose.php on line 739

hootphil · 2014-04-23T01:19:00.000Z

Yes, I pushed a fix for this to Phirehose today and https://github.com/DarrenCook already merged into master (thanks!). See https://github.com/fennb/phirehose/pull/63 and https://groups.google.com/forum/#!topic/phirehose-users/AytivyXTxrw

step3profit · 2014-04-23T02:05:04.000Z

Thanks for that, btw.

jikiwi · 2014-04-23T06:53:10.000Z

I’ve just had exactly the same issue and this is now fixed thanks to your fix on Phirehose.

Thanks

d0ri0 · 2014-04-23T07:47:15.000Z

Thank you for the info and the fast fix.

CraigLyonsEIS · 2014-04-23T13:39:24.000Z

Thanks for the quick fix!

stunxcom · 2014-04-23T14:11:21.000Z

Thanks @hootphil we needed this quick fix for stunx.com!

felipeiasi · 2014-04-23T14:59:01.000Z

Tks tks tks!!! Thanks to your fix on Phirehose.

brianbedard · 2014-04-23T16:19:12.000Z

You’re awesome.

roelfrenkema · 2014-04-24T05:41:44.000Z

Lol my phirehose is not recognisable as phirehose anymore. I will have to adjust manually had it fixed in 2 minutes though but now I can apply a final solution, thanks devs.

LogicalArthur · 2014-04-25T23:37:59.000Z

Hey everyone. Thanks for your patience while we investigated this issue.

Take a look at http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2

In particular, note “Field names are case-insensitive.”.

We’re very happy to hear that Phirehose was patched, and we encourage others to update their clients and libraries to adhere to the standard and not hard-code a particular capitalization for header field names.

hootphil · 2014-04-25T23:57:54.000Z

Thanks for looking into this Arthur. After checking the specs in more detail, I realized that the document I previously referred to states this case-insensitivity as well (http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-26#section-3.2).

Good to get clarification on this, and at least this discussion will serve as a reference for other developers.

Have a good weekend,
Phil