Invalid content-encoding (deflate) responses

[takke]

Hi there,

For example, when we send a request to https://api.twitter.com/1.1/* we expect to receive JSON, recently we have started receiving content with the header “Content-Encoding: deflate” at random.

Here is the evidence of this problem:

intercept[https://api.twitter.com/1.1/lists/statuses.json?since_id=929183416876089344&count=101&page=1&list_id=1960479&include_entities=true&include_ext_alt_text=true&tweet_mode=extended][GET]
--req. headers:
Authorization: *** SECRET ***
X-Twitter-Client-Version: 4.0.6
X-Twitter-Client-URL: http://twitter4j.org/en/twitter4j-4.0.6.xml
X-Twitter-Client: Twitter4J
User-Agent: twitter4j http://twitter4j.org/ /4.0.6
Accept-Encoding: gzip

--res. headers:
cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
content-disposition: attachment; filename=json.json
content-encoding: deflate
content-type: application/json;charset=utf-8
date: Sat, 11 Nov 2017 03:30:22 GMT
expires: Tue, 31 Mar 1981 05:00:00 GMT
last-modified: Sat, 11 Nov 2017 03:30:22 GMT
pragma: no-cache
server: tsa_m
set-cookie: personalization_id="***=="; Expires=Mon, 11 Nov 2019 03:30:22 UTC; Path=/; Domain=.twitter.com
set-cookie: lang=ja; Path=/
set-cookie: guest_id=v1%***; Expires=Mon, 11 Nov 2019 03:30:22 UTC; Path=/; Domain=.twitter.com
status: 200 OK
strict-transport-security: max-age=631138519
x-access-level: read-write-directmessages
x-connection-hash: 30adcc0319ac0a5e63055033a424ee1c
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-rate-limit-limit: 900
x-rate-limit-remaining: 888
x-rate-limit-reset: 1510371855
x-response-time: 202
x-transaction: 00832bbb000ac3d4
x-twitter-response-tags: BouncerCompliant
x-xss-protection: 1; mode=block

We captured via interceptor of OkHttp3 on Nexus5X (Android 8.0)
Connection type: HTTP/2

And we dumped the bare response body but we cannot detect the type of binary:

deflate_sample_20171111_1513.dat

Can you help us?

This problem may be related to the following topic:

Inconsistent content encoding responses

Hi there, For example, when we send a request to https://ads-api.twitter.com/2/accounts we expect to receive JSON, recently we have started receiving content with the header “Content-Encoding: deflate” at random. Sometimes the response is JSON, sometimes it is encoded. This behavior happens throughout the ads API endpoints. Can you help us? Thanks, Holly

Thanks,

[andypiper]

Thanks for this - we are aware of the inconsistency for some requests and endpoints and are working to resolve. In the meantime, you can try forcing the use of HTTP/1.1 and use the Accept-Encoding: None header to request that the API returns uncompressed data if this is an issue for you.

[takke]

Thank you for your quick reply. We’ve relieved to hear you already be aware of this issue.

Because many Android users are using it now, so we can not take those solutions.

However, its frequency of occurrence has been low in recent weeks, so I look forward to it in the future.

Thanks,