V1 async reports missing required headers

dan_royo · 2016-04-20T21:13:28.629Z

I’m playing around with the new asynchronous routes, and I’ve noticed that all of the final report links (https://ton.twimg.com/advertiser-api-async-analytics/XXXX) return gzip’ed results as per the docs, but the response is missing the “Content-Encoding: gzip” header on the response.

Due to this header’s absence, tools which would automatically decompress the response and show the user the JSON response instead return encoded garbage. Examples of impacted tools & libraries include: Chrome Dev Tools, Postman, Httpie, and the Requests library for Python.

Per RFC 2616 (https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html), this header is mandatory, and must be included when gzip encoding is applied to a response.

Screen Shot 2016-04-20 at 5.03.17 PM.png1306x868 102 KB

Screen Shot 2016-04-20 at 5.07.09 PM.png1323x1086 87.1 KB

hwz · 2016-04-20T22:14:33.281Z

We’re in agreement that a response header indicating the the filetype should be sent, and have been working internally on a fix here.

After the change announced in https://twitter.com/AdsAPI/status/721128510773796865 is rolled out next week, the correct headers should be returned as well.

dan_royo · 2016-04-20T22:29:31.059Z

Glad to hear it! Thanks for the response!

majoritasdev · 2016-05-12T11:13:45.378Z

This has not been fixed yet. The header required is Content-Encoding: gzip, as @dan_royo said, not “content-type: application/gzip” as we receive now; and the extension “.json.gz” is also not enough. Please take a look at the headers for a cURL request to such a stats file, which contains the Accept-Encoding: gzip request header as it should, but the response headers are missing the Content-Encoding: gzip header:

* About to connect() to ton.twimg.com port 443 (#38)
*   Trying 104.244.43.71...
* Connected to ton.twimg.com (104.244.43.71) port 443 (#38)
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_128_CBC_SHA
* Server certificate:
* 	subject: CN=*.twimg.com,OU=Twitter Security,O="Twitter, Inc.",L=San Francisco,ST=California,C=US
* 	start date: Apr 22 00:00:00 2016 GMT
* 	expire date: Apr 27 12:00:00 2017 GMT
* 	common name: *.twimg.com
* 	issuer: CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
> GET /advertiser-api-async-analytics/[sensitive-information].json.gz HTTP/1.1
Host: ton.twimg.com
Accept: */*
Accept-Encoding: gzip

< HTTP/1.1 200 OK
< Date: Thu, 12 May 2016 10:41:28 GMT
< Server: Apache
< content-md5: 1Z5PbdZV7Qn7TQyMGn9cfQ==
< content-type: application/gzip
< etag: "1Z5PbdZV7Qn7TQyMGn9cfQ=="
< expires: Thu, 19 May 2016 09:00:28 GMT
< last-modified: Thu, 12 May 2016 08:55:16 GMT
< x-connection-hash: 0170bcd18baf77858fc355817d55560a
< x-response-time: 9
< x-ton-expected-size: 525
< x-ton-expires: Thu, 19 May 2016 08:55:16 GMT
< Cache-Control: max-age=31536000
< Content-Length: 525
< Accept-Ranges: bytes
< Via: 1.1 varnish
< Age: 6060
< X-Served-By: cache-tw-fra1-cr1-16-TWFRA1
< X-Cache: HIT
< X-Content-Type-Options: nosniff
< 
* Connection #38 to host ton.twimg.com left intact

@hwz: Could you please send this further to the technical team to fix? Thank you!

dan_royo · 2016-05-12T13:22:55.155Z

Yeah, apologies for not verifying this earlier, but @majoritasdev is correct. This still isn’t to spec, the Content-Encoding header is still missing.

majoritasdev · 2016-05-16T08:31:45.534Z

@ Twitter Staff: can someone please see to this? This is important to be able to work with the reports in a docs-compliant and future proof manner.

Thank you.

majoritasdev · 2016-05-19T10:49:17.290Z

Sometimes, when I want to get the analytics data from the job’s url, this is what I receive from the server:

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
  <head>
    <title>503 backend read error</title>
  </head>
  <body>
    <h1>Error 503 backend read error</h1>
    <p>backend read error</p>
    <h3>Guru Mediation:</h3>
    <p>Details: cache-tw-nyc1-cr1-1-TWNYC1 1463654226 2911472397</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>

dan_royo · 2016-05-20T14:40:39.579Z

Based on the radio silence, I’m assuming this isn’t going to get fixed.

majoritasdev · 2016-05-20T14:43:54.944Z

Well, it should be, because it’s obviously a bug!

wangdao · 2017-08-29T03:57:36.295Z

yeah,I also have the similar problem. when I get ‘.json.gz’, it return this:

{‘X-Content-Type-Options’: ‘nosniff’, ‘Content-Type’: ‘text/html’, ‘Server’: ‘ECS (dfw/F4AF)’, ‘Date’: ‘Tue, 29 Aug 2017 03:33:24 GMT’, ‘Content-Length’: ‘357’}

It lead to the following error: File “/usr/lib/python3.5/site-packages/twitter_ads/http.py”, line 73, in perform
response = self.__oauth_request()
response_obj = Response(response.status_code, response.headers, body=response.raw, raw_body=raw_response_body)
File “/usr/lib/python3.5/site-packages/twitter_ads/http.py”, line 144, in init
self._body = json.loads(raw_response_body)
File “/usr/lib64/python3.5/json/init.py”, line 312, in loads
s.class.name))
TypeError: the JSON object must be str, not ‘bytes’