In async response sometime I am getting invalid gzip file

[idreaminfinite]

My script run to get stats data for campaigns

When I do multiple request for getting 20 campaign data for week in one request
sometime I am getting

IOError: Not a gzipped file

I am checking job status and when its success… then only I am downloading it…

here my request are quite often (1 in 15 seconds )

sometime I observe sometime its not …is it depend on traffic at twitter end…?

What might be reason behind this???

[andypiper]

(moving to the Ads API category)

[JBabichJapan]

It is likely that the request to download the file from our blob server is throwing an error, please check the HTTP status code of the actual file download and add a retry logic there (I have heard that it will fail occasionally and usually works upon a retry, we are tracking the instability with a ticket internally).

[ChrisNiebel]

I am always getting an invalid gzip when using GET /1/stats/jobs/accounts/:account_id. I am checking the status code and it is SUCCESS. It appears that the first few bytes and the last few bytes are correct but the file cannot be opened using any zip tool. This is my .NET code:

        var request = (HttpWebRequest)WebRequest.Create(analyticsResults.url);
        var response = request.GetResponse();

        var responseStream = new System.IO.StreamReader(response.GetResponseStream());
        var responseString = responseStream.ReadToEnd();

        var fileGuid = Guid.NewGuid().ToString("N");
        var entityName = analyticsResults.entity == "LINE_ITEM" ? "LineItem" : "PromotedTweet";
        var fileName = $"{dropOffLocation}\\Twitter{entityName}Metrics_{fileGuid}.json.gz";

        using (var stream = new StreamWriter(fileName))
        {
            stream.Write(responseString);
            stream.Flush();
        }

[brooklynmike]

Hello Chris,

I’m happy to help, but first I’d like to get some additional information. Below are the first two steps of the async analytics flow.

1- Create the job using the POST /1/stats/jobs/accounts/:account_id endpoint

$ twurl -X POST -H ads-api.twitter.com "/1/stats/jobs/accounts/18ce53zb651?entity=CAMPAIGN&entity_ids=86onf&granularity=TOTAL&metric_groups=ENGAGEMENT&placement=ALL_ON_TWITTER&start_time=2017-04-24&segmentation_type=DEVICES&platform=0" | jq
{
  "request": {
    "params": {
      "start_time": "2017-04-24T04:00:00Z",
      "segmentation_type": "DEVICES",
      "entity_ids": [
        "86onf"
      ],
      "end_time": "2017-04-25T04:00:00Z",
      "placement": "ALL_ON_TWITTER",
      "granularity": "TOTAL",
      "entity": "CAMPAIGN",
      "platform": "0",
      "metric_groups": [
        "ENGAGEMENT"
      ]
    }
  },
  "data_type": "job",
  "data": {
    "start_time": "2017-04-24T04:00:00Z",
    "segmentation_type": "DEVICES",
    "url": null,
    "id_str": "860558303541067776",
    "entity_ids": [
      "86onf"
    ],
    "end_time": "2017-04-25T04:00:00Z",
    "country": null,
    "placement": "ALL_ON_TWITTER",
    "id": 860558303541067800,
    "expires_at": null,
    "account_id": "18ce53zb651",
    "status": "PROCESSING",
    "granularity": "TOTAL",
    "entity": "CAMPAIGN",
    "created_at": "2017-05-05T18:14:22Z",
    "platform": "0",
    "updated_at": "2017-05-05T18:14:22Z",
    "metric_groups": [
      "ENGAGEMENT"
    ]
  }
}

2- Use the id_str from step 1 in a request to the GET /1/stats/jobs/accounts/:account_id endpoint

$  twurl -H ads-api.twitter.com "/1/stats/jobs/accounts/18ce53zb651?job_ids=860558303541067776"
{
  "request": {
    "params": {
      "job_ids": [
        860558303541067800
      ]
    }
  },
  "data_type": "job",
  "next_cursor": null,
  "data": [
    {
      "start_time": "2017-04-24T04:00:00Z",
      "segmentation_type": "DEVICES",
      "url": "https://ton.twimg.com/advertiser-api-async-analytics/OUkKN4O1YnjEnE131HOxhNkCoklzqGcF4we340km6PVCadeunTw_jJq2gbFOljYraeL0BdceWSaH41rhCrHMQkm-4wNT5ynOeM_1LLGJztDPBgsZOuQG35Aayc79lMCO.json.gz",
      "id_str": "860558303541067776",
      "entity_ids": [
        "86onf"
      ],
      "end_time": "2017-04-25T04:00:00Z",
      "country": null,
      "placement": "ALL_ON_TWITTER",
      "id": 860558303541067800,
      "expires_at": "2017-05-07T18:14:24Z",
      "account_id": "18ce53zb651",
      "status": "SUCCESS",
      "granularity": "TOTAL",
      "entity": "CAMPAIGN",
      "created_at": "2017-05-05T18:14:22Z",
      "platform": "0",
      "updated_at": "2017-05-05T18:14:24Z",
      "metric_groups": [
        "ENGAGEMENT"
      ]
    }
  ]
}

Can you include the output of the second step into the thread? That will enable us to ensure that we’re able to successfully download the gz file.

[ChrisNiebel]

I was using the POST/GET as you specify. The problem was I was saving the response to disk and could not open the gzip file using any utility. I ended up using .NET’s GZip serializer to unzip it in memory then store it to disk. /shrug