TON Resumable Upload header location bug

ton-upload
ton

#1

Found a couple of inconsistencies with the TON upload docs (specifically used in the tailored audience flow).

https://dev.twitter.com/rest/ton/resumable
Docs denote the header location as Location: /1.1/ton/bucket/{bucket}/SzFxGfAg_Zj.mp4

The actual header location in the response looks something like this (for ta_partner bucket used for tailored aud list uploads): /ta_partner/{bucket}/SzFxGfAg_Zj.mp4.
Directly using this header in POST accounts/:account_id/tailored_audience_changes request (https://dev.twitter.com/ads/reference/post/accounts/%3Aaccount_id/tailored_audience_change) results in a 400 with error {"code":"INVALID_PARAMETER","message":"Expected valid file path, got \"\/ta_partner\/{bucket}\/SzFxGfAg_Zj.mp4\" for input_file_path","parameter":"input_file_path"}*.

The header location returned from the initialization request and used for the subsequent chunk uploads look like this: /1.1/ton/data/ta_partner/{bucket}/SzFxGfAg_Zj.mp4?resumable=true&resumeId=406297.

I looked at the pattern of this path and tacked the /1.1/ton/data prefix to the original header location and tried using that in the input_file_path request param to POST accounts/:account_id/tailored_audience_changes: /1.1/ton/data/ta_partner/{bucket}/SzFxGfAg_Zj.mp4 – which worked and returned a 201.

It looks like there is an actual bug with the final returned location header in the TON resumable upload flow, which omits the required /1.1/ton/data prefix in the file path. And a discrepancy in the twitter api docs where the sample response looks different from what is actually returned (or what the location should look like). The single upload docs page response is also wrong https://dev.twitter.com/rest/ton/single-chunk


#2

Thanks for raising this issue. It might be the same thing that was blocking TON upload via our Ruby and Python SDK - since changing TON behavior is going to take us a while we ended up working around it with this workaround:

The problem at least when I debugged it then was the parameter part (?resumable=true&resumeId=406297), and stripping that off allows it to work. I’ve re-raised this issue internally and will try to get the core of the issue fixed.

If you could please help confirm if it’s the same issue as that or if it’s a different issue after all then we will debug again and try to repro what’s going on.

Thanks for the detailed report!

John


#3

Hmmm no, this looks like a different issue. I’m not sure how the resumable upload flow looks for non-ta_partner bucket uploads, but what I’m seeing is specific to the tailored audience flow. The faulty location header that I’m referring to is from the response of the final/last chunk in the series of resumable uploads (last response in the docs here). The fix that I made in my code was to use the response header location prefixed with /1.1/ton/data as the input_file_path param in the POST accounts/:account_id/tailored_audience_changes request. That solved the 400s for me!


#4

Hi, sorry for the delay here.

Unfortunately I haven’t been able to repro the issue with “/1.1/ton/data” being missing.

What I do repro (with https://github.com/twitterdev/ton-upload/blob/master/ton_upload reference script)
Command to run:
./ton_upload --trace -m upload -b ta_partner -f ~/handlesmedium.txt
(File > 30MB)

Is that the final location is being returned like this:
Uploading chunk 43 (44040192-45088767/45836603)…
Uploading chunk 44 (45088768-45836602/45836603)…
File stored at /1.1/ton/data/ta_partner/####/jLTg0Sq2eAOQu5X.txt?resumable=true&resumeId=265580

I can guess that the location as “/1.1/ton/data/ta_partner/####/jLTg0Sq2eAOQu5X.txt?resumable=true&resumeId=265580”, if passed to tailored_audience_changes (with the ? part) will get a similar error message to the one in your original post (maybe API is confused or only reports part of the URL as parsed?). Could you please double check the location and let us know if you definitely still see the prefix missing? (we might have to work on this with you offline if so).

Thanks,

John


#5

Yeah I definitely see the prefix missing (and I’m not seeing any extra params). I’m developing in Go, so I wrote my own uploading script (not using the ruby script). Manually adding the prefix has been the only way I’ve been able to get our script working to load an audience into twitter.

The location header from the first multi-chunk TON initialization call response used for the subsequent chunk upload requests:
/1.1/ton/data/ta_partner/<ID_OMITTED>/<FILE_NAME_OMITTED>.csv?resumable=true&resumeId=244401

Headers from a response from the final TON chunk upload look something like this (omitted some stuff):
201 Created 201 HTTP/2.0 2 0 map[Content-Length:[0] Domain=.twitter.com; Path=/; Expires=Fri, 08-Feb-2019 00:37:46 UTC] Strict-Transport-Security:[max-age=631138519] X-Content-Type-Options:[nosniff] X-Rate-Limit-Reset:[1486515135] Date:[Wed, 08 Feb 2017 00:37:46 GMT] Location:[/ta_partner/<ID_OMITTED>/<FILE_NAME_OMITTED>.csv] X-Rate-Limit-Limit:[90000] X-Rate-Limit-Remaining:[89931] X-Response-Time:[90] X-Tsa-Request-Body-Time:[255]