Problem with audience manager trying to create an audience through API

[liviutudor]

Issueing the following request I now encounter a HTTP 400 (Bad request):

POST /1.1/ton/bucket/ta_partner?resumable=true HTTP/1.1
Authorization: ..
X-TON-Content-Length: 11
X-TON-Content-Type: text/comma-separated-values
Content-Type: text/comma-separated-values
Content-Length: 0
Host: ton.twitter.com
User-Agent: Apache-HttpClient/4.4.1 (Java/1.8.0_40)
Accept-Encoding: gzip,deflate
Connection: close

When is a HTTP 400 issued? and what is wrong with the above request?

[brandonmblack]

@liviutudor the ton-upload script actually does do a multi-part resumable upload but it’s automatically determined based on the file size. If the upload file is > 8MB it will automatically switch to a multi-part upload.

The logic fork on file size is here:

twitterdev/ton-upload/blob/master/ton_upload#L210-L214

Here’s the code that initiates a multi-part upload (POST request):

twitterdev/ton-upload/blob/master/ton_upload#L118-L138

As you can see, the header’s specified on the initial multi-part request are:

• X-TON-Content-Type (correct MIME type)
• X-TON-Content-Length (file size)
• X-TON-Expires (can be up to 7 days)
• Content-Length (always 0 on the initial POST)
• Content-Type (same as X-TON-Content-Type)

The remaining chunks get sent in subsequent requests as follows (PUT request):

twitterdev/ton-upload/blob/master/ton_upload#L164-L170

For the individual chunk requests, only Content-Type, Content-Length and Content-Range headers are set. In addition, the PUT request with each chunk must contain the location value returned from the initial POST where your multipart session was created.

If your initial POST to create a multi-part session returned a 403 or failed for any other reason, the subsequent PUT requests for each chunk will return a 400 due to that location not being valid. I believe the issue in your initial POST request is that content length is set to 325 and should be set to zero.

Also note that application/x-www-form-urlencoded is never allowed and will be treated as a bad request. Looking at your request trace, you aren’t setting this and you’re ok here but its a common pitfall worth pointing out.

[liviutudor]

Hi @brandonmblack, this is very helpful!

Looking at my code the only header I’m missing seems to be the X-TON-Expires is this header optional or mandatory?

I haven’t tested yet the chunks upload since the initial request fails, so the initial 403 cannot be “blamed” for that since it is during the initial request this is failing…

Also, I am running the test using a small file (11 bytes) – is this the reason why I am failing my tests as the file is under 8Mb? Is this a requirement that all files under 8Mb need to upload via non-resumeable uploads?

I have just run another requests of a file over 8Mb (83,886,080 bytes) and still getting HTTP 400 –

HttpResponseProxy{HTTP/1.1 400 Bad Request [cache-control: no-cache, connection: close, content-length: 0, date: Mon, 18 May 2015 23:11:12 GMT, server: tsa_a, set-cookie: guest_id=v1%3A143199067272724883; Domain=.twitter.com; Path=/; Expires=Wed, 17-May-2017 23:11:12 UTC, strict-transport-security: max-age=631138519, x-connection-hash: 6d029f685454d711e3aa95c6f639f9e3, x-rate-limit-limit: 90000, x-rate-limit-remaining: 89999, x-rate-limit-reset: 1431991572, x-response-time: 10] [Content-Length: 0,Chunked: false]}

The headers all look correct on this side so I am only suspecting that I need the X-TON-Expires header? Is this what is causing the failure?

POST /1.1/ton/bucket/ta_partner?resumable=true HTTP/1.1 
Authorization: ...
 X-TON-Content-Length: 83886080
 X-TON-Content-Type: text/comma-separated-values
 Content-Type: text/comma-separated-values
 Content-Length: 0
 Host: ton.twitter.com
 User-Agent: Apache-HttpClient/4.4.1 (Java/1.8.0_40)
 Accept-Encoding: gzip,deflate
 Connection: close

[liviutudor]

OK so follow up on this:

adding the X-TON-Expires header seems to fix the initial upload for me when the file is over 8Mb. If the file is smaller then I get a HTTP 400 (Bad request).

A few things here:

• this header is NOT being mentioned at all in the TON docco however it seems it is required – and as such I suggest strongly the docco is updated.
• it is not possible it seems to perform a resume-able upload for files under 8Mb. This seems to be a random limit since the documentation talks about 64Mb. Also, the docco is vague about whether resumeable uploads can be used or not for smaller files:

For files less than 64MB, non-resumable may be used. For files greater than or equal to 64MB, resumable must be used.

• The above suggests that we can (but don’t have to!) use a non-resumeable upload for under 64Mb, whereas in fact I gather that it should read: "For files less than 64Mb non-resumeable MUST be used…"
• the headers are referred to sometimes as X-TON-… and sometimes as X-Ton-… I guess this means that the Twitter API is NOT case sensitive when it comes to these headers. This easily causes confusion as the HTTP protocol specifies that the headers ARE in fact case sensitive.

I am in the process now of testing the non-resumeable upload too and the chunks upload and will loop back in this with details. I would like in the meanwhile to get clarifications on the above.

Thanks,

[dodnert]

It is a huge oversight that X-TON-Expires header is not documented as a required header. We wasted a lot of time debugging because of that critical requirement not being mentioned in the online documentation. Please fix this!

[brandonmblack]

@liviutudor @dodnert

Bear in mind that the TON API is used for many things, not just TA list uploads. It’s a generic API for simple blob storage used by a number of products both internal and external to Twitter. This is unfortunately also why error messages from this API are a bit obscure and difficult to decipher.

The X-TON-Expires header is only required on some buckets and the documentation you’re referring too is generic in purpose and meant for the TON API in general. For the ta_partner bucket we do always require the X-TON-Expires header. In our example script on GitHub we always set the X-TON-Expires header and this is considered a safe practice since buckets that don’t require the header will simply ignore it.

Regarding case-sensitivity of this header, per the HTTP/1.1 spec (RFC2616) all HTTP headers are always case-insensitive:

Each header field consists of a name followed by a colon (":") and the field value. Field names are case-insensitive.

As noted already, Content-Type can never be set to application/x-www-form-urlencoded for any bucket and all requests with this value will always be rejected. While the ta_partner bucket accepts all other valid MIME types, some select TON API buckets do place additional restrictions on the Content-Type values allowed.

While the docs do need to remain generic and multi-purpose since this is a separate API used by many things, what would be useful is a table with each bucket, whether or not the X-TON-Expires header is required and the Content-Type values each bucket will accept.

There’s no minimum size required for multi-part uploads, but the maximum size for a single chunk PUT request is 64MB (however, we recommend using a smaller chunk size like 32MB for better results). You could indeed only implement the multi-part upload and for smaller file sizes and just do the initial POST request followed by a single PUT chunk request with the correct byte size specified.

The truth is though, as you’re both experiencing now, the TON API is a bit difficult to get the hang of when you’re first starting out and notorious for being a less-than-friendly interface for developers to work with. In the long run, you’ll likely see us move away from this as a solution and towards more developer-friendly APIs (eg. something like a batch upload for TA lists).

[liviutudor]

Hi @brandonmblack, your help with this is much appreciated!

I agree with the others commenting on this thread that indeed the X-Ton-Expires is a big deal, having it left out of the docco and I totally get the fact that the TON api is quite generic, it does come across as trying to be “one-size-fits-all” sort of solution. It’s precisely that it has so many applicability areas that all these things should be documented.

I think the idea with the table listing which buckets have special requirements / restrictions is a good start – either that or mention in each API using TON explicitely the requirements on the bucket.

Also what would help I think is if the TON API (and other API’s for that matter) explicitely state that all the parameters are to be passed in the URL rather than including them in the body; I know they state that the application/x-ww-form are not permitted but this is not as obvious as clearly communicating that.

The particular cases around file sizes when uploading a single chunk or a resumeable upload makes me a bit nervous as based on this discussion, the documentation and my tests, it seems to be another area where each bucket + API + (quite likely) connection speed can render such different results. For instance, the API returns a max-chunk-size and the docco recommends we don’t exceed it, however it then goes on to suggest that we choose a chunk size that we can send over the wire in 30 seconds – this to me reads more like a must based on the previous findings. Your suggestion to choose a lower size than the max-chunk-size also makes me wonder about what other oddities are out there hidden in the TON API. Anything else you can think of with regards to multiple chunks uploads?

[dodnert]

Thanks, @brandonmblack. I have just two things I’d like to point out in response.

1. There is no documentation of the X-TON-Expires header anywhere on dev.twitter.com that I could find. This needs to be fixed.

2. On the TON API overview page, you will find the sentence “For Tailored Audiences, please use the bucket name ta_partner.” Would it be so difficult to add one more sentence that reads “For Tailored Audience uploads, please make sure that the X-TON-Expires header is defined with the date set to ???.”

[brandonmblack]

@dodnert done.

https://dev.twitter.com/rest/ton

[dodnert]

@brandonmblack - Cool!

BTW, is there documentation anywhere about the date string syntax that is acceptable in this X-header?

[liviutudor]

@dodnert the date string is a standard http date header
@brandonmblack thanks for updating the docco’s

[brandonmblack]

@dodnert @liviutudor’s comment is correct. We use the HTTP standard for representing date times in headers. You can read more about that here.

Here’s the relevant section:

HTTP applications have historically allowed three different formats for
the representation of date/time stamps:

Sun, 06 Nov 1994 08:49:37 GMT ; RFC 822, updated by RFC 1123
Sunday, 06-Nov-94 08:49:37 GMT ; RFC 850, obsoleted by RFC 1036
Sun Nov 6 08:49:37 1994 ; ANSI C’s asctime() format

The first format is preferred as an Internet standard and represents
a fixed-length subset of that defined by RFC 1123 [8] (an update to
RFC 822 [9]). The second format is in common use, but is based on the
obsolete RFC 850 [12] date format and lacks a four-digit year.
HTTP/1.1 clients and servers that parse the date value MUST accept
all three formats (for compatibility with HTTP/1.0), though they MUST
only generate the RFC 1123 format for representing HTTP-date values
in header fields. See section 19.3 for further information.

Additionally, make sure you note that the RFC also requires all times always be in GMT / UTC:

All HTTP date/time stamps MUST be represented in Greenwich Mean Time
(GMT), without exception. For the purposes of HTTP, GMT is exactly
equal to UTC (Coordinated Universal Time).

[liviutudor]

OK so having gotten past the issue with TON I am still encountering issues now with the Tailored Audiences API – the dreaded 403:

HTTP/1.1 403 Forbidden [content-disposition: attachment; filename=json.json, content-type: application/json;charset=utf-8, date: Tue, 19 May 2015 20:23:31 GMT, server: tsa_a, set-cookie: guest_id=v1%3A143206701154609507; Domain=.twitter.com; Path=/; Expires=Thu, 18-May-2017 20:23:31 UTC, strict-transport-security: max-age=631138519, x-access-level: read-write, x-connection-hash: fd85961944b8bd7cecaf2977de6dd1fa, x-content-type-options: nosniff, x-frame-options: SAMEORIGIN, x-response-time: 23, x-runtime: 0.01229, x-transaction: 36c9b81e714421b7, x-xss-protection: 1; mode=block

As I understand this is not an issue with OAuth but rather with having access to this API?

[liviutudor]

@brandonmblack are you aware of any special requirements (headers etc) with the Tailored Audiences API?
The error response clearly states this as I get:

UNAUTHORIZED_CLIENT_APPLICATION / The client application making this request does not have access to this API

[brandonmblack]

@liviutudor looks like this might be another instance of your app not being setup correctly. Apologies this wasn’t handled correctly in the first attempt. Can you give it another try?

[liviutudor]

@brandonmblack Thanks for promptly jumping on this! I can confirm now I was able to create, delete, retrieve an audience now.
Testing next uploading changes to it – my guess is with TON and TA working this should be fine too but I will post here an update to confirm.

[liviutudor]

@brandonmblack OK so I am getting a 401 when POST-ing an audience change which suggests an OAuth issue. However, the same code used for all the other POST/GET blocks applies so I’m not entirely sure why that could happen.
Are there any particulars about POST accounts/:account_id/tailored_audience_changes ? (headers or parameters that are needed but not supplied in the docco?
If it helps diagnozing this, the audience details used in last test:

id=saqt
name=testing_audience_IhcldTpotjCMuCWDVQxEVQ8bNReoUHCDOfkBB9ncTQ

(I’ve deleted the audience in the meanwhile btw)

[liviutudor]

Update here – found another problem with the docco: for making changes to the audiences, there is a input_file_path parameter in the URL, which is a path from (previously) uploaded to TON. All the examples on that page suggest using the value from TON unencoded – e.g. /1.1/ton/data/ta_partner/3wyo1gwuqF7_Lhr (see https://dev.twitter.com/ads/reference/post/accounts/%3Aaccount_id/tailored_audience_change) however doing so fails with a 401 as the parameter when “unpacked” on the other side gets mangled. Instead, this parameter needs to be URL-encoded (and this has an effect on the OAuth headers). Probably need to update the docco to reflect this.

I can confirm though that create / make changes / delete / retrieve audiences works now on my end. (We haven’t implemented any of the other operations so might start a separate thread at the time if we find issues with it.)

Thank you so much for your help with this, @brandonmblack !

[brandonmblack]

@liviutudor glad I could help and glad to see you up and running.

[pykler]

@brandonmblack actually the initial post, if the X-Content-Length is too small (<< 8MB, e.g. 1KB) the initial post fails so you will not be able to do a multi part upload for small files. Same logic that is in the TON upload script needs to be kept.