Problem with audience manager trying to create an audience through API


#1

I seem to be having a problem creating an audience from my Java app which deals with tailored audiences. It works fine connecting to the Ads API, and I am able to query the existing audiences fine. However, when I am trying to create an audience, I get a 403 / Forbidden and the error is:

{“errors”:[{“code”:“UNAUTHORIZED_CLIENT_APPLICATION”,“message”:“The client application making this request does not have access to this API”}],“request”:{“params”:{}}}

Any ideas what causes this? As I said querying the audiences works absolutely fine.

Thanks,

Liv


#2

I do keep wondering, with things like CREATE, the documentation (https://dev.twitter.com/ads/reference/post/accounts/%3Aaccount_id/tailored_audiences) suggests placing the parameters in the URL rather than inside the HTTP POST. Is that definitely the case or is the documentation inacurate?


#3

When you get a 401 response, that’s generally an issue with how you’re using OAuth in the request. However, when you see a 403 it means your request was technically sound, but your app or the user token provided doesn’t have permission to do the thing you’re attempting to do.

A couple of questions:

  1. Has the user token been revoked? Often times this type of error occurs because the user has revoked the token that was granted to this application. This can be confirmed by trying other types of requests. For example, if you’re also seeing 401s when you try to call GET /0/accounts it’s likely this is the case.

  2. Does the app you’re working with have Ads API access? While the program is open to many new developers, app white-listing is still required.


#4

Thank you for the clarifications on that! I was wondering what the difference in between 401 and 403 are – so it’s good to know that the OAuth is fine (which makes sense, since the GET request to retrieve the audiences works).
However, this leads me now to the question about why this is happening. The token hasn’t been revoked (as I said GET audiences for instance works).
Could be that the app has been granted then limited access – how do I check this? All of this is happening inside a Netflix (corporate) account so I would hope we do have access? We have access to the Audience Management tools in the ads console – are these 2 not tied in together?


#5

Also I see the same 403 gets issued when I issue a TON “write” request. Is TON access based on a whitelist too?


#6

What’s the application ID for the app you’re seeing these failures with?


#7

It’s the Netflix Ads API app – based on the URL in apps.twitter.com I’m guessing the ID is 7751317. The owner ID is 16573941


#8

@liviutudor looks like your application wasn’t setup on our end correctly. You don’t currently have permission to use the TON API which is why you’re seeing 403s when you attempt to upload.

Can you try again and let me know if you see a different result?


#9

@liviutudor your app is now setup correctly and permissions on Twitter’s end should no longer be the issue.

What bucket are you currently attempting to upload too?

However, the TON API requires some very specific headers, content-type, etc to be set on each request and is often difficult for developers to get started with.

I’d like to have to try to request again with our ton-upload tool / example script (outside of your own implementation) so we can confirm if the issue is indeed a permission issue or an issue within your own test code.

Please try the following (Ruby 2.0 recommend) and let me know wha the results are:

git clone https://github.com/twitterdev/ton-upload.git
cd ton-upload
bundle install
./ton_upload --mode upload --bucket ta_partner --file /path/to/file

#10

This is what i get running the ton_upload script:

lgml-ltudor3QC:ton-upload > ./ton_upload --mode upload --bucket ta_partner --file ~/Documents/test.csv 
WARNING: The rest_client gem is deprecated and will be removed from RubyGems. Please use rest-client gem instead.
./ton_upload:31:in `<main>': undefined method `[]' for nil:NilClass (NoMethodError)
lgml-ltudor3QC:ton-upload > 

Ruby version:

lgml-ltudor3QC:ton-upload > ruby -version
ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14]

#11

Regarding the headers/etc I followed the documentation provided about TON and Tailored Audiences – is there anything that is not included there?


#12

@liviutudor before you can run the ton-upload script you need to have twurl setup on your machine. The ton-upload script uses your OAuth config from twurl to make requests and we often require using twurl to trouble shoot API issues (so it’s good in general to have this setup).

gem install twurl
twurl authorize --consumer-key key --consumer-secret secret

Once you have that installed and you’ve setup OAuth for twurl, you should be able to repeat your test with the ton-upload script and get past the error you were seeing.


#13

Thanks @brandonmblack – that worked and the ton_upload script uploaded correctly.
My Java code is still receiving 403 on POST requests only – all the GET requests are going through.
Is there a way to see all the ton_upload exchange of headers so I can try and inspect what my test is doing wrong?

Scrap that I’ve figured out the -t parameter…


#14

OK so a few questions on this:

  • the ton_upload script doesn’t do resumeable uploads which is what I’m dealing with – is there another script to emulate that?

  • I have gone through the docco again (https://dev.twitter.com/rest/ton/resumable) to ensure I have all the headers and I DO now supply all the headers from the documentation

  • however, the ton_upload doesn’t seem to – this is a header trace:

      POST https://ton.twitter.com/1.1/ton/bucket/ta_partner HTTP/1.1
      Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
      Accept: */*
      User-Agent: OAuth gem v0.4.7
      Content-Type: text/comma-separated-values
      Content-Length: 325
      X-Ton-Expires: Thu, 28 May 2015 21:03:09 GMT
      Authorization: ....
      Connection: close
      Host: ton.twitter.com
    

There is no X-Ton-Content-Length, or X-TON-Content-Type :frowning:

So are these required or not?


#15

Issueing the following request I now encounter a HTTP 400 (Bad request):

POST /1.1/ton/bucket/ta_partner?resumable=true HTTP/1.1
Authorization: ..
X-TON-Content-Length: 11
X-TON-Content-Type: text/comma-separated-values
Content-Type: text/comma-separated-values
Content-Length: 0
Host: ton.twitter.com
User-Agent: Apache-HttpClient/4.4.1 (Java/1.8.0_40)
Accept-Encoding: gzip,deflate
Connection: close

When is a HTTP 400 issued? and what is wrong with the above request?


#16

@liviutudor the ton-upload script actually does do a multi-part resumable upload but it’s automatically determined based on the file size. If the upload file is > 8MB it will automatically switch to a multi-part upload.

The logic fork on file size is here:

Here’s the code that initiates a multi-part upload (POST request):

As you can see, the header’s specified on the initial multi-part request are:

  • X-TON-Content-Type (correct MIME type)
  • X-TON-Content-Length (file size)
  • X-TON-Expires (can be up to 7 days)
  • Content-Length (always 0 on the initial POST)
  • Content-Type (same as X-TON-Content-Type)

The remaining chunks get sent in subsequent requests as follows (PUT request):

For the individual chunk requests, only Content-Type, Content-Length and Content-Range headers are set. In addition, the PUT request with each chunk must contain the location value returned from the initial POST where your multipart session was created.

If your initial POST to create a multi-part session returned a 403 or failed for any other reason, the subsequent PUT requests for each chunk will return a 400 due to that location not being valid. I believe the issue in your initial POST request is that content length is set to 325 and should be set to zero.

Also note that application/x-www-form-urlencoded is never allowed and will be treated as a bad request. Looking at your request trace, you aren’t setting this and you’re ok here but its a common pitfall worth pointing out.


Facing 403 Forbidden error in ton api
Tailored audiences and ton/ta_data?
#17

Hi @brandonmblack, this is very helpful!

Looking at my code the only header I’m missing seems to be the X-TON-Expires is this header optional or mandatory?

I haven’t tested yet the chunks upload since the initial request fails, so the initial 403 cannot be “blamed” for that since it is during the initial request this is failing…

Also, I am running the test using a small file (11 bytes) – is this the reason why I am failing my tests as the file is under 8Mb? Is this a requirement that all files under 8Mb need to upload via non-resumeable uploads?

I have just run another requests of a file over 8Mb (83,886,080 bytes) and still getting HTTP 400 –

HttpResponseProxy{HTTP/1.1 400 Bad Request [cache-control: no-cache, connection: close, content-length: 0, date: Mon, 18 May 2015 23:11:12 GMT, server: tsa_a, set-cookie: guest_id=v1%3A143199067272724883; Domain=.twitter.com; Path=/; Expires=Wed, 17-May-2017 23:11:12 UTC, strict-transport-security: max-age=631138519, x-connection-hash: 6d029f685454d711e3aa95c6f639f9e3, x-rate-limit-limit: 90000, x-rate-limit-remaining: 89999, x-rate-limit-reset: 1431991572, x-response-time: 10] [Content-Length: 0,Chunked: false]}

The headers all look correct on this side so I am only suspecting that I need the X-TON-Expires header? Is this what is causing the failure?

POST /1.1/ton/bucket/ta_partner?resumable=true HTTP/1.1 
Authorization: ...
 X-TON-Content-Length: 83886080
 X-TON-Content-Type: text/comma-separated-values
 Content-Type: text/comma-separated-values
 Content-Length: 0
 Host: ton.twitter.com
 User-Agent: Apache-HttpClient/4.4.1 (Java/1.8.0_40)
 Accept-Encoding: gzip,deflate
 Connection: close

#18

OK so follow up on this:

adding the X-TON-Expires header seems to fix the initial upload for me when the file is over 8Mb. If the file is smaller then I get a HTTP 400 (Bad request).

A few things here:

  • this header is NOT being mentioned at all in the TON docco however it seems it is required – and as such I suggest strongly the docco is updated.
  • it is not possible it seems to perform a resume-able upload for files under 8Mb. This seems to be a random limit since the documentation talks about 64Mb. Also, the docco is vague about whether resumeable uploads can be used or not for smaller files:

For files less than 64MB, non-resumable may be used. For files greater than or equal to 64MB, resumable must be used.

  • The above suggests that we can (but don’t have to!) use a non-resumeable upload for under 64Mb, whereas in fact I gather that it should read: "For files less than 64Mb non-resumeable MUST be used…"
  • the headers are referred to sometimes as X-TON-… and sometimes as X-Ton-… I guess this means that the Twitter API is NOT case sensitive when it comes to these headers. This easily causes confusion as the HTTP protocol specifies that the headers ARE in fact case sensitive.

I am in the process now of testing the non-resumeable upload too and the chunks upload and will loop back in this with details. I would like in the meanwhile to get clarifications on the above.

Thanks,


#19

It is a huge oversight that X-TON-Expires header is not documented as a required header. We wasted a lot of time debugging because of that critical requirement not being mentioned in the online documentation. Please fix this!


#20

@liviutudor @dodnert

Bear in mind that the TON API is used for many things, not just TA list uploads. It’s a generic API for simple blob storage used by a number of products both internal and external to Twitter. This is unfortunately also why error messages from this API are a bit obscure and difficult to decipher.

The X-TON-Expires header is only required on some buckets and the documentation you’re referring too is generic in purpose and meant for the TON API in general. For the ta_partner bucket we do always require the X-TON-Expires header. In our example script on GitHub we always set the X-TON-Expires header and this is considered a safe practice since buckets that don’t require the header will simply ignore it.

Regarding case-sensitivity of this header, per the HTTP/1.1 spec (RFC2616) all HTTP headers are always case-insensitive:

Each header field consists of a name followed by a colon (":") and the field value. Field names are case-insensitive.

As noted already, Content-Type can never be set to application/x-www-form-urlencoded for any bucket and all requests with this value will always be rejected. While the ta_partner bucket accepts all other valid MIME types, some select TON API buckets do place additional restrictions on the Content-Type values allowed.

While the docs do need to remain generic and multi-purpose since this is a separate API used by many things, what would be useful is a table with each bucket, whether or not the X-TON-Expires header is required and the Content-Type values each bucket will accept.

There’s no minimum size required for multi-part uploads, but the maximum size for a single chunk PUT request is 64MB (however, we recommend using a smaller chunk size like 32MB for better results). You could indeed only implement the multi-part upload and for smaller file sizes and just do the initial POST request followed by a single PUT chunk request with the correct byte size specified.

The truth is though, as you’re both experiencing now, the TON API is a bit difficult to get the hang of when you’re first starting out and notorious for being a less-than-friendly interface for developers to work with. In the long run, you’ll likely see us move away from this as a solution and towards more developer-friendly APIs (eg. something like a batch upload for TA lists).