Data processing differently via API vs. direct upload to audience

api

#1

When we create audience and add contacts via API, when processing finishes result is always ‘audience too small’. But, when we upload the same data to create new audience directly on Twitter audience manager, data processes correctly and thousands of contacts are matched (audience enters ready state on completion of processing).
This is from audience manager:

We have tried this with two separate data sets with same results. Please let us know if any further information is needed. Any assistance you can offer will be greatly appreciated.

Thanks,
Frank Lacour


#2

Hey @silverpop_ea

Thank you for bringing this to our attention. In order to investigate the issue further, can you provide us with the following details:

  • The Ads account id
  • The id’s for the Audiences that have issues
  • The id’s for the Audiences uploaded via the UI
  • The timestamp when these audiences were uploaded

In addition, please note that Audiences may take up to a few days in order to be processed, which may account for the delay.

Thanks!


#3

@imit8me,
Thanks for the quick reply! The following is the requested information:
ad account id: 18ce53z8vuo
audiences:
Api upload:
id: 1lz35
timestamp: 2016-12-19 14:58:08

id: 1m3z3
timestamp: 2016-12-19 16:14:31

For the audiences created by direct upload in the UI, the creation timestamp is not in the response when we make a get audiences Api call, so we do not have a way of capturing or storing timstamps from audiences created in Twitter audience manager that get shared back to our app. These are the audiences created via Twitter audience manager UI:
id: 1m40k
date created: 2016-12-19
size: 3220

id: 1m61r
date created: 2016-12-20
size: 2053

I am waiting for the audience to either show ‘ready’ or ‘audience too small’ in Twitter audience manager. Processing should have completed at that time, yes? Please let us know if any further information is needed. Any assistance or information you can offer will be greatly appreciated.

Thanks,
Frank Lacour


#4

Hi,
I just wanted to check if there was any information or update in regards to the reported issue. Anything you can offer will be greatly appreciated. Thanks!


#5

Thanks for the follow up, @silverpop_ea, and for providing the audience details.

For 1lz35 and 1m3z3, zero users were matched, which is why the response shows "reasons_not_targetable": [TOO_SMALL]. This suggests an issue with the audience files.

A few questions:

  • Have you successfully uploaded an audience—one that shows a READY status—via the API before? If so, is there anything different between those instance and these?
  • Were the entries—in this case, emails—hashed?

#6
  • Well, 1lz35 was created with the same set of data that 1m40k was generated from and 1m3z3 was the same set of data used for 1m61r, but 1m40k and 1m61r were created in the audience manager UI and the other two via API. So, there should be matches, but I am suspecting a problem with the data we are trying to send to the API (we are still getting a success response, which makes troubleshooting fun).
  • I was not the original developer of this functionality in our app, and older data has been purged from our test account (so do not readily have ids of previous ‘ready’ audiences, but that part of the code has not changed for a pretty long time, so there should be no difference between previous instances and those reported in this thread).
  • The code was written to use the raw entries, but I have tried hashing the data as well. The results are the same: ‘audience too small’.

Any further information you can offer will be greatly appreciated.

–Frank Lacour


#7

Agreed since we’re seeing zero matches for 1lz35 and 1m3z3. When did you try using the hashed data? Do you have IDs for those? Have you tried using the scripts found here for hashing the audience files?


#8

I was not 100% sure which audience(s) were created using hashed values, so I created 1m5v0 to be absolutely sure. Will advise when it has finished processing.


#9

Great! Thanks, @silverpop_ea. We’ll keep an eye on this, too.

In the meantime, could you please provide an example of the requests and responses used to:

  • hash the entries in the audience file
  • upload the audience file
  • create the placeholder tailored audience
  • add the records to the tailored audience based on the uploaded audience file

Thanks!


#10

Apologies for the delay.

  • The hashing is taking place within our PHP app (no request or response to provide). We are using the PHP language function hash() as hash(‘sha256’, $email_address) when encoding the email addresses.

  • file upload:

request:

POST /1.1/ton/bucket/ta_partner HTTP/1.1
Host: ton.twitter.com
Accept: */*
Content-Type: text/comma-separated-values
Content-Length: 2392040
X-TON-Expires: Mon, 2 Jan 2017 16:59:22 GMT
Authorization: OAuth oauth_consumer_key="XY35yOB3u2XzvV0j74jNfpCqw", oauth_nonce="100af6c201e7e582d2b983da433b7f74cfc8bc00f852b5afcab3c946468ba5e5104902fee59a22d9cd5a29585e312d3ee0431d9369b3d59c112e01e538532719", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1482944362", oauth_version="1.0", oauth_token="3186774122-BTAGH0L5xUtP8XBi0Bq7xDb1h6NfSzdMdr5JcEv", oauth_signature="LNaWKRHkFOcyhqpo%2FFQ8jSIU2xk%3D

response:

HTTP/1.1 201 Created
content-length: 0
content-type: application/octet-stream
date: Wed, 28 Dec 2016 15:00:14 GMT
location: /1.1/ton/data/ta_partner/3186774122/K1MzZuWjVNDhnhq
server: tsa_b
set-cookie: guest_id=v1%3A148293721406184322; Domain=.twitter.com; Path=/; Expires=Fri, 28-Dec-2018 15:00:14 UTC
strict-transport-security: max-age=631138519
x-connection-hash: e1b484270be3280c6b60f8f0e8c5efb9
x-content-type-options: nosniff
x-rate-limit-limit: 50
x-rate-limit-remaining: 49
x-rate-limit-reset: 1482938114
x-response-time: 211
x-tsa-request-body-time: 1567
  • audience creation:

request:

POST /1/accounts/18ce53z8vuo/tailored_audiences?account_id=18ce53z8vuo&name=radishbo-ya+twitter+audience+hashed+TESTING+-+EMAIL&list_type=EMAIL HTTP/1.1
Host: ads-api.twitter.com
Accept: */*
Authorization: OAuth oauth_consumer_key="XY35yOB3u2XzvV0j74jNfpCqw",oauth_nonce="3d63d6fe95fc436e55049c6a54b79e5d6a3db3ee32b8f8e9e3e6df21b801e1dd38ce7a55ecb88eb0f4dde532c64a686c41878107605767338bb34f4aed43775b",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1482944362",oauth_version="1.0",oauth_token="31867741
22-BTAGH0L5xUtP8XBi0Bq7xDb1h6NfSzdMdr5JcEv",oauth_signature="mXbj9p%2Fs1u4U4mzVRH23acrcjrw%3D"

response:

{  
   "data_type":"tailored_audience",
   "data":{  
      "targetable":false,
      "name":"radishbo-ya twitter audience hashed TESTING - EMAIL",
      "targetable_types":[  
         "CRM",
         "EXCLUDED_CRM"
      ],
      "audience_type":"CRM",
      "permission_level":"READ_WRITE",
      "is_owner":true,
      "id":"1mk5q",
      "reasons_not_targetable":[  
         "PROCESSING",
         "TOO_SMALL"
      ],
      "list_type":"EMAIL",
      "created_at":"2016-12-28T15:00:12Z",
      "updated_at":"2016-12-28T17:19:22Z",
      "partner_source":"OTHER",
      "deleted":false,
      "audience_size":null
   },
   "request":{  
      "params":{  
         "account_id":"18ce53z8vuo",
         "name":"radishbo-ya twitter audience hashed TESTING - EMAIL",
         "list_type":"EMAIL"
      }
   }
}
  • add records:

request:

POST /1/accounts/18ce53z8vuo/tailored_audience_changes?account_id=18ce53z8vuo&tailored_audience_id=1mk5q&input_file_path=%2F1.1%2Fton%2Fdata%2Fta_partner%2F3186774122%2FayJbsnB6DtAaRIT&operation=ADD HTTP/1.1
Host: ads-api.twitter.com
Accept: */*
Authorization: OAuth oauth_consumer_key="XY35yOB3u2XzvV0j74jNfpCqw",oauth_nonce="449a255ab64257f45a7bf06f1f45e3e3aea90a64955fbd21fb809ab5f171a79cb8e13b011eb54384460180845b860beecc36203f38e37f4e8be6db9a3b08582b",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1482944424",oauth_version="1.0",oauth_token="3186774122-BTAGH0L5xUtP8XBi0Bq7xDb1h6NfSzdMdr5JcEv",oauth_signature="YWQvTcnwhA34zCL1nPeHeRu88Hc%3D"

response:

{  
   "data_type":"tailored_audience_change",
   "data":{  
      "input_file_path":"\/ta_partner\/3186774122\/jBck60eCiHTgDWv",
      "tailored_audience_id":"1mk5q",
      "state":"COMPLETED",
      "id":"728bx",
      "operation":"ADD"
   },
   "request":{  
      "params":{  
         "tailored_audience_id":"1mk5q",
         "input_file_path":"\/ta_partner\/3186774122\/jBck60eCiHTgDWv",
         "operation":"ADD",
         "account_id":"18ce53z8vuo"
      }
   }
}

Please let me know if any further information is needed. Thanks!


#11

This is great, @silverpop_ea!

The issue might be with the way these are hashed. The hashed contents must be base64-encoded. You could use the base64_encode() function. See this post for more information. Once you create a new audience with this change, please let us know if it resolves the issue.

Thanks for taking the time to provide these details!


#12

I created an audience via API where the emails were hashed w/ sha256 and then base64 encoded. It is still processing. I’ll update when processing has finished.


#13

Sounds good. Thanks, @silverpop_ea!


#14

The audience has finished processing, and unfortunately the result was the same: audience too small. The audience ID is 1mjfc.
The email addresses were hashed and encoded in PHP as follows:
base64_encode(hash(‘sha256’, $email));
This same data has produced matches when creating an audience in the audience manager UI. Any information or assistance you can offer will be greatly appreciated.


#15

Thanks for the update, @silverpop_ea.

The base64 encoding in the linked post, above, is for the oauth_body_hash. Sorry about the confusion on that one.

So, the hashing of the entries in the audience file is likely correct. (You can confirm by seeing whether the results of using the hash() function match the results of using one of the these scripts.)

The issue, then, must be with the TON API. Have you tried making a GET request on the returned location header value to see if what you get is what you expect? Note that the hashed data should be sent as the raw post body in the request.

In the meantime, we’ll keep digging on our end. You can also checkout the PHP SDK.


#16

In case it helps, the following Python example shows how to successfully* upload the audience file.

import os

import requests
from requests_oauthlib import OAuth1


FILE = "hashed_user_ids.txt"
SIZE = str(os.path.getsize(FILE))
CONSUMER_KEY = ""
CONSUMER_SECRET = ""
ACCESS_TOKEN = ""
ACCESS_TOKEN_SECRET = ""

with open(FILE, "rb") as ta:
    data = ta.read()

auth = OAuth1(CONSUMER_KEY, CONSUMER_SECRET,
              ACCESS_TOKEN, ACCESS_TOKEN_SECRET)

headers = {"content-type" : "text/plain",
           "content-length" : SIZE,
           "x-ton-expires" : "Mon, 05 Jan 2017 00:00:01 GMT"}

r = requests.post("https://ton.twitter.com/1.1/ton/bucket/ta_partner",
                  data=data, auth=auth, headers=headers)
>>> r.status_code
201
>>> r.headers
{  
  "x-rate-limit-remaining":"89999",
  "x-response-time":"25",
  "content-length":"0",
  "x-content-type-options":"nosniff",
  "x-connection-hash":"48d5ec31a0e2aac8041ae085a9bf0231",
  "set-cookie":"guest_id=v1%3A148316869190993094; Domain=.twitter.com; Path=/; Expires=Mon, 31-Dec-2018 07:18:11 UTC",
  "strict-transport-security":"max-age=631138519",
  "x-tsa-request-body-time":"0",
  "server":"tsa_a",
  "location":"/1.1/ton/data/ta_partner/2417045708/ax3EumENxdlSVir.txt",
  "date":"Sat, 31 Dec 2016 07:18:11 GMT",
  "x-rate-limit-limit":"90000",
  "content-type":"text/plain",
  "x-rate-limit-reset":"1483169591"
}

* Success, in this case, isn’t judged by the response code. Rather, we’ve verified that the entries in the audience file are matched with active Twitter users and that the audience shows a “READY” status—that is, "targetable": true.


#17

I’ve been trying to test a subsequent get call to the returned upload location, but keep getting auth required message (which I’m sure is a problem with oauth header; still working that out), but have another question as well: when you say ‘Note that the hashed data should be sent as the raw post body in the request’, that is for the post for uploading the file, not the subsequent get call you recommended, right? We are using curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata) before curl executes the upload to set the hashed data as the post body and do receive a 201 response when the file is uploaded. Any information you can offer will be greatly appreciated.


#18

I was able to sort out the headers. The GET call to the location returned by the TON upload Api call is returning a 200 OK response (curl output below). From this perspective, it appears the TON upload is successful. I have also validated the hashed values in the uploaded file match what is produced by the sample scripts in the twitterdev/ads-platform-tools package on github (from link above). Any further assistance, suggestion, or information you can offer will be greatly appreciated.

  • About to connect() to ton.twitter.com port 443 (#2)
  • Trying 199.16.156.58…
  • Connected to ton.twitter.com (199.16.156.58) port 443 (#2)
  • warning: ignoring value of ssl.verifyhost
  • skipping SSL peer certificate verification
  • SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  • Server certificate:
  •   subject: CN=ton.twitter.com,OU=Twitter Security,O="Twitter, Inc.",L=San Francisco,ST=California,C=US
    
  •   start date: Feb 02 00:00:00 2016 GMT
    
  •   expire date: Oct 01 12:00:00 2018 GMT
    
  •   common name: ton.twitter.com
    
  •   issuer: CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
    

POST /1.1/ton/bucket/ta_partner HTTP/1.1
Host: ton.twitter.com
Accept: /
Content-Type: text/comma-separated-values
Content-Length: 335
X-TON-Expires: Tue, 10 Jan 2017 20:09:27 GMT
Authorization: OAuth …

  • upload completely sent off: 335 out of 335 bytes
    < HTTP/1.1 201 Created
    < content-length: 0
    < content-type: application/octet-stream
    < date: Thu, 05 Jan 2017 20:09:27 GMT
    < location: /1.1/ton/data/ta_partner/3186774122/CYhbk7-4ovbSjgs
    < server: tsa_b
    < set-cookie: guest_id=v1%3A148364696788168711; Domain=.twitter.com; Path=/; Expires=Sat, 05-Jan-2019 20:09:27 UTC
    < strict-transport-security: max-age=631138519
    < x-connection-hash: 623e2d61633a14f9f5a48c4c1e7b1c33
    < x-content-type-options: nosniff
    < x-rate-limit-limit: 50
    < x-rate-limit-remaining: 43
    < x-rate-limit-reset: 1483647473
    < x-response-time: 23
    < x-tsa-request-body-time: 0
    <

  • Connection #2 to host ton.twitter.com left intact

  • About to connect() to ton.twitter.com port 443 (#3)

  • Trying 199.16.156.58…

  • Connected to ton.twitter.com (199.16.156.58) port 443 (#3)

  • warning: ignoring value of ssl.verifyhost

  • skipping SSL peer certificate verification

  • SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

  • Server certificate:

  •   subject: CN=ton.twitter.com,OU=Twitter Security,O="Twitter, Inc.",L=San Francisco,ST=California,C=US
    
  •   start date: Feb 02 00:00:00 2016 GMT
    
  •   expire date: Oct 01 12:00:00 2018 GMT
    
  •   common name: ton.twitter.com
    
  •   issuer: CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
    

GET /1.1/ton/data/ta_partner/3186774122/CYhbk7-4ovbSjgs HTTP/1.1
Host: ton.twitter.com
Accept: /
Content-Type: text/comma-separated-values
Content-Length: 0
X-TON-Expires: Tue, 10 Jan 2017 20:09:28 GMT
Authorization: OAuth …

< HTTP/1.1 200 OK
< accept-ranges: bytes
< content-length: 335
< content-md5: BP3xMKcvQ+pKXirr8TFhlg==
< content-type: application/octet-stream
< date: Thu, 05 Jan 2017 20:09:28 GMT
< etag: “BP3xMKcvQ+pKXirr8TFhlg==”
< expires: Thu, 12 Jan 2017 20:09:28 GMT
< last-modified: Thu, 05 Jan 2017 20:09:27 GMT
< server: tsa_b
< set-cookie: guest_id=v1%3A148364696898329027; Domain=.twitter.com; Path=/; Expires=Sat, 05-Jan-2019 20:09:28 UTC
< strict-transport-security: max-age=631138519
< surrogate-key: ta_partner
< x-connection-hash: b4025bf9ecbd0eee1f1fa102b76d889a
< x-content-type-options: nosniff
< x-rate-limit-limit: 50
< x-rate-limit-remaining: 42
< x-rate-limit-reset: 1483647473
< x-response-time: 12
< x-ton-expected-size: 335
< x-ton-expires: Tue, 10 Jan 2017 20:09:27 GMT
<


#19

That’s correct—I’m referring to when you make a request to POST /1.1/ton/bucket/ta_partner.

That’s great as it helps us eliminate that as a possible issue.


What do you see when you make a GET request using the location header value? What you should see is one hashed entity per line, as follows.

$ twurl -H ton.twitter.com "/1.1/ton/data/ta_partner/2417045708/gsc5WiuqPaXRSUA.txt"
c2bcd645a4baf0db743103c72231b1424b7601664d8ce44cacc614549f822a41
17fbc61536f4a0169924772b3d7eb83691978d83e3c2480e669e3b29e8749961
...
e32f5b173a82318a67a3bd6a5932109dacdc43b4b97614c73f46abab86f80873

Because we expect one hashed value per line.

If, instead, the response contains newline characters, it’s not correct.

# indicates that the data was not sent correctly
c2bcd645a4baf0db743103c72231b1424b7601664d8ce44cacc614549f822a41\n17fbc61536f4a0169924772b3d7eb83691978d83e3c2480e669e3b29e8749961\n...\ne32f5b173a82318a67a3bd6a5932109dacdc43b4b97614c73f46abab86f80873\n

Uploading the audience list

Let’s say we’re just trying to send three hashed entities.

In the request

# incorrectly sends data
$ twurl -X POST -H ton.twitter.com "/1.1/ton/bucket/ta_partner" --header "Content-Type: text/plain" --header "X-TON-Expires: Sat, 14 Jan 2017 06:01:00 GMT" -d "c2bcd645a4baf0db743103c72231b1424b7601664d8ce44cacc614549f822a41\n17fbc61536f4a0169924772b3d7eb83691978d83e3c2480e669e3b29e8749961\ne32f5b173a82318a67a3bd6a5932109dacdc43b4b97614c73f46abab86f80873\n"

This request mirrors the one in the Python example, above. The file data that we’re sending in that example includes the newline characters.

However, while this is a perfectly reasonable way to construct the request at the command line, the data will not be sent correctly (when using bash). The reason is that, “A Posix standard shell does not interpret C escapes.”

$ echo "hello\nworld"
hello\nworld

See this Stack Overflow post for more details.

As the linked post explains, there is a way to have bash recognize the newline characters: using the $'...' quotation form.

# correctly sends data
twurl -X POST -H ton.twitter.com "/1.1/ton/bucket/ta_partner" --header "Content-Type: text/plain" --header "X-TON-Expires: Sat, 14 Jan 2017 06:01:00 GMT" -d $'c2bcd645a4baf0db743103c72231b1424b7601664d8ce44cacc614549f822a41\n17fbc61536f4a0169924772b3d7eb83691978d83e3c2480e669e3b29e8749961\ne32f5b173a82318a67a3bd6a5932109dacdc43b4b97614c73f46abab86f80873\n'

From a file

We can do better and directly send the data from a file.

# file contents
$ cat hashed_user_ids.txt
c2bcd645a4baf0db743103c72231b1424b7601664d8ce44cacc614549f822a41
17fbc61536f4a0169924772b3d7eb83691978d83e3c2480e669e3b29e8749961
...
e32f5b173a82318a67a3bd6a5932109dacdc43b4b97614c73f46abab86f80873

(Note: no actual ellipses, just one hashed user ID per line.)

We’ll use a subshell to get the file contents: "$(<hashed_user_ids.txt)".

$ twurl -t -X POST -H ton.twitter.com "/1.1/ton/bucket/ta_partner" --header "Content-Type: text/plain" --header "X-TON-Expires: Sat, 14 Jan 2017 06:01:00 GMT" -d "$(<hashed_user_ids.txt)"
opening connection to ton.twitter.com:443...
opened
starting SSL for ton.twitter.com:443...
SSL established
<- "POST /1.1/ton/bucket/ta_partner HTTP/1.1\r\nContent-Type: text/plain\r\nX-Ton-Expires: Sat, 14 Jan 2017 06:01:00 GMT\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nAccept: */*\r\nUser-Agent: OAuth gem v0.5.1\r\nAuthorization: OAuth oauth_body_hash=\"Zu1XQVum1avNt7OuWd%2F3U7qDfkI%3D\", oauth_consumer_key=\"...\", oauth_nonce=\"ZXLIrFsPX89nSY8IWTSry3IKOmegeZkhnDyHAN7Ss\", oauth_signature=\"s6IK0HKKDPfUh3jABYfYZhicD6g%3D\", oauth_signature_method=\"HMAC-SHA1\", oauth_timestamp=\"1483863050\", oauth_token=\"...\", oauth_version=\"1.0\"\r\nConnection: close\r\nHost: ton.twitter.com\r\nContent-Length: 6499\r\n\r\n"
<- "c2bcd645a4baf0db743103c72231b1424b7601664d8ce44cacc614549f822a41\n17fbc61536f4a0169924772b3d7eb83691978d83e3c2480e669e3b29e8749961\ne32f5b173a82318a67a3bd6a5932109dacdc43b4b97614c73f46abab86f80873"
-> "HTTP/1.1 201 Created\r\n"
-> "connection: close\r\n"
-> "content-length: 0\r\n"
-> "content-type: text/plain\r\n"
-> "date: Sun, 08 Jan 2017 08:10:50 GMT\r\n"
-> "location: /1.1/ton/data/ta_partner/2417045708/gsc5WiuqPaXRSUA.txt\r\n"
-> "server: tsa_b\r\n"
-> "set-cookie: guest_id=v1%3A148386305067403704; Domain=.twitter.com; Path=/; Expires=Tue, 08-Jan-2019 08:10:50 UTC\r\n"
-> "strict-transport-security: max-age=631138519\r\n"
-> "x-connection-hash: e0724f963fa2564e3964b3f8ab00b530\r\n"
-> "x-content-type-options: nosniff\r\n"
-> "x-rate-limit-limit: 90000\r\n"
-> "x-rate-limit-remaining: 89999\r\n"
-> "x-rate-limit-reset: 1483863950\r\n"
-> "x-response-time: 30\r\n"
-> "x-tsa-request-body-time: 72\r\n"
-> "\r\n"
reading 0 bytes...
-> ""
read 0 bytes
Conn close

You’ll then use /1.1/ton/data/ta_partner/2417045708/gsc5WiuqPaXRSUA.txt as the input_file_path value in the subsequent request to the POST accounts/:account_id/tailored_audience_changes endpoint.


Tailored Audience files uploads failing - 411 length required
#20

So, it appears at some point a third-party package our app uses changed default behavior to enclose fields in a csv file with double quotes. After correcting this, I was able to create audience, add contacts from an uploaded file, and have the audience enter ready state and show the expected number of matches upon completion of processing. Thanks for all your help!