Uploading data to TON API fails using PHP cURL

sdk

#1

Hey guys,

I am helping develop a twitter ads SDK for the PHP community and I’m stuck on the TON API for creating both single chunk uploads, and multi-chunk uploads. If I make the request using TWURL like so:

twurl -t -H ton.twitter.com /1.1/ton/bucket/ta_partner -X POST -A “Content-Type: text/plain”
-A “X-TON-Expires: Sat, 02 Jul 2016 17:19:02 GMT” -A “Content-Length: 12” --data "aaa@test.com"

The request returns a successful response as expected.

However this EXACT request sent over cURL in PHP fails…

    $headers = [
        'Authorization: OAuth oauth_version="1.0", oauth_nonce="xx", oauth_timestamp="1467394021",     oauth_consumer_key="xx", oauth_token="xx", oauth_signature_method="HMAC-SHA1", oauth_signature="xx"',
        'Content-Type: text/plain',
        'Content-Length: 12',
    ];
    $postfields = ['aaa@test.com'];
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_VERBOSE, true);
    curl_setopt($ch, CURLOPT_URL, 'https://ton.twitter.com/1.1/ton/bucket/ta_partner');
    curl_setopt($ch, CURLOPT_HEADER, true);
    curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_POST, 1);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $postfields);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 1);
    $result = curl_exec($ch);

with:

POST /1.1/ton/bucket/ta_partner HTTP/1.1
Host: ton.twitter.com
Accept: /
Content-Length: 12
Authorization: OAuth oauth_version=“1.0”, oauth_nonce=“xx”, oauth_timestamp=“1467397328”, oauth_consumer_key=“xx”, oauth_token=“xx”, oauth_signature_method=“HMAC-SHA1”, oauth_signature="xx"
Expect: 100-continue
Content-Type: text/plain; boundary=------------------------b222ecc0fec7f864

< HTTP/1.1 100 Continue
< HTTP/1.1 403 Forbidden
< content-length: 0
< date: Fri, 01 Jul 2016 18:22:07 GMT

  • Server tsa_a is not blacklisted
    < server: tsa_a
    < set-cookie: guest_id=v1%3A146739732730564146; Domain=.twitter.com; Path=/; Expires=Sun, 01-Jul-2018 18:22:07 UTC
    < strict-transport-security: max-age=631138519
    < x-connection-hash: 21da24649d20d073f1f0b8a879a2c786
    < x-content-type-options: nosniff
    < x-response-time: 7
    < x-tsa-request-body-time: 44
  • HTTP error before end of send, stop sending
    <
  • Closing connection 0

#2

Could someone help us here?

is oauth_body_hash necessary?


#3

@JBabichJapan @majoritasdev

I don’t know if some of you could help us. @Blackburn29 is helping me on the library, trying to implement TA upload, I’ve tryed on my own last 2 days and I always get an 403 error.

I ask @JBabichJapan because is part of Twitter Staff and @majoritasdev because is a PHP developer and he acomplished this with Zend Framework Module.

I’ve read a lot of post in the forum, related to Audience Upload with PHP and most people had the same problem.

I’ve followed this post Ads Targeting - upload CSV file to TON bucket and tryed to put X-TON-Expires as date('r', strtotime('+6 day')) as @majoritasdev remarked.

I know is not a credentials issue, since if I change the endpoint everything works perfect.

I’ve tryed a lot of things->

Accept: / - Accept: application/json

my txt file is a file with one email "aaa@test.com"

I send the request to cURL with Content-Type: text/plain.

I’ve tryed with hashed and non hashed data.

curl_setopt($ch, CURLOPT_POSTFIELDS, "aaa@text.com";
curl_setopt($ch, CURLOPT_POSTFIELDS, hash('sha256',trim("aaa@text.com"); And a lot of different variants.

Could you help us?

@majoritasdev could you share a cURL verbose output of a TA upload (to TON host) with a fake email like I’ve sent before.

Thanks!


#4

Hi @hector_borras! The payload should be sent as the raw body of the request, not as a POST field.


#5

Yeah the first thing that came up when I looked at the first post is that X-Ton-Expires is missing, here is a summary of the headers you need to send for initial request of multi part one:

X-TON-Content-Type with value text/plain
X-TON-Content-Length, the size (in bytes) of the file to be uploaded
X-TON-Expires, a HTTP-date (RFC 1123 [8]-date) when the uploaded segment file should expire; we recommend 7 days from the current time
Content-Length with value 0
Authorization Header

Generally we point people to https://github.com/twitterdev/ton-upload/blob/master/ton_upload as a reference implementation but newer one is: https://github.com/twitterdev/twitter-ruby-ads-sdk/blob/master/lib/twitter-ads/http/ton_upload.rb

Another common issue is for the oauth_signature or even oauth_nonce to be invalid. We’ve heard of some developers struggling to develop support for this because OAuth library not seeming to be compatible with what our API expects there.

The ton_api permission is a separate one from Ads API so if it’s a “true” permissions issue then you don’t have access to ta_partner bucket (but I wouldn’t expect you to be able to use twurl in that case…).


#6

Thanks @majoritasdev,

Buy how could I do that?

Everywhere I search about how to send the data with PHP cURL says that I should use:
curl_setopt($ch, CURLOPT_POSTFIELDS, $rawData) ;

Thanks


#7

Thanks @JBabichJapan , First of all I’m trying to upload a TA with only one record, not using the chunked upload (I’ll cover it when I acomplish to upload a simple file)

I’m following this because I feel more comfortable with python:

But in the end, the Request object send data with a body parameter and after it, the lib sends it as
data = self.options.get('body', None)

How can I check if I have the correct permissions for this endpoint/api?

Thanks!


#8

@hector_borras: try this: http://programmers.stackexchange.com/questions/199580/how-can-i-send-data-without-postfields-in-php-curl#199586


#9

I’ll try, thanks!


#10

I’ve readed that but reading this code:
curl_setopt($ch, CURLOPT_HTTPHEADER, array( 'Content-Type: application/json', 'Content-Length: ' . strlen($data_string)) );

and this one:
`
$data = array(“name” => “Hagrid”, “age” => “36”);
$data_string = json_encode($data);

$ch = curl_init(‘http://api.local/rest/users’);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, “POST”);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data_string);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
‘Content-Type: application/json’,
'Content-Length: ’ . strlen($data_string))
);

$result = curl_exec($ch);
`

It’s more or less what I’m doing, sending it as:

curl_setopt($ch, CURLOPT_POSTFIELDS, $data_string);

But, wait, maybe the thing is replace:

curl_setopt($ch, CURLOPT_POST, 1); --> curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");

Is it a big difference?

Thanks!


#11

I don’t know, maybe it does. Try to see if it makes any difference.


#12

@hector_borras The code on my branch on Git is based on the ruby script that @JBabichJapan posted above. Still having the same 403 errors. I think this has to do with the postfields in cURL being formatted incorrectly


#13

I’ve used the example post above to initalize a multi-chunk upload for a 31MB file of sha256 hashed emails with the following headers:

        [
            'X-TON-Content-Type'   => 'text/plain',
            'X-TON-Content-Length' => 32500000,
            'X-TON-Expires'        => 'Tue, 12 Jul 2016 15:26:11 GMT',
            'Content-Length'       => 0,
            'Content-Type'         => 'text'plain',
        ]

and I still get a 403 forbidden error. I dont see this to be an authentication error either, as my PHP and TWURL requests are sending the exact same Authorization: header (except twurl has the oauth_body_hash)

Twurl (Working, getting 201 Created):

Authorization:
OAuth oauth_body_hash=“ZmqMusMfLNliIT45yUqjwqfwUQ0%3D”,
oauth_consumer_key=“xx”,
oauth_nonce=“qrLoitjRvHgEwdvZi40Xo2CLImOPXXsYzICB4gi3k8”,
oauth_signature=“xx”,
oauth_signature_method=“HMAC-SHA1”,
oauth_timestamp=“1467732784”,
oauth_token=“xx”,
oauth_version=“1.0”

PHP: (Failing, 403 Forbidden)

Authorization:
oauth_consumer_key=“xx”,
oauth_nonce=“13d4b96a5e21615ec1b9d4877051b220”,
oauth_signature=xx"
oauth_signature_method=“HMAC-SHA1”,
oauth_timestamp=“1467732371”,
oauth_token=“xx”,
OAuth oauth_version=“1.0”,


#14

I don’t know what could be happening.

I’ve downloaded twitterads-python-sdk and debuged the creation of a TA (it works and I can see the audience in the dashboard).

Those are the headers:

`
POST /1.1/ton/bucket/ta_partner HTTP/1.1
Host: ton.twitter.com
Content-Length: 27
Accept-Encoding: gzip, deflate
x-ton-expires: Fri, 15 Jul 2016 18:24:14 GMT
Accept: /
user-agent: twitter-ads version: 1.1.0 platform: Python 2.7 (CPython/linux2)
Connection: keep-alive
content-type: text/plain
Authorization: OAuth oauth_nonce=“55603487433575853881467743056”, oauth_timestamp=“1467743056”, oauth_version=“1.0”, oauth_signature_method=“HMAC-SHA1”, oauth_consumer_key=“CONSUMER”, oauth_token=“TOKEN”, oauth_signature=“SIGNATURE”

hborrasaleixandre@gmail.com
`

Headers on my request:

POST /1.1/ton/bucket/ta_partner HTTP/1.1 Host: ton.twitter.com User-Agent: TwitterAds SDK v0.1.0 Accept-Encoding: gzip, deflate Accept: */* Authorization: OAuth oauth_version="1.0", oauth_nonce="494db26331f46a46dfd73211d5ef5726", oauth_timestamp="1467745832", oauth_consumer_key="CONSUMER", oauth_token="TOKEN", oauth_signature_method="HMAC-SHA1", oauth_signature="SIGNATURE" x-ton-expires: Fri, 15 Jul 2016 19:10:32 GMT content-type: text/plain Connection: keep-alive Content-Length: 27

Any guess?


#15

The only thing I can think of this something in the way the cURL works through PHP and that the data is not being sent over in the body correctly. Other than that, I’m out of ideas @hector_borras


#16

Hi folks,

I checked out the php-sdk branch with ton_upload support and tried to debug this, as far as I can tell from diffing with successful run with our Ruby script the major difference is that PHP SDK is not outputting the oauth_body_hash field. Especially because the post contains a payload I wouldn’t be surprised to see that this is required to make sure the payload hasn’t been tampered with.

I would give that a look, I can find some articles mentioning this related to PHP libraries:

Let me know if this solves the issue!

Thanks,

John


#17

Thank you ver much @JBabichJapan for your effort.

Searching about an alternative to cURL, I’ve found pecl_http, with this article:

looks similar to your post.

This afternoon (I’m in spain) I’ll check it, hopefully it will work!

Thanks again!


#18

@hector_borras you should look into

http://docs.guzzlephp.org/en/latest/overview.html

This is a great library. I use it all the time


#19

I’ll try with this one too!!

I’ll be really happy if I can finish this today!


#20

OMG!

EUREKA!!!

PS:

Well, after the euphoria,

The problem was the oauth_body_hash, for cURL is a required parameter. You need to encode base64 the result of encode in sha1 the content.

'oauth_body_hash' => base64_encode(sha1($content));

After that, it works. I’ll see when I’ll need to split the content in different chunks.

Thanks to everybody!!!


Data processing differently via API vs. direct upload to audience