/media/upload rejecting mp4 videos

[Keatran]

So I’m using https://github.com/abraham/twitteroauth to connect to twitter and this works for uploading images but whenever I try and upload a video it fails no matter what I try.

Test Code:

$s3Path = $imagePath = "https://s3-eu-west-1.amazonaws.com/---OMITED---".($media->path);
$fileExt = explode(".", $media->path);
$fileExt = end($fileExt);

$tmpPath = sys_get_temp_dir()."/".sha1(time())."_".microtime().".{$fileExt}");
$fh = fopen($tmpPath, "w");
fwrite($fh, $s3->download($media->path));
fclose($fh);

$response = $this->getConnection()->upload("media/upload", array("media" => $tmpPath), true);
//unlink($tmpPath);
error_log(json_encode($response));
error_log("Trying to upload {$tmpPath}");
return $response->media_id;

Debugging:

[Tue Apr 10 18:25:32.413313 2018] [php7:notice] [pid 14880] [client 172.31.21.62:49071] {"request":"\\/1.1\\/media\\/upload.json","error":"media type unrecognized."}, referer: http://localhost:3000/posts
[Tue Apr 10 18:25:32.413456 2018] [php7:notice] [pid 14880] [client 172.31.21.62:49071] Trying to upload /tmp/a818f4f38b6a529d8a95cc6afd6f10cd7900ed3a_8693c50f3748a490698d0d9941e32daed8016478.mp4, referer: http://localhost:3000/posts
[Tue Apr 10 18:25:32.632256 2018] [php7:notice] [pid 14880] [client 172.31.21.62:49071] stdClass Object\n(\n    [status] => failure\n    [error] => Array\n        (\n            [code] => 187\n            [message] => Status is a duplicate.\n            [connected] => 1\n        )\n\n)\n, referer: http://localhost:3000/posts

I have made sure the video is using the format supplied in the last response to Unable to upload .MP4 file,

The video uploads to twitter fine when I use the twitter.com’s post creator. but every time I try and upload it to Twitter using the 1.1/media/upload endpoint’s I always get back

{"request":"/1.1/media/upload.json","error":"media type unrecognized."}

The funny thing is the exact same code is used for uploading images and it works perfectly.

I’m not sure what else to try or if this is a bug in twitters rules on media uploads. via the API?

[bobber205]

I think they made non mp4 videos a policy awhile back

[Keatran]

ok do you know what format do they do want / accept , as MPEG video with AAC audio encoding is supposed to be an MP4? and why does this page specificly state video/mp4 in there documentation?
https://developer.twitter.com/en/docs/media/upload-media/uploading-media/chunked-media-upload

[andypiper]

Are you able to provide a link to an example file that you’re attempting to upload via the API? It’s pretty hard to tell what might be wrong otherwise.

My debugging approach is usually to try the Python large video upload sample, and also to compare with the expected specifications having used a tool like mp4box to examine the file.

[Keatran]

Here is a video i have been tested with.

https://s3-eu-west-1.amazonaws.com/beacon.public-upload/website_4/54d99d711d1e019a00da83af0216612f048b7653.mp4

So now i have tested this using the media uploader built in Python and it work’s so i thought the problem is with the Abraham connector so i went over to the issues on that project to submit a bug report. https://github.com/abraham/twitteroauth/issues/646

So following on from this i have been debugging the hell out of the Abrahams Twitter oAuth connector and found out there is most deffinatly a problem with Twiters media API, however i dont understand why this would work when using the python uploader… so unless python is doing something wrong with binary data it makes no sense. i don’t know how python works very well as i rearly use but understand its syntax enough to understand what that uploader is doing.

When calling Status with a loads of debugging:

 private function uploadMediaChunked($path, array $parameters)
    {
        $initParams = $this->mediaInitParameters($parameters);
        $init = $this->http('POST', self::UPLOAD_HOST, $path, $initParams);
        // Append
        $segmentIndex = 0;
        $media = fopen($parameters['media'], 'rb');
        $totalSegmants = ceil($initParams['total_bytes']/$this->chunkSize);
        $totalSent = 0;
        while (!feof($media))
        {
            $bin = fread($media, $this->chunkSize);
            error_log("segment ".($segmentIndex+1)."/{$totalSegmants} is of size ".strlen($bin)." bytes ");
            $this->http('POST', self::UPLOAD_HOST, 'media/upload', [
                'command' => 'APPEND',
                'media_id' => $init->media_id_string,
                'segment_index' => $segmentIndex++,
                'media' => $bin
            ]);
            $totalSent += strlen($bin);
            error_log("sent {$totalSent}/{$initParams['total_bytes']}");
        }
        fclose($media);
        // Finalize
        $finalize = $this->http('POST', self::UPLOAD_HOST, 'media/upload', [
            'command' => 'FINALIZE',
            'media_id' => $init->media_id_string
        ]);

        //check media is read
        while(true){
                error_log("sending STATUS requet for {$init->media_id_string}");
                $data = $this->http('POST', self::UPLOAD_HOST, 'media/upload', [
                        'command' => 'STATUS',
                        'media_id' => $init->media_id_string,
                ]);
                error_log("state = ".json_encode($data));
                if($data->processing_info->state != 'in_progress'){
                        if($data->processing_info->state == "failed"){
                                error_log("failed to upload media to twitter {$data->processing_info->error->message}");
                                throw new \Exception("Upload media to twitter failed \r\n {$data->processing_info->error->message}");
                        }
                        break;
                }
                if(isset($data->processing_info)){
                        sleep($data->processing_info->check_after_secs);
                }
        }
        return $finalize;
    }

The function above was previously using 'media_data' => json_encode(fread($media, $this->chunkSize)); and did not work i changed it to use binary data as im on a PHP7 server so binary data is fully supported so less memory & CPU usage as im not json_encoding anymore.

I get

[Wed Apr 18 11:57:35.045339 2018] sending STATUS requet for 986574402333573122, referer: *OMITED*
[Wed Apr 18 11:57:35.247701 2018] state = {"errors":[{"code":38,"message":"media parameter is missing."}]}, referer: *OMITED*

And according to https://developer.twitter.com/en/docs/media/upload-media/api-reference/get-media-upload-status i should not be sending media and i have no binary data left to send so telling me to send media, makes no sense i dont under stand why the Media API on a status request is sending this as a response.

This appears to be triggered by another problem that im not 100% certain on again seems to be the media API getting it wrong

this is what php output when trying to upload the file: size is 1446304 from the filesize function

[Wed Apr 18 12:34:02.996705 2018] File /tmp/f54897f638f4614f6232fb4ec35dece8479813e7_d470686180bd7bb6a4b8591b364dd7f73b14caf9.mp4 size is 1446304 bytes, referer: *OMITTED*

logs of the upload as it is chunked size is in bytes

[Wed Apr 18 12:34:03.234387 2018] segment 1/6 is of size 250000 bytes , referer: *OMITTED*
[Wed Apr 18 12:34:05.419806 2018] sent 250000/1446304, referer: *OMITTED*
[Wed Apr 18 12:34:05.421065 2018] segment 2/6 is of size 250000 bytes , referer: *OMITTED*
[Wed Apr 18 12:34:07.655178 2018] sent 500000/1446304, referer: *OMITTED*
[Wed Apr 18 12:34:07.655411 2018] segment 3/6 is of size 250000 bytes , referer: *OMITTED*
[Wed Apr 18 12:34:09.794782 2018] sent 750000/1446304, referer: *OMITTED*
[Wed Apr 18 12:34:09.795031 2018] segment 4/6 is of size 250000 bytes , referer: *OMITTED*
[Wed Apr 18 12:34:12.012933 2018] sent 1000000/1446304, referer: *OMITTED*
[Wed Apr 18 12:34:12.013173 2018] segment 5/6 is of size 250000 bytes , referer: *OMITTED*
[Wed Apr 18 12:34:14.195218 2018] sent 1250000/1446304, referer: *OMITTED*
[Wed Apr 18 12:34:14.195435 2018] segment 6/6 is of size 196304 bytes , referer: *OMITTED*
[Wed Apr 18 12:34:15.930540 2018] sent 1446304/1446304, referer: *OMITTED*
[Wed Apr 18 12:34:16.201088 2018] sending STATUS requet for 986583635326324737, referer: *OMITTED*
[Wed Apr 18 12:34:16.405301 2018] state = {"errors":[{"code":38,"message":"media parameter is missing."}]}, referer: *OMITTED*
[Wed Apr 18 12:34:16.405453 2018] {"request":"\\/1.1\\/media\\/upload.json","error":"Segments do not add up to provided total file size."}, referer: *OMITTED*

So from the above logs i did the maths,

   250,000 * 5  = 1,250,000
 + 196,304
                = 1,446,304 

And just to verify on windows before uploaded the video it says the size is: 1,446,304 bytes.

So why do i get a response of: {"request":"\\/1.1\\/media\\/upload.json","error":"Segments do not add up to provided total file size."} quite clearly they add up when im sending them to twitter…

[Keatran]

Sorry for the double post but I wanted this issue to be separate as it is a separate issue but same overall system, this connector also has the ability to use the standard/media/upload

But this works like (and i mean works as i can use it to upload a small image)

private function uploadMediaNotChunked($path, array $parameters)
    {
        $file = file_get_contents($parameters['media']);
        $base = base64_encode($file);
        $parameters['media'] = $base;
        //$parameters['media'] = $file;
        return $this->http('POST', self::UPLOAD_HOST, $path, $parameters);
    }

As you can see the media parameter is holding base64 encoded data and not binary data yet the documentation linked above show this is wrong yet this method works and if i switch the commented code around to:

//$base = base64_encode($file);
//$parameters['media'] = $base;
$parameters['media'] = $file;

It fails so something is clearly wrong with this as it would say not only is the documentation is wrong in the parameter list, but also wrong in that Twitter does not support binary data as the doc say binary data should be media and media_data must be used for base64.