FINALIZE step in media chunk upload giving 400 error message?

[Adi_t2021]

I am trying to upload image or video using chunk uploading steps step 1 and 2 i.e INIT and APPEND works fine (I also checked the logs) but in FINALIZE step getting 400 bad request. Can someone please tell me what I am missing or doing something wrong?

STEP1: INIT

const splitBuffer = require('../splitBuffer');
const fetch = require('cross-fetch');
const crypto = require('crypto');
const OAuth = require('oauth-1.0a');

const oauth = OAuth({
    consumer: {
        key: process.env.TWITTER_CONSUMER_KEY,
        secret: process.env.TWITTER_CONSUMER_SECRET
    },
    signature_method: 'HMAC-SHA1',
    hash_function: (baseString, key) => crypto.createHmac('sha1', key).update(baseString).digest('base64')
});

const token = {
    key: '',
    secret: ''
};

const initUpload = async (buffer) => {
    // Init
    const bufferLength = Buffer.byteLength(buffer);

    const data = { 'command': 'INIT', 'total_bytes': bufferLength, 'media_type': 'image/png' };
    const requestBody = {
        url: `https://upload.twitter.com/1.1/media/upload.json`,
        method: 'POST',
        data: data
    }

    const authHeader = oauth.toHeader(oauth.authorize(requestBody, token));

    const body = percentEncode(querystring.stringify(data));

    const result = await fetch(`https://upload.twitter.com/1.1/media/upload.json`, {
        method: 'POST',
        headers: {
            Authorization: authHeader["Authorization"],
            'Content-Type': 'application/x-www-form-urlencoded',
            'Accept': 'application/json'
        },
        body: body,
    });

    return result.json();
}

STEP 2: APPEND

const appendMedia = async (buffer, chunkSize, media_id_string) => {
    const parts = splitBuffer(buffer, chunkSize);

    const uploads = parts.map(async (part, index) => {
        const data = { 'command': 'APPEND', 'media_id': media_id_string, 'segment_index': index, media_data: part }
        const requestBody = {
            url: `https://upload.twitter.com/1.1/media/upload.json`,
            method: 'POST',
            data: data
        }

        const authHeader = oauth.toHeader(oauth.authorize(requestBody, token));

        const body = percentEncode(querystring.stringify(data));

        const responseData = await fetch(`https://upload.twitter.com/1.1/media/upload.json`, {
            method: 'POST',
            headers: {
                Authorization: authHeader["Authorization"],
                'Content-Type': 'application/x-www-form-urlencoded',
                'Accept': 'application/json'
            },
            body: body,
        });

        return responseData;
    });

    const finalResult = await Promise.all(uploads).then(() => media_id_string);
    return finalResult;
}

STEP 3: FINALIZE

const finalize = async (media_id_string) => {
    const data = { 'command': 'FINALIZE', 'media_id': media_id_string }
    const requestBody = {
        url: `https://upload.twitter.com/1.1/media/upload.json`,
        method: 'POST',
        data: data
    }

    const authHeader = oauth.toHeader(oauth.authorize(requestBody, token));

    const body = percentEncode(querystring.stringify(data));

    const result = await fetch(`https://upload.twitter.com/1.1/media/upload.json`, {
        method: 'POST',
        headers: {
            Authorization: authHeader["Authorization"],
            'Content-Type': 'application/x-www-form-urlencoded',
            'Accept': 'application/json'
        },
        body: body
    });

    const answer = await result.json();
}

Routes:

router.post('/twitter/media', async (req, res) => {
    try {
        const image = await axios.get('https://ik.imagekit.io/XXXXXXXXX/test-upload_9OP0H7LEl.png', { responseType: 'arraybuffer' });
        const base64Image = new Buffer.from(image.data).toString('base64');

        const result = await initUpload(base64Image);
        const { media_id_string } = result;

        console.log("init media_id_string", media_id_string);

        const result2 = await appendMedia(base64Image, 1024, media_id_string);

        const response = await finalize(media_id_string);
        console.log("response", response);

        res.status(200).send({ status: "ok" });
    } catch (error) {
        console.log("error", error)
        res.status(403).json({ message: "Missing, invalid, or expired tokens" });
    }
});

I referred few stackoverflow posts and twitter community answers but none of them is working for me. I tried to increase/decrease file chunk size. I also tried to use multipart/form-data instead of application/x-www-form-urlencoded for step 3 FINALIZE but that is also not working. Image which I am using is 1.6mb size. Please suggest what I am missing. Thanks.
Note: In docs one thing is mentioned:

Because the method uses multipart POST, OAuth is handled differently. POST or query string parameters are not used when calculating an OAuth signature basestring or signature. Only the oauth_* parameters are used``` am I facing issue because of this ??

[IgorBrigadir]

Does uploading on the web work for the same file?

How about this python example? work on the same media? GitHub - twitterdev/large-video-upload-python: Sample Python code for uploading video up to 140 seconds and/or up to 512Mb.

[Adi_t2021]

Solution 1:
I was able to solve this video chunk upload using this nodejs script mentioned on StackOverflow: javascript - How do you upload a chunked video to twitter using node - Stack Overflow

[andypiper]

Did you figure out the specific difference to your code, or did you just use the code from StackOverflow directly? Would be good to know what the issue was here.

[gauloics]

quote

@Adi_t2021

I am trying to upload image or video using chunk uploading steps step 1 and 2 i.e INIT and APPEND works fine (I also checked the logs) but in FINALIZE step getting 400 bad request. Can someone please tell me what I am missing or doing something wrong?

STEP1: INIT

const splitBuffer = require('../splitBuffer');
const fetch = require('cross-fetch');
const crypto = require('crypto');
const OAuth = require('oauth-1.0a');

const oauth = OAuth({
    consumer: {
        key: process.env.TWITTER_CONSUMER_KEY,
        secret: process.env.TWITTER_CONSUMER_SECRET
    },
    signature_method: 'HMAC-SHA1',
    hash_function: (baseString, key) => crypto.createHmac('sha1', key).update(baseString).digest('base64')
});

const token = {
    key: '',
    secret: ''
};

const initUpload = async (buffer) => {
    // Init
    const bufferLength = Buffer.byteLength(buffer);

    const data = { 'command': 'INIT', 'total_bytes': bufferLength, 'media_type': 'image/png' };
    const requestBody = {
        url: `https://upload.twitter.com/1.1/media/upload.json`,
        method: 'POST',
        data: data
    }

    const authHeader = oauth.toHeader(oauth.authorize(requestBody, token));

    const body = percentEncode(querystring.stringify(data));

    const result = await fetch(`https://upload.twitter.com/1.1/media/upload.json`, {
        method: 'POST',
        headers: {
            Authorization: authHeader["Authorization"],
            'Content-Type': 'application/x-www-form-urlencoded',
            'Accept': 'application/json'
        },
        body: body,
    });

    return result.json();
}

STEP 2: APPEND

const appendMedia = async (buffer, chunkSize, media_id_string) => {
    const parts = splitBuffer(buffer, chunkSize);

    const uploads = parts.map(async (part, index) => {
        const data = { 'command': 'APPEND', 'media_id': media_id_string, 'segment_index': index, media_data: part }
        const requestBody = {
            url: `https://upload.twitter.com/1.1/media/upload.json`,
            method: 'POST',
            data: data
        }

        const authHeader = oauth.toHeader(oauth.authorize(requestBody, token));

        const body = percentEncode(querystring.stringify(data));

        const responseData = await fetch(`https://upload.twitter.com/1.1/media/upload.json`, {
            method: 'POST',
            headers: {
                Authorization: authHeader["Authorization"],
                'Content-Type': 'application/x-www-form-urlencoded',
                'Accept': 'application/json'
            },
            body: body,
        });

        return responseData;
    });

    const finalResult = await Promise.all(uploads).then(() => media_id_string);
    return finalResult;
}

STEP 3: FINALIZE

const finalize = async (media_id_string) => {
    const data = { 'command': 'FINALIZE', 'media_id': media_id_string }
    const requestBody = {
        url: `https://upload.twitter.com/1.1/media/upload.json`,
        method: 'POST',
        data: data
    }

    const authHeader = oauth.toHeader(oauth.authorize(requestBody, token));

    const body = percentEncode(querystring.stringify(data));

    const result = await fetch(`https://upload.twitter.com/1.1/media/upload.json`, {
        method: 'POST',
        headers: {
            Authorization: authHeader["Authorization"],
            'Content-Type': 'application/x-www-form-urlencoded',
            'Accept': 'application/json'
        },
        body: body
    });

    const answer = await result.json();
}

Routes:

router.post('/twitter/media', async (req, res) => {
    try {
        const image = await axios.get('https://ik.imagekit.io/XXXXXXXXX/test-upload_9OP0H7LEl.png', { responseType: 'arraybuffer' });
        const base64Image = new Buffer.from(image.data).toString('base64');

        const result = await initUpload(base64Image);
        const { media_id_string } = result;

        console.log("init media_id_string", media_id_string);

        const result2 = await appendMedia(base64Image, 1024, media_id_string);

        const response = await finalize(media_id_string);
        console.log("response", response);

        res.status(200).send({ status: "ok" });
    } catch (error) {
        console.log("error", error)
        res.status(403).json({ message: "Missing, invalid, or expired tokens" });
    }
});

I referred few stackoverflow posts and twitter community answers but none of them is working for me. I tried to increase/decrease file chunk size. I also tried to use multipart/form-data instead of application/x-www-form-urlencoded for step 3 FINALIZE but that is also not working. Image which I am using is 1.6mb size. Please suggest what I am missing. Thanks.
Note: In docs one thing is mentioned:

Because the method uses multipart POST, OAuth is handled differently. POST or query string parameters are not used when calculating an OAuth signature basestring or signature. Only the oauth_* parameters are used``` am I facing issue because of this ??

Try use this library https://github.com/PLhery/node-twitter-api-v2/blob/master/doc/v1.md#upload-medias

[Adi_t2021]

Thanks I will try it out and will let you know

[Adi_t2021]

Hi @andypiper I have one question related to twitter media chunk upload API. I tried to download image or video from some url that url can be s3 or imgur or imagekit https://imagekit.io/
In 99% cases I have a imagekit url for image or video. Now if I convert it into buffer and encode which can be used for chunk media upload then it does not work.
code snippet for downloading image or video from url:

const downloadImageFromUrl = async () => {
    const video = await axios.get(`https://i.imgur.com/rfYwI5n.mp4`, { responseType: 'arraybuffer' });
    const buffer = Buffer.from(video.data, 'binary').toString('base64');
    return buffer;
}

Now as above code was not working. I tried to download image or video from same url using this code snippet:

const https = require('https');
const fs = require('fs');

function saveImageToDisk(url, path) {
    const localpath = fs.createWriteStream(path);

    const response = https.get(url, function (res) {
        res.pipe(localpath);
    });

    console.log("response", response);
}

saveImageToDisk('https://i.imgur.com/rfYwI5n.mp4', "hello" + ".mp4")

console.log("done")

Now once image is downloaded and stored on disk I then read that same file and use it in media chunk upload API endpoint. This time entire flow works fine i.e INIT, APPEND and FINALIZE works fine.
code snippet for reading file:
require('fs').readFileSync(mediaPath, { encoding: 'base64' })

In short I think that after downloading and uploading image to twitter does not work. It only works when uploading a local file. Can you please suggest if I am doing anything wrong in my code? See these messages for reference:

1. Unable to use chucked media upload / mediaUploadAppend: "Could not authenticate you" · Issue #63 · FeedHive/twitter-api-client · GitHub
2. Unable to use chucked media upload / mediaUploadAppend: "Could not authenticate you" · Issue #63 · FeedHive/twitter-api-client · GitHub
   Is this issue because of twitter does not support that file or am I doing something wrong with nodejs file buffer? I have limited knowledge of node file buffers. Can you please let me know if this issue is specific to twitter APIs. Thanks

[andypiper]

I am not sure and I’ve not tested this myself, but it does sound like something might be changing the file in the download process. If you are able to download the file via e.g. a browser, and upload with your code, and it works, then that would suggest there is something happening to change the file if you download the file with code.