POST Upload chunked INIT command gets media parameter is missing error

[ePirat]

Sorry if my advice sounded unfriendly, it was not intended to be that way! Unfortunately I can’t provide you with Java Code examples but I can clarify how exactly you need to encode data.

INIT

The init command can be sent with a Content-Type of application/x-www-form-urlencoded which looks like key=value&key2=value2

APPEND

The append command has to be sent with a Content-Type of multipart/form-data and encoded as that properly, in twurl it looks like it’s actually application/x-www-form-urlencoded but that is only how the data is given on command line, it will not use that for the actual encoding.
There are quite a few posts about that on Stackoverflow about it, this for example.
If you really want to implement it yourself, have a look at RFC2388 which describes how it works.
If you have further questions, feel free to ask.

FINALZE

For finalize you can use application/x-www-form-urlencoded again.

[andypiper]

I know the media API team is also keen to get our documentation into better shape so hopefully this will all improve over time.

[BinCodinLong]

Hi ePirat and andypiper if you’re around today.

I have un-methodized my code that composes the http message I send for the INIT.
Sorry it’s not pretty – I took out everything but the essentials. Anyway, here it is:

HttpParams httpParams = new BasicHttpParams();
HttpProtocolParams.setVersion(httpParams, HttpVersion.HTTP_1_1);
HttpProtocolParams.setContentCharset(httpParams, UTF_8);
HttpProtocolParams.setUserAgent(httpParams, HTTP_CORE_1_1);
HttpProtocolParams.setUseExpectContinue(httpParams, false);
httpParams.setParameter("command", "INIT")
          .setParameter("media_type", "image/jpeg")
          .setParameter("total_bytes", 6046 )
  
HttpHost httpHost =new HttpHost(UPLOAD_TWITTER_COM, 443);
DefaultHttpClientConnection httpClientConnection = new DefaultHttpClientConnection();

Socket socket = null;
// initialize the HTTPS connection
SSLContext sslcontext = SSLContext.getInstance("TLS");
sslcontext.init(null, null, null);
SSLSocketFactory ssf = sslcontext.getSocketFactory();
socket = ssf.createSocket();
socket.connect(new InetSocketAddress(httpHost.getHostName(), httpHost.getPort()), 0);
httpClientConnection.bind(socket, httpParams);
// see utilty method below
String authorization_header_string = getAuthorizationHeaderString(
            null, twitterConsumerKey, oauthNonce
            , oauthSignature, oauthTimestamp, encode(oauthTokenAccess), ONE_POINT_ZERO, null
    );
	
BasicHttpEntityEnclosingRequest httpEntityEnclosingRequest =
        new BasicHttpEntityEnclosingRequest( "POST", "https://upload.twitter.com/1.1/media/upload.json" );
httpEntityEnclosingRequest.setParams( httpParams );
httpEntityEnclosingRequest.addHeader("Authorization", authorization_header_string);

HttpContext httpContext = new BasicHttpContext(null);
httpContext.setAttribute(ExecutionContext.HTTP_CONNECTION, httpClientConnection);
httpContext.setAttribute(ExecutionContext.HTTP_TARGET_HOST, httpHost);

HttpProcessor httpProcessor = getHttpProcessor();
HttpRequestExecutor httpRequestExecutor = new HttpRequestExecutor();
httpRequestExecutor.preProcess(httpEntityEnclosingRequest, httpProcessor, httpContext);
HttpResponse httpResponse = httpRequestExecutor.execute(
        httpEntityEnclosingRequest, httpClientConnection, httpContext);
httpResponse.setParams(httpParams);
httpRequestExecutor.postProcess(httpResponse, httpProcessor, httpContext);

String responseStatusLine = httpResponse.getStatusLine().toString();
Log.d(Constants.TOURMAKER, TAG + " runTwitterUpdate responseStatusLine = " + responseStatusLine);
String responseBody = EntityUtils.toString( httpResponse.getEntity() );
Log.d( Constants.TOURMAKER, TAG + " runTwitterUpdate responsebody = " + responseBody );
if( !isRunTwitterUpdateResponseOK(responseStatusLine, responseBody ) ) return;

/**
* Compose the so-called authorization header string
* @param callback
* @param consumerKey
* @param nonce
* @param signature
* @param timeStamp
* @param token
* @return
*/
@NonNull
private String getAuthorizationHeaderString( String callback, String consumerKey
      , String nonce, String signature, String timeStamp, String token, String version, String status ) {
String cb = callback != null ? OAUTH_CALLBACK + EQUAL + callback + COMMA : EMPTY;
String s = signature != null ? COMMA + OAUTH_SIGNATURE + EQUAL + QUOTE + encode(signature) + QUOTE : EMPTY;
String t = token != null ? COMMA + OAUTH_TOKEN + EQUAL + QUOTE + token + QUOTE : EMPTY;
String v = oauthTokenVerifier != null ? COMMA + OAUTH_VERIFIER + EQUAL + QUOTE + oauthTokenVerifier + QUOTE : EMPTY;
String st = status != null ? COMMA + STATUS + EQUAL + QUOTE + encode( status ) + QUOTE : EMPTY;
String authorization_header_string = OAUTH + BLANK
        + cb
        + OAUTH_CONSUMER_KEY + EQUAL + QUOTE + consumerKey + QUOTE
        + COMMA + OAUTH_SIGNATURE_METHOD + EQUAL + QUOTE + HMAC_SHA1 + QUOTE
        + s
        + COMMA + OAUTH_TIMESTAMP + EQUAL + QUOTE + timeStamp + QUOTE
        + COMMA + OAUTH_NONCE + EQUAL + QUOTE + nonce + QUOTE
        + t
        + v
        + st
        + COMMA + OAUTH_VERSION + EQUAL + QUOTE + version + QUOTE;
Log.d(Constants.TOURMAKER, TAG + " authorization_header_string = " + authorization_header_string);
return authorization_header_string;
}

Starting at line 6 of the above code I do:
httpParams.setParameter(“command”, “INIT”)
.setParameter(“media_type”, “image/jpeg”)
.setParameter(“total_bytes”, 6046 )
That’s the part I’m unsure of; is that how one is supposed to send command=INIT, etc.?
Is media_type=“image/jpeg” the reason I get the “media parameter is missing” error?

I hope someone can help me out here. Thanx. You know, I think I should start a new thread that specifically hilites this code. I suspect this thread is stale now. What do you think?

[ePirat]

To be honest, I don’t think this question is suitable for this forum, as it’s not Twitter related but about how to do standard HTTP POST encodings in Java. You might get much more results if you ask on Stackoverflow!

[BinCodinLong]

Hi ePirat, Andy Piper;
I don’t agree; I think this question does belong in the twitter forum. The question is not about how to do standard POSTs with httpcore; it’s about what does the twitter server require – exactly what does it require – and where does the twitter server require it?

The sticking-point is evidently three things: the command=INIT, the media_type=image/jpeg, and the total_bytes=6046 (the last two being “for instances,” of course).
The question only twitter can answer is where do they require them: in a header, in the message body, in an entity…?
This is not a general question; it’s about what twitter software requires.

I think I’ll start a new thread in this forum, and in StackOverflow.

[andypiper]

Those three parameters should be posted (that order is fine) as the data in a POST request - the twurl example on the upload media page shows this being done. The missing piece for me is gaps in my knowledge over how you would do this in httpcore in Java.

[ePirat]

The sticking-point is evidently three things: the command=INIT, the media_type=image/jpeg, and the total_bytes=6046 (the last two being “for instances,” of course). The question only twitter can answer is where do they require them: in a header, in the message body, in an entity…?

I already explained that in an earlier post:

The init command can be sent with a Content-Type of application/x-www-form-urlencoded

So following your example, this would look like:

command=INIT&media_type=image%2Fjpeg&total_bytes=6046

[BinCodinLong]

Hi again,
ePirat, oh, if only it were that simple; I would have been done a long time ago.

Andy Piper, can you ask one of the server-side programmers to help? It shouldn’t take much of his/her time.
I’ve tried several approaches; I’ll show them to you below. Take them to one of the programmers and ask which
approach is correct, if either; and why both approaches get “media parameter is missing.” response.
Here’s the first approach:

    BasicHttpEntityEnclosingRequest httpEntityEnclosingRequest =
            new BasicHttpEntityEnclosingRequest( POST, TWITTER_ENDPOINT_UPLOAD_MEDIA );

    // I tried the following line, but it causes authorization required failure
    //new BasicHttpEntityEnclosingRequest( POST, TWITTER_ENDPOINT_UPLOAD_MEDIA+"?command=INIT&media_type=image/jpeg&total_bytes=6046" );

    // Since we setContentType("application/x-www-form-urlencoded"),
    // it seems reasonable that we should send a UrlEncodedFormEntity.
    List<NameValuePair> formparams = new ArrayList<NameValuePair>();
    formparams.add(new BasicNameValuePair("command", "INIT"));
    formparams.add(new BasicNameValuePair("media_type", "image/png"));
    formparams.add(new BasicNameValuePair("total_bytes", "6046"));
    UrlEncodedFormEntity urlEncodedFormEntity = new UrlEncodedFormEntity(formparams, UTF_8);
    httpEntityEnclosingRequest.setEntity( urlEncodedFormEntity );
    urlEncodedFormEntity.setContentType("application/x-www-form-urlencoded");

Here’s the second approach:

    StringEntity stringEntity;
    stringEntity = new StringEntity( "command=INIT&media_type=image%2Fjpeg&total_bytes=6046", UTF_8);
    stringEntity.setContentType("application/x-www-form-urlencoded");
	httpEntityEnclosingRequest.setEntity(stringEntity);

Both approaches proceed on to this code:

    httpEntityEnclosingRequest.setParams(httpParams);
    httpEntityEnclosingRequest.addHeader("Authorization", authorization_header_string);
    httpEntityEnclosingRequest.addHeader("Content-Type","x-www-form-urlencoded" );
    HttpContext httpContext = getHttpContext(httpHost, httpClientConnection);
    HttpResponse httpResponse = getHttpResponse(
            httpParams, httpClientConnection, httpContext, httpEntityEnclosingRequest);

Both approaches get the response:
{“errors”:[{“code”:38,“message”:“media parameter is missing.”}]}.

Do this for me and I’ll send you a box of candy every day for a year,
or if you prefer I’ll buy you a yacht.

[ePirat]

I’ve just checked again with twurl and even with cURL:

twurl -t -H upload.twitter.com "/1.1/media/upload.json" -d "command=INIT&media_type=image/jpeg&total_bytes=6046" 

works correctly here and returns:

{"media_id":702275033650044928,"media_id_string":"702275033650044928","expires_after_secs":86400}

You can try it manually using cURL (use the OAuth tool at the bottom of this page) too. Given that this works fine, there has to be some issue with your code, finding and fixing that is not related at all to Twitter development but, as I already said, a general Java HTTP coding question and better suited to ask at Stackoverflow, as this is definitely not a problem with the Twitter Servers or API. (Unless, of course, manually testing that with twurl or cURL will give you the same error)

[BinCodinLong]

andyPiper:
Is there any chance you can talk with someone that knows the server code, or must I give up on this forum?

[andypiper]

Our media API team do watch this forum but I don’t think this is an issue on the backend. We know that it is possible to upload media with a curl/twurl command with all of the parameters specified, so it’s a matter of figuring out what’s going on in your Java client-side code to cause the request to not be formatted as expected.

Take a look at this patch to the Twitter4j code which adds chunked media upload support. It might give you some clues about how to format the URL parameters correctly. I realise you’ve explained you can’t use Twitter4j directly but this might be helpful in understanding the way other Java developers have implemented something similar.

[BinCodinLong]

andyPiper:
Thanks. I was unaware of this code. It could be just what I need.

[BinCodinLong]

Today I’m trying the twurl at Uploading Media.
The image Vietnam2.jpg is width 640 x height 360 x Bit depth 24.
I have tried the same image as png.
Twurl version is 0.9.3 and ruby version is 2.4.5.1
I always get the following failure:
C:\tools\ruby22\bin>twurl -H upload.twitter.com -X POST “/1.1/media/upload.json” --file “C:/Vietnam2
.jpg” --file-field “media” -t
opening connection to upload.twitter.com:443…
opened
starting SSL for upload.twitter.com:443…
SSL established
<- “POST /1.1/media/upload.json HTTP/1.1\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
\r\nAccept: /\r\nUser-Agent: OAuth gem v0.4.7\r\nContent-Type: multipart/form-data, boundary=“00T
wurl514810677127570030lruwT99”\r\nAuthorization: OAuth oauth_body_hash=“dkffoHbE4r7%2B%2FMHUEJ8i1s
tpIM8%3D”, oauth_consumer_key=“ai0QfDr5Dn6EaMMYak4f0z7bc”, oauth_nonce=“VpTRutZzwR6w8uUpdGFXAy6b
yER64rfwNBCp3Obvg”, oauth_signature=“WGqsQnbTpvFD76nvWu7MbSXsL2A%3D”, oauth_signature_method=“HM
AC-SHA1”, oauth_timestamp=“1456844084”, oauth_token=“3312874117-z40Ty2K8jKxzjhLBqXNqhnQ0yMxI5qVT
5LmNuLa”, oauth_version=“1.0”\r\nConnection: close\r\nHost: upload.twitter.com\r\nContent-Length:
248\r\n\r\n”
<- “–00Twurl514810677127570030lruwT99\r\nContent-Disposition: form-data; name=“media”; filename=
“Vietnam2.jpg”\r\nContent-Type: application/octet-stream\r\n\r\n\xFF\xD8\xFF\xE0\x00\x10JFIF\x00\x0
1\x01\x01\t\t\x00\x00\xFF\xDB\x00C\x00\x06\x04\x05\x06\x05\x04\x06\x06\x05\x06\a\a\x06\b\n\x10\n\n
\t\t\n\x14\x0E\x0F\f\x10\x17\x14\x18\x18\x17\x14\x16\x16\r\n–00Twurl514810677127570030lruwT99–\r\n
”
-> “HTTP/1.1 400 Bad Request\r\n”
-> “cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0\r\n”
-> “connection: close\r\n”
-> “content-disposition: attachment; filename=json.json\r\n”
-> “content-encoding: gzip\r\n”
-> “content-length: 98\r\n”
-> “content-type: application/json;charset=utf-8\r\n”
-> “date: Tue, 01 Mar 2016 14:54:37 GMT\r\n”
-> “expires: Tue, 31 Mar 1981 05:00:00 GMT\r\n”
-> “last-modified: Tue, 01 Mar 2016 14:54:37 GMT\r\n”
-> “pragma: no-cache\r\n”
-> “server: tsa_a\r\n”
-> “set-cookie: lang=en; Path=/\r\n”
-> “set-cookie: guest_id=v1%3A145684407746817050; Domain=.twitter.com; Path=/; Expires=Thu, 01-Mar-2
018 14:54:37 UTC\r\n”
-> “status: 400 Bad Request\r\n”
-> “strict-transport-security: max-age=631138519\r\n”
-> “vary: Origin\r\n”
-> “x-access-level: read-write\r\n”
-> “x-connection-hash: 9ae37f95e24206de92104e10d44adcee\r\n”
-> “x-frame-options: SAMEORIGIN\r\n”
-> “x-rate-limit-limit: 415\r\n”
-> “x-rate-limit-remaining: 411\r\n”
-> “x-rate-limit-reset: 1456846253\r\n”
-> “x-response-time: 34\r\n”
-> “x-transaction: cbcc78893b7773f5\r\n”
-> “x-tsa-request-body-time: 13\r\n”
-> “x-twitter-response-tags: BouncerCompliant\r\n”
-> “x-xss-protection: 1; mode=block\r\n”
-> "\r\n"
reading 98 bytes…
-> “”
-> “\x1F\x8B\b\x00\x00\x00\x00\x00\x00\x00\xAAVJ-*\xCA/*V\xB2\x8A\xAEVJ\xCEOIU\xB2262\xD1Q\xCAM-.NL
a\xF2\x94B2R\x15\xCA\x12s2S\x12K2\xF3\xF3\x14\xF2\xD3\x14rSS2\x13\x152S\x8A\x15\xD2\x123sRS\xF4\x94j
ck\x01\x00\x00\x00\xFF\xFF\x03\x00\xC1(\xB1.I\x00\x00\x00”
{“errors”:[{“code”:324,“message”:“The validation of media ids failed.”}]}read 98 bytes
Conn close
C:\tools\ruby22\bin>twurl -v
0.9.3

C:\tools\ruby22\bin>gem -v
2.4.5.1

What’s up?

[andypiper]

Not sure - our media testing tool here -> https://django-rest-apis.herokuapp.com/media/photo may help.

[BinCodinLong]

I get Server Error (500) no matter whether I upload with php, Java, anything…
BTW, I tried to tell twurl to use a proxy with -P 127.0.0.1:8888 so that I could observe sessions in Fiddler. but I just get the following:
C:\tools\ruby22\bin>twurl -H upload.twitter.com -X POST “/1.1/media/upload.json” --file “C:/Vietnam2
.jpg” --file-field “media” -t -P 127.0.0.1:8888
C:/tools/ruby22/lib/ruby/2.2.0/uri/rfc3986_parser.rb:66:in split': bad URI(is not URI?): 127.0.0.1: 8888 (URI::InvalidURIError) from C:/tools/ruby22/lib/ruby/2.2.0/uri/rfc3986_parser.rb:72:inparse’
from C:/tools/ruby22/lib/ruby/2.2.0/uri/common.rb:226:in parse' from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/oauth-0.4.7/lib/oauth/consumer.rb:307:increa
te_http’
from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/oauth-0.4.7/lib/oauth/consumer.rb:92:in `http’

    from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/twurl-0.9.3/lib/twurl/oauth_client.rb:193:in `

configure_http!'
from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/twurl-0.9.3/lib/twurl/oauth_client.rb:62:in i nitialize' from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/twurl-0.9.3/lib/twurl/oauth_client.rb:24:inn
ew’
from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/twurl-0.9.3/lib/twurl/oauth_client.rb:24:in l oad_client_for_username_and_consumer_key' from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/twurl-0.9.3/lib/twurl/oauth_client.rb:48:inl
oad_default_client’
from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/twurl-0.9.3/lib/twurl/oauth_client.rb:17:in l oad_from_options' from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/twurl-0.9.3/lib/twurl/cli.rb:25:indispatch’
from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/twurl-0.9.3/lib/twurl/cli.rb:21:in run' from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/twurl-0.9.3/bin/twurl:4:in<top (required)>'
from C:/tools/ruby22/bin/twurl:23:in load' from C:/tools/ruby22/bin/twurl:23:in'
I tried to enclose the IP address in quotes and single-quotes as well, to no avail.
Do you know how to direct twurl to a proxy?

[andypiper]

twurl -P <proxy>

[BinCodinLong]

What does <proxy> mean?
I just did it:
C:\tools\ruby22\bin>twurl -H upload.twitter.com -X POST “/1.1/media/upload.json” --file “C:/Vietnam2
.jpg” --file-field “media” -t -P <127.0.0.1:8888>
The syntax of the command is incorrect.

[andypiper]

Using <pointy brackets> is the standard way of saying this is a parameter you should fill in - so drop the < and > from your command and just enter the IP and port (may need to be quoted since you have a port there, I’m not certain). BTW I got that syntax from twurl -h.

[BinCodinLong]

Hi:
I was just writing back to you to say exactly what you just said about the general convention <something>; and I too got the same info as you from twurl -h of course.

What’s lacking is an explanation: Should User do twurl -P 127.0.0.1:8888; such that twurl will from then on send its requests through the proxy until User turns proxy off with maybe
twurl -P followed by nothing.
Or does User add -P 127.0.0.1:8888 to each of his command lines, like this:
C:\tools\ruby22\bin>twurl -H upload.twitter.com -X POST "/1.1/media/upload.json" --file "C:/Vietnam2.jpg" --file-field "media" -t -P 127.0.0.1:8888

Do you follow me? Documentation has to be explicit and thorough.

I’ve done many things:
twurl -P 127.0.0.1:8888 and
`twurl -P “127.0.0.1:8888” and many more.
All result in twurl merely printing out its help message with no indication it is going to start using the proxy.

twurl -H upload.twitter.com -X POST "/1.1/media/upload.json" --file "C:/Vietnam2 .jpg" --file-field "media" -t -P 127.0.0.1:8888
results in the error I sent you several messages back.

Anyway, we’re kinda getting off-topic; this whole proxy thing was just a BTW.
The real issue today is that
twurl -H upload.twitter.com -X POST "/1.1/media/upload.json" --file "C:/Vietnam2 .jpg" --file-field "media" -t
fails.

Maybe we should get back to that.

[BinCodinLong]

Andy:
I just went to my twitter timeline, and one of the https://django-rest-apis.herokuapp.com/media/photo uploads did indeed work. The image and text are there on my timeline.

So, we can perhaps disregard my Server Error (500) report.

Also, lets stop thinking about the proxy business.

Le’s get back to the original issue at message 18:
twurl -H upload.twitter.com -X POST "/1.1/media/upload.json" --file "C:/Vietnam2 .jpg" --file-field "media" -t
is failing.