POST Upload chunked INIT command gets media parameter is missing error

android
java

#1

Using twitter4j etc. blows my dex limit; so I have to roll my own twitter requests.
I’ve succeeded all the way through getting an access token and doing a statuses update.
Now I need to upload a photo, so I’m trying the https://dev.twitter.com/rest/reference/post/media/upload-chunked .
I’m not doing Update_with_media because that’s deprecated.
Here’s how I set the parameters; I’m not sure it’s the right way:
HttpParams httpParams = new BasicHttpParams();
HttpProtocolParams.setVersion(httpParams, HttpVersion.HTTP_1_1);
HttpProtocolParams.setContentCharset(httpParams, UTF_8);
HttpProtocolParams.setUserAgent(httpParams, HTTP_CORE_1_1);
HttpProtocolParams.setUseExpectContinue(httpParams, false);
httpParams.setParameter(“command”, “INIT”)
.setParameter(“media_type”, “image/png” )
.setParameter(“total_bytes”, 6046 );

Below is the debugger’s display of the entityEnclosingRequest I’m sending to
"https://upload.twitter.com/1.1/media/upload.json".
I get a: {“errors”:[{“code”:38,“message”:“media parameter is missing.”}]} response.
Can anybody help me out?

httpEntityEnclosingRequest = {BasicHttpEntityEnclosingRequest@6185}
entity = {StringEntity@6203}
content = {byte[43]@6258}
chunked = false
contentEncoding = null
contentType = {BasicHeader@6259} "Content-Type: application/x-www-form-urlencoded"
shadow$klass = {Class@6175} "class org.apache.http.entity.StringEntity"
shadow$monitor = -2063798403
method = {String@6204} "POST"
requestline = null
uri = {String@6078} "https://upload.twitter.com/1.1/media/upload.json"
headergroup = {HeaderGroup@6205}
params = {BasicHttpParams@6181}
parameters = {HashMap@6230} size = 7
0 = {HashMap$HashMapEntry@6234} “http.protocol.expect-continue” -> "false"
1 = {HashMap$HashMapEntry@6235} “media_type” -> "image/png"
2 = {HashMap$HashMapEntry@6236} “http.useragent” -> "HttpCore/1.1"
3 = {HashMap$HashMapEntry@6237} “command” -> "INIT"
4 = {HashMap$HashMapEntry@6238} “http.protocol.version” -> "HTTP/1.1"
5 = {HashMap$HashMapEntry@6239} “total_bytes” -> "6046"
6 = {HashMap$HashMapEntry@6240} “http.protocol.content-charset” -> "UTF-8"
shadow$klass = {Class@1707} "class org.apache.http.params.BasicHttpParams"
shadow$monitor = -2116729771
shadow$klass = {Class@6174} "class org.apache.http.message.BasicHttpEntityEnclosingRequest"
shadow$monitor = -2106404655


#2

If you have a look at the docs, you see in the second paragraph:

Ensure your POST is a multipart/form-data request.

but it seems you don’t do that:

Please try setting the correct content type and encode the data correctly.


#3

If you have a look at the docs, you see in the second paragraph:
See our uploading-media guide for more on using this endpoint.
There, you will find twurl examples for INIT, APPEND, and FINALIZE.
They all use Content-Type: application/x-www-form-urlencoded, and they work.

Basically, my question is how does one encode into the http request the command, media_type, and total_bytes data? Your answer:
Please try setting the correct content type and encode the data correctly.
boils down to “correctly”; that is not a helpful answer.
If you know how to encode these requests correctly, then please tell me how.


#4

Hi both! I think @ePirat was suggesting you double-check the encoding piece, not a direct criticism :slight_smile: but I can see how it might be read.

@BinCodinLong I’m afraid I don’t know the answer to your question directly either, but I’m going to move this thread over to the Media APIs category. Some of our team with more knowledge in these areas monitor that category a bit more often, so hopefully someone can help to explain what is going on here.

Peace! :bird:


#5

Thanks for the help! I hope somebody can give very specific – java code – advice.


#6

Sorry if my advice sounded unfriendly, it was not intended to be that way! Unfortunately I can’t provide you with Java Code examples but I can clarify how exactly you need to encode data.

INIT

The init command can be sent with a Content-Type of application/x-www-form-urlencoded which looks like key=value&key2=value2

APPEND

The append command has to be sent with a Content-Type of multipart/form-data and encoded as that properly, in twurl it looks like it’s actually application/x-www-form-urlencoded but that is only how the data is given on command line, it will not use that for the actual encoding.
There are quite a few posts about that on Stackoverflow about it, this for example.
If you really want to implement it yourself, have a look at RFC2388 which describes how it works.
If you have further questions, feel free to ask.

FINALZE

For finalize you can use application/x-www-form-urlencoded again.


#7

I know the media API team is also keen to get our documentation into better shape so hopefully this will all improve over time.


#8

Hi ePirat and andypiper if you’re around today.

I have un-methodized my code that composes the http message I send for the INIT.
Sorry it’s not pretty – I took out everything but the essentials. Anyway, here it is:

HttpParams httpParams = new BasicHttpParams();
HttpProtocolParams.setVersion(httpParams, HttpVersion.HTTP_1_1);
HttpProtocolParams.setContentCharset(httpParams, UTF_8);
HttpProtocolParams.setUserAgent(httpParams, HTTP_CORE_1_1);
HttpProtocolParams.setUseExpectContinue(httpParams, false);
httpParams.setParameter("command", "INIT")
          .setParameter("media_type", "image/jpeg")
          .setParameter("total_bytes", 6046 )
  
HttpHost httpHost =new HttpHost(UPLOAD_TWITTER_COM, 443);
DefaultHttpClientConnection httpClientConnection = new DefaultHttpClientConnection();

Socket socket = null;
// initialize the HTTPS connection
SSLContext sslcontext = SSLContext.getInstance("TLS");
sslcontext.init(null, null, null);
SSLSocketFactory ssf = sslcontext.getSocketFactory();
socket = ssf.createSocket();
socket.connect(new InetSocketAddress(httpHost.getHostName(), httpHost.getPort()), 0);
httpClientConnection.bind(socket, httpParams);
// see utilty method below
String authorization_header_string = getAuthorizationHeaderString(
            null, twitterConsumerKey, oauthNonce
            , oauthSignature, oauthTimestamp, encode(oauthTokenAccess), ONE_POINT_ZERO, null
    );
	
BasicHttpEntityEnclosingRequest httpEntityEnclosingRequest =
        new BasicHttpEntityEnclosingRequest( "POST", "https://upload.twitter.com/1.1/media/upload.json" );
httpEntityEnclosingRequest.setParams( httpParams );
httpEntityEnclosingRequest.addHeader("Authorization", authorization_header_string);

HttpContext httpContext = new BasicHttpContext(null);
httpContext.setAttribute(ExecutionContext.HTTP_CONNECTION, httpClientConnection);
httpContext.setAttribute(ExecutionContext.HTTP_TARGET_HOST, httpHost);

HttpProcessor httpProcessor = getHttpProcessor();
HttpRequestExecutor httpRequestExecutor = new HttpRequestExecutor();
httpRequestExecutor.preProcess(httpEntityEnclosingRequest, httpProcessor, httpContext);
HttpResponse httpResponse = httpRequestExecutor.execute(
        httpEntityEnclosingRequest, httpClientConnection, httpContext);
httpResponse.setParams(httpParams);
httpRequestExecutor.postProcess(httpResponse, httpProcessor, httpContext);

String responseStatusLine = httpResponse.getStatusLine().toString();
Log.d(Constants.TOURMAKER, TAG + " runTwitterUpdate responseStatusLine = " + responseStatusLine);
String responseBody = EntityUtils.toString( httpResponse.getEntity() );
Log.d( Constants.TOURMAKER, TAG + " runTwitterUpdate responsebody = " + responseBody );
if( !isRunTwitterUpdateResponseOK(responseStatusLine, responseBody ) ) return;

/**
* Compose the so-called authorization header string
* @param callback
* @param consumerKey
* @param nonce
* @param signature
* @param timeStamp
* @param token
* @return
*/
@NonNull
private String getAuthorizationHeaderString( String callback, String consumerKey
      , String nonce, String signature, String timeStamp, String token, String version, String status ) {
String cb = callback != null ? OAUTH_CALLBACK + EQUAL + callback + COMMA : EMPTY;
String s = signature != null ? COMMA + OAUTH_SIGNATURE + EQUAL + QUOTE + encode(signature) + QUOTE : EMPTY;
String t = token != null ? COMMA + OAUTH_TOKEN + EQUAL + QUOTE + token + QUOTE : EMPTY;
String v = oauthTokenVerifier != null ? COMMA + OAUTH_VERIFIER + EQUAL + QUOTE + oauthTokenVerifier + QUOTE : EMPTY;
String st = status != null ? COMMA + STATUS + EQUAL + QUOTE + encode( status ) + QUOTE : EMPTY;
String authorization_header_string = OAUTH + BLANK
        + cb
        + OAUTH_CONSUMER_KEY + EQUAL + QUOTE + consumerKey + QUOTE
        + COMMA + OAUTH_SIGNATURE_METHOD + EQUAL + QUOTE + HMAC_SHA1 + QUOTE
        + s
        + COMMA + OAUTH_TIMESTAMP + EQUAL + QUOTE + timeStamp + QUOTE
        + COMMA + OAUTH_NONCE + EQUAL + QUOTE + nonce + QUOTE
        + t
        + v
        + st
        + COMMA + OAUTH_VERSION + EQUAL + QUOTE + version + QUOTE;
Log.d(Constants.TOURMAKER, TAG + " authorization_header_string = " + authorization_header_string);
return authorization_header_string;
}

Starting at line 6 of the above code I do:
httpParams.setParameter(“command”, “INIT”)
.setParameter(“media_type”, “image/jpeg”)
.setParameter(“total_bytes”, 6046 )
That’s the part I’m unsure of; is that how one is supposed to send command=INIT, etc.?
Is media_type=“image/jpeg” the reason I get the “media parameter is missing” error?

I hope someone can help me out here. Thanx. You know, I think I should start a new thread that specifically hilites this code. I suspect this thread is stale now. What do you think?


#9

To be honest, I don’t think this question is suitable for this forum, as it’s not Twitter related but about how to do standard HTTP POST encodings in Java. You might get much more results if you ask on Stackoverflow!


#10

Hi ePirat, Andy Piper;
I don’t agree; I think this question does belong in the twitter forum. The question is not about how to do standard POSTs with httpcore; it’s about what does the twitter server require – exactly what does it require – and where does the twitter server require it?

The sticking-point is evidently three things: the command=INIT, the media_type=image/jpeg, and the total_bytes=6046 (the last two being “for instances,” of course).
The question only twitter can answer is where do they require them: in a header, in the message body, in an entity…?
This is not a general question; it’s about what twitter software requires.

I think I’ll start a new thread in this forum, and in StackOverflow.


#11

Those three parameters should be posted (that order is fine) as the data in a POST request - the twurl example on the upload media page shows this being done. The missing piece for me is gaps in my knowledge over how you would do this in httpcore in Java.


#12

I already explained that in an earlier post:

So following your example, this would look like:

command=INIT&media_type=image%2Fjpeg&total_bytes=6046

#13

Hi again,
ePirat, oh, if only it were that simple; I would have been done a long time ago.

Andy Piper, can you ask one of the server-side programmers to help? It shouldn’t take much of his/her time.
I’ve tried several approaches; I’ll show them to you below. Take them to one of the programmers and ask which
approach is correct, if either; and why both approaches get “media parameter is missing.” response.
Here’s the first approach:

    BasicHttpEntityEnclosingRequest httpEntityEnclosingRequest =
            new BasicHttpEntityEnclosingRequest( POST, TWITTER_ENDPOINT_UPLOAD_MEDIA );

    // I tried the following line, but it causes authorization required failure
    //new BasicHttpEntityEnclosingRequest( POST, TWITTER_ENDPOINT_UPLOAD_MEDIA+"?command=INIT&media_type=image/jpeg&total_bytes=6046" );

    // Since we setContentType("application/x-www-form-urlencoded"),
    // it seems reasonable that we should send a UrlEncodedFormEntity.
    List<NameValuePair> formparams = new ArrayList<NameValuePair>();
    formparams.add(new BasicNameValuePair("command", "INIT"));
    formparams.add(new BasicNameValuePair("media_type", "image/png"));
    formparams.add(new BasicNameValuePair("total_bytes", "6046"));
    UrlEncodedFormEntity urlEncodedFormEntity = new UrlEncodedFormEntity(formparams, UTF_8);
    httpEntityEnclosingRequest.setEntity( urlEncodedFormEntity );
    urlEncodedFormEntity.setContentType("application/x-www-form-urlencoded");

Here’s the second approach:

    StringEntity stringEntity;
    stringEntity = new StringEntity( "command=INIT&media_type=image%2Fjpeg&total_bytes=6046", UTF_8);
    stringEntity.setContentType("application/x-www-form-urlencoded");
	httpEntityEnclosingRequest.setEntity(stringEntity);

Both approaches proceed on to this code:

    httpEntityEnclosingRequest.setParams(httpParams);
    httpEntityEnclosingRequest.addHeader("Authorization", authorization_header_string);
    httpEntityEnclosingRequest.addHeader("Content-Type","x-www-form-urlencoded" );
    HttpContext httpContext = getHttpContext(httpHost, httpClientConnection);
    HttpResponse httpResponse = getHttpResponse(
            httpParams, httpClientConnection, httpContext, httpEntityEnclosingRequest);

Both approaches get the response:
{“errors”:[{“code”:38,“message”:“media parameter is missing.”}]}.

Do this for me and I’ll send you a box of candy every day for a year,
or if you prefer I’ll buy you a yacht.


#14

I’ve just checked again with twurl and even with cURL:

twurl -t -H upload.twitter.com "/1.1/media/upload.json" -d "command=INIT&media_type=image/jpeg&total_bytes=6046" 

works correctly here and returns:

{"media_id":702275033650044928,"media_id_string":"702275033650044928","expires_after_secs":86400}

You can try it manually using cURL (use the OAuth tool at the bottom of this page) too. Given that this works fine, there has to be some issue with your code, finding and fixing that is not related at all to Twitter development but, as I already said, a general Java HTTP coding question and better suited to ask at Stackoverflow, as this is definitely not a problem with the Twitter Servers or API. (Unless, of course, manually testing that with twurl or cURL will give you the same error)


#15

andyPiper:
Is there any chance you can talk with someone that knows the server code, or must I give up on this forum?


#16

Our media API team do watch this forum but I don’t think this is an issue on the backend. We know that it is possible to upload media with a curl/twurl command with all of the parameters specified, so it’s a matter of figuring out what’s going on in your Java client-side code to cause the request to not be formatted as expected.

Take a look at this patch to the Twitter4j code which adds chunked media upload support. It might give you some clues about how to format the URL parameters correctly. I realise you’ve explained you can’t use Twitter4j directly but this might be helpful in understanding the way other Java developers have implemented something similar.


#17

andyPiper:
Thanks. I was unaware of this code. It could be just what I need.


#18

Today I’m trying the twurl at Uploading Media.
The image Vietnam2.jpg is width 640 x height 360 x Bit depth 24.
I have tried the same image as png.
Twurl version is 0.9.3 and ruby version is 2.4.5.1
I always get the following failure:
C:\tools\ruby22\bin>twurl -H upload.twitter.com -X POST “/1.1/media/upload.json” --file “C:/Vietnam2
.jpg” --file-field “media” -t
opening connection to upload.twitter.com:443
opened
starting SSL for upload.twitter.com:443
SSL established
<- “POST /1.1/media/upload.json HTTP/1.1\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
\r\nAccept: /\r\nUser-Agent: OAuth gem v0.4.7\r\nContent-Type: multipart/form-data, boundary=“00T
wurl514810677127570030lruwT99”\r\nAuthorization: OAuth oauth_body_hash=“dkffoHbE4r7%2B%2FMHUEJ8i1s
tpIM8%3D”, oauth_consumer_key=“ai0QfDr5Dn6EaMMYak4f0z7bc”, oauth_nonce=“VpTRutZzwR6w8uUpdGFXAy6b
yER64rfwNBCp3Obvg”, oauth_signature=“WGqsQnbTpvFD76nvWu7MbSXsL2A%3D”, oauth_signature_method=“HM
AC-SHA1”, oauth_timestamp=“1456844084”, oauth_token=“3312874117-z40Ty2K8jKxzjhLBqXNqhnQ0yMxI5qVT
5LmNuLa”, oauth_version=“1.0”\r\nConnection: close\r\nHost: upload.twitter.com\r\nContent-Length:
248\r\n\r\n”
<- “–00Twurl514810677127570030lruwT99\r\nContent-Disposition: form-data; name=“media”; filename=
“Vietnam2.jpg”\r\nContent-Type: application/octet-stream\r\n\r\n\xFF\xD8\xFF\xE0\x00\x10JFIF\x00\x0
1\x01\x01\t\t\x00\x00\xFF\xDB\x00C\x00\x06\x04\x05\x06\x05\x04\x06\x06\x05\x06\a\a\x06\b\n\x10\n\n
\t\t\n\x14\x0E\x0F\f\x10\x17\x14\x18\x18\x17\x14\x16\x16\r\n–00Twurl514810677127570030lruwT99–\r\n

-> “HTTP/1.1 400 Bad Request\r\n”
-> “cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0\r\n”
-> “connection: close\r\n”
-> “content-disposition: attachment; filename=json.json\r\n”
-> “content-encoding: gzip\r\n”
-> “content-length: 98\r\n”
-> “content-type: application/json;charset=utf-8\r\n”
-> “date: Tue, 01 Mar 2016 14:54:37 GMT\r\n”
-> “expires: Tue, 31 Mar 1981 05:00:00 GMT\r\n”
-> “last-modified: Tue, 01 Mar 2016 14:54:37 GMT\r\n”
-> “pragma: no-cache\r\n”
-> “server: tsa_a\r\n”
-> “set-cookie: lang=en; Path=/\r\n”
-> “set-cookie: guest_id=v1%3A145684407746817050; Domain=.twitter.com; Path=/; Expires=Thu, 01-Mar-2
018 14:54:37 UTC\r\n”
-> “status: 400 Bad Request\r\n”
-> “strict-transport-security: max-age=631138519\r\n”
-> “vary: Origin\r\n”
-> “x-access-level: read-write\r\n”
-> “x-connection-hash: 9ae37f95e24206de92104e10d44adcee\r\n”
-> “x-frame-options: SAMEORIGIN\r\n”
-> “x-rate-limit-limit: 415\r\n”
-> “x-rate-limit-remaining: 411\r\n”
-> “x-rate-limit-reset: 1456846253\r\n”
-> “x-response-time: 34\r\n”
-> “x-transaction: cbcc78893b7773f5\r\n”
-> “x-tsa-request-body-time: 13\r\n”
-> “x-twitter-response-tags: BouncerCompliant\r\n”
-> “x-xss-protection: 1; mode=block\r\n”
-> "\r\n"
reading 98 bytes…
-> “”
-> “\x1F\x8B\b\x00\x00\x00\x00\x00\x00\x00\xAAVJ-*\xCA/*V\xB2\x8A\xAEVJ\xCEOIU\xB2262\xD1Q\xCAM-.NL
a\xF2\x94B2R\x15\xCA\x12s2S\x12K2\xF3\xF3\x14\xF2\xD3\x14rSS2\x13\x152S\x8A\x15\xD2\x123sRS\xF4\x94j
ck\x01\x00\x00\x00\xFF\xFF\x03\x00\xC1(\xB1.I\x00\x00\x00”
{“errors”:[{“code”:324,“message”:“The validation of media ids failed.”}]}read 98 bytes
Conn close
C:\tools\ruby22\bin>twurl -v
0.9.3

C:\tools\ruby22\bin>gem -v
2.4.5.1

What’s up?


#19

Not sure - our media testing tool here -> https://django-rest-apis.herokuapp.com/media/photo may help.


#20

I get Server Error (500) no matter whether I upload with php, Java, anything…
BTW, I tried to tell twurl to use a proxy with -P 127.0.0.1:8888 so that I could observe sessions in Fiddler. but I just get the following:
C:\tools\ruby22\bin>twurl -H upload.twitter.com -X POST “/1.1/media/upload.json” --file “C:/Vietnam2
.jpg” --file-field “media” -t -P 127.0.0.1:8888
C:/tools/ruby22/lib/ruby/2.2.0/uri/rfc3986_parser.rb:66:in split': bad URI(is not URI?): 127.0.0.1: 8888 (URI::InvalidURIError) from C:/tools/ruby22/lib/ruby/2.2.0/uri/rfc3986_parser.rb:72:inparse’
from C:/tools/ruby22/lib/ruby/2.2.0/uri/common.rb:226:in parse' from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/oauth-0.4.7/lib/oauth/consumer.rb:307:increa
te_http’
from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/oauth-0.4.7/lib/oauth/consumer.rb:92:in `http’

    from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/twurl-0.9.3/lib/twurl/oauth_client.rb:193:in `

configure_http!'
from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/twurl-0.9.3/lib/twurl/oauth_client.rb:62:in i nitialize' from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/twurl-0.9.3/lib/twurl/oauth_client.rb:24:inn
ew’
from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/twurl-0.9.3/lib/twurl/oauth_client.rb:24:in l oad_client_for_username_and_consumer_key' from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/twurl-0.9.3/lib/twurl/oauth_client.rb:48:inl
oad_default_client’
from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/twurl-0.9.3/lib/twurl/oauth_client.rb:17:in l oad_from_options' from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/twurl-0.9.3/lib/twurl/cli.rb:25:indispatch’
from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/twurl-0.9.3/lib/twurl/cli.rb:21:in run' from C:/tools/ruby22/lib/ruby/gems/2.2.0/gems/twurl-0.9.3/bin/twurl:4:in<top (required)>'
from C:/tools/ruby22/bin/twurl:23:in load' from C:/tools/ruby22/bin/twurl:23:in'
I tried to enclose the IP address in quotes and single-quotes as well, to no avail.
Do you know how to direct twurl to a proxy?