Issue with the encoding of the forward slash character for chunked media upload

oauth
video
rest

#1

My application needs to be able to upload video to Twitter. To initiate a chunked upload, the application needs to make a POST request like this:

https://upload.twitter.com/1.1/media/upload.json?command=INIT&media_type=video/mp4&total_bytes=5471907

The trouble I am having relates to the encoding of the forward slash character in the media_type=video/mp4 parameter. My application is written in Scala and it uses the Spray.io HTTP client library for creating requests and parsing responses. We have our own implementation of OAuth 1.0a for signing the requests.

By experimenting with Twitter’s tools, it appears that Twitter requires the forward slash to be percent-encoded as %2F in the HTTP request. In other words, it expects the request URL to be exactly this:

https://upload.twitter.com/1.1/media/upload.json?command=INIT&media_type=video%2Fmp4&total_bytes=5471907

However, the Spray library will not allow us to encode the parameter string like this. If we specify the parameter value using a /, then it remains a / when the request is sent. If we change the / to %2F in the parameter value, then Spray thinks that the % character itself needs to be percent encoded. Either way, we end up with a 401 Unauthorized response from Twitter.

The Spray architect believes that RFC 3986 does not require the forward slash to be percent-encoded. Click here to see where he is quoted as saying this.

First, I want to ask the Developer community if anyone else is using Scala and Spray and has found a way to work around this limitation in Spray? Is there some way to get Spray to encode the parameter string the way that Twitter requires? Or are we going to have to use some different HTTP client library to make this one REST request?

I would appreciate it if someone from Twitter could comment on this question: why does Twitter require that the forward slash be percent-encoded? I understand that the OAuth algorithm requires that character to be percent encoded when calculating the signature, but it seems to me that requiring the percent-encoding of that character on the HTTP request itself conflicts with RFC 3986.


#2

I think this quote from RFC 3986, section 3.4 is particularly relevant to this topic.

The characters slash ("/") and question mark ("?") may represent data within the query component. *[...]* However, as query components are often used to carry identifying information in the form of "key=value" pairs and one frequently used value is a reference to another URI, it is sometimes better for usability to avoid percent-encoding those characters.

#3

I have found a workaround. This Twitter endpoint currently accepts the parameters passed in the request body using Content-Type: application/x-www-form-urlencoded. When we do this, the request successfully returns the 202 Accepted response that we need.