DM Media download results in redirection loop

[yfilyk]

Hi, that’s me again here. Running in circles (literally).

I have DM media url: https://ton.twitter.com/1.1/ton/data/dm/920660133419642886/920660123240067074/7hNzfpxZ.png .

Works fine in browser. However every time I try to download it using any http library (Net::Http, OpenURI, Faraday, Httparty) - I get into redirection loop. Request to media url returns status 302, redirects to the same url, request to which gives status 302 and redirects to the same bloody url!

Since I’m not exactly sure how this is wired on your side, could you give me a hint about what I should do with second request (after first 302) to get 200 with it and not 302 again?

Please and Thank You

[joncipriano]

Not sure why you may seeing a 302, but images in DMs are private like the DMs themselves. So as you need to provide authentication information to access or send a DM, you need to to do exactly the same when requesting the image. It works in a browser, because you are probably logged in to Twitter with the account that has access to the DM image.

[yfilyk]

Thanks for you answer Jon.
Actually browser gets 302 redirect first time as well, this redirect points to the same image url and so following the redirect browser goes to the same url but gets http status 200 and a file this time. Screenshot from chrome debugger tool as confirmation:

First Request:

Redirect

Same thing happens in Postman when I send signed request with user tokens. First request is redirected to the same media url and following the redirect we get the file with status 200.
However using http libraries after being redirected with 302 I get 302 again, so get redirected again, and again, in endless loop. Log:

irb(main):006:0> response = HTTParty.get(url, follow_redirects: true, headers: { 'Authorization'=>'OAuth oauth_consumer_key=\"****\",oauth_token=\"***-***\",oauth_signature_method=\"HMAC-SHA1\",oauth_timestamp=\"1508410566\",oauth_nonce=\"***\",oauth_version=\"1.0\",oauth_signature=\"****%3D\"' })
HTTParty::RedirectionTooDeep: HTTP redirects too deep

In case access tokens were wrong - I’d get 401 error I believe.

I was going to ask how exactly do you define whether to return status 302 and redirect or status 200 and file when both times request goes to the same URL?

Hope this helps. Thanks!

[joncipriano]

I don’t see any redirects myself in chrome or using postman. I know of a few developer referencing images in DMs without issue in production at the moment, as well. Avoiding the time to set up a ruby env appropriately, I was able to successfully download an image in my own DM using Node’s request library without issue.

var request = require('request')
var fs = require("fs")

// twitter authentication
var twitter_oauth = {
  consumer_key: 'redacted',
  consumer_secret: 'redacted',
  token: 'redacted',
  token_secret: 'redacted'
}

// update request options
request_options = {
  url: 'redacted', // URL to image in DM
  oauth: twitter_oauth
}

// download image
request.get(request_options).pipe(fs.createWriteStream('image.jpg')).on('close', function () {
  console.log('Download complete.')
})

[yfilyk]

Thanks for sharing the code Jon.

I executed it on my machine and it works very well. I’ll try to translate it to ruby as best as I can.
I guess something is just not right with the code Postman generated for me. Here’s what code Postman generates using Node request library:

var request = require("request");

var options = { method: 'GET',
  url: 'https://ton.twitter.com/1.1/ton/data/dm/920660133419642886/920660123240067074/7hNzfpxZ.png',
  headers: { authorization: 'OAuth oauth_consumer_key=\\"redacted\\",oauth_token=\\"redaceted\\",oauth_signature_method=\\"HMAC-SHA1\\",oauth_timestamp=\\"1508417698\\",oauth_nonce=\\"redacted\\",oauth_version=\\"1.0\\",oauth_signature=\\"redacted%3D\\"' } };

request(options, function (error, response, body) {
  if (error) throw new Error(error);

  console.log(body);
});

And here’s the output:

↪ node nodescript.js
(node:1586) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 pipe listeners added. Use emitter.setMaxListeners() to increase limit
/Users/admin/Downloads/nodescript.js:30
  if (error) throw new Error(error);
             ^

Error: Error: Exceeded maxRedirects. Probably stuck in a redirect loop https://ton.twitter.com/1.1/ton/data/dm/920660133419642886/920660123240067074/7hNzfpxZ.png

Will investigate why generated code looks like this and why it doesn’t work.

UPD:
Changing “headers: { authorization: …” bit to “oauth: twitter_oauth” makes all the difference, I wonder why

UPD 2:
Got it working. Ruby code:

require 'faraday'
require 'simple_oauth'

hdr = SimpleOAuth::Header.new('GET', 'https://ton.twitter.com/1.1/ton/data/dm/920660133419642886/920660123240067074/7hNzfpxZ.png', {}, { consumer_key: 'redacted',consumer_secret: 'redacted',token: 'redacted',token_secret: 'redacted' })

response = Faraday.get do |req|
  req.url('https://ton.twitter.com/1.1/ton/data/dm/920660133419642886/920660123240067074/7hNzfpxZ.png')
  req.headers['Authorization'] = hdr.to_s
end

puts response.body # raw image

Still don’t know why header generated by either Postman or Twitter Gem did not work but I guess it doesn’t matter now.

Many Thanks Jon, you code was very inspiring

[joncipriano]

Good to hear! This is interesting indeed, but glad you got it worked out.

My postman generated javascript resulted in the same result, but I was able to get it to work by modifying the headers object (added missing single quotes and removed unnecessary escaped double quotes).

var request = require("request");

var options = { method: 'GET',
  url: 'https://ton.twitter.com/1.1/ton/data/dm/redacted.jpg:large',
  headers: 
   { 'postman-token': 'b4704b56-3b54-40f4-1899-d8ad3f0d8e13', 
     'cache-control': 'no-cache',
     'authorization': 'OAuth oauth_consumer_key="redacted",oauth_token="redacted",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1508517740",oauth_nonce="redacted",oauth_version="1.0",oauth_signature="redacted"' } };

request(options, function (error, response, body) {
  if (error) throw new Error(error);

  console.log('success');
});

[CZeniark]

Thanks! you made my day ahha Thank GOD for you