Can you get Public Timeline using oauth by only using curl and openssl in Unix shell


#1

After reading several articles on oauth in twitter it seems the basic process is:
One time event - Register App using a valid twitter account and get 4 static strings (Consumer Key and Secret and Access token and secret)
Every time you want to access a public timeline:

  1. Generate one-time only strings (oauth_nonce, oauth_signature, oauth_timestamp) using some of above 4 static strings
  2. Use static strings (Consumer Key and Access token) and one-time only strings (oauth_nonce, oauth_signature, oauth_timestamp) to access public timeline

Is it as straightforward as this and if so can I use openssl for step 1 and curl for step 2.
If it is this straightforward can someone post an example using openssl and curl, but if it is not this simple can someone expand on what is missing from my process.
At the moment I am not using oauth and so from a unix shell I can just use:
curl https://api.twitter.com/1/statuses/user_timeline.rss?screen_name=twitterapi

But as I understand, this method will no longer work from March 2013 so I need to use oauth. I was hoping I can do this without have to use PHP, python, Ruby, C++, etc

Thanks
Mike


#2

It is as straight forward as I outlined above, except you have to percent encode some of the information - here’s how to do it:
Create an App at https://dev.twitter.com/apps, (as I am just getting tweets from the command line, the Website field isn’t relevant so I just entered http://www.google.co.uk) and in the “Details” tab click on “Create my access token” and then go to the “OAuth tool” tab of your application and enter Request URL and query, so for https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=twitterapi you would just enter URL “https://api.twitter.com/1.1/statuses/user_timeline.json” and query “screen_name=twitterapi” and click on "See OAuth signature for this request"
This gives an example of the Signature base string and the cURL command and the cURL command would work, but only for a limited time as it contains a timestamp, but if you replace the timestamp and nonce, then you can reuse the Signature base string.
The oauth_timestamp is just the number of seconds since the Unix epoch for which you can use "date +%s"
The oauth_nonce is any unique alphanumeric string, so I base64 encoded the timestamp+HH:MM:SS+Nanoseconds and stripped out +, = and / as follows:
date +%s%T%N | openssl base64 | sed -e s’/[+=/]//g’
So if you generate the timestamp and nonce then you can reuse the “Signature base string” and the “Signature base string” is used to create the oauth_signature as follows using openssl:
echo -n $sig_base_string | openssl dgst -sha1 -hmac key -binary | openssl base64 | sed -e s’/+/%2B/’ -e s’///%2F/’ -e s’/=/%3D/'
The key is the Consumer secret, followed by an ampersand character ‘&’, followed by the Access token secret where both secrets must be percent encoded which basically means replacing any non-alphanumeric keys with %hexcode. The resulting signature also has to be percent encoded and as this is base64 encoded, I know the only non-alphanumeric keys that could be present are +, =, / so the “sed” replaces them with %hexcode.
So then you just run the example curl code replacingyour generated oauth_nonce, oauth_signature and oauth_timestamp

Below is an example script of just 5 lines where I have used a parameter for screen_name so you can use this script for screen names other than “twitterapi”:
screen_name=twitterapi
timestamp=date +%s
nonce=date +%s%T%N | openssl base64 | sed -e s'/[+=/]//g'

signature=echo -n 'GET&https%3A%2F%2Fapi.twitter.com%2F1.1%2Fstatuses%2Fuser_timeline.json&oauth_consumer_key%3DQ4hmbd34xSSymdZNPLVzpA%26oauth_nonce%3D'$nonce'%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D'$timestamp'%26oauth_token%3D204448593-GHvNZ5FqMETOgH3QVV3M1AZElzyGNXsaGiq2TXRE%26oauth_version%3D1.0%26screen_name%3D'$screen_name | openssl dgst -sha1 -hmac 'xzuHvl9PkWlbOqUotjXLxf7nsKfPQKiI2skwCNAy0&JHwxCOYDaI4jCdALc9938Lu7piqaNodjByubQhFY' -binary | openssl base64 | sed -e s'/+/%2B/' -e s'/\//%2F/' -e s'/=/%3D/'

curl --get ‘https://api.twitter.com/1.1/statuses/user_timeline.json’ --data “screen_name=$screen_name” --header ‘Authorization: OAuth oauth_consumer_key=“Q4hmbd34xSSymdZNPLVzpA”, oauth_nonce="’$nonce’", oauth_signature="’$signature’", oauth_signature_method=“HMAC-SHA1”, oauth_timestamp="’$timestamp’", oauth_token=“204448593-GHvNZ5FqMETOgH3QVV3M1AZElzyGNXsaGiq2TXRE”, oauth_version=“1.0”’ --verbose

I’ll probably improve this code by using parameters for keys, but it works.
One thing to note is that I used:
https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=twitterapi
rather than
https://api.twitter.com/1/statuses/user_timeline.rss?screen_name=twitterapi

which is used in the opening post as the whole point of using oauth is that you won’t be able to use Twitter API version 1 from March and you will have to use 1.1 and this does not support rss format, so I now need to parse the json format which looks more difficult than rss format.

The App details for the above code example are (I have now deleted App so the keys are no longer valid for the above code):
Consumer key Q4hmbd34xSSymdZNPLVzpA
Consumer secret xzuHvl9PkWlbOqUotjXLxf7nsKfPQKiI2skwCNAy0

Access token 204448593-GHvNZ5FqMETOgH3QVV3M1AZElzyGNXsaGiq2TXRE
Access token secret JHwxCOYDaI4jCdALc9938Lu7piqaNodjByubQhFY
Access level Read-only

The Signature base string and cURL command generated by OAuth tool were:
Signature base string
GET&https%3A%2F%2Fapi.twitter.com%2F1.1%2Fstatuses%2Fuser_timeline.json&oauth_consumer_key%3DQ4hmbd34xSSymdZNPLVzpA%26oauth_nonce%3D63e82bd2a63c4d7a2ff6a265ec1c3bfe%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1359414714%26oauth_token%3D204448593-GHvNZ5FqMETOgH3QVV3M1AZElzyGNXsaGiq2TXRE%26oauth_version%3D1.0%26screen_name%3Dtwitterapi

cURL command
curl --get ‘https://api.twitter.com/1.1/statuses/user_timeline.json’ --data ‘screen_name=twitterapi’ --header ‘Authorization: OAuth oauth_consumer_key=“Q4hmbd34xSSymdZNPLVzpA”, oauth_nonce=“63e82bd2a63c4d7a2ff6a265ec1c3bfe”, oauth_signature=“Fg5ZMBKu%2Ff6j8VT3pwJgIxFq%2BA4%3D”, oauth_signature_method=“HMAC-SHA1”, oauth_timestamp=“1359414714”, oauth_token=“204448593-GHvNZ5FqMETOgH3QVV3M1AZElzyGNXsaGiq2TXRE”, oauth_version=“1.0”’ --verbose

Mike


#3

Thanks @mikebounds! This is EXACTLY what I was looking for. I create a script that does this and share it in github (https://github.com/gianu/latest_tweets).

Thanks again!


#4

Very helpful. Thanks!


#5

How do you send the request in urlencoded format? It seems it is in cgi-bin format.


#6

I am not sure what you mean , the signature sent in the request is URL encoded by the
sed -e s’/+/%2B/’ -e s’///%2F/’ -e s’/=/%3D/’ at the end.

I noticed that the command used to set signature is not shown as enclosed in back quotes - I went to edit, but backquotes are there, but don’t show, so you need a back quote between signature and = and one after the sed statement at the end. But sgianazza gianu has provided the script using parameter for keys which I had already done after writing my original post, but hadn’t got round to uploading.

Mike


#7

Hello, thanks a lot for this tutorial, it is working great for me. I took the @sgianazza script from github and tried to modify it to instead of reading tweets, posting tweets with an image attached.

The resource URL is:
https://api.twitter.com/1.1/statuses/update_with_media.json

and the requests:
media[]=/photo.png&status=Test

And I followed the documentation:
https://dev.twitter.com/docs/api/1.1/post/statuses/update_with_media

The scripts gives me an error that says that i couldn’t be authenticated. But the original script of @sgianazza works fine. I’ve searched more info about the error here but nothing seemed to work. It would be great if someone that knows what is going on could help me, and other people that maybe needs the script.

And finally the script is:

consumer_key=KEY
consumer_secret=KEY
oauth_token=KEY
oauth_secret=KEY

timestamp=date +%s

nonce=date +%s%T555555555 | openssl base64 | sed -e s'/[+=/]//g'

signature_base_string=“POST&https%3A%2F%2Fapi.twitter.com%2F1.1%2Fstatuses%2Fupdate_with_media.json&media%3D%252Fphoto.png%26oauth_consumer_key%3D${consumer_key}%26oauth_nonce%3D${nonce}%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D${timestamp}%26oauth_token%3D${oauth_token}%26oauth_version%3D1.0%26status%3DTest”

signature_key="${consumer_secret}&${oauth_secret}"

oauth_signature=echo -n ${signature_base_string} | openssl dgst -sha1 -hmac ${signature_key} -binary | openssl base64 | sed -e s'/+/%2B/' -e s'/\//%2F/' -e s'/=/%3D/'

header=“Authorization: OAuth oauth_consumer_key=”${consumer_key}", oauth_nonce="${nonce}", oauth_signature="${oauth_signature}", oauth_signature_method=“HMAC-SHA1”, oauth_timestamp="${timestamp}", oauth_token="${oauth_token}", oauth_version=“1.0"”

result=curl --request 'POST' 'https://api.twitter.com/1.1/statuses/update_with_media.json' --data 'media%5B0%5D=%2Fphoto.png&status=Test' --header "Content-Type: application/x-www-form-urlencoded" --header "${header}" --verbose

echo $result

And the output is:

  • About to connect() to api.twitter.com port 443 (#0)
  • Trying 199.16.156.231… % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0connected
  • Connected to api.twitter.com (199.16.156.231) port 443 (#0)
  • successfully set certificate verify locations:
  • CAfile: none
    CApath: /etc/ssl/certs
  • SSLv3, TLS handshake, Client hello (1):
    } [data not shown]
  • SSLv3, TLS handshake, Server hello (2):
    { [data not shown]
  • SSLv3, TLS handshake, CERT (11):
    { [data not shown]
  • SSLv3, TLS handshake, Server finished (14):
    { [data not shown]
  • SSLv3, TLS handshake, Client key exchange (16):
    } [data not shown]
  • SSLv3, TLS change cipher, Client hello (1):
    } [data not shown]
  • SSLv3, TLS handshake, Finished (20):
    } [data not shown]
  • SSLv3, TLS change cipher, Client hello (1):
    { [data not shown]
  • SSLv3, TLS handshake, Finished (20):
    { [data not shown]
  • SSL connection using RC4-SHA
  • Server certificate:
  •    subject: C=US; ST=California; L=San Francisco; O=Twitter, Inc.; OU=Twitter Security; CN=api.twitter.com
    
  •    start date: 2013-04-08 00:00:00 GMT
    
  •    expire date: 2013-12-31 23:59:59 GMT
    
  •    subjectAltName: api.twitter.com matched
    
  •    issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)09; CN=VeriSign Class 3 Secure Server CA - G2
    
  •    SSL certificate verify ok.
    

POST /1.1/statuses/update_with_media.json HTTP/1.1
User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.15 libssh2/1.2.6
Host: api.twitter.com
Accept: /
Content-Type: application/x-www-form-urlencoded
Authorization: OAuth oauth_consumer_key=“KEY”, oauth_nonce=“KEY”, oauth_signature=“KEY”, oauth_signature_method=“HMAC-SHA1”, oauth_timestamp=“KEY”, oauth_token=“KEY”, oauth_version="1.0"
Content-Length: 37

} [data not shown]
< HTTP/1.1 401 Unauthorized
< content-length: 63
< content-type: application/json; charset=utf-8
< date: Fri, 25 Oct 2013 20:53:47 UTC
< server: tfe
< set-cookie: guest_id=v1%3A138273442725534346; Domain=.twitter.com; Path=/; Expires=Sun, 25-Oct-2015 20:53:47 UTC
< strict-transport-security: max-age=631138519
<
{ [data not shown]
158 63 100 63 0 37 111 65 --:–:-- --:–:-- --:–:-- 150* Connection #0 to host api.twitter.com left intact

  • Closing connection #0
  • SSLv3, TLS alert, Client hello (1):
    } [data not shown]
    {“errors”:[{“message”:“Could not authenticate you”,“code”:32}]}

#8

It looks like you’re including the “status” parameter as part of your signature base string – when doing a multipart POST, the OAuth signature base string should have oauth_* parameters exclusively.

[node:22194] is a newer guide that you may find helpful.


#9

Thanks, finally I installed twurl and made a script with it, and it works great!


#10

I need a help here… I am following the same procedure provided by @mikebounds but I am trying to do a POST.

I am using the OAuth tool in my apps profile and I added:

Request URL:
https://api.twitter.com/1.1/statuses/update.json

and in the Request Query:
status=HELLO&display_coordinates=false

However, I am receiving “HTTP/1.1 401 Unauthorized” and the permission is set to “READ/WRITE and DIRECT Messages”

Someone have idea what is really going on ? The same GET api called by @mikebounds worked for me but the POST no way!


#11

I tried to execute my cURL command generated from my app created on Twitter. But keeps timing out and says Failed to connect to the URL specified. what could the problem be? I have installed the cURL v7.33 and am trying to execute the command from the Command prompt meant for the Windows 7 x64bit… Please help!!


Thanks,
Hariharan S


#12

How can I modify this to do a search instead of focusing on one particular user ?


#13

Hi,

could you please help me, I installed twurl in CentOS 6.5, but whenever i try to do ‘twurl authorize --consumer-key XXXXXX–consumer-secret XXXXXX’, but i always get ‘Authorization failed. Check that your consumer key and secret are correct, as well as username and password.’, is there anything I am missing here? i really wanna start using twurl instead of php libs I currently use.

Thanks


#14

Excellent explanation, thanks. Very useful.

I have had the same issue as @manoelramon2011 - I could GET but not POST, despite having the correct read/write permissions set in the application.

It turns out that if your access tokens were generated as read-only then they need to be regenerated once the permission has been changed to read/write. Otherwise you will only be able to read with them, despite the application being listed as having write permissions.

This manifested as 401 unauthorized errors for me.

Hope this help.