Logstash Twitter Authorisation error

[DnB_LeighP]

Anyone got any experiance of using Twitter with Logstash?

I’ve looked everywhere to find an answer to the below error but can only find a firewall issue (so tried outside the firewall) and the time stamp being different between the twitter server and my machine (machine timing set correctly). Can anyone advise what’s causing the below authorisation error (I’ve also checked the twitter app and settings are correct and working).

Logstash startup completed

←[33m {:exception=>Twitter::Error::Unauthorized, :backtrace=>[“C:/logstash-1.5.1
/vendor/bundle/jruby/1.9/gems/twitter-5.12.0/lib/twitter/streaming/response.rb:2
1:in on_headers_complete’”, “org/ruby_http_parser/RubyHttpParser.java:370:in<
<’”, “C:/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/twitter-5.12.0/lib/twitter/
streaming/response.rb:16:in <<’”, “C:/logstash-1.5.1/vendor/bundle/jruby/1.9/ge
ms/twitter-5.12.0/lib/twitter/streaming/connection.rb:22:instream’”, “C:/logst
ash-1.5.1/vendor/bundle/jruby/1.9/gems/twitter-5.12.0/lib/twitter/streaming/clie
nt.rb:116:in request’”, “C:/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/twitter
-5.12.0/lib/twitter/streaming/client.rb:36:infilter’”, “C:/logstash-1.5.1/vend
or/bundle/jruby/1.9/gems/logstash-input-twitter-0.1.6/lib/logstash/inputs/twitte
r.rb:88:in run’”, “C:/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/logstash-core
-1.5.1-java/lib/logstash/pipeline.rb:176:ininputworker’”, “C:/logstash-1.5.1/v
endor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:17
0:in `start_input’”], :level=>:warn}←[0m

I’m using the below twitter config:-

input {
twitter {
consumer_key => "xxxxxxxxxxxxxxxxxx"
consumer_secret => "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
oauth_token => "xxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
oauth_token_secret => "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
keywords => ["xxxxxxxxxxx"]
full_tweet => true
}
}

output {
stdout { codec => dots }
elasticsearch {
protocol => "http"
host => "localhost"
index => "twitter"
document_type => "tweet"
template => "twitter_template.json"
template_name => "twitter"
}
}

Thanks

[DnB_LeighP]

I’ve tried this with a different set of Twitter app details and now get Connection refused. Any ideas?

←[33mConnection refused - Connection refused {:exception=>#<Errno::ECONNREFUSED:
Connection refused - Connection refused>, :backtrace=>[“org/jruby/ext/socket/Ru
byTCPSocket.java:126:in initialize'", "org/jruby/RubyIO.java:853:innew’”, “C:
/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/twitter-5.12.0/lib/twitter/streamin
g/connection.rb:16:in stream'", "C:/logstash-1.5.1/vendor/bundle/jruby/1.9/gems /twitter-5.12.0/lib/twitter/streaming/client.rb:116:inrequest’”, “C:/logstash-
1.5.1/vendor/bundle/jruby/1.9/gems/twitter-5.12.0/lib/twitter/streaming/client.r
b:36:in filter'", "C:/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/logstash-inpu t-twitter-0.1.6/lib/logstash/inputs/twitter.rb:88:inrun’”, “C:/logstash-1.5.1/
vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:1
76:in inputworker'", "C:/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/logstash-c ore-1.5.1-java/lib/logstash/pipeline.rb:170:instart_input’”], :level=>:warn}←[
0m

[Martin_J_Ed]

Hi Leigh

I’ve got the same issue and get the following:

{:exception=>Twitter::Error::Unauthorized, :backtrace=>["/home/administrator/logstash/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/twitter-5.12.0/lib/twitter/streaming/response.rb:21:in on_headers_complete'", "org/ruby_http_parser/RubyHttpParser.java:370:in<<’", “/home/administrator/logstash/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/twitter-5.12.0/lib/twitter/streaming/response.rb:16:in <<'", "/home/administrator/logstash/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/twitter-5.12.0/lib/twitter/streaming/connection.rb:22:instream’”, “/home/administrator/logstash/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/twitter-5.12.0/lib/twitter/streaming/client.rb:116:in request'", "/home/administrator/logstash/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/twitter-5.12.0/lib/twitter/streaming/client.rb:36:infilter’”, “/home/administrator/logstash/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/logstash-input-twitter-1.0.1/lib/logstash/inputs/twitter.rb:88:in run'", "/home/administrator/logstash/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:177:ininputworker’”, “/home/administrator/logstash/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:171:in `start_input’”], :level=>:warn}

Did you get anywhere with your question you’d posted in August? I’ll have a look at github too and will let you know in case I’ll get answers.

Regards
Martin

[DnB_LeighP]

Hi Martin. Unfortunately still not managed to resolve this. I believe it may be a local machine configuration issue however I’m trying to prove that out. As soon as I get to the bottom of it I will post a solution on here.

[Martin_J_Ed]

Thank you Leigh,

Much appreciated and thanks for the quick reply. I’ll let you know as well in case I find something.

Have a good weekend.

Regards
Martin

[andypiper]

I’m not familiar with the tools you’re using, but one thing of note is that from the crash reports, Logstash is embedding the Ruby twitter gem at version 5.12, which is about 18 months old. That said, I don’t see anything in the changelog that refers to auth or network changes in the intervening versions, but I do know that Twitter API endpoints no longer support older SSL versions or weak SHA certificates. Beyond those hints, I’m not sure what to suggest, I’m afraid.

[Martin_J_Ed]

Thank you very much.

I’ve tried different versions, i.e. 1.4.3, 1.5.3 and 1.5.4
All give me the same result. I’m pretty sure when I was running 1.4.? in May this year it was working fine.

Very likely it’s a security certificate the Twitter API doesn’t like.

I’ll keep in touch if I find anything.

Regards
Martin

[Martin_J_Ed]

Quick update,

I’ve used the same twitter config at home when directly connected to the internet via EE (UK ISP).
That worked fine, I was able to source some tweets.

But when I’m at work it doesn’t work. We use a proxy as most companies do. A colleague checked for me and it doesn’t look like logstash is using the proxy. I’ll try the logstash forum.

Leigh, any chance you’re using a proxy too?

[DnB_LeighP]

Sounds like different problems. I have the same problem when going direct over the internet but when I run the same config on a colleagues machine it works fine.

For me it looks like it could be something to do with my local machine setup. I’ll let you know if I get any further.

[Triconex]

Hello!

Check your system date and time. I found my one to be wrong. After I corrected it, no more Twitter Error Auth messages. All credits to https://gist.github.com/rmoff/400ee2783a2d3f14c412

Good luck

[andypiper]

Ah, glad that fixed it for you! There’s a time element involved in OAuth so this makes perfect sense.

[Audrey_Neveu]

Hi all,

I’m facing the same issue and I know for sure it’s not coming from my credentials as I’m using them in two other configuration files on the same environnement and it works perfectly.

The only differences with the last one is that I’m using follows instead of keywords as I want to retrieve the twitter feed from an account and not only from tags.

Here is the input part of my configuration file:
input { twitter { consumer_key => "xxx" consumer_secret => "xxx" oauth_token => "xxx" oauth_token_secret => "xxx" follows => [ "@elastic" ] full_tweet => true } }

And here is my stacktrace

{:exception=>Twitter::Error::Unauthorized, :backtrace=>["/Users/audrey/Dev/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/twitter-5.15.0/lib/twitter/streaming/response.rb:21:inon_headers_complete'", "org/ruby_http_parser/RubyHttpParser.java:370:in <<'", "/Users/audrey/Dev/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/twitter-5.15.0/lib/twitter/streaming/response.rb:16:in<<'", "/Users/audrey/Dev/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-input-twitter-2.2.0/lib/logstash/inputs/twitter/patches.rb:31:in stream'", "/Users/audrey/Dev/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-input-twitter-2.2.0/lib/logstash/inputs/twitter/patches.rb:58:inrequest'", "/Users/audrey/Dev/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/twitter-5.15.0/lib/twitter/streaming/client.rb:37:in filter'", "/Users/audrey/Dev/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-input-twitter-2.2.0/lib/logstash/inputs/twitter.rb:128:inrun'", "/Users/audrey/Dev/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:206:in inputworker'", "/Users/audrey/Dev/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:199:instart_input'"], :options=>nil, :level=>:warn}`

Perhaps I’m not using the follows attribute the right way? I’ve tried with or without the @ before the id but it makes no difference.

I’ve also tried to add use_samples => true as mentionned by @mobcdi and it works but indexed tweets are not relevant.

Any help highly appreciated