Flume Authentication Error 401

[shanemeister1]

I searched the internet for hours trying to solve this problem. I found several discussion threads on the subject, but none seem to provide a fix.

When I run the following flume-ng command:
flume-ng agent --conf-file flume.conf --name TwitterAgent -Dflume.root.logger=INFO,console

I get the following error:
17/03/13 05:54:34 INFO twitter4j.TwitterStreamImpl: 401:Authentication credentials (https://dev.twitter.com/pages/auth) were missing or incorrect. Ensure that you have set valid consumer key/secret, access token/secret, and the system clock is in sync.

I know my keys are ok. I tested it by pulling tweets with a python script, and they work fine. I set my computer clock with ntp.

Here is my flume.conf file:

# Naming the components on the current agent.
TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = twitterSink

# Describing/Configuring the source
TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.consumerKey = <consumerKey>
TwitterAgent.sources.Twitter.consumerSecret = <consumerSecret>
TwitterAgent.sources.Twitter.accessToken = <accessToken> 
TwitterAgent.sources.Twitter.accessTokenSecret =  <accessTokenSecret> 

# Describing/Configuring the sink
TwitterAgent.sinks.twitterSink.type = org.apache.flume.sink.kafka.KafkaSink
TwitterAgent.sinks.twitterSink.kafka.topic = data
TwitterAgent.sinks.twitterSink.kafka.bootstrap.servers = localhost:9092
TwitterAgent.sinks.twitterSink.channel = channel1
TwitterAgent.sinks.twitterSink.kafka.flumeBatchSize = 20
TwitterAgent.sinks.twitterSink.kafka.producer.acks = 1
TwitterAgent.sinks.twitterSink.kafka.producer.linger.ms = 1
TwitterAgent.sinks.twitterSink.kafka.producer.compression.type = snappy


# Describing/Configuring the channel
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 100

# Binding the source and sink to the channel
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sinks.twitterSink.channel = MemChannel

Any help will be greatly appreciated.
Thanks!

[andypiper]

Thanks for confirming that you’ve tested the keys / time via an alternative script, that’s helpful. It sound like this must be isolated to the way Flume is connecting, then.

Is your Flume set up pulling in the most recent version of twitter4j?

[shanemeister1]

@andypiper Thanks for the quick response.
That is one thing I wanted to try. I downloaded twitter4j-4.0.4.zip from twitter4j.org but don’t know how to install it. I am not a Java programmer (more of hack).

What I did do was unzipped it, located the three jar files and placed them in /usr/local/flume/lib directory where the 3.x jars were located. I also downloaded the twitter-4j-4.0,5-SNAPSHOT.zip, but I was not able to find any information on where to go from here.

Again, any help is greatly appreciated!

[andypiper]

The last time this came up there was a fairly protracted thread about it and it did require technical intervention. At least one poster offered their own version of the Flume jar (use at your own discretion).

In the longer term it would be good to encourage the Apache Flume maintainers to get their code updated to v4.x of Twitter4J as the non-https, v1 streaming endpoints are no longer supported. It looks like an issue was raised with them back in October but it has not been updated so far.

[shanemeister1]

@andypiper Yes, I read the lengthy thread and it didn’t appear that anyone had a good answer there other than to determine that it was a bug and needed to be fixed. I am really surprised that Apache Flume would let this one sit for so long. Must be really busy.
Anyway, thanks for your help, and I will have to reevaluate my use of Flume in the architecture.
THANKS!