Exception in thread "Twitter4J Async Dispatcher[0]" while fetching tweets using flume

[priyalcss]

following is the error.

Exception in thread “Twitter4J Async Dispatcher[0]” java.lang.NoSuchMethodError: twitter4j.MediaEntity.getMediaURLHttps()Ljava/lang/String;
at org.apache.flume.source.twitter.TwitterSource.extractRecord(TwitterSource.java:258)
at org.apache.flume.source.twitter.TwitterSource.onStatus(TwitterSource.java:157)
at twitter4j.StatusStreamImpl.onStatus(StatusStreamImpl.java:78)
at twitter4j.AbstractStreamImplementation$1.run(AbstractStreamImplementation.java:107)
at twitter4j.internal.async.ExecuteThread.run(DispatcherImpl.java:114)
2017-08-10 11:36:58,290 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:/etc/flume/conf/tweets.conf for changes
2017-08-10 11:37:28,292 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:/etc/flume/conf/tweets.conf for changes
2017-08-10 11:37:51,099 (Twitter Stream consumer-1[Receiving stream]) [INFO - twitter4j.internal.logging.SLF4JLogger.info(SLF4JLogger.java:83)] Stream closed.
2017-08-10 11:37:51,101 (Twitter Stream consumer-1[Receiving stream]) [ERROR - org.apache.flume.source.twitter.TwitterSource.onException(TwitterSource.java:331)] Exception while streaming tweets
Stream closed.Relevant discussions can be found on the Internet at:
http://www.google.co.jp/search?q=70971d2e or
http://www.google.co.jp/search?q=000683fe
TwitterException{exceptionCode=[70971d2e-000683fe 70971d2e-000683ad], statusCode=-1, retryAfter=-1, rateLimitStatus=null, featureSpecificRateLimitStatus=null, version=2.2.6}
at twitter4j.AbstractStreamImplementation.handleNextElement(AbstractStreamImplementation.java:167)
at twitter4j.StatusStreamImpl.next(StatusStreamImpl.java:67)
at twitter4j.TwitterStreamImpl$TwitterStreamConsumer.run(TwitterStreamImpl.java:442)
Caused by: javax.net.ssl.SSLException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:596)
at sun.security.ssl.InputRecord.read(InputRecord.java:532)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:552)
at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:609)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:696)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3375)
at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:238)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at twitter4j.AbstractStreamImplementation.handleNextElement(AbstractStreamImplementation.java:86)
… 2 more

[andypiper]

Caused by: javax.net.ssl.SSLException: SSL peer shut down incorrectly

Sounds like an SSL connection problem. What version of Twitter4J is in use here?

[priyalcss]

Version of Twitter4j is 2.2.6

I am using twitter4j-core-2.2.6.jar , twitter4j-media-support-2.2.6.jar , twitter4j-stream-2.2.6.jar and flume-sources-1.0-SNAPSHOT.jar files.

[andypiper]

That version of Twitter4J is very old and probably doesn’t support the switch to TLS over SSL that happened a couple of years ago.

[priyalcss]

I tried using Twitter4j-3.0.4 and Twitter4j-3.0.5 but still i got the same error.

[priyalcss]

In flume-env.sh file , i added the CLASSPATH

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk.x86_64
export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"

FLUME_CLASSPATH="/usr/hdp/2.5.0.0-1245/flume/lib/*"

export HIVE_HOME=/usr/lib/hive
export HCAT_HOME=/usr/lib/hive-hcatalog

And here is the flume command which i used for execution,

flume-ng agent -n TwitterAgent --conf /etc/flume/conf -f /etc/flume/conf/tweets.conf -Dflume.root.logger=DEBUG,console -Dtwitter4j.streamBaseURL=https://stream.twitter.com/1.1/

[andypiper]

We don’t directly support either Flume or Twitter4J, they are both provided by third parties. The last time I came across issues with Flume I think it was traced to the project using a very old version of Twitter4J (current release is 4.0.5 or 4.0.6 I think). I found a couple of StackOverflow posts that talk about the specific Java error that you’re seeing, and that would seem to suggest that it might be SSL/TLS, which is what I suspected.

I’m not able to reproduce this locally, and I’ve verified that I’m able to connect to the streaming API endpoint using twurl, so I know there is no issue there.

I would recommend raising the issue with the Flume project directly, unless someone else on these forums is able to suggest what might be wrong here.