Twitter Streaming API not working with twitter4j and Apache Flume

[saad_9_1]

Hi, Our development team uses Apache Flume along with twitter4j library to consume Twitter streaming API methods. The Streaming service is not working for some time. The authentication is not working with
twitter4j, as the streaming api url is shown to be invalid. Are there any recommendations or information about how to resolve the problem.
Thanks

[andypiper]

What Streaming API URL are you using?

[saad_9_1]

https://stream.twitter.com/1.1/statuses/filter.json

The problem is in authentication. The error says URI is invalid or resouirce requested such as the user doesnot exist.Unknown uri

When tried from https://dev.twitter.com/rest/tools/console , the error says connection limit exceeded.

[umersafeer]

i am working on project sentimental analysis of twitter logs. i am accessing tweets through twitter API using apache flume. it is working fine for me couple of days ago but since 4th may 2016 it shows that error…INFO twitter4j.TwitterStreamImpl: 404:The URI requested is invalid or the resource requested, such as a user, does not exist.
16/05/07 10:36:05 INFO twitter4j.TwitterStreamImpl: Waiting for 10000 milliseconds…
kindly help .i also check this https://dev.twitter.com/overview/status site it shows the stream.api services are disruption… So kindly help me to get rid of that situation.

[IgorBrigadir]

Another thing to check is twitter4j version in your pom.xml latest is 4.0.4:

<dependency> <groupId>org.twitter4j</groupId> <artifactId>twitter4j-stream</artifactId> <version>4.0.4</version> </dependency>

[andypiper]

i also check this https://dev.twitter.com/overview/status site it shows the stream.api services are disruption…

Unfortunately that page is not accurate at the moment - there is no disruption, so the chances are that your code may be pointing at the wrong URL somewhere. You’d need to check inside the Twitter4J or Flume libraries I expect. Apologies for this confusion.

[umersafeer]

evey thing is fine working correctly 4 days ago . every libraries placed in correct directory. so this is not the issue of twitter 4j library.

[andypiper]

Which URL is reported as being invalid?

What is your code doing to connect to the endpoints?

Which version of Twitter4J are you using? Looking at https://github.com/apache/flume/blob/trunk/pom.xml it looks like it pulls in Twitter4J 3.0.3 which is very old and may well be using the old endpoints that were just shutdown around 5 days ago.

Unfortunately we’re not able to manage every piece of third party code that connects to our API, so I don’t have time to debug this in detail for you, but I think that is what you should be looking at.

[umersafeer]

@saad_9_1 do you get the solution i have facing same issue…!

[umersafeer]

flume-source-1.0.SNAPSHOT.jar is the flume directory and i have twitter4j-3.0.3 libraries placed in flume lib folder… should i update twitter4j version and from where i could get them???

[andypiper]

I’m afraid I’m not familiar enough with Flume to know whether Twitter4J 3.0.3 has an issue. You can find the project at http://twitter4j.org though.

[umersafeer]

this is actually my error:

2016-05-10 19:38:09,638 (Twitter Stream consumer-1[Establishing connection]) [DEBUG - twitter4j.internal.logging.SLF4JLogger.debug(SLF4JLogger.java:67)] connection: close
2016-05-10 19:38:09,640 (Twitter Stream consumer-1[Establishing connection]) [DEBUG - twitter4j.internal.logging.SLF4JLogger.debug(SLF4JLogger.java:67)] Unknown URL. See Twitter Streaming API documentation at http://dev.twitter.com/pages/streaming_api
2016-05-10 19:41:11,961 (Twitter Stream consumer-1[Establishing connection]) [INFO - twitter4j.internal.logging.SLF4JLogger.info(SLF4JLogger.java:83)] 404:The URI requested is invalid or the resource requested, such as a user, does not exist.
Unknown URL. See Twitter Streaming API documentation at http://dev.twitter.com/pages/streaming_api

[andypiper]

It seems like something is swallowing the URL / URI rather than reporting it, so I don’t know what the 404 is referring to.

[umersafeer]

its working fine couple of days ago streaming tweets through api effectiviely. so tell me what to do in the situation like ?//

[andypiper]

its working fine couple of days ago streaming tweets through api effectiviely.

Yes, but a few days ago the old endpoints were disabled.

so tell me what to do in the situation

I’m really sorry but I don’t have the bandwidth to dig into this issue myself - I have suggested the likely causes, I think you’ll need to dive into the code to understand what it was doing in interacting with the Twitter API and what you’ll need to change.

[umersafeer]

thats sound great although anyone answering… could you plz defined endpoints???

[andypiper]

The Streaming API and endpoints are documented here.

You might also want to test your tokens and keys using the twurl tool to make a regular connection to confirm that there isn’t some issue with your network connection or something else.

[andypiper]

the error says connection limit exceeded

if you are using the console I would recommend using twurl from the command line to test instead.

[_FlyHigh_high]

Did you solve the problem? I have the exact same issue.

[Naveed23]

Team… Having same issue … basically the approach to use flume sources snapshot was given by Cloudera and for use cases esp sentimental analysis … Can twitter come up with step by step procedure to resolve this ??