Can twitter Streaming API be used with Apache Flume


#1

Hi all,

I am new to twitter APIs, I have used the search API with the Apache Flume but I am not sure if we can use streaming API with Flume.

So can anyone tell me if I can use the streaming API with Apache Flume?


#2

I haven’t personally used Flume, but from reading the documentation it appears that it does indeed work with the Streaming API. This looks like example code to listen to the sample endpoint.


#3

I know 2 flume connectors for the twitter streaming endpoint (see below). There’s surely more available and/or you can code one yourself.

org.apache.flume.source.twitter.TwitterSource
this one is usually already part of your hadoop distrib

com.cloudera.flume.source.TwitterSource
this one must be downloaded from cloudera’s website

Hope this helps,
Eric.