How can streaming application have different message queues?


#1

Considering the example mentioned on your GitHub project (https://github.com/twitter/hbc/blob/master/hbc-example/src/main/java/com/twitter/hbc/example/FilterStreamExample.java), lets take a scenario:

I have 3 users: User A, User B and User C

User A: wants all the tweets for "Brazil"
User B: wants all the tweets for "Germany"
User C: wants all the tweets for “San Francisco”

From example on Github

BlockingQueue queue = new LinkedBlockingQueue(10000);
StatusesFilterEndpoint endpoint = new StatusesFilterEndpoint();
// add some track terms
endpoint.trackTerms(Lists.newArrayList(“twitterapi”, “#yolo”));

Authentication auth = new OAuth1(consumerKey, consumerSecret, token, secret);
// Authentication auth = new BasicAuth(username, password);

// Create a new BasicClient. By default gzip is enabled.
Client client = new ClientBuilder()
        .hosts(Constants.STREAM_HOST)
        .endpoint(endpoint)
        .authentication(auth)
        .processor(new StringDelimitedProcessor(queue))
        .build();

// Establish a connection
client.connect();

Now if I need a different messageQueue for each user, I will have to create 3 client and connect 3 times. When I do that, I see error as

[0m[0m16:04:06,946 INFO  [com.twitter.hbc.httpclient.BasicClient] (default task-6) New connection executed: tweetStream-client, endpoint: /1.1/statuses/filter.json?delimited=length&stall_warnings=true
[0m[0m16:04:06,947 INFO  [com.twitter.hbc.httpclient.ClientBase] (hosebird-client-io-thread-0) tweetStream-client Establishing a connection
[0m[33m16:04:07,660 WARN  [com.twitter.hbc.httpclient.ClientBase] (hosebird-client-io-thread-0) tweetStream-client Error connecting w/ status code - 420, reason - Enhance Your Calm
[0m[0m16:04:12,662 INFO  [com.twitter.hbc.httpclient.ClientBase] (hosebird-client-io-thread-0) tweetStream-client Done processing, preparing to close connection
[0m[0m16:04:12,665 INFO  [com.twitter.hbc.httpclient.ClientBase] (hosebird-client-io-thread-0) tweetStream-client Establishing a connection
[0m[31m16:04:12,665 ERROR [com.twitter.hbc.httpclient.ClientBase] (hosebird-client-io-thread-0) tweetStream-client Unknown exception while establishing connection to https://stream.twitter.com/1.1/statuses/filter.json?delimited=length&stall_warnings=true: java.lang.IllegalStateException: Connection pool shut down
    at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:167) [httpcore-4.2.4.jar:4.2.4]
    at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:199) [httpcore-4.2.4.jar:4.2.4]
    at org.apache.http.impl.conn.PoolingClientConnectionManager.requestConnection(PoolingClientConnectionManager.java:188) [httpclient-4.2.5.jar:4.2.5]
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:448) [httpclient-4.2.5.jar:4.2.5]
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) [httpclient-4.2.5.jar:4.2.5]
    at org.apache.http.impl.client.DecompressingHttpClient.execute(DecompressingHttpClient.java:137) [httpclient-4.2.5.jar:4.2.5]
    at org.apache.http.impl.client.DecompressingHttpClient.execute(DecompressingHttpClient.java:108) [httpclient-4.2.5.jar:4.2.5]
    at com.twitter.hbc.httpclient.RestartableHttpClient.execute(RestartableHttpClient.java:86) [hbc-core-2.0.2.jar:]
    at com.twitter.hbc.httpclient.Connection.connect(Connection.java:44) [hbc-core-2.0.2.jar:]
    at com.twitter.hbc.httpclient.ClientBase.establishConnection(ClientBase.java:179) [hbc-core-2.0.2.jar:]
    at com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:142) [hbc-core-2.0.2.jar:]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_05]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_05]
    at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_05]

[0m[0m16:04:12,667 INFO  [com.twitter.hbc.httpclient.ClientBase] (hosebird-client-io-thread-0) tweetStream-client exit event - Connection pool shut down
[0m[33m16:04:12,667 WARN  [com.twitter.hbc.httpclient.ClientBase] (hosebird-client-io-thread-0) tweetStream-client failed to establish connection properly
[0m[0m16:04:12,667 INFO  [com.twitter.hbc.httpclient.ClientBase] (hosebird-client-io-thread-0) tweetStream-client Done processing, preparing to close connection
[0m[0m16:04:12,667 INFO  [com.twitter.hbc.httpclient.ClientBase] (hosebird-client-io-thread-0) tweetStream-client Shutting down httpclient connection manager

I tried to identify why this error occurs when I do this and find https://dev.twitter.com/docs/streaming-apis/connecting which says

The client has connected too frequently. For example, an endpoint returns this status if:

A client makes too many login attempts in a short period of time.
Too many copies of an application attempt to authenticate with the same credentials.

My question is simple:
With one registered application and using streaming-api, how can I have separate queues for different items to track without connecting to the client every time. This is so that my application works maintains independent queues for each track-to-follow and without hitting Rate Limits

Thanks a lot for your help in advance