Best way to track user-specific (home) and filter streams?



I am working on developing an application that will allow users to dynamically include twitter streams onto a web page. We are utilizing a backend server that is doing most of the heavy lifting, and it handles the connection to the twitter servers. I am still trying to hash out the details of how to operate this service within the bounds of Twitter’s developer policies while maximizing throughput of statuses. The specific requirements I am following are:

  • A User can have their own ‘home’ feed of tweets

  • A User can have a ‘search’ feed of tweets for a particular keyword, hashtag, etc

  • Ideally, we would like this data to be pushed in real time.

  • However, if this is not possible, it should also be acceptable to fall back on a REST api (and simulate streaming on our end), if this increases the ‘throughput’ of statuses retrieved.

From what I understand having tinkered with the User Stream API, it seems there is a hard-limit of around 5-10 streams per App consumer key (if I am not mistaken). My questions are as follows:

  • For the purposes of the requirements mentioned above, is it feasible to use the User Stream API to 1) receive push updates for 1000 clients or less? 500? 100? Or is it locked-in at just 5-10?

  • In the same vein, would it be a preferable design to try to use a user-agnostic “keyword” stream API (aka User A and User B both want to receive tweets about #WorldCup, so we consolidate this into one streaming API link)?

  • If maximum throughput of statuses is the goal, bearing in mind the statuses are 1) from the user’s own ‘home’ feed or 2) from a keyword/hashtag ‘filter’ feed, should I be using the Stream API or the REST API?

Last thing I’ll mention is that the design of the application I am working on would highly benefit from being as real-time as possible, so obviously the Stream API is preferable.