I am trying to build an application that will capture user streams at scale.
Ideally, users will signup for my app by using the sign in with Twitter auth flow. The app will then monitor their feed (tweets from everyone they follow). The app then processes each of those tweets and flags tweets that meet certain criteria.
My initial thought was to create a new user stream for every user that signed up. For each stream, I would check each tweet in real-time. The problem is with scalability.
I realize that I may have to give up the “real time” monitoring for a less-expensive polling option using the REST API instead of the User Stream API.
My question is what is the best way to achieve constant user stream monitoring at scale?