I was hoping to get some guidance on building an architecture for a cloud based twitter analysis service. Suppose I want to build a platform where users can see some analytics based on certain keywords/hashtags they choose. Users can track keywords for days or months and analyse their own respective results. Now to do so I was thinking of employing streaming API to get real time feeds for these keywords. Now suppose I have 100 users following new keywords on a daily basis. I cannot have a single stream for following reasons:
1). Rate limit of 1 users keywords affecting other users part of the stream
2). Restarting stream multiple times a day whenever a new keyword is added
Another option is to create a NEW stream for each user, but then:
1). I have to create a new twitter app with API key and access token for each stream
2). Number of streams grow exponentially and I believe my IP gets blacklisted
Can someone help me on what architecture needs to be followed ideally? I know getting data from Gnip/DataSift may be an option but I dont have that sort of budget.
Looking forward to your insights