Also, I want to share my frustration just to be sure the platform team understands why and how people are/could be consuming the service. I can’t speak for everyone, but what i’m trying to build is EXTREMELY low volume, except twitter offers me no way to get this information.
The only data i’m trying to get from twitter is knowing when a user favorites a tweet so my app can react to it. When I first approached the problem, site streams made perfect sense. It was literally, less than 30 lines of code. We’re talking 3-4 favorites a DAY from my authorized users. From what I can tell the REST and STREAM apis are being architected to solve a single use case, rebuilding status timelines. I’m not sure if that’s an internal business goal, (you only want people building consumption clients), or something else.
I went from 30 lines of code, which if filtered, consumed only literal bytes of data (only looking for uid and timestamp of favorite) to constantly fetching (deep crawl) to the absolute maximum amount allowed # of requests, in order to have a pseudo real time system. Not only do I have to slam twitter as often as possible to find data that may change ONCE in a single week, i now have to build out this extremely complex polling system. If you you are wondering why I have to slam twitter with a deep paginated crawl as often as possible… How to get all new favorites of user foo since last fetch?
Is there anyway you can meet us in the middle? I’m not sure if believe you that it’s to keep abuse in check, it would be pretty doable to restrict and then bless the access keys of your partners. I’m an individual developer with 10 users who is now hitting you with 9000 REST requests an hour, instead of 6 open streams (would only be 1 if i had site streams)
You definitely need to fix the documentation ASAP. At the very least, remove any copy/messaging that says build with user streams first before applying. Below the header on user streams you should also mention “If you are using this endpoint to apply for site streams, stop now.” I think it would be nice if you also stopped beating around the bush with user streams and clarify that they are intended for end clients only and if you have any implementation that does not run from the client, you’re doing it wrong. I can’t think of a single context where 6 connection limit would ever make sense from a single source.
The messaging is waaaaaaaaaaaay to casual. They should really be taken down and passed around directly to your partners. Having them out there, admitting to favored partners, and telling the rest of us to reach out to gnip sales is a poopy feeling