Clarification regarding User Stream Connection limitations

[redskins80]

The documentation for connection limitations for user streams seem ambiguous to me. Here is my specific question.

We need to open a user stream for every user (end user) on our platform. Every connection we open will be using a different user access token. But all connections will be using the same OAuth Application token. And also lets assume all connections will be from the same server IP address.

What kind of rate limiting will I be subjected to?

[pi_kumar]

In this case it is likely that you will receive the default user stream limit which about 6/min. Can you provide some more information around why you are using user stream vs site stream?

[redskins80]

Thanks for taking the time for answering my question.

In short, we are using user streams because site steams are not accepting applications anymore.

Currently we plan to work around this entire issue by having 5-6 user streams open on a single IP and use multiple EC2 instances to accommodate more streams. So you can safely assume we will have at some point around 50-60 streams connected over roughly around 10 systems (10 IPs). But this would only work if we are sure that Twitter does not rate limit at all on the basis of OAuth application token. Also, is this method ethical and would Twitter be OK with this?

So, although the documentation doesn’t suggest it, we still need to confirm whether twitter ever rate limits streams based on OAuth application token.

If it doesn’t, then we can go for the workaround using multiple EC2 instances. Else using user streams will be difficult. We will have to resort to polling the REST APIs.

[ayushchd]

Could we please have some clarification regarding this? I have seen a lot of similar questions around here but none have a clear answer.

[rromanchuk]

We’re basically screwed. What you’re asking for is site streams, but site streams are dead. Even twitter says “build on user streams first, then apply for site streams” It’s complete waste of developers’ time and needs to be removed from documentation. It’s back to complicated polling architecture and slamming the REST API.

The 6 connections “limit” for user streams is completely bogus. If i’m opening user streams with valid tokens, why does twitter care how many I have open? I would really love some feedback internally from twitter about why this restriction exists.

They are clearly uncomfortable with site streams, but i’m not exactly sure what they are afraid of.

[ayushchd]

Pretty worrying to see that Twitter devs are right out ignoring this thread. It would be really appreciated if they could just clarify the doubts. Even if the restrictions are stringent, that’s fine, we would just like to have a first-hand confirmation from them.

[rromanchuk]

Not sure if admin or community, but my other post just got flagged

https://twittercommunity.com/t/low-volume-actions-favorites-need-to-be-searchable-by-date-or-at-least-have-favorited-at-in-meta-response/50067/1

Same problem many developers have who work with anything besides timelines

Favorites List Doesn't Add Newly-Favorited Tweets To The Top

I'm working on an app that grabs all of a user's favorited tweets upon signup and later requests a user's newly-favorited tweets upon user request. Using twitter's Working with timelines documentation, it's pretty straight-forward to grab a user's current favorited tweets using the max_id parameter. Which starts at the very top of the list and works it's way down. The documentation also explains how to use the since_id param to grab new tweets added to a timeline i.e. GET statuses / user_t…

And still no response to

Please remove all mention of site streams from your documentation - don't waste developers time

Hi Twitter, I love you, but it's really sadistic that you still have site stream documentation plastered everywhere. I just wasted a colossal amount of time building an app with user streams, in the hopes that one day, site streams could be a real option. After stalking all my friends who work at twitter for months trying to find who is at the other end of that google form, it's pretty clear that this is dead in the water, or 100% shall grant (giant-biz-dev-deals). Can you please remove this do…

I’m honestly not trying to be a dick, but if twitter doesn’t want to support streams for internal business reasons, they need to communicate this a lot more clearly. I’m beginning to think twitter doesn’t know what twitter is doing.

[rromanchuk]

Can you provide some more information around why you are using user stream vs site stream?

Because we don’t have access to site streams…

[ayushchd]

Could you please update on this?

[andypiper]

Apologies for the lack of responses to this thread while some of the team have been away from the office.

The public and user streams are indeed limited on based on tokens and per-IP-based connection restrictions (perhaps 6 connections before your connections are recycled). This is in addition to the 1% of traffic cap that exists for these streams, and repeated reconnection attempts will lead to 420 error codes.

As I explained on another thread this week, we also have a developer agreement and policy that states you should not circumvent the limits by creating multiple instances of your apps etc. This is to attempt to ensure fair access and a stable platform for everyone. We’re increasingly proactive in reviewing application activity to maintain a clean platform that everyone gets to enjoy.

In short, the public streaming API was not built for high volume, complex filter connections and is not intended to be a full enterprise solution.

As @rromanchuk has said here and on another thread, we don’t have any plans to expand the site streams program. I’ve personally taken his comments on board and plan to revise the docs to hopefully reduce confusion in these areas. It is less simple to completely remove the documentation, as a number of partners are using those APIs and still need the reference docs available. That said, we will de-emphasise them.

I realise that these responses will be frustrating. We do offer a paid solution (GNIP) which I’m aware doesn’t fulfil everyone’s requirements particularly when it comes to cost, but these are the options available.

If you’d like additional / elevated access or permissions for your app, we have Policy Support forms where you are able to describe the purpose and intent of your application so that we can evaluate the requirement. I’ll be open about this too, and point out that elevated access is granted less frequently as the platform grows, stability is paramount, and as we strive to reduce the number of bad actors and other frustrating misuses of the system.

[rromanchuk]

Also, I want to share my frustration just to be sure the platform team understands why and how people are/could be consuming the service. I can’t speak for everyone, but what i’m trying to build is EXTREMELY low volume, except twitter offers me no way to get this information.

The only data i’m trying to get from twitter is knowing when a user favorites a tweet so my app can react to it. When I first approached the problem, site streams made perfect sense. It was literally, less than 30 lines of code. We’re talking 3-4 favorites a DAY from my authorized users. From what I can tell the REST and STREAM apis are being architected to solve a single use case, rebuilding status timelines. I’m not sure if that’s an internal business goal, (you only want people building consumption clients), or something else.

I went from 30 lines of code, which if filtered, consumed only literal bytes of data (only looking for uid and timestamp of favorite) to constantly fetching (deep crawl) to the absolute maximum amount allowed # of requests, in order to have a pseudo real time system. Not only do I have to slam twitter as often as possible to find data that may change ONCE in a single week, i now have to build out this extremely complex polling system. If you you are wondering why I have to slam twitter with a deep paginated crawl as often as possible… How to get all new favorites of user foo since last fetch?

Is there anyway you can meet us in the middle? I’m not sure if believe you that it’s to keep abuse in check, it would be pretty doable to restrict and then bless the access keys of your partners. I’m an individual developer with 10 users who is now hitting you with 9000 REST requests an hour, instead of 6 open streams (would only be 1 if i had site streams)

You definitely need to fix the documentation ASAP. At the very least, remove any copy/messaging that says build with user streams first before applying. Below the header on user streams you should also mention “If you are using this endpoint to apply for site streams, stop now.” I think it would be nice if you also stopped beating around the bush with user streams and clarify that they are intended for end clients only and if you have any implementation that does not run from the client, you’re doing it wrong. I can’t think of a single context where 6 connection limit would ever make sense from a single source.

The messaging is waaaaaaaaaaaay to casual. They should really be taken down and passed around directly to your partners. Having them out there, admitting to favored partners, and telling the rest of us to reach out to gnip sales is a poopy feeling

[ayushchd]

Thanks for responding, Andy. I would still like to clarify, that moving ahead, are you completely shutting down the Streaming API for developers? Or is there still an off-chance that if we build a POC with user-streams and apply, we might still get through the program?

On a side note, it seems quite unfair that there are some developers who have gotten access to the site streams program and now the developers are just blocked. Or is it that have you started charging those developers for the same? Or are you revoking access for existing devs? Would be great if you could share some more details regarding the site streams.