Proper use and care of the Streaming API


#1

I’m looking for documentation, discussion forum posts, and personal experience describing what’s the intended way for a good Twitter partner developer to use the Streaming API. How many connections can a single application make, where each connection would be authorized by a different user? There’s really nothing in the official docs, but obviously the limit can’t be infinite. So it must be somewhere between 0 and infinite – is there any official number between 0 and infinity? If not a hard number, is there a rule-of-thumb?

I believe for Site Streams the number is 100,000, since that’s the maximum number of oauth tokens that can exist for a single application (for certain types of applications; the limit is higher I assume for other types, though the on-line docs don’t say). Well, maybe its less than 100,000 simultaneous connections – you wouldn’t have all 100K users connected at the same time… So maybe its 10K? 1K? 100? 10?

Is there any similar limit number for Public streams (filter stream, etc.)?

Similarly, if I have multiple applications registered under a single developer account, what are the per-developer limits on the Streaming APIs (Site Streams, Public streams, etc.)? I’m sure a single developer can’t have an unlimited number of connections spread across an unlimited number of applications. So what should a developer assume they can do while still staying a good Twitter partner?

I’m looking for sufficient specific, actionable information so that I can design the future scaling of applications and user bases.


#2

Answering myself w.r.t. number of connections per application: see https://dev.twitter.com/discussions/921. Streaming API is designed to be a server-to-server, back-end technology, supporting a very small number your application to Twitter, though it sounds like there is some tolerance for short-term parallel stream to cover certain circumstances.

I truly wish the Streaming API documentation were updated with this information – it seems extremely vital to the intended use of the Streaming API. The documentation implies something quite different: that you can have one stream per app-authorized account!


#3

Hi Brian,

Glad you asked these questions and were able to find some of your own answers. We can definitely formulate the documentation a bit better in this area, and I’ll make sure we get this context more clearly in there. As with most of our API documentation, it’s all grown rather organically by a small team that sometimes is too close to the subject to see “what’s missing.”

The Streaming API auth and rate limiting model was constructed in a pre-OAuth-at-Twitter world. When we added OAuth support to the API, the model didn’t really change – though perhaps it should have. Someday we’ll likely migrate to a model based on application-only auth, taking the “user context” out of the equation entirely, arriving at a more pure application-as-application model.

Some quick info about the Streaming API:

  • The Streaming API didn’t used to support OAuth. It only supported Basic Auth. Under the basic auth model, the application connecting and the user used to represent that application were one and the same thing.

  • Then the Streaming API started supporting OAuth. But the model of what the proper “user” context to use with the Streaming API did not – the user and the application doing the connecting are still one-and-the-same, they’re just connecting through the “vehicle” of a registered OAuth application record.

  • Permissions to use the Streaming API are granted to specific accounts, which represent the concept of the application-streaming API relationship.

  • Someday we’ll alter this model to be more application-centric, in that applications have roles in the Streaming API and users become factored out. You would use application-only auth to connect to a single stream.twitter.com stream in this case and the connecting context thus becomes much clearer.

  • The single stream an application connects to should encompass all possible criteria being matched on behalf of its end-users – track, follow, locations, etc.

  • We will disconnect a stream if a duplicate connection is made. In the case of OAuth, duplicate is determined by the OAuth credentials used to issue the request. There is some leeway in timing multiple simultaneous connections, but not much.

User Streams & Site Streams, while technically part of our family of Streaming APIs, have a very very different authentication model than everything above. The application used to connect with is more than just a vehicle, it’s a core part of the context and “conversation” of capabilities and permissions. You can connect a “small handful” (our cute way of saying not many, but we’re not telling you exactly how many) of user streams belonging to different users, but it’s not an API meant to be used in a server-side context for more than just personal use.

The beta Site Streams product has its own set of evolving connection criteria, and it’s a bit more complicated than other streams. The 100,000 token limit isn’t really germane to the subject – that token limit only applies to Twitter Clients that replicate the Twitter user experience, and site streams supports a number of different application types besides Twitter clients. A single connection can stream on behalf of up to 1,000 users represented by access tokens in the connecting application, with the maximum number of connections (and thus supported users) determined on a mostly case-by-case basis in the beta.

So when we’re talking about connecting to the Streaming API at stream.twitter.com, the user and application connecting are still one-and-the-same. A connecting application might have a registered application on dev.twitter.com/apps that is uses. It might not. But either way, the user used when connecting represents the application in totality, and the “registered application” is really just a vehicle without its own identity in the connection.

You’ll be best served by sticking to the concept of “one account, one application, one open connection” when connecting to the stream.twitter.com endpoints. If there are other distinct applications you plan to use with stream.twitter.com, I would strive to follow the same 1:1:1 concept for it.

You may find that at times stream.twitter.com lets you get away with more open connections here or there, but that behavior shouldn’t be counted on. Thanks again for pointing out the lack of clarity on this in our streaming docs; we’ll get to work on improving that.

Thanks,
Taylor


#4

Attention @episod Taylor Singletary:

I created a website and I have been using the User Stream for about a year , and now I want some of my friends to log onto my site so they can also stream content. the problem is that when they log onto it seems that only one user can stream at a time. from what I read that is normal. now I switched over to ‘SITE STREAM’ to get my desired results. Taylor, am I in the right direction?


#5

I created a website and I have been using the User Stream for about a year , and now I want some of my friends to log onto my site so they can also stream content. the problem is that when they log onto it seems that only user stream connection can exist? now I switched over to ‘SITE STREAM’ to get my desired results. Taylor, am I in the right direction? also how can I obtain permission to the SITE STREAM API? Thank You Tay


#6

okay will fix it