Streaming API: Only one standing connection per account?

LeoTheCTO · 2011-09-16T14:40:12.000Z

Similar to the other questions. If we are building a service that has authenticated clients (either oAuthed to our API key – or they could setup their own API key) each tracking a small # of #tags, we should not open one streaming connection / client (from our server)… Right? If we build one tracking list on one stream connection for all clients, obviously (1) we will need to keep reconnecting with a new tracking list each time any client makes a change… which is probably OK (2) after a bunch of clients we will hit the limit of 400 tags / connection. When we get to this point, is the standard Twitter policy to push people off to GNIP/DataSift? Or could we get an upgrade for our single oAuthed API key?

godawful · 2011-09-21T13:34:54.000Z

I came here to ask the same question as it happens. Can someone address this issue?

Thanks!

Wuup · 2011-09-21T14:51:26.000Z

Is it possible / acceptable to use the streaming api via a PHP server for all of an applicaitons users. (server side) instead of client slide. In theory this would have to open too many streams which is frowned upon with guidelines?

“b) excessive connection attempts will result in an automatic ban of the IP.”

Is the streaming API only for desktop apps / client side applications. If it was server side, would it have to be one stream per user, or is there a way to open a stream for all users? (assume not).

Any advice, thoughts would be useful.

episod · 2011-09-21T15:14:29.000Z

Hi @Wuup,

There are a few different ways to approach this – but essentially when you’re thinking about the core Streaming API (not User Streams or Site Streams), it’s essentially a server-to-server technology and the user context that you’re operating in isn’t your end-user context, it’s a user dedicated to your application for working with the Streaming API. You work to keep connections open for as long as possible and queue up changes to the stream (based on your end-user’s requests) so that you limit reconnects to as seldom as possible. In this case, you’d open up a single connection to stream on behalf of many of your end users.

Site Streams operates in a similar context: you’re still primarily connecting as a user who represents the application (and owns the application) – however, you’re also providing a list of users you’re acting on behalf of (and have access tokens for). You’re then streamed the equivalent of the “me feed” from User Streams for each user you’re acting on behalf of.

The core streaming API is limited in the % of firehose you can have access to, the number of distinct query terms, queried users, geographical areas. For a lot of developers, the basic limits are sufficient for their user base – for others greater limits become ideal, or other forms of compromise like prioritizing one class of end user over another.

Wuup · 2011-09-22T08:23:46.000Z

Gotcha,

Thanks.

“For a lot of developers, the basic limits are sufficient for their user base – for others greater limits become ideal, or other forms of compromise like prioritizing one class of end user over another.”

The basic limits are free? But the greater limits are ‘pay for’ via resellers?

@episod - I have put through an application for site streams - thanks for your advice, we feel this is the right way for us going forward.

episod · 2011-09-21T15:26:48.000Z

This can be a complex issue. GNIP and Datasift provide data for a number of use caes, but one of the primary areas they serve is for tweets in non-display context or tweets in very limited (paywall, for example) contexts.

My recommendation: use the Search API for ad-hoc queries by your end-users. If they decide they want a query to be recurring, that they want to watch it from “now on” – add it to a queue you have for the next time you add additional terms to your outstanding streaming connection. Consider tiering access to your service to further allocate/direct resources.

A queuing system becomes pretty necessary when you’re working with the Streaming API. You want to limit reconnects as much as possible – one way to do that is to use two streams (different users) and use them alternately.

When you run into a situation where the basic limits are no longer adequate (not just because you want lots of breathing room ), send an email to api@twitter.com explaining in detail how you’re currently using the streaming API, information about who your users are and how they access your product, what they do with tweets within your product, and how heightened streaming limits will aid your ability to serve our mutual users. Often we’ll recommend you go to GNIP or DataSift, if it’s appropriate for your scenario.

It never hurts to make sure that tweets are fully actionable in your application: reply, retweet, favorite, follow; that your application meets display guidelines and so on.

Hope this helps you think about this. The Streaming API and OAuth API keys have very few relationships with each other when it comes to this stuff.

me_bx · 2011-12-22T16:06:42.000Z

Hi @episod,
"you can run two streaming Twitter clients from your machine pretty easily."
Is it still possible? I have failed maintaining two filter stream connections in parallel, they got disconnected all the time. It looked to me like twitter API was disconnecting me, but might be related to the client library I am using. Any opinion on this? Each stream was following approx. 1000 twitter Ids.

Thanks,

mehdi

strdevus · 2012-09-13T18:13:48.000Z

I have the same issue. Running sample stream and a filter stream with the same credential seem to disconnect each other. I’m using phirehose library. The filter stream follows 1400 ids. @episod, can you confirm that we can run two streams concurrently? Thanks!

kurrik · 2012-09-14T17:16:56.000Z

The disconnect logic has been improved since his post. You can now maintain one connection per account. Connecting with the same credentials will kick off the oldest connection.

vaughnjeffrey · 2012-09-27T16:30:39.000Z

Can I use 2 Twitter developer accounts, one for the streaming API access and another for giving my users oauth access to sign up/publish posts to Twitter?

Is this against Twitter’s rules?

episod · 2012-09-27T17:47:48.000Z

As long as you’re striving to maintain a single open connection to the Streaming API, it doesn’t matter if it’s the same account as the one that owns the application you use for REST API access – though it’s often better to have them be the same.

vaughnjeffrey · 2012-09-27T18:13:56.000Z

Why is it better to have them be the same?

episod · 2012-09-27T18:22:10.000Z

To better identify that they’re associated with each other. Twitter doesn’t have a way for you to tell us that two accounts are part of the same application, developer, or user.

adriaan_pelzer · 2012-12-18T16:23:26.000Z

Another question on this topic:

Say, for example, we would want to scale a filter/track stream connection beyond 400 keywords, or a filter/follow connection beyond 5000 users. What would be the best approach to doing this? Or does that bring us into “term violation” territory?

I want to figure out wether I should future-proof my app by adding back end functionality to add more accounts to open streams with, or should I state 5000 users/400 keywords as a hard limit?

ProbandoC · 2013-12-06T19:09:23.000Z

I’m currently developing a new application using the Twitter Stream API provided by you thanks to the library Twitter4J. I’m testing the app each five minutes, or less. Every time I stop the app, then the connection goes down and when it is launched again, a new connection is made to the twitter servers. Could this result in a ban?

Trade_List · 2013-12-21T10:17:47.000Z

I’m a bit confused. My application users will each initiate their own stream connection for tracking filter/status for different queries.

The problem is as soon as two users connect simultaneously, the first user gets disconnected. Is there anyway around this? Or am I doing it wrong. Like the other posted said, one user in dev everything works fine. As soon as you add a 2nd user, then you get disconnected.

Trade_List · 2013-12-21T10:19:32.000Z

How do we avoid this? Obviously its unrealistic to only have one connection when your app has multiple users.

ChrisSLock · 2014-03-18T23:27:38.000Z

So I understand that I cannot open 2 streams with the same credentials. My use case is that I want to have one statuses/filter open and one statuses/sample open. Will one disconnect the other or am I safe since they are distinct streams?

kuAshit · 2014-04-01T12:23:28.000Z

I am currently new to twitter4j application . Currently while developing an app I noticed of getting into rate limit frequently. Is it possible to change the ouath tokens while the app is running. I mean for example I have three sets of tokens,if one set of token get into rate limit then I change it with other one without stopping my application.And those tokens are generated separately.
I know it sounds a bit crazy but is this possible would need a reply …
Thanks in advance

system · 2014-09-02T14:31:16.008Z