How to avoid to be blocked/banned in a multi-app scenario?

[GMaxera]

We have multiple clients using our product at the same time. Each client has its own Twitter App, and we are planning to add Streaming APIs for let clients track somethings even when they are not logged into our system.
The server (and hence the IP) where all streaming API connection originates are the same for all clients. But each connection to the streaming API use different consumer key and secret for each client. Hence, we ensure that from our server will be no more than one connection using the same consumer key and secret.

Occassionaly, a client may change the terms to track and we will disconnect and reconnect to the stream.

I’m wondering if doing so we will blocked (or banned) from using Streaming API because it’s not clear in the documentation how limits are calculated.
Are we allowed to create as many connection as we need if all of them use a unique consumer key and secret ?
What is the fastest rate at which disconnect and reconnect will not arise a blockage ?
The disconnection/reconnection are calculated by IP or by consumer key-secret ?

[andypiper]

In this kind of use case I would suggest taking a look at the Gnip data products from Twitter that allow you to dynamically inject new PowerTrack filter rules without a reconnection.

[GMaxera]

Sorry, but this answer doesn’t help.
GNIP is not free … and the website doesn’t report any price at all … and when we asked a quotation for their services the lower price is 4000$/month !!
That price is 10 times greater than our gross income with 10 clients running for a month.

So, in these case what we can do ?? Just give up ??

I’m not pretending to have everything for free … I just want to know my boundaries and limits clearly so that we can craft an offer to our clients.
On the other side, Twitter offer the Streaming API for free but of course with limitations.
I just want to know exactly what I can do to avoid to be blocked.

I know that:

• I cannot open more than one connection with same consumer key-secret authentication
• I cannot disconnet/reconnect too often (I suppose once a day it’s not considered too often)

What I don’t know is:

• I can create 20 streaming connection with different consumer key-secret from the same IP ?
• If these 20 connection disconnect and reconnect at the same time, Twitter will count then as an abuse of the service ? (because all reconnection originates from the same IPs?)
  So, basically, I want to know if in same ways Twitter calculate the limits using the IP or not.

Finally, a personal point of view that’s off topic: I can understand Twitter try to force developers to use GNIP and earn money from us … but the price is too high and if they will not offer a starting price for small businesses, we only have two alternative: use free APIs, or do not offer to our client any Twitter functionality and switch to our social platforms.

[SocialOpinions]

I am in a similar situation, I spoke to Gnip and the price for us right now is restrictive. Let me know how you get on.

Regards.

[andypiper]

Thanks for the feedback and comments here. Let me directly answer the question (with apologies for not giving something you felt could be a helpful answer the first time around), and then talk about the alternative.

I know that: - I cannot open more than one connection with same consumer key-secret authentication - I cannot disconnet/reconnect too often (I suppose once a day it’s not considered too often)

What I don’t know is: - I can create 20 streaming connection with different consumer key-secret from the same IP ? - If these 20 connection disconnect and reconnect at the same time, Twitter will count then as an abuse of the service ? (because all reconnection originates from the same IPs?)So, basically, I want to know if in same ways Twitter calculate the limits using the IP or not.

What you’re essentially asking is, what the limitations are on the Streaming API, and how to avoid them if possible.

• the Streaming API provides 1% of the total firehose
• the Streaming API is free to use, and has no specific SLA for support or service
• apps are limited to one or two connections per consumer key. There are additional limits in place to avoid having developers try to avoid the restriction by creating a number of different apps on the same IP (this is per the section of the Developer Agreement that talks about access restrictions, II.B). So to address your question, yes, if you tried to do this, it would be detected and your apps may stop being able to connect.
• there are limitations on the length of filter queries, and in the numbers of terms and users you can track on a single connection. These numbers are in the API documentation.
• we request that you don’t constantly connect and reconnect to the endpoint, per the section on reconnecting to the streaming API in the overview documentation. This means that if you are constantly wanting to add terms or users to a filter track query, your users won’t have a great experience as you’re likely to get held back from connecting too often. Once a day probably would be OK, I’ve had my own apps and IPs restricted for doing it too often within a few hours, for instance.

So that (I hope) covers your question.

The reason I mentioned Gnip in my original answer was for two reasons, which I feel are good ones:

• PowerTrack is amazingly flexible, and allows you to modify rules on-the-fly, without any connection issues.
• it is fully-supported and offers redundancy and resiliency, so the other restrictions I mentioned are not a problem, i.e. it is a solid solution for building your own business offerings where you can rely on performance and support.

Fully understand that this is not an option for everyone. I apologise again that you didn’t feel that my initial response answered your question. I hope this clears up any confusion I created.

Always appreciate the feedback on our API and platform offerings, and I’ll be sure to share your thoughts with our team - that’s my role, as your developer advocate.

[GMaxera]

Thank you very much

Very detailed

Regarding GNIP, we really like to use it in the future, but it’s difficult. The problem is that GNIP doesn’t offer a pricing plan that allow to startup new products. It would be great if GNIP had a pay-per-usage plan.