Strategies for mitigating Search API rate limits for application-only oauth

[josephimt]

Hello, thanks you in advance for your help.

My team is having some difficulty figuring out how we can mitigate Search API rate limits by having our authorized application make requests on behalf of users. Please see this help doc: https://dev.twitter.com/docs/auth/application-only-auth (“requests made on behalf of users will not deplete from the rate limits available through app-only auth”).

Here is the scenario:

1. My company creates and hosts mobile websites for events. We have multiple customers and manage multiple events simultaneously. We host these mobile websites from a single IP address.

2. One commonly requested feature is an embedded Twitter stream. Example #1: A trade conference may want their official Twitter stream to appear on the homepage of the mobile website. Example #2: A music festival may want Twitter streams for each band to appear on the band’s official page.

3. In the past (with basic authorization) we accomplished this using the Search API requesting tweets from a specific Twitter user and/or search term. We would construct the request as needed and then publish it. Worked like a charm.

4. With oauth, the approach described in #3 no longer works. So we are switching to the new approach using application-only oauth. We’ve got the formatting all in place (adhering to Twitter’s requirements) and have begun testing.

5. During testing we hit the rate limit for obvious reasons–testing artificially creates demand. But we anticipate this is going to be a major issue in production as well–we may be offering this same feature to multiple customers simultaneously, or to one larger customer (e.g., an art studio tour or music festival) with periods of peak demand.

6. It appears from the documentation cited above that we could mitigate (if not avoid) this issue by submitting Search API requests for each mobile website (customer) using user-based authorization—our authorized app makes the request from our IP but on behalf of our customers. This does not avoid rate limits, but it mitigates the risk of a shutdown while also ensuring customers “pay for their own usage”.

Question #1: Is what we’re thinking of doing in #6 actually possible or are we misinterpreting the documentation?

Question #2: If this is possible, how the heck do we do this?

Thank you!
Joseph

[episod]

Hi Joseph,

The rate limits for app-only auth are meant to solve more basic use cases. In API v1.1, the best way to expand your use of the API to meet your user demand is by requiring users to authenticate and leveraging their access tokens when making requests on their behalf. You’ll then have their rate limits to work with, each isolated from each other. You’ll also have what’s left over in app-only auth to handle sidecar use cases. In this case probably, your “user” is some representation of each event itself.

You’ll also want to look into the Streaming API. Instead of repeatedly issuing the same recurring queries for a conference or event, you would open a single longer-lived stream on behalf of all your events and receive the tweets matching in real time. Then you bucket them by event after processing.

Using the REST & Streaming APIs together is the best way to be flexible in a scenario like this.

[josephimt]

Hi Taylor, thanks for the response. That makes sense (user-by-user authentication with the "users’ being a rep) and is one of the scenarios we talked about. We were just concerned you’d think this was gaming the system; we’ll assume from your response it is not.

Re: streaming, we also looked briefly at this but thought it an extreme solution for us at this time. It’s nice to know we can scale up into this if necessary.

Thanks again!