Twitter, please let us pay you for higher API limits

schammy · 2013-02-19T22:47:13.000Z

I run a web analytics service that includes a Twitter analytics feature, where you can input a search term and we regularly poll the API for new results. We’ve had this feature for almost 4 years now and over that time we’ve had over 16,000 searches created on our end, currently 7500 of those are active.

The current search API 1.0 limits have worked fine for us even as we have scaled up to this many searches. We essentially grab a random subset of the searches every 5 minutes and hit the API until you tell us the limits have been reached. Then the script exits and we wait the proper amount of requested time until starting up again.

I don’t know how many queries per hour we send to the API but it’s definitely several thousand at least.

The new search API 1.1 is currently listed as being throttled to 180 per 15 minutes, so 720 per hour. This is really going to limit the usefulness of our Twitter integration, as it will take 10 hours to run through and update all of the searches we’ve added, and that will only get worse.

I know, you all want to cry me a river. But my point is this.

We are perfectly willing and able to pay money for higher API limits, as I’m sure many other developers are as well. I wouldn’t even bat an eye at paying Twitter $1,000/month for considerably higher limits, say 3000 or 5000 queries per hour. Just 1,000 developers paying you this amount of money would generate $1,000,000 of revenue per month, or $12,000,000 per year, and this is a very conservative estimate in my opinion.

Please consider doing this. You will make us developers very happy and you will generate a good amount of extra revenue as well.

PLEASE…

episod · 2013-02-19T23:17:50.000Z

API v1.1’s search limits don’t sound like they should stop you from doing what you want to be doing.

Each user of your application would have 180 queries they could issue to search/tweets per 15 minutes. If you haven’t been requiring your end users to log in with their Twitter account until now, which is understandable considering the Search API didn’t previously support authentication, you’ll now want to start.

You may need to rethink your application model to accomodate the user auth requirement, but you shouldn’t need additional queries per user to accomplish that.

schammy · 2013-02-21T04:31:05.000Z

I hadn’t considered that, not a bad idea, however it does add a lot more complexity on our end and is also a barrier between a user setting up a search on our end.

But regardless of that, I am still perplexed that Twitter refuses to make money off their API. It seems like such an obvious source of revenue.

mattrogowski · 2013-02-21T15:08:32.000Z

Hijacking your thread a bit but I have a similar issue.

I’m building an app for people to import their archives to view historical stats and stuff (people love stats, right?) and also want to load their mentions and retweeted tweets too (and favourited tweets but there’s no easy way of doing that at all). Anyway, loading who actually retweeted all of your retweeted tweets is a nightmare, 15 requests per 15 minutes took over 7 hours for my ~450 retweeted tweets. The API’s great but things like this can limit its uses, albeit for an admitted edge-case.

I’m only an individual dev working on a personal project and don’t really have the funds to pay for higher API limits and would rather be able to bat my eyelids to get some higher limits for one endpoint, but I think you’re missing a trick with people who are willing to pay.

Unless we can already request higher rate limits for a specific endpoint?

jnylund · 2013-02-21T20:48:19.000Z

The place that the new API is really going to kill analytics companies is network analysis. There is no alternative method for getting the list of followers for a particular account so that a de-duped network size can be computed. Can you please give us some relief here?

guitaraffair · 2013-02-21T21:06:44.000Z

We’re in the same boat. At least @schammy’s solution of paying for elevated limits would create a environment where everyone would have the ability to keep their businesses going. As it is, it’s just a black-hole application process, where Twitter will arbitrarily pick the winners and losers. Since I don’t see the big analytics companies complaining, I’m assuming they are going to get elevated levels while the startups who don’t yet have a big name will be put at a market disadvantage because we will no longer be able to provide certain analytics functions to our customers. There’s an element of openness and fairness that is lacking from this process IMO.

musigma_mustang · 2013-03-25T12:57:46.000Z

We need to do real-time network analysis based on certain topics and need the followers and friends data for this. And we need it real-time. We cannot really get the users to authenticate and user their access tokens for their follower data since this is not an analysis for a specific user. We would be willing to pay to get the follower data at a higher rate limit. Is there a way to get this? Does Twitter have any plans to sell or resell this data?

kurtismullins · 2013-03-26T12:15:55.000Z

You guys can use Gnip (and there’s another re-seller) but it isn’t cheap, by a long shot.

mattrogowski · 2013-03-26T12:20:41.000Z

Unfortunately for a personal project I can’t do that. I’d happily sacrifice all access to unused endpoints to get a higher limit on another one but doubt that’s possible.

zafercalik · 2013-06-07T05:38:09.000Z

if i want to query twitter via search api 1.1 and not to exceed rate limit, cant i do collect many users user base token and use them in order. For example collect 100 users token and so i can have 1800 query limit for 15 minutes ?