API Limitations

gordol · 2013-03-22T10:44:02.000Z

I’m building a pretty large app for a client, and had some questions about heavy API usage.

Basically this app is going to aggregate feeds from various sources. My client estimates around 900 follow-able users will be in this system to start out, with more being added over time. He wants to update the feed data every 15 minutes, so we would need to update one user feed per second, assuming 900 feeds and a 15 minute TTL. As the requests take a few seconds to complete, we would then need to load balance across a few servers to tackle the queue asynchronously.

My question is, should I be worried about quota errors or hitting any kind of limitations? If so, what are our options?

Thanks in advance…

-Gordo L

episod · 2013-03-22T14:57:23.000Z

You definitely should make sure you and your client fully understand the documented rate limits in API v1.1 and the menu of platform options we offer – including Streaming APIs. I would recommend delving into this site’s documentation before planning what the app’s requirements are. Be sure that you’re also developing according to display requirements.

gordol · 2013-03-23T05:45:00.000Z

https://dev.twitter.com/docs/embedded-timelines
"Embedded timelines are not subject to traditional Twitter API rate limits, and are designed to scale with your site as your traffic and audience grows. By using Twitter Widgets, you are agreeing to Twitter’s Developer Rules of the Road."

This is exactly what we need, except not for our own profile… Is there an easy way to programmatically create an embedded timeline widget given a specific twitter’s user ID or username?

gordol · 2013-03-23T03:58:10.000Z

The user_timeline endpoint allows 300 requests per 15 minutes. That seems awfully small, only allowing one request every 3 seconds. If we have 900 feeds to update that means it will take around an hour and a half per cycle.

Is there any way for us to get an increased quota? Is there a bulk API?

gordol · 2013-03-23T06:25:09.000Z

I’ve got it working without the API by using a widget, so this isn’t an issue any longer since it’s purely client side.

gordol · 2013-03-23T14:00:00.000Z

Actually nevermind, just noticed this: https://twitter.com/support/status/307211042121973760

gordol · 2013-03-23T14:00:38.000Z

So, I’m out of luck I guess. The API limits are way too low and there’s no way to programatically create widgets or embed a widget by userID or username.

140dev · 2013-03-23T14:07:30.000Z

Taylor’s suggestion was to use the streaming API. That will collect all tweets for up to 5,000 users with no rate limits: https://dev.twitter.com/docs/api/1.1/post/statuses/filter

That should handle your request, and it will be in real-time.

gordol · 2013-03-23T14:18:00.000Z

What if I want to pull public feeds without requiring a user to authenticate?

140dev · 2013-03-23T14:24:21.000Z

The streaming API does not require all the account owners to authorize. You do need to have at least 1 authorized account that you control that makes the API connection, but that can collect tweets on anyone, except protected accounts. You should read the documentation on this.

gordol · 2013-03-23T14:28:58.000Z

Okay, I know technically that I could do that, but I wasn’t sure whether that was allowed and I didn’t see it explicitly stated in the docs.