API Limitations


#1

I’m building a pretty large app for a client, and had some questions about heavy API usage.

Basically this app is going to aggregate feeds from various sources. My client estimates around 900 follow-able users will be in this system to start out, with more being added over time. He wants to update the feed data every 15 minutes, so we would need to update one user feed per second, assuming 900 feeds and a 15 minute TTL. As the requests take a few seconds to complete, we would then need to load balance across a few servers to tackle the queue asynchronously.

My question is, should I be worried about quota errors or hitting any kind of limitations? If so, what are our options?

Thanks in advance…

-Gordo L


#2

You definitely should make sure you and your client fully understand the documented rate limits in API v1.1 and the menu of platform options we offer – including Streaming APIs. I would recommend delving into this site’s documentation before planning what the app’s requirements are. Be sure that you’re also developing according to display requirements.


#3

https://dev.twitter.com/docs/embedded-timelines
"Embedded timelines are not subject to traditional Twitter API rate limits, and are designed to scale with your site as your traffic and audience grows. By using Twitter Widgets, you are agreeing to Twitter’s Developer Rules of the Road."

This is exactly what we need, except not for our own profile… Is there an easy way to programmatically create an embedded timeline widget given a specific twitter’s user ID or username?


#4

The user_timeline endpoint allows 300 requests per 15 minutes. That seems awfully small, only allowing one request every 3 seconds. If we have 900 feeds to update that means it will take around an hour and a half per cycle.

Is there any way for us to get an increased quota? Is there a bulk API?


#5

I’ve got it working without the API by using a widget, so this isn’t an issue any longer since it’s purely client side.


#6

Actually nevermind, just noticed this: https://twitter.com/support/status/307211042121973760


#7

So, I’m out of luck I guess. The API limits are way too low and there’s no way to programatically create widgets or embed a widget by userID or username.


#8

Taylor’s suggestion was to use the streaming API. That will collect all tweets for up to 5,000 users with no rate limits: https://dev.twitter.com/docs/api/1.1/post/statuses/filter

That should handle your request, and it will be in real-time. :slight_smile:


#9

What if I want to pull public feeds without requiring a user to authenticate?


#10

The streaming API does not require all the account owners to authorize. You do need to have at least 1 authorized account that you control that makes the API connection, but that can collect tweets on anyone, except protected accounts. You should read the documentation on this.


#11

Okay, I know technically that I could do that, but I wasn’t sure whether that was allowed and I didn’t see it explicitly stated in the docs.