Why is Twitter quota for app-auth greater for some endpoints and less for others, compared to user-auth?

oauth

#1

Twitter allows developers to use app-auth rather than user-auth, but sometimes the API quotas for app-auth are better than with user-auth, and sometimes they’re worse - why is that?

From https://developer.twitter.com/en/docs/basics/rate-limits.html :

In the table above you can see that if you want to search tweets, it’s better to use app-auth, but if you want to simply do bulk lookup by id, it’s better to use user-auth.


#2

As I replied on Stack Overflow.

Here's the thing: app-only auth is really intended as a stop-gap or fill-in measure. What I mean by this is, imagine you have 10 users that are all signed in to your app and you are using their tokens to access the API; what if one of these users is super popular and you need a few more calls. You can then use the application-level app-only auth to make a few additional search requests (for example) to "top up" what might run out of requests per user. If you have 10 users that are all super popular, you might use all of the app-level requests to backfill for them quite fast.


#3

Thanks Andy for that explanation - I can see how for the use-case you describe app-auth might make sense as a fill-in measure. I work on the Guardian’s in-house analytics tool Ophan - it uses the Twitter API for these purposes:

  • resolving referring urls to specific Tweets (using /search/tweets)
  • reading the timelines of Guardian-brand accounts like @guardian, @GuardianAus, etc to see what Tweets have been posted
  • to display those tweets to Guardian staff on the Ophan analytics dashboard frontend (often using /statuses/lookup - because of GDPR, we no longer store the full body of those tweets in our own systems)

For us there is really only one user of our our Twitter ‘app’ - Ophan itself - so it seemed that app-auth was most appropriate:

  • the Ophan service itself doesn’t really correspond to a twitter user (unless you count @guardian)
  • using app-auth gave us the highest /search/tweets quota, which despite our best efforts to reduce API usage, we definitely needed. We didn’t realise when we switched over to app-auth that we were also reducing our /statuses/lookup quota to 1/3 of it’s previous value - we assumed, incorrectly, that the quota levels for app-auth were designed to accommodate a single ‘service’ entity.

What would you advise is the best way for us to deal with shortages of /statuses/lookup quota? If you’re in the London area, you’d be very welcome to come over to our Guardian offices in Kings Cross so we can discuss how we can make better use of the Twitter API?


#4

You can try using both app auth and user auth together: it makes authenticating calls slightly more complicated but not that much.

At a minimum, you have the app “owner” - the developer account that owns the app automatically gets keys generated under “Keys & tokens”, so you have 180+450 calls every 15 minutes for /search/tweets

I’ve a hacky way for displaying tweets, if storing them is a restriction, it might be worth doing something like this:

Instead of looking up tweets to display from API, add <blockquote> tags with permalinks to tweets into whatever page that’s displaying them like this:

<html>
<body>

<blockquote class="twitter-tweet"><a href="https://twitter.com/Interior/status/463440424141459456"></a></blockquote>
<blockquote class="twitter-tweet"><a href="https://twitter.com/guardian/status/1103716184959131648"></a></blockquote>
<blockquote class="twitter-tweet"><a href="https://twitter.com/guardian/status/1103715446388416512"></a></blockquote>
<blockquote class="twitter-tweet"><a href="https://twitter.com/i/status/1103715320198545409"></a></blockquote>

<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

</body>
</html>

This way, if you know the tweet id you want, you never have to store any tweet content on your systems at all, and you don’t have to load it yourself through the API.

widgets.js will load the page and replace all the <blockquote> tags with proper tweet embeds. (that example won’t work if viewing that html as a local html file but will work when loaded from a server, as twitter widgets won’t load in a browser with content security policy by default or something like that)

If a <blockquote> contains an <a> with an href with a tweet id (you don’t even need to get the user correct - like the last entry in that example), widgets.js can render them in the browser just fine. To load those dynamically you might need to call it manually after creating the blockquotes see https://developer.twitter.com/en/docs/twitter-for-websites/javascript-api/overview and https://developer.twitter.com/en/docs/twitter-for-websites/javascript-api/guides/javascript-api

Hope that helps!


#5

You can try using both app auth and user auth together: it makes authenticating calls slightly more complicated but not that much.

That makes sense - we’ll probably end up doing that!

The blockquote method works well if you want to display the whole tweet, though it’s not so good if you want to have a compact representation of the tweet, which we do in quite a few places - it turns out for us that we because we cautiously didn’t store the timestamp of the tweet (playing it safe with GDPR data retention) we often need to perform a lookup when we’re rendering our most minimal representation of a tweet, which just includes the account & timestamp.


#6

Hi Roberto,
How about if you ask your users (person looking at the page, editor, owner etc) to authenticate to your app (read only) and store tokens for them.
Then you can use their context to query the API for the data they are looking at (statuses/lookup), and only use the app context for backend related queries (search/tweets, and if that’s not enough you can always backfill with the premium API)

PS. I want to point out that blocking could be an issue here, so if your users block others they won’t see their tweets, which could be a good thing or a bad thing depending on how you look at it