Querying for deleted promoted tweets is too slow

rate-limits
creative

#1

Hey all :slight_smile:

I’m showing metrics per-creative (card) for a couple of accounts, in order to show consistent metrics I need to query the promoted_tweets endpoint with the “with_deleted” flag set to true (and later querying the metrics stats on those ids…).

The problem with that is that for some accounts this process does not scale :smiley: since the promoted_tweets endpoint (with_deleted=1) returns the promoted tweets since the beginning of time. we have some accounts with more than 25k promoted_tweets at this point and with the rate-limits those queries are never-ending.

Ideally I would like to be able to query only for those promoted_tweets (including deleted) that had any stats in the specified time-range, but afaik there is no such endpoint.

Alternatively, adding a deleted_at flag to the promoted_tweets endpoint (and allowing to sort on it) could also potentially solve this problem - is it possible for you to expose this field via the api?

Do you guys have any other ideas for a solution?

Thanks


#2

@dimapv I think the solution is to reduce the surface area of what you’re trying to do.

In more extreme cases, I’ve seen single advertiser accounts that had upwards of 300k-500k+ promoted tweets. Obviously, at a certain point enumerating all of those promoted tweets over and over again every time I want to do something isn’t going to be a winning strategy – as you’re finding out yourself now. You’ll have to smartly store some of the more repetitive data you’re pulling and reduce the number of items you need to fetch.

Take a look at how the Ads API allows sorting:
https://dev.twitter.com/ads/basics/sorting

My recommendation is to persist (in your own database) some basic information and the IDs for the promoted tweets you want to interact with later. Build yourself an automated job that on a regular interval ingests promoted tweets by calling GET /promoted_tweets for each account account with with_deleted=true&sort_by=created_at-desc.

twurl -H ads-api.twitter.com "/0/accounts/abc123/promoted_tweets?with_deleted=true&sort_by=created_at-desc"

This will return all promoted tweets regardless of delete status, sorted in descending order by the creation time. Your promoted tweet collector job can keep track of the last time it successfully ran and stop paging through promoted tweets in the API response once it gets to items created before it’s last run. Every time your job runs, it will only have to fetch the latest promoted tweet info for each account.

For real-time stats (non-historical, in the last 7 days) you could also be more intelligent here by looking at the campaign and line item. You’ll still need to gather a list of promoted tweet IDs, but using the campaign and line item you can exclude stats requests for promoted tweets that don’t belong to any campaigns that would have had activity in the last 7 days (stats data older than that won’t change).


#3

Thanks @brandonmblack! that’s a great suggestion and one that I’ve considered actually, the downside is that when scraping new accounts that I’ve not seen before (with existing historical data) I will not be able to receive the latest metrics for deleted promoted tweets (for example: scraping the last 7 days of a new account).

a full traversal of all promoted_tweets could take days which is not acceptable for my use-case, what would you recommend in such a case?


#4

@dimapv there’s a solution for historical stats data coming soon (next week) but unfortunately I can’t say much about it now. Even with a high volume of promoted tweets, reading those promoted tweets created or active within the last 7 days then using that to pull stats for the same time period shouldn’t take too long at all.

As for ingesting the promoted tweets themselves (not the stats data, but the IDs and other data) it shouldn’t take days at a standard tier level but right now at a developer level it may be a little restrictive. The above recommendation is still the way to go and once you graduate to standard tier of the program you would be able to read as many as 250K promoted tweets up to 1000 at a time per request in about 30 mins and stay under the rate limit.

twurl -H ads-api.twitter.com "/0/accounts/abc123/promoted_tweets?sort_by=created_at-desc&count=1000"

Related documentation:
https://dev.twitter.com/ads/basics/rate-limiting


#5

Thanks @brandonmblack!

How would you pull only the active promoted_tweets of the last 7 days (as that’s exactly what would solve my problem)? or is that something that will be possible only with the new solution that is not available yet?

did not know that you launched the new tiers, we will apply to the standard access soon. :slightly_smiling: