@dimapv I think the solution is to reduce the surface area of what you’re trying to do.
In more extreme cases, I’ve seen single advertiser accounts that had upwards of 300k-500k+ promoted tweets. Obviously, at a certain point enumerating all of those promoted tweets over and over again every time I want to do something isn’t going to be a winning strategy – as you’re finding out yourself now. You’ll have to smartly store some of the more repetitive data you’re pulling and reduce the number of items you need to fetch.
Take a look at how the Ads API allows sorting:
https://dev.twitter.com/ads/basics/sorting
My recommendation is to persist (in your own database) some basic information and the IDs for the promoted tweets you want to interact with later. Build yourself an automated job that on a regular interval ingests promoted tweets by calling GET /promoted_tweets for each account account with with_deleted=true&sort_by=created_at-desc.
twurl -H ads-api.twitter.com "/0/accounts/abc123/promoted_tweets?with_deleted=true&sort_by=created_at-desc"
This will return all promoted tweets regardless of delete status, sorted in descending order by the creation time. Your promoted tweet collector job can keep track of the last time it successfully ran and stop paging through promoted tweets in the API response once it gets to items created before it’s last run. Every time your job runs, it will only have to fetch the latest promoted tweet info for each account.
For real-time stats (non-historical, in the last 7 days) you could also be more intelligent here by looking at the campaign and line item. You’ll still need to gather a list of promoted tweet IDs, but using the campaign and line item you can exclude stats requests for promoted tweets that don’t belong to any campaigns that would have had activity in the last 7 days (stats data older than that won’t change).
