We’re running a rather popular Twitter Analytics service that works really well for most people. Lately, we see more and more accounts with 1M+ followers sign up (including Twitter executives/investors/etc.) and we fail on scanning their accounts.
We run a two-stage process:
Stage 1: Use following/ids to get a complete list of IDs for the user’s followers. This one goes through fine mostly, also for large accounts (few hours).
Stage 2: Run a full scan (once!) of all IDs using users/lookup to understand the follower base demographics (most popular followers, VIP followers, etc.)
Stage 2 requires a few minutes to a day at most for smaller accounts (<100K followers).
For large accounts (500K up), we are unable to scan through since most requests fail with a 502. Our tool then throttles down to not 100 users/lookup requests, but tries 50, then 10, then 1 and this often goes through or detects a damaged record and skips it. It throttles back up to 100 requests afterwards.
My question is, and I would need a definite answer to proceed with our business or shut it down, is it feasible to think this issue with the 502 errors will be addressed in the near future? We are trying to build an analytics business on top of Twitter’s platform and are very happy with the API and the data we can get through it, we are adhering to every limit and restriction and work very hard to be a ‘nice API user’ in every possible way.
Is the 502 a simple reason for overload on Twitter’s side? What is the best way to work around this undocumented limitation? I lose a rate limited request every time I get a 502 (which should not happen since I got nothing back for my request!).
