Clarification request: users/lookup returns 0 to 100 results when requesting 100 results

[fruji]

This goes along with a recently closed issue (link: https://dev.twitter.com/discussions/10460#comment-22594) when users/lookup returned invalid responses.

Now, especially with larger accounts, I request up to 100 IDs and receive either 1 ID, or 100 IDs. It seems to be toggling between these two, but for larger accounts, when reaching deeper into the follower bucket (towards the end), it drops to 1 response per 100 ID requests. I will process the 1 response, and request the next 100 (shifting the requested list basically by 1) and it will return another 1 ID.

Has this got to do with poisoned/invalid IDs in the result set?

My major issue is that I basically burn through the 300+ API requests quickly for accounts with 1M+ followers very fast once it drops to 1 ID responses.

Just curious if there is a work around, or if this is what we have to be working with for now until a change is made on that front.

Thanks again.

[episod]

Is this a case of you walking through followers or followings for specific users and getting to the oldest user IDs in the collection that may represent now-deleted users? Can you share some of the IDs you’re working with that exhibit this? Is it really a problem with users/lookup as much as you’d just prefer whatever methods you’re sourcing these IDs from to just not hand you these IDs you can’t work with?

[fruji]

I’ll add IDs in a bit, but here’s what we’re doing:

We provide detailed follower-base analytics (VIPs following you, most influential by followers, etc.). User authenticates, we scan a his follower base with (rate-limit auto-scaling) requests for “followers/ids”. If exhausted, we back off and wait and return later. This endpoint is wonderful, supplies us with 5.000 IDs a pop, we can easily scan through very large accounts (our users with 1M+ followers). Once we have that (i.e. a full list of everyone following this person), we need details to run analytics. So we scan through the large list of IDs and use “users/lookup” with 100 IDs supplied on each call, also rate-limit adjusted. If an error occurs, we reduce to 50 IDs, if still fails, we reduce to 10 IDs, if still fails, we reduce to 1 ID and if that fails we’ll remove this ID forever (poisoned) and scale back up to 100 IDs requested on next call.

The issue is here:
We noticed that the request for 100 IDs (on larger accounts, somewhere towards the middle/end of bucket) via users/lookup responds with only 1 ID, instead of 100. Ok, that’s fine, maybe 99 invalid since last followers/ids request an hour ago? Maybe a reason to back off and request only 1 next call? Not sure. If 99 were dead accounts and therefore not returned, that would be fine (although why return them with followers/ids right before, then…). But the strange thing is that when I keep bugging the API with more 1 ID calls (instead of 10,50 or 100 IDs max), it will supply them, just only 1, but will go further forever almost it seems until it hits rate limit (which is fast, after 350 calls basically in that case).

(1) [followers/ids] Get all follower IDs for a user, 5000/call, take care of rate limits. Loop until done.
(2) DONE with all IDs, temporarily saved on our side. Now process these for details.
(3) [users/lookup] Request details for 100 IDs [example IDs: 5000,5001,5002,5003…5099]
(4) API responds with 1 ID instead of 100. Ok, save details for that one.
(5) [users/lookup] Request details for 100 IDs [now: 5001,5002,5003…5100]
(6) API responds with 1 ID instead of 100. Ok, save details for one.
(7) [users/lookup] Request details for 100 IDs [now:5002,5003,5004…5101]
(8) API responds with 1 ID instead of 100. Ok, save details for one.

So it seems 5001,5002,5003 were actually valid. I still only got 1 ID (ID=5000) on the first call. So I figured, as said earlier, it might be a smart way for you guys to deal with traffic / response times (if time out, just return 1 ID instead of none) – but even if I dial down the requested IDs to 50, or 10, it will just return 1 ID for these larger accounts, towards the end.

So, if you have an (authenticated) account like @sacca - it’s basically impossible to run users/lookup and get a complete picture of his most popular followers.

I’ll compile a list of IDs and a better explanation, but this is what we’re seeing right now. Thanks again, your support here on this forum is fantastic.

[fruji]

Oh, and we’re fine if this takes up to multiple days on the first run and we go through users/lookup in slow-motion. We understand you can’t just offer a complete dump at an instant. No issue there.

What I do, on a re-scan (once we have that user in our system), is basically only use followers/ids to detect NEW/GONE ids and just refresh the ones that are new with users/lookup. So we don’t do this whole shebang every day. It’s only once per new user. But is a pain for everyone signing up with us for over 1M followers (starts at around 200k-300k accounts usually, they have to wait for multiple days to be processed the first time).

[fruji]

Alright, hold off until I can post a clear sequence of IDs in a reproducible manner. We’re getting 99 IDs returned now of 1 sometimes, and other requests time out 502 (probably page size issue I assume). Either way, I’ll wait until I see a clear example of the issue and then report back, don’t want to waste your time before I have it reproduced in a clear setting.