Cant get all favorite tweets by rest api

[leroyzhou]

I want to get all favorite tweets about 23.3k for one user 。
I use the https://api.twitter.com/1.1/favorites/list.json api with max_id, but only got about 3k+ favorites .
How can I get the all 23.3k favorites of the user?

[twit]

I see this same bug. After 18 pages of favorites (using the oldest favorite.id from the prior page as the max_id parameter to get the next page), the API seems to just break and return a truncated list. I seem to get 3580 total favorites before it breaks.

[twit]

Sometimes pages return less than the full 200 count, so the exact number of distinct tweets is lower, in the 3100-3300 range. The exact number differs from user to user, but seems to be non-deterministic via any external pattern.

[twit]

After much more work/investigation, I came across this possibly related notice. On https://dev.twitter.com/rest/reference/get/statuses/user_timeline it says “This method can only return up to 3,200 of a user’s most recent Tweets.”

There is no such hard limitation mentioned on https://dev.twitter.com/rest/reference/get/favorites/list but it’s extremely suspicious that the bug seems to happen at about the same number. Perhaps the bug is an intended limitation that is missing from the documentation?

[twit]

This same limitation seems to exist on the twitter.com website. Scrolling down, you only get to see ~3000 favorites and then the site stops loading more. It looks like the only way to access older favorites or tweets (for yourself) is to delete 3000 new ones and then perhaps the old ones will be visible again? For a user other than yourself that wouldn’t be possible though.

Can anyone confirm this issue?

[twit]

It looks like the ~3000 favorites that are shown are the ones with the most recent tweet IDs. Strangely this seems to mean that if you favorite an old tweet (older than the ~3000th most recent favorite), then it’s immediately lost at the bottom of the list where it can never be viewed. Is this the expected behavior? Any way around this?

[twit]

Side note, in case anyone else wondered: “Your Twitter archive” (as available on the https://twitter.com/settings/account page) doesn’t contain an archive of your Twitter favorites either.

[twit]

This seems to be an issue that goes beyond the API itself. As per my last comment, even searching the main twitter site, it appears there’s no way to find favorites (even ones you just clicked moments ago), if it happens to be for an old tweet ID.

I see online a number of scripts that people have written to delete all the favorites in order to collect them and archive them, so that seems to be the only answer.

[twit]

Turns out archiving via deletion doesn’t seem to work either. As you delete the favorites, fewer and fewer favorites are returned by the https://dev.twitter.com/rest/reference/get/favorites/list API. There seems to be some other undocumented bug or “feature” that prevents finding your old favorites. Will the API eventually recover and show the old favorites?

[twit]

I’m not sure how long I should wait. The hope was once I archived the new ones the system would eventually provide access to the older ones. Anyone know…?

[mattrogowski]

You’ll never be able to get them. I’m in the same situation as the OP and you for favourites; I got about 18 pages worth and the same number as you, then nothing.

The reason the pages don’t always return 200 tweets is because, as per the docs, deleted and unavailable tweets are removed after they’ve been queried. So, if you’ve deleted 10 of your last 200 tweets, when you try and load 200, you’ll get 190. Clearly the index is still kept somewhere and still counts towards the initial 200 total.

I imagine it’s the same with favourites, so I can only guess the link to that tweet being a favourite is still stored somewhere and once you’ve favourited it, even if you un-favourite it, it’ll still get removed from the results. This makes sense; if you have followed someone for ages, unfollow them, and follow them again, you don’t appear at the top of their followers list, you appear way down the list where you were before; clearly that link is still kept somewhere, and it must be the same with favourites, i.e once the link is made, it’s only marked as ‘inactive’ if you then undo it.

I created a topic here Tweet index limits about the limits, because the 3,200 limit has been present for years, way back to when the Fail Whale was a daily occurrence; Twitter’s infrastructure has come a long way since then but unfortunately these API limits are still imposed.

I think the official response from Twitter, if one was ever given, would be “tough, this is how it is”.

¯_(ツ)_/¯