Paging is not possible with /statuses/retweeters/ids.json

restapi
rest
retweet
api

#1

(This is also available as a gist: https://gist.github.com/jeremylow/3e4d6b0a18370f7444bb84ccdc136f2d)

Problem

For endpoint /statuses/retweeters/ids.json, there is no way to page through ids returned by API. No combination of setting cursor or count parameters has any effect on return value of previous_cursor, next_cursor, etc. Documentation is here:

https://dev.twitter.com/rest/reference/get/statuses/retweeters/ids

Furthermore, although count parameter is honored, is not documented.

Info

Example tweet is here: https://twitter.com/jack/statuses/397 (I thought since this is quite old, tweeters may have deleted/suspended accounts such to bring the count < 100, so this tweet was used as well https://twitter.com/POTUS/status/757763701101490176 with > 540K retweeters.)

System setup:

Python 3.5.1
pyenv 20160629-19-ga761ff6
pip 8.1.2 from /home/jeremy/.pyenv/versions/3.5.1/envs/python-twitter-3.5.1/lib/python3.5/site-packages (python 3.5)
requests.__version__ == 2.10.0

No query string

GET url: https://api.twitter.com/1.1/statuses/retweeters/ids.json?id=397

b'{"ids":[2766771192,1018421041,65555663,2883171442,2495447557,2329303118,400773576,2951536813,2354858683,2805218958,2921074850,717950795,2298780054,2956277463,2932914681,594152187,2149827450,1280652686,255149618,2253500994,1154184554,1028143183,2959583048,2812607934,2950529331,2228419408,2209549874,506047908,1321986698,1678837003,1391372328,876733711,2810075863,1092858866,1190783960,1462374198,2601544682,2904210737,2974097771,2954764264,2174326285,2579330610,2312150604,2935300107,2880845950,2491777656,2904960699,2504816257,2792158449,2966376340,2923958981,487115006,1155588638,2937463192,1701275521,375847468,87824707,2831786788,738630438,2920496680,1260643814,542483377,173699661],"next_cursor":0,"next_cursor_str":"0","previous_cursor":0,"previous_cursor_str":"0"}'

Setting cursor=-1 has no effect

GET url: https://api.twitter.com/1.1/statuses/retweeters/ids.json?id=397&cursor=-1

b'{"ids":[2766771192,1018421041,65555663,2883171442,2495447557,2329303118,400773576,2951536813,2354858683,2805218958,2921074850,717950795,2298780054,2956277463,2932914681,594152187,2149827450,1280652686,255149618,2253500994,1154184554,1028143183,2959583048,2812607934,2950529331,2228419408,2209549874,506047908,1321986698,1678837003,1391372328,876733711,2810075863,1092858866,1190783960,1462374198,2601544682,2904210737,2974097771,2954764264,2174326285,2579330610,2312150604,2935300107,2880845950,2491777656,2904960699,2504816257,2792158449,2966376340,2923958981,487115006,1155588638,2937463192,1701275521,375847468,87824707,2831786788,738630438,2920496680,1260643814,542483377,173699661],"next_cursor":0,"next_cursor_str":"0","previous_cursor":0,"previous_cursor_str":"0"}'

Setting count < [# retweeters] has no effect

GET url: https://api.twitter.com/1.1/statuses/retweeters/ids.json?id=397&count=5

b'{"ids":[2766771192,1018421041,65555663,2883171442,2495447557],"next_cursor":0,"next_cursor_str":"0","previous_cursor":0,"previous_cursor_str":"0"}'

Setting count to False has no effect.

GET url: https://api.twitter.com/1.1/statuses/retweeters/ids.json?id=397&count=False

b'{"ids":[2766771192,1018421041,65555663,2883171442,2495447557,2329303118,400773576,2951536813,2354858683,2805218958,2921074850,717950795,2298780054,2956277463,2932914681,594152187,2149827450,1280652686,255149618,2253500994,1154184554,1028143183,2959583048,2812607934,2950529331,2228419408,2209549874,506047908,1321986698,1678837003,1391372328,876733711,2810075863,1092858866,1190783960,1462374198,2601544682,2904210737,2974097771,2954764264,2174326285,2579330610,2312150604,2935300107,2880845950,2491777656,2904960699,2504816257,2792158449,2966376340,2923958981,487115006,1155588638,2937463192,1701275521,375847468,87824707,2831786788,738630438,2920496680,1260643814,542483377,173699661],"next_cursor":0,"next_cursor_str":"0","previous_cursor":0,"previous_cursor_str":"0"}'

GET url: https://api.twitter.com/1.1/statuses/retweeters/ids.json?id=397&count=False&cursor=-1

b'{"ids":[2766771192,1018421041,65555663,2883171442,2495447557,2329303118,400773576,2951536813,2354858683,2805218958,2921074850,717950795,2298780054,2956277463,2932914681,594152187,2149827450,1280652686,255149618,2253500994,1154184554,1028143183,2959583048,2812607934,2950529331,2228419408,2209549874,506047908,1321986698,1678837003,1391372328,876733711,2810075863,1092858866,1190783960,1462374198,2601544682,2904210737,2974097771,2954764264,2174326285,2579330610,2312150604,2935300107,2880845950,2491777656,2904960699,2504816257,2792158449,2966376340,2923958981,487115006,1155588638,2937463192,1701275521,375847468,87824707,2831786788,738630438,2920496680,1260643814,542483377,173699661],"next_cursor":0,"next_cursor_str":"0","previous_cursor":0,"previous_cursor_str":"0"}'

Setting count to false has no apparent effect.

GET url: https://api.twitter.com/1.1/statuses/retweeters/ids.json?id=397&count=false

b'{"ids":[2766771192,1018421041,65555663,2883171442,2495447557,2329303118,400773576,2951536813,2354858683,2805218958,2921074850,717950795,2298780054,2956277463,2932914681,594152187,2149827450,1280652686,255149618,2253500994,1154184554,1028143183,2959583048,2812607934,2950529331,2228419408,2209549874,506047908,1321986698,1678837003,1391372328,876733711,2810075863,1092858866,1190783960,1462374198,2601544682,2904210737,2974097771,2954764264,2174326285,2579330610,2312150604,2935300107,2880845950,2491777656,2904960699,2504816257,2792158449,2966376340,2923958981,487115006,1155588638,2937463192,1701275521,375847468,87824707,2831786788,738630438,2920496680,1260643814,542483377,173699661],"next_cursor":0,"next_cursor_str":"0","previous_cursor":0,"previous_cursor_str":"0"}'

GET url: https://api.twitter.com/1.1/statuses/retweeters/ids.json?id=397&count=false&cursor=-1

b'{"ids":[2766771192,1018421041,65555663,2883171442,2495447557,2329303118,400773576,2951536813,2354858683,2805218958,2921074850,717950795,2298780054,2956277463,2932914681,594152187,2149827450,1280652686,255149618,2253500994,1154184554,1028143183,2959583048,2812607934,2950529331,2228419408,2209549874,506047908,1321986698,1678837003,1391372328,876733711,2810075863,1092858866,1190783960,1462374198,2601544682,2904210737,2974097771,2954764264,2174326285,2579330610,2312150604,2935300107,2880845950,2491777656,2904960699,2504816257,2792158449,2966376340,2923958981,487115006,1155588638,2937463192,1701275521,375847468,87824707,2831786788,738630438,2920496680,1260643814,542483377,173699661],"next_cursor":0,"next_cursor_str":"0","previous_cursor":0,"previous_cursor_str":"0"}'

Setting cursor to random number has no effect

GET url: https://api.twitter.com/1.1/statuses/retweeters/ids.json?id=757763701101490176&cursor=2

b'{"ids":[1051830008,143177815,706875716340686849,114917463,759298774069940224,2848697050,3990477979,722783383647055872,339951175,2647581335,2803545588,2330749884,758573389183262721,975501115,2587732459,498230047,2361850597,4769403156,29107855,20385450,2680014975,190878038,2812446430,702121650683297793,131752228,3861073699,702216241554071553,1461937674,3136820547,360197633,578156311,759264117957984264,35084722,2229515437,602505803,245583615,485241166,795395245,2903684148,1317930270,759095590508695552,3099653994,2311236067,1925706102,2315961918,2176514426,838962740,2260951883,2489791377,1379253114,197281207,2235300116,2502852487,3432053626,122133687,176591472,816289867,1858415006,757729951546118144,1132831700,2233480638,1491635718,840979369,112603618,759228995481239554,22859238,2290702819,3065505324,788488850,759242112240058374,64611681,35664127,333843236,2545637816,172985987,1063614912,274777522,750503536006406144,1858630219,688513,1251867318,2878557819,2564723035,724128202470264832,738880291],"next_cursor":0,"next_cursor_str":"0","previous_cursor":0,"previous_cursor_str":"0"}'

compare with:

GET url: https://api.twitter.com/1.1/statuses/retweeters/ids.json?id=757763701101490176&cursor=-1

b'{"ids":[1051830008,143177815,706875716340686849,114917463,759298774069940224,2848697050,3990477979,722783383647055872,339951175,2647581335,2803545588,2330749884,758573389183262721,975501115,2587732459,498230047,2361850597,4769403156,29107855,20385450,2680014975,190878038,2812446430,702121650683297793,131752228,3861073699,702216241554071553,1461937674,3136820547,360197633,578156311,759264117957984264,35084722,2229515437,602505803,245583615,485241166,795395245,2903684148,1317930270,759095590508695552,3099653994,2311236067,1925706102,2315961918,2176514426,838962740,2260951883,2489791377,1379253114,197281207,2235300116,2502852487,3432053626,122133687,176591472,816289867,1858415006,757729951546118144,1132831700,2233480638,1491635718,840979369,112603618,759228995481239554,22859238,2290702819,3065505324,788488850,759242112240058374,64611681,35664127,333843236,2545637816,172985987,1063614912,274777522,750503536006406144,1858630219,688513,1251867318,2878557819,2564723035,724128202470264832,738880291],"next_cursor":0,"next_cursor_str":"0","previous_cursor":0,"previous_cursor_str":"0"}'

Potential Issue with Statuses/Retweeters
#2

Thanks for the examples!

count=False shouldn’t work, setting cursors to anything apart from “-1” or the specific integer returned by next_cursor or previous_cursor also shouldn’t work - so that’s fine.

But setting count=10 & cursor=-1 and not getting a next cursor when there are more results looks like a bug. But it’s not that severe - since you can get the maximum available results with count=100 in a single call anyway.

Since the endpoint returns 100 latest results, and then filters suspended / private accounts afterwards - even for a tweet with 540k retweets you may get <100 results - (i got 87 with 757763701101490176 tweet)

Hope that helps!


#3

Yeah, it’s definitely a bug. I was looking into this issue:

https://github.com/bear/python-twitter/issues/306

Since the documentation is incorrect, it’s a problem for our library.


#4

Looks like this still hasn’t been resolved. There’s no way to get more than the first 100 retweeter IDs for a tweet.


#5

They say it’s intention and that the docs shouldn’t give indication that it can be paginated.


#6

So the fact that it actually gives a “next_cursor” and “previous_cursor” value, but those are 0… also not indicative of a broken feature?


#7

You might think so, but these are actually the result of some internal code paths that cause timelines to be hydrated with cursor values. I agree that they should not be returned in this case, but this is older code which was built before the current code stack and is less straightforward to remediate. I’d love to have this cleaned up, but that’s not a high priority at this time.


#8

Could we bump this in priority? Seems like a pretty glaring omission from your API considering the importance of re-tweets in disseminating (dis)-information throughout your platform. While I am sure your search algorithm is proprietary, I am willing to bet that the re-tweet count of a status plays a role in its search ranking.

Since there is no way to obtain a complete list of all users who re-tweeted a particular status, I have to take your word on the validity of the re-tweet count. Even if the count is accurate, how many of the underlying accounts are valid (ie not bots)? How many of those accounts exist solely to re-tweet particular users? When were these accounts created? Right now there is no way of obtaining that information.

Like it or not, Twitter has become an avenue of political discourse. Thus far, Twitter has proven unable or unwilling to put safeguards in place ensuring this discourse is not hijacked or manipulated. For transparencies sake, it is imperative you give your API users the access they need to perform some analytics on how your platform is being used.


#9

Is there a ticket somewhere that I can watch regarding this issue?


#10

I’ve explained clearly that this is not a deliberate omission and that this is an historical fact of the API. The current roadmap is at https://trello.com/b/myf7rKwV/twitter-api-platform-roadmap

There is an API for engagement data and that is available commercially.

We are unable to assist with this issue further via this thread. Thank you.


#11