Nondeterminstic behavior for lists/members/create_all

rate-limits
rest

#11

They say you have to add 100 at a time
https://dev.twitter.com/rest/reference/post/lists/members/create_all

I’m doing that and sleeping 100 seconds between subsequent calls but it still gets stuck after few times. Hoping @andypiper gets back to us, cheers.


#12

I first noticed this behavior around the beginning of July along with others that work with lists extensively. At the time, I assumed it to be a new, secret rate-limit (or at least was not updated in the docs, I know docs can trail behind). But, glad to see this thread since it is still happening and the docs have not been updated, if indeed it is a rate limit of sorts.

I can confirm that the call behaves differently with a identical set of user IDs, which was why I assumed it was some type of limiting.

I can also confirm that once it starts happening (not all users added in the create_all) then attempts in the native Twitter Apps/site to add a member to a list also fail, again pointing to some limit on the user in the number of memberships that can be created in a given period. The error message in the Twitter web site is “Your account may not be allowed to perform this action. Please refresh the page and try again.”


Lists/members/create_all not adding people to a list
#13

I’ve ran in the exact same problem.

Can’t figure the origin: not related to ids, number of calls, existing list or not, and no error code.
It just worked fine 2 months ago (well, not after creating several listsx). Now, it just add very few people (sometimes 0, sometimes ~ 20). So I need to to make the call dozen of times to add up to 3k members.

Other issues related:



#14

@andypiper Have you had a chance to dig into this? Let me know if you need more sample requests/description.


#15

Has anyone found a fix for this or a workaround? I’m unable to use create_all to add accounts. As others have said, there appears to be some type of limiting on Twitter’s end or just an outright bug. Sometimes 1 account out of my list will get added, sometimes 0, etc. It is very erratic.

I may just use the create endpoint to see if that works better, but adding the 1,000 max for a list would be a lot of unnecessary API calls.

Also, I’m getting a 200 response back from the API even though none of the ids in my list are ever added. It’s very strange.


#16

I’m getting this same behavior using the REST API. In addition, attempts to add any users to a list on the account I use for testing also result in the “your account may not be allowed to do this” error mentioned above.

Please? Is there a rate limit or limitation to be aware of so that we’re not triggering this result?

We’re limiting our calls to this API to once each 10 minutes, and a max of 100 users added to the list each time. Should be well below the API limits as published.


#17

I’m experiencing the same behavior here:

  • 200 ‘OK’ response from the API but members frequently not added
  • No rates limit appear to change after I call the create_all endpoint
  • As soon as I notice an issue, any attempt to add a user to a list through the twitter website result in the “your account may not be allowed to do this” message to pop

Am I missing something about this particular endpoint ? is there a limit on the number of users one can add to lists for a given amount of time ?


#18

We’re experiencing the exact same thing, for what it’s worth


#19

Can someone from Twitter put the time to at least confirm the existence of this bug? It’s been almost half a year and a lot of people are reproducing this issue.


#20

6 months later, same problem.

@andypiper please deliver us from this hell.


#21

Any updates @andypiper? It seems to be getting worse, and by that I mean more restrictive in how frequently members can be added. This double secret probation is frustrating for many folks I’m sure. Why not publish what the limits are? (assuming it’s a secret rate limit, because after a year, I can think of no other reason for this to continue).


#22

We’ve also ran into this problem as well. Even the curl command from the generate signature widget on https://dev.twitter.com/rest/reference/post/lists/members/create_all often adds no users to the list. This will happen repeatedly and often takes over an hour of retries (working well within the rate limits) gradually filling up the list with the twitter users we want in it.

In our case, the api returns no errors, it simply just doesnt add the users to the list.


#23

This is still happening. Can anyone from official team confirm? Please? @andypiper


#24

I’m unable to confirm any issues with this endpoint beyond the sporadic reports on this thread over an 18 month period. If you are able to share any further details then we may be able to investigate.


#25

There were examples provided over a year ago. If we provide you samples will you respond?


#26

Part of the question here is whether the accounts in question have (now or previously) been used for bulk automation of the lists endpoints. Bulk automation is generally against the rules and can cause app keys to be marked as spammy.

Certainly if there are verifiable / reproducible test cases and calls that consistently cause issues, then this is something that can be looked into, but bear in mind that the basic Twitter APIs don’t have any specific guarantees of success, so all we can do is make general investigations based on the information we are provided. A lot of the previous discussion on this thread is “me too” without specific failing test cases or code snippets and that is difficult to debug.


#27

I understand there can be many factors, I’ve just seen the same behavior described repeatedly and have myself reproduced across a number of different apps and account combinations.

As an example, let’s try to create a list of 2095 members (just looking here for samples: https://toplists.xyz/) and creating a copy in our own account to make further updates to it (i.e. fork the list like forking code).

https://twitter.com/susanbeebe/lists/social-media-smarties

First get the members of the existing list:

https://api.twitter.com/1.1/lists/members.json?owner_screen_name=susanbeebe&include_entities=false&skip_status=true&slug=social-media-smarties&count=5000

That would give us 2095 users in the response. Next, we want to execute create_all calls for those 2095 users. The docs indicate you can pass 100 users at a time, but that results in “HTTP 503 - Twitter is temporarily over capacity.” errors frequently, so for my test, let’s use batches of 20. This endpoint is not rate-limited, so we will not get an error during the process of making those 105 calls.

After running this, I end up with a list that now was 1104 members, nowhere near the 2095 in the source list. Additionally, whenever I now try to manually add somebody to a list, any list for my account, I get an error in the Twitter web app’s UI: “Your account may not be allowed to perform this action. Please refresh the page and try again.”

Here’s my sample ruby program that encountered no errors:

require "twitter"

client = Twitter::REST::Client.new do |config|
  config.consumer_key        = ""
  config.consumer_secret     = ""
  config.access_token        = ""
  config.access_token_secret = ""
end

puts "Retrieving list members..."
source_members = client.list_members(
  "susanbeebe", "social-media-smarties", { count: 2100, skip_status: false }
)
source_names = source_members.map(&:screen_name)
puts "#{source_names.count} in list"

puts "Create new list..."
new_list = client.create_list("create_all-test", { mode: "private" })

puts "Adding list members"
source_names.each_slice(20) do |slice|
  client.add_list_members(new_list.id, slice)
  puts "  -- added #{slice.count} members"
  sleep rand(5)
end

#28

Thanks this is a useful test case. I’ll see what I can discover based on this (but given my limited time that maybe take a while, patience appreciated as usual). Thank you!


#30

I have the exact same problem.

This is the “member_count” result of a very similar call @ryanwi has been using. For the first 1000 members it seems to work without problem, afterwards only a couple of the users get actually added to the list.

I am adding a 100 users per call, but after 10 calls no more real users are added.

“member_count”:96
"member_count":193
"member_count":291
"member_count":384
"member_count":477
"member_count":570
"member_count":672
"member_count":770
"member_count":861
"member_count":959
"member_count":1039
"member_count":1039
"member_count":1052
"member_count":1049
"member_count":1066
"member_count":1052
"member_count":1070
"member_count":1070
"member_count":1070
"member_count":1053
"member_count":1070
"member_count":1070
"member_count":1053
"member_count":1070
"member_count":1068
"member_count":1068
"member_count":1070
"member_count":1068
"member_count":1070
"member_count":1070
"member_count":1071
"member_count":1054
"member_count":1078
"member_count":1076
"member_count":1078
"member_count":1062
"member_count":1079
"member_count":1090
"member_count":1090
"member_count":1090
"member_count":1090
"member_count":1077
"member_count":1094
"member_count":1092
"member_count":1092
"member_count":1095
"member_count":1111
"member_count":1113
"member_count":1096
"member_count":1113


#31

For anyone interested / having the same problem:

The rate limit seems to be at arround ~1000 users that are being added to a list and resets every 24 hours.

Adding less than a 100 users per call or doing longer breaks between calls makes absolutely no difference. Once the limit is reached you are not able to add more users to a list, even if it is a brand new one. And you get a “Your account may not be allowed to perform this action. Please refresh the page and try again.” error within twitters web interface.

I couldn’t find any other solution than adding only a maximum of ~1000 users per day to a list.