I’m using the API to scrape the tweets of users from a list. While this process works for quite a while, after several hours, I get an error that tells me that the user is suspended (Code 63) - but when I look up these users manually, they have not been suspended. Any idea what issue is causing this? Could it possibly be a rate-limiting issue gone awry?
That might be possible, but I’ve never seen or heard of this before. Are you able to share any code snippets and examples that might reproduce the issue you’re seeing?
Here is the code - I run it from a list of usernames in a ‘ids’. It usually takes several hours for the error to appear.
for person in ids: try: #print person rate_limit_status = twitter_stream.application.rate_limit_status() remaining = rate_limit_status["resources"]["users"]["/users/show/:id"]["remaining"] if remaining<5: print "Sleeping (Rate Limit)" time.sleep(900) username = (twitter_stream.users.show(user_id=person))['screen_name'] print username i1 = (twitter_stream.friends.ids(screen_name=username)) ids_friends = i1['ids'] i2 = (twitter_stream.followers.ids(screen_name=username)) ids_followers = i2['ids'] for id1 in ids_friends: if id1 in ids_followers: if id1 not in ids: ids.append(id1) #print ids for id_num in ids: if id_num not in marked: p = (twitter_stream.users.show(user_id=id_num)) username = p['screen_name'] protected = p['protected'] marked.append(id_num) if not protected: iterator = twitter_stream.statuses.user_timeline(screen_name=username,count=32000) print username if username not in read_main: screen_names.append(username) f_main.write(username + "\n") f = open("origin/"+ username + ".txt", "a") for tweet in iterator: f.write(json.dumps(tweet)+'\n') f.close() except Exception as e: print e print "Rate Limiting - Sleeping" time.sleep(900)
Have you been able to isolate specific users this always happens with, or is this random?
Aside from always occurring a few hours into the program, it’s random. Once it occurs for one user, I am never able to scrape accurately until I kill the program and restart - every following user also appears to be suspended, even when they are not.