Understanding Twitter API Rate Limits

[vdirects]

I am trying to log information (name, user name, bio, etc.) of every follower for a specific user using Python

However, after 300 followers, I receive a rate limit error message.

Can someone explain why this is and how I can maximize the number of followers returned? Shouldn’t this number be higher, like 5,000?

How can I scan all of my followers if I have thousands, or millions of them?

[andypiper]

You can find information on our rate limits in the documentation. You’ll need to build-in logic to your app which waits between fetching groups of followers to avoid hitting the rate limit. You can use GET users/lookup instead of GET users/show to get more users at once.

[vdirects]

Even with built-in logic, gathering someone’s entire list of followers (if they have thousands) would take hours/days/weeks to complete at 15 requests per window.

• If one waits one minute in between requests does the counter reset to add 1 more request? Or do you have to wait 15 whole minutes after the 15th request

• What is considered a “request” in the API? I am not sure why I’m only getting 300 users when scanning one person’s list of followers when my app only calls get_user (screen_name) once, and then followers()

[andypiper]

do you have to wait 15 whole minutes after the 15th request

The request window means that if you have (say) 180 calls to a resource in a 15 minute window, and make 100 calls in 1 minute, then wait 5 minutes, and then make another 80 calls in 1 minute… (thus using up your allocation) you still have to wait another 8 minutes before the 15 minute / 180 call counter is reset. You’ll find information about the number of calls you have remaining, and time before the counter is reset, in the X-Rate-Limit HTTP headers.

What is considered a “request” in the API?

One call to a single endpoint is a request to that endpoint.

I’m not sure where the number of “300 users” is coming from. The followers/list endpoint returns batches of up to 20-200 users per screen_name, and is cursored.

The rate limits on the user objects are typically much lower than those for timelines, fetching Tweets, and searching. There are a number of good reasons for this and this is unlikely to change, so if you need to do what you are describing for users with large follower numbers I’m afraid it will take some time within the rate limits.

[vdirects]

thank you.

Here is an example of the python code I"m using. Are you able to tell where I’m getting 300 results from? I’m only making one get request (get_user). Where are 15 requests coming from? Shouldn’t this count as one request? Or are you able to tell me how I can maximize the results?

client = tweepy.API(auth)

followers = tweepy.Cursor(client.followers, id=screenName)
for follower in followers.items():
    info=[]
    name =follower.name
    screen_name = follower.screen_name

    userId = userId + 1

    info.append(userId)
    info.append(name)
    info.append(screen_name)

    csvFile = open('followers.csv','a')
    newFile =csv.writer(csvFile) #imported csv
    newFile.writerow(info)
    #close file
    csvFile.close()

[IgorBrigadir]

The “300 results” you are getting come from:

followers = tweepy.Cursor(client.followers, id=screenName)

this calls /followers/list 15 times(the rate limit), each retrieving 20 users (the default number of results per page) - giving you 300 results.

Use this instead:

followers = tweepy.Cursor(client.followers, id=screenName, count=200)

This will return 200 followers per call, which means you can retrieve 3000 followers per 15 minute window (15 calls each retrieving 200 users)

Alternatively - using Application Authentication, you can call this endpoint 30 times every 15 min. See https://dev.twitter.com/rest/public/rate-limits

If this is still too slow, combining /followers/ids with /users/lookup might be better, for example: Imagine you want to retrieve all of andypiper’s 10.6K followers:

First, use /followers/ids with count=5000 parameter - (This uses up 3 of 15 calls in a rate limit window)
Now you have ~10600 user ids, use /users/lookup retrieving info for 100 users per call (~106 calls from 180 limit)

[vdirects]

Use this instead:

followers = tweepy.Cursor(client.followers, id=screenName, count=200)

Thanks. Though, this still stops at 300. I’ve also tried:

 followers = tweepy.Cursor(client.followers, id=screenName).items(200)

Also, I am using Application Auth. I will attempt the followers/ids method.

[vdirects]

Turns out I had:

tweets = follower.statuses_count

Which, apparently can only make 300 requests. Now I am up to 3,000, but I can’t get the number of tweets of each follower.

[rchoi]

This topic was automatically closed after 36 hours. New replies are no longer allowed.