Why am I limited to requesting ~390,000 user look ups?

tweepy
limits

#1

I am trying to get the profiles from a list with ~1million user IDs. I am using tweepy and relevant code fragment :

auth = tweepy.AppAuthHandler(consumerkey,consumersecret)
api = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True)
followersL=[]

for i in range(0, len(followersIDL), 100):
    while True:
        try : 
            followersL.extend(api.lookup_users(user_ids=followersIDL[i:i+100]))
            time.sleep(3)
        except tweepy.TweepError as error :
            print("...Exception : api_code {} len(followersL) = {} : {}".format(                   
                error.__dict__['api_code'],len(followersL),                         
                time.strftime("%a, %d %b %Y %H:%M:%S ", time.localtime())))
            time.sleep(300)
            continue
        break   

After collecting about 390,000 profiles, I get stuck in the exception catching part of the loop. I’ve extended the time.sleep(300) -> time.sleep(3600*2) and this still has not helped matters. The relevant exception is :

tweepy.error.TweepError: Failed to send request: HTTPSConnectionPool(host='api.twitter.com', port=443): Max retries exceeded with url: /1.1/users/lookup.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x1c5976240>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))

I am perplexed by this issue b/c I think that I’m respecting the user/lookup limits by sleeping 3 seconds between requests.

QUESTION : How do I get past this apparent absolute limit of ~390k user profile lookups?


#3

That is really strange as I have never come up against any issues when using the tweepy wait_on_rate_limit constraint, it might be good to check if you are running anything else in parallel. Another possible problem is that you exceeded the rate limit in a very short period of time before in which case I have found I have had issues with twitter allowing access (normally waiting a day sorts it out). Hope this helps (sorry for the double post)


#4

I’m not exceeding the published limits (either 900 requests per 15minutes or 300 requests per 15minutes). Notice the sleep(3). I’ve even extended it up to 10s between requests.


#7

@lee34422566- We might be able to help you. Please reach out to our enterprise sales team for more details.


#8

Thanks, I filled out an application over a week ago and have not heard back. How long does it take to hear back from them?


#9

It should be pretty quickly. Have you since heard back from them?


#10

I have not heard back.

Thanks.


#11

What is the company name and email address that you applied with?


#12

I don’t recall which company name I used. Probably something like “Personal.” Could I send you the email address privately so that it isn’t on this public forum? Thanks.


#13

I’ll start a private message with you now.
Thanks!


#14