Why am I limited to requesting ~390,000 user look ups?



I am trying to get the profiles from a list with ~1million user IDs. I am using tweepy and relevant code fragment :

auth = tweepy.AppAuthHandler(consumerkey,consumersecret)
api = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True)

for i in range(0, len(followersIDL), 100):
    while True:
        try : 
        except tweepy.TweepError as error :
            print("...Exception : api_code {} len(followersL) = {} : {}".format(                   
                time.strftime("%a, %d %b %Y %H:%M:%S ", time.localtime())))

After collecting about 390,000 profiles, I get stuck in the exception catching part of the loop. I’ve extended the time.sleep(300) -> time.sleep(3600*2) and this still has not helped matters. The relevant exception is :

tweepy.error.TweepError: Failed to send request: HTTPSConnectionPool(host='api.twitter.com', port=443): Max retries exceeded with url: /1.1/users/lookup.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x1c5976240>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))

I am perplexed by this issue b/c I think that I’m respecting the user/lookup limits by sleeping 3 seconds between requests.

QUESTION : How do I get past this apparent absolute limit of ~390k user profile lookups?


That is really strange as I have never come up against any issues when using the tweepy wait_on_rate_limit constraint, it might be good to check if you are running anything else in parallel. Another possible problem is that you exceeded the rate limit in a very short period of time before in which case I have found I have had issues with twitter allowing access (normally waiting a day sorts it out). Hope this helps (sorry for the double post)


I’m not exceeding the published limits (either 900 requests per 15minutes or 300 requests per 15minutes). Notice the sleep(3). I’ve even extended it up to 10s between requests.