503 response status code problem

[DiploOffDuty]

Hi,

I’ve written code in python for followers and friends scraping. Everything was working perfectly until I’ve tried to download accounts from @setavakfi account (>60,000 followers). After 40 calls (8000 accounts total - 200 per call) I’m getting 503 error (happens in every try). I’ve tried (multiple times in last two days) to re-start script once again from the last saved cursor (1574812962976647290) but got same response status code immediately (503). Response content is: {“errors”:[{“message”:“Over capacity”,“code”:130}]}).
Requests are user authenticated and made with 61 seconds sleep time to avoid hitting the 15 minute limit.
So far it seems that problem is only related to one account (@setavakfi). Calls to other accounts with 60,000+ followers work perfectly. Has anyone encounter same error?

[semifor]

I was able to successfully fetch all of @setavakfi’s followers with 341 API calls. I’m using perl’s Twitter::API library with trait RetryOnError. The key difference may simply be retries.

Any 5XX error is generally retryable immediately. Twitter::API uses progressive backoff. On a 5XX error, it delays 250ms and retries the call. On each subsequent 5XX failure, it doubles the delay and retries again, up to 5 times.

Try adding a retry loop with progressive backoff.

It does seem strange that you consistently get a 503 after 40 calls, though. In my experience, there’s no consistency to the 503 errors—they generally occur quite randomly.

[DiploOffDuty]

Thanks for reply. I have added simple progressive retry loop. Same results. Still getting api error code 130 “Over capacity”. Tried with different twitter account and different ISP and same error occurs.

I’m pasting my code - can somebody test it and check what seems to be a problem?

Thanks in advance!

import tweepy
import time
import csv
import sys
import random

consumer_key = ''
consumer_secret = ''
access_token = ''
access_token_secret = ''

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True, retry_errors=set([401, 404, 500, 502, 503, 504]))

account = 'setavakfi'
log_file = '{}_followers.csv'.format(account)
retry = 0
errorCount = 0

tweepy_cursor = tweepy.Cursor(api.followers, screen_name=account, count=200, cursor=1574812962976647290).pages()
followers_count = []
while True:
    try:
        retry =0
        user = tweepy_cursor.next()
        cursor = tweepy_cursor.next_cursor
        followers_count += user
        print 'Retrieved {} followers accounts'.format(len(followers_count))
        print 'Current cursor: {}'.format(cursor)
        with open(log_file, 'ab') as fd:
            writer = csv.writer(fd)
            for i, user in enumerate(user): 
                writer.writerow([str("@"+user.screen_name), unicode(user.name).encode('utf-8'), str(user.lang), unicode(user.location).encode('utf-8'), str(user.geo_enabled)])
        print "Resting..."
        time.sleep(random.randint(60, 70)  
                    
    except tweepy.TweepError as e:
        print "Error code: {} with message: {}".format(e.api_code, e.message[0]['message'])
        errorCount +=1
        retry += 1
        print 'Retrying in {} seconds'.format(60+retry*5) 
        time.sleep(60+retry*5)
        if retry == 10:
            break

    except StopIteration:
        break   
    
print 'Done with {} errors'.format(errorCount)