Rate Limited when uploading a tailored audience using python

[stephenjwz]

Hi

I am trying to create a tailored audience based on a list of hashed emails.
It seems to work with small lists (they’re showing as processing but no errors showed up), but if they’re big (> 60mb at least, one at 90mb is definitely failing) I am tending to get the below error.
As I understand it, twitter_ads creates chunks from the file I provide then uploads them one at a time? Is it trying to do this too quickly? I am using time.sleep() to space out requests to create lists, but is there something I need to do within the individual requests to make them take their time?

Any advice appreciated, thanks

Stephen

Traceback (most recent call last):
  File "G:\Decision_Science\Scripts\twitterTailoredAudience.py", line 58, in <module>
    audience = TailoredAudience.create(account, name+'twithash.csv', name, TA_LIST_TYPES.EMAIL)
  File "C:\Python27\lib\site-packages\twitter_ads\audience.py", line 31, in create
    getattr(audience, '__update_audience__')(upload.perform(), list_type, TA_OPERATIONS.ADD)
  File "C:\Python27\lib\site-packages\twitter_ads\http.py", line 260, in perform
    self.__upload_chunk(location, chunk_size, bytes, bytes_start, bytes_read)
  File "C:\Python27\lib\site-packages\twitter_ads\http.py", line 315, in
    __upload_chunk domain=self._DEFAULT_DOMAIN, headers=headers, body=bytes).perform()
  File "C:\Python27\lib\site-packages\twitter_ads\http.py", line 72, in perform
    raise Error.from_response(response)
twitter_ads.error.RateLimit: <RateLimit object at 0x3d75878 code=429 details=None

[juanshishido]

Thanks for reaching out, @stephenjwz. What version of the Python SDK are you using? In #109, which is associated with version 1.2.1, we increased the chunk size from 1MB to 64MB.

[stephenjwz]

Hi @juanshishido

I think 1.21.1 - I get the below if I try to update. I am still getting the same error.

Requirement already up-to-date: twitter_ads in c:\python27\lib\site-packages
Requirement already up-to-date: python-dateutil in c:\python27\lib\site-packages
(from twitter_ads)
Requirement already up-to-date: pyyaml in c:\python27\lib\site-packages (from tw
itter_ads)
Requirement already up-to-date: requests-oauthlib in c:\python27\lib\site-packag
es (from twitter_ads)
Requirement already up-to-date: six>=1.5 in c:\python27\lib\site-packages (from
python-dateutil->twitter_ads)
Requirement already up-to-date: requests>=2.0.0 in c:\python27\lib\site-packages
(from requests-oauthlib->twitter_ads)
Requirement already up-to-date: oauthlib>=0.6.2 in c:\python27\lib\site-packages
(from requests-oauthlib->twitter_ads)
Requirement already up-to-date: chardet<3.1.0,>=3.0.2 in c:\python27\lib\site-pa
ckages (from requests>=2.0.0->requests-oauthlib->twitter_ads)
Requirement already up-to-date: certifi>=2017.4.17 in c:\python27\lib\site-packa
ges (from requests>=2.0.0->requests-oauthlib->twitter_ads)
Requirement already up-to-date: idna<2.6,>=2.5 in c:\python27\lib\site-packages
(from requests>=2.0.0->requests-oauthlib->twitter_ads)
Requirement already up-to-date: urllib3<1.23,>=1.21.1 in c:\python27\lib\site-pa
ckages (from requests>=2.0.0->requests-oauthlib->twitter_ads)

[juanshishido]

@stephenjwz: I was able to successfully upload a 78MB tailored audience list file. This might be difficult to debug, but one thing that might help is getting the full trace of the requests. To do this, when you’re instantiating the Client class, use trace=True:

>>> client = Client(CONSUMER_KEY,
                    CONSUMER_SECRET,
                    ACCESS_TOKEN,
                    ACCESS_TOKEN_SECRET,
                    trace=True)

---

Just for reference, in order to check what version you’re using, you can do:

>>> import twitter_ads
>>> twitter_ads.__version__
'1.2.2'

[stephenjwz]

Finally got around to testing this! Got the below error, which resembles what I get when trying to make too big of a http request normally?

Traceback (most recent call last):
File “S:\pythonWIP\twitterTeams.py”, line 103, in
audience = TailoredAudience.create(account, name+‘twithash.csv’, name, TA_LI
ST_TYPES.EMAIL)
File “C:\Python27\lib\site-packages\twitter_ads\audience.py”, line 31, in crea
te
getattr(audience, ‘update_audience’)(upload.perform(), list_type, TA_OPE
RATIONS.ADD)
File “C:\Python27\lib\site-packages\twitter_ads\http.py”, line 245, in perform

response = self.__upload(resource, open(self._file_path, 'rb').read())

File “C:\Python27\lib\site-packages\twitter_ads\http.py”, line 284, in __uploa
d
domain=self._DEFAULT_DOMAIN, headers=headers, body=bytes).perform()
File “C:\Python27\lib\site-packages\twitter_ads\http.py”, line 70, in perform
response = self.__oauth_request()
File “C:\Python27\lib\site-packages\twitter_ads\http.py”, line 95, in _oauth
request
files=files, stream=stream)
File “C:\Python27\lib\site-packages\requests\sessions.py”, line 555, in post
return self.request(‘POST’, url, data=data, json=json, **kwargs)
File “C:\Python27\lib\site-packages\requests\sessions.py”, line 508, in reques
t
resp = self.send(prep, **send_kwargs)
File “C:\Python27\lib\site-packages\requests\sessions.py”, line 618, in send
r = adapter.send(request, **kwargs)
File “C:\Python27\lib\site-packages\requests\adapters.py”, line 490, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (‘Connection aborted.’, error("(-1, ‘Unexpe
cted EOF’)",))
DEBUG:paramiko.transport:EOF in transport thread
INFO:paramiko.transport.sftp:[chan 0] sftp session closed

[imit8me]

Hey @stephenjwz,

Can you provide us the exact script you’ve used as well as details around the audience itself?

Thanks!

[juanshishido]

@stephenjwz: We’ve identified the issue here. I’ve submitted a pull request (#122) and hope to merge it in soon. Thanks for your patience and for raising this.

[stephenjwz]

Thats brilliant! sorry I haven’t been too forthcoming with replies, keep having to move onto other projects

[juanshishido]

@stephenjwz: We’ve released the update to the SDK. To update:

$ pip install twitter-ads --upgrade

Please let us know whether this resolved the issue. Thanks!