Analytics Jobs Taking a Long Time to Complete

[tboilard]

We are seeing asynchronous analytics jobs, requesting 3 days worth of data, taking over 1 hour to complete. Our polling task times out after 1 hour (with a Job status of PROCESSING) so we are not sure how long the jobs actually take to complete. This seems to be a relatively new phenomenon.

Here is some information on some recent jobs that failed. If you need any other information please let me know. Any insight into this would be appreciated.

Thanks for your help,
Tom Boilard

TwAnalyticsPollError: Job ID: 783209242736156672 Status: PROCESSING Job Data: {u’status’: u’PROCESSING’, u’created_at’: u’2016-10-04T07:36:49Z’, u’placement’: u’ALL_ON_TWITTER’, u’account_id’: u’9ukwn’, u’metric_groups’: [u’ENGAGEMENT’, u’BILLING’, u’VIDEO’, u’MEDIA’, u’WEB_CONVERSION’], u’url’: None, u’country’: None, u’start_time’: u’2016-10-04T04:00:00Z’, u’expires_at’: None, u’entity’: u’LINE_ITEM’, u’platform’: None, u’id_str’: u’783209242736156672’, u’granularity’: u’DAY’, u’entity_ids’: [u’3fl1f’, u’3hcwb’, u’3hcwy’, u’3hczj’, u’3hd0p’, u’3hdf8’, u’3hdgg’, u’3hdhg’, u’3hdi2’, u’3hf4a’, u’3hf4w’, u’3hf5a’, u’3hfiu’, u’3hftw’, u’3hfwl’, u’3hg3s’, u’3hg4z’, u’3hg5n’, u’3hg82’, u’3hg8r’], u’segmentation_type’: u’LANGUAGES’, u’id’: 783209242736156672, u’end_time’: u’2016-10-05T04:00:00Z’}

TwAnalyticsPollError: Job ID: 783208991493062656 Status: PROCESSING Job Data: {u’status’: u’PROCESSING’, u’created_at’: u’2016-10-04T07:35:49Z’, u’placement’: u’ALL_ON_TWITTER’, u’account_id’: u’9ukwn’, u’metric_groups’: [u’ENGAGEMENT’, u’BILLING’, u’VIDEO’, u’MEDIA’, u’WEB_CONVERSION’], u’url’: None, u’country’: None, u’start_time’: u’2016-10-04T04:00:00Z’, u’expires_at’: None, u’entity’: u’CAMPAIGN’, u’platform’: None, u’id_str’: u’783208991493062656’, u’granularity’: u’DAY’, u’entity_ids’: [u’3fxar’, u’3fxbh’, u’3w0qn’, u’3w61d’, u’3w61r’, u’3w800’, u’3w80k’, u’3w80x’, u’3w8a7’, u’3wdl4’, u’3wdlt’, u’4fflo’, u’5im3n’, u’5im5i’], u’segmentation_type’: u’PLATFORMS’, u’id’: 783208991493062656, u’end_time’: u’2016-10-05T04:00:00Z’}

TwAnalyticsPollError: Job ID: 783208990859821056 Status: PROCESSING Job Data: {u’status’: u’PROCESSING’, u’created_at’: u’2016-10-04T07:35:49Z’, u’placement’: u’ALL_ON_TWITTER’, u’account_id’: u’9ukwn’, u’metric_groups’: [u’ENGAGEMENT’, u’BILLING’, u’VIDEO’, u’MEDIA’, u’WEB_CONVERSION’], u’url’: None, u’country’: None, u’start_time’: u’2016-10-04T04:00:00Z’, u’expires_at’: None, u’entity’: u’CAMPAIGN’, u’platform’: None, u’id_str’: u’783208990859821056’, u’granularity’: u’DAY’, u’entity_ids’: [u’3crve’, u’3efl4’, u’3eflm’, u’3efo6’, u’3efpb’, u’3eg3n’, u’3eg4u’, u’3eg5u’, u’3eg6g’, u’3ehr8’, u’3ehru’, u’3ehs8’, u’3ei46’, u’3eif4’, u’3eihs’, u’3eiou’, u’3eiq1’, u’3eiqp’, u’3eit2’, u’3eitq’], u’segmentation_type’: u’PLATFORMS’, u’id’: 783208990859821056, u’end_time’: u’2016-10-05T04:00:00Z’}

TwAnalyticsPollError: Job ID: 783094681009881088 Status: PROCESSING Job Data: {u’status’: u’PROCESSING’, u’created_at’: u’2016-10-04T00:01:35Z’, u’placement’: u’ALL_ON_TWITTER’, u’account_id’: u’18ce53xpx8t’, u’metric_groups’: [u’ENGAGEMENT’, u’BILLING’, u’VIDEO’, u’MEDIA’, u’WEB_CONVERSION’], u’url’: None, u’country’: None, u’start_time’: u’2016-10-03T07:00:00Z’, u’expires_at’: None, u’entity’: u’CAMPAIGN’, u’platform’: None, u’id_str’: u’783094681009881088’, u’granularity’: u’DAY’, u’entity_ids’: [u’4qxjk’, u’4qxjt’, u’4qxjv’, u’4qxjx’, u’4qxk5’, u’4qxkq’, u’4qxkr’, u’4qxks’, u’4qxkt’, u’4qxkv’, u’4r7ws’, u’4rfkv’, u’4rmcr’, u’4scko’, u’4sdpw’, u’4seh7’, u’4suq1’, u’4thqf’, u’4tkow’, u’4tqtc’], u’segmentation_type’: None, u’id’: 783094681009881088, u’end_time’: u’2016-10-04T07:00:00Z’}

TwAnalyticsPollError: Job ID: 783112642722488320 Status: PROCESSING Job Data: {u’status’: u’PROCESSING’, u’created_at’: u’2016-10-04T01:12:58Z’, u’placement’: u’ALL_ON_TWITTER’, u’account_id’: u’18ce54dkeul’, u’metric_groups’: [u’ENGAGEMENT’, u’BILLING’, u’VIDEO’, u’MEDIA’, u’WEB_CONVERSION’], u’url’: None, u’country’: None, u’start_time’: u’2016-10-03T23:00:00Z’, u’expires_at’: None, u’entity’: u’PROMOTED_TWEET’, u’platform’: None, u’id_str’: u’783112642722488320’, u’granularity’: u’DAY’, u’entity_ids’: [u’ygyw9’, u’ygywa’, u’ygywb’, u’ygywc’, u’ygywd’, u’ygywe’, u’ygywf’, u’ygywg’, u’ygywh’, u’yn4a0’, u’yyyoa’, u’yz0i3’, u’yz0i4’, u’yz0i5’, u’yz0i6’, u’yz0i7’, u’yz0i8’, u’yz0i9’, u’yz0ia’, u’yz0ib’], u’segmentation_type’: u’LOCATIONS’, u’id’: 783112642722488320, u’end_time’: u’2016-10-04T23:00:00Z’}

[JBabichJapan]

Hi,

Thanks for this report - based on multiple reports on the forum here I took a look into this and I can confirm there is some delay going on, it appears to have started to recover already. The system is not invulnerable to partners hitting it for huge amounts of data in short periods of time which is one reason we continue to ask partners to follow best practices and not request data in redundant ways. Another possible reason is a temporary issue with backend - neither case usually last for more than a day.

We will continue to monitor the performance and post an announcement here if it does not recover.

Thanks,

John

[JBabichJapan]

Hi,

As of today the performance of async analytics seems back to normal levels - please post a new thread if you are continuing to see issues that you feel are unacceptable (including specifics helps a lot).

For polling, my expectation is that all jobs will be marked as either SUCCEED or be marked as FAILURE after 60 minutes. If this is not happening please let us know so I can follow up.

Thanks,

John

[dwillis]

I have some jobs that have been in Processing state for over an hour.

Are you still having issues?

1 of the jobs is 784023716158140416

“status”:“PROCESSING”,“granularity”:“DAY”,“entity”:“CAMPAIGN”,“created_at”:“2016-10-06T13:33:15Z”,"

My time zone is 4 hours behind gmt

This job has been running for 2 hours and remains in Processing state the entire time.

I have polled for it a half dozen times for the past 2 hours.

[JBabichJapan]

Hey, unfortunately it is difficult for us to investigate job-by-job status. We are still investigating this with high priority but it would help to confirm if you are still seeing degraded performance on your jobs.

Thanks

[tboilard]

HI John,

Thanks for your attention to this matter. We really appreciate it. We are
still seeing jobs taking longer than an hour (We timeout after 120 retries
at a 30 second cadence) so once again we are not sure exactly how long the
jobs are taking or wether they are even finishing.

Below is a screen grab of the last 60 hours from rollbar showing when we
are seeing timeouts (EST). If you need any other information please let me
know.

Thanks,
Tom Boilard