Sat May 06 2017 - Unable to download tweets for this date


#1

I payed for a subscription for the full-archive.

Using this query: {“companies”:[‘3MNews’,‘AmexBusiness’,‘Chevron’,‘CocaColaCo’,‘DowDuPontCo’,‘HomeDepot’,‘JNJNews’,‘Nike’,‘pfizer_news’,‘UnitedHealthGrp’,‘VerizonNews’,‘WaltDisneyCo’],
“fromDate”:[‘200706100000’],
“toDate”:[‘201811290000’]}

It was working well but suddenly It stopped at May 6. There, I wasted 146 request in blanck and I suspect it’s correlated.

INFO - Successfully downloaded --> From: Sat May 06 09:01:27 +0000 2017 To: Sat May 06 09:01:30 +0000 2017 - 2018-11-29 19:07:39
INFO - Successfully downloaded --> From: Sat May 06 09:01:24 +0000 2017 To: Sat May 06 09:01:27 +0000 2017 - 2018-11-29 19:07:42
INFO - Successfully downloaded --> From: Sat May 06 09:01:20 +0000 2017 To: Sat May 06 09:01:24 +0000 2017 - 2018-11-29 19:07:46
INFO - Successfully downloaded --> From: Sat May 06 09:01:17 +0000 2017 To: Sat May 06 09:01:20 +0000 2017 - 2018-11-29 19:07:49
INFO - Successfully downloaded --> From: Sat May 06 09:01:14 +0000 2017 To: Sat May 06 09:01:17 +0000 2017 - 2018-11-29 19:07:53
INFO - Successfully downloaded --> From: Sat May 06 09:01:10 +0000 2017 To: Sat May 06 09:01:14 +0000 2017 - 2018-11-29 19:07:57
INFO - Successfully downloaded --> From: Sat May 06 09:01:07 +0000 2017 To: Sat May 06 09:01:10 +0000 2017 - 2018-11-29 19:08:00
INFO - Successfully downloaded --> From: Sat May 06 09:01:04 +0000 2017 To: Sat May 06 09:01:07 +0000 2017 - 2018-11-29 19:08:04
INFO - Successfully downloaded --> From: Sat May 06 09:01:01 +0000 2017 To: Sat May 06 09:01:04 +0000 2017 - 2018-11-29 19:08:07
INFO - Successfully downloaded --> From: Sat May 06 09:00:57 +0000 2017 To: Sat May 06 09:01:01 +0000 2017 - 2018-11-29 19:08:11
INFO - Successfully downloaded --> From: Sat May 06 09:00:54 +0000 2017 To: Sat May 06 09:00:57 +0000 2017 - 2018-11-29 19:08:14

I had downloaded 150000 tweets without any problem before this query, so my app is working well.
Also, I tried to download using May 05 to the from date and worked well.
Ideas? Is there any special problem with May 06?

Thank you.


#2

Thank you for submitting a new topic.

I will dig in to see if I can find out what is going on here.


#3

After doing some investigations, I found that I was able to get through all of May 6th, so it isn’t that specific date that caused this situation.

This is still something that I’d like to put some deeper thought into. I am going to sync with our product team to see if they can investigate further.


#4

A couple of questions to help our team investigate:

What did the payload of these May 6th requests look like?

With the query, are you looking for Tweets that mention those companies, or Tweets that are posted by those companies? Or maybe something differently?

Did you manually stop the May 6th loop, or did it eventually end up moving on to a later time period?


#5

I was looking for tweets posted by those companies using from: operator. I stopped manually the process. The payload returned were 0 tweets + next token + dates.

It’s quite strange because I could download perfectly from 09 Feb 2008 to 05 May 2017 and from 07 May 2017 to 29 Nov 2018. But I could not get any data from 06 May 2017.
Thank you for your help.


#6

Could you please provide an example payload from May 6th?