Mismatch between requests used and tweets downloaded - Search API

[DBProject3]

I had used 53 request from the premium search API but I only downloaded 760 tweets. If i’m not wrong with sandbox you can download 100 tweets/ request, that means that I should had got 5300 tweets not 760.

Revising my Logs, I can see that for exemple: WaltDisneyCo, I had download from From: Tue Oct 09 17:32:59 +0000 2018 To: Tue Oct 23 00:28:46 +0000 2018 (9 tweets). Afterwards, The next request jumped From: Tue Oct 09 17:32:59 +0000 2018 To: Tue Oct 23 00:28:46 +0000 2018 (9 tweets). Seems like each request only download tweets between months not taking in account they were not enough to fill in the request with 100 tweets.

My query : “query”:“from:398282459”, “fromDate”:“201101010000”, “toDate”:“201811010000”

Any idea what could had happened?

Thank you.

[Aurelia]

Hi @DBProject3 - thanks for writing in. I was able to reproduce and, indeed, this is not the kind of behaviour that we would expect. I am going to raise this with our Product team and I will let you know when I have more information.

Thank you for your patience and apologies for any inconvenience caused.

[DBProject3]

Hi @ AureliaSpecker,

I understand what is said here: Premium Search | Full Archive.

But my logs do not make sense anyways. 30 days rule are not accomplished.

INFO - Successfully downloaded --> From: Tue Oct 09 17:32:59 +0000 2018 To: Tue Oct 23 00:28:46 +0000 2018 - 2018-11-17 13:38:40 (9 tweets)

INFO - Successfully downloaded --> From: Wed Sep 12 20:59:30 +0000 2018 To: Wed Sep 26 19:41:34 +0000 2018 - 2018-11-17 13:38:41 (8 tweets)

INFO - Successfully downloaded --> From: Wed Aug 15 23:09:47 +0000 2018 To: Thu Sep 06 18:34:11 +0000 2018 - 2018-11-17 13:38:44 (5 tweets)

INFO - Successfully downloaded --> From: Mon Jul 16 18:27:33 +0000 2018 To: Fri Aug 10 20:37:19 +0000 2018 - 2018-11-17 13:38:47 (8 tweets)

¿Is it possible that the 30 day rule is applied to the last date downloaded in the previous request?

This way I’m losing tweets, because the API already knows that there are no tweets in these days, but It still look for them in the next request.

I mean, If I start on 31/10. There are only tweets until 09/10. That means that between 09/10 - 31/09 there are no tweets (30 days rule). But the next request will look for tweets between 09/10 - 09/09 even knowing that there are no tweets between 09/10 - 31/09.

Do you know something more about this topic?

Thank you, a lot, for your answer and patience.

[Aurelia]

Hi @DBProject3 - thanks for sending this additional detail. I am still looking into this and I will let you know when I have more information. Thank you for your patience.

[DBProject3]

Sorry for my insistence, but ¿do you know somehting more about the topic?

I would like to buy 1000 request for the search API before December starts. But I can’t download what I want if I do not know how exactly works or if there are an error.

Thank you

[Aurelia]

Hi @DBProject3 - unfortunately no, we are still looking into this issue. Please expect a delay due to Thanksgiving. I’m sorry about any inconvenience caused and thank you for your patience.

[LauraW]

I think this is relevant to this post, because I believe I am also having a similar problem with the premium full archive. I’m sending this request:
screen_names=[‘JasonCrowCO6’]
max_count=500
fromDate=“201704110000”
toDate=“201807240000”
query_name=‘from:’+screen_names[sn]

search_params = {‘query’: query_name,
‘toDate’: toDate,
‘fromDate’: fromDate,
‘maxResults’: 500
}
“requestParameters” of the return tweet object shows this:
{‘maxResults’: 500, ‘fromDate’: ‘201704110000’, ‘toDate’: ‘201807240000’}

I’m only getting one month back of requests even though I should be getting 500. I only received tweets from Jul 21 2018 to Jun 23 2018, even though a “next” token is being produced.

As additional info–which I don’t think is the source of the problem after a lot of digging–I just switched to Premium API, but my code was working well with the Sandbox FullArchive before I switched. I made the changes to the Dev Env it was looking for and changed my tokens and key, but it is not responding with the right number of tweets (not even 100, which I was at least getting in Sandbox FullArchive).

[Aurelia]

Hi @LauraW - this doesn’t seem to be related. I reproduced your query and I was able to get information from earlier that Jun 23 2018, using the next token to paginate through the data. The first ‘page’ of data returns Tweets going back to Jun 23 2018. However, after that you should be able to retrieve Tweets from April 2017, as indicated by the fromDate parameter in your query.

Please read our documentation on pagination here.

[LauraW]

I read it numerous times, but must be missing something. I am still only getting one month at a time and just wasted 13 requests getting to the end of the time. Why is it not giving the maximum results (500)? It gave me the full 100 tweets each time when I used the free version.

[DBProject3]

Any news about the main topic @Aurelia?

[hmz]

@LauraW at the bottom of your payload, do you see a ‘next’ token?

E.g. "next": "eyJhdXRo...

Premium users get 500 Tweets per data requests, but Premium Sandbox users get 100 Tweets per data request.

Hamza

[DBProject3]

I payed for a subscription for the full-archive.

Using this query: {“companies”:[‘3MNews’,‘AmexBusiness’,‘Chevron’,‘CocaColaCo’,‘DowDuPontCo’,‘HomeDepot’,‘JNJNews’,‘Nike’,‘pfizer_news’,‘UnitedHealthGrp’,‘VerizonNews’,‘WaltDisneyCo’],
“fromDate”:[‘200706100000’],
“toDate”:[‘201811290000’]}

It was working well but suddenly It stopped at May 6. There, I wasted 146 request in blanck and I suspect it’s correlated. It is not able to found any tweet in the next 30 days and it start another time from the las date to look for tweets in the next 30 days. But the next request start the same day but just a few secons after. Here u can see my logs.

INFO - Successfully downloaded --> From: Sat May 06 09:01:27 +0000 2017 To: Sat May 06 09:01:30 +0000 2017 - 2018-11-29 19:07:39
INFO - Successfully downloaded --> From: Sat May 06 09:01:24 +0000 2017 To: Sat May 06 09:01:27 +0000 2017 - 2018-11-29 19:07:42
INFO - Successfully downloaded --> From: Sat May 06 09:01:20 +0000 2017 To: Sat May 06 09:01:24 +0000 2017 - 2018-11-29 19:07:46
INFO - Successfully downloaded --> From: Sat May 06 09:01:17 +0000 2017 To: Sat May 06 09:01:20 +0000 2017 - 2018-11-29 19:07:49
INFO - Successfully downloaded --> From: Sat May 06 09:01:14 +0000 2017 To: Sat May 06 09:01:17 +0000 2017 - 2018-11-29 19:07:53
INFO - Successfully downloaded --> From: Sat May 06 09:01:10 +0000 2017 To: Sat May 06 09:01:14 +0000 2017 - 2018-11-29 19:07:57
INFO - Successfully downloaded --> From: Sat May 06 09:01:07 +0000 2017 To: Sat May 06 09:01:10 +0000 2017 - 2018-11-29 19:08:00
INFO - Successfully downloaded --> From: Sat May 06 09:01:04 +0000 2017 To: Sat May 06 09:01:07 +0000 2017 - 2018-11-29 19:08:04
INFO - Successfully downloaded --> From: Sat May 06 09:01:01 +0000 2017 To: Sat May 06 09:01:04 +0000 2017 - 2018-11-29 19:08:07
INFO - Successfully downloaded --> From: Sat May 06 09:00:57 +0000 2017 To: Sat May 06 09:01:01 +0000 2017 - 2018-11-29 19:08:11
INFO - Successfully downloaded --> From: Sat May 06 09:00:54 +0000 2017 To: Sat May 06 09:00:57 +0000 2017 - 2018-11-29 19:08:14

I had downloaded 150000 tweets without any problem before this query, so my app is working well.

Ideas?

Thank you.

[LauraW]

Hamza, please notice my previous note. I am using Premium Full Archive. I am not even getting 100 tweets back per request.I am picking up the next token each time through and feeding it back in. As stated, my app worked previously and when I moved to premium it stopped producing full amount of tweets in the return. DBProject3 and I have been waiting a long time for actual help. How can we get some help? We are both paying for this service. I can’t keep waiting days in between responses, I am trying to complete a project.

[DBProject3]

I realy need a solution. I’m waisting payed request on Sat May 06 09:00:54 +0000 2017. I Added McDonaldsCorp to be sure that the API could found tweets on Sat May 06 but it do not work it.

If I start on 05 May 2017 it works well. Is there any problem at 06 May 2017?

[LeBraat]

@LauraW - Sorry we let this get out of control. Will you please start a new topic related to your issue? This topic relates to something different from what you are working on.

[LeBraat]

@DBProject3 - The original reason for this topic is on our product team’s radar, but they haven’t had the chance to investigate yet. I will keep you updated.

As for the more recent situation, will you please submit a new topic? We try to keep each topic to be working through a single issue, and I’m pretty sure the two issues that you’ve raised here are unrelated.

[DBProject3]

Okei I will open a new topic.

Thank you.

[LeBraat]

Thank you. Sorry for the inconvenience.

[LauraW]

Thanks. Sorry @DBProject3 , I didn’t mean to distract from your issue. It sounded similar initially to me.

This is a new post I started on this topic this morning.

https://twittercommunity.com/t/premium-full-archive-not-returning-500-tweets/117829

[LeBraat]

Thanks @LauraW! Checking now.