Search API returning less result than usual

[camonmon]

Thanks Andy, we’ll be waiting.

[andypiper]

So you all know that we are indeed tracking this, our internal ticket is PREL-17886 and there are a few moving parts.

Please bear with us while we work on this. Thank you for your understanding and patience.

[camonmon]

Thanks Andy! We will stay tuned. Hopefully it gets fixed soon.

[reynaldborer]

According to our metrics, the issue was solved last night around 02:00 UTC, we are now seeing the same amount of tweets returned as before. Does any of you confirm the resolution?

Andy, is it fixed internally?

[camonmon]

We are seeing it better. However, when you look for keywords or Hashtags that are very heavy (Amazon, #Twitter) the results are less than the limits. It seems to be fixed for hashtags or keywords with not heavy load of tweets but for the big ones, it only returns 3000 or so.

Anyway, thanks Andy, you’re a true rockstar.

[reynaldborer]

Right sorry, I was comparing wrong metrics. There is indeed a clear improvement, but content is still missing, although far less than before.

[PruebasMolonas]

I agree with @reynaldborer the results the API are returning are not complete. It is more than before but not complete.

Looking forward to a fix here.

[andypiper]

Thanks for the patience.

Here’s what I can share - after we looked into what you reported here, we did fix an issue that had caused a bunch of Tweets to be excluded from the index (apologies), and you would have seen the change a little after 02:00 UTC, which matches @reynaldborer’s experience above.

If there is still a smaller result for certain search terms then it may be unrelated to the original issue, and I can see what else might be happening. Please bear in mind that although we are worldwide, we’re not a 24x7 team so I may not be able to share any more updates for a couple of days. Glad to hear that the original dip was at least resolved by the fix I referred to just above.

[reynaldborer]

Thanks for the update @andypiper!

[PruebasMolonas]

Yes, thanks Andy.

[camonmon]

Hi @andypiper any news about this. It seems to go better, but still many tweets missing. Maybe something to do with the #SB50? Thanks

[andypiper]

There is no correlation here with #SB50.

As I understand it, the primary issue here was caused by a temporary problem with a batch of Tweets being accidentally labelled as spam and excluded from the index, as I mentioned above. We fixed that issue and corrected that batch so that they were indexed correctly.

I’m not aware of any other search quality issue at the moment, so I’d need more detail of what you’re seeing. Remember that the public search API is not a complete index of all accounts and Tweets, is optimised for recency, and only covers a limited timeframe.

[impetusashwin]

Hi, I am using the Search API for finding most recent tweets. I am not receiving the “max_id” for getting the next results after few requests (sometimes after 17 requests and sometimes after 45 requests). I am using app-only authentication and the rate limit is also not reached (450 req / 15min window).

Can someone tell if i am missing something in the API request. I am trying to retrieve most recent tweets for past 7 days using max_id. Thanks!

[andypiper]

FYI to complete the loop here, I’ve just seen another small change committed that should hopefully have closed out any lingering issues related to this fault.

[PruebasMolonas]

Hi Andy

This is giving problems again. Other topics have been opened in the forum. The SEARCH API is giving problems, returning less tweets.

[andypiper]

The other topic you commented on was from a few weeks ago. If there is an issue today, please can you provide some additional specifics (terms, types of content you’re searching for, etc) in order for me to check in with the team? Thanks.

[PruebasMolonas]

Hi Andy, thanks!

For example the hashtag #HUBDAY. I tried all different tools: hashtracking, Tweet Binder, tweetreach and so and all of them return only 50 to 60 tweets from the last hours when the hashtag has lot more.

[andypiper]

I had confirmation from the team that there were a few issues with that term that should be settling out.

[PruebasMolonas]

Thanks Andy. We will stay tuned. Thanks @andypiper

[PruebasMolonas]

Hi @andypiper any news about this? We see that the API is returning less results in many hashtags. Would you need more info about those hashtags? Thanks