Unreliable hashtag search

[timkindberg]

I have an app which has been successfully searching through the API for posts based on their hashtags for about a couple of years, but which has recently been getting erratic results. For example, up until about Wed/Thu it would return tweets containing #AfrikaEye2018 (a film festival). Since then, it returns nothing. It has been retrieving tweets hashtagged #ClimateChange for months, but recently it does so unreliably. If I do a search from a user account with these hashtags, I largely see what I expect to see, although even that doesn’t produce all the posts that have in fact been made.

Has something changed in the API or terms of service? In case it’s relevant, my query seeks only tweets containing videos. I haven’t yet sought approval for the account from which the API searches take place. As I say, it’s been working for two years or so. The API rate limits have not been reached.

Many thanks for any help.

Tim

[LeBraat]

Hey @timkindberg,

If you are using the standard search/tweets endpoint, then you should note the following statement on that API reference page:

Please note that Twitter’s search service and, by extension, the Search API is not meant to be an exhaustive source of Tweets. Not all Tweets will be indexed or made available via the search interface.

We have not made any changes to this endpoint in a long time, so it is strange to hear about this change in behavior.

What I can recommend to you is to apply for a developer account and start working with the newer premium Search API which will return a complete set of results, unlike the standard search/tweets endpoint.

[timkindberg]

Thanks for your reply. I am using the search/tweets endpoint. I’m aware of the terms of service that you cite. It’s striking, however, that the results have suddenly become very unreliable, whereas I’ve found them to be reliable in practice for retrieving recent tweets for the last two years. E.g. literally nothing from #AfrikaEye2018 is being returned as of a few days ago; and the results for #ClimateChange tend to be a few consecutive tweets, then a larger gap, then a few consecutive tweets, etc.

Has anyone else noticed this behaviour?

In any case, I’ll apply for a developer account

[footieengland]

Just done a test of this myself with the tag #AfrikaEye2018 it returns results from today back to the 6th Nov. Are you posting the recent attribute with the query ?

[timkindberg]

Thanks for looking into this (I’ve been travelling hence I’m only just replying). I’m not aware of a ‘recent’ attribute; I’m querying for the hashtag with a ‘native_video’ filter. Is that what you used? As mentioned, it’s exactly the same query I’ve used successfully for over two years.

[Aurelia]

Hi @timkindberg - This behaviour does indeed seem a little odd. However, I have not been able to reproduce on my side. Could you please provide the exact query that you’re running and give more detail as to the timeframe: when exactly did you stop receiving Tweets for your request and how long did this last for? Has this happened again since you first noticed it?

[footieengland]

#AfrikaEye2018

Hi Tim

Hope you enjoyed the travelling. The recent attribute is documented here
https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets

When I search using filter:native_video #AfrikaEye2018 using the API it just returns a single result using recent and 2 results if change it to mixed.

However if I pass the query on twitter.com using the advanced search it only returns 6 results.

https://twitter.com/search?vertical=default&q=filter%3Anative_video%20%23AfrikaEye2018%20&src=typd&qf=off&lang=en-gb

So it would seem there isn’t a huge pool of tweets containing that hashtag that is filterable by native_video in the first place. As we know the standard API doesn’t return an exhaustive list of tweets anyway.

Regards

[timkindberg]

I’ve tried again with a fresh hashtag #PMQs. My query is:
https://api.twitter.com/1.1/search/tweets.json?q=%23pmqs+exclude%3Anativeretweets+exclude%3Aretweets%2Bfilter%3Anative_video&result_type=recent&count=100&since_id=1065264376142286849&tweet_mode=extended

NB I update since_id after the first call. I’ve retrieved 51 tweets over the last few tens of minutes but the API is now steadfastly ignoring every tweet since, including test tweets I’ve posted from a couple of accounts, ones that show up in search on the Android app.

[timkindberg]

It’s just returned, posted :

Mergers & Acquistions (MergeAndAcq)

The previous tweet returned, 45 mins prior, was:

LeShankEntertainment (leshankent)

I can find the former, not the latter, in the Android app search. There are plenty of videos (including the two I posted) in between.

[timkindberg]

Could someone from Twitter please comment on the new de facto unreliability of this type of search? I’m using only hashtags with a low (one per several minutes/hours) arrival rate, so it’s unlikely to be a resourcing issue.

[andypiper]

Most of the team are currently out for holidays so unfortunately it may take a few days to dig deeper. We appreciate your patience. Do you have access to the premium API free sandbox to try to reproduce the behaviour?

[timkindberg]

Thanks, and looking forward to hearing from you. I’ve just been approved as a Twitter developer so I’ll look at the sandbox now.

[andypiper]

Just one thing I’ll caution is that your sandbox requests per month are significantly lower than on the standard API, so please be careful how frequently you hit it! I’m sorry that I’ve not had the bandwidth to try to reproduce myself in the past couple of days. Thanks for any insight you can provide from your own testing, in advance.

[system]

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.