Search api results return a small amount of tweets

[DinosLamprop]

Hello community,

I gather tweets every day for the past month for my thesis using only the trending topics of each day. The last couple of weeks i noticed that some of the trending topics do not return the expected volume of tweets, as return from the trends query of the api. I retried the next day with no results.

Any suggestions?

Thank you in advance.

P.S. i use tweepy library.

[andypiper]

It’s very hard to say without some example code and search terms, and a sense of what your expectation might be?

[DinosLamprop]

My expectations are around the number of tweets that is shown for the trend at the twitter app and the trend call from the api. I use the same code for multiple terms and i get good results. Because the same trend shows up every week ( as it is a tv show) i use the search term every week with normal results.

If you insist in showing code i can share it.

[AdamGDunn]

Given the timing of when this happened, I’m assuming the reason why is related to this change at Twitter: https://developer.twitter.com/en/docs/tweets/search/overview/standard

Note the keyword “sampling” in the description. My suggestion is to ask your supervisor to pay for a Premium account and do it quickly so that you don’t risk data loss. And good luck with your thesis.

[DinosLamprop]

Thank you for your response. Is it normal for the sample to be that small? The twitter app on my phone was showing over 1000 tweets for the topic and i could only get 30-50.

[AdamGDunn]

Unfortunately I am not sure. I have also been looking around for information about what changes were made to Standard Search when Premium was introduced and I don’t see a definitive answer anywhere. It’s not clear if we have lost the ability to access all tweets matching a search query, and if we have, what proportions of tweets we can now access. It’s also not clear whether repeated searches will return a different sample, or if it will return from within a pre-defined sample like in the streaming API.

[DinosLamprop]

I cannot tell either. From my experience some topics return a number of tweets close to the expected one after searching again, but for some other topics that was not the case.

[andypiper]

No changes have been made to standard search or any of the other standard APIs as a result of the premium launch.

[AdamGDunn]

Can you please clarify what the following sentence means?
"This means that some Tweets and users may be missing from search results."
https://developer.twitter.com/en/docs/tweets/search/overview/standard

Specifically, I’m interested in quantifying the word “some” and understanding whether the ones missing are a random selection, or whether it depends on the vagaries of the search terms and the frequencies with which they are used. Is it possible to know in advance which types of searches are going to be more affected by missing Tweets?

Thanks.

[andypiper]

Again, I want to reassure that nothing has changed on this endpoint and it has remained similar in behaviour since the Twitter Search API came online ~7 years ago.

The language is deliberately vague, as the algorithm and indexing has never been fully defined externally. To answer the specific question, some users and terms, specifically those created very recently or heavily flagged as potentially abusive or spammy, may be excluded from the limited 7 day index for standard search. Equally, at times of load some Tweets or users may be excluded from this index. It is not possible to describe every single condition under which Tweets may not be included.

The behaviour of the standard search endpoint has categorically not changed as a result of the premium API; the new endpoint provides net new functionality built on the enterprise platform.

[AdamGDunn]

Thanks! That is good news.