How to get more search results with the API?

[alfageeek]

I’m putting together a little app to help people find stolen tweets, and I’ve found that while the search interface on the web produces tons of results, the search rest API only seems to give very recent stuff. I’ve tried playing with the result_type and nothing helps.

The results I’m seeing are never more than a week or two old, so if someone stole a tweet from a month ago, it doesn’t appear in the API search results, but it’s plain as day on the web search.

(I’m basically getting 0 results for most searches in the API, and exactly the same query gives a hundred on the web site)

Is there a trick to doing a twitter search via API that will yield more results?

Is there any expectation of when the search API will start to catch up to the web in terms of utility?

[boxnumber03]

There is no “trick” for getting more results, at least according to the search documentation. And, if you need to go back further than about a week, the expectation is that you purchase this service from one of Twitter’s certified partners.

(Curious, what’s a “stolen tweet”?)

[andypiper]

Is there a trick to doing a twitter search via API that will yield more results?

Is there any expectation of when the search API will start to catch up to the web in terms of utility?

The Search API is targeted at recent tweets (around 7 days) rather than a comprehensive index of all historical data. That kind of information is available via Gnip, or other certified data partners. There are no plans for this to change at the current time.

[alfageeek]

(Curious, what’s a “stolen tweet”?)

A “stolen tweet” is when somebody (usually in “funny twitter”) writes something really clever, and then other people copy and paste that tweet without attribution as though they came up with it themselves. A lot of people find this kind of plagiarism really bothersome, and would rather not follow people who do that.

(It’s also a violation of twitter’s terms of service to copy other people’s tweets, but twitter clearly doesn’t care, since a lot of “blue check” verified accounts contain nothing but stolen tweets.)

[alfageeek]

The Search API is targeted at recent tweets (around 7 days) rather than a comprehensive index of all historical data. That kind of information is available via Gnip, or other certified data partners. There are no plans for this to change at the current time.

Oh! Gee, you think you might have put that in the documentation, so I didn’t waste so much time trying to get results that didn’t suck!

I read a couple other threads here on the forum where other “Twitter Staff” said that you would be making a better search (like the twitter.com “search” feature) available via the API. I guess they were wrong?

This is a public service app, not anything I want to monetize, so using a paid service like Gnip wouldn’t work for me.

[andypiper]

Let me clarify a little!

you think you might have put that in the documentation, so I didn’t waste so much time trying to get results

The search API is clearly documented and we have statements in the docs that make it clear that the index is not comprehensive, e.g. via the search API overview page https://dev.twitter.com/rest/public/search (my italics)

The Twitter Search API is part of Twitter’s v1.1 REST API. It allows queries against the indices of recent or popular Tweets and behaves similarily to, but not exactly like the Search feature available in Twitter mobile or web clients, such as Twitter.com search.

Before getting involved, it’s important to know that the Search API is focused on relevance and not completeness. This means that some Tweets and users may be missing from search results.

On the other point

I read a couple other threads here on the forum where other “Twitter Staff” said that you would be making a better search (like the twitter.com “search” feature) available via the API. I guess they were wrong?

I’m not sure which threads you are referring to specifically. The most recent set of questions related to the update to make Twitter.com search historically comprehensive, and we clearly stated that there are no plans to make a change to the search API to make it equivalent at this time.

That kind of information is available via Gnip, or other certified data partners.

Gnip is indeed part of Twitter - there are alternatives such as Datasift if you prefer. We do realise that the fact that the public RESTful search API is not 100% comprehensive is disappointing to some developers, but there are no immediate plans for this to change. In addition, the set of additional API tools offered by data partners to interrogate the complete Tweet index are substantially more capable in terms of date ranges, locations, trends etc. so there is added value there.

[andypiper]