Geocoded search queries returning subset of data

mastermaq · 2013-03-27T20:39:38.000Z

For at least the last twelve hours now, the Search API has been returning a tiny fraction of the data it used to for geocoded queries! Has something broken or is there a change that needs to be made to the queries?

johnbarratt · 2013-03-27T21:11:00.000Z

We are noticing the same thing here, with a 10x drop in tweets coming through since just after 20:10UTC on the 27th. Seems that tweets (ironically) aren’t being geocoded for identification in the search results. Only those tweets with explicit geotag (Place or lat/lon from GPS), or a explicit lat/lon in the user profile are coming through with a geo search.

So previously if someone had ‘Melbourne, Australia’ in their profile, they would come up in a geocoded search over Melbourne, but now they don’t, only if they have attached a geotag to tweets, or put an explicit lat lon in their profile (typically due to older clients who did it pre-geotag days).

You can see this in the public search results as well, eg:

https://twitter.com/search/realtime?q=near%3Amelbourne&src=typd

Most tweets have the little location icon, and those that don’t, you will find a lat/lon in the user profile.

drmcameron · 2013-03-28T03:08:53.000Z

Yes, our search results have flat-lined at about the same time. It’s hurting our application badly.
We tested with both V1.1 and V1 client calls and they’re both exhibiting same low-volume behavior.

matevzg · 2013-03-28T15:53:50.000Z

We are seeing a major drop on the number of results for any query that has a geo filter applied. Any confirmation on this search team?

fokxxy · 2013-03-28T17:23:48.000Z

We have had an incredible drop in geocode search results as well. Started around the same time as you guys. No idea why but we are getting a fraction of what we used to get in results.

episod · 2013-03-28T18:00:30.000Z

We recently changed providers of reverse geocoding data – there are some expected differences in coverage and accuracy that we will continue working to minimize, but a overall decrease in these specific kind of results is expected.

Please note that these differences should only apply to tweets that are fuzzily matched to a location based on the user’s unstructured string input to their profile location field – Tweets that are actually explicitly marked with geo data are unaffected.

mastermaq · 2013-03-28T18:06:18.000Z

Thanks for the response. We’re not talking about a minor decrease here though! Results for one of my queries went from an average of around 55,000 tweets per day to less than 1500. That’s a significant impact. To me it’s broken!

episod · 2013-03-28T18:09:52.000Z

Can you share a sample of the recurring queries that you’re servicing where you’ve noticed such a dramatic drop? Thanks!

mastermaq · 2013-03-28T18:17:59.000Z

Sure:

/1.1/search/tweets.json?geocode=53.540941%2C-113.493698%2C50mi&count=100&q=%2A&result_type=recent&include_entities=true

episod · 2013-03-28T20:58:20.000Z

Thanks, very helpful. I’ll update this thread when I have more info to share.

johnbarratt · 2013-03-28T21:00:32.000Z

Interesting, so there are actually no results that have been geocoded in the result set, so if you have changed providers, then they aren’t providing any functionality for search at the moment.

This has resulted in at least a 10x drop in results. Only a relatively small portion of people have either a lat/lon in their profile, or attached to their tweet. The above search I provided for the web search interface is mirroring the problem :

https://twitter.com/search/realtime?q=near%3Amelbourne&src=typd

Thanks!

matevzg · 2013-03-29T07:03:33.000Z

The following query…

(":)" OR “:))” OR “:-)” OR “: )” OR “:D” OR “=)” OR “;-)” OR “;)” OR “;))” OR “:(” OR “:-(” OR “: (” OR “;-(” OR “;(” OR “;((”) AND (geocode:“50.86594,10.30086,400km”)

…was returning approx. 25.000 results / day. Since March 27th it is returning 1.000 results / day. Other geocode based query are exhibiting the same behavior making them unusable.

StiaanMaree · 2013-03-29T11:52:00.000Z

Hi Taylor,

I’m also noticing a drop to less than 10% like everyone else, if it helps, here is another example query:

https://api.twitter.com/1.1/search/tweets.json?geocode=-31.203405%2C26.938477%2C1350km&count=100&result_type=recent&q=i’m

episod · 2013-03-29T14:48:26.000Z

Thanks, our search team has identified a bug that’s arose as part of the transition. It may take some time to resolve. I’ll update this thread when I have another update to share. Thanks everyone for the examples!

matevzg · 2013-03-29T16:14:11.000Z

Taylor thank you for your prompt investigation. Could you please give an example of a time frame this will be solved in, so we could have something credible for our customers/users?

episod · 2013-03-29T22:07:26.000Z

It looks like it won’t get resolved for about a week – hopefully quicker.

johnbarratt · 2013-03-30T00:20:48.000Z

Given it appears it may still take quite some time, is it possible for you to rollback the code to the known working geocoding, until this is properly resolved with the new setup? Particularly given this is seriously affecting so many services? Thanks!

TVBookClubbers · 2013-03-30T07:32:40.000Z

Am I right in assuming this issue explains why the ‘near this place’ filter on twitter advanced search is now only filtering about 10% of normal content? Is this problem effecting all users?

matevzg · 2013-03-30T23:09:14.000Z

Yes, as it seems it should be affecting all users.

TVBookClubbers · 2013-03-31T08:01:43.000Z

Many thanks for confirming that. I’m surprised there isn’t more internet chatter about this bug. There must be many developers/users pulling their hair out with frustration. At least the twitter people are ‘on the case’. Hope a solution is found sooner than a week!.