Search API returning (very) sparse geocode results

[johnbarratt]

From about 2014-11-20 23:00 UTC (about 4hrs ago) we noticed that search requests with a geocode parameter are now typically returning around 10 (0-20) results when 100 were requested, and where there is well over 100 available.

Previously they would return close to the full 100 (with just the odd deleted ones removed I imagine).

This means now that if you search back in time now by paginating, you hit a zero result page rather quickly, even though there clearly are more available back ‘past’ the empty result set, and still within the week.

A very simple example that reproduces this problem is this request which searches for mentions of ‘melbourne’ within 1000km of Melbourne. Just now it only returned 11 results, but clearly there are many, many thousands of these over the last week :

/1.1/search/tweets.json?q=melbourne&geocode=-37.8143,144.963,1000km&count=100 

Here is a similar query for only 10 tweets, but it only returns 1 tweet, with the results below.

/1.1/search/tweets.json?q=melbourne&geocode=-37.8143,144.963,1000km&count=10

{"statuses":[{"metadata":{"iso_language_code":"en","result_type":"recent"},"created_at":"Fri Nov 21 02:59:01 +0000 2014","id":535628421330894848,"id_str":"535628421330894848","text":"Turn it up Melbs. @ Melbourne Convention and Exhibition Centre http:\/\/t.co\/OkslwOtYei","source":"\u003ca href=\"http:\/\/instagram.com\" rel=\"nofollow\"\u003eInstagram\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":212357078,"id_str":"212357078","name":"Rob Frost","screen_name":"Rob_618","location":"Adelaide, Australia","profile_location":null,"description":"You are a man. Act appropriately.","url":null,"entities":{"description":{"urls":[]}},"protected":false,"followers_count":216,"friends_count":151,"listed_count":6,"created_at":"Fri Nov 05 21:20:53 +0000 2010","favourites_count":1,"utc_offset":37800,"time_zone":"Adelaide","geo_enabled":true,"verified":false,"statuses_count":3603,"lang":"en","contributors_enabled":false,"is_translator":false,"is_translation_enabled":false,"profile_background_color":"131516","profile_background_image_url":"http:\/\/abs.twimg.com\/images\/themes\/theme14\/bg.gif","profile_background_image_url_https":"https:\/\/abs.twimg.com\/images\/themes\/theme14\/bg.gif","profile_background_tile":true,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/462496323376144384\/329FQ-iF_normal.jpeg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/462496323376144384\/329FQ-iF_normal.jpeg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/212357078\/1399103716","profile_link_color":"009999","profile_sidebar_border_color":"EEEEEE","profile_sidebar_fill_color":"EFEFEF","profile_text_color":"333333","profile_use_background_image":true,"default_profile":false,"default_profile_image":false,"following":false,"follow_request_sent":false,"notifications":false},"geo":{"type":"Point","coordinates":[-37.82490005,144.95366416]},"coordinates":{"type":"Point","coordinates":[144.95366416,-37.82490005]},"place":{"id":"01864a8a64df9dc4","url":"https:\/\/api.twitter.com\/1.1\/geo\/id\/01864a8a64df9dc4.json","place_type":"city","name":"Melbourne","full_name":"Melbourne, Victoria","country_code":"AU","country":"Australia","contained_within":[],"bounding_box":{"type":"Polygon","coordinates":[[[144.593741856,-38.433859306],[145.512528832,-38.433859306],[145.512528832,-37.5112737225],[144.593741856,-37.5112737225]]]},"attributes":{}},"contributors":null,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[],"symbols":[],"user_mentions":[],"urls":[{"url":"http:\/\/t.co\/OkslwOtYei","expanded_url":"http:\/\/instagram.com\/p\/vpVeFPEuUF\/","display_url":"instagram.com\/p\/vpVeFPEuUF\/","indices":[63,85]}]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"lang":"en"}],"search_metadata":{"completed_in":0.022,"max_id":535628421330894848,"max_id_str":"535628421330894848","query":"melbourne","refresh_url":"?since_id=535628421330894848&q=melbourne&geocode=-37.8143%2C144.963%2C1000km&include_entities=1","count":10,"since_id":0,"since_id_str":"0"}}

Please let me know if you need any more information to help resolve.

Thanks,

JB

[robert_lowe]

Having the same issue here. Only returning tweets that are specifically geotagged.
Results are not falling back to profile location as per previously.

Please Help!

[johnbarratt]

One further data point that may help track down and fix this issue. Running the query above without a search term, gets even less results (3) even though the result set is a superset of the query with a term.

eg. this broad query only returns 3 results :

/1.1/search/tweets.json?geocode=-37.8143,144.963,1000km&count=100

Whereas this more specific one returns 11 :

/1.1/search/tweets.json?geocode=-37.8143,144.963,1000km&count=100&q=melbourne

Thanks,

JB

[emilioastarita]

I can confirm this having the same issue.

[PhilipLJohnson]

We are also experiencing this issue.

[isaach]

Thanks for the reports, everyone. We’re looking into some issues with geo-related searches; stay tuned.

[robert_lowe]

Has there been any progress made on this?

[isaach]

Yes. But I expect it’ll be at least a few days before we get a fix out. Thanks for whatever patience you can muster in the meantime.

[boibucom]

Same issue here, its around 1 week now with geocode function not working. could it be related somehow to the new Twitter search capabilities announced 1 week ago ?

[Yunxi1987]

The problem still exist. When will this be fixed?

[andypiper]

As @isaach mentioned just 2 days ago, we are aware of the issue and working to resolve it, but it may be a few days. We apologise for the disruption to your applications, and appreciate your patience. We’re hopeful that we can get a fix deployed soon, but some things take time

[isaach]

Hi all, an update on this issue: there are in fact two separate aspects at work

1. Low search coverage of explicitly-geotagged tweets; and
2. Lack of fallback to profile location for geo searches

Yesterday we deployed a fix for (1) and you should see improvement already in the API. We’re working on a fix for (2) as well; I don’t yet have a timeline but will keep you posted.

[lampos]

Have you managed to develop any timeline for issue (2)?

[andypiper]

No news to share on this at present, but we are aware and continue to work on it.

[judotens]

this issue still exist

[andypiper]

As we said, we have resolved part of the issue, but the fallback to profile location for geo searches has not yet been finalised.

This is part of a significant set of changes on the search backend, so it may take some time for the second part of the issue to be resolved. Unfortunately we need to set an expectation of weeks rather than days before a full solution is likely to be in place.

We are very sorry for the inconvenience this is causing. This is obviously a very frustrating situation for you all in our developer community, but there’s only a limited rate at which we can get this particular issue finalised as there are more parts in motion than may be apparent.

Thank you so much for your patience and understanding!

[RobinMeisel]

Thanks for the work you are doing to rectify the issue, however in the interim our Twitter Ads campaigns for targeted audiences is suffering as we are unable to geolocate individuals to include in the campaigns. Is there an alternate source of twitter data to obtain geo-located searches?

[azrael]

Thanks to @isaach for the head’s up on this thread (started my own thread a couple of weeks ago). It’s been about three weeks. Thanks for working hard on this guys And would love it if it were back up by Christmas

[markzohar]

Are you certain that issue #1 has been fixed? We’re still seeing very low search coverage for explicitly geo-tagged tweets where we used to receive a significantly larger data set before. Any further updates would be most appreciated. Thanks!

[lampos]

+1