Search by THREE: Location, Historical Time AND Keyword


#1

Hi,

I am trying to download tweets that: (A) contain a set of keywords, (b) were posted within certain distance of a location, and © For a certain time period (a few months in 2010).

However, I get no tweets the moment I add the date field!! I tried various combinations and it is sad that there is no clear documentation available online.

(I am using the Twitter Fullsearch api)

PRODUCT = ‘fullarchive’
LABEL = XX

SEARCH_TERM = ‘fever point_radius:[8.6160669,-10.06117800000004 40km]’
r = api.request(‘tweets/search/%s/:%s’ % (PRODUCT, LABEL),
{‘query’:SEARCH_TERM, ‘maxResults’: 500, ‘toDate’: ‘201002020000’, ‘fromDate’: ‘201001010000’})


#2

With just ‘fever’ and without the location, I get tweets in the given date range. This could imply that my label is correct.

With just the location and the date range, I get no tweets.

With keywords and location and the date range, I get no tweets.

With just the location (and no date range and keywords), I get exactly one tweet from 2018.


#4

Hi @aadi_joshi, I was able to reproduce the situation: I, too, receive 0 Tweets when combining the time-range, keyword ‘fever’ and location. Let me dig into this and I will let you know when I have more information.


#5

Bear in mind that only a very small percentage of Tweets have location data associated (typically only 1-2% have location attached). I’d speculate that this may be why your query doesn’t return Tweets when location is added to your criteria.


#6

There’s some ambiguity in the docs too:

“Full Archive Search” Timeline
https://developer.twitter.com/en/docs/tweets/search/guides/fas-timeline
says 2010-03-06 is when Geo starts matching, but for “Historical Powertrack” https://developer.twitter.com/en/docs/tweets/batch-historical/guides/hpt-timeline.html it’s 2011-09-01

  • Geo : Native geo data prior to 9/1/2011 is not available in Historical Powertrack. As a result, all operators reliant on this geo data will not be supported for jobs with a timeframe prior to this date.

I’m not actually sure which applies to the fullarchive endpoint, but i’m assuming it’s /fas-timeline ?


#7

Yes, the Full-archive Search API and the Historical PowerTrack products have different archives and different 'start dates" for when Tweet geo metadata begins in the archive.

As stated in those docs, the Search API archive has geo metadata beginning around March 2010.