Twitter Search API retrieving very less tweets for mentions compared to what I see in website

search

#1

I am trying to retrieve all tweets with particular mention. When I search on twitter website there are more than 2000 tweets but when I use python REST API, I am able to retreive only 35 tweets. Moreover sometime back the same code used to retreive all the tweets. Can’t figure out what is the problem. Please find code below:

def get_mentions_of_startup(mention):
    filename = get_mentions_file_name(mention)
    print filename
    
    maxTweets = 10000000 # Some arbitrary large number
    tweetsPerQry = 100  # this is the max the API permits

    # If results from a specific ID onwards are reqd, set since_id to that ID.
    # else default to no lower limit, go as far back as API allows
    sinceId = None

    # If results only below a specific ID are, set max_id to that ID.
    # else default to no upper limit, start from the most recent tweet matching the search query.
    max_id = -1L

    tweetCount = 0
    api = twitter
    print("Downloading max {0} tweets".format(maxTweets))
    with open(filename, 'w') as f:
        while tweetCount < maxTweets:
            try:
                print max_id
                if (max_id <= 0):
                    if (not sinceId):
                        print "Calling user_timeline 1"
                        new_tweets = api.search(q=mention, count=tweetsPerQry)
                    else:
                        print "Calling user_timeline 2"
                        print sinceId
                        new_tweets = api.search(q=mention, count=tweetsPerQry,
                                            since_id=sinceId)
                else:
                    if (not sinceId):
                        print "Calling user_timeline 3. Max id="
                        print max_id
                        new_tweets = api.search(q=mention, count=tweetsPerQry,
                                            max_id=str(max_id-1))
                    else:
                        print "Calling user_timeline 4. Max id="
                        print max_id
                        print "since_id="
                        print sinceId
                        new_tweets = api.search(q=mention, count=tweetsPerQry,
                                            max_id=str(max_id - 1),
                                            since_id=sinceId)
                if not new_tweets:
                    print("No more tweets found")
                    break
                for tweet in new_tweets:
                   f.write(jsonpickle.encode(tweet._json, unpicklable=False) + '\n')
                tweetCount += len(new_tweets)
                print("Downloaded {0} tweets".format(tweetCount))
                max_id = new_tweets[-1].id
            except tweepy.TweepError as e:
                    # Just exit if any error
                    print("some error : " + str(e))
                    break
    print ("Downloaded {0} tweets, Saved to {1}".format(tweetCount, filename))

get_mentions_of_startup("LookUpHQ")

#2

If you check the documentation, you’ll see that the Search API will only return up to 7 days of results, and is not a complete index. The search on the website covers the whole Tweet index.


#3

Hi,

Thanks for quick response.
Is there any trick to collect more tweets i.e. at least tweets dating back to 6 months?
Is it not possible to retrieve all tweets with mentions even using max_id and since_id?

Thanks,
Regards,


#4

Unfortunately that’s not possible in the public API. If you want a more comprehensive search API then the Gnip data products provide full archive and 30 day search options, but these are typically commercial enterprise options.


#5