Twitter Search API retrieving very less tweets for mentions compared to what I see in website



I am trying to retrieve all tweets with particular mention. When I search on twitter website there are more than 2000 tweets but when I use python REST API, I am able to retreive only 35 tweets. Moreover sometime back the same code used to retreive all the tweets. Can’t figure out what is the problem. Please find code below:

def get_mentions_of_startup(mention):
    filename = get_mentions_file_name(mention)
    print filename
    maxTweets = 10000000 # Some arbitrary large number
    tweetsPerQry = 100  # this is the max the API permits

    # If results from a specific ID onwards are reqd, set since_id to that ID.
    # else default to no lower limit, go as far back as API allows
    sinceId = None

    # If results only below a specific ID are, set max_id to that ID.
    # else default to no upper limit, start from the most recent tweet matching the search query.
    max_id = -1L

    tweetCount = 0
    api = twitter
    print("Downloading max {0} tweets".format(maxTweets))
    with open(filename, 'w') as f:
        while tweetCount < maxTweets:
                print max_id
                if (max_id <= 0):
                    if (not sinceId):
                        print "Calling user_timeline 1"
                        new_tweets =, count=tweetsPerQry)
                        print "Calling user_timeline 2"
                        print sinceId
                        new_tweets =, count=tweetsPerQry,
                    if (not sinceId):
                        print "Calling user_timeline 3. Max id="
                        print max_id
                        new_tweets =, count=tweetsPerQry,
                        print "Calling user_timeline 4. Max id="
                        print max_id
                        print "since_id="
                        print sinceId
                        new_tweets =, count=tweetsPerQry,
                                            max_id=str(max_id - 1),
                if not new_tweets:
                    print("No more tweets found")
                for tweet in new_tweets:
                   f.write(jsonpickle.encode(tweet._json, unpicklable=False) + '\n')
                tweetCount += len(new_tweets)
                print("Downloaded {0} tweets".format(tweetCount))
                max_id = new_tweets[-1].id
            except tweepy.TweepError as e:
                    # Just exit if any error
                    print("some error : " + str(e))
    print ("Downloaded {0} tweets, Saved to {1}".format(tweetCount, filename))



If you check the documentation, you’ll see that the Search API will only return up to 7 days of results, and is not a complete index. The search on the website covers the whole Tweet index.



Thanks for quick response.
Is there any trick to collect more tweets i.e. at least tweets dating back to 6 months?
Is it not possible to retrieve all tweets with mentions even using max_id and since_id?



Unfortunately that’s not possible in the public API. If you want a more comprehensive search API then the Gnip data products provide full archive and 30 day search options, but these are typically commercial enterprise options.