#1

Hey all, I’m making calls to my full archive endpoint, but I’m fairly certain I’m only seeing tweets from the past 30 days. I’ve been trying with users who I know have tweets that contain certain strings, and after doing some testing, I’m only seeing the tweets that have been recent, not their entire history. I’m not seeing any errors from the endpoint, just a 200 with either no tweets, or the most recent tweets.

When I try to paste my created tweet query into Twitter’s own advanced search, I am seeing the ‘Uh-Oh! Something went wrong on our end’ message. In addition, I can see this in the error_log in the network tab:
Error: Unhandled API error from RICH_TIMELINE: InvalidRequestUrl 47"

All my query is, is a long list of words, with a from operator. So it will look like:

(apple OR orange OR banana) (from:your_account), but with about 90 words in that list. I made sure that it didn’t exceed the 1024 character limit.

Any idea why its not returning the entire history of tweets here? I’ve already used 24 of my paid requests trying to fix this, I really don’t want to continue testing this if I’m going to use up more requests… Thanks in advance!

#2

Thanks so much for reaching out here. Do you have a code sample you can share with us here?

#3

Sure, just for building and the sending the request itself?

#4

Yes, thanks in advance.

#5

Hey Jessica,

Sorry for the delay, had to get back on my personal computer. So here is the url I’m calling:

public static String FULL_ARCHIVE_URL = "https://api.twitter.com/1.1/tweets/search/fullarchive/dev.json";

And here is the code that builds my requests:

public MyQueryRequest buildQuery(String screenname) throws MyServiceException{
    if(StringUtils.isEmpty(screenname)) {
        //TODO: log and throw an error
    }


    MyQueryRequest req = new MyQueryRequest();
    String q = req.getQuery();
    q = q.concat("(");

    List<Word> words = wordsRepository.findAll();
    Iterator iterator = words.iterator();
    Word nextWord = (Word) iterator.next();
    while(iterator.hasNext()) {
        q = q.concat(nextWord.getText()).concat(" OR ");
        nextWord = (Word) iterator.next();
    }
    //for the last word, don't add 'OR'
    //also add the 'from'
    q = q.concat(nextWord.getText()).concat(") ");

    //append fund operator
    q = q.concat(buildFromOperation(screenname));

    if(q.length() > 1024) {
        throw new MyServiceException("Query is too long, user's screenname should not be this long..");
    }
    req.setQuery(q);

    return req;
}

I just tested this locally, and I attached the query that this code produced. For purposes of testing this again, I modified the first two words I’m searching for to ‘honk’, and ‘commuting’, and I searched my own Twitter history. I used these (both sadly about driving) because I used tweeted the word ‘honk’ a few days ago, and tweeted about ‘commuting’ in 2016.

Honk: https://twitter.com/nicky_robby/status/1172614123760365569
‘Commuting’: https://twitter.com/nicky_robby/status/781830976561438720

After calling the endpoint, I get a 200 successful response, and can see only the ‘honk’ instance returned.

Here is the full value of the query I’m sending in MyQueryRequest:

(honk OR commuting OR Apricot OR Avocado OR Banana OR Bilberry OR Blackberry OR Blackcurrant OR Blueberry OR Boysenberry OR Currant OR Cherry OR Cloudberry OR Coconut OR Cranberry OR Cucumber OR Damson OR Date OR Dragonfruit OR Durian OR Elderberry OR Feijoa OR Fig OR Goji OR Gooseberry OR Grape OR Raisin OR Grapefruit OR Guava OR Honeyberry OR Huckleberry OR Jabuticaba OR Jackfruit OR Jostaberry OR Jujube OR Kiwifruit OR Kumquat OR Lemon OR Lime OR Loganberry OR Loquat OR Longan OR Lychee OR Mango OR Mangosteen OR Marionberry OR Melon OR Cantaloupe OR Honeydew OR Watermelon OR Mulberry OR Nectarine OR Nance OR Orange OR Clementine OR Mandarine OR Tangerine OR Papaya OR Passionfruit OR Peach OR Pear OR Persimmon OR Plantain OR Plum OR Prune OR Pineapple OR Pineberry OR Plumcot OR Pomegranate OR Pomelo OR Purple) (from:nicky_robby)

Here is the body I recieved

{"results":[{"created_at":"Fri Sep 13 20:52:47 +0000 2019","id":1172614123760365569,"id_str":"1172614123760365569","text":"Firing off a nice justified honk of the horn is one of life\u2019s purest pleasures","source":"\u003ca href=\"http:\/\/twitter.com\/download\/iphone\" rel=\"nofollow\"\u003eTwitter for iPhone\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":15362686,"id_str":"15362686","name":"Nick","screen_name":"nicky_robby","location":"philadelphia","url":null,"description":null,"translator_type":"none","derived":{"locations":[{"country":"United States","country_code":"US","locality":"Philadelphia","region":"Pennsylvania","sub_region":"Philadelphia County","full_name":"Philadelphia, Pennsylvania, United States","geo":{"coordinates":[-75.16379,39.95234],"type":"point"}}]},"protected":false,"verified":false,"followers_count":714,"friends_count":834,"listed_count":12,"favourites_count":10935,"statuses_count":6379,"created_at":"Wed Jul 09 05:39:47 +0000 2008","utc_offset":null,"time_zone":null,"geo_enabled":true,"lang":null,"contributors_enabled":false,"is_translator":false,"profile_background_color":"C0DEED","profile_background_image_url":"http:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png","profile_background_image_url_https":"https:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png","profile_background_tile":true,"profile_link_color":"0084B4","profile_sidebar_border_color":"FFFFFF","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/995804535649177600\/rOmal_WS_normal.jpg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/995804535649177600\/rOmal_WS_normal.jpg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/15362686\/1535718829","default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"is_quote_status":false,"quote_count":0,"reply_count":1,"retweet_count":0,"favorite_count":4,"entities":{"hashtags":[],"urls":[],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"filter_level":"low","lang":"en","matching_rules":[{"tag":null}]}],"requestParameters":{"maxResults":100,"fromDate":"201908180000","toDate":"201909172324"}}

Any help on this or see something I’m doing wrong? Sorry for the lengthy response and thanks in advance for the help

#6

Twitter advanced web search is more limited than Premium, so unfortunately it doesn’t make a good way to test the query in this case. Since fullarchive calls are expensive, i usually fall back on using 30day endpoint to test.

It sounds like you’re not getting the full results as a result of pagination Premium search APIs | Docs | Twitter Developer Platform Premium API will paginate on 100 tweets, or 500 if it’s a paid premium subscription, or 30 days of tweets, whatever comes first.

By default, fullarchive returns tweets from the last 30 days, to get older ones you’d have to specify fromDate and toDate but remember that these are request parameters, not query parameters Premium search APIs | Docs | Twitter Developer Platform so your search query will not change, but you will have to supply the time range, then you’ll see a next token in your results you can use to paginate - currently your results do not contain a next token because there is only 1 result in the (default) specified time range.

#7

Thanks for the reply, that makes sense.

So in my case, if I’m using a paid premium subscription, and if I were to include a large time range as request parameters, will it search only the first 500 tweets in that range for those keywords? Then return 0 tweets found, and a next token for me to continue searching tweets in batches of 500?

Or will it search the entire time range of tweets for those keywords, then only return a next token if more than 500 tweets are found?

#8

In your case, specify the full time range, and it will search the entire specified range, return the first page (500 tweets, or the first 30 days worth of tweets) of results with a next token which you can use to retrieve the next set of results.

So if your range is 10 months, and if there was less than 500 tweets a month in your query, at a minimum you will use 10 requests.

Using a paid premium tier you can get estimated counts too - Premium search APIs | Docs | Twitter Developer Platform this might help you plan your queries or estimate how many requests you could use.

1 Like
#9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.