Twitter academic API not getting historical tweets

[FHaouari]

Hello,
Could you please advise about the following. I tried to collect the conversation for historical tweets but I am not able to get them. I am able to get only the ones posted in the last 7 days. Could you please advise why since we are supposed to get the historical tweets using the academic research API.

Many thanks in advance.

[IgorBrigadir]

If you have been approved for the new academic access and have a project on the dashboard with the academic limits on https://developer.twitter.com/en/portal/dashboard then it should work, but the academic search API is a different endpoint than the usual v2 search - what code / library are you using and how are you making the calls (what parameters etc)

[FHaouari]

Thanks for your reply. For getting the full archive it worked perfectly fine using this code. However what I am looking for is getting the replies to the old tweets which was introduced in V2. I can get only the replies for the tweets posted in the last 7 days. I am using the below command in my python code. Is collecting the replies part of the academic reasearch API?

   command ='curl --request GET \
   --url "https://api.twitter.com/2/tweets?ids='+str(tweetID)+'&tweet.fields=author_id,conversation_id,created_at,in_reply_to_user_id,referenced_tweets,public_metrics&expansions=author_id,in_reply_to_user_id,referenced_tweets.id&user.fields=name,username,created_at,description,verified,public_metrics,pinned_tweet_id" \
   --header "Authorization: Bearer '+bearer_token+'"'

[IgorBrigadir]

Running a curl command in python is not the best way of doing this.

To use Twitter-API-v2-sample-code/full-archive-search.py at main · twitterdev/Twitter-API-v2-sample-code · GitHub to et conversations, you need to specify conversation_id in the query like:

query_params = {'query': 'conversation_id:1334987486343299072','tweet.fields': 'author_id'}

Search Tweets - How to build a query

[FHaouari]

Thanks a lot for your help. I used the query you shared but I am not getting an output. I appreciate your advise

200
{
    "meta": {
        "result_count": 0
    }
}

Here is the code

import requests
import os
import json

consumer_key=''
consumer_secret_key=''
bearer_token=''
search_url = "https://api.twitter.com/2/tweets/search/all"

query_params = {'query': 'conversation_id:1334987486343299072','tweet.fields': 'author_id'}

def create_headers(bearer_token):
    headers = {"Authorization": "Bearer {}".format(bearer_token)}
    return headers


def connect_to_endpoint(url, headers, params):
    response = requests.request("GET", search_url, headers=headers, params=params)
    print(response.status_code)
    if response.status_code != 200:
        raise Exception(response.status_code, response.text)
    return response.json()


def main():
    headers = create_headers(bearer_token)
    json_response = connect_to_endpoint(search_url, headers, query_params)
    print(json.dumps(json_response, indent=4, sort_keys=True))

if __name__ == "__main__":
    main()

[FHaouari]

@FHaouari

def connect_to_endpoint(url, headers, params):
    response = requests.request("GET", search_url, headers=headers, params=params)
    print(response.status_code)
    if response.status_code != 200:
        raise Exception(response.status_code, response.text)
    return response.json()


def main():
    headers = create_headers(bearer_token)
    json_response = connect_to_endpoint(search_url, headers, query_params)
    print(json.dumps(json_response, indent=4, sort_keys=True))

if __name__ == "__main__":
    main()

If I have old tweet Ids how to get their replies? as per my knowledge the conversation_id is a new param.

[IgorBrigadir]

I don’t have academic access yet to check myself but it works for more recent tweets in /search/recent endpoint. What happens when you specify a start_time and end_time or since_id?

[FHaouari]

This query below works fine
query_params = {'query': 'covid','max_results':'10','start_time':'2020-12-31T15:00:00Z','end_time':'2021-01-05T15:00:00Z','tweet.fields': 'author_id'}

[FHaouari]

I tried using the recent search. I can get the replies of tweets posted withing 7 days but not for old tweets.

[IgorBrigadir]

That’s odd, i would have expected the conversation_id: operator to work - maybe someone from twitter can investigate?

[codevid1]

hello @FHaouari ,

I have the same problem compiling the comments of historical tweets. did you get any solution? thanks for your attention.

[IgorBrigadir]

@codevid1 what exact code / commands are you running? and what query?

This is an old thread, and retrieving conversations can now be done with twarc: twarc2 (en) - twarc

See here to get set up:

Help for beginners

I would begin here, to get the basics of setting up a command line utility up: Collect Twitter Data with Twarc! · Learn Twarc! That twarc tutorial doesn’t use the academic access endpoint, but it’s good practice to get familiar with using tools to get data. There are more tutorials here: Tutorials - twarc and a livestream recording here: TwitterDev: twarc2 and Academic Search - PeerTubeFR It will be better to use twarc2 for data crawling: to install twarc2, pip3 install --upgrade twarc then…

[Blanca_C_Fi]

Hi @IgorBrigadir , I know this is an old post, but I believe retrieving historical conversations is still not possible. I have an academic account, but I still can not retrieve old conversations. I have used both CURL, python queries and TWARC, but none of them return anything.

[IgorBrigadir]

What was the exact twarc command you ran?