How can I scrape twitter data for some years?

[NazaruddinLaOde]

Dear Sir/madam

Morning
How can I scrappe twitter data from January 01, 2020 to December 30, 2022?
Thank you

[LoneWolfgangK]

I believe that you need Academic access. Otherwise, you can only retrieve Tweets from the past week.

Alternatively, if you have Tweet IDs, you can rehydrate them. Maybe I’m mistaken, but I don’t think that there is any time limit on this.

I guess this is probably not the answer you wanted. I wish you the best

[IgorBrigadir]

Without Academic Access, the only other alternative is to use the old v1.1 Premium API, for which you need elevated access. Search API | Twitter API | Docs | Twitter Developer Platform but this is totally different and separate to v2 Academic Access, and the old v1.1 search entirely.

[NazaruddinLaOde]

Thank you for reply. I have academic access

[LoneWolfgangK]

That’s good. You can use twarc. This should get you started:

Help for beginners

I would begin here: Tutorial - twarc or the video tutorial here: twarc Demonstration - YouTube There are more tutorials here: Resources - twarc and another livestream recording here: TwitterDev: twarc2 and Academic Search - PeerTubeFR There’s an older tutorial to get the basics of setting up a command line utility Collect Twitter Data with Twarc! · Learn Twarc! that twarc tutorial doesn’t use the academic access endpoint, but it’s good practice to get familiar with using command line tools to …

Follow these tutorials. Try it out. Let us know if you have a more specefic question.

[NazaruddinLaOde]

Thank you for the suggestion. I am usingRSt udio. Is twarc compatible with RStudio?

[IgorBrigadir]

twarc command line tools themselves should be run inside a terminal / command prompt - not in R, but R should be able to load the generated CSV from twarc without any issues.

[NazaruddinLaOde]

so, to run twarc, I should use python, right?

[LoneWolfgangK]

Yes, you use Python. It’s very straightforward after you set up.

Try it out. If you get stuck, someone will help you

[IgorBrigadir]

@NazaruddinLaOde

to run twarc, I should use python, right?

You need python installed, but you will not need to code any python. Twarc runs as a command line tool - so you type commands into a prompt, not python.

[NazaruddinLaOde]

The python looks like this. How to start?

[IgorBrigadir]

No, this is a python prompt - you type python code in here, which is not what you want. But this is good that Python is definitely installed on your machine.

For twarc, you only need to have python installed, and use a command line to type a command in. On windows, try opening PowerShell for example.

See here for a demo:

twarc Demonstration

(ignore the $ because on windows it will look different, but you can type the exact same commands into Powershell - you may also need to replace twarc2 with twarc2.exe but i don’t have Windows to double check this)

[NazaruddinLaOde]

After typing (twarc), it looks like this:

[IgorBrigadir]

Oh, sorry, no, you don’t need to type (twarc) in either - just the actual twarc2 command, like

twarc2 search

Also you are still typing this into a Python prompt, not a Command Line - it may be worth starting here: Tutorial - twarc