Do you have a sample list of your queries?
So we are working to extract multiple queries (with two groups of words occurring at the same time; thus double parenthesis)
(\“johnson & johnson's\”)(word1, word2)
In that case, in the text file, that should be:
("johnson johnson s") (word1 word2)
& ampersands and ' apostrophes are stripped from text in search, so they should be removed. Quotes are not escaped when writing queries in a text file, only when writing them in command line. And there should be a space, which is an implicit logical AND between the two parentheses. There are no commas in queries, if you want logical OR, it’s like: (word1 OR "two words" OR word3) in case there’s a 2 word phrase.
Instead of the above python code, you can do the exact same thing in command line:
!twarc2 searches --archive --start-time "2021-06-28" --end-time "2021-11-28" queries_auto.txt vax_results.jsonl
!twarc2 csv vax_results.jsonl vax_output.csv
That’s if you want everything in one file.
To write to separate files using your code, change the line:
with open("vax_results.jsonl", "a+") as f:
to something unique to the query, since you’re enumerating them anyway, like
with open(f"vax_results_query_{count}.jsonl", "a+") as f:
Then you’ll have to change the open("vax_results.jsonl", "r") and open("vax_output.csv", "a") below to the same file: open(f"vax_results_query_{count}.jsonl", "r") and open(f"vax_output_query_{count}.csv", "a") when converting to CSV. And then indent all the CSV code from converter down to be under the main for .. enumerate() loop, to get a bunch of different files like:
vax_results_query_0.jsonl
vax_output_query_0.csv
vax_results_query_1.jsonl
vax_output_query_1.csv
vax_results_query_2.jsonl
vax_output_query_2.csv
