#1

Having issues with the searchtweets-v2 package in python and using the Academic Research product track for a full_archive. Bellow is my current code:

from searchtweets import ResultStream, gen_rule_payload, load_credentials

search_args = load_credentials(filename="~/.twitter_keys",
                    yaml_key="search_tweets_api",
                    env_overwrite=False)

x = "(from:amazon OR from:AmazonKindle OR from:amazonmusic OR from:alexa99 OR from:amazonluna OR " \
        "from:PrimeVideo OR from:amazonpay OR from:SportsOnPrime OR from:amazonfiretv OR from:amazonprimenow OR " \
        "from:AmazonPhotos OR from:amazonappstore OR from:primegaming OR from:AmazonFresh OR from:Xbox OR " \
        "from:XboxGamePass OR from:XboxGamePassPC) -is:reply -is:retweet"

query = gen_rule_payload(x, results_per_call=100, start_time='2008-05-01')

rs = ResultStream(request_parameters=query,
                  max_results=10000000,
                  max_pages=100000,
                  **search_args)
print(rs)

tweets = list(rs.stream())

import pandas as pd
import pandas

pd.DataFrame(pandas.json_normalize(tweets)).to_csv('raw_amazon_xbox_data.csv', encoding='utf-8')`Preformatted text`

I get the following error:
ImportError: cannot import name ‘gen_rule_payload’ from ‘searchtweets’

In the thead linked bellow it is recommended to not have searchtweets and searchtweets-v2 downloaded when recieving this error. I’ve included a picture of my package list to show I only have searchtweets-v2 downloaded.

image
image730×538 8.69 KB

#2

I would try to do the same thing in twarc command line, it does more tweet specific processing when converting to CSV too,

Make sure it has the bearer token:

twarc2 configure

Search and save as json:

twarc2 search --archive --start-time "2008-05-01" "(from:amazon OR from:AmazonKindle ...) -is:retweet" results.jsonl

Install the CSV plugin with

pip install --upgrade twarc-csv

And you’ll be able to convert in the command line:

twarc2 csv results.json results.csv