KEY ERROR on yalm file for searchtweets

[STEMxicanEd]

I am trying to set up searchtweets with premium sandbox

I created a yalm file that looks like this

search_tweets_api:
    account_type: premium
    endpoint: https://api.twitter.com/1.1/tweets/search/fullarchive/<>.json
    consumer_key: 	<>
    consumer_secret: <>

and a config file that looks like this

[search_rules]
from_date = 2006-03-30
to_date = 2018-06-23
pt_rule = #realtimechem

[search_params]
results_per_call = 100
max_results = 500

[output_params]
save_file = True
filename_prefix = realtimechem
results_per_file = 10000000

and I have tried running my query like this:

 python tools/search_tweets.py --config-file acs.config --credential-file-key twitter_keys.yaml

However when I run I get:

ERROR:searchtweets.credentials:cannot read file ~/.twitter_keys.yaml
ERROR:searchtweets.credentials:Account type is not specified and cannot be inferred.
Please check your credential file, arguments, or environment variables
for issues. The account type must be ‘premium’ or ‘enterprise’.

Traceback (most recent call last):
File “tools/search_tweets.py”, line 196, in
main()
File “tools/search_tweets.py”, line 159, in main
env_overwrite=args_dict[“env_overwrite”])
File “/anaconda3/lib/python3.6/site-packages/searchtweets/credentials.py”, line 186, in load_credentials
parsed_vars = _parse_credentials(merged_vars, account_type=account_type)
File “/anaconda3/lib/python3.6/site-packages/searchtweets/credentials.py”, line 82, in _parse_credentials
raise KeyError
KeyError

I am new to python and I am truly lost

[andypiper]

I think you’ve got a mistake in the way you’re calling the helper script.

Try this:

python tools/search_tweets.py --config-file acs.config --credential-file twitter_keys.yaml --credential-file-key search_tweets_api

[STEMxicanEd]

Thanks Andy this started running and got this error

Traceback (most recent call last):
File “tools/search_tweets.py”, line 196, in
main()
File “tools/search_tweets.py”, line 159, in main
env_overwrite=args_dict[“env_overwrite”])
File “/anaconda3/lib/python3.6/site-packages/searchtweets/credentials.py”, line 178, in load_credentials
yaml_vars = _load_yaml_credentials(filename=filename, yaml_key=yaml_key)
File “/anaconda3/lib/python3.6/site-packages/searchtweets/credentials.py”, line 34, in _load_yaml_credentials

Is making me think that the problem is my yaml file and that i created it incorrectly although not sure how

[binary_aaron]

Hi @STEMxicanEd! I have a few questions/thoughts for you:

1 - what version of searchtweets do you have?
2 - Do you mind sharing where you created your yaml file? It seems like the program cannot find the file.
3 - calling the command-line application as search_tweets.py --.... is the correct way to use it in recent versions of the package.

Thanks!

[STEMxicanEd]

Hi @binary_aaron

1. I had installed from pip but I installed via github as the readme suggested and everything went well
2. both my keys and yaml file are in ~search-tweets-python-master/ (I actually realized I created them wrong so created them with configparser and from the yaml package in python)
3. i ran it like this

python tools/search_tweets.py --config-file acs.cfg --credential-file twitter_keys.yaml --credential-file-key search_tweets_api

and didn’t complained about cfg or yaml but got

File “tools/search_tweets.py”, line 10, in
from searchtweets import (ResultStream,
ImportError: No module named searchtweets

I am so confused

[STEMxicanEd]

Ok wait I used python3 tools/search_tweets.py --config-file acs.cfg --credential-file twitter_keys.yaml --credential-file-key search_tweets_api

and then it started running and the error seems to be my yaml file

Traceback (most recent call last):
File “tools/search_tweets.py”, line 196, in
main()
File “tools/search_tweets.py”, line 159, in main
env_overwrite=args_dict[“env_overwrite”])
File “/Users/pvincentruz/Box Sync/CnEn/twitter_social_networks/project_management/tweet_scrapping/search-tweets-python/searchtweets/credentials.py”, line 178, in load_credentials
yaml_vars = _load_yaml_credentials(filename=filename, yaml_key=yaml_key)
File “/Users/pvincentruz/Box Sync/CnEn/twitter_social_networks/project_management/tweet_scrapping/search-tweets-python/searchtweets/credentials.py”, line 34, in _load_yaml_credentials
search_creds = yaml.load(f)[yaml_key]
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/yaml/init.py”, line 75, in load
return loader.get_single_data()
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/yaml/constructor.py”, line 37, in get_single_data
return self.construct_document(node)
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/yaml/constructor.py”, line 41, in construct_document
data = self.construct_object(node)
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/yaml/constructor.py”, line 86, in construct_object
data = constructor(self, node)
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/yaml/constructor.py”, line 414, in construct_undefined
node.start_mark)
yaml.constructor.ConstructorError: could not determine a constructor for the tag ‘tag:yaml.org,2002:python/object/apply:ruamel_yaml.comments.CommentedMap’
in “twitter_keys.yaml”, line 1, column 1

so I went back to how to create it

import yaml
twitter_keys = """
search_tweets_api:
    account_type: premium
    endpoint: https://api.twitter.com/1.1/tweets/search/fullarchive/<>.json
    consumer_key: AIJ....
    consumer_secret: Rm......
"""
with open('twitter_keys.yml', 'w') as yaml_file:
    yaml.dump(twitter_keys, yaml_file, default_flow_style=False)

And created a file named:

twitter_keys.yml

so I called:

python3 tools/search_tweets.py --config-file acs.cfg --credential-file twitter_keys.yml --credential-file-key search_tweets_api

Traceback (most recent call last):
File “tools/search_tweets.py”, line 196, in
main()
File “tools/search_tweets.py”, line 159, in main
env_overwrite=args_dict[“env_overwrite”])
File “/Users/pvincentruz/Box Sync/CnEn/twitter_social_networks/project_management/tweet_scrapping/search-tweets-python/searchtweets/credentials.py”, line 178, in load_credentials
yaml_vars = _load_yaml_credentials(filename=filename, yaml_key=yaml_key)
File “/Users/pvincentruz/Box Sync/CnEn/twitter_social_networks/project_management/tweet_scrapping/search-tweets-python/searchtweets/credentials.py”, line 34, in _load_yaml_credentials
search_creds = yaml.load(f)[yaml_key]
TypeError: string indices must be integers

Thanks for the patience

[STEMxicanEd]

Hey guys @binary_aaron my husband finally had the time to sit down and walk thourgh the debugging he solved the problem and it was on my yalm file you can close this issue thanks thanks so much

[StophersRAWR]

Sorry answered my own question… haha gotta be careful with extra key inputs…

[LeBraat]