Search_tweets.py error


#1

Hi,
When I execute search_tweets.py (copied the contents into test.py) as shown below I receive an error which doesn’t tell me much. Has anyone run into this issue? I am new to this and I appreciate someone can assist.

python test.py --credential-file PATH-OF-CREDENTIALS-FILE --config-file PATH-OF-CONFIG-FILE --filename-prefix test_search

–Result I get --------
Traceback (most recent call last):
File “test.py”, line 207, in
main()
File “test.py”, line 201, in main
for tweet in stream:
File “C:\Python37\lib\site-packages\searchtweets\utils.py”, line 140, in write_result_stream
yield from write_ndjson(_filename, stream)
File “C:\Python37\lib\site-packages\searchtweets\utils.py”, line 95, in write_ndjson
for item in data_iterable:
File “C:\Python37\lib\site-packages\searchtweets\result_stream.py”, line 208, in stream
self.execute_request()
File “C:\Python37\lib\site-packages\searchtweets\result_stream.py”, line 260, in execute_request
rule_payload=self.rule_payload)
File “C:\Python37\lib\site-packages\searchtweets\result_stream.py”, line 101, in retried_func
error_message = resp.json()[“error”][“message”]
KeyError: ‘error’


#2

I don’t have a copy of your code, but this error means that resp.json() is returning an object that does not have a property called “error”. You can try printing resp.json() and see what it actually is returning.


#3

thank you for the suggestion. The response I get when printing resp.json() is below

{‘errors’: [{‘message’: ‘Sorry, that page does not exist’, ‘code’: 34}]}

This is the endpoint I am using in YAML
https://api.twitter.com/1.1/search/search30days1.json


#4

Looks like you are missing the ‘s’ in ‘errors’. But that’s not all. The value of ‘errors’ is an array. Try this:

error_message = resp.json()[“errors”][0][“message”]


#5

After adding [0] I received the error below

requests.exceptions.HTTPError

This is the code I downloaded from https://github.com/twitterdev/search-tweets-python by the way. I am executing search_tweets.py which uses results_stream.py (the file I keep editing to get the correct error message)

Thanks again for the help


#6

So, after printing more responses and a few other parameters I found out the actual error is “ERROR:searchtweets.result_stream:HTTP Error code: 404: Sorry, that page does not exist”

Double-checking the endpoint, I noticed I was missing ‘30day’ in the link. After providing the correct endpoint I am able to get the result.

Cheers


#7

I am also trying to get data from premium search API and currently using its sandbox version. But I failed to execute the code given at https://github.com/twitterdev/search-tweets-python. Can u please provide me the code that you are using to crawl the dataset. I will replace the keys and other things with mine.

Thank you


#8

What went wrong? What did you try?


#9

What is the error? Is the code failing or running but throwing an error?

open result_stream.py in PythonInstallationPath\Lib\site-packages\searchtweets and print the arguments and json response to see what the error is. something like below (around row 78)

    while True:
        try:
            resp = func(*args, **kwargs)
            print(args, kwargs) ##add these print statements
            print (resp.json())

#10

Thanks for your reply and sorry for late response,

When I run search_tweets.py, it is giving following error

return {**dict1, **dict2}
^
SyntaxError: invalid syntax

Another important point is that when we install searchtweets using pip, search_tweets.py is not installed. It is only available with the github zip file


#11

Actually, I am unable to understand their use description given in readme file. When I am running search_tweets.py file, it is giving the following error

File “/usr/local/lib/python2.7/dist-packages/searchtweets/utils.py”, line 80
return {**dict1, **dict2}
^
SyntaxError: invalid syntax

What I have to do. I am stuck now


#12

You have python version 2.7, you need Python 3.3 or later to run search_tweets.py

If you have python 3.3 or higher and pip install search_tweets worked without errors then you may have multiple versions of python installed (not uncommon) so you’ll need to make sure you create a python 3 environment for this.