Hello,

I have an academic research account, but when I try to retrieve conversations even of recent tweets (within 7 days) like so

twurl -j “/2/tweets/search/recent?query=conversation_id:1398361598062854144&tweet.fields=in_reply_to_user_id,author_id,created_at,conversation_id&max_results=100”

It does not work and I keep getting this message.

{
  "title": "Unauthorized",
  "type": "about:blank",
  "status": 401,
  "detail": "Unauthorized"
}

OR if I try this:

def get_data(url):
headers = {‘Authorization’: "Bearer " + “AAAAAAAAAAAAAAAAAAAAAxxx” }
response = requests.get(url, headers=headers)
response_data = response.json()
return response_data

get_data(/2/tweets/search/all?query=“TwitterDev”;max_results=10’)

I get his message:

client_id': '2xxxxxx',
 'detail': 'When authenticating requests to the Twitter API v2 endpoints, you must use keys and tokens from a Twitter developer App that is attached to a Project. You can create a project via the developer portal.',
 'registration_url': 'developer.twitter.com/en/docs/projects/overview',
 'title': 'Client Forbidden',
 'required_enrollment': 'Standard Basic',
 'reason': 'client-not-enrolled',
 'type': '//api.twitter.com/2/problems/client-forbidden'

The app is within a project so I am not sure what else could be causing this.

Thank you for your help

Academic access endpoint supports Application Only authentication, (bearer token only). So in twurl, you can use the bearer token instead of the access tokens like this

twurl --bearer "/2/tweets/search/recent?query=conversation_id:1398361598062854144&tweet.fields=in_reply_to_user_id,author_id,created_at,conversation_id&max_results=100"

I can’t immediately tell what’s wrong with the other code but i recommend using twarc GitHub - DocNow/twarc: A command line tool (and Python library) for archiving Twitter JSON instead for academic access: Help for beginners - #2 by IgorBrigadir it can be used as both a command line tool or as a python library Help with full archive search with Tweepy or easy Python alternative - #4 by IgorBrigadir

The app should also be in an “Academic” project, not a “Standard” one - in case you applied and got a developer account for academic use case, it does not automatically grant you access to the academic endpoint: Twitter Developer Access - twarc

1 Like

Thank you for the quick response.

I tried using the twurl endpoint you suggested. However, I am getting the error message below
No available bearer token found for consumer_key:xxx

That consumer key is an old key. I have regenerated the tokens at least twice since then. I am not sure why even if I close and restart the command prompt, it keeps referring to that consumer key even when I try using

twurl authorize --bearer --consumer-key --consumer-secret

Do you have an idea why that may be?

Thank you

It might be reading old configs or something? is there a file called .twurlrc (usually hidden file)

You could try to delete or rename this file and try to run

twurl authorize --bearer --consumer-key Aa..zz --consumer-secret Aa...zz

again,or edit the file if you’re comfortable editing YAML.

1 Like

Thank you so much. It seems like that helped me at least retrieve recent conversations. However, when I try to access historical data using

twurl --bearer "/2/tweets/search/all?query=conversation_id:1224970162459172865&tweet.fields=in_reply_to_user_id,author_id,created_at,conversation_id&max_results=100"

I get this error

{"client_id":"2xx","detail":"When authenticating requests to the Twitter API v2 endpoints, you must use keys and tokens from a Twitter developer App that is attached to a Project. You can create a project via the developer portal.","registration_url":"https://developer.twitter.com/en/docs/projects/overview","title":"Client Forbidden","required_enrollment":"Standard Basic","reason":"client-not-enrolled","type":"https://api.twitter.com/2/problems/client-forbidden"}

Is that the appropriate endpoint to retrieve past conversations?

What does your apps projects dashboard look like? https://developer.twitter.com/en/portal/projects-and-apps should have “Academic Research” not “Standard” and the project app should be the one you’re using in twurl. Also, i think for conversations you have to specify start_date.

If that looks ok, then another thing to check is the default app used by twurl

twurl accounts

should list the apps - and the default should be the app that is attatched to the Academic Access project, not the Standard one.

However, i don’t seem to be able to retrieve conversations for 1224970162459172865 tweet - it might be a temporary error or it might be something else wrong. 1398361598062854144 works.

Hello,

Yes, 1398361598062854144 worked for me as well but 1224970162459172865 is older than 7 days, I tried with other older tweet_ids (using search/all) and it did not work.

The account default account used by twurl is the one that should be associated with the Academic Research account. But I do not have access to the dashboard since this is my colleague’s account.

So I ran test_api.py as suggested in #469
but I got this error

{“title”:“Unauthorized”,“type”:“about:blank”,“status”:401,“detail”:“Unauthorized”}

I am guessing this might mean it is not attached to the Academic Research app but I am waiting on my colleague to confirm.

In the meantime I also tried running

twarc2 conversations TEST.txt TEST.jsonl

but got the error

Unable to parse 400 error as JSON: Bad Request

Then I tried running

import datetime
import itertools
import twarc

from twarc.client2 import Twarc2
from twarc.expansions import flatten

# Your bearer token here
t = Twarc2(bearer_token="TOKEN")

# Start and end times must be in UTC
start_time = datetime.datetime(2006, 3, 21, 0, 0, 0, 0, datetime.timezone.utc)
end_time = datetime.datetime(2021, 5, 27, 0, 0, 0, 0, datetime.timezone.utc)

# search_results is a generator, max_results is max tweets per page, 500 max for full archive search.
search_results = t.search_all(query="conversation_id:1224970162459172865", start_time=start_time, end_time=end_time, max_results=10)

# Get just 1 page of results instead of iterating over everything in search_results:
# To get everything use `for page in search_results:`
for page in itertools.islice(search_results, 1):
    # Do something with the page of results:
    # print(page)
    # or alternatively, "flatten" results returning 1 tweet at a time, with expansions inline:
    for tweet in flatten(page)['data']:
        # Do something with the tweet
        print(tweet)

But I got this error

NameError: name 'requires_app_auth' is not defined

Thanks for troubleshooting - what OS and version of python are you running? and what’s the output of pip list ?

Before running any twarc2 commands, make sure you run

twarc2 configure

When copy pasting the bearer token, sometimes half of it gets left out because it contains a % sign, so make sure you copy all of it. In the command line you may not see the output, but paste it in and enter and it will work.

As for the python snippet: There is a small fix required to that python code because twarc was updated, so this works for me:

import datetime
import itertools
import twarc

from twarc.client2 import Twarc2
from twarc import ensure_flattened

# Your bearer token here
t = Twarc2(bearer_token="AAA...zzz")

# Start and end times must be in UTC
start_time = datetime.datetime(2006, 3, 21, 0, 0, 0, 0, datetime.timezone.utc)
end_time = datetime.datetime(2021, 5, 27, 0, 0, 0, 0, datetime.timezone.utc)

# search_results is a generator, max_results is max tweets per page, 500 max for full archive search.
search_results = t.search_all(query="conversation_id:1224970162459172865", start_time=start_time, end_time=end_time, max_results=10)

# Get just 1 page of results instead of iterating over everything in search_results:
# To get everything use `for page in search_results:`
for page in itertools.islice(search_results, 1):
    # Do something with the page of results:
    # print(page)
    # or alternatively, "flatten" results returning 1 tweet at a time, with expansions inline:
    for tweet in ensure_flattened(page):
        # Do something with the tweet
        print(tweet)

I get 3 tweets from this - as expected https://twitter.com/AFPFactCheck/status/1224970162459172865

1 Like

Thank you so much for the follow-up.

OS: 19041.985
Python 3.8.3
I am using Jupyter Notebook and Anaconda Prompt

pip list
Package                            Version
---------------------------------- -------------------
alabaster                          0.7.12
anaconda-client                    1.7.2
anaconda-navigator                 1.9.12
anaconda-project                   0.8.3
appdirs                            1.4.4
argh                               0.26.2
asn1crypto                         1.3.0
astroid                            2.4.2
astropy                            4.0.1.post1
atomicwrites                       1.4.0
attrs                              19.3.0
Automat                            20.2.0
autopep8                           1.5.3
Babel                              2.8.0
backcall                           0.2.0
backports.functools-lru-cache      1.6.1
backports.shutil-get-terminal-size 1.0.0
backports.tempfile                 1.0
backports.weakref                  1.0.post1
bcrypt                             3.1.7
beautifulsoup4                     4.9.1
bitarray                           1.4.0
bkcharts                           0.2
bleach                             3.1.5
bokeh                              2.1.1
boto                               2.49.0
Bottleneck                         1.3.2
brotlipy                           0.7.0
certifi                            2020.6.20
cffi                               1.14.0
chardet                            3.0.4
click                              7.1.2
click-config-file                  0.6.0
click-plugins                      1.1.1
cloudpickle                        1.5.0
clyent                             1.2.2
colorama                           0.4.3
comtypes                           1.1.7
conda                              4.8.5
conda-build                        3.18.11
conda-package-handling             1.7.0
conda-verify                       3.4.2
configobj                          5.0.6
constantly                         15.1.0
contextlib2                        0.6.0.post1
cryptography                       2.9.2
cssselect                          1.1.0
cycler                             0.10.0
Cython                             0.29.14
cytoolz                            0.10.1
dask                               2.20.0
decorator                          4.4.2
defusedxml                         0.6.0
diff-match-patch                   20200713
distlib                            0.3.1
distributed                        2.20.0
docutils                           0.16
ekphrasis                          0.5.1
entrypoints                        0.3
et-xmlfile                         1.0.1
fastcache                          1.1.0
filelock                           3.0.12
flake8                             3.8.3
Flask                              1.1.2
fsspec                             0.7.4
ftfy                               5.8
future                             0.18.2
gensim                             3.8.3
gevent                             20.6.2
glob2                              0.7
gmpy2                              2.0.8
greenlet                           0.4.16
h5py                               2.10.0
HeapDict                           1.0.1
html5lib                           1.1
hyperlink                          20.0.1
idna                               2.10
imageio                            2.9.0
imagesize                          1.2.0
importlib-metadata                 1.6.1
incremental                        17.5.0
intervaltree                       3.0.2
ipykernel                          5.3.2
ipython                            7.16.1
ipython-genutils                   0.2.0
ipywidgets                         7.5.1
isort                              4.3.21
itemadapter                        0.1.1
itsdangerous                       1.1.0
jdcal                              1.4.1
jedi                               0.17.1
Jinja2                             2.11.2
joblib                             0.16.0
json5                              0.9.5
jsonschema                         3.2.0
jupyter                            1.0.0
jupyter-client                     6.1.6
jupyter-console                    6.1.0
jupyter-core                       4.6.3
jupyterlab                         2.1.5
jupyterlab-server                  1.2.0
keyring                            21.2.1
kiwisolver                         1.2.0
krippendorff                       0.4.0
lazy-object-proxy                  1.4.3
libarchive-c                       2.9
llvmlite                           0.32.1
locket                             0.2.0
lxml                               4.5.2
MarkupSafe                         1.1.1
matplotlib                         3.2.2
mccabe                             0.6.1
menuinst                           1.4.16
mistune                            0.8.4
mkl-fft                            1.1.0
mkl-random                         1.1.1
mkl-service                        2.3.0
mock                               4.0.2
more-itertools                     8.4.0
mpmath                             1.1.0
msgpack                            1.0.0
multipledispatch                   0.6.0
navigator-updater                  0.2.1
nbconvert                          5.6.1
nbformat                           5.0.7
networkx                           2.4
nltk                               3.5
nose                               1.3.7
notebook                           6.0.3
numba                              0.49.1
numexpr                            2.7.1
numpy                              1.18.5
numpydoc                           1.1.0
oauthlib                           3.1.0
olefile                            0.46
openpyxl                           3.0.4
packaging                          20.4
pandas                             1.0.5
pandocfilters                      1.4.2
paramiko                           2.7.1
parsel                             1.6.0
parso                              0.7.0
partd                              1.1.0
path                               13.1.0
pathlib2                           2.3.5
pathtools                          0.1.2
patsy                              0.5.1
pep8                               1.7.1
pexpect                            4.8.0
pickleshare                        0.7.5
Pillow                             7.2.0
pip                                20.1.1
pkginfo                            1.5.0.1
pluggy                             0.13.1
ply                                3.11
prometheus-client                  0.8.0
prompt-toolkit                     3.0.5
Protego                            0.1.16
psutil                             5.7.0
py                                 1.9.0
pyasn1                             0.4.8
pyasn1-modules                     0.2.7
pycodestyle                        2.6.0
pycosat                            0.6.3
pycparser                          2.20
pycurl                             7.43.0.5
PyDispatcher                       2.0.5
pydocstyle                         5.0.2
pyflakes                           2.2.0
Pygments                           2.6.1
PyHamcrest                         2.0.2
pylint                             2.5.3
PyNaCl                             1.4.0
pyodbc                             4.0.0-unsupported
pyOpenSSL                          19.1.0
pyparsing                          2.4.7
pyreadline                         2.1
pyrsistent                         0.16.0
PySocks                            1.7.1
pytest                             5.4.3
python-dateutil                    2.8.1
python-jsonrpc-server              0.3.4
python-language-server             0.34.1
python-twitter                     3.5
pytz                               2020.1
PyWavelets                         1.1.1
pywin32                            227
pywin32-ctypes                     0.2.0
pywinpty                           0.5.7
PyYAML                             5.3.1
pyzmq                              19.0.1
QDarkStyle                         2.8.1
QtAwesome                          0.7.2
qtconsole                          4.7.5
QtPy                               1.9.0
queuelib                           1.5.0
rauth                              0.7.3
redis                              3.5.3
regex                              2020.6.8
requests                           2.24.0
requests-oauthlib                  1.3.0
rope                               0.17.0
Rtree                              0.9.4
ruamel-yaml                        0.15.87
scikit-image                       0.16.2
scikit-learn                       0.23.1
scipy                              1.5.0
Scrapy                             2.3.0
seaborn                            0.10.1
Send2Trash                         1.5.0
service-identity                   18.1.0
setuptools                         49.2.0.post20200714
simplegeneric                      0.8.1
singledispatch                     3.4.0.3
sip                                4.19.13
six                                1.15.0
smart-open                         3.0.0
snowballstemmer                    2.0.0
sortedcollections                  1.2.1
sortedcontainers                   2.2.2
soupsieve                          2.0.1
Sphinx                             3.1.2
sphinxcontrib-applehelp            1.0.2
sphinxcontrib-devhelp              1.0.2
sphinxcontrib-htmlhelp             1.0.3
sphinxcontrib-jsmath               1.0.1
sphinxcontrib-qthelp               1.0.3
sphinxcontrib-serializinghtml      1.1.4
sphinxcontrib-websupport           1.2.3
spyder                             4.1.4
spyder-kernels                     1.9.2
SQLAlchemy                         1.3.18
statsmodels                        0.11.1
sympy                              1.6.1
tables                             3.6.1
tblib                              1.6.0
termcolor                          1.1.0
terminado                          0.8.3
testpath                           0.4.4
threadpoolctl                      2.1.0
toml                               0.10.1
toolz                              0.10.0
tornado                            6.0.4
tqdm                               4.47.0
traitlets                          4.3.3
twarc                              2.1.2
tweet-preprocessor                 0.6.0
Twisted                            20.3.0
typing-extensions                  3.7.4.2
ujson                              1.35
unicodecsv                         0.14.1
urllib3                            1.25.9
virtualenv                         20.4.1
w3lib                              1.22.0
watchdog                           0.10.3
wcwidth                            0.2.5
webencodings                       0.5.1
Werkzeug                           1.0.1
wheel                              0.34.2
widgetsnbextension                 3.5.1
win-inet-pton                      1.1.0
win-unicode-console                0.5
wincertstore                       0.2
wordcloud                          1.8.0
wrapt                              1.11.2
xlrd                               1.2.0
XlsxWriter                         1.2.9
xlwings                            0.19.5
xlwt                               1.3.0
xmltodict                          0.12.0
yapf                               0.30.0
zict                               2.0.0
zipp                               3.1.0
zope.event                         4.4
zope.interface                     4.7.1

I had run

twarc2 configure

before but I reran it after verifying the bearer token was correct using

curl --user “ABC:DEF” --data “grant_type=client_credentials” “https://api.twitter.com/oauth2/token

Then I pasted the bearer token to a text file to ensure it was complete.

twarc2 configure
Please enter your Bearer Token (leave blank to skip to API key configuration):
(Optional) Add API keys and secrets for user mode authentication [y or n]? n

Output

Your keys have been written to C:\Users\x\AppData\Roaming\twarc\config


✨ ✨ ✨  Happy twarcing! ✨ ✨ ✨

I ran the updated code you provided me and I now I got

HTTPError: 403 Client Error: Forbidden for url:

Does this just confirm the app is not attached to the Academic Research track?

Yeah i think you covered all the bases on your end. I can’t think of anything else to check.

Thanks for the debug info, this will help later for twarc development. We might try to see if we can put in more helpful error messages.

Thank you so much for your time and patience!