#1

Hi, I’m quite new to API stuff. I need to scrape more than 10K tweets. Can anyone tell me how to deal with the “next_token” function? My code is as follows:

bearer_token=' old one'
search_url = "https://api.twitter.com/2/tweets/search/all"

query_params = {'query':'(5G lang:en -is:retweet -is:reply) OR #5G',
                'max_results':'500',
                'expansions': 'geo.place_id,author_id',
                'tweet.fields': 'text,author_id,created_at,public_metrics,geo,lang',
                'user.fields': 'id,name,username,created_at,location,public_metrics',
                'place.fields': 'country_code,country,geo,id,place_type,name'
               }

After the finishing the above process, I go regenerate bear_token from develpers website and copy paste to the first line and run the code again ?

bearer_token='new'

The postman seems pointing that I can put “next_token” into the “query_params” section. But I don’t know how.

#2

bearer_token is for authentication, it won’t change. You do not need to regenerate this once you have it.

next_token is used for pagination, and is included as a url parameter (same as max_results) eachrequest will give you the next_token you should use for the next request. Pagination | Docs | Twitter Developer Platform

I recommend starting with twarc, where this is all done for you:

1 Like