I am trying to retrieve all tweets by Elon Musk in the past using the Twitter Timeline API. I have not added any ‘excludes’ query param, so I am expecting it to return 100 results and a ‘next_token’ for accessing the next 100 results (pagination).
However, I am getting only 22 results in the response.
curl --location --request GET 'https://api.twitter.com/2/users/44196397/tweets?max_results=100' \
--header 'Authorization: Bearer $BEARER_TOKEN'
max_results is more like an upper limit than a target result, because deleted tweets are removed after retrieval.
I don’t know why you’re not getting a next_token to paginate though, that seems like an error
Update 1: I am getting 100 responses (as expected) when using the cURL command.
However, both Postman and the requests library in Python are giving me 22 results
1 Like
That’s even stranger. What’s the full curl command (with your tokens redacted)
Is the curl command and postman requests using the same authentication? Bearer token or consumer keys and Access tokens?
Yes, both are using Bearer token. I generated the cURL command from Postman itself
1 Like
Update 2: In one out of five attempts (random), I am now getting 100 results with the ‘next_token’. (both Postman & cURL)
Whenever, I get 100 results, I also get a ‘next_token’ for pagination. On using the ‘pagination token’, I always get 0 results in my next request.
In four out of five attempts, I get 22 results without the ‘next_token.’ ((both Postman & cURL) )
1 Like
ah ok - if it’s random, it’s likely an eventual consistency issue - you may be hitting a load balancer that’s sending you to an index with an error. Unfortunately, it’s very hard to safeguard and work around these. What worked for me in the past was retrying several minutes later, or filling in gaps using a combination of the v1.1 Timeline API, and then hydrating using the tweets lookup in v2.
