#1

Search Tweets uses a parameter called next_token

User Timeline uses a parameter called pagination_token

Why?

#2

That happened because at the time we didn’t have an agreed way to name parameters in a consistent way. The team is aware of this, and we will fix this in a future version of the Twitter API.

Here’s a look behind the scenes if you’re curious why this happened.

As you can expect, many teams work on many things separately but roughly at the same time. In our case, for simplicity, let’s say there was a team who worked on Search, and another who worked on Timelines. Those two developments were happening separately but in parallel. While we had (and still have) a way to have all endpoints return the same payload, we were still building a the process to have all endpoints accept same or similar parameters.

Search was launched first, and it uses the next_token notation for a few reasons:

  • It’s more self descriptive, because it reflects what it’s returned in the meta payload
  • You can only paginate forward
  • It’s similar to the next parameter accepted by older versions of Search.

Shortly after that, we started working on a number of other endpoints, and we realized that developers may need to paginate back to a previous results page, and so next_token would be confusing. And so this is how we came up naming that parameter pagination_token: its name does not prescribe a cursoring direction, and “pagination” clearly marks it as a cursor, not as an access token.

More importantly, one big thing happened: we created a centralized set of design and naming principles that every team can consult and use to name things consistently. And so now we have a centralized place to name things in the same way, which is why all the other endpoints in the platform are accepting the same parameters and work in the same way.

Why we didn’t fix Search after making these decisions? According to our versioning principles, replacing next_token with pagination_token requires to trigger a breaking change. So while we cannot simply hot swap one parameter name with another, we will be able to do so then it’s time to release the next version of the Twitter API.

I hope this gives you some background not just on how we ship things, but also on how we learn and grow from feedback and past mistakes.

5 Likes
#3

Really interesting thanks. Perhaps add pagination_token as an alias of next_token in the Search API and then update the docs accordingly?

2 Likes
#4

I like that! I’ll pass that suggestion on to the team!

2 Likes
#5

I was coming here to create a new topic on this issue as well. Luckily, I searched and found this one before I submitted my own.

Though the individual documentation pages for the various endpoints may be specific as to the name of the pagination token parameter, I feel that an update to the Pagination documentation would be beneficial.

Is the search endpoint the only one that uses the parameter next_token in v2 API?

Thanks!