#1

Hi all!

Just found an unexpected behavior in the official twitter-api-typescript-sdk - request.ts when using full search endpoint. As it is stated, the full search (Academic Research Access) has an extra 1 tpsec rate limit. Digging into the sdk code, the fetchWithRetries() check if res.status == 429 to detect throttling, and after that, fires a timeout in order to wait to the x-rate-limt-reset refresh.

  if (res.status === 429 && max_retries > 0) {
    const rateLimitReset = Number(res.headers.get("x-rate-limit-reset"));
    const timeTillReset = rateLimitReset * 1000 - Date.now();
    await new Promise((resolve) => setTimeout(resolve, timeTillReset));
    return fetchWithRetries(url, init, max_retries - 1);
  }

The problem is that pagination returns a 429 error if the query is sent in less time than 1 sec, but, the throttling is fired to the next reset time window (about 10/15 min) each time a page is requested in the iterator code. Taking into account that the sdk does not distinguishes between api calls, this leaves the library pretty unusable in a fullSearch-academic scenario.

I workarounded it by changing the timeout (timeTillReset) to an exponential backoff scheme, in order to not affect other non-rate limited endpoints while maintaining the rate limit protection, and disabling the decreasing of the max_retries parameter (as it is going to be fired each time a page is requested).

Hope it will be addressed in future updates of the library.
Cheers!

1 Like
#2

Thanks for sharing this - would you be willing to open this as an Issue on the GitHub repository? :pray:

#3

Sure!

also sent a PR to the GitHub repository.

2 Likes