#1

When using the full-archive search (and potentially even more endpoints) the x-rate-limit-limit attribute of the header shows a value of 3000. Simultaneously the x-rate-limit-remaining attribute is decrementing from that value. After exactly 300 requests the endpoint reports a Too-Many-Requests error. Internally, there seems to be still the old rate-limit of 300 requests in 15 minutes however the header reports the wrong limit number. Strangely enough after receiving this error it’s own error request header reports the correct x-rate-limit-remaining of 0 and x-rate-limit-limit of 300.

2 Likes
#2

As it turns out even after changing the code to wait until the next time reset when reaching x-rate-limit-remaining == 2700 the error occurs again in the next iteration at 2701. It seems like the remaining limit is decreasing from 300 with each iteration of the crawl and wait routine.

2 Likes
#3

Seems to be fixed since last weekend.

2 Likes