Let’s take these two questions separately.
The rate limits are on a per-endpoint, per token basis. So you get x calls to endpoint A as well as x calls to endpoint B (and x calls to endpoint C, D etc) within each window, using the same token. You can prove this to yourself by tracing your API calls and inspecting the X-Rate-Limit HTTP headers on the responses, which will count down on a per endpoint basis. You don’t need to sum up all of the calls across all endpoints inside the window.
In the case of the followers/list endpoint, the rate limits are relatively low, so you would (probably) have to cache the results in stages and perform the query over a longer time period in order to build out all of the followers list across all of the pages of results. I’m not aware of any alternative method to fetch that data.