According to GET followers/ids:
At this time, results are ordered with the most recent following first — however, this ordering is subject to unannounced change and eventual consistency issues.
While the ordering is not guaranteed by the API, it has been ordered this way since the launch of this endpoint so it’s relatively safe to rely on. You can start with cursor=-1 and pageinate through the 5000 most recent follower ids until they majority of them start overlapping with your existing list of followers.
https://api.twitter.com/1.1/followers/ids.json?screen_name=harry_styles&cursor=-1