I have a question regarding extracting all the user’s followers
Twitter API provides pretty nice api for extracting info about user’s followers, but of course the amount of the data in the result is limited and a paging mechanism to process all the result is implemented. So far we haven’t found a way to search directly using Twitter API among the followers. Therefore we were forced to implement the search functionality ourselves. In order to be able to make correct searches ,we first had to extract info(name and avatars) for all the user’s fowollers. IT appeared to be a heavy job, because the best method we found so far is by using https://dev.twitter.com/docs/api/1/get/followers/ids to get the user followers ids (~5000 ids per request) and then https://dev.twitter.com/docs/api/1/get/users/lookup for every id to get the follower’s name and the avatar. (100 results per request). We of course cache the data. The cache expires after 15 minutes.
So far so good. The search is working.We have implemented our way to page the cached data and so on. The problem appears when the user has above 2000 followers. In order to collect all the data we have to make at least 21 requests, which significantly delays the initial load of our form and I guess by using this approach with more users, accessing the form, we can soon reach the allowed request rate limit.
So the question is: What is the best way (fast and with less requests) to implement a search among the user’s followers. I guess in order to guarantee correct search result we should still be able to get all user followers data before the user is able to make the search ,so partially loaded followers is not an option.
Thanks in advance.