Problem with twitter4j-getFollowersIDs

[tkbalajiprasad]

Hi guys,
I’m working on a project which analyses closure patterns in social networks.
Part of my requirement is to collect followers and following IDs of thousands of users under scrutiny.

I have a problem with rate limit exceeding 350 requests/hour.
With just 4-5 requests my limit is exceeding - ie, when the number of followers I collected exceeds the 350 mark.
ie, if I have 7 members each having 50 followers, then when I collect the follower details of just 7 members, my rate exceeds.(7*50 = 350).

But my requirements involves collecting the follower-following details of close to 16,000 members.
How can I achieve this? Why is each follower of a user treated as a separate request?
I expected collecting the followers of a member means 1 request… Can I get this behavior some how?

I’ve shown part of the code below… It is directly taken from the examples provided with twitter4j.
It runs perfectly fine until the the number of followers collected > 350.
(And if a single user has more than 350 followers, again this fails to run)

<standard oauth authentication code>
TwitterFactory tf = new TwitterFactory(cb.build());
            Twitter twitter = tf.getInstance();
          long cursor = -1;
            IDs ids;
            User user;
            System.out.println("Listing following ids.");
            do {
                ids = twitter.getFollowersIDs(username, cursor);
                for (long id : ids.getIDs()) {
                    user = twitter.showUser(id);
                    if (!userFollowerNames.contains(user.getScreenName())) {                   
                        userFollowerNames.add(user.getScreenName());
                    }
                    //System.out.println(id);
                }
            } while ((cursor = ids.getNextCursor()) != 0);

error-
“Failed to get list members: 400:The request was invalid. An accompanying error message will explain why. This is the status code will be returned during rate limiting (https://dev.twitter.com/pages/rate-limiting).
error - Rate limit exceeded. Clients may not make more than 350 requests per hour.”

[episod]

While I can’t help you with Twitter4J specifically, I see that you’re using twitter.showUser which I assume corresponds to user/show in the REST API. There’s another related method called users/lookup which taken a collection of user IDs or screen names, 100 at a time, and allows you to “hydrate” them to user objects. That approach would be wiser for you. You may want to scale back your requirements to something more manageable for you with the API limits, or add a longer duration to your study to give you enough time to collect the data.

[tkbalajiprasad]

@episod - Thanks a lot for your reply… I see that there is a lookupUsers method which takes a collection of screen names and returns a list of “User” objects… But this user object gives info like number of followers, number of friends etc- But it doesn’t serve my purpose of getting the actual screen names of follwers/friends. I’m I missing some thing here.

[SdiPondy]

@tkbalajiprasad i am also expecting the same result and getting the issue. So Please share the solution if you found it .

[007sankey]

@episod Can I get list of followers who joined today ? Specifically I do not want list of old followers, I would need like as soon as someone joined me, I can fetch his/her details in my app. Do you have any such API or Idea to do this ??

[crawler_tweets]

@tkbalajiprasad,

Did you find out a way to solve this issue?

-Vinay Vishwanatha

[nhattan2012]

You should try Paging!