I am a Software Engineering Master’s student @ Faculdade de Engenharia da Universidade do Porto, currently working on my Master’s thesis “Feature selection for automatic hate speech detection in text”. In other words, I am testing new features for hate speech detection in Twitter comments.
Most of the approaches target text classification itself, but my goal is to investigate how (Twitter) user profiling can improve the classification of tweets. For that matter, one of the features that is crucial for my research is to model the users’ social graph aka social network analysis.
The drawback in generating a user social network is that the API rate limits are REALLY low for the information required. My algorithm is quite simple, it only considers 1 layer of depth, i.e. I check which users a certain user follows and is followed by, and for those I check whether they follow themselves or not. So, for each user I make a few requests:
- List of followers
- List of friends
- For each follower I check if he follows/is followed by each friend by accesing the ‘show_friendship’ parameter.
The API rate limits to list the friends and followers are 15 per 15 minutes and for the ‘show_friendships’ it is 180 requests per 15 min.
Practically speaking, for an user with 67 followers and 13 friends, it takes more than 1 hour (!!) to generate his social graph. Considering my sample contains 1k users, it would take 1000 hours to generate all the required social graphs. And that considering each user has a maximum of 67 + 13 connections, which is quite low.
Finally, is there a way for my API rate limit to be increased? I am desperate enough to even pay for a temporary raise in my request limit for both friends/followers list and ‘show_friendship’ paramters. Kindly please.