Before I dig deep into how to collect my data, I thought it’s better I ask if it’s possible with current rate limits.
I aim to capture the social connections (following, follower) for a selected number of tweeters in a community (let’s say ~100,000 users from the same city) on a daily basis. The goal is have snapshots of the social graph on daily basis. I tried GET users/search with q=Boston. But that really is not location based search. On top of that, only the first 1000 user ids in Boston are available. How do you recommend I get this static list of users who live in the same city? Once I have that, I can use GET friends/ids and GET followers/ids to construct the social graph. But I have to repeat this on a daily basis and I am sure I would hit the rate limit. Any suggestions? I don’t think I can get this information from the streaming API.
I also would like to collect the tweets and retweets for these selected ~100,000 users on a daily basis. The goal is to know who tweeted on what day, and whether they retweeted from one of their friends.
I know I can collect tweets from the streaming API, but it’s a random sample, and I can’t enforce it to stream tweets from a static list of users. Can I?