Downloading a network of tweets

[DublinPhox]

Hi all,

I am fairly new to twitter API’s and just wanted to get some feedback on my planned approach to download a network of tweets before I go ahead and try implement it. Additionally, some of you may have better/alternative ideas which would be good to hear.

I have a list of, say 100 people that I am interested in. I would like to download their recent (3200) tweets along with all of their follower’s tweets. After researching the streaming and search API I have concluded they are not feasible to accomplish this. So I was thinking this may work…

1. Get my list of accounts that I am interested in - say 100 accounts
2. Create a list of their followers using Get Followers/list
3. Create lists using POST lists/members/create- you can have up to 1000 lists with 500 members in each.
4. Use the lists in (3) to download the tweets using GET lists/statuses. Using the lists feature should help me get around the rate limit of 180/15 mins as each list would count as one request (please correct me if this is not true!).

My 100 accounts are all from the same country so it would be unlikely that there would be more than 5 million (1000*5000) total accounts in the network.

Can anyone comment on this approach - feasibility, better ways of doing it etc?

Thanks in advance.

[andypiper]

help me get around the rate limit of 180/15 mins

The developer policy is super clear on this - do not attempt to circumvent the rate limits.

If you are using the APIs as documented (and from a brief glance you are, and this is fine), then I do not see an issue. However if you are actively seeking to “get around the rate limit” then we really cannot help you here, and would recommend our commercial solutions instead.

[DublinPhox]

No that wasn’t what I meant at all - excuse my bad wording. I completely respect the rate limits. What I should have said was that using lists would make it more efficient to search multiple accounts. Thanks for the feedback anyway.

[andypiper]

Got it, and you’re right, I was just checking your wording! API wise if you’re within the limits and using a creative approach like this I’m not aware of an issue with the idea you’ve outlined.

[DanielCHood]

Your lists strategy would work but not efficiently. Analyzing millions of records consistently with GET requests is a poor plan regardless of rate limits. You should look at streaming data (limits are again going to be an issue so you may want to look at using gnip – if this is a hobby project I’m not sure it’s feasible).

[DublinPhox]

Thanks for the response Daniel. Is it possible to stream tweets from specific users?

[When2plus2is5]

Here you go:

Consuming streaming data