Hey Josh, appreciate that things can be confusing here, especially if you’re new to it / not necessarily a coder and just want to do some analysis.
To clarify a couple of points - the REST API will enable you to query back over a seven day period (search) or up to 3200 Tweets (timeline). Streaming is an “always on” delivery mechanism that delivers up to 1% of the Twitter firehose in realtime, i.e. you’ll get future Tweets, but you can’t look backwards. Using the filter option you’ve specified, you’d get all the Tweets from the user ID you are following as they are posted.
Another thing I’m not sure about is your reference to 500+ columns - I think a Tweet object should be substantially smaller than that in terms of properties, and it hasn’t changed a lot over the past 12 months (the exception being the new extended format Tweets which enable the full 140 characters to be used for text). Saying that, there will be Tweets with different values - if a Tweet has media attached then the extended_entities object will be included.
On the question of how to connect to the streaming API, realistically you need to write a bit of code to sit and listen to it (and yes, that could be your laptop being open and listening / running the code). There are a large number of client libraries available to do this. You might find it straightforward to write something in Python, for example, depending on your experience (tweepy has a simple interface) - or, using Node.js and the twit module, something simple like this:
var Twit = require('twit')
// get keys and tokens from apps.twitter.com
var T = new Twit({
consumer_key: 'xxxx',
consumer_secret: 'xxxx',
access_token: 'xxxx',
access_token_secret: 'xxx',
timeout_ms: 60*1000, // optional HTTP request timeout to apply to all requests.
})
// define the ID of the user we are interested in
var userID = '786491';
// open a stream following events from that user ID
var stream = T.stream('statuses/filter', { follow: ( userID ) });
stream.on('tweet', function (tweet) {
// compare the user ID inside the Tweet object we passed in
// to check it matches
if (tweet.user.id == userID) {
console.log("this was sent by the user we want to track")
// now do something else like save to a file
} else {
console.log(tweet.user.id + " - " + tweet.user.screen_name)
// so we can ignore it
}
});
