Extract more than 200 rows

#1

Hi
I’ve just started to look at extracting data from Twitter using SSIS. The 3rd party Twitter tool allows me to extract data using the Twitter REST API.

As an initial test I want to just download the tweets on my timeline using Statuses/User Timeline. It works beautifully but it only brings down 200 rows at a time. I have more tweets than that on my timeline.

From what I have read I think this is to do with the API having to read through pages of my timeline to retrieve tweets.

Any advice on what I would need to do to pull down all of my tweets?

At some point I will want to pull data down for a certain account, hashtag etc. and I assume I will come across a similar obstacle?

Grateful for any help.

Many thanks
Nav

#2

To get more data than is provided by a single request you need to use the cursor (next_cursor, next_cursor_str) fields and send that value in the next request. It’s the way Twitter paginates their content.

When you get to the point where you want to pull data for other accounts, I recommend looking at https://dev.twitter.com/rest/reference/get/statuses/user_timeline. This will allow you to send other user_id or screen_name fields and pull their timeline. To search by hashtag and other fields you will need to the search api which is located here: https://dev.twitter.com/rest/public/search

#3

Hi
Thank you for your reply, very helpful. I will look up how to use the cursor fields and hopefully work it out.

Cheers
Nav