4 options for web scraping - twitter


#1

hi everybody,

As i understood there is 4 option for web scraping :

  1. pay a sum of money and get access to most of the twitts
  2. use the twwuter api for developers and get information about accounts you own
    3.use the open public api which includes 2 possibilities :slight_smile:
    a.get 1% of all public twitts
    b.get specific twitts that match a search word
  3. pay a not-so-large sum of money and get access to specific features that aren’t uncluded in the public twitter api

please help me figure out what are pricing for each option and whats are the conditions/limitations for each of them

regrdless to your helful answer i would like to know to what adress should i reffer this question - email or link would be really helful as well.


#2

Hi @lev6521 - actually “web scraping” usually refers to the process of programmatically reading information from web sites, and that’s against our Developer Policy unfortunately, as you should be using the API.

The commercial Gnip APIs from Twitter provide access to all (not “most”) public Tweets (i.e. anything that is not otherwise from a protected account, potentially going back to the first Tweet in 2006). You’re free to contact Gnip for more information. Typically these are enterprise level APIs, with full data fidelity and replay capabilities.

Here I think you’re confused about the difference between the public REST and Streaming APIs. In the case of the REST API, you can get information about accounts (followers, profile images, etc) as well as their most recent Tweets, or use a search which covers 7 days of history. In the case of the Streaming API, you have a 1% limit on the firehose for any filter you set. In all of these cases, access is free.

I don’t know what you mean by this - what features are you looking for specifically? A few features may be available to Twitter Official Partners e.g. whitelisted access to the Ads API, but otherwise I’m not sure what you mean - and I don’t think this would help you with “web scraping”.

I guess the most important question here is, what would you like to be able to do with the Twitter platform? It sounds like you want access to some Tweet data, is that correct? If you can be specific, then we can help you to understand the options in more detail if needed.


#3

(also, I’m moving this to our Developer Teatime category, as it is not a technical question that is specific to the Streaming API)


#4

please help me figure Which options are in section one and four. That is if I want to download more than one percent, what price and what percentage I get in return.

If you do not address to whom I can turn.


#5

As I said above, question 4 is not an option as there is no such set of products (and you haven’t clarified which “specific features that are not included in the public API” you meant).

I already posted the link to the page where you can contact Gnip. That’s what you should do if you would like to access more than a 1% realtime stream. You will have access to up to 100% of the firehose, based on your filter terms.


#6

I asked the above link on Thursday. No one answered.