Can I scrape Twitter's HTML for Conversation purposes?

[adriancrepaz]

Hi,

As you’re obviously aware, the related_tweets endpoint was depreciated with the release of your REST V1.1 API.
This has caused many of us headaches when wanting to fetch a conversion of tweets.

I’ve looked at the HTML markup for your mobile site, for a public tweet (conversation), and it would be very easy to parse the HTML to return a list of tweets and/or tweetIds.

Would you forbid me doing this?
If so, could you please make an exception and allow this, seeing as features we’ve previously built are now broken.

I realise we could possibly fetch a conversation of tweets using the in_reply_to_tweet field, but this seems like a lot of calls when I could just get the data I need in one request.

Thanks,
Adrian

[johnfoobar]

I had a similar question for my application. According to Twitter’s terms of service, “scraping the Services without the prior consent of Twitter is expressly prohibited”. How could I get permission for my company’s app to scrape Twitter? The problem would also be solved if the view conversation api was open to the public. View Conversations- API Support There is a twitter api as a service which provides this. It’s called Topsy http://api.topsy.com/doc/resources/content/conversation/. Something like that would be nice, but I’m wondering how they get the conversation themselves. If my company were to hook up to your firehouse would we then have the ability to view conversations? Any help would be appreciated even clues of how to get around this. If you have any ideas yourself. Thanks.

[andypiper]

Web scraping and automation is 100% prohibited under the Developer Agreement and Policy, and Terms of Service, and efforts to extract data from the platform via these methods may result in permanent suspension from the service, blacklisting of relevant IPs, and other penalties.

[andypiper]