"The Account Activity API will eventually replace the User streams"

[dekisu]

NOPENOPENOPE. This will kill third party desktop/mobile clients or at least force them to do polling (with all its the disadvantages and missing features).

Am I reading this wrong?

[HeroicSocialApp]

hootsuite in jeopardy here?

[andypiper]

Thanks for the feedback on the roadmap and plan ahead.

That’s definitely not the intent of this change and eventual replacement.

The current streaming APIs (user streams and site streams) have not been very scalable and have a number of limitations - not the least of which is that site streams is not generally available to a majority of developers.

I’ve personally been using the Account Activity API for a while now to process direct messages for some bot work, and the responsiveness at the moment is very nice. We’ll build out additional features in that API for other activity types over time before it is labelled “GA” and we begin to remove the existing streaming options.

I’ll certainly point out that today’s announcements and the roadmap ahead is focused on the core Twitter data platform so none of the things we’ve specifically talked about so far is aimed at client applications.

That said, I can think of some architectural patterns that would enable a webhook-based interaction to continue to work with a client application. It would require work, but that’s one of the reasons why we want to be transparent about where we are going, and to enable a long deprecation / migration period to all developers that depend on the existing API patterns and behaviours.

[andypiper]

As with all of our discussions in these forums, we’re not able to speculate or comment on individual applications for a number of reasons, developer privacy being a fundamental one. I hope the clarification I’ve provided about the Account Activity API here gives additional context, and we’re looking forward to discussing more about the webhooks architecture as more developers start to try it out.

[HeroicSocialApp]

Same here @andypiper and thank you. I’ve already filed my application for access to it but understand the team is probably pretty swamped right now! But great job on everything, we’re very excited about the direction!

[dekisu]

Thanks for the quick reply. Glad to hear that this is not the intent, and I can certainly see the appeal of a webhook based API for certain use cases, but I’m still not sure what this means to me as a desktop userstream client.

Particularly:

[…] and we begin to remove the existing streaming options.

I’ll certainly point out that today’s announcements and the roadmap ahead is focused on the core Twitter data platform so none of the things we’ve specifically talked about so far is aimed at client applications

If it’s not aimed at client applications but the streaming APIs will still be removed, then what is aimed at client applications?

That said, I can think of some architectural patterns that would enable a webhook-based interaction to continue to work with a client application

Theoretically, we could set up our own infrastructure with a web server listening for webhooks of all of users, and provide our own streaming-equivalent server.

But we’re just an open source client application operating on a $0 budget, and we kinda value the privacy of our users, we don’t want to have to add a man-in-the-middle proxy to allow them to have streaming-like functionality.

I also fail to see how this is more efficient - sending outgoing HTTP requests for every single tweet or tweet event received by every user? I just see a ton of extra overhead, in size and in latency.

Of course, if the long term plan is to kill applications that are dangerously close to the core twitter client experience, I can see how that’s not a big concern.

[andypiper]

We understand and hear the concerns you’re communicating here. As I mentioned, the existing public streaming APIs have not proven to be particularly scalable over the long term (particularly site streams) - so we’re faced with a choice about how to move forward.

In this case, we’re choosing to invest in an alternative manner of delivering near realtime events and activities, similar to those available via the current APIs; but, yes, we are changing the model to webhook-based delivery.

This happens to pair particularly well with the new Direct Message platform where (for example) developers may wish to build server-side intelligence like bots, and that’s one reason why these events are the first ones available. As we’ve mentioned in the forum post about this, this pattern also makes things a lot more flexible and reliable for us to develop further, unlike the legacy long-lived socket connection streaming APIs.

If you take a look at the roadmap you’ll see where we’re intending to invest over the coming months around the API, and at this moment there’s nothing that is specific to client applications, apart from more robust search and more advanced direct message functionality (both of which have wide applicability).

[seirl]

We understand that the new API you’re developing is better for some use cases like bots, but this is irrelevant to the issue. This thread is about third party clients support. We are concerned that you deleting the streaming APIs will kill third-party clients.

You said:

That’s definitely not the intent of this change

And then:

at this moment there’s nothing that is specific to client applications

Are you realizing that, if this API is discontinued, the effect it will have is to kill third party clients? If so, we would appreciate that you state that clearly instead of making everyone wait to see if you decide to care in your roadmap.

Are you killing third party clients, yes or no?

[andypiper]

Nothing in today’s announcements or in the roadmap changes any of our previous positions or policies relating to client apps.

I’ve already mentioned that there are alternative models to consider that would enable you to consume the events in a similar way to the way you do today - I also realise that this is not what everyone wants to hear.

There’s a significant road ahead while we build Account Activity API, the new developer experience we’ve discussed in our blog post, and in providing a full deprecation story around the current streaming APIs.

We’ll be interested to hear more about the kinds of things you want to achieve on the platform, and how we can help you to do so within the direction of the roadmap and our plans.

[dekisu]

We’ll be interested to hear more about the kinds of things you want to achieve on the platform, and how we can help you to do so within the direction of the roadmap and our plans.

Sure:

• A standalone twitter api client
• Running in a computer or phone that doesn’t have a fixed IP address or publicly reachable inbound ports
• That doesn’t require end users to get their own API key
• That doesn’t depend on non-twitter server infrastructure or any sort of proxies
• That displays tweets as soon as they are posted
• That is faster than having to poll home_timeline.json every 60 seconds (since that’s all that’s allowed by the rate limits)
• That can, in one way or the other, receive all the events currently available in userstreams, including updates of follows, lists, mutes, blocks, favs, etc

And yes, this means I’d be open to more polling endpoints if they had reasonably generous rate limits, despite how annoying it would be to rewrite the clean stream handling code into something that polls the same thing every time and merges state. But polling every 60 seconds is useless for twitter. And I don’t think you all would be happy with even more polling.

I still have some hope that there will be some way to do this because tweetdeck exists and depends on the userstream. So hopefully you’ll find a solution that works for tweetdeck and can also be used by everyone else with similar requirements.

As an aside: thanks for being upfront about this and having proper plans for the deprecation of these features. I’m not happy about this, but it’s infinitely better than what some other companies have done.

[dekisu]

This post sets a date to kill user streams, Tuesday June 19, 2018.

The account activity API continues to be completely unsuitable. Add to the requirements above:

• That handles more than 35 users

I can live with less events and I guess we’ll have to tolerate things not appearing instantly, but AAAPI can’t work. It doesn’t even have events for follows, which is the main thing twitter clients consume.

So, polling it is, then.

home_timeline.json is still as slow as always, with 15 reqs / 15 mins. Meanwhile, for the same endpoint, tweetdeck gets 225 reqs / 15 mins and polls every four seconds.

I also note that the new DM endpoint, direct_messages/events/list, says “15/user and 750/app”, which aside from the same slow polling rate, it sounds like it’s limited to 50 users per app, meaning that if i switch to this API my users will instantly get rate limited.

Can you do something to improve this situation?

Thanks.

[whattheshot]

From what I understand, the follow event will still be available, it’s only the unfollow event that disappears.

[andypiper]

At this time, the only suitable alternative will be a polling mechanism (which I personally understand will not provide the same level of responsiveness as the legacy - but not sustainable - streaming option.

The comment on Account Activity API not having follow events is not accurate, so far as I know - where did you get the information that follow events are not available in the new format?

Your comment on TweetDeck is not quite relevant here, as TweetDeck is web-based, and also runs on Twitter’s internal infrastructure on the server side. Yes, there are client-side events but this is much more complex than that.

We want to be transparent about what is currently in the pipeline for the API platform, and we listened closely to all of your requests throughout the year. We’ve been evaluating the opportunity to build something that would maintain the features you’re asking for - but at the current time, we have nothing to announce that would satisfy all of them, and the existing infrastructure for User and Site Streams is no longer sustainable.

Do please keep an open mind, and continue to discuss your interests with us. Per our statement of direction and publication of the roadmap in April, we continue to invest in the Twitter data platform, and we’re excited about the features we can add for API consumers over time.

[dekisu]

Thanks for the reply.

The comment on Account Activity API not having follow events is not accurate, so far as I know - where did you get the information that follow events are not available in the new format?

Sorry, ambiguous wording on my side. I was referring to this part from the migration guide:

Another key difference you will notice in regards to data being delivered is that Twitter will no longer send events from people that you follow on Twitter. This was an intentional change and is not something we plan to alter going forward

Emphasis mine. What I had in mind when I wrote that was the with=followings parameter of streams, the default for the user stream behavior of adding messages from the home timeline, and something that is no longer available in AAAPI.

Your comment on TweetDeck is not quite relevant here, as TweetDeck is web-based, and also runs on Twitter’s internal infrastructure on the server side. Yes, there are client-side events but this is much more complex than that.

Well, I opened tweetdeck and the network tab of the browser inspector and saw home_timeline.json requests every 4 seconds. Practically the same thing we’re going to have to do, just with much generous rate limits.

I can understand it makes sense to give these privileges to twitter-controlled applications since it’s easy to adapt their code if necessary without breaking API promises, but other than that, I don’t see a big difference between tweetdeck’s polling and our polling. Both are just trying to request updates as often as possible. Am I missing something?

So my main question is: How likely is it that we could get improved polling rate limits, considering this is the only replacement to userstreams?

[richardhyland]

This is something I’d like to see as well. Mentions has a rate limit of 75 which would allow a poll almost every 5 seconds but for Home timeline we can only poll once per minute with current rate limits

Increasing the Home timeline to 75 or more when user streams goes away would provide nearly a viable alternative

[ePirat]

Well, I opened tweetdeck and the network tab of the browser inspector and saw home_timeline.json requests every 4 seconds. Practically the same thing we’re going to have to do, just with much generous rate limits.

Yes, Twitter always had very generous rate limits in their own clients which are very different from the rate limits that everyone else has to deal with.

I am really hoping that Twitter will offer a solution for standalone clients that replace User streams or at least make the rate limits for polling more generous, else we will be in the same situation that we were years ago, were clients had to carefully poll the API to not exceed the rate limits…

[andypiper]

We are exploring options internally, but do not have specific announcements or changes to provide today. If changes are made, these are likely to relate to rate limits for polling. There is no anticipated direct webhook analogue to the functionality referred to on the user stream API.

As announced last April, the developer platform roadmap is oriented towards data access, and not towards providing 1:1 mapping of client app features.

[jrhunt]

Hi @andypiper - here is twitter’s current timeline as I understand it.

“We are providing notice to all Twitter developers that on Tuesday June 19, 2018 we are retiring the following services and endpoints”…

Which means many apps / services need to migrate to the Account Activity webhooks APIs before June 19th… yet these APIs are still in beta. I have applied for access to this beta for a number of apps to no avail - it seems I was whitelisted for some completely different service and not the Account Activity API Beta. I can no longer submit an additional application to apply.

There are currently 5 months before you remove the APIs our current apps depend on… yet we cannot currently even use the APIs that are supposed to replace them. I beg, on behalf of your enthusiastic developer community, that you give at least 6 months from the time the APIs are generally available, to the the time you remove the old APIs. I think Twitter is making a very poor decision to remove existing APIs before the vast majority of developers even have access to the new ones.

Rewriting applications and services to webhooks is non-trivial in many cases. It takes time and effort. Scaling webhooks is a different architecture from scaling streams.

We recognize that twitter is working hard to provide a set of APIs that allows both twitter and customers of twitter to grow and scale but breaking backwards compatibility like this is something that should be done ONLY when there’s a suitable replacement that we have had time to build on.

[happycamper]

Hi @jrhunt -

Thanks for your feedback and discussion, we are learning and iterating during this beta in preparation for a later GA. Also note that for larger and more complex/feature rich integrations, we do have an enterprise version of the Account Activity API (here’s more). You can get in touch with an account rep by applying here: https://developer.twitter.com/en/enterprise-application

[happycamper]