What is the actual limit for statuses/filter endpoint

[DmitryDzhus]

The documentation for statuses/filter Streaming API endpoint says that 5000 userids is a limit for the amount of followed accounts: https://dev.twitter.com/streaming/reference/post/statuses/filter.

However in reality I’m getting 503 Service Temporarily Unavailable error when using lists with 1426 accounts or longer:

"(Status {statusCode = 503, statusMessage = \"Service Temporarily Unavailable\"},[(\"connection\",\"close\"),(\"content-length\",\"0\"),(\"date\",\"Mon, 16 Nov 2015 11:41:52 GMT\"),(\"server\",\"tsa\"),(\"x-connection-hash\",\"027f64fea03b4680629d88b07ab62a92\")])"

It works if I decrease the list length to 1425 or less. This behaviour is consistent regardless of what OAuth tokens I use (I’ve tested with multiple tokens unused by other applications) or what IP of the connecting box is. I had an idea of splitting my list in some 1000-1400 long chunks and establishing a streaming API connection for every chunk, but in this case I get 200 OK response for the first connection and 420 Embrace your calm for the rest.

The cutoff value of 1425 is oddly specific, does anybody have any possible explanation to this?

The official example is not helpful, because it limits the number of followed users to just 100: https://github.com/twitter/hbc/blob/master/hbc-core/src/main/java/com/twitter/hbc/core/endpoint/SitestreamEndpoint.java#L32

If 1425 is the actual hard limit per single API token, shouldn’t the API documentation be updated to reflect this? It’s misleading otherwise.

[Bedon292]

Do you know, are you using POST or GET? If you are using GET then the limit is likely because you are creating a URL that is too long. As you said the docs say 5000 ids is the max, so I have to assume that GET vs POST is the problem. That will depend on the client you are using, and you might want to dig into the docs / source and see what they are using.

[DmitryDzhus]

Hello, I’ve checked and the client sends POST request to statuses/filter.json, so I’m ruling out URL being too long.

[DmitryDzhus]

Does anybody know if getting commercial access to Twitter feeds comes with technical support for cases like this?

[andypiper]

Apologies for the delayed responses here, I’ve been out for a couple of weeks.

Firstly, I’m not aware of a hard limit at 1425 user IDs, I believe it should be 5000 per the docs, but obviously it sounds like you’re having an issue and it does sound odd. Is there any chance you could provide the full track parameter (I realise this could be long!) and preferably a code sample - can you confirm that regardless of which users are included in the list, it always seems to have this limit?

You shouldn’t make additional stream connections - this is documented in the Connections information - this will indeed lead to 420 rate limiting errors.

Regarding the example you linked to, the reason it tops out at 100 is that you’re looking at a Site Streams sample, which does limit the initial connection to 100 users in a filter, and uses Control Streams to upgrade that to up to 1000 (this is irrelevant as Site Streams is at capacity/not available to you).

The User Streams endpoint should indeed support larger filter value.

Regarding “commercial access” - via Gnip, our enterprise data platform, you would get access to direct support, rather than the community that we provide for the public APIs via these forums.

[mktearlybird]

Hi, I’m facing an identical issue of having the streaming api (statuses/filter) limit the number of followed ID’s to approx 1430 rather than 5000. I am also using POST, however it isn’t clear when using POST if the filter parameters should still be in the url, or whether they should be sent in the body of the request.

When sending them in the url we get the limit above. When sending in the body of the post message I’m not able to get the api to respond - I get an “unauthorised” response despite the same credentials being used.

My guess is that we are supposed to send them in the body in order to reduce the length of the URL. Would you be able to verify this please and provide some details such as encoding etc if you are able to get it to work?

We have a commercial Gnip feed but we need to use the statuses/filter for compliance messages. I will reference this post in my discussions with gnip.

Thanks

Danny

[mktearlybird]

To save someone else some hassle, after further testing we found that for large numbers of follows or keywords you need to post them as post-parameters rather than in the URL. This also alters the authentication requirements hence the unauthorised response at first.

[DmitryDzhus]

Indeed, I double checked and in fact the library my application is using wrongly included API parameters in the request string. I’ve changed it to put parameters in request body and it works with up to 5000 followed accounts now, just as expected!

The issue is resolved.

[andypiper]