Discussion for "Changes to following caps and message ordering for Site and User Streams"

kurrik · 2013-03-15T20:31:17.000Z

Use this thread to discuss [node:15873].

hca5 · 2013-03-15T21:13:53.000Z

Hi Arne, in the context of Site Streams, will the warning message you describe be a top-level message in the stream, or will it be packaged in a for_user envelope?

roly426 · 2013-03-15T21:33:53.000Z

It will likely be top level. I’m basing that on all other warning messages being top level and this new one does include a user_id.

kurrik · 2013-03-15T21:58:50.000Z

Yeah, top level as the warning applies to the entire stream. I’ll update the messages documentation to indicate this. Thanks for the insightful question!

aa_enezi · 2013-03-16T04:45:05.000Z

{
“warning”: {
“code”: “FOLLOWS_OVER_LIMIT”,
“message”: “The requested user follows more accounts than the maximum supported by this streaming endpoint. Only a subset of 10000 followed accounts are included in this stream.”,
“user_id”: <user_id>
}
}

tbbuck · 2013-03-21T15:02:55.000Z

Regarding users with over 10,000 followers on Site Streams, is there - or will there be - any way to specify a list of followers that we’d like to guarantee are included?

We have a few use cases where it doesn’t necessarily matter if we can’t include all of the followers, just as long as a much smaller subset of them (say, 20) can be guaranteed to be there.

Alternatively, is there any option to have our account whitelisted, and not have this limit imposed?

Many thanks,
Tom.

episod · 2013-03-21T15:06:16.000Z

No, this is not a planned feature at this time but it is an interesting request. There is no way to have the constraint waived for a Site Streams consumer at this time.

tbbuck · 2013-03-21T15:16:13.000Z

OK thanks - can you keep me informed if this changes, or you need testers for anything similar to this?

One other question: are you set on making it a random subset of followers? It’d be a huge improvement for me if it would always be the first 10,000 ordered by some date, or even by their id - basically, just something that we have half a chance of predicting

dtran320 · 2013-03-25T22:17:53.000Z

Seeing an extra ‘}’ character in the ‘friends’ event when we receive this warning:

{"for_user":,"message":{"warning":{"code":"FOLLOWS_OVER_LIMIT","message":"The requested user follows more accounts than the maximum supported by this streaming endpoint. Only a subset of 10000 followed accounts are included in this stream.","user_id":}} {"friends":[<10,000 ids REDACTED>]}}

roly426 · 2013-03-25T22:21:36.000Z

We’re getting the same error. The first message is missing a ‘}’ at the end.

kurrik · 2013-03-25T22:24:23.000Z

Thanks for the report.

dtran320 · 2013-03-26T00:51:44.000Z

Hey Arne,

Any updates on this? We pushed out a hotfix to just ignore improperly formatted JSON in any single event for now.

Is there anything I can do to provide more info/help you guys debug this?

Best,
David

kurrik · 2013-03-26T00:56:23.000Z

We rolled back sitestreams to prior behavior - you should not see the issue any more at this point. Are you still getting the invalid messages?