Discussion for "Changes to following caps and message ordering for Site and User Streams"


#1

Use this thread to discuss [node:15873].


#2

Hi Arne, in the context of Site Streams, will the warning message you describe be a top-level message in the stream, or will it be packaged in a for_user envelope?


#3

It will likely be top level. I’m basing that on all other warning messages being top level and this new one does include a user_id.


#4

Yeah, top level as the warning applies to the entire stream. I’ll update the messages documentation to indicate this. Thanks for the insightful question!


#5

{
“warning”: {
“code”: “FOLLOWS_OVER_LIMIT”,
“message”: “The requested user follows more accounts than the maximum supported by this streaming endpoint. Only a subset of 10000 followed accounts are included in this stream.”,
“user_id”: <user_id>
}
}


#6

Regarding users with over 10,000 followers on Site Streams, is there - or will there be - any way to specify a list of followers that we’d like to guarantee are included?

We have a few use cases where it doesn’t necessarily matter if we can’t include all of the followers, just as long as a much smaller subset of them (say, 20) can be guaranteed to be there.

Alternatively, is there any option to have our account whitelisted, and not have this limit imposed?

Many thanks,
Tom.


#7

No, this is not a planned feature at this time but it is an interesting request. There is no way to have the constraint waived for a Site Streams consumer at this time.


#8

OK thanks - can you keep me informed if this changes, or you need testers for anything similar to this?

One other question: are you set on making it a random subset of followers? It’d be a huge improvement for me if it would always be the first 10,000 ordered by some date, or even by their id - basically, just something that we have half a chance of predicting :slight_smile:


#9

Seeing an extra ‘}’ character in the ‘friends’ event when we receive this warning:

{"for_user":,"message":{"warning":{"code":"FOLLOWS_OVER_LIMIT","message":"The requested user follows more accounts than the maximum supported by this streaming endpoint. Only a subset of 10000 followed accounts are included in this stream.","user_id":}} {"friends":[<10,000 ids REDACTED>]}}

#10

We’re getting the same error. The first message is missing a ‘}’ at the end.


#11

Thanks for the report.


#12

Hey Arne,

Any updates on this? We pushed out a hotfix to just ignore improperly formatted JSON in any single event for now.

Is there anything I can do to provide more info/help you guys debug this?

Best,
David


#13

We rolled back sitestreams to prior behavior - you should not see the issue any more at this point. Are you still getting the invalid messages?