MSC4471: Streaming ephemeral event updates for room events

[bradtgmurray]

Rendered

Primarily built for AI/LLM use cases where the model generates responses slowly and incrementally and readers want to read the event while it's generated but not pollute the event graph long term once the response is complete, but I think also could be generally useful.

Heavily based upon the work of @batuhan and @tulir in mautrix/go#473, but I tweaked a few things while drafting the MSC that I hope are seen as simplifications or improvements.

[turt2live]

Implementation requirements:

• Client (sender)
• Client (renderer/subscriber)

[dozro]

you should maybe link to the rendered version like other MSCs do, in your PR description. so adding [Rendered](https://github.com/bradtgmurray/matrix-spec-proposals/blob/event-streams/proposals/4471-event-streams.md) to your PR description :3

[gingershaped]

It's worth noting that there are a lot of use cases for this MSC besides streaming LLM responses to clients. Bots in general could take advantage of this mechanism for displaying the progress of long-running tasks like CI jobs and batch processing work, which could perhaps be noted in the proposal itself to further dispel any concerns that this is only useful for LLMs.

[ara4n]

i literally needed this last night for experimenting with streaming transcriptions of matrixrtc calls

[batuhan]

To add to the point of this being useful beyond LLM's, live location sharing was one of the future features we were considering when we created our version of this MSC at Beeper.

[kegsay]

I do not think it is a good idea to add yet another encryption layer on top of Matrix. I understand the reasoning, but to me it still feels like a bad smell to do this when we have an adjacent encryption layer which we're choosing not to use.

[bradtgmurray]

@kegsay Can you clarify which adjacent encryption scheme you think would be better suited?

I hesitated to use olm/megolm or anything with a rachet since these messages will be quite frequent and probably vulnerable to gaps (hence the gap handling), so using a similar scheme to encrypted media felt reasonable.

[kegsay]

We already have an encryption layer in the form of Megolm.

If I understand correctly, your concern is for scenarios like:

• Your phone is offline and there's 1000 messages sent
• Your phone goes online and you.. can't decrypt the messages because the ratchet is too far advanced?

I don't think this is actually a problem, though I am not a cryptographer. The main limitation is out of order to-device messages, where we only hold onto a certain number of keys. Matrix intentionally serves to-device messages to clients in a FIFO-style queue to avoid this issue, so I'm struggling to identify what concrete issue you're trying to avoid by not using megolm.

My main problem with your approach is it's an entirely new scheme, even if it may look and feel similar to encrypted media, it still creates lots of problematic surfaces around replay attacks, key rotation semantics, authenticity guarantees, etc. Just saying "If the room event containing the stream descriptor is encrypted, the descriptor is already visible only to clients which can decrypt that room event." isn't really good enough imo, particularly if it means it's leaking event content (the partially assembled LLM response).

[kegsay]

I like the pattern of:

• Send PDU when starting stream
• Bunch of stream updates which clients may or may not be able to read
• Send final PDU when stream finished.

[kegsay]

Missing "Security Considerations".

In particular, there are DoS risks here with clients who maliciously subscribe many devices to blow up the subscriber count.

[kegsay]

Honestly I would get rid of this and just always replace. The expected size of these updates are what, 2KB total? You're capped at 64KB anyway and these streams demand a certain level of network connectivity such that I am not sure it's worth the added complexity of handling sequences etc. If you drop it, you can just blindly apply the last stream update you receive down the wire (LWW style) and be done with it.

[bradtgmurray]

Elaborated below, but I'm designing for a case where the event is being updated 4-5 times a second with 10-20 characters being added each time. My bad for not including that design parameter in the MSC in the first place.

[kegsay]

There is a painful amount of overlap here machinery wise with MSC4354: Sticky Events. Earlier designs for sticky events did in fact use EDUs. If sticky events used EDUs then I think this would mostly obsolete the need for this MSC.

As such, I wonder if the right call is to build off sticky events to make it support EDUs rather than go off and invent a whole new delivery mechanism. You could then reuse things like the sync token field which sticky events needs, reuse the LWW semantics on clients, reuse the section in the /sync response which is separate from timeline, etc.

This then lets designers choose which properties they want: if they want convergence and persistence because they need to maintain RTC call state, use sticky events in PDU mode. If they are okay losing convergence and want ephemerality, then use sticky events in EDU mode.

We would probably need to be restrictive of the EDU type, lest it basically become MSC2477: User-defined ephemeral events in rooms, but I think architecturally this feels cleaner.

[bradtgmurray]

Thanks for the feedback! That's a fair flag, and I spent a good chunk of time yesterday reading through Sticky Events and considering the approach. I'm not against it, but I think I did a poor job of outlining the requirements here (and I'll edit the MSC to include this in a moment).

The goal here is to enable an experience where the observed event is appearing to mutate in real time. For ChatGPT for example, if you pop open the Chrome devtools, you can see it using server-sent events to send chunks of updates that are 10-20 characters long 4-5 times a second, and then the frontend mutates this into making the updates appear a character at a time. You could imagine this same frequency of updates being useful for Matthew's transcription case, or a case where we could actually tail a build log where we want the event to be updated "live". However, this generates such volume of traffic that it would be inappropriate to hit a mobile device with every update upon foregrounding, just because this is happening in a room that the user isn't even looking at.

The goal of this MSC is for client devices to opt-in and say yes, I want and need this large amount of traffic to enable this "real time" use case. For the use cases I'm imagining, it's likely that only a handful of users and only a single one of their devices will be observing the event in real time, so let's design a scheme where only those devices get the high amount of traffic. If someone arrives late, they can late subscribe and get brought up to speed and be real-time from there, but then if they leave (unfocus the room, not actually leave) then their client can stop the traffic flow by unsubscribing from that event stream.

Note for other usecases such a live location sharing or CI sharing the state of a build having less frequent (every few seconds) is probably fine, in which case something like Sticky Events where every device in the room gets every update is probably fine.

I'm curious if we can merge this with Sticky Events and have those properties, but I think the key thing is the opt-in, which I think makes it structurally different that Sticky Events.

[kegsay]

However, this generates such volume of traffic that it would be inappropriate to hit a mobile device with every update upon foregrounding, just because this is happening in a room that the user isn't even looking at.

..which is sorta the whole point with MSC4186 - the whole point there is that you only subscribe to rooms you're actively viewing, and only get room summaries otherwise. That's intrinsically per-device as the SSS connection is device scoped. In fact, this proposal is arguably worse for traffic over federation due to each visible device needing to send subscribe/unsubscribe signals over federation, vs just receiving the data and letting the receiving HS determine client visibility based on what clients request via their local HS.

A compounding factor is that these updates are to-device messages, which unfortunately have that intentional FIFO property I mentioned on the encryption comment. This means your streams would actively block the delivery of room keys preventing events from being decrypted. You can't selectively drain the queue per event type (e.g. "only drain room keys to-device messages").

Hopefully you can see why I'm questioning this proposal: the problems you're solving are already solved in slightly different ways with MSCs approaching FCP, and the proposal is introducing edges which don't yet have good answers.

[bradtgmurray]

The questioning is completely warranted! I appreciate the thoughtful review. Honestly, at Beeper we never moved to Sliding Sync, so I kind of overlook how sliding sync could also address this by scoping down who receives the traffic.

I still don't love ptuting these incremental updates in the event graph though. Correct me if I'm wrong, but with sticky events, they'll be in their own portion of the sync payload during their sticky period, but additionally if you paginate back you'll also see the events where the sticky period have lapsed. If we have a use cases where events are being streamed, and each message got a few dozen incremental streamed updates while it was being sent, I don't really love those events just hanging around in the pagination history forever. We could potentially have pagination skip those events? But that feels clunky as well.

My goal here is "send updates only to clients that want it, only during the period that the updates are relevant, and then once the message has stopped being updated that data is gone forever".

[kegsay]

You're right, if they were PDUs. The idea is that you could have them be EDUs and reuse the same section of the sync response, thus when the stickiness expires they really are gone gone and don't leave a trace in the DAG.

[bradtgmurray]

Ah I understand what you're saying.

I think this MSC may be too pragmatic and I've basically optimized for not requiring homeserver changes, so I've smuggled what I needed through existing channels (todevice). We could get a cleaner design by making the event shape more like sticky events, which makes sense to me.

I still think that the subscribe/unsubscribe semantics are pretty important, which to my mind could make it a different thing than the existing sticky event shape. How do you feel about a "subscribable" sticky event? Or should we just let publishers publish at a high rate regardless of subscribers and have sliding sync do the work of avoiding delivering events to clients that don't want it? We could also only deliver these types of events over sliding sync and not deliver them at all in sync v2?

[kegsay]

Or should we just let publishers publish at a high rate regardless of subscribers and have sliding sync do the work of avoiding delivering events to clients that don't want it?

Yes, let them publish because it is unclear if there is a benefit to having subscription logic at the federation layer or not (sub/unsub control messages via to-device). It's an optimisation we can add later but can't remove if it turns out to be too complex.

We could also only deliver these types of events over sliding sync and not deliver them at all in sync v2?

Yes, we could do that.

[bradtgmurray]

First PR to start implementing this is up at matrix-org/matrix-rust-sdk#6607

[bradtgmurray]

Hey all, I'm thinking of taking another run at this based on the feedback I've heard so far.

Goal: Enable event senders the ability to rapidly update an event they sent for a short (< 1 hour) period of time after they send it, with a target of 5 updates per second and potentially 1000s of updates for each event. Clients that aren't actively observing the event during this period do not see these updates, but can start receiving the traffic at any time during the update window. The updates do not go into the room DAG, but the final state of the event does.

Tweaks from the proposal as it is today:

• Distribution mechansim: Instead of each subscriber getting it's own unique stream of to-device events from the publisher, the publisher sends a new entity that represents its homeserver. These get distributed to all users in the room over federation using EDUs. These events do not get distributed over sync using the default sync filters. Clients can opt into the room's streaming updates they're currently viewing by mutating their sync filter. The client's homeserver needs to hold onto all the updates for the duration of the streaming period, and if a client starts receiving streaming updates late for an event, all previous updates are replayed by the homeserver to the client over sync. Note: I'm still not sure that this is the same thing as an "emphemeral sticky event", it feels like it has pretty different semantics.

• Encryption: Let's use megolm for these updates from the publisher instead. To avoid exhausting the rachet for the megolm key already being used by that device to send to the room, let's use a new megolm private key for each set of streaming updates. The private key can be included in the payload of the original streaming event to avoid having to pay the overhead of distributing a key that clients may not use. The megolm key will have an additional field that is an expires_at time, which allows clients to throw away the key once the stream expires, as the final edit to the event to remove the streaming capability will be encrypted with the original room key. This megolm key for the particular stream should not be backed up to key backup, it can be retrieved at any time by decrypting the original event again.

Looking at the alternative design proposed above, I'm not sure if it's better than the originally proposed approach.

Pros: Clients are simpler (homeservers do resyncing by playback), encryption uses an existing scheme. Traffic between the publisher and client and publisher's homserver and other homeservers is lower when there's multiple interested devices on the receivers homeservers.

Cons: Servers are more complicated to store the new entity type, Server-to-server gets a new entity type, sync is more complicated, servers now receive the traffic regardless if they have subscribing devices, replay means that clients potentially receive more traffic in cases where the stream is replaced multiple times (the server can't mutate the stream at all on behalf of clients as they're encrypted once for all subscribers with megolm).

I think I want to continue experimenting with the to-device approach, it just feels to me that in the low subscriber case where subscribers come and go the data being distributed can be optimized quite a bit by the publisher being subscriber-aware, but of course I still want to continue the discussion.