From 9cbe4481b7e2eb4ee765611cd6bb6888ff1698e1 Mon Sep 17 00:00:00 2001 From: Timo K Date: Fri, 10 May 2024 12:32:12 +0200 Subject: [PATCH 01/62] draft Signed-off-by: Timo K --- proposals/4143-MatrixRTC.md | 150 ++++++++++++++++++++++++++++++++++++ 1 file changed, 150 insertions(+) create mode 100644 proposals/4143-MatrixRTC.md diff --git a/proposals/4143-MatrixRTC.md b/proposals/4143-MatrixRTC.md new file mode 100644 index 00000000000..fba0fcaf5ff --- /dev/null +++ b/proposals/4143-MatrixRTC.md @@ -0,0 +1,150 @@ +# MSC4143: MatrixRTC + +This MSC defines the modules with which the matrix real time system is build with. + +The MatrixRTC specification is separated into different modules. + +- The MatrixRTC room state that defines the state of the real time application.\ + It is the source of truth for: + - Who is part of a session + - Who is connected via what technology/backend + - Metadata per device used by other participants to decide whether the streams + from this source are of interest / need to be subscribed. +- The RTC backend. + - It defines how to connect the participating peers. + - Livekit is the standard for this as of writing. + - Defines how to connect to a server/other peers, how to update the connection, + how to subscribe to different streams... + - Another planned backend is a full mesh implementation based on MSC3401. +- The RTCSession types (application) have their own per application spec. + - Calls can be done with an application of type `m.call` see (TODO: link call msc) + - The application defines all the details of the RTC experience: + - How to interpret the metadata of the member events. + - What streams to connect to. + - What data in which format to sent over the RTC channels. + +This MSC will focus on the matrix room state which can be seen as the most high +level signalling of a call: + +## Proposal + +Each RTC session is made out of a collection of `m.rtc.member` events. +Each `m.rtc.member` event defines the application type: `application` +and a `call_id`. And is stored in a state event of type `m.rtc.member`. +The first element of the state key is the `userId` and the second the `deviceId`. +(see [this proposal for state keys](https://github.com/matrix-org/matrix-spec-proposals/pull/3757#issuecomment-2099010555) +for context about second/first state key.) + +### The MatrixRTC room state + +Everything required for working MatrixRTC +(current session, sessions history, join/leave events, ...) only +require one event type. + +A complete `m.rtc.member` state event looks like this: + +```json +// event type: m.rtc.member +// event key: ["@user:matrix.domain", "DEVICEID"] +{ + "m.application": "m.my_session_type", + "m.call_id": "", + "focus_active": {...FOCUS_A}, + "foci_preferred": [ + {...FOCUS_1}, + {...FOCUS_2} + ] +} +``` + +giving us the information, that user: `@user:matrix.domain` with device `DEVICEID` +is part of an RTCSession of type `m.call` in the scope/sub-session `""` (empty +string as call id) connected over `FOCUS_A`. This is all information that is needed +for another room member to detect the running session and join it. + +There is **no event** to represent a session. This event would include shared +information where it is not trivial to decide who has authority over it. +Instead the session is a computed value based on `m.rtc.member` events. +The list of events with the same `m.application` and `m.call_id` represent one session. +This array allows to compute fields like participant count, start time ... + +Sending an empty `m.rtc.member` event represents a leave action. +Sending a well formatted `m.rtc.member` represents a join action. + +Based on the value of `m.application`, the event might include additional parameters +required to provide additional session parameters. + +> A thirdRoom like experience could include the information of an approximate position +on the map, so that clients can omit connecting to participants that are not in their +area of interest. + +#### Historic sessions + +Since there is no singe entry for a historic session (because of the owner ship discussion), +historic sessions need to be computed and most likely cached on the client. + +Each state event can either mark a join or leave: + +- join: `prev_state.m.application != current_state.m.application` && + `prev_state.m.call_id != current_state.m.call_id` && + `current_state.m.application != undefined` + (where an empty `m.rtc.member` event would imply `state.m.application == undefined`) +- leave: `prev_state.m.application != current_state.m.application` && + `prev_state.m.call_id != current_state.m.call_id` && + `current_state.m.application == undefined` + +Based on this one can find user sessions. (The range between a join and a leave +event) of specific times. +The collection of all overlapping user sessions with the same `call_id` and +`application` define one MatrixRTC history event. + +### The RTC backend + +`foci_active` and `foci_preferred` are used to communicate + +- how a user is connected to the session (`foci_active`) +- what connection method this user knows about would like to connect with. + +The only enforced parameter of a `foci_preferred` or `foci_active` is `type`. +Based on the focus type a different amount of parameters might be needed to, +communicate how to connect to other users. +`foci_preferred` and `foci_active` can have different parameters so that it is, +possible to use a combination of the two to figure our that everyone is connected +with each other. + +Only users with the same type can connect in one session. If a frontend does +not support the used type they cannot connect. + +Each focus type will get its own MSC in which the detailed procedure to get from +the foci information to working webRTC connections to the streams of all the +participants is explained. + +- [`livekit`](www.example.com) TODO: create `livekit` focus MSC and add link here. +- [`full_mesh`](https://github.com/matrix-org/matrix-spec-proposals/pull/3401) +TODO: create `full-mesh` focus MSC based on[MSC3401](https://github.com/matrix-org/matrix-spec-proposals/pull/3401) +and add link here. + +### The RTCSession types (application) + +Each session type might have its own specification in how the different streams +are interpreted and even what focus type to use. This makes this proposal extremely +flexible. A Jitsi conference could be added by introducing a new `m.application` +and a new focus type and would be MatrixRTC compatible. It would not be compatible +with applications that do not use the Jitsi focus but clients would know that there +is an ongoing session of unknown type and unknown focus and could display/represent +this in the user interface. + +To make it easy for clients to support different RTC session types, the recommended +approach is to provide a matrix widget for each session type, so that client developers +can use the widget as the first implementation if they want to support this RTC +session type. + +Each application should get its own MSC in which the all the additional +fields are explained and how the communication with the possible foci is +defined: + +- [`m.call`](www.example.com) TODO: create `m.call` MSC and add link here. + +## Potential issues +## Alternatives +## Security considerations \ No newline at end of file From b2b4e5ef4d70367dcdd0c3c82e6c420f3c662a7b Mon Sep 17 00:00:00 2001 From: Timo K Date: Mon, 13 May 2024 13:55:57 +0200 Subject: [PATCH 02/62] lowercase filename Signed-off-by: Timo K --- proposals/{4143-MatrixRTC.md => 4143-matrix-rtc.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename proposals/{4143-MatrixRTC.md => 4143-matrix-rtc.md} (100%) diff --git a/proposals/4143-MatrixRTC.md b/proposals/4143-matrix-rtc.md similarity index 100% rename from proposals/4143-MatrixRTC.md rename to proposals/4143-matrix-rtc.md From 8c1340a69284cc113de1123c181b9847425ee7ad Mon Sep 17 00:00:00 2001 From: Timo Date: Wed, 15 May 2024 00:06:48 +0200 Subject: [PATCH 03/62] add note about this relying on MSC3757 --- proposals/4143-matrix-rtc.md | 35 ++++++++++++++++++++++------------- 1 file changed, 22 insertions(+), 13 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index fba0fcaf5ff..69acac5f282 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -1,20 +1,20 @@ # MSC4143: MatrixRTC -This MSC defines the modules with which the matrix real time system is build with. +This MSC defines the modules with which the matrix real time system is build. The MatrixRTC specification is separated into different modules. - The MatrixRTC room state that defines the state of the real time application.\ - It is the source of truth for: + It is the source of truth for: - Who is part of a session - Who is connected via what technology/backend - Metadata per device used by other participants to decide whether the streams - from this source are of interest / need to be subscribed. + from this source are of interest / need to be subscribed. - The RTC backend. - It defines how to connect the participating peers. - Livekit is the standard for this as of writing. - Defines how to connect to a server/other peers, how to update the connection, - how to subscribe to different streams... + how to subscribe to different streams... - Another planned backend is a full mesh implementation based on MSC3401. - The RTCSession types (application) have their own per application spec. - Calls can be done with an application of type `m.call` see (TODO: link call msc) @@ -57,6 +57,13 @@ A complete `m.rtc.member` state event looks like this: } ``` +> [!NOTE] +> This relies on [MSC3757](https://github.com/matrix-org/matrix-spec-proposals/pull/3757). +> We need to have one state event per device, hence multiple "non-overwritable" state +> events per user. +> +> More specifically this uses the approach outlined in this [comment](https://github.com/matrix-org/matrix-spec-proposals/pull/3757#issuecomment-2099010555). + giving us the information, that user: `@user:matrix.domain` with device `DEVICEID` is part of an RTCSession of type `m.call` in the scope/sub-session `""` (empty string as call id) connected over `FOCUS_A`. This is all information that is needed @@ -75,8 +82,8 @@ Based on the value of `m.application`, the event might include additional parame required to provide additional session parameters. > A thirdRoom like experience could include the information of an approximate position -on the map, so that clients can omit connecting to participants that are not in their -area of interest. +> on the map, so that clients can omit connecting to participants that are not in their +> area of interest. #### Historic sessions @@ -86,12 +93,12 @@ historic sessions need to be computed and most likely cached on the client. Each state event can either mark a join or leave: - join: `prev_state.m.application != current_state.m.application` && - `prev_state.m.call_id != current_state.m.call_id` && - `current_state.m.application != undefined` + `prev_state.m.call_id != current_state.m.call_id` && + `current_state.m.application != undefined` (where an empty `m.rtc.member` event would imply `state.m.application == undefined`) - leave: `prev_state.m.application != current_state.m.application` && - `prev_state.m.call_id != current_state.m.call_id` && - `current_state.m.application == undefined` + `prev_state.m.call_id != current_state.m.call_id` && + `current_state.m.application == undefined` Based on this one can find user sessions. (The range between a join and a leave event) of specific times. @@ -121,8 +128,8 @@ participants is explained. - [`livekit`](www.example.com) TODO: create `livekit` focus MSC and add link here. - [`full_mesh`](https://github.com/matrix-org/matrix-spec-proposals/pull/3401) -TODO: create `full-mesh` focus MSC based on[MSC3401](https://github.com/matrix-org/matrix-spec-proposals/pull/3401) -and add link here. + TODO: create `full-mesh` focus MSC based on[MSC3401](https://github.com/matrix-org/matrix-spec-proposals/pull/3401) + and add link here. ### The RTCSession types (application) @@ -146,5 +153,7 @@ defined: - [`m.call`](www.example.com) TODO: create `m.call` MSC and add link here. ## Potential issues + ## Alternatives -## Security considerations \ No newline at end of file + +## Security considerations From 8c800a634f977037e9e2d126c1e4cbd223dcbfb1 Mon Sep 17 00:00:00 2001 From: Timo Date: Fri, 17 May 2024 15:37:58 +0200 Subject: [PATCH 04/62] remove the m.prefix from fields. --- proposals/4143-matrix-rtc.md | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 69acac5f282..b0bce477921 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -47,8 +47,10 @@ A complete `m.rtc.member` state event looks like this: // event type: m.rtc.member // event key: ["@user:matrix.domain", "DEVICEID"] { - "m.application": "m.my_session_type", - "m.call_id": "", + "application": "m.my_session_type", + "call_id": "", + "device_id": "DEVICEID", + "created_ts": Time | undefined, "focus_active": {...FOCUS_A}, "foci_preferred": [ {...FOCUS_1}, @@ -69,16 +71,24 @@ is part of an RTCSession of type `m.call` in the scope/sub-session `""` (empty string as call id) connected over `FOCUS_A`. This is all information that is needed for another room member to detect the running session and join it. +We include the device_id in the member content to not rely on the exact format of the state key. +In case [MSC3757](https://github.com/matrix-org/matrix-spec-proposals/pull/3757) is used it would not +be the second element of the state key array. + +`created_ts` is an optional property that caches the time of creation. It is not required +for an event that, has not yet been updated, there the `origin_server_ts` is used. +Once the event gets updated the origin_server_ts needs to be copied into the `created_ts` field. + There is **no event** to represent a session. This event would include shared information where it is not trivial to decide who has authority over it. Instead the session is a computed value based on `m.rtc.member` events. -The list of events with the same `m.application` and `m.call_id` represent one session. +The list of events with the same `application` and `m.call_id` represent one session. This array allows to compute fields like participant count, start time ... Sending an empty `m.rtc.member` event represents a leave action. Sending a well formatted `m.rtc.member` represents a join action. -Based on the value of `m.application`, the event might include additional parameters +Based on the value of `application`, the event might include additional parameters required to provide additional session parameters. > A thirdRoom like experience could include the information of an approximate position @@ -92,13 +102,13 @@ historic sessions need to be computed and most likely cached on the client. Each state event can either mark a join or leave: -- join: `prev_state.m.application != current_state.m.application` && +- join: `prev_state.application != current_state.application` && `prev_state.m.call_id != current_state.m.call_id` && - `current_state.m.application != undefined` - (where an empty `m.rtc.member` event would imply `state.m.application == undefined`) -- leave: `prev_state.m.application != current_state.m.application` && + `current_state.application != undefined` + (where an empty `m.rtc.member` event would imply `state.application == undefined`) +- leave: `prev_state.application != current_state.application` && `prev_state.m.call_id != current_state.m.call_id` && - `current_state.m.application == undefined` + `current_state.application == undefined` Based on this one can find user sessions. (The range between a join and a leave event) of specific times. @@ -135,7 +145,7 @@ participants is explained. Each session type might have its own specification in how the different streams are interpreted and even what focus type to use. This makes this proposal extremely -flexible. A Jitsi conference could be added by introducing a new `m.application` +flexible. A Jitsi conference could be added by introducing a new `application` and a new focus type and would be MatrixRTC compatible. It would not be compatible with applications that do not use the Jitsi focus but clients would know that there is an ongoing session of unknown type and unknown focus and could display/represent From 1dcbfce23a6d6b4e99fc338a19d2489c50c15f14 Mon Sep 17 00:00:00 2001 From: Timo Date: Wed, 5 Jun 2024 15:59:48 +0200 Subject: [PATCH 05/62] update --- proposals/4143-matrix-rtc.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index b0bce477921..1fa274987c6 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -44,7 +44,7 @@ require one event type. A complete `m.rtc.member` state event looks like this: ```json -// event type: m.rtc.member +// event type: "m.rtc.member" // event key: ["@user:matrix.domain", "DEVICEID"] { "application": "m.my_session_type", @@ -66,7 +66,7 @@ A complete `m.rtc.member` state event looks like this: > > More specifically this uses the approach outlined in this [comment](https://github.com/matrix-org/matrix-spec-proposals/pull/3757#issuecomment-2099010555). -giving us the information, that user: `@user:matrix.domain` with device `DEVICEID` +This gives us the information, that user: `@user:matrix.domain` with device `DEVICEID` is part of an RTCSession of type `m.call` in the scope/sub-session `""` (empty string as call id) connected over `FOCUS_A`. This is all information that is needed for another room member to detect the running session and join it. @@ -77,10 +77,18 @@ be the second element of the state key array. `created_ts` is an optional property that caches the time of creation. It is not required for an event that, has not yet been updated, there the `origin_server_ts` is used. + +> [!NOTE] +> We introduce `created_ts()` as the notation for `created_ts ?? origin_server_ts` + Once the event gets updated the origin_server_ts needs to be copied into the `created_ts` field. +An existing `created_ts` field implies that this is a state event updating the current session +and a missing `created_ts` field implies that it is a join state event. +All membership events that belong to one member session can be grouped with the index +`created_ts()`+`device_id`. This is why the `m.rtc.member` events deliberately do NOT include a `membership_id`. -There is **no event** to represent a session. This event would include shared -information where it is not trivial to decide who has authority over it. +Other then the membership sessions, there is **no event** to represent a rtc session (containing all members). +This event would include shared information where it is not trivial to decide who has authority over it. Instead the session is a computed value based on `m.rtc.member` events. The list of events with the same `application` and `m.call_id` represent one session. This array allows to compute fields like participant count, start time ... From 813a21ad12f5aa8ee6aaa754bc599d624e5db609 Mon Sep 17 00:00:00 2001 From: Timo Date: Thu, 6 Jun 2024 18:02:07 +0200 Subject: [PATCH 06/62] add foci_preferred well known section --- proposals/4143-matrix-rtc.md | 52 ++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 1fa274987c6..6ad9f037466 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -149,6 +149,50 @@ participants is explained. TODO: create `full-mesh` focus MSC based on[MSC3401](https://github.com/matrix-org/matrix-spec-proposals/pull/3401) and add link here. +#### Sourcing `foci_preferred` + +At some point participants have to decide/propose which focus they use. +Based on the focus type and usecase choosing a `foci_preferred` can be different. +If possible these guidelines should be obeyed: + +- If there is a relation between the `focus_active` and a preferred focus (`type: livekit` is an example for this) + it is recommended to copy _the preferred focus that relates to the current `focus_active`_ of other participants to the start of the `foci_preferred` array of the member event. + (The exact definition of: _the preferred focus that relates to the current `focus_active`_ is part of the specification for each focus type. For `full_mesh` for example there is no such thing as: _the preferred focus that relates to the current `focus_active`_ ) +- Homeservers can proposes `preferred_foci` via the well known. An array of preferred foci is provided behind the well known key `m.rtc_foci`: + ```json + "m.rtc_foci": [ + { + "type":"livekit", + "livekit_service_url":"https://my_livekit_focus.domain" + } + ] + ``` + Those proposals from **your own** homeserver should come next in the `foci_preferred` list of the member event. +- Clients also have the option to configure a preferred foci even though this is not recommended (see below). + Those come last in the list. + +The rational for those guidelines are as following: + +- It is always desired to have as little focus switches as possible. + That is why the highest priority is to prefer the focus that is already in use +- MatrixRTC is designed around the same culture that makes matrix possible: + A large amount of infrastructure in form of homeservers is provided by the users. + For MatrixRTC the same is thea goal. To achieve a stable and healthy ecosystem + rtc infrastructure should be thought of as a part of a homeserver. It is very similar + to a turn server: mostly traffic and little cpu load. + To not end up in a world where each user is only using one central SFU but where the traffic + is split over multiple SFU's it is important that we leverage the SFU distribution on the + homeserver distribution. + For this reason the second guideline is to lookup the prefferred foci from the homeserver well_known +- Looking up the prefferred foci from a client is toxic to a federated system. If the majority of users + decide to use the same client all of the users will use one Focus. This destroys the passive security mechanism, that each instance is not an interesting attack vector since it is only a fraction of the network. + Additionally it will result in poor performance if every user on matrix would use the same Focus. + There are cases where this is acceptable: + - Transitioning to MatrixRTC. Here it might be beneficial to have a client that has a fallback Focus + so calls also work with homeservers not supporting it. + - For testing purposes where a different Focus should be tested but one does not want to touch the .well_known + - For custom deployments that benefit from having the Focus configuration on a per client basis instead of per homeserver. + ### The RTCSession types (application) Each session type might have its own specification in how the different streams @@ -175,3 +219,11 @@ defined: ## Alternatives ## Security considerations + +## Unstable prefix + +The state events and the well_known key introduced in this MSC use the unstable prefix +`org.matrix.msc4143.` instead of `m.` as used in the text. + +Possible values inside the `m.rtc.member` event (like `m.call`) will use a prefix defined in the +related PR (TODO create and link `m.call` application type PR) From d65e42e06c5490bbf48b211144e31829aee789b8 Mon Sep 17 00:00:00 2001 From: Timo Date: Thu, 20 Jun 2024 11:53:47 +0200 Subject: [PATCH 07/62] update to reference to msc [MSC4158](https://github.com/matrix-org/matrix-spec-proposals/pull/4158) --- proposals/4143-matrix-rtc.md | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 6ad9f037466..aa003afe698 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -158,15 +158,7 @@ If possible these guidelines should be obeyed: - If there is a relation between the `focus_active` and a preferred focus (`type: livekit` is an example for this) it is recommended to copy _the preferred focus that relates to the current `focus_active`_ of other participants to the start of the `foci_preferred` array of the member event. (The exact definition of: _the preferred focus that relates to the current `focus_active`_ is part of the specification for each focus type. For `full_mesh` for example there is no such thing as: _the preferred focus that relates to the current `focus_active`_ ) -- Homeservers can proposes `preferred_foci` via the well known. An array of preferred foci is provided behind the well known key `m.rtc_foci`: - ```json - "m.rtc_foci": [ - { - "type":"livekit", - "livekit_service_url":"https://my_livekit_focus.domain" - } - ] - ``` +- Homeservers can proposes `preferred_foci` via the well known. An array of preferred foci is provided behind the well known key `m.rtc_foci`. This is defined in [MSC4158](https://github.com/matrix-org/matrix-spec-proposals/pull/4158). They are related and it is recommended to also read [MSC4158](https://github.com/matrix-org/matrix-spec-proposals/pull/4158) with this MSC. Those proposals from **your own** homeserver should come next in the `foci_preferred` list of the member event. - Clients also have the option to configure a preferred foci even though this is not recommended (see below). Those come last in the list. From 4ab679affc6fc08522a29b8ab804f91333e27023 Mon Sep 17 00:00:00 2001 From: Timo Date: Tue, 2 Jul 2024 17:17:27 +0200 Subject: [PATCH 08/62] add a "Reliability requirements for the room state" section --- proposals/4143-matrix-rtc.md | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index aa003afe698..2064de305ef 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -103,6 +103,37 @@ required to provide additional session parameters. > on the map, so that clients can omit connecting to participants that are not in their > area of interest. +#### Reliability requirements for the room state + +Room state is a very well suited place to store the data for a MatrixRTC session +if allows: + +- The client to determine current ongoing sessions without loading history for every room. + Or doing additional work other then the sync loop that needs to run anyways. +- The client can compute/access data of past sessions without any additional redundant data. +- Sessions (start/end/participant count) are federated and there is not redundant data storage that + could result in conflicts, or can get out of sync. The room state events are part of the dag and this + is solved like any other PDU in matrix. + +A chellanging circumstance with using the room state to represent a session is +the disconnection behaviour. If the client disconnects from a call because of a network issue, +an application crash or a user forcefully quitting the client, the room state cannot be updated anymore. +The client is required to leave by sending a new empty state which cannot happen once connection is lost. + +If the state is not updated correctly we end up with incorrect session end timestamps a room state that is not correctly representing the current RTC session state. Historic and current MatrixRTC session data would be broken. + +For an acceptable solution, the following requirements need to be taken into consideration: + +- Room state is set to empty if the client looses connection. (A heardbeat like system is desired) +- The best source of truth for a call participation is a working connection to the SFU. + It is desired that the disconnect of the SFU is connected to the room state. +- It should be possible to updated the room state without the client being online. +- All this should be compatible when matrix uses cryptographic identities. + +[MSC4340](https://github.com/matrix-org/matrix-spec-proposals/pull/4140) proposes a concept to +delay the leave events until one of the leave conditions (heartbeat or SFU disconnect) occur +and fulfil all of the these requirements. + #### Historic sessions Since there is no singe entry for a historic session (because of the owner ship discussion), @@ -185,7 +216,7 @@ The rational for those guidelines are as following: - For testing purposes where a different Focus should be tested but one does not want to touch the .well_known - For custom deployments that benefit from having the Focus configuration on a per client basis instead of per homeserver. -### The RTCSession types (application) +### The RTC Session types (application) Each session type might have its own specification in how the different streams are interpreted and even what focus type to use. This makes this proposal extremely From 6d84256f60c6d39469786a582e9e2b2d7901ab40 Mon Sep 17 00:00:00 2001 From: Timo Date: Wed, 11 Sep 2024 19:10:42 +0200 Subject: [PATCH 09/62] use current state key format --- proposals/4143-matrix-rtc.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 2064de305ef..004f5aa61e3 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -45,7 +45,7 @@ A complete `m.rtc.member` state event looks like this: ```json // event type: "m.rtc.member" -// event key: ["@user:matrix.domain", "DEVICEID"] +// event key: "@user:matrix.domain_DEVICEID" { "application": "m.my_session_type", "call_id": "", @@ -63,8 +63,6 @@ A complete `m.rtc.member` state event looks like this: > This relies on [MSC3757](https://github.com/matrix-org/matrix-spec-proposals/pull/3757). > We need to have one state event per device, hence multiple "non-overwritable" state > events per user. -> -> More specifically this uses the approach outlined in this [comment](https://github.com/matrix-org/matrix-spec-proposals/pull/3757#issuecomment-2099010555). This gives us the information, that user: `@user:matrix.domain` with device `DEVICEID` is part of an RTCSession of type `m.call` in the scope/sub-session `""` (empty From d68e94262e8af7df7509a4b583945be2a00015ea Mon Sep 17 00:00:00 2001 From: Timo Date: Wed, 11 Sep 2024 19:12:32 +0200 Subject: [PATCH 10/62] add `expires_after` --- proposals/4143-matrix-rtc.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 004f5aa61e3..729a5b7e67a 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -51,6 +51,7 @@ A complete `m.rtc.member` state event looks like this: "call_id": "", "device_id": "DEVICEID", "created_ts": Time | undefined, + "expires_after": Duration, "focus_active": {...FOCUS_A}, "foci_preferred": [ {...FOCUS_1}, From 3b19b49bff758e1018ea8bef2dad52d2e54778b2 Mon Sep 17 00:00:00 2001 From: Timo Date: Wed, 11 Sep 2024 19:12:58 +0200 Subject: [PATCH 11/62] add section about send order (delayed leave event -> join event) --- proposals/4143-matrix-rtc.md | 37 +++++++++++++++++++++++++++++++----- 1 file changed, 32 insertions(+), 5 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 729a5b7e67a..433f0d60468 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -119,7 +119,8 @@ the disconnection behaviour. If the client disconnects from a call because of a an application crash or a user forcefully quitting the client, the room state cannot be updated anymore. The client is required to leave by sending a new empty state which cannot happen once connection is lost. -If the state is not updated correctly we end up with incorrect session end timestamps a room state that is not correctly representing the current RTC session state. Historic and current MatrixRTC session data would be broken. +If the state is not updated correctly we end up with incorrect session end timestamps a room state that is not +correctly representing the current RTC session state. Historic and current MatrixRTC session data would be broken. For an acceptable solution, the following requirements need to be taken into consideration: @@ -133,6 +134,25 @@ For an acceptable solution, the following requirements need to be taken into con delay the leave events until one of the leave conditions (heartbeat or SFU disconnect) occur and fulfil all of the these requirements. +A matrixRTC client has to first send/schedule the following delayed leave event: + +```json +// event type: "m.rtc.member" +// event key: "@user:matrix.domain_DEVICEID" +{ + "leave_reason": "CONNECTION_LOST" +} +``` + +only after that the actual state event can be sent, so that we guarantee that the state will be empty eventually. +The `leave_reason` is added so clients can be more verbal about why a user disconnected from a call. + +Receiving clients will be able to detect if this order was not followed with the `has_delayed_overwrite: true` +unsigned property. If the property is missing the event is invalid. + +This also invalides delayed leave events that are send with a valid membership content. They do not contain the +`has_delayed_overwrite: true` unsigned property. + #### Historic sessions Since there is no singe entry for a historic session (because of the owner ship discussion), @@ -186,9 +206,15 @@ Based on the focus type and usecase choosing a `foci_preferred` can be different If possible these guidelines should be obeyed: - If there is a relation between the `focus_active` and a preferred focus (`type: livekit` is an example for this) - it is recommended to copy _the preferred focus that relates to the current `focus_active`_ of other participants to the start of the `foci_preferred` array of the member event. - (The exact definition of: _the preferred focus that relates to the current `focus_active`_ is part of the specification for each focus type. For `full_mesh` for example there is no such thing as: _the preferred focus that relates to the current `focus_active`_ ) -- Homeservers can proposes `preferred_foci` via the well known. An array of preferred foci is provided behind the well known key `m.rtc_foci`. This is defined in [MSC4158](https://github.com/matrix-org/matrix-spec-proposals/pull/4158). They are related and it is recommended to also read [MSC4158](https://github.com/matrix-org/matrix-spec-proposals/pull/4158) with this MSC. + it is recommended to copy _the preferred focus that relates to the current `focus_active`_ of other participants to + the start of the `foci_preferred` array of the member event. + (The exact definition of: _the preferred focus that relates to the current `focus_active`_ is part of the + specification for each focus type. For `full_mesh` for example there is no such thing as: _the preferred focus that + relates to the current `focus_active`_ ) +- Homeservers can proposes `preferred_foci` via the well known. An array of preferred foci is provided behind the well + known key `m.rtc_foci`. This is defined in [MSC4158](https://github.com/matrix-org/matrix-spec-proposals/pull/4158). + They are related and it is recommended to also read + [MSC4158](https://github.com/matrix-org/matrix-spec-proposals/pull/4158)with this MSC. Those proposals from **your own** homeserver should come next in the `foci_preferred` list of the member event. - Clients also have the option to configure a preferred foci even though this is not recommended (see below). Those come last in the list. @@ -207,7 +233,8 @@ The rational for those guidelines are as following: homeserver distribution. For this reason the second guideline is to lookup the prefferred foci from the homeserver well_known - Looking up the prefferred foci from a client is toxic to a federated system. If the majority of users - decide to use the same client all of the users will use one Focus. This destroys the passive security mechanism, that each instance is not an interesting attack vector since it is only a fraction of the network. + decide to use the same client all of the users will use one Focus. This destroys the passive security mechanism, that + each instance is not an interesting attack vector since it is only a fraction of the network. Additionally it will result in poor performance if every user on matrix would use the same Focus. There are cases where this is acceptable: - Transitioning to MatrixRTC. Here it might be beneficial to have a client that has a fallback Focus From a278ff85c6d69f04e8551c6ed0bd8b11ffb7eddb Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 25 Nov 2024 09:04:59 +0000 Subject: [PATCH 12/62] language fixes --- proposals/4143-matrix-rtc.md | 108 +++++++++++++++++------------------ 1 file changed, 54 insertions(+), 54 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 433f0d60468..4fe6fc01a77 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -1,6 +1,6 @@ # MSC4143: MatrixRTC -This MSC defines the modules with which the matrix real time system is build. +This MSC defines the modules with which the MatrixRTC (Matrix Real Time Communication) signalling system is built. The MatrixRTC specification is separated into different modules. @@ -23,14 +23,14 @@ The MatrixRTC specification is separated into different modules. - What streams to connect to. - What data in which format to sent over the RTC channels. -This MSC will focus on the matrix room state which can be seen as the most high +This MSC will focus on the Matrix room state, which can be seen as the most high level signalling of a call: ## Proposal -Each RTC session is made out of a collection of `m.rtc.member` events. +Each RTC session is made out of a collection of `m.rtc.member` state events. Each `m.rtc.member` event defines the application type: `application` -and a `call_id`. And is stored in a state event of type `m.rtc.member`. +and a `call_id`. The first element of the state key is the `userId` and the second the `deviceId`. (see [this proposal for state keys](https://github.com/matrix-org/matrix-spec-proposals/pull/3757#issuecomment-2099010555) for context about second/first state key.) @@ -43,7 +43,7 @@ require one event type. A complete `m.rtc.member` state event looks like this: -```json +```json5 // event type: "m.rtc.member" // event key: "@user:matrix.domain_DEVICEID" { @@ -80,17 +80,17 @@ for an event that, has not yet been updated, there the `origin_server_ts` is use > [!NOTE] > We introduce `created_ts()` as the notation for `created_ts ?? origin_server_ts` -Once the event gets updated the origin_server_ts needs to be copied into the `created_ts` field. +Once the event gets updated, the origin_server_ts needs to be copied into the `created_ts` field. An existing `created_ts` field implies that this is a state event updating the current session and a missing `created_ts` field implies that it is a join state event. All membership events that belong to one member session can be grouped with the index `created_ts()`+`device_id`. This is why the `m.rtc.member` events deliberately do NOT include a `membership_id`. Other then the membership sessions, there is **no event** to represent a rtc session (containing all members). -This event would include shared information where it is not trivial to decide who has authority over it. +Such an event would include shared information, and deciding who has authority over that is not trivial. Instead the session is a computed value based on `m.rtc.member` events. The list of events with the same `application` and `m.call_id` represent one session. -This array allows to compute fields like participant count, start time ... +This array allows to compute fields such as participant count, start time, etc. Sending an empty `m.rtc.member` event represents a leave action. Sending a well formatted `m.rtc.member` represents a join action. @@ -98,45 +98,46 @@ Sending a well formatted `m.rtc.member` represents a join action. Based on the value of `application`, the event might include additional parameters required to provide additional session parameters. -> A thirdRoom like experience could include the information of an approximate position +> A [thirdroom](https://thirdroom.io)-like experience could include the information of an approximate position > on the map, so that clients can omit connecting to participants that are not in their > area of interest. #### Reliability requirements for the room state -Room state is a very well suited place to store the data for a MatrixRTC session -if allows: +Room state is a very well suited place to store the data for a MatrixRTC session, as +it allows: -- The client to determine current ongoing sessions without loading history for every room. - Or doing additional work other then the sync loop that needs to run anyways. +- The client to determine current ongoing sessions without loading history for every room, + or doing additional work other than the sync loop that needs to run anyway. - The client can compute/access data of past sessions without any additional redundant data. - Sessions (start/end/participant count) are federated and there is not redundant data storage that could result in conflicts, or can get out of sync. The room state events are part of the dag and this is solved like any other PDU in matrix. -A chellanging circumstance with using the room state to represent a session is -the disconnection behaviour. If the client disconnects from a call because of a network issue, -an application crash or a user forcefully quitting the client, the room state cannot be updated anymore. +A challenge with using the room state to represent a session is disconnection behaviour. +If the client disconnects from a call because of a network issue, +an application crash, or a user forcefully quitting the client - then the room state cannot be updated any more. The client is required to leave by sending a new empty state which cannot happen once connection is lost. -If the state is not updated correctly we end up with incorrect session end timestamps a room state that is not +If the state is not updated correctly we end up with incorrect session end timestamps, and a room state that is not correctly representing the current RTC session state. Historic and current MatrixRTC session data would be broken. For an acceptable solution, the following requirements need to be taken into consideration: -- Room state is set to empty if the client looses connection. (A heardbeat like system is desired) +- Room state is set to empty if the client loses connection. (A heardbeat like system is desired) - The best source of truth for a call participation is a working connection to the SFU. It is desired that the disconnect of the SFU is connected to the room state. -- It should be possible to updated the room state without the client being online. -- All this should be compatible when matrix uses cryptographic identities. +- It should be possible to update the room state without the client being online. +- All of this should still work when Matrix uses cryptographic identities (e.g. + [MSC4080](https://github.com/matrix-org/matrix-spec-proposals/pull/4080)). -[MSC4340](https://github.com/matrix-org/matrix-spec-proposals/pull/4140) proposes a concept to +[MSC4140](https://github.com/matrix-org/matrix-spec-proposals/pull/4140) proposes a concept to delay the leave events until one of the leave conditions (heartbeat or SFU disconnect) occur and fulfil all of the these requirements. -A matrixRTC client has to first send/schedule the following delayed leave event: +A MatrixRTC client has to first send/schedule the following delayed leave event: -```json +```json5 // event type: "m.rtc.member" // event key: "@user:matrix.domain_DEVICEID" { @@ -144,19 +145,19 @@ A matrixRTC client has to first send/schedule the following delayed leave event: } ``` -only after that the actual state event can be sent, so that we guarantee that the state will be empty eventually. +Subsequently, the actual state event can be sent, so that we guarantee that the state will be empty eventually. The `leave_reason` is added so clients can be more verbal about why a user disconnected from a call. -Receiving clients will be able to detect if this order was not followed with the `has_delayed_overwrite: true` +Receiving clients will be able to detect if the delayed event request was recognised by the presence of the `has_delayed_overwrite: true` unsigned property. If the property is missing the event is invalid. -This also invalides delayed leave events that are send with a valid membership content. They do not contain the +This also invalidates delayed leave events that are send with a valid membership content. They do not contain the `has_delayed_overwrite: true` unsigned property. #### Historic sessions -Since there is no singe entry for a historic session (because of the owner ship discussion), -historic sessions need to be computed and most likely cached on the client. +Since there is no single entry for a historic session (because of the ownership ambiguity), +historic sessions need to be computed on the client. Each state event can either mark a join or leave: @@ -168,14 +169,14 @@ Each state event can either mark a join or leave: `prev_state.m.call_id != current_state.m.call_id` && `current_state.application == undefined` -Based on this one can find user sessions. (The range between a join and a leave -event) of specific times. +Based on this one can find user sessions. The range between a join and a leave +event gives the specific times and duration of the session. The collection of all overlapping user sessions with the same `call_id` and `application` define one MatrixRTC history event. ### The RTC backend -`foci_active` and `foci_preferred` are used to communicate +`foci_active` and `foci_preferred` are used to communicate: - how a user is connected to the session (`foci_active`) - what connection method this user knows about would like to connect with. @@ -190,9 +191,8 @@ with each other. Only users with the same type can connect in one session. If a frontend does not support the used type they cannot connect. -Each focus type will get its own MSC in which the detailed procedure to get from -the foci information to working webRTC connections to the streams of all the -participants is explained. +Each focus type will get its own MSC, describing how to get from the foci +information to establishing WebRTC connections for all participants. - [`livekit`](www.example.com) TODO: create `livekit` focus MSC and add link here. - [`full_mesh`](https://github.com/matrix-org/matrix-spec-proposals/pull/3401) @@ -202,7 +202,7 @@ participants is explained. #### Sourcing `foci_preferred` At some point participants have to decide/propose which focus they use. -Based on the focus type and usecase choosing a `foci_preferred` can be different. +Based on the focus type and use case choosing a `foci_preferred` can be different. If possible these guidelines should be obeyed: - If there is a relation between the `focus_active` and a preferred focus (`type: livekit` is an example for this) @@ -214,46 +214,46 @@ If possible these guidelines should be obeyed: - Homeservers can proposes `preferred_foci` via the well known. An array of preferred foci is provided behind the well known key `m.rtc_foci`. This is defined in [MSC4158](https://github.com/matrix-org/matrix-spec-proposals/pull/4158). They are related and it is recommended to also read - [MSC4158](https://github.com/matrix-org/matrix-spec-proposals/pull/4158)with this MSC. + [MSC4158](https://github.com/matrix-org/matrix-spec-proposals/pull/4158) with this MSC. Those proposals from **your own** homeserver should come next in the `foci_preferred` list of the member event. - Clients also have the option to configure a preferred foci even though this is not recommended (see below). Those come last in the list. -The rational for those guidelines are as following: +The rationale for these guidelines are: -- It is always desired to have as little focus switches as possible. - That is why the highest priority is to prefer the focus that is already in use -- MatrixRTC is designed around the same culture that makes matrix possible: - A large amount of infrastructure in form of homeservers is provided by the users. - For MatrixRTC the same is thea goal. To achieve a stable and healthy ecosystem - rtc infrastructure should be thought of as a part of a homeserver. It is very similar +- It is always desired to have as few focus switches as possible. + That is why the highest priority is to prefer the focus that is already in use. +- MatrixRTC is designed around the same architecture as the rest of Matrix, with + conversations being powered by many homeservers from across the network. + MatrixRTC has the same goal. To achieve a stable and healthy ecosystem + RTC infrastructure should be thought of as a part of a homeserver. It is very similar to a turn server: mostly traffic and little cpu load. To not end up in a world where each user is only using one central SFU but where the traffic - is split over multiple SFU's it is important that we leverage the SFU distribution on the - homeserver distribution. - For this reason the second guideline is to lookup the prefferred foci from the homeserver well_known -- Looking up the prefferred foci from a client is toxic to a federated system. If the majority of users - decide to use the same client all of the users will use one Focus. This destroys the passive security mechanism, that + is split over multiple SFU's it is important that we leverage the SFU distribution similarly to the + distribution of homeservers. + For this reason the second guideline is to lookup the preferred foci from the homeserver's well_known. +- Looking up the preferred foci from a client is toxic to a federated system. If the majority of users + decide to use the same client all of the users will use one focus. This destroys the passive security mechanism that each instance is not an interesting attack vector since it is only a fraction of the network. - Additionally it will result in poor performance if every user on matrix would use the same Focus. + Additionally it will result in poor performance if every user on Matrix would use the same focus. There are cases where this is acceptable: - - Transitioning to MatrixRTC. Here it might be beneficial to have a client that has a fallback Focus + - Transitioning to MatrixRTC. Here it might be beneficial to have a client that has a fallback focus so calls also work with homeservers not supporting it. - - For testing purposes where a different Focus should be tested but one does not want to touch the .well_known + - For testing purposes where a different focus should be tested but one does not want to touch the .well_known - For custom deployments that benefit from having the Focus configuration on a per client basis instead of per homeserver. ### The RTC Session types (application) -Each session type might have its own specification in how the different streams +Each session type can have its own specification in how the different streams are interpreted and even what focus type to use. This makes this proposal extremely -flexible. A Jitsi conference could be added by introducing a new `application` +flexible. For instance, a Jitsi conference could be added by introducing a new `application` and a new focus type and would be MatrixRTC compatible. It would not be compatible with applications that do not use the Jitsi focus but clients would know that there is an ongoing session of unknown type and unknown focus and could display/represent this in the user interface. To make it easy for clients to support different RTC session types, the recommended -approach is to provide a matrix widget for each session type, so that client developers +approach is to provide a Matrix widget for each session type, so that client developers can use the widget as the first implementation if they want to support this RTC session type. From f97e4e64cd2be6c46fd582ce09dd5e35d90cc814 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 25 Nov 2024 09:48:06 +0000 Subject: [PATCH 13/62] clarify that non-empty delayed membership events are invalid --- proposals/4143-matrix-rtc.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 4fe6fc01a77..5b950c5fc7a 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -151,8 +151,8 @@ The `leave_reason` is added so clients can be more verbal about why a user disco Receiving clients will be able to detect if the delayed event request was recognised by the presence of the `has_delayed_overwrite: true` unsigned property. If the property is missing the event is invalid. -This also invalidates delayed leave events that are send with a valid membership content. They do not contain the -`has_delayed_overwrite: true` unsigned property. +This also ensures that delayed leave events that are incorrectly sent with a non-empty membership content are invalidated, +as they will not contain the `has_delayed_overwrite: true` unsigned property. #### Historic sessions From b626d512f07a3d7a65351515c40d11010a4f8927 Mon Sep 17 00:00:00 2001 From: Hugh Nimmo-Smith Date: Tue, 17 Dec 2024 07:10:42 +0000 Subject: [PATCH 14/62] Latest --- proposals/4143-matrix-rtc.md | 668 ++++++++++++++++++++++++++--------- 1 file changed, 503 insertions(+), 165 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 5b950c5fc7a..293a206ff3b 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -1,57 +1,101 @@ # MSC4143: MatrixRTC -This MSC defines the modules with which the MatrixRTC (Matrix Real Time Communication) signalling system is built. +MatrixRTC is short for Matrix real time communication. +This MSC defines the modules with which the Matrix real time system is built. -The MatrixRTC specification is separated into different modules. +MatrixRTC specifies how a real time session is described in a room and how matrix users can connect to +a session. -- The MatrixRTC room state that defines the state of the real time application.\ +The MatrixRTC specification is separated into different modules: + + +- The MatrixRTC room state that defines the state of the real time session.\ It is the source of truth for: - Who is part of a session - Who is connected via what technology/backend - Metadata per device used by other participants to decide whether the streams from this source are of interest / need to be subscribed. -- The RTC backend. +- The MatrixRTC backend. + - Allows for multiple backend implementations to be used. + - It defines how to discover the available backend(s). - It defines how to connect the participating peers. - - Livekit is the standard for this as of writing. - Defines how to connect to a server/other peers, how to update the connection, how to subscribe to different streams... - - Another planned backend is a full mesh implementation based on MSC3401. -- The RTCSession types (application) have their own per application spec. - - Calls can be done with an application of type `m.call` see (TODO: link call msc) + - A proposal utilising LiveKit is the standard for this as of writing. + - Another planned backend is a full mesh implementation based on [MSC3401](https://github.com/matrix-org/matrix-spec-proposals/pull/3401). +- The MatrixRTC application. + - Each application type can have it's own spec. + - Voice and video conferencing can be done with an application of type `m.call` - The application defines all the details of the RTC experience: - How to interpret the metadata of the member events. - What streams to connect to. - What data in which format to sent over the RTC channels. + - What MatrixRTC backends are supported. +- End-to-end encryption of media streams -This MSC will focus on the Matrix room state, which can be seen as the most high -level signalling of a call: +This MSC will focus on the Matrix room state which is responsible for the high +level signalling of a RTC session: ## Proposal -Each RTC session is made out of a collection of `m.rtc.member` state events. -Each `m.rtc.member` event defines the application type: `application` -and a `call_id`. -The first element of the state key is the `userId` and the second the `deviceId`. -(see [this proposal for state keys](https://github.com/matrix-org/matrix-spec-proposals/pull/3757#issuecomment-2099010555) -for context about second/first state key.) +Each RTC session is made out of a collection of `m.rtc.member` room state events. +Each `m.rtc.member` event defines who (the `member`) is a participant of which session (the `session`). ### The MatrixRTC room state -Everything required for working MatrixRTC +All data related to a MatrixRTC session (current session, sessions history, join/leave events, ...) only -require one event type. +requires one event type. -A complete `m.rtc.member` state event looks like this: +(current session, sessions history, join/leave events, ...) only +require one event type:. -```json5 +We use a set of `m.rtc.member` (one for each participant) state events to represent a session. + +based on the content a `m.rtc.member` state event can either represent a connected or a disconnected member. + +#### Joining a session + +Sending a well-formed `m.rtc.member` event that describes a connected state for a state key that is not yet used or contains a disconnected `m.rtc.member` event represents a join action. + +The fields are as follows: + +- `member` required object - describes the participant of the RTC session: + - `id` required string - a unique identifier for this session membership as defined above. Recommended to be a UUID. It can be reused if the user leaves and rejoins the session. + It should be unique across all devices of the user. TODO: define grammar + - `device_id` required string - the Matrix device ID of the device that is joining the session. This is used when sending + [to-device messages](https://spec.matrix.org/v1.11/client-server-api/#send-to-device-messaging). + - `user_id` required string - the Matrix user ID of the user that is joining the session. This is needed as we cannot rely + on the owner of state event as it might have been modified by an admin or similar. +- `session` required object - an object that is used to uniquely identify this session across RTC member events +of the Matrix room: + - `application` required string - a recognised application type. e.g. `m.call` as linked below + - additional fields as defined by the application type +- `created_ts` - timestamp in milliseconds since UNIX epoch. + - this should **not** be present the first time that the `m.rtc.member` event is sent. + - if the `m.rtc.member` event is sent again, the `created_ts` should be populated with the `origin_server_ts` + that was given to the previous version of the state event. +- `focus_active` required Focus object - specifies the algorithm that defines how to choose a Focus for this member. See below for details. +- `foci_preferred` required array of Focus objects - specifies the input data for this algorithm contributed by this member. See below for details. + +Additional fields may be added depending on the application type. + +A full `m.rtc.member` state event for a joined member looks like this: + +```json // event type: "m.rtc.member" -// event key: "@user:matrix.domain_DEVICEID" +// state key: see next section for definition { - "application": "m.my_session_type", - "call_id": "", - "device_id": "DEVICEID", + "session": { + "application": "m.call" + // further fields for the application + }, + "member": { + "id": "xyzABCDEF10123", + "device_id": "DEVICEID", + "user_id": "@user:matrix.domain" + }, "created_ts": Time | undefined, - "expires_after": Duration, "focus_active": {...FOCUS_A}, "foci_preferred": [ {...FOCUS_1}, @@ -60,19 +104,9 @@ A complete `m.rtc.member` state event looks like this: } ``` -> [!NOTE] -> This relies on [MSC3757](https://github.com/matrix-org/matrix-spec-proposals/pull/3757). -> We need to have one state event per device, hence multiple "non-overwritable" state -> events per user. - -This gives us the information, that user: `@user:matrix.domain` with device `DEVICEID` -is part of an RTCSession of type `m.call` in the scope/sub-session `""` (empty -string as call id) connected over `FOCUS_A`. This is all information that is needed -for another room member to detect the running session and join it. - -We include the device_id in the member content to not rely on the exact format of the state key. -In case [MSC3757](https://github.com/matrix-org/matrix-spec-proposals/pull/3757) is used it would not -be the second element of the state key array. +This gives us the information, that user: `@user:matrix.domain` with member ID `DEVICEID_m:call_123456789` +is part of a session identified by `{}` using application of type `m.call` connected over `FOCUS_A`. +This is sufficient information for another room member to detect the running session and join it. `created_ts` is an optional property that caches the time of creation. It is not required for an event that, has not yet been updated, there the `origin_server_ts` is used. @@ -80,56 +114,120 @@ for an event that, has not yet been updated, there the `origin_server_ts` is use > [!NOTE] > We introduce `created_ts()` as the notation for `created_ts ?? origin_server_ts` -Once the event gets updated, the origin_server_ts needs to be copied into the `created_ts` field. +Once the event gets updated the `origin_server_ts` needs to be copied into the `created_ts` field. An existing `created_ts` field implies that this is a state event updating the current session and a missing `created_ts` field implies that it is a join state event. All membership events that belong to one member session can be grouped with the index -`created_ts()`+`device_id`. This is why the `m.rtc.member` events deliberately do NOT include a `membership_id`. +`created_ts()`+`state_key`. This is why the `m.rtc.member` events deliberately do NOT include something akin to a `membership_id`. -Other then the membership sessions, there is **no event** to represent a rtc session (containing all members). -Such an event would include shared information, and deciding who has authority over that is not trivial. +Other then the membership sessions, there is **no event** to represent a RTC session (containing all members). +This event would include shared information where it is not trivial to decide who has authority over it. Instead the session is a computed value based on `m.rtc.member` events. -The list of events with the same `application` and `m.call_id` represent one session. -This array allows to compute fields such as participant count, start time, etc. - -Sending an empty `m.rtc.member` event represents a leave action. -Sending a well formatted `m.rtc.member` represents a join action. +The list of events with the same `session` content represent one session. +This array allows to compute fields like participant count, start time etc. Based on the value of `application`, the event might include additional parameters -required to provide additional session parameters. +to provide additional session parameters. -> A [thirdroom](https://thirdroom.io)-like experience could include the information of an approximate position +> A [Third Room](https://thirdroom.io) like experience could include the information of an approximate position > on the map, so that clients can omit connecting to participants that are not in their > area of interest. +#### State key for `m.rtc.member` + +The state key is generated from the `member` field of the `m.rtc.member` event. + +We want to choose a state key that is compatible with whichever state protection proposal is accepted to ensure that +users cannot modify one another's sessions. + +For [MSC3757](https://github.com/matrix-org/matrix-spec-proposals/pull/3757) we generate the state key by +concatenating the following strings: + +- the Matrix ID of the user +- an `_` (underscore) +- the `member`.`id` field + +For example with a `member`.`id` of `xyzABCDEF10123` for user `@user:matrix.domain` the state key would be `@user:matrix.domain_xyzABCDEF10123`. + +For a client parsing the state key we would treat anything before the first `_` as the Matrix ID of the user +and anything after as the `member`.`id` field. + +#### Leaving a session + +Sending an empty `m.rtc.member` event represents a leave action. The state key must be the same as boefore + +There is an optional `leave_reason` field that can be used to provide a reason for leaving the session: + +- `leave_reason` optional string - one of: `lost_connection` + +An example of leaving a session where the user explicitly disconnects: + +```json +// event type: "m.rtc.member" +// state key: "@user:matrix.domain_xyzABCDEF10123" +{ +} +``` + +The client should use the `prev_content` field of the [room state event](https://spec.matrix.org/v1.11/client-server-api/#room-event-format) +to determine the details of the leave event. + +For example: + +```json +// event type: "m.rtc.member" +// state key: "@user:matrix.domain_xyzABCDEF10123" +{ + "content": { + "leave_reason": "lost_connection" + }, + "prev_content": { + "session": { + "application": "m.call", + "call_id": "" + }, + "member": { + "id": "xyzABCDEF10123", + "device_id": "DEVICEID", + "user_id": "@user:matrix.domain" + }, + "created_ts": 123456, + "focus_active": {...FOCUS_A}, + "foci_preferred": [ + {...FOCUS_1}, + {...FOCUS_2} + ] + } +} +``` + #### Reliability requirements for the room state -Room state is a very well suited place to store the data for a MatrixRTC session, as -it allows: +Room state is a very well suited place to store the data for a MatrixRTC session. +It allows: -- The client to determine current ongoing sessions without loading history for every room, - or doing additional work other than the sync loop that needs to run anyway. +- The client to determine current ongoing sessions without loading history for every room. + Or doing additional work other then the sync loop that needs to run anyways. - The client can compute/access data of past sessions without any additional redundant data. - Sessions (start/end/participant count) are federated and there is not redundant data storage that - could result in conflicts, or can get out of sync. The room state events are part of the dag and this - is solved like any other PDU in matrix. + could result in conflicts, or can get out of sync. The room state events are part of the DAG and this + is solved like any other Persistent Data Unit (PDU) in Matrix. -A challenge with using the room state to represent a session is disconnection behaviour. -If the client disconnects from a call because of a network issue, -an application crash, or a user forcefully quitting the client - then the room state cannot be updated any more. +However, a challenging circumstance with using the room state to represent a session is +the disconnection behaviour. If the client disconnects from a call because of a network issue, +an application crash or a user forcefully quitting the client, the room state cannot be updated anymore. The client is required to leave by sending a new empty state which cannot happen once connection is lost. -If the state is not updated correctly we end up with incorrect session end timestamps, and a room state that is not +If the state is not updated correctly we end up with a room state that is not correctly representing the current RTC session state. Historic and current MatrixRTC session data would be broken. For an acceptable solution, the following requirements need to be taken into consideration: -- Room state is set to empty if the client loses connection. (A heardbeat like system is desired) +- Room state is set to empty if the client looses connection. (A heartbeat like system is desired) - The best source of truth for a call participation is a working connection to the SFU. - It is desired that the disconnect of the SFU is connected to the room state. -- It should be possible to update the room state without the client being online. -- All of this should still work when Matrix uses cryptographic identities (e.g. - [MSC4080](https://github.com/matrix-org/matrix-spec-proposals/pull/4080)). + It is desired that the disconnect of the member on the SFU gets propagated to the room state. +- It should be possible to updated the room state without the client being online. +- All this should be compatible when Matrix uses cryptographic identities. [MSC4140](https://github.com/matrix-org/matrix-spec-proposals/pull/4140) proposes a concept to delay the leave events until one of the leave conditions (heartbeat or SFU disconnect) occur @@ -137,142 +235,382 @@ and fulfil all of the these requirements. A MatrixRTC client has to first send/schedule the following delayed leave event: -```json5 +```json // event type: "m.rtc.member" -// event key: "@user:matrix.domain_DEVICEID" +// state key: "@user:matrix.domain_xyzABCDEF10123" { - "leave_reason": "CONNECTION_LOST" + "leave_reason": "lost_connection" } ``` -Subsequently, the actual state event can be sent, so that we guarantee that the state will be empty eventually. +only after that the actual state event can be sent, so that we guarantee that the state will be empty eventually. The `leave_reason` is added so clients can be more verbal about why a user disconnected from a call. +It allows to communicate with other participants in a session if the user has disconnected intentionally or lost connection. -Receiving clients will be able to detect if the delayed event request was recognised by the presence of the `has_delayed_overwrite: true` -unsigned property. If the property is missing the event is invalid. - -This also ensures that delayed leave events that are incorrectly sent with a non-empty membership content are invalidated, -as they will not contain the `has_delayed_overwrite: true` unsigned property. - -#### Historic sessions +#### Session history -Since there is no single entry for a historic session (because of the ownership ambiguity), -historic sessions need to be computed on the client. +Since there is no single entry for a historic session (because of the ownership discussion), +historic sessions need to be computed and most likely cached on the client. Each state event can either mark a join or leave: -- join: `prev_state.application != current_state.application` && - `prev_state.m.call_id != current_state.m.call_id` && - `current_state.application != undefined` - (where an empty `m.rtc.member` event would imply `state.application == undefined`) -- leave: `prev_state.application != current_state.application` && - `prev_state.m.call_id != current_state.m.call_id` && - `current_state.application == undefined` +- join: `prev_state.session != current_state.session` && + `current_state.session != undefined` + (where an empty `m.rtc.member` event would imply `state.session == undefined`) +- leave: `prev_state.session != current_state.session` && + `current_state.session == undefined` -Based on this one can find user sessions. The range between a join and a leave -event gives the specific times and duration of the session. -The collection of all overlapping user sessions with the same `call_id` and -`application` define one MatrixRTC history event. +Based on this one can find user sessions. (The range between a join and a leave +event) of specific times. +The collection of all overlapping user sessions with the same `session` contents +define one MatrixRTC history event. ### The RTC backend -`foci_active` and `foci_preferred` are used to communicate: +Backend **infrastructure** in this context can be anything that can serve as the backend for a +MatrixRTC session. In most cases this is a SFU. But also a full mesh implementation could +be an infrastructure. Not all kind of infrastructure require a way of sourcing a backend resource +(e.g. full-mesh). In this MSC we only refer to infrastructure where it is necessary to have access to additional +data to participate in the MatrixRTC session. -- how a user is connected to the session (`foci_active`) -- what connection method this user knows about would like to connect with. +The backend is referred to as a Focus or as Foci in plural. -The only enforced parameter of a `foci_preferred` or `foci_active` is `type`. -Based on the focus type a different amount of parameters might be needed to, -communicate how to connect to other users. -`foci_preferred` and `foci_active` can have different parameters so that it is, -possible to use a combination of the two to figure our that everyone is connected -with each other. +Note that these backends are independent of the application (e.g. `m.call`) being used in the session. + +A Focus is represented as a JSON object with one mandatory field: + +- `type` required string: The type of the Focus as defined by an RTC backend.. + +Additional fields will be present depending on `type`. Only users with the same type can connect in one session. If a frontend does not support the used type they cannot connect. -Each focus type will get its own MSC, describing how to get from the foci -information to establishing WebRTC connections for all participants. - -- [`livekit`](www.example.com) TODO: create `livekit` focus MSC and add link here. -- [`full_mesh`](https://github.com/matrix-org/matrix-spec-proposals/pull/3401) - TODO: create `full-mesh` focus MSC based on[MSC3401](https://github.com/matrix-org/matrix-spec-proposals/pull/3401) - and add link here. - -#### Sourcing `foci_preferred` - -At some point participants have to decide/propose which focus they use. -Based on the focus type and use case choosing a `foci_preferred` can be different. -If possible these guidelines should be obeyed: - -- If there is a relation between the `focus_active` and a preferred focus (`type: livekit` is an example for this) - it is recommended to copy _the preferred focus that relates to the current `focus_active`_ of other participants to - the start of the `foci_preferred` array of the member event. - (The exact definition of: _the preferred focus that relates to the current `focus_active`_ is part of the - specification for each focus type. For `full_mesh` for example there is no such thing as: _the preferred focus that - relates to the current `focus_active`_ ) -- Homeservers can proposes `preferred_foci` via the well known. An array of preferred foci is provided behind the well - known key `m.rtc_foci`. This is defined in [MSC4158](https://github.com/matrix-org/matrix-spec-proposals/pull/4158). - They are related and it is recommended to also read - [MSC4158](https://github.com/matrix-org/matrix-spec-proposals/pull/4158) with this MSC. - Those proposals from **your own** homeserver should come next in the `foci_preferred` list of the member event. -- Clients also have the option to configure a preferred foci even though this is not recommended (see below). - Those come last in the list. - -The rationale for these guidelines are: - -- It is always desired to have as few focus switches as possible. - That is why the highest priority is to prefer the focus that is already in use. -- MatrixRTC is designed around the same architecture as the rest of Matrix, with - conversations being powered by many homeservers from across the network. - MatrixRTC has the same goal. To achieve a stable and healthy ecosystem - RTC infrastructure should be thought of as a part of a homeserver. It is very similar - to a turn server: mostly traffic and little cpu load. - To not end up in a world where each user is only using one central SFU but where the traffic - is split over multiple SFU's it is important that we leverage the SFU distribution similarly to the - distribution of homeservers. - For this reason the second guideline is to lookup the preferred foci from the homeserver's well_known. -- Looking up the preferred foci from a client is toxic to a federated system. If the majority of users - decide to use the same client all of the users will use one focus. This destroys the passive security mechanism that - each instance is not an interesting attack vector since it is only a fraction of the network. - Additionally it will result in poor performance if every user on Matrix would use the same focus. - There are cases where this is acceptable: - - Transitioning to MatrixRTC. Here it might be beneficial to have a client that has a fallback focus - so calls also work with homeservers not supporting it. - - For testing purposes where a different focus should be tested but one does not want to touch the .well_known - - For custom deployments that benefit from having the Focus configuration on a per client basis instead of per homeserver. - -### The RTC Session types (application) - -Each session type can have its own specification in how the different streams -are interpreted and even what focus type to use. This makes this proposal extremely -flexible. For instance, a Jitsi conference could be added by introducing a new `application` -and a new focus type and would be MatrixRTC compatible. It would not be compatible -with applications that do not use the Jitsi focus but clients would know that there -is an ongoing session of unknown type and unknown focus and could display/represent +Each Focus type will get its own MSC in which the detailed procedure to get from +the foci information to working WebRTC connections to the streams of all the +participants is explained. + +Foci are represented in three places: + +- `focus_active` of `m.rtc.member` state event - specifies the algorithm that defines how to choose a Focus for this member. +- `foci_preferred` of `m.rtc.member` state event- specifies the input data for this algorithm contributed by this member. +- `m.rtc_foci` of the `.well-known/matrix/client` - specifies the list of available Foci for the homeserver. + +The `focus_active` algorithm needs to be designed so that all participants converge to the same SFU/Focus. + +The following Focus `type` values are defined: + +- `livekit` - a backend using the [LiveKit](https://livekit.io/) SFU as described in + [MSC4195](https://github.com/matrix-org/matrix-spec-proposals/pull/4195). +- `full_mesh` - a backend using a full-mesh approach based on [MSC3401](https://github.com/matrix-org/matrix-spec-proposals/pull/3401). + +#### Choosing the value of `foci_preferred` for the `m.rtc.member` state event + +At some point session participants have to decide/propose which Focus they will use. + +Based on the Focus type and application choosing the method by which the contents of the `foci_preferred` field on the `m.rtc.member` +can be different. + +There are three guidelines which should be obeyed by a client when building the `foci_preferred` list: + +1. It is always desired to have as few Focus switches as possible. + +If there are other participants on the session (i.e. other `m.rtc.member` events) the client should calculate what the Focus it should connect to +based on the `m.rtc.member` events for the existing participants. +This should happen reactively on each `m.rtc.member` state event change. +Each MatrixRTC frontend is responsible that it can deal with focus switches based on changing state gracefully. It is part of the design of MatrixRTC and a requirement for a eventually consistent distributed system. + +The calculated Focus should then be present at the start of the `foci_preferred` list. + +2. The client should lookup the suggested foci from the homeserver `.well-known/matrix/client` as defined below. + +MatrixRTC is designed around the same culture that makes Matrix possible: A large amount of infrastructure in the form of homeservers is provided by the users. + +To achieve a stable and healthy ecosystem backend RTC infrastructure should be thought of as a part of a homeserver. + +It is very similar to a TURN server: mostly traffic and little CPU load. + +To not end up in a world where each user is only using one central SFU but where the traffic +is split over multiple SFU's it is important that we leverage the SFU distribution on the +homeserver federation. + +These proposals from **your own** homeserver should come next in the `foci_preferred` list of the member event. + +3. Clients should not use a hard-coded Focus. + +Looking up the preferred Foci from a client is toxic to a federated system. If the majority of users +decide to use the same client all of the users will use one Focus. This destroys the passive security mechanism, that +each instance is not an interesting attack vector since it is only a fraction of the network. +Additionally it will result in poor performance if every user on Matrix would use the same Focus. + +However, there are cases where this is acceptable: + +- Transitioning to MatrixRTC. Here it might be beneficial to have a client that has a fallback Focus + so calls also work with homeservers not supporting it. +- For testing purposes where a different Focus should be tested but one does not want to touch the .well-known +- For custom deployments that benefit from having the Focus configuration on a per client basis instead of per homeserver. + +Therefore, if a client does use a hard-coded Focus it should come last in the `foci_preferred` list. + +#### Discovery of Foci using `.well-known/matrix/client` + +> [!NOTE] +> Backend **infrastructure** in this context can be anything that can serve as the backend for a +> MatrixRTC session. In most cases this is a SFU. But also a full mesh implementation could +> be an infrastructure. Not all kind of infrastructure require a way of sourcing a backend resource +> (e.g. full-mesh). In this MSC we only refer to infrastructure where it is necessary to have access to additional +> data to participate in the MatrixRTC session. + +We use a `m.rtc_foci` key in the homeserver `.well-known/matrix/client` that can be used to expose +a sorted (by priority) list of Focus description objects. + +For example in generic form: + +```json +{ + "m.rtc_foci": [ + { + "type": "some-focus-type", + "additional-type-specific-field": "https://my_focus.domain", + "another-additional-type-specific-field": ["with", "Array", "type"] + } + ] +} +``` + +Or a concrete example for a `livekit` Focus: + +```json +{ + "m.rtc_foci": [ + { + "type":"livekit", + "livekit_service_url":"https://livekit-jwt.call.element.io" + } + ] +} +``` + +### The RTC application types + +Each application type might have its own specification in how the different streams +are interpreted and even what Focus type to use. This makes this proposal extremely +flexible. A Jitsi conference could be added by introducing a new `application` +and a new Focus type and would be MatrixRTC compatible. It would not be compatible +with applications that do not use the Jitsi Focus but clients would know that there +is an ongoing session of unknown type and unknown Focus and could display/represent this in the user interface. -To make it easy for clients to support different RTC session types, the recommended -approach is to provide a Matrix widget for each session type, so that client developers -can use the widget as the first implementation if they want to support this RTC -session type. +To make it easy for clients to support different application types, the recommended +approach is to provide a Matrix widget for each application type. This way the +client developers can use the widget as the first implementation if they want to +support this RTC application type. Each application should get its own MSC in which the all the additional fields are explained and how the communication with the possible foci is defined: -- [`m.call`](www.example.com) TODO: create `m.call` MSC and add link here. +- `m.call` - voice and video conferencing described by [MSC4196](https://github.com/matrix-org/matrix-spec-proposals/pull/4196). + +#### Interoperability between applications + +There is a use-case in which a `m.call` app might want to participate in a session of type (application) `custom-call-with-more-features`. A native mobile matrix client might support `m.call` and is at hand to join the feature rich application/session. + +There could be fallback mechanisms but the most flexible approach is to treat it per application type. If it makes sense for an application type to fully conform to `m.call` a client that can connect to an `m.call` RTC session (application) could claim that it is also compatible with `custom-call-with-more-features` . It is than the job of the `custom-call-with-more-features` session type (application) to define some kind of feature list so that it can tell if users are joining with an m.call client or a dedicated `custom-call-with-more-features` client. +### End-to-end encryption of media streams + +We define how the key material is shared between the participants of the call to facilitate end-to-end encryption of the media streams. + +The backend (e.g. LiveKit) MSC defines how the key material is actually used. + +#### Shared password + +A shared password may be used to encrypt the media streams sent via the RTC backend that has been distributed ahead of time to the participants. + +For example, it could be in the query parameter of a private URL attached to a calendar invitation. + +#### Per-participant sender key + +A participant can share it's chosen key with other participants by sending Matrix [to-device messaging](https://spec.matrix.org/v1.11/client-server-api/#send-to-device-messaging) to the other participants. + +The key is sent as an event of type `m.rtc.encryption_keys` as an encrypted to-device message. + +The device ID that is being sent to is the `member`.`device_id` from the `m.rtc.member` events. + +The event contains the following fields: + +- `session` required object: The contents of the `session` from the `m.rtc.member` event. +- `member` required object: The contents of the `member` from the corresponding `m.rtc.member` event. +- `keys` required array of objects: The sender keys to be distributed to the participant: + - `key` required string: The base64 encoded key material. + - `index` required int: The index of the key to distinguish it from other keys. This must be a between 0 and 255 inclusive. + In some implementations of MatrixRTC this may correspond to the `keyID` field of the WebRTC [SFrame](https://www.w3.org/TR/webrtc-encoded-transform/#sframe) header. + - `invalidates_key_index` optional int: The index of the key that is invalidated by this key. If this is set, the application should invalidate the key identified + by `invalidates_key_index` once it receives a frame with the new `index`. This is to protect against an exfiltrated key being used to forge frames. + - `invalidates_after_ms` optional int: The number of milliseconds after the key identified by `invalidates_key_index` is invalidated by this key even if no frames + are received. Again, this is to protect against an exfiltrated key being used to forge frames. + +Depending on the RTC application, additional fields may be added to this event. + +An example to-device event: + +```json5 +// event type: "m.rtc.encryption_keys" +{ + "session": { + "application": "m.call", + "call_id": "", + "scope": "m.room" + }, + "member": { + "id": "xyzABCDEF10123", + "device_id": "DEVICEID", + "user_id": "@user:matrix.domain" + }, + "room_id": "!roomid:matrix.domain", + "keys": [ + { + "index": 10, + "key": "base64encodedkey", + "invalidates_key_index": 9, + "invalidates_after_ms": 5000 + }, + ], +} +``` + +On receipt of the `m.rtc.encryption_keys` event the application can associate the received key with the RTC session by matching the `session` and `member` contents with the corresponding `m.rtc.member` event. + +When the application joins the session it should send the key to all the existing participants. + +To ensure forward secrecy and post compromise security, the key material should be rotated (i.e. a new key generated) when a participant joins or leaves the session. + +Key rotation is done as follows: + +- the sending application generates the new key material for the participant. +- the sending application sends the new key material to all the participants with a new `index` value and `invalidates_key_index` set to the current `index`. +- the receiving application stores the new key material for the specified `index`. +- the sending application continues to use the old/current key to encrypt media. +- the sending application waits for a period of time. The default should be 3 seconds. + It is possible to overwrite this on a per application basis in case an application has specific requirements on security or wants to minimize missed stream data. + Also negotiation approaches can be defined where the RTC application uses data channels to communicate if everyone has received the next key. +- the sending application starts to use the new key to encrypt media. +- the receiving application invalidates the existing key with the `invalidates_key_index` value. + +### Discovery/negotiation of application types + +Problem: If a user wants to make a call to a user or room, then which call/application options should the client present to the user? + +This should also take account of non-MatrixRTC calling: legacy 1:1 VoIP, room state widget for Jitsi. + +TODO: write up notes. ## Potential issues ## Alternatives +### One state event per user + +[MSC3401](https://github.com/matrix-org/matrix-spec-proposals/pull/3401) proposed to have one state event per user with that state event containing an array of memberships. + +This introduces two problems: + +- potential inconsistency where one user device overwrites the state of another device during a concurrent update. +- when handling client disconnects the MSC3757 proposal could not be used as you would not know what the correct + state is at the time of the disconnect. + +### One state event per device + +This would mean not using `member`.`id` in the state key anymore. Race conditions can be solved by the client which would need to manage multiple sessions at once. + +### A separate system not associated with Matrix accounts + +This MSC proposes to combine the MatrixRTC backend infrastructure with the homeserver. +Other sources where the backend could be sourced from are: + +- A separate system not associated with Matrix accounts. + (you would need a Matrix account + a "LiveKit provider" account for example) +- The client could bring its own backend link. +- A centralized solution. + +The centralized solution would not fit to Matrix. A separate system would match the distributed +nature of Matrix but would not match the user experience goals for MatrixRTC calls. + +The client defining the SFU that is used, is the current solution. This causes the issue, that clients +in general are less distributed than homeservers. There is only a limited set of clients that a large +percentage of users use. +Using this as the source for the infrastructure would result in just a handful of very large infrastructure +hosts. +This is harder to scale and it is harder to justify who is covering the costs. (For Matrix homeservers, this +is an already solved problem where there are individuals, communities and institutions that have their own individual +solutions and answers for how and why they provide the infrastructure.) + +### `m.rtc.encryption_keys` room event + +Earlier iterations of this MSC used an encrypted `m.rtc.encryption_keys` room event to distribute the per-participant sender keys. + +Whilst reducing traffic by only needing to send one event per participant, this approach does not allow for perfect forward secrecy +as the keys are stored in the room history. + +The encrypted content of the `m.rtc.encryption_keys` event was as follows: + +```json +{ + "session": { + "application": "m.call", + "call_id": "" + }, + "member": { + "id": "xyzABCDEF10123", + "device_id": "DEVICEID", + "user_id": "@user:matrix.domain" + }. + "keys": [ + { + "index": 0, + "key": "base64encodedkey" + }, + ], +} +``` + ## Security considerations +### Discoverability of infrastructure + +The `.well-known/matrix/client` is publicly readable, hence everyone can read and know +about the infrastructure which could lead to resource "stealing". +Each infrastructure however has their own authentication mechanism defined in the infrastructure specification. +Those mechanisms for instance can use a service to interact with the homeserver and based on that decide to allow users +to use the infrastructure. + +This is defined in the respective infrastructure MSC. + +### Forward secrecy for end-to-end encryption of media streams + +The considerations to ensure forward secrecy are described in the [End-to-end encryption of media streams](#end-to-end-encryption-of-media-streams) +section above. + +### End-to-end media encryption key rotation lag + +The proposed key rotation semantics does mean that a participant could continue to decrypt media that was sent in the three seconds after +leaving the session. + ## Unstable prefix -The state events and the well_known key introduced in this MSC use the unstable prefix -`org.matrix.msc4143.` instead of `m.` as used in the text. +Use `org.matrix.msc3401.call.member` as the state event type in place of `m.rtc.member`. + +For discovery via `.well-known/matrix/client` the prefix `org.matrix.msc4158.rtc_foci` is used in place of `m.rtc_foci`. + +Use `io.element.call.encryption_keys` in place of the `m.rtc.encryption_keys` room event and to-device event types. + +## Dependencies + +This proposal depends on +[MSC3757: Restricting who can overwrite a state event](https://github.com/matrix-org/matrix-spec-proposals/pull/3757) +to provide access control for the decentralised management of call membership state. However, an alternative such +as [MSC3779: "Owned" State Events](https://github.com/matrix-org/matrix-spec-proposals/pull/3779) could be used instead with +some adaptations. -Possible values inside the `m.rtc.member` event (like `m.call`) will use a prefix defined in the -related PR (TODO create and link `m.call` application type PR) +This proposal also depends on [MSC4140: Cancellable delayed events](https://github.com/matrix-org/matrix-spec-proposals/pull/4140) +to provide a mechanism for clients to ensure that they can update the room state even if they lose connection. From ec9fa8b8cac2d75d9133684a8def1343f443aba2 Mon Sep 17 00:00:00 2001 From: Hugh Nimmo-Smith Date: Tue, 17 Dec 2024 07:41:46 +0000 Subject: [PATCH 15/62] json5 for legibility --- proposals/4143-matrix-rtc.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 293a206ff3b..86de34e176f 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -82,7 +82,7 @@ Additional fields may be added depending on the application type. A full `m.rtc.member` state event for a joined member looks like this: -```json +```json5 // event type: "m.rtc.member" // state key: see next section for definition { @@ -162,7 +162,7 @@ There is an optional `leave_reason` field that can be used to provide a reason f An example of leaving a session where the user explicitly disconnects: -```json +```json5 // event type: "m.rtc.member" // state key: "@user:matrix.domain_xyzABCDEF10123" { @@ -174,7 +174,7 @@ to determine the details of the leave event. For example: -```json +```json5 // event type: "m.rtc.member" // state key: "@user:matrix.domain_xyzABCDEF10123" { @@ -235,7 +235,7 @@ and fulfil all of the these requirements. A MatrixRTC client has to first send/schedule the following delayed leave event: -```json +```json5 // event type: "m.rtc.member" // state key: "@user:matrix.domain_xyzABCDEF10123" { @@ -366,7 +366,7 @@ a sorted (by priority) list of Focus description objects. For example in generic form: -```json +```json5 { "m.rtc_foci": [ { @@ -380,7 +380,7 @@ For example in generic form: Or a concrete example for a `livekit` Focus: -```json +```json5 { "m.rtc_foci": [ { @@ -554,7 +554,7 @@ as the keys are stored in the room history. The encrypted content of the `m.rtc.encryption_keys` event was as follows: -```json +```json5 { "session": { "application": "m.call", From 50db42af969cc41ba86fc97068f29fca50c2f7f4 Mon Sep 17 00:00:00 2001 From: Timo K Date: Wed, 30 Jul 2025 15:45:11 +0200 Subject: [PATCH 16/62] be more specific on `member.id` Signed-off-by: Timo K --- proposals/4143-matrix-rtc.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 86de34e176f..61f003f0b95 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -61,8 +61,13 @@ Sending a well-formed `m.rtc.member` event that describes a connected state for The fields are as follows: - `member` required object - describes the participant of the RTC session: - - `id` required string - a unique identifier for this session membership as defined above. Recommended to be a UUID. It can be reused if the user leaves and rejoins the session. - It should be unique across all devices of the user. TODO: define grammar + - `id` required string - a unique identifier for this session membership. + It should be reused if the user leaves and rejoins the session from the same device. + It has to be unique for: + - each devices of the user + - each session (application + optional application specific id if multiples + sessions with the same application are possible, defined per application) + The proposed grammar for the id is: `_` - `device_id` required string - the Matrix device ID of the device that is joining the session. This is used when sending [to-device messages](https://spec.matrix.org/v1.11/client-server-api/#send-to-device-messaging). - `user_id` required string - the Matrix user ID of the user that is joining the session. This is needed as we cannot rely From 9bc444c2927f9d825ddeb2a725dd1f6a30854069 Mon Sep 17 00:00:00 2001 From: Timo K Date: Thu, 31 Jul 2025 14:10:39 +0200 Subject: [PATCH 17/62] add id to session Signed-off-by: Timo K --- proposals/4143-matrix-rtc.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 61f003f0b95..17df6d3beb0 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -92,7 +92,8 @@ A full `m.rtc.member` state event for a joined member looks like this: // state key: see next section for definition { "session": { - "application": "m.call" + "application": "m.call", + "id": UUID | ApplicationSpecificDefaultEnum // further fields for the application }, "member": { From 33231dcd4bfc9a2bb6ee5b715cdfe39f226ce6ff Mon Sep 17 00:00:00 2001 From: Timo K Date: Thu, 31 Jul 2025 14:10:59 +0200 Subject: [PATCH 18/62] calrify that the state key cannot be used as the source of truth for device id and application Signed-off-by: Timo K --- proposals/4143-matrix-rtc.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 17df6d3beb0..50fb80a65ff 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -158,9 +158,14 @@ For example with a `member`.`id` of `xyzABCDEF10123` for user `@user:matrix.doma For a client parsing the state key we would treat anything before the first `_` as the Matrix ID of the user and anything after as the `member`.`id` field. +The state key should/can **never** be used to imply any information about the user device or application. +Even thought the proposed format is`_`, +the state key only has to fulfill be unique in regards to device, application and application_id. + #### Leaving a session -Sending an empty `m.rtc.member` event represents a leave action. The state key must be the same as boefore +Sending an empty `m.rtc.member` event represents a leave action +for the associated rtc membership. There is an optional `leave_reason` field that can be used to provide a reason for leaving the session: From cea4cdd536ecc889058c9a4ef1e8a8159c92a277 Mon Sep 17 00:00:00 2001 From: Timo <16718859+toger5@users.noreply.github.com> Date: Fri, 8 Aug 2025 10:51:29 +0200 Subject: [PATCH 19/62] Update proposals/4143-matrix-rtc.md Co-authored-by: Johannes Marbach --- proposals/4143-matrix-rtc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 50fb80a65ff..37f4b336079 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -126,7 +126,7 @@ and a missing `created_ts` field implies that it is a join state event. All membership events that belong to one member session can be grouped with the index `created_ts()`+`state_key`. This is why the `m.rtc.member` events deliberately do NOT include something akin to a `membership_id`. -Other then the membership sessions, there is **no event** to represent a RTC session (containing all members). +Other than the membership sessions, there is **no event** to represent a RTC session (containing all members). This event would include shared information where it is not trivial to decide who has authority over it. Instead the session is a computed value based on `m.rtc.member` events. The list of events with the same `session` content represent one session. From d61969a9a3696b6c54d7987b1643b5bc03670927 Mon Sep 17 00:00:00 2001 From: Timo K Date: Tue, 26 Aug 2025 10:46:19 +0200 Subject: [PATCH 20/62] clarify what is part of this MSC and what can be found in other MSC's Signed-off-by: Timo K --- proposals/4143-matrix-rtc.md | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 37f4b336079..2bbb975abfb 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -8,14 +8,27 @@ a session. The MatrixRTC specification is separated into different modules: - -- The MatrixRTC room state that defines the state of the real time session.\ +- **The MatrixRTC room state** that defines the state of the real time session.\ It is the source of truth for: - Who is part of a session - Who is connected via what technology/backend - Metadata per device used by other participants to decide whether the streams from this source are of interest / need to be subscribed. -- The MatrixRTC backend. +- Key sharing for rtc data and media + - Everyone needs a secret for any other participant to encrypt media and other + real time data. + - There can be multiple keys or just one shared with the whole call. + The keys can get changed over time or stay the same during the whole session. + At the end every participant needs one valid key for every other participant + at any time of the session. + - This MSC also defines how keys are shared. + +This MSC will focus on the Matrix room state, which is responsible for +the high level signalling of a RTC session and the key sharing. + +The other modules are defined by other MSC's: + +- The MatrixRTC backend/real time data transport. - Allows for multiple backend implementations to be used. - It defines how to discover the available backend(s). - It defines how to connect the participating peers. @@ -31,10 +44,6 @@ The MatrixRTC specification is separated into different modules: - What streams to connect to. - What data in which format to sent over the RTC channels. - What MatrixRTC backends are supported. -- End-to-end encryption of media streams - -This MSC will focus on the Matrix room state which is responsible for the high -level signalling of a RTC session: ## Proposal From 4ad39614f11e7c7e682af7c14ac333c505f65abe Mon Sep 17 00:00:00 2001 From: fkwp Date: Tue, 14 Oct 2025 14:31:47 +0200 Subject: [PATCH 21/62] major rewrite addressing a lot of feedback from offline discussions. --- proposals/4143-matrix-rtc.md | 1400 +++++++++++++++++++++++----------- 1 file changed, 964 insertions(+), 436 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 2bbb975abfb..8c8c5a5f037 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -1,584 +1,1083 @@ # MSC4143: MatrixRTC -MatrixRTC is short for Matrix real time communication. -This MSC defines the modules with which the Matrix real time system is built. +This MSC defined Matrix real-time communication, in short MatrixRTC. This is the base layer to build +real-time systems on top of Matrix. -MatrixRTC specifies how a real time session is described in a room and how matrix users can connect to -a session. +MatrixRTC specifies how a real-time session is described in a room and how Matrix users can connect +to a session. The MatrixRTC specification is separated into different modules: -- **The MatrixRTC room state** that defines the state of the real time session.\ - It is the source of truth for: - - Who is part of a session - - Who is connected via what technology/backend - - Metadata per device used by other participants to decide whether the streams - from this source are of interest / need to be subscribed. -- Key sharing for rtc data and media - - Everyone needs a secret for any other participant to encrypt media and other - real time data. - - There can be multiple keys or just one shared with the whole call. - The keys can get changed over time or stay the same during the whole session. - At the end every participant needs one valid key for every other participant - at any time of the session. - - This MSC also defines how keys are shared. - -This MSC will focus on the Matrix room state, which is responsible for -the high level signalling of a RTC session and the key sharing. - -The other modules are defined by other MSC's: - -- The MatrixRTC backend/real time data transport. - - Allows for multiple backend implementations to be used. - - It defines how to discover the available backend(s). - - It defines how to connect the participating peers. - - Defines how to connect to a server/other peers, how to update the connection, - how to subscribe to different streams... - - A proposal utilising LiveKit is the standard for this as of writing. - - Another planned backend is a full mesh implementation based on [MSC3401](https://github.com/matrix-org/matrix-spec-proposals/pull/3401). -- The MatrixRTC application. - - Each application type can have it's own spec. - - Voice and video conferencing can be done with an application of type `m.call` - - The application defines all the details of the RTC experience: - - How to interpret the metadata of the member events. - - What streams to connect to. - - What data in which format to sent over the RTC channels. - - What MatrixRTC backends are supported. +* **The MatrixRTC State** — defines the state of a real-time session and serves as the source of + truth for: + * Which participants are part of a session + * Which RTC technology each participant is connected through + * Which kind and how many sessions can take place in this room. + * Per-device metadata used by other participants to decide whether streams from that source are of + interest and should be subscribed to. +* **Discovery of RTC Transports** + * The actual transport for real-time data provided by the Matrix deployment (e.g. Selective + Forwarding Units (SFUs) or assisted by STUN/TURN) + * Supports real-time communication exchange, with or without backend infrastructure + * Works across topologies such as SFU, P2P, or MCU +* **The E2EE key Sharing for RTC Data** + * Each participant requires a secret in order to publish encrypted real-time data + * Every participant must hold a valid key for every other participant at all times during the + session to decrypt their media + * Sessions may use multiple keys or a single shared key across all participants; keys may rotate + during the session + * This MSC also specifies how E2EE keys are distributed + +This MSC focuses on the three aspects above, the other modules are defined by accompanying MSC's: + +* **MatrixRTC Transports** + * Supports multiple transport implementations + * Defines how to connect to peers or servers, update connections, and subscribe to streams + * A LiveKit-based transport is the current standard proposal (MSC4195: MatrixRTC Transport using + LiveKit backend) + * Another planned transport is a full-mesh implementation based on + [MSC3401](https://github.com/matrix-org/matrix-spec-proposals/pull/3401) +* **MatrixRTC Applications** + * Each application type can have its own specification + * Example: voice and video conferencing via an application of type `m.call` + * Defining finer-grained signalling UX, e.g., ringing, declining, stopping ringing + * Specifies the full RTC experience, including: + * How to interpret member event metadata + * What streams to connect to + * What data to send over RTC channels and in which format + * Which MatrixRTC transports are supported ## Proposal -Each RTC session is made out of a collection of `m.rtc.member` room state events. -Each `m.rtc.member` event defines who (the `member`) is a participant of which session (the `session`). +MatrixRTC provides a framework for building real-time communication on top of Matrix. At its core, +it introduces the concept of **applications**, which define the semantics of a real-time experience, +and **slots**, which act as containers where those experiences take place within a room. +Participants join as **members** within slots, and their overlap defines a **session**. -### The MatrixRTC room state +This proposal defines how MatrixRTC slots are opened and closed, how MatrixRTC members join and +leave, and how these interactions together form the lifecycle of a MatrixRTC session. -All data related to a MatrixRTC session -(current session, sessions history, join/leave events, ...) only -requires one event type. +The proposal also specifies how: -(current session, sessions history, join/leave events, ...) only -require one event type:. +* **Applications** describe the type of RTC activity (e.g. a call, a shared document, or a real-time + game) and may define additional metadata and constraints. +* **Slots** are represented in room state (`m.rtc.slot` identified by a unique `slot_id`) and govern + what kind of application may run, along with permissions and configuration. +* **Membership** (`m.rtc.member` [MSC4354 Sticky + Events](https://github.com/matrix-org/matrix-spec-proposals/pull/4354)) provides a precise record + of who is actively participating, and under which transports and devices. +* **Sessions** emerge from the overlap of active members within an open slot, with clear start and + end conditions. +* **Transports** define how participants exchange media, supporting multiple backends such as SFUs, + MCUs, or peer-to-peer. +* **End-to-end encryption** ensures secure media exchange, including mechanisms for key sharing and + rotation. -We use a set of `m.rtc.member` (one for each participant) state events to represent a session. +This design deliberately separates **slot management** (opening/closing slots, requiring higher +power levels) from **slot participation** (joining as a member, requiring lower power levels). This +separation gives applications a robust surface for conflict resolution, clear ownership of events, +and flexibility across use cases such as always-on spaces, scheduled meetings, or ephemeral +conversations. -based on the content a `m.rtc.member` state event can either represent a connected or a disconnected member. +The following sections specify these components in detail. -#### Joining a session +### MatrixRTC Application Types -Sending a well-formed `m.rtc.member` event that describes a connected state for a state key that is not yet used or contains a disconnected `m.rtc.member` event represents a join action. +An **application type** defines the semantics of a MatrixRTC session, such as a real-time game, a +voice/video call, or a shared document. Each application type has its own specification describing: -The fields are as follows: +* How different RTC streams are interpreted. +* Which MatrixRTC transport types are supported or required. -- `member` required object - describes the participant of the RTC session: - - `id` required string - a unique identifier for this session membership. - It should be reused if the user leaves and rejoins the session from the same device. - It has to be unique for: - - each devices of the user - - each session (application + optional application specific id if multiples - sessions with the same application are possible, defined per application) - The proposed grammar for the id is: `_` - - `device_id` required string - the Matrix device ID of the device that is joining the session. This is used when sending - [to-device messages](https://spec.matrix.org/v1.11/client-server-api/#send-to-device-messaging). - - `user_id` required string - the Matrix user ID of the user that is joining the session. This is needed as we cannot rely - on the owner of state event as it might have been modified by an admin or similar. -- `session` required object - an object that is used to uniquely identify this session across RTC member events -of the Matrix room: - - `application` required string - a recognised application type. e.g. `m.call` as linked below - - additional fields as defined by the application type -- `created_ts` - timestamp in milliseconds since UNIX epoch. - - this should **not** be present the first time that the `m.rtc.member` event is sent. - - if the `m.rtc.member` event is sent again, the `created_ts` should be populated with the `origin_server_ts` - that was given to the previous version of the state event. -- `focus_active` required Focus object - specifies the algorithm that defines how to choose a Focus for this member. See below for details. -- `foci_preferred` required array of Focus objects - specifies the input data for this algorithm contributed by this member. See below for details. +For example, a [Third Room](https://thirdroom.io) like experience could include information about +the virtual scene, e.g., a place, available objects or weather conditions. -Additional fields may be added depending on the application type. +This modular design makes MatrixRTC flexible. For example, a Jitsi-based conference could be +supported by defining a new application type and a corresponding RTC transport. While such a session +would not be compatible with Matrix clients that do not implement the Jitsi transport, those clients +would still be able to detect the presence of an unsupported application and transport type, and +handle it with a sensible user interface. -A full `m.rtc.member` state event for a joined member looks like this: +The minimum Application definition consists of a simple JSON object -```json5 -// event type: "m.rtc.member" -// state key: see next section for definition +```json { - "session": { - "application": "m.call", - "id": UUID | ApplicationSpecificDefaultEnum - // further fields for the application - }, - "member": { - "id": "xyzABCDEF10123", - "device_id": "DEVICEID", - "user_id": "@user:matrix.domain" - }, - "created_ts": Time | undefined, - "focus_active": {...FOCUS_A}, - "foci_preferred": [ - {...FOCUS_1}, - {...FOCUS_2} - ] + "application": { + "type": "m.call", // The # character MUST NOT appear in a valid type string. + // Optional: application-specific metadata + "m.call.spatial_audio": true + } } ``` -This gives us the information, that user: `@user:matrix.domain` with member ID `DEVICEID_m:call_123456789` -is part of a session identified by `{}` using application of type `m.call` connected over `FOCUS_A`. -This is sufficient information for another room member to detect the running session and join it. +The `type` field MUST be a string that does not contain the `#` character. The type field SHOULD +follow the [*Common Namespaced Identifier +Grammar*](https://spec.matrix.org/v1.16/appendices/#common-namespaced-identifier-grammar). -`created_ts` is an optional property that caches the time of creation. It is not required -for an event that, has not yet been updated, there the `origin_server_ts` is used. +Each application type MUST have its own MSC, which specifies the additional fields and defines how +communication with the corresponding transport is handled. For example: -> [!NOTE] -> We introduce `created_ts()` as the notation for `created_ts ?? origin_server_ts` +* `m.call` — voice and video calling, as defined in + [MSC4196](https://github.com/matrix-org/matrix-spec-proposals/pull/4196). -Once the event gets updated the `origin_server_ts` needs to be copied into the `created_ts` field. -An existing `created_ts` field implies that this is a state event updating the current session -and a missing `created_ts` field implies that it is a join state event. -All membership events that belong to one member session can be grouped with the index -`created_ts()`+`state_key`. This is why the `m.rtc.member` events deliberately do NOT include something akin to a `membership_id`. +To facilitate interoperability, ideally each application type should provide a Matrix widget that +can serve as a reference implementation for clients. This allows client developers to support new +application types by embedding or integrating the widget, without having to implement the full +application logic themselves. -Other than the membership sessions, there is **no event** to represent a RTC session (containing all members). -This event would include shared information where it is not trivial to decide who has authority over it. -Instead the session is a computed value based on `m.rtc.member` events. -The list of events with the same `session` content represent one session. -This array allows to compute fields like participant count, start time etc. +### MatrixRTC Slot and Constraining Slots -Based on the value of `application`, the event might include additional parameters -to provide additional session parameters. +A **MatrixRTC slot** is the container for MatrixRTC members and temporal overlapping MatrixRTC +members form a MatrixRTC session. Each slot is identified by a unique `slot_id`, tied to a specific +`application` type, and represented by the `m.rtc.slot` state event with the `slot_id` as the state +key. -> A [Third Room](https://thirdroom.io) like experience could include the information of an approximate position -> on the map, so that clients can omit connecting to participants that are not in their -> area of interest. +> [!NOTE] Conceptually, a slot is like a virtual meeting room: MatrixRTC sessions occur within +> slots, and multiple sessions may occur consecutively without changing the slot’s configuration. -#### State key for `m.rtc.member` +Slots are part of the shared room state and can only be created or modified by authorized users with +sufficient `power_level`. Clients MUST react on and respect latest `m.rtc.slot` definitions as +defined by the Matrix room state at all times. The ability to open and close slots with different +patterns enables a wide variety of use cases, from always-on shared spaces to short lived scheduled +meetings. -The state key is generated from the `member` field of the `m.rtc.member` event. +Slots are named by the RTC app, but are expected to typically be ordinals (e.g. equivalent to line 1 +or line 2 on a telephone). If the app needs to define its slots out of band (e.g. mapping them to +widget IDs) then it can use those IDs as names. However, given a slot is the mechanism around which +sessions converge, it must have a predictable name. Unpredictable IDs such as session IDs should +never be used to name a slot. -We want to choose a state key that is compatible with whichever state protection proposal is accepted to ensure that -users cannot modify one another's sessions. +#### Opening a MatrixRTC Slot -For [MSC3757](https://github.com/matrix-org/matrix-spec-proposals/pull/3757) we generate the state key by -concatenating the following strings: +A slot is opened by sending an `m.rtc.slot` state event with `state_key = slot_id`. This event +authorises MatrixRTC members that intend to participate in the slot and follows the schema below: -- the Matrix ID of the user -- an `_` (underscore) -- the `member`.`id` field +```json +// Example: an open slot with application-specific metadata +{ + "application": { + "type": "m.call", + // optional: app specific slot metadata + "m.call.id": UUID, // Note your application must handle rollback due to state resolution + "m.call.voice_only": true + } +} -For example with a `member`.`id` of `xyzABCDEF10123` for user `@user:matrix.domain` the state key would be `@user:matrix.domain_xyzABCDEF10123`. +state_key: "m.call#ROOM" // slot_id +``` -For a client parsing the state key we would treat anything before the first `_` as the Matrix ID of the user -and anything after as the `member`.`id` field. +**Field description:** -The state key should/can **never** be used to imply any information about the user device or application. -Even thought the proposed format is`_`, -the state key only has to fulfill be unique in regards to device, application and application_id. +* **state key**: The state key of the `m.rtc.slot` state event, referred to as the `slot_id`, serves + as the slot name. +* `application` An application JSON object, which **MUST** specify the application type and MAY + include additional fields which **constrain** the application (e.g., restricting a call to be + voice-only). **Note** those additional fields can be used in combination with MatrixRTC member + ones subject to the application specific MSC. -#### Leaving a session +The `slot_id` of an open slot acts like a virtual address where participants are allowed to meet. +The grammar for the `slot_id` is a well formed state key confined such that members of different +MatrixRTC applications never occupy the same slot according to: -Sending an empty `m.rtc.member` event represents a leave action -for the associated rtc membership. +``` +{application.type}#{application_slot_id} +``` -There is an optional `leave_reason` field that can be used to provide a reason for leaving the session: +Where -- `leave_reason` optional string - one of: `lost_connection` +* `application.type` is the `type` field in the application JSON object +* The `#` character MUST NOT be used in either `application.type` or `application_slot_id` +* `application_slot_id` is the application-specific slot ID. Each application MSC defines its own + schema (e.g., `ROOM`, `1`, `2`) to allow multiple parallel slots of the same type according to the + application requirements. -An example of leaving a session where the user explicitly disconnects: +This grammar MUST never be used to parse `slot_ids`; it exists only to namespace the `state_key`. -```json5 -// event type: "m.rtc.member" -// state key: "@user:matrix.domain_xyzABCDEF10123" -{ -} +#### Closing a MatrixRTC Slot + +To close a slot, the corresponding `m.rtc.slot` state event is updated with empty content which +removes all remaining MatrixRTC members, for example: + +```json +// Empty content represents a closed slot +{} + +state_key: "m.call#ROOM" // slot_id ``` -The client should use the `prev_content` field of the [room state event](https://spec.matrix.org/v1.11/client-server-api/#room-event-format) -to determine the details of the leave event. +#### Examples of MatrixRTC Slot Usage -For example: +In the following example three different slot use-cases are depicted +``` +m.rtc.slot[id_1] {content_1} ...████████████████████████████████████████████... +m.rtc.slot[id_2] {content_2} ███████████████████ -```json5 -// event type: "m.rtc.member" -// state key: "@user:matrix.domain_xyzABCDEF10123" -{ - "content": { - "leave_reason": "lost_connection" - }, - "prev_content": { - "session": { - "application": "m.call", - "call_id": "" - }, - "member": { - "id": "xyzABCDEF10123", - "device_id": "DEVICEID", - "user_id": "@user:matrix.domain" - }, - "created_ts": 123456, - "focus_active": {...FOCUS_A}, - "foci_preferred": [ - {...FOCUS_1}, - {...FOCUS_2} - ] - } -} +m.rtc.slot[id_state_res] {content_ABC} ████XXXXXXXXXXXXXXXXXXXXXXXXXXXXX +m.rtc.slot[id_state_res] {content_XYZ} █████████████████████████████ + | + | + Rollback due to Matrix state resolution + + +Time ────────────────────────────────────────────────────► ``` -#### Reliability requirements for the room state +* `id_1` comprises a long-lived slot suited for a Discord-style experience or a shared whiteboard + where people can hop on and hop off as desired. +* `id_2` suited for a scheduled conference meeting, where the state event might be managed using + cancellable delayed events (MSC4140) or a bot. +* `id_state_res` This `m.rtc.slot` state event is subject to Matrix state resolution. Hence, it is + divided potentially into two intervals. If a rollback occurs during an active MatrixRTC session, + application logic MUST handle conflicts (e.g., a rolled-back UUID). -Room state is a very well suited place to store the data for a MatrixRTC session. -It allows: +#### Recommended User Experience -- The client to determine current ongoing sessions without loading history for every room. - Or doing additional work other then the sync loop that needs to run anyways. -- The client can compute/access data of past sessions without any additional redundant data. -- Sessions (start/end/participant count) are federated and there is not redundant data storage that - could result in conflicts, or can get out of sync. The room state events are part of the DAG and this - is solved like any other Persistent Data Unit (PDU) in Matrix. +The lifecycle of a slot follows four distinct states: -However, a challenging circumstance with using the room state to represent a session is -the disconnection behaviour. If the client disconnects from a call because of a network issue, -an application crash or a user forcefully quitting the client, the room state cannot be updated anymore. -The client is required to leave by sending a new empty state which cannot happen once connection is lost. +``` +Closed +Open + ├─ Active --- at least one m.rtc.member is connected + └─ Inactive --- no member is connected +``` -If the state is not updated correctly we end up with a room state that is not -correctly representing the current RTC session state. Historic and current MatrixRTC session data would be broken. +Slots may transition between these states as a slot is opened or closed, or as users connect or disconnect. -For an acceptable solution, the following requirements need to be taken into consideration: +While **participation** in a MatrixRTC session is possible if a **slot is open**, **management** of +slots depends on the user’s **power level**. Clients MUST respect these distinctions in the user +experience. -- Room state is set to empty if the client looses connection. (A heartbeat like system is desired) -- The best source of truth for a call participation is a working connection to the SFU. - It is desired that the disconnect of the member on the SFU gets propagated to the room state. -- It should be possible to updated the room state without the client being online. -- All this should be compatible when Matrix uses cryptographic identities. +For **open** slots in a room, clients SHOULD clearly represent the current slot’s activity state: -[MSC4140](https://github.com/matrix-org/matrix-spec-proposals/pull/4140) proposes a concept to -delay the leave events until one of the leave conditions (heartbeat or SFU disconnect) occur -and fulfil all of the these requirements. +* **Active:** clients SHOULD indicate which slots are currently active in a room (e.g., by showing + participant count, showing an icon, list entry, or join button). +* **Inactive:** clients SHOULD indicate that the slot is available and provide a way for a user to + join as the first member (e.g., via a “Call” button or placeholder icon). -A MatrixRTC client has to first send/schedule the following delayed leave event: +**Slot management** UX SHOULD only exposed to users who have sufficient power level to manage slots: -```json5 -// event type: "m.rtc.member" -// state key: "@user:matrix.domain_xyzABCDEF10123" -{ - "leave_reason": "lost_connection" -} -``` +* **Closed slot:** The UX SHOULD provide controls to manage the slot (e.g., opening or modifying the + slot) +* **Open slot:** The UX SHOULD provide controls to perform management (e.g., closing or modifying + the slot) -only after that the actual state event can be sent, so that we guarantee that the state will be empty eventually. -The `leave_reason` is added so clients can be more verbal about why a user disconnected from a call. -It allows to communicate with other participants in a session if the user has disconnected intentionally or lost connection. +Multiple active slots may exist in the same room if the application type supports them. -#### Session history +Clients MAY present an option to room administrators to enable specific applications in the room by +adding a slot and a type to room state. Future MSCs may change the defaults for new rooms to enable +RTC applications by default. -Since there is no single entry for a historic session (because of the ownership discussion), -historic sessions need to be computed and most likely cached on the client. +### MatrixRTC Membership -Each state event can either mark a join or leave: +A MatrixRTC membership is represented by a sticky `m.rtc.member` event +([MSC4354](https://github.com/matrix-org/matrix-spec-proposals/pull/4354) Sticky Events). These +events describe a participant’s presence in an MatrixRTC slot and provide sufficient metadata for +other room members to detect and join the same slot. -- join: `prev_state.session != current_state.session` && - `current_state.session != undefined` - (where an empty `m.rtc.member` event would imply `state.session == undefined`) -- leave: `prev_state.session != current_state.session` && - `current_state.session == undefined` +The membership events are implemented as a key-value store according to the addendum section of +[MSC4354 Sticky Events](https://github.com/matrix-org/matrix-spec-proposals/pull/4354) giving the +same semantics as Matrix room state. If the room is encrypted `m.rtc.member` events are also +encrypted and vice versa. -Based on this one can find user sessions. (The range between a join and a leave -event) of specific times. -The collection of all overlapping user sessions with the same `session` contents -define one MatrixRTC history event. +An `m.rtc.member` sticky event can be either **Connected** or **Disconnected** -### The RTC backend +* **Connected**, if + * **Slot open**: the `sticky_key` of the event needs to match the `member.id` AND the `slot_id` + matches the state\_key of the slot event. + * **Content:** MUST match the JSON content schema for connecting to a slot (see below). + * The sender is still a member of the room (not kicked / left) + * **Event is sticky**: The sticky event needs not to be expired as described in [MSC4354 Sticky + Events](https://github.com/matrix-org/matrix-spec-proposals/pull/4354) +* **Disconnected**, if + * **Not Connected:** any of the required conditions to be connected are not met. + * **Content:** SHOULD match the JSON content schema for disconnecting from a slot (see below). If + it does not match, clients MUST handle the lack of information gracefully. -Backend **infrastructure** in this context can be anything that can serve as the backend for a -MatrixRTC session. In most cases this is a SFU. But also a full mesh implementation could -be an infrastructure. Not all kind of infrastructure require a way of sourcing a backend resource -(e.g. full-mesh). In this MSC we only refer to infrastructure where it is necessary to have access to additional -data to participate in the MatrixRTC session. +#### Connecting to a MatrixRTC Slot -The backend is referred to as a Focus or as Foci in plural. +A valid `m.rtc.member` event as a prerequisite for connecting to a slot has the following schema: -Note that these backends are independent of the application (e.g. `m.call`) being used in the session. +```json +// event type: "m.rtc.member" +{ + "slot_id": "m.call#.ROOM", + "application": { + "type": "m.call", + // further fields for the application (optional) + "m.call.id": UUID + }, + "member": { + "id": "xyzABCDEF0123" // UUID random/anonymise/unique (external) service identifier + "claimed_device_id": "DEVICEID" + "claimed_user_id": "@user:matrix.domain" + }, + "m.relates_to":{ // an updated m.rtc.member event MUST reference the first m.rtc.member + rel_type: "m.reference", // event you sent for this call. You should omit if this is the first event. + event_id: "$join_event_id" + }, + "rtc_transports": [ + {...TRANSPORT_1}, + {...TRANSPORT_2} + ], + "sticky_key": "xyzABCDEF0123" // same as member.id + "versions": [ + "v0", + "example.mscXXXX.asymmetric_encryption" + ], +} +``` -A Focus is represented as a JSON object with one mandatory field: +**Field explanations:** + +* `slot_id` — The slot this member belongs to. +* `application` — Is a JSON object that identifies the MatrixRTC application type; Besides the + mandatory `type` field which needs to match the MatrixRTC slot identified by `slot_id` it **may** + include application-specific metadata (see MatrixRTC Application Types). For example, a [Third + Room](https://thirdroom.io) style experience could include approximate position data on a map, + allowing clients to avoid connecting to participants outside their area of interest. + **Note** application-specific metadata as part of the MatrixRTC slot `application` object have + precedence over the MatrixRTC member ones. +* `m.relates_to` — The `m.relates_to` field optionally references the initial connect event, + distinguishing connect events from updates. Clients SHOULD include this field when sending updates + to an existing `m.rtc.member` event to ensure continuity in membership lifecycle tracking, enable + accurate historical reconstruction, and allow deriving the participant’s start time using the + `origin_server_ts` of the referenced event. +* `member` — Uniquely identifies this participation instance; includes: + * `id` — UUID to distinguish multiple participants, even for the same device. This ID can also + serve as a canonical identifier for certain MatrixRTC transports, helping to prevent metadata + leakage. The ID MUST be unique for each connect event. + * `claimed_device_id` — Matrix device identifier. + * `claimed_user_id` — Matrix user ID. +* `rtc_transports` — List of objects describing how to access this participant’s media streams. See + [MatrixRTC Transport](#matrixrtc-transport) for the correct object format. +* `sticky_key` — A unique key used to track this membership across updates. The key persists for the + lifetime of the MatrixRTC session and is a copy of the `member.id` field[^1]. +* `versions` — Protocol versions and capabilities supported by the client. + +For the example above, user `@user:matrix.domain` with `member.id = xyzABCDEF0123` is part of the +slot `m.call#ROOM`, using an application of type `m.call`, and connected over `TRANSPORT_1`. This +information is sufficient for other room members to detect the running slot and join the session. -- `type` required string: The type of the Focus as defined by an RTC backend.. +> [!NOTE] +> to stay connected: The client MUST maintain participation by sending a new sticky event, before an +`m.rtc.member` sticky event expires (i.e., `sticky.duration_ms <= 0`). **It is strongly +recommended** that the update of the `m.rtc.member` sticky event is scheduled sufficiently ahead of +the event timing out (e.g., 5 minutes) to minimize potential connection state “flipping effects”. -Additional fields will be present depending on `type`. +#### Disconnect from a MatrixRTC Slot -Only users with the same type can connect in one session. If a frontend does -not support the used type they cannot connect. +A valid `m.rtc.member` event as a prerequisite for disconnecting from a slot has the following schema: -Each Focus type will get its own MSC in which the detailed procedure to get from -the foci information to working WebRTC connections to the streams of all the -participants is explained. +``` +// event type: "m.rtc.member" +{ + "slot_id": "m.call#ROOM", // MUST + "m.relates_to":{ // SHOULD + rel_type: "m.reference", + event_id: "$join_event_id" + }, + "disconnect_reason": { // SHOULD + "class": "server_error", + "reason": "ice_failed", + "description": "Failed to establish peer-to-peer connection via ICE", + } + "sticky_key": "xyzABCDEF0123" +} +``` -Foci are represented in three places: +> [!NOTE] +> Any other content for the same `sticky_key` that is not a valid connect event is also treated as a +> disconnect event. However, it is lacking important metadata useful for UX. + +**Field explanations:** + +- `slot_id`: The slot this member belongs to. +- `m.relates_to`: Clients SHOULD include the `m.relates_to` field referencing the original + `m.rtc.member` join event when disconnecting. This facilitates accurate session history and + retrospective computations by fetching the relation to determine the participant’s original join + time or associated metadata. +- `sticky_key:` The sticky key from the disconnecting MatrixRTC member. +- `disconnect_reason`: The `disconnect_reason` object is **optional** and provides additional + context when a participant disconnects from a call. It is only meaningful if the user has + **previously attempted to connect** (i.e., has sent at least one valid `m.rtc.member` event for + the slot) This ensures that the disconnection reason refers to a real connection lifecycle rather + than a pre-join cancellation. + +`disconnect_reason` F**ield explanations:** + +| Field | Type | Required | Description | +| ----- | ----- | ----- | ----- | +| `class` | string | ✅ | High-level category of the disconnection or error. | +| `reason` | string | ✅ | Machine-readable identifier of the specific cause. | +| `description` | string | ⚪ | Optional human-readable explanation providing additional context. | + +**Class categories and examples:** + +| Class | Example Reason | Description / When Used | +| ----- | ----- | ----- | +| user\_action | `hangup` | Participant intentionally ended the call after joining. | +| | `switch_device` | User moved the session to another device mid-call. | +| client\_error | `media_error` | Failed to capture or transmit audio/video after joining. | +| | `transport_failure` | Local ICE/DTLS setup failed despite a successful `m.rtc.member` event. | +| | `encryption_error` | Failed to set up E2EE for the media channel after connecting. | +| server\_error | `ice_failed` | ICE negotiation could not complete due to network/server issues. | +| | `dtls_failed` | DTLS handshake failed. | +| | `network_error` | Temporary network outage caused the connection to drop. | +| redirection | `call_transferred` | Call was redirected to another slot, device, or user. | +| | `moved_temporarily` | Session temporarily moved (e.g., server migration). | +| permanent\_failure | `codec_mismatch` | Participant cannot decode/encode the call media. | +| | `unsupported_features` | Session requested unsupported capabilities. | + +**Note: (Pre-join)** In situations where a client never successfully connects to a call (for +example, if the user is busy or declines a MatrixRTC session), a dedicated **sticky event** is +required to convey the participant’s status. + +#### Lifecycle of a MatrixRTC Membership {#lifecycle-of-a-matrixrtc-membership} + +The MatrixRTC membership is a collection of linked `m.rtc.member` events. With the definition of a +**Connected** and a **Disconnected** MatrixRTC membership from above: + +* The **connect transition** occurs when the `m.rtc.member` collection becomes Connected. Due to the + definition of Connected this happens at `max(start_slot_ts, start_content_ts)` where: + * `start_content_ts:` is the time the connect `m.rtc.member` event is sent. + * `start_slot_ts:` is the time the slot is opened. +* The **disconnect transition** occurs when the `m.rtc.member` collection becomes Disconnected. Due + to the definition of Disconnected this happens at `min(end_slot_ts, end_content_ts, + end_sticky_ts).` where: + * `end_content_ts:` is the time the disconnect `m.rtc.member` event is sent. + * `end_slot_ts:` is the time the slot is closed. + * `end_sticky_ts` is the time the stickiness of the connected event expires. -- `focus_active` of `m.rtc.member` state event - specifies the algorithm that defines how to choose a Focus for this member. -- `foci_preferred` of `m.rtc.member` state event- specifies the input data for this algorithm contributed by this member. -- `m.rtc_foci` of the `.well-known/matrix/client` - specifies the list of available Foci for the homeserver. +``` + (Connect) (Update) (Disconnect) + + m.rtc.member ───────► m.rtc.member ──── ... ───► m.rtc.member + ^ | | + ├─────────────────────┘ m.relation | + | | + └────────────────────────────────────────────────┘ m.relation + References original join event + + + [---------------- Connected ------- ... --------][--- Disconnected ... +Time ───────────────────────────────────────────────────────────────────────► +``` -The `focus_active` algorithm needs to be designed so that all participants converge to the same SFU/Focus. +> [!NOTE] +> Updates are optional. They are only required if the participation lasts longer than the sticky +event’s timeout (`sticky.duration_ms`) or the member changes their `m.rtc.member` event (for example +to change transport). In such cases, participants MUST refresh their membership by sending a new +sticky event update. -The following Focus `type` values are defined: +#### Reliability for Real-Time Applications -- `livekit` - a backend using the [LiveKit](https://livekit.io/) SFU as described in - [MSC4195](https://github.com/matrix-org/matrix-spec-proposals/pull/4195). -- `full_mesh` - a backend using a full-mesh approach based on [MSC3401](https://github.com/matrix-org/matrix-spec-proposals/pull/3401). +For real-time applications, it is important to maintain an accurate view of which participants are +connected and for how long. In particular: -#### Choosing the value of `foci_preferred` for the `m.rtc.member` state event +* **Precise membership tracking** — RTC applications rely on a high-resolution understanding of the + lifetime of each `m.rtc.member` object. Knowing exactly when a participant joins or leaves is + critical for computing session state. +* **Handling (unintentionally) disconnected clients** — A client that loses network connectivity + must not be treated as continuously connected. Otherwise, the session state may become + inconsistent or appear to include phantom participants. -At some point session participants have to decide/propose which Focus they will use. +Clients MUST handle network disconnects or crashes by sending empty leave events once reconnection +is possible, or by using cancellable delayed leave events +([MSC4140](https://github.com/matrix-org/matrix-spec-proposals/pull/4140)). -Based on the Focus type and application choosing the method by which the contents of the `foci_preferred` field on the `m.rtc.member` -can be different. +Clients SHOULD use cancellable delayed events to implement a “deadman switch” for MatrixRTC +membership by sending a leave event ahead of the join event as a delayed event with a reasonable +delay (e.g., 15–30 seconds). The client then periodically resets the delayed event’s timer. If the +timer expires due to a missing reset, the leave event is automatically emitted, marking the +participant as disconnected and ensuring accurate session state even in cases of sudden +disconnection, crashes, or network failures. -There are three guidelines which should be obeyed by a client when building the `foci_preferred` list: +### MatrixRTC Session -1. It is always desired to have as few Focus switches as possible. +A **MatrixRTC Session** is defined as the period of overlapping, **Connected** `m.rtc.member` events +that share the same slot identified by `slot_id`. In other words, a session represents the span of +time during which a potentially changing set of one or more participants is continuously connected +to the same MatrixRTC slot. -If there are other participants on the session (i.e. other `m.rtc.member` events) the client should calculate what the Focus it should connect to -based on the `m.rtc.member` events for the existing participants. -This should happen reactively on each `m.rtc.member` state event change. -Each MatrixRTC frontend is responsible that it can deal with focus switches based on changing state gracefully. It is part of the design of MatrixRTC and a requirement for a eventually consistent distributed system. +Note the Connected status of the m.rtc.member event is already bounded to an open slot -The calculated Focus should then be present at the start of the `foci_preferred` list. +Example of a MatrixRTC session: This diagram illustrates overlapping member intervals defining the +MatrixRTC session lifetime, bounded by the slot being open. -2. The client should lookup the suggested foci from the homeserver `.well-known/matrix/client` as defined below. +``` + Session Start Session End + | | +m.rtc.member[0] |████████████████████████ ████████████████████████|████ + |^ connect disconnect ^ ^ connect disconnect ^| + | | +m.rtc.member[1] | ███████████████████████████████████████ | + | ^ connect disconnect ^ | + | | +m.rtc.member[2] | ████████████████████████ | + | ^ connect disconnect ^ | + | | +Slot (open) [*********************************************************] + | | +Session lifetime [***************************************************] + + +Time ───────────────────────────────────────────────────────────────────► +``` -MatrixRTC is designed around the same culture that makes Matrix possible: A large amount of infrastructure in the form of homeservers is provided by the users. +The lifetime of a MatrixRTC session is implicitly defined by the union of all **connected** member +intervals, bounded by the slot’s open and close times: -To achieve a stable and healthy ecosystem backend RTC infrastructure should be thought of as a part of a homeserver. +* **Session start:** the first member connects +* **Session end:** the last connected member disconnects -It is very similar to a TURN server: mostly traffic and little CPU load. +#### On Session Identifiers -To not end up in a world where each user is only using one central SFU but where the traffic -is split over multiple SFU's it is important that we leverage the SFU distribution on the -homeserver federation. +A MatrixRTC session identifier MAY be retrospectively derived by XORing the `event_id` of all +associated `m.rtc.member` events. -These proposals from **your own** homeserver should come next in the `foci_preferred` list of the member event. +Alternatively, a pre-defined MatrixRTC session identifier MAY be included in the associated +`m.rtc.slot` state event. Because Matrix state events are always subject to state resolution, +applications MUST handle potential conflicts gracefully — for example, by rolling back conflicting +state changes or employing a coordination mechanism to prevent conflicts in the first place. A +typical implementation might involve a bot maintaining a Matrix room as the sole administrator, +opening a slot for a scheduled video conference, and including a unique `m.call.id` in the +`application` JSON object. When a pre-defined session identifier is used, only one MatrixRTC session +SHOULD exist within that slot. -3. Clients should not use a hard-coded Focus. +In essence, *slots* exist to support concurrent RTC sessions within the same room. The sessions +within each slot may or may not have an stated session ID depending on whether the app is able to +switch ID mid-session (or display multiple concurrent sessions meaningfully). -Looking up the preferred Foci from a client is toxic to a federated system. If the majority of users -decide to use the same client all of the users will use one Focus. This destroys the passive security mechanism, that -each instance is not an interesting attack vector since it is only a fraction of the network. -Additionally it will result in poor performance if every user on Matrix would use the same Focus. +* If an application supports switching or displaying multiple sessions concurrently, participants + SHOULD explicitly declare the session they belong to. +* If it does not (for example, in typical group VoIP scenarios), participants may omit a session ID + and implicitly join the default, anonymous session associated with that slot. -However, there are cases where this is acceptable: +**Implementation note:** +This behavior may lead to user confusion in edge cases such as network partitions. For example, +users who thought they were in one session during a network partition and then find themselves +suddenly merged into another (unified) session, and so teleported into someone else’s conversation. -- Transitioning to MatrixRTC. Here it might be beneficial to have a client that has a fallback Focus - so calls also work with homeservers not supporting it. -- For testing purposes where a different Focus should be tested but one does not want to touch the .well-known -- For custom deployments that benefit from having the Focus configuration on a per client basis instead of per homeserver. +A more robust design might require all calls to have explicit session identifiers. In cases where +conflicting identifiers are detected shortly after call setup, clients could tear down the local +session and converge on the session that wins the race. -Therefore, if a client does use a hard-coded Focus it should come last in the `foci_preferred` list. +If conflicts occur later during a session, the better user experience may be to treat them as two +concurrent calls within the same room, allowing users to choose which session to join. In practice, +this could be implemented as a prompt (e.g., *“There’s already a call happening — do you want to +join it?”*), potentially automated to accept by default when no other participants remain in the +local session. -#### Discovery of Foci using `.well-known/matrix/client` +#### Session history + +As MatrixRTC does not emit a single event describing past sessions, historic MatrixRTC sessions MUST +be reconstructed from room history. Clients SHOULD cache reconstructed sessions to avoid repeated, +expensive history pagination. + +In general, all required information is present as part of the Matrix DAG: + +* **Sticky events** are persistently recorded in the Matrix DAG. +* **`m.rtc.slot` state event transitions**, even when affected by state resolution, are visible as + part of the timeline. + +Counterintuitively, **state resolution is not a problem** — RTC sessions occurred in real life, and +their existence remains valid regardless of how the room state later resolves. + +However, for this reconstruction process to work **reliably and deterministically**, the Matrix +protocol requires **consistent message ordering (on a homeserver basis)** across both state and +timeline events. This remains an open problem and is discussed further in the +[*Impact of Message Ordering wrt. Session History*](#impact-of-message-ordering-wrt.-session-history) +section. As a temporary workaround, the reconstruction process uses `origin_server_ts` ordering. + +General concept +* Paginate backward through room history to collect relevant events: + * `m.rtc.slot` events (slot open/close boundaries) + * `m.rtc.member` events (connects, updates, disconnects) +* Correlate `m.rtc.slot` events with surrounding `m.rtc.member` events to reconstruct membership + intervals. +* Handle incomplete membership events: clients may fail to send proper connect/disconnect events; + using slot boundaries helps limit incorrect membership data. + +Note: due to the two-tier power level model (who may manage slots vs participation), using slot +boundaries to filter membership events is typically sufficient to avoid many classes of spurious or +invalid records. + +Step-by-step reconstruction algorithm + +1. Gather all `m.rtc.slot` events from the timeline and determine the slot open/close intervals + (i.e. transitions where `m.rtc.slot` content becomes non-empty → slot open; becomes empty → slot + closed). +2. For each open slot interval \[`slot_start, slot_end`\], as given by `origin_server_ts` at the + slot status transitions: + * Collect `m.rtc.member` events whose connect/update/disconnect activity intersects the slot + interval. + * Cluster related `m.rtc.member` events by their `sticky_key` + * For each member event cluster the `membership_interval` \= \[`membership_start, + membership_end`\] is given by: + * `membership_start = max(start_slot_ts, start_content_ts)` as defined in [Lifecycle of a MatrixRTC Membership](#lifecycle-of-a-matrixrtc-membership) + * `membership_end = min(end_slot_ts, end_content_ts, end_sticky_ts)` as defined in [Lifecycle of a MatrixRTC Membership](#lifecycle-of-a-matrixrtc-membership) + * Drop all `membership_interval`s where `membership_start >= membership_end`. + * Merge all overlapping or contiguous `membership_interval`s into a single (continuous) + interval `collection`. Within a given slot, **MatrixRTC sessions cannot overlap**; all + intersecting intervals form a single unified session. +3. For each `collection` of overlapping `membership_interval`s, a (historical) MatrixRTC session + exists with the following properties: + * Slot identified by `slot_id` in the interval \[`slot_start, slot_end`\]. + * Starts at `min(m.membership_start for m in collection)` + * Ends at `max(m.membership_end for m in collection)` + * Individual MatrixRTC membership intervals given by `((m.membership_start, m.membership_end) for m in collection)` + +In general, there may be more than one `collection` of overlapping membership intervals, resulting +in multiple consecutive MatrixRTC sessions within a single slot. + +### MatrixRTC Transport {#matrixrtc-transport} + +MatrixRTC, by design, supports multiple transport implementations, each defined in its own MSC. +Transports describe how clients connect to peers or servers, manage connections, publish and +subscribe to media streams. A transport can represent different RTC backends such as SFUs, MCUs, or +full-mesh peer-to-peer topologies. + +Currently, [MSC4195: MatrixRTC Transport using LiveKit backend](https://github.com/matrix-org/matrix-spec-proposals/pull/4195) +is the primary proposal, while a full-mesh transport based on +[MSC3401](https://github.com/matrix-org/matrix-spec-proposals/pull/3401) is also planned. +Each transport defines its connection model, supported topologies, and any additional requirements +for participating clients. + +#### RTC Transports in `m.rtc.member` Events + +Each `m.rtc.member` event contains an `rtc_transports` array, which allows other clients to +determine how to connect to, and exchange real-time data with that participant. This array includes +at least one transport JSON object, each containing at least the following required field: + +* `type` (string) — Identifies the RTC transport. +* Additional fields may be present depending on the RTC transport `type`. + +The `versions` field of the `m.rtc.member` event SHOULD be used to determine which transports are +supported by all participants. Clients SHOULD publish their RTC data using at least one transport +supported by their application, constrained to the set of transports supported by all participants. +Clients MAY additionally publish their media via multiple transports. + +To render a MatrixRTC session, a client MUST be prepared to connect to as many transports as there +are members currently connected. For bandwidth efficiency, clients are recommended to subscribe to +only one transport per member. + +The exact procedures for subscribing to and publishing real-time data are defined in the dedicated +MSCs for each transport type. + +#### Discovery of RTC Transports + +MatrixRTC requires a mechanism for clients to discover which RTC transports — such as an SFU or TURN +server — are available in their Matrix deployment. To support this, homeservers expose an endpoint +that returns a list of JSON objects. Each transport-description object represents an RTC transport +as defined by the corresponding MSC. > [!NOTE] -> Backend **infrastructure** in this context can be anything that can serve as the backend for a -> MatrixRTC session. In most cases this is a SFU. But also a full mesh implementation could -> be an infrastructure. Not all kind of infrastructure require a way of sourcing a backend resource -> (e.g. full-mesh). In this MSC we only refer to infrastructure where it is necessary to have access to additional -> data to participate in the MatrixRTC session. +> RTC Transports in this context can be anything that can serve as the backend for a MatrixRTC +> session. In most cases this is a SFU. But also a full mesh implementation could be a transport. +> Not all kinds of RTC transports require a way of sourcing a backend resource (e.g. a +> peer-to-peer-solution). In this MSC we only refer to transport where it is necessary to have +> access to additional data to participate in the MatrixRTC session. -We use a `m.rtc_foci` key in the homeserver `.well-known/matrix/client` that can be used to expose -a sorted (by priority) list of Focus description objects. +The endpoint: `GET /_matrix/client/v1/rtc/transports` is used to expose a sorted (by priority) list +of Transport description objects. -For example in generic form: +Response format: -```json5 +```json { - "m.rtc_foci": [ + "rtc_transports": [ { - "type": "some-focus-type", - "additional-type-specific-field": "https://my_focus.domain", + "type": "some-transport-type", + "additional-type-specific-field": "https://my_transport.domain", "another-additional-type-specific-field": ["with", "Array", "type"] } ] } ``` -Or a concrete example for a `livekit` Focus: +Concrete example for a `livekit_multi_sfu` transport: -```json5 +```json { - "m.rtc_foci": [ + "rtc_transports": [ { - "type":"livekit", - "livekit_service_url":"https://livekit-jwt.call.element.io" + "type":"livekit_multi_sfu", + "livekit_service_url":"https://matrix-rtc.example.com/livekit/jwt" } ] } ``` -### The RTC application types - -Each application type might have its own specification in how the different streams -are interpreted and even what Focus type to use. This makes this proposal extremely -flexible. A Jitsi conference could be added by introducing a new `application` -and a new Focus type and would be MatrixRTC compatible. It would not be compatible -with applications that do not use the Jitsi Focus but clients would know that there -is an ongoing session of unknown type and unknown Focus and could display/represent -this in the user interface. - -To make it easy for clients to support different application types, the recommended -approach is to provide a Matrix widget for each application type. This way the -client developers can use the widget as the first implementation if they want to -support this RTC application type. - -Each application should get its own MSC in which the all the additional -fields are explained and how the communication with the possible foci is -defined: - -- `m.call` - voice and video conferencing described by [MSC4196](https://github.com/matrix-org/matrix-spec-proposals/pull/4196). - -#### Interoperability between applications - -There is a use-case in which a `m.call` app might want to participate in a session of type (application) `custom-call-with-more-features`. A native mobile matrix client might support `m.call` and is at hand to join the feature rich application/session. +**Fields:** -There could be fallback mechanisms but the most flexible approach is to treat it per application type. If it makes sense for an application type to fully conform to `m.call` a client that can connect to an `m.call` RTC session (application) could claim that it is also compatible with `custom-call-with-more-features` . It is than the job of the `custom-call-with-more-features` session type (application) to define some kind of feature list so that it can tell if users are joining with an m.call client or a dedicated `custom-call-with-more-features` client. -### End-to-end encryption of media streams +* `rtc_transports` — array of transport-description representing the available RTC transports + offered by the homeserver. +* Each object in the array MUST conform to the JSON schema defined for its `type` (e.g. + `livekit_multi_sfu` in MSC4195). -We define how the key material is shared between the participants of the call to facilitate end-to-end encryption of the media streams. +Clients SHOULD use this list to determine which RTC transports to connect to and may advertise their +selected transports according to the respective MSC in the `rtc_transports` field of their +`m.rtc.member` events. -The backend (e.g. LiveKit) MSC defines how the key material is actually used. +### End-to-end Encryption of RTC Data -#### Shared password +This section defines how key material is shared between participants in MatrixRTC to enable +end-to-end encryption of RTC data. -A shared password may be used to encrypt the media streams sent via the RTC backend that has been distributed ahead of time to the participants. +MatrixRTC sessions initiated in a **non-encrypted room** will remain non-encrypted. Sessions started +in **encrypted rooms** will be encrypted. -For example, it could be in the query parameter of a private URL attached to a calendar invitation. +Every member in an encrypted MatrixRTC slot maintains a unique sender key, which is securely shared +with all other members. This sender key (secure random bytes) MUST be used by the RTC application to +derive and stretch any secrets they need to encrypt/authenticate the data/media uploaded by this +user. -#### Per-participant sender key +Keys are distributed to the RTC members. The RTC membership is based on top of the room membership +and inherits its security properties. -A participant can share it's chosen key with other participants by sending Matrix [to-device messaging](https://spec.matrix.org/v1.11/client-server-api/#send-to-device-messaging) to the other participants. +The key exchange depends on the underlying secure peer-to-peer olm channel (via to-device message) +for key distribution. -The key is sent as an event of type `m.rtc.encryption_keys` as an encrypted to-device message. +This ensures that only members in the RTC slot can decrypt data/media; other people, even if in the +room, will never get the key material. -The device ID that is being sent to is the `member`.`device_id` from the `m.rtc.member` events. +The RTC transport MSC (e.g., +[MSC4195: MatrixRTC Transport using LiveKit backend](https://github.com/matrix-org/matrix-spec-proposals/pull/4195)) +specifies how the key material is actually used. For example, the RTC client may generate a secure +random 256-bit key, which the application can then use to derive the required secrets (using HKDF or +other key-stretching methods as appropriate). -The event contains the following fields: +#### Key Distribution {#key-distribution} -- `session` required object: The contents of the `session` from the `m.rtc.member` event. -- `member` required object: The contents of the `member` from the corresponding `m.rtc.member` event. -- `keys` required array of objects: The sender keys to be distributed to the participant: - - `key` required string: The base64 encoded key material. - - `index` required int: The index of the key to distinguish it from other keys. This must be a between 0 and 255 inclusive. - In some implementations of MatrixRTC this may correspond to the `keyID` field of the WebRTC [SFrame](https://www.w3.org/TR/webrtc-encoded-transform/#sframe) header. - - `invalidates_key_index` optional int: The index of the key that is invalidated by this key. If this is set, the application should invalidate the key identified - by `invalidates_key_index` once it receives a frame with the new `index`. This is to protect against an exfiltrated key being used to forge frames. - - `invalidates_after_ms` optional int: The number of milliseconds after the key identified by `invalidates_key_index` is invalidated by this key even if no frames - are received. Again, this is to protect against an exfiltrated key being used to forge frames. +When joining a MatrixRTC slot, each participant shares a generated E2EE key with the other members +(`m.rtc.member`) of the same slot by sending an **encrypted** [Matrix to-device +message](https://spec.matrix.org/v1.11/client-server-api/#send-to-device-messaging). The key is +transmitted via an event of type `m.rtc.encryption_key`. The **target device ID** is taken from the +`member`.`device_id` field of the recipient’s `m.rtc.member` event. The event follows the following +schema: -Depending on the RTC application, additional fields may be added to this event. - -An example to-device event: - -```json5 -// event type: "m.rtc.encryption_keys" +``` +// event type: "m.rtc.encryption_key" { - "session": { - "application": "m.call", - "call_id": "", - "scope": "m.room" - }, - "member": { - "id": "xyzABCDEF10123", - "device_id": "DEVICEID", - "user_id": "@user:matrix.domain" - }, "room_id": "!roomid:matrix.domain", - "keys": [ - { + "member.id": "xyzABCDEF0123" + "media_key": { "index": 10, - "key": "base64encodedkey", - "invalidates_key_index": 9, - "invalidates_after_ms": 5000 - }, - ], + "key": "base64encodedkey" + }, } ``` -On receipt of the `m.rtc.encryption_keys` event the application can associate the received key with the RTC session by matching the `session` and `member` contents with the corresponding `m.rtc.member` event. - -When the application joins the session it should send the key to all the existing participants. - -To ensure forward secrecy and post compromise security, the key material should be rotated (i.e. a new key generated) when a participant joins or leaves the session. +**Field explanations:** + +* `slot_id` required string: `slot_id` of the slot this key is related to. +* `member.id` required string: The `member.id` from the target `m.rtc.member` event. Note, because + `member.id` is globally unique per member instance, it is sufficient to disambiguate multiple key + events for the same device, even if they use the same `media_key.index` value. +* `media_key` The media key to use to decrypt the participant media: + * `key` required string: The base64 encoded key material. + * `index` required int: The index of the key to distinguish it from other keys. This must be + between 0 and 255 inclusive. In some implementations of MatrixRTC this may correspond to the + `keyID` field of the WebRTC [SFrame](https://www.w3.org/TR/webrtc-encoded-transform/#sframe) + header. + * `format` the key export format, `0` for the raw bytes base64 encoded. +* consecutive sessions within the same slot. +* Depending on the RTC application, additional fields may be added to this event. + +Upon receipt, any `m.rtc.encryption_key` to-device event sent in cleartext SHOULD be discarded. The +receiving client SHOULD use the Olm decryption metadata to determine the sender’s `user_id`, +`device_id`, and the device’s verification state. Sender information is considered *claimed* unless +the device is verified. + +The client SHOULD apply the same acceptance policy for these to-device messages as it would for room +messages or session setup. For example, if the client is configured to *exclude insecure devices*, +then keys received from such devices MUST also be excluded. + +The corresponding `m.rtc.member` is determined by matching its `member.id` with the one from +`m.rtc.encryption_key`. + +Clients SHOULD rotate their keys to ensure **confidentiality** whenever a participant joins or +leaves the slot. The **key rotation** process is as follows: + +* The sending application generates the new key material for the local participant. +* The sending application sends the new key material to all other participants with a new `index` value. +* The receiving application stores the new key material for the specified `index`. +* The sending application continues to use the old/current key to encrypt media. +* After a short delay `delayBeforeUse,` default: 5 seconds), the sending application switches to the + new key. + * It is possible to overwrite the default delay on a per application basis in case an application + has specific requirements on security or wants to minimize missed stream data + * Also negotiation can be used over data channels to confirm all participants have received the new key. + +**Note:** Rotating a key will require to re-send a to-device message to all the participants, and +this is expensive. Clients SHOULD minimize key exchange traffic for rapid joiners/leavers. + +For rapid new joiners: A key rotation grace period (`keyRotationGracePeriod`) is used, if a new +member joins during the grace period, then the same key can be used and shared just to that new +member.. + +For rapid leavers: The `delayBeforeUse` period is used to coalesce any membership change occurring +in that period and then a single key rotation is scheduled afterward. + +`keyRotationGracePeriod` must be greater than `delayBeforeUse` or it will have no effects (default +10s and 5s) + +A future version of key exchange could introduce ratcheting, this would reduce the key traffic for +new joiners (only sends the ratcheted key to the new joiners instead of rotating to all). + +#### Shared Key alternative + +For big calls in a room, it might be interesting to use a shared key system instead of a per-sender +key system. +In order to use a shared key, a new encrypted room event (`m.rtc.shared_encryption_key`) should be +sent in the room, and the slot should be updated to include the `event_id` of the shared key event. + +```json +// Example: an open slot with shared key encryption +{ + "application": { + "type": "m.call", + // optional: app specific slot metadata + "m.call.id": UUID, + "m.shared_key_event": "$000" + } +} +state_key: "m.call#ROOM" // slot_id +``` -Key rotation is done as follows: +The shared key event: +``` + "event_id": "$000", + "origin_server_ts": 1759827668867, + "type": "m.rtc.shared_encryption_key" + "content": { + "key": "base64encodedkey" + }, +``` -- the sending application generates the new key material for the participant. -- the sending application sends the new key material to all the participants with a new `index` value and `invalidates_key_index` set to the current `index`. -- the receiving application stores the new key material for the specified `index`. -- the sending application continues to use the old/current key to encrypt media. -- the sending application waits for a period of time. The default should be 3 seconds. - It is possible to overwrite this on a per application basis in case an application has specific requirements on security or wants to minimize missed stream data. - Also negotiation approaches can be defined where the RTC application uses data channels to communicate if everyone has received the next key. -- the sending application starts to use the new key to encrypt media. -- the receiving application invalidates the existing key with the `invalidates_key_index` value. +**Pros**: Scales for big calls as there is no need to distribute n keys via to-device and to rotate +keys on joiner/leavers +**Cons**: Less security, every room member (and future members) have the key materials even if they +didn’t actively join the call. No automatic key rotation, so if the shared key is compromised past +recording could be decrypted. -### Discovery/negotiation of application types +Only users with enough power level (moderator, admin) can update the slot in order to enable shared +keys call. -Problem: If a user wants to make a call to a user or room, then which call/application options should the client present to the user? +Clients should detect when the slot is configured for a shared key, and then use the shared key +instead of generating and sharing sender keys. -This should also take account of non-MatrixRTC calling: legacy 1:1 VoIP, room state widget for Jitsi. +### Scope and Responsibilities of MatrixRTC Applications -TODO: write up notes. +This MSC defines the **foundational protocol for real-time communication (RTC)** in Matrix — +establishing the primitives and data structures needed for RTC functionality. It deliberately **does +not define a complete calling experience**, leaving that to specific MatrixRTC applications that +build on top of this foundation. Each such application can define its own UX expectations, +signalling behaviour, and feature set. -## Potential issues +Implementations, especially chat clients, are expected to support one or more of these applications +as they are defined by future MSCs. +The first of these is expected to be the +**`m.call` application ([MSC4196](https://github.com/matrix-org/matrix-spec-proposals/pull/4196))**. +Clients SHOULD treat this application as a special case and present a familiar call experience that +abstracts away the underlying slot and membership model. -## Alternatives +Common UX patterns include: -### One state event per user +* A “Call” button in the room header. +* Ongoing call information displayed in the room header. +* Moderation-style wording for slot management, e.g. *“Allow users to start a call.”* +* Permission-related messaging, e.g. *“You are not allowed to start a call in this room.”* -[MSC3401](https://github.com/matrix-org/matrix-spec-proposals/pull/3401) proposed to have one state event per user with that state event containing an array of memberships. +This MSC’s goal is to define the **core RTC protocol**, not to specify every user-facing behaviour. +It delegates **user experience, call control, and signalling flows** to individual RTC applications +layered on top. For example, pre-call signalling (ringing, declining, etc.) is handled outside the +RTC session itself, using membership and slot application data and other events (TODO: link related +MSCs). -This introduces two problems: +Importantly, **MatrixRTC can be used by custom or experimental RTC applications** without those +applications being merged into the Matrix specification. The only difference is that a *merged* RTC +application may impose additional interoperability requirements on clients, whereas *non-merged* +applications can still operate using the same MatrixRTC primitives. -- potential inconsistency where one user device overwrites the state of another device during a concurrent update. -- when handling client disconnects the MSC3757 proposal could not be used as you would not know what the correct - state is at the time of the disconnect. +## Potential issues -### One state event per device +### Shared State for a MatrixRTC Session using Matrix Primitives + +In the context of Matrix’s eventual consistency, we evaluated options for representing shared state +in MatrixRTC and identified the following trade-offs: + +* For a **conflict-free approach**: +* We **avoid** shared state such as a session identifier altogether (at least not a priori). +* Each participant state is independent, and the session is computed as the aggregate of + `m.rtc.member` events. +* This sacrifices the simplicity of a UUID-like session identifier but avoids rollback problems due + to state resolution +* A good example of this approach is a room-based call or videoconference where the set of active + participants defines the session. +* For **shared state** (e.g., introducing a session identifier that all participants agree on): + * In any distributed system, if multiple participants attempt the same action at the same time, + there is a risk of *glare* (a race condition). One side will “win,” and the other may need to + roll back. This is not specific to Matrix: it is a well-known problem across telephony protocols + such as PSTN, GSM, SS7, SIP, or even early rotary exchanges. + * Using Matrix state resolution can lead to **rollbacks**, which is problematic for real-time + communication, since rolling back is not matching the semantics of real-time. + * The Matrix protocol does **not provide a consensus primitive** to prevent such rollbacks. + * As a result, any shared state implementation must accept the possibility of rollbacks and handle + them at the application layer. + +From this we conclude: both approaches are valid within MatrixRTC: + +* For the actual session state the **conflict-free** approach is well suited. It should be avoided + to rollback calls and should not allow a participant to initiate such a rollback and break the + current ongoing session. +* For permissions and (shared) metadata, **shared state** is the desired concept. By decoupling the + concept of a **session** (people communicating in a meeting room with a predefined `slot_id`) and + the **context** itself (the slot), we can have a conflict-free session concept but a centrally + managed context with potential state rollbacks that do not necessarily interfere with the session + itself. + +### MatrixRTC Membership Heartbeat Keep Alive + +If an RTC Transport is involved like a SFU then per definition the SFU has a very good understanding +of the connectivity of the individual participants. In that case it would be nice to be able to +delegate the ownership of the delayed leave event to that infrastructure component. However, +practical implementation has so far proven that the current implementation of delayed events are +also sufficient in adverse network conditions. So for now it's good enough and considered as an +optimisation for a future MSC. + +### Discovery and Negotiation of Application Types + +MatrixRTC does not currently define how clients should discover or negotiate which real-time +applications are available in a given room or between users. For example, when placing a call, it is +unclear whether the client should offer a MatrixRTC application, legacy 1:1 VoIP, or a room widget +such as Jitsi. Even in the case of similar applications (e.g., multiple call-capable MatrixRTC +apps), the proposal does not specify how a client should decide which one to launch. Application +discovery and negotiation are therefore left to a future MSC. + +### Interoperation Between Different RTC Transports + +MatrixRTC currently lacks a defined mechanism for interoperability across different RTC backends +(e.g., SFUs, MCUs, peer-to-peer). As a result, all participants in the same slot must share at least +one common transport implementation. This can lead to fragmentation if clients or Matrix sites +support disjoint sets of transports. A more complete interop solution is left as future work. + +### Impact of Message Ordering wrt. Session History {#impact-of-message-ordering-wrt.-session-history} + +As described above, all information required to reconstruct historic MatrixRTC sessions is available +within the Matrix DAG: + +* **Sticky events** are persistently recorded in the DAG. +* **`m.rtc.slot` state event transitions**, even when affected by state resolution, are visible as + part of the timeline. + +However, for this reconstruction process to function **reliably and deterministically**, the Matrix +protocol requires **consistent message ordering** on a **homeserver basis** across both state and +timeline events.This homeserver-centric ordering is crucial: due to netsplits or federation delays, +the room state may temporarily diverge between homeservers, even though it will eventually converge. +A concept such as [stitched +ordering](https://codeberg.org/andybalaam/stitched-order/src/branch/main/msc/msc.md) would satisfy +these requirements. Furthermore, **state resolution results** — even though they do not need to be +persisted in the DAG — **must be presented to clients as part of the timeline**. This ensures an +immutable and complete historical view per homeserver. + +To accurately represent RTC session history as perceived in-real-life by participants, ordering +should ideally rely on `received_server_ts` (which is not available today) rather than +`origin_server_ts`. While `origin_server_ts` can serve as a practical workaround for now, it does +not necessarily reflect the experienced order of events in the presence of latency, netsplits, or +federation lag. + +Counterintuitively, **state resolution for `m.rtc.slot` is not a problem** — RTC sessions occur in +real life, and their existence remains valid regardless of how room state later resolves. What *is* +important, however, is that each `m.rtc.slot` event, at its position in the DAG, **satisfies the +applicable authorisation rules** at that point in time to ensure the reconstructed session history +is consistent with valid room state transitions. -This would mean not using `member`.`id` in the state key anymore. Race conditions can be solved by the client which would need to manage multiple sessions at once. +## Alternatives -### A separate system not associated with Matrix accounts +### Represent MatrixRTC Members as regular Matrix State -This MSC proposes to combine the MatrixRTC backend infrastructure with the homeserver. -Other sources where the backend could be sourced from are: +Sticky events are used because they combine persistence with efficiency -- A separate system not associated with Matrix accounts. - (you would need a Matrix account + a "LiveKit provider" account for example) -- The client could bring its own backend link. -- A centralized solution. +* They do have similar delivery guarantees as state events and are reliably passed down the sync + loop for the duration of their stickiness – no pagination required in the event of gappy syncs +* They allow clients to compute the current membership (and session) state reliably without + contributing to long-term room state bloating. +* As they are persisted in the DAG like regular events, they can also be used to reconstruct + previous sessions, supporting session history and retrospective analysis. -The centralized solution would not fit to Matrix. A separate system would match the distributed -nature of Matrix but would not match the user experience goals for MatrixRTC calls. +Furthermore and In contrast to Matrix room state (as of writing this MSC), sticky events can be +already encrypted improving security significantly. -The client defining the SFU that is used, is the current solution. This causes the issue, that clients -in general are less distributed than homeservers. There is only a limited set of clients that a large -percentage of users use. -Using this as the source for the infrastructure would result in just a handful of very large infrastructure -hosts. -This is harder to scale and it is harder to justify who is covering the costs. (For Matrix homeservers, this -is an already solved problem where there are individuals, communities and institutions that have their own individual -solutions and answers for how and why they provide the infrastructure.) +### Organising MatrixRTC Members (`m.rtc.member`)as an Array per User -### `m.rtc.encryption_keys` room event +[MSC3401](https://github.com/matrix-org/matrix-spec-proposals/pull/3401) proposed to have one state +event per user with that state event containing an array of RTC memberships. -Earlier iterations of this MSC used an encrypted `m.rtc.encryption_keys` room event to distribute the per-participant sender keys. +This introduces two problems: -Whilst reducing traffic by only needing to send one event per participant, this approach does not allow for perfect forward secrecy -as the keys are stored in the room history. +- Potential inconsistency where one user device overwrites the state of another device during a + concurrent update. +- When handling client disconnects, +- [MSC4140 delayed events](https://github.com/matrix-org/matrix-spec-proposals/pull/4140) **cannot + reliably maintain an accurate membership state**. This is because, at the time a delayed event is + scheduled, the current membership array may change before the event is actually emitted, making it + impossible to predict the correct state in advance. + +### Using the Device Name as State Key for MatrixRTC Members (`m.rtc.member`) + +Using the client’s **device name** instead of `member.id` as the state key for the `m.rtc.member` +event may lead to **race conditions** if the same device joins a MatrixRTC slot multiple times. + +This scenario is realistic in cases such as: + +* multiple widgets joining the same slot to provide different perspectives on the session (e.g., + moderator view vs. participant view), or +* a native mobile application establishing an additional connection (e.g., alongside a webview) for + purposes such as screen sharing. + +### Alternative Transport Provisioning Models + +This MSC proposes that the Matrix site setup is also responsible for providing the MatrixRTC +infrastructure. However, other sources for providing MatrixRTC transport could be considered, +including: + +* **A separate system not associated with a Matrix account.** + For example, users might require both a Matrix account and a separate “LiveKit provider” account. + This approach is difficult to achieve across federation, since all users participating in a + MatrixRTC session would need an account at the external service provider. +* **Client-provided transport.** + The client itself could define and operate the SFU used for a session. + This approach represents a middle ground but introduces several challenges: most users rely on a + small number of popular clients, meaning only a handful of transport infrastructures would + ultimately serve the majority of traffic. + This concentration makes the system harder to scale and raises questions about cost, governance, + and accountability for maintaining the infrastructure. +* **A centralized solution.** + A single shared service could provide transport for all MatrixRTC users. + While simple to deploy, this model is **not Matrix-idiomatic**, as it contradicts the + decentralized design principles of the protocol. + +In contrast, Matrix homeservers already have diverse answers to the question of infrastructure +provision — individuals, communities, and institutions independently operate homeservers for their +own reasons and at their own expense. Replicating this **distributed sustainability model** for +MatrixRTC transport remains the most natural direction. + +### E2EE Key Distribution via Room Event (`m.rtc.encryption_keys`) + +Earlier iterations of this MSC used an encrypted `m.rtc.encryption_keys` room event to distribute +the per-participant sender keys. + +Whilst reducing traffic by only needing to send one event per participant to the homeserver, this +approach does not allow for perfect forward secrecy as the keys are persisted in the room history. The encrypted content of the `m.rtc.encryption_keys` event was as follows: -```json5 +``` { "session": { "application": "m.call", - "call_id": "" + "id": "" }, "member": { "id": "xyzABCDEF10123", @@ -594,43 +1093,72 @@ The encrypted content of the `m.rtc.encryption_keys` event was as follows: } ``` -## Security considerations +## Extensibility considerations + +This MSC introduces a completely new concept to Matrix. The proposal is designed to be as abstract +and flexible as possible. While it is expected to replace legacy Matrix calling mechanisms, it does +**not** attempt to replicate or redefine how traditional calls work. Instead, it focuses on modeling +the **core principles** required for real-time communication sessions. -### Discoverability of infrastructure +The design process drew heavily from real-world use cases — most notably the multi-year [Element +Call](https://github.com/element-hq/element-call) project. Other RTC application types were also +examined to identify common requirements and to test whether the proposal could be naturally +extended to support Matrix’s broader ecosystem of use cases.. -The `.well-known/matrix/client` is publicly readable, hence everyone can read and know -about the infrastructure which could lead to resource "stealing". -Each infrastructure however has their own authentication mechanism defined in the infrastructure specification. -Those mechanisms for instance can use a service to interact with the homeserver and based on that decide to allow users -to use the infrastructure. +### Slot constraints -This is defined in the respective infrastructure MSC. +Additional constraints may be defined for the `m.rtc.slot` event. Examples could include: + +* **matrix ID list** – Restricting participation to a specific set of users. +* **power level constraint** – Limiting access based on user power levels or roles. + +These and other potential constraints are **not** part of this MSC and will be addressed in a +follow-up proposal. However, this MSC has been designed to remain compatible with such extensions. +Future constraint definitions would likely exist alongside the `"application"` key within the +`m.rtc.slot` event structure. + +## Security considerations + +### Discoverability of RTC Infrastructure + +RTC infrastructure details are disseminated to all participants through `m.rtc.member` events. This +transparency means that anyone part of a MatrixRTC session can view and understand the +infrastructure, which could potentially lead to unauthorized resource use. However, each +infrastructure type defines its own authentication mechanisms, as detailed in its specific MSC. +These mechanisms may involve a service interacting with the homeserver to determine whether a user +is authorized to utilize the infrastructure. ### Forward secrecy for end-to-end encryption of media streams -The considerations to ensure forward secrecy are described in the [End-to-end encryption of media streams](#end-to-end-encryption-of-media-streams) -section above. +The considerations to ensure forward secrecy are described in the [Key +Distribution](#key-distribution) section above. ### End-to-end media encryption key rotation lag -The proposed key rotation semantics does mean that a participant could continue to decrypt media that was sent in the three seconds after -leaving the session. +The proposed key rotation semantics does mean that a participant could continue to decrypt media +that was sent in the three seconds after leaving the session. ## Unstable prefix -Use `org.matrix.msc3401.call.member` as the state event type in place of `m.rtc.member`. +Use `org.matrix.msc4143.rtc.member` as the state event type in place of `m.rtc.member`. -For discovery via `.well-known/matrix/client` the prefix `org.matrix.msc4158.rtc_foci` is used in place of `m.rtc_foci`. +For MatrixRTC Transport discovery via `GET` endpoint use +`/_matrix/client/unstable/org.matrix.msc4143/rtc/transports` instead of +`/_matrix/client/v1/rtc/transports` -Use `io.element.call.encryption_keys` in place of the `m.rtc.encryption_keys` room event and to-device event types. +Use `org.matrix.msc4143.rtc.encryption_key` in place of the `m.rtc.encryption_key` room event and +to-device event types. ## Dependencies -This proposal depends on -[MSC3757: Restricting who can overwrite a state event](https://github.com/matrix-org/matrix-spec-proposals/pull/3757) -to provide access control for the decentralised management of call membership state. However, an alternative such -as [MSC3779: "Owned" State Events](https://github.com/matrix-org/matrix-spec-proposals/pull/3779) could be used instead with -some adaptations. +This proposal depends on [MSC4354 Sticky +Events](https://github.com/matrix-org/matrix-spec-proposals/pull/4354) to provide room state similar +semantics without the drawback of contributing to room state bloating. + +This proposal also depends on [MSC4140: Cancellable delayed +events](https://github.com/matrix-org/matrix-spec-proposals/pull/4140) to provide a mechanism for +clients to ensure that they can update the room state even if they lose connection. -This proposal also depends on [MSC4140: Cancellable delayed events](https://github.com/matrix-org/matrix-spec-proposals/pull/4140) -to provide a mechanism for clients to ensure that they can update the room state even if they lose connection. +[^1]: Because `member.id` is generally unique, it serves as a reliable candidate for `sticky_key`, + preventing undesired collisions. Clients can use it to distinguish new member participation from + updates to existing RTC members. From 1fbd843e6b11e2ab0aa8b987384504a180b3d570 Mon Sep 17 00:00:00 2001 From: Timo K Date: Tue, 14 Oct 2025 14:53:50 +0200 Subject: [PATCH 22/62] json5 code block formats Signed-off-by: Timo K --- proposals/4143-matrix-rtc.md | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 8c8c5a5f037..bc13706e4b6 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -100,7 +100,7 @@ handle it with a sensible user interface. The minimum Application definition consists of a simple JSON object -```json +```json5 { "application": { "type": "m.call", // The # character MUST NOT appear in a valid type string. @@ -152,7 +152,7 @@ never be used to name a slot. A slot is opened by sending an `m.rtc.slot` state event with `state_key = slot_id`. This event authorises MatrixRTC members that intend to participate in the slot and follows the schema below: -```json +```json5 // Example: an open slot with application-specific metadata { "application": { @@ -179,7 +179,7 @@ The `slot_id` of an open slot acts like a virtual address where participants are The grammar for the `slot_id` is a well formed state key confined such that members of different MatrixRTC applications never occupy the same slot according to: -``` +```json5 {application.type}#{application_slot_id} ``` @@ -198,7 +198,7 @@ This grammar MUST never be used to parse `slot_ids`; it exists only to namespace To close a slot, the corresponding `m.rtc.slot` state event is updated with empty content which removes all remaining MatrixRTC members, for example: -```json +```json5 // Empty content represents a closed slot {} @@ -208,6 +208,7 @@ state_key: "m.call#ROOM" // slot_id #### Examples of MatrixRTC Slot Usage In the following example three different slot use-cases are depicted + ``` m.rtc.slot[id_1] {content_1} ...████████████████████████████████████████████... m.rtc.slot[id_2] {content_2} ███████████████████ @@ -297,7 +298,7 @@ An `m.rtc.member` sticky event can be either **Connected** or **Disconnected** A valid `m.rtc.member` event as a prerequisite for connecting to a slot has the following schema: -```json +```json5 // event type: "m.rtc.member" { "slot_id": "m.call#.ROOM", @@ -368,7 +369,7 @@ the event timing out (e.g., 5 minutes) to minimize potential connection state A valid `m.rtc.member` event as a prerequisite for disconnecting from a slot has the following schema: -``` +```json5 // event type: "m.rtc.member" { "slot_id": "m.call#ROOM", // MUST @@ -683,7 +684,7 @@ of Transport description objects. Response format: -```json +```json5 { "rtc_transports": [ { @@ -697,7 +698,7 @@ Response format: Concrete example for a `livekit_multi_sfu` transport: -```json +```json5 { "rtc_transports": [ { @@ -756,7 +757,7 @@ transmitted via an event of type `m.rtc.encryption_key`. The **target device ID* `member`.`device_id` field of the recipient’s `m.rtc.member` event. The event follows the following schema: -``` +```json5 // event type: "m.rtc.encryption_key" { "room_id": "!roomid:matrix.domain", @@ -832,7 +833,7 @@ key system. In order to use a shared key, a new encrypted room event (`m.rtc.shared_encryption_key`) should be sent in the room, and the slot should be updated to include the `event_id` of the shared key event. -```json +```json5 // Example: an open slot with shared key encryption { "application": { @@ -846,7 +847,8 @@ state_key: "m.call#ROOM" // slot_id ``` The shared key event: -``` + +```json5 "event_id": "$000", "origin_server_ts": 1759827668867, "type": "m.rtc.shared_encryption_key" @@ -1073,7 +1075,7 @@ approach does not allow for perfect forward secrecy as the keys are persisted in The encrypted content of the `m.rtc.encryption_keys` event was as follows: -``` +```json5 { "session": { "application": "m.call", From 60e23d90ea7bbdc8c50cbc875a01487c880901a8 Mon Sep 17 00:00:00 2001 From: Timo K Date: Tue, 14 Oct 2025 14:56:53 +0200 Subject: [PATCH 23/62] add unstable m.rtc.slot prefix. Signed-off-by: Timo K --- proposals/4143-matrix-rtc.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index bc13706e4b6..75c22645a9e 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -1142,7 +1142,8 @@ that was sent in the three seconds after leaving the session. ## Unstable prefix -Use `org.matrix.msc4143.rtc.member` as the state event type in place of `m.rtc.member`. +Use `org.matrix.msc4143.rtc.member` as the sticky event type in place of `m.rtc.member`. +Use `org.matrix.msc4143.rtc.slot` as the state event type in place of `m.rtc.slot`. For MatrixRTC Transport discovery via `GET` endpoint use `/_matrix/client/unstable/org.matrix.msc4143/rtc/transports` instead of From 680ef7dd65bbfbc275cc3177f36286e94c021fa9 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Wed, 15 Oct 2025 01:51:55 +0200 Subject: [PATCH 24/62] fix grammar & link the MSC --- proposals/4143-matrix-rtc.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 75c22645a9e..b82da2dd56e 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -1,7 +1,9 @@ # MSC4143: MatrixRTC -This MSC defined Matrix real-time communication, in short MatrixRTC. This is the base layer to build -real-time systems on top of Matrix. +## Overview + +This MSC defines MatrixRTC: how to set up real-time communication sessions over Matrix. +This is the base layer to build real-time systems on top of Matrix. MatrixRTC specifies how a real-time session is described in a room and how Matrix users can connect to a session. @@ -28,14 +30,15 @@ The MatrixRTC specification is separated into different modules: during the session * This MSC also specifies how E2EE keys are distributed -This MSC focuses on the three aspects above, the other modules are defined by accompanying MSC's: +This MSC focuses on the three aspects above, the other modules are defined by accompanying MSCs: * **MatrixRTC Transports** * Supports multiple transport implementations * Defines how to connect to peers or servers, update connections, and subscribe to streams - * A LiveKit-based transport is the current standard proposal (MSC4195: MatrixRTC Transport using + * A LiveKit-based transport is the current standard proposal + ([MSC4195](https://github.com/matrix-org/matrix-spec-proposals/pull/4195): MatrixRTC Transport using LiveKit backend) - * Another planned transport is a full-mesh implementation based on + * Another planned transport is a full-mesh WebRTC implementation based on [MSC3401](https://github.com/matrix-org/matrix-spec-proposals/pull/3401) * **MatrixRTC Applications** * Each application type can have its own specification From b62f75b21db2c683e9093c0bd9fdc37afb3cc8b6 Mon Sep 17 00:00:00 2001 From: fkwp Date: Fri, 12 Dec 2025 17:11:54 +0100 Subject: [PATCH 25/62] more informatino on the purpose of `member.id` --- proposals/4143-matrix-rtc.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index b82da2dd56e..ce1b12e9dee 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -347,9 +347,10 @@ A valid `m.rtc.member` event as a prerequisite for connecting to a slot has the accurate historical reconstruction, and allow deriving the participant’s start time using the `origin_server_ts` of the referenced event. * `member` — Uniquely identifies this participation instance; includes: - * `id` — UUID to distinguish multiple participants, even for the same device. This ID can also - serve as a canonical identifier for certain MatrixRTC transports, helping to prevent metadata - leakage. The ID MUST be unique for each connect event. + * `id` — UUID to distinguish multiple participations, even for the same user and same device. This + ID can also serve as a canonical identifier for certain MatrixRTC transports, helping to prevent + metadata leakage if used as additional entropy, e.g., `SHA-256(user_id|device_id|member.id)`. + The ID MUST be unique for each connect event. * `claimed_device_id` — Matrix device identifier. * `claimed_user_id` — Matrix user ID. * `rtc_transports` — List of objects describing how to access this participant’s media streams. See From 50f83318669f8c8b98f92c559490c4a663f4cd6f Mon Sep 17 00:00:00 2001 From: fkwp Date: Fri, 12 Dec 2025 17:15:24 +0100 Subject: [PATCH 26/62] markdown cleanup --- proposals/4143-matrix-rtc.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index ce1b12e9dee..aa5e554ba3f 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -408,7 +408,7 @@ A valid `m.rtc.member` event as a prerequisite for disconnecting from a slot has the slot) This ensures that the disconnection reason refers to a real connection lifecycle rather than a pre-join cancellation. -`disconnect_reason` F**ield explanations:** +`disconnect_reason` **Field explanations:** | Field | Type | Required | Description | | ----- | ----- | ----- | ----- | @@ -437,7 +437,7 @@ A valid `m.rtc.member` event as a prerequisite for disconnecting from a slot has example, if the user is busy or declines a MatrixRTC session), a dedicated **sticky event** is required to convey the participant’s status. -#### Lifecycle of a MatrixRTC Membership {#lifecycle-of-a-matrixrtc-membership} +#### Lifecycle of a MatrixRTC Membership The MatrixRTC membership is a collection of linked `m.rtc.member` events. With the definition of a **Connected** and a **Disconnected** MatrixRTC membership from above: @@ -591,7 +591,7 @@ their existence remains valid regardless of how the room state later resolves. However, for this reconstruction process to work **reliably and deterministically**, the Matrix protocol requires **consistent message ordering (on a homeserver basis)** across both state and timeline events. This remains an open problem and is discussed further in the -[*Impact of Message Ordering wrt. Session History*](#impact-of-message-ordering-wrt.-session-history) +[*Impact of Message Ordering wrt. Session History*](#impact-of-message-ordering-wrt-session-history) section. As a temporary workaround, the reconstruction process uses `origin_server_ts` ordering. General concept @@ -635,7 +635,7 @@ Step-by-step reconstruction algorithm In general, there may be more than one `collection` of overlapping membership intervals, resulting in multiple consecutive MatrixRTC sessions within a single slot. -### MatrixRTC Transport {#matrixrtc-transport} +### MatrixRTC Transport MatrixRTC, by design, supports multiple transport implementations, each defined in its own MSC. Transports describe how clients connect to peers or servers, manage connections, publish and @@ -752,7 +752,7 @@ specifies how the key material is actually used. For example, the RTC client may random 256-bit key, which the application can then use to derive the required secrets (using HKDF or other key-stretching methods as appropriate). -#### Key Distribution {#key-distribution} +#### Key Distribution When joining a MatrixRTC slot, each participant shares a generated E2EE key with the other members (`m.rtc.member`) of the same slot by sending an **encrypted** [Matrix to-device @@ -968,7 +968,7 @@ MatrixRTC currently lacks a defined mechanism for interoperability across differ one common transport implementation. This can lead to fragmentation if clients or Matrix sites support disjoint sets of transports. A more complete interop solution is left as future work. -### Impact of Message Ordering wrt. Session History {#impact-of-message-ordering-wrt.-session-history} +### Impact of Message Ordering wrt. Session History As described above, all information required to reconstruct historic MatrixRTC sessions is available within the Matrix DAG: From d53175b48b0b615c38482da08aa155254038fc2b Mon Sep 17 00:00:00 2001 From: Robin Date: Fri, 23 Jan 2026 15:41:31 +0100 Subject: [PATCH 27/62] Remove stray . from slot ID --- proposals/4143-matrix-rtc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index aa5e554ba3f..cd18832a3fa 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -304,7 +304,7 @@ A valid `m.rtc.member` event as a prerequisite for connecting to a slot has the ```json5 // event type: "m.rtc.member" { - "slot_id": "m.call#.ROOM", + "slot_id": "m.call#ROOM", "application": { "type": "m.call", // further fields for the application (optional) From dc605bfb899310a2b710741a5c8deb96131fb6e7 Mon Sep 17 00:00:00 2001 From: fkwp Date: Fri, 23 Jan 2026 16:39:17 +0100 Subject: [PATCH 28/62] cleanup field description for to-device message --- proposals/4143-matrix-rtc.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index cd18832a3fa..9625b9bdb1a 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -785,8 +785,7 @@ schema: between 0 and 255 inclusive. In some implementations of MatrixRTC this may correspond to the `keyID` field of the WebRTC [SFrame](https://www.w3.org/TR/webrtc-encoded-transform/#sframe) header. - * `format` the key export format, `0` for the raw bytes base64 encoded. -* consecutive sessions within the same slot. +* `format` the key export format, `0` for the raw bytes base64 encoded. * Depending on the RTC application, additional fields may be added to this event. Upon receipt, any `m.rtc.encryption_key` to-device event sent in cleartext SHOULD be discarded. The From 7907635a54080fd35739910818e26f5d6edbe8c3 Mon Sep 17 00:00:00 2001 From: fkwp Date: Fri, 23 Jan 2026 16:39:41 +0100 Subject: [PATCH 29/62] minor changes to the m.rtc.encryption_key to-device format --- proposals/4143-matrix-rtc.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 9625b9bdb1a..ffe467b8a00 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -765,18 +765,18 @@ schema: // event type: "m.rtc.encryption_key" { "room_id": "!roomid:matrix.domain", - "member.id": "xyzABCDEF0123" + "member_id": "xyzABCDEF0123", "media_key": { - "index": 10, - "key": "base64encodedkey" + "index": 10, + "key": "base64encodedkey", }, + "version": "0" } ``` **Field explanations:** -* `slot_id` required string: `slot_id` of the slot this key is related to. -* `member.id` required string: The `member.id` from the target `m.rtc.member` event. Note, because +* `member_id` required string: The `member.id` from the target `m.rtc.member` event. Note, because `member.id` is globally unique per member instance, it is sufficient to disambiguate multiple key events for the same device, even if they use the same `media_key.index` value. * `media_key` The media key to use to decrypt the participant media: From c032f039216b02d80b5c897633cc6995f0d678b6 Mon Sep 17 00:00:00 2001 From: Robin Date: Wed, 11 Mar 2026 20:26:14 +0100 Subject: [PATCH 30/62] Update transport type to match latest MSC4195 revision --- proposals/4143-matrix-rtc.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index ffe467b8a00..9c86838b345 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -700,14 +700,14 @@ Response format: } ``` -Concrete example for a `livekit_multi_sfu` transport: +Concrete example for a `livekit` transport: ```json5 { "rtc_transports": [ { - "type":"livekit_multi_sfu", - "livekit_service_url":"https://matrix-rtc.example.com/livekit/jwt" + "type": "livekit", + "livekit_service_url": "https://matrix-rtc.example.com/livekit/jwt" } ] } @@ -718,7 +718,7 @@ Concrete example for a `livekit_multi_sfu` transport: * `rtc_transports` — array of transport-description representing the available RTC transports offered by the homeserver. * Each object in the array MUST conform to the JSON schema defined for its `type` (e.g. - `livekit_multi_sfu` in MSC4195). + `livekit` in MSC4195). Clients SHOULD use this list to determine which RTC transports to connect to and may advertise their selected transports according to the respective MSC in the `rtc_transports` field of their From 866ca090e95e17b82da4614c23e9cdd24aa5466e Mon Sep 17 00:00:00 2001 From: fkwp Date: Wed, 15 Apr 2026 11:45:25 +0200 Subject: [PATCH 31/62] Require authentication for MatrixRTC transport discovery endpoint --- proposals/4143-matrix-rtc.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 9c86838b345..e663c178053 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -672,9 +672,11 @@ MSCs for each transport type. #### Discovery of RTC Transports MatrixRTC requires a mechanism for clients to discover which RTC transports — such as an SFU or TURN -server — are available in their Matrix deployment. To support this, homeservers expose an endpoint -that returns a list of JSON objects. Each transport-description object represents an RTC transport -as defined by the corresponding MSC. +server — are available in their Matrix deployment. To support this, homeservers expose an +authenticated endpoint that returns a list of JSON objects. The endpoint is authenticated as the RTC +transport information is only useful to local clients; remote clients obtain transport information +via `m.rtc.member` events. Each transport-description object represents an RTC transport as defined +by the corresponding MSC. > [!NOTE] > RTC Transports in this context can be anything that can serve as the backend for a MatrixRTC From 3855617a4ffe67ef2186397253ff5f4716818481 Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Fri, 5 Jun 2026 13:48:04 +0200 Subject: [PATCH 32/62] Fix [!NOTE] syntax --- proposals/4143-matrix-rtc.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index e663c178053..b60f4d454b0 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -135,7 +135,8 @@ members form a MatrixRTC session. Each slot is identified by a unique `slot_id`, `application` type, and represented by the `m.rtc.slot` state event with the `slot_id` as the state key. -> [!NOTE] Conceptually, a slot is like a virtual meeting room: MatrixRTC sessions occur within +> [!NOTE] +> Conceptually, a slot is like a virtual meeting room: MatrixRTC sessions occur within > slots, and multiple sessions may occur consecutively without changing the slot’s configuration. Slots are part of the shared room state and can only be created or modified by authorized users with From be629da3e879c1a11a50e29e2aba6433752d9960 Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Fri, 5 Jun 2026 13:48:56 +0200 Subject: [PATCH 33/62] Use "power level" rather than "`power_level`" to avoid ambiguity --- proposals/4143-matrix-rtc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index b60f4d454b0..dc5365e9f4d 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -140,7 +140,7 @@ key. > slots, and multiple sessions may occur consecutively without changing the slot’s configuration. Slots are part of the shared room state and can only be created or modified by authorized users with -sufficient `power_level`. Clients MUST react on and respect latest `m.rtc.slot` definitions as +sufficient power level. Clients MUST react on and respect latest `m.rtc.slot` definitions as defined by the Matrix room state at all times. The ability to open and close slots with different patterns enables a wide variety of use cases, from always-on shared spaces to short lived scheduled meetings. From 9c17cc624fd005538f023cc9f4e75b633cd6eb78 Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Mon, 8 Jun 2026 09:19:40 +0200 Subject: [PATCH 34/62] Avoid using MUST with respect to creating MSCs per application --- proposals/4143-matrix-rtc.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index dc5365e9f4d..6321ebf8902 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -117,8 +117,9 @@ The `type` field MUST be a string that does not contain the `#` character. The t follow the [*Common Namespaced Identifier Grammar*](https://spec.matrix.org/v1.16/appendices/#common-namespaced-identifier-grammar). -Each application type MUST have its own MSC, which specifies the additional fields and defines how -communication with the corresponding transport is handled. For example: +Each application type specifies the additional fields and defines how communication with the +corresponding transport is handled. Concrete applications types are introduced in separate proposals +such as: * `m.call` — voice and video calling, as defined in [MSC4196](https://github.com/matrix-org/matrix-spec-proposals/pull/4196). From 29a3f30b207f6492889034a3731002e3eb6e67bf Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Mon, 8 Jun 2026 09:23:53 +0200 Subject: [PATCH 35/62] Remove recommendation to provide widget implementations for application types --- proposals/4143-matrix-rtc.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 6321ebf8902..bfc59e63494 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -124,11 +124,6 @@ such as: * `m.call` — voice and video calling, as defined in [MSC4196](https://github.com/matrix-org/matrix-spec-proposals/pull/4196). -To facilitate interoperability, ideally each application type should provide a Matrix widget that -can serve as a reference implementation for clients. This allows client developers to support new -application types by embedding or integrating the widget, without having to implement the full -application logic themselves. - ### MatrixRTC Slot and Constraining Slots A **MatrixRTC slot** is the container for MatrixRTC members and temporal overlapping MatrixRTC From e3083de2d8997354692da0d514d999846d9782a3 Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Mon, 8 Jun 2026 10:10:27 +0200 Subject: [PATCH 36/62] Clarify that RTC member refers to m.rtc.member events --- proposals/4143-matrix-rtc.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index bfc59e63494..60051111ea7 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -172,8 +172,9 @@ state_key: "m.call#ROOM" // slot_id as the slot name. * `application` An application JSON object, which **MUST** specify the application type and MAY include additional fields which **constrain** the application (e.g., restricting a call to be - voice-only). **Note** those additional fields can be used in combination with MatrixRTC member - ones subject to the application specific MSC. + voice-only). **Note** those additional fields may be extended through fields in the `application` + content block of `m.rtc.member` events. If a field occurs in both, the value from the `m.rtc.slot` event + takes precedence. The `slot_id` of an open slot acts like a virtual address where participants are allowed to meet. The grammar for the `slot_id` is a well formed state key confined such that members of different From f92827386e93c564459c617bacb04a214ce46d61 Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Mon, 8 Jun 2026 10:11:00 +0200 Subject: [PATCH 37/62] Fix number of states --- proposals/4143-matrix-rtc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 60051111ea7..26e0eeb97e2 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -234,7 +234,7 @@ Time ───────────── #### Recommended User Experience -The lifecycle of a slot follows four distinct states: +The lifecycle of a slot follows three distinct states: ``` Closed From 730296fc19fcbcdae224d5056f26788791d62f31 Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Mon, 8 Jun 2026 10:13:29 +0200 Subject: [PATCH 38/62] Downgrade UX description to examples --- proposals/4143-matrix-rtc.md | 24 ++++++++---------------- 1 file changed, 8 insertions(+), 16 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 26e0eeb97e2..6fa6911b0e6 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -246,22 +246,14 @@ Open Slots may transition between these states as a slot is opened or closed, or as users connect or disconnect. While **participation** in a MatrixRTC session is possible if a **slot is open**, **management** of -slots depends on the user’s **power level**. Clients MUST respect these distinctions in the user -experience. - -For **open** slots in a room, clients SHOULD clearly represent the current slot’s activity state: - -* **Active:** clients SHOULD indicate which slots are currently active in a room (e.g., by showing - participant count, showing an icon, list entry, or join button). -* **Inactive:** clients SHOULD indicate that the slot is available and provide a way for a user to - join as the first member (e.g., via a “Call” button or placeholder icon). - -**Slot management** UX SHOULD only exposed to users who have sufficient power level to manage slots: - -* **Closed slot:** The UX SHOULD provide controls to manage the slot (e.g., opening or modifying the - slot) -* **Open slot:** The UX SHOULD provide controls to perform management (e.g., closing or modifying - the slot) +slots depends on the user’s **power level**. Clients should be aware of this distinction when constructing +their user interfaces. + +For **open** and **active** slots, for instance, clients could display the participant count, an icon, +a list entry, or join button. For **open** but **inactive** slots, in turn, clients could indicate that the +slot is available and provide a way for a user to join as the first member (e.g., via a “Call” button or +placeholder icon). Finally, for users who have sufficient power level to manage slots, clients could +display a slot management UI with controls to open, close or modify slots. Multiple active slots may exist in the same room if the application type supports them. From c7fcea0a00a1bafe621dd02883f0fcc6cf939373 Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Mon, 8 Jun 2026 10:18:12 +0200 Subject: [PATCH 39/62] Remove duplicate paragraph --- proposals/4143-matrix-rtc.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 6fa6911b0e6..8284320ea8d 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -575,9 +575,6 @@ In general, all required information is present as part of the Matrix DAG: * **`m.rtc.slot` state event transitions**, even when affected by state resolution, are visible as part of the timeline. -Counterintuitively, **state resolution is not a problem** — RTC sessions occurred in real life, and -their existence remains valid regardless of how the room state later resolves. - However, for this reconstruction process to work **reliably and deterministically**, the Matrix protocol requires **consistent message ordering (on a homeserver basis)** across both state and timeline events. This remains an open problem and is discussed further in the From 72f3c4ca0e6d437a7bdc8e3ef0f4d5f2b4b91678 Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Mon, 8 Jun 2026 10:19:22 +0200 Subject: [PATCH 40/62] Linkify --- proposals/4143-matrix-rtc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 8284320ea8d..4d02526bc23 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -707,7 +707,7 @@ Concrete example for a `livekit` transport: * `rtc_transports` — array of transport-description representing the available RTC transports offered by the homeserver. * Each object in the array MUST conform to the JSON schema defined for its `type` (e.g. - `livekit` in MSC4195). + `livekit` in [MSC4195](https://github.com/matrix-org/matrix-spec-proposals/pull/4195)). Clients SHOULD use this list to determine which RTC transports to connect to and may advertise their selected transports according to the respective MSC in the `rtc_transports` field of their From 6dd4f50d827b46c6d83dcac2c93b45a3e19a541a Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Mon, 8 Jun 2026 10:20:16 +0200 Subject: [PATCH 41/62] Fix property name --- proposals/4143-matrix-rtc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 4d02526bc23..ad5aae1f26c 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -747,7 +747,7 @@ When joining a MatrixRTC slot, each participant shares a generated E2EE key with (`m.rtc.member`) of the same slot by sending an **encrypted** [Matrix to-device message](https://spec.matrix.org/v1.11/client-server-api/#send-to-device-messaging). The key is transmitted via an event of type `m.rtc.encryption_key`. The **target device ID** is taken from the -`member`.`device_id` field of the recipient’s `m.rtc.member` event. The event follows the following +`member.claimed_device_id` field of the recipient’s `m.rtc.member` event. The event follows the following schema: ```json5 From 73efe9387fd966202bc590e2b2798500b327b1fb Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Mon, 8 Jun 2026 10:22:39 +0200 Subject: [PATCH 42/62] Fix bullets --- proposals/4143-matrix-rtc.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index ad5aae1f26c..79cb43af258 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -1012,8 +1012,7 @@ This introduces two problems: - Potential inconsistency where one user device overwrites the state of another device during a concurrent update. -- When handling client disconnects, -- [MSC4140 delayed events](https://github.com/matrix-org/matrix-spec-proposals/pull/4140) **cannot +- When handling client disconnects, [MSC4140 delayed events](https://github.com/matrix-org/matrix-spec-proposals/pull/4140) **cannot reliably maintain an accurate membership state**. This is because, at the time a delayed event is scheduled, the current membership array may change before the event is actually emitted, making it impossible to predict the correct state in advance. From d840122b746ee1b46dcc848f95caff71e12d442a Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Mon, 8 Jun 2026 13:03:02 +0200 Subject: [PATCH 43/62] Relocate slot ID grammar Co-authored-by: Johannes Marbach --- proposals/4143-matrix-rtc.md | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 79cb43af258..8bd3ff69df9 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -141,12 +141,6 @@ defined by the Matrix room state at all times. The ability to open and close slo patterns enables a wide variety of use cases, from always-on shared spaces to short lived scheduled meetings. -Slots are named by the RTC app, but are expected to typically be ordinals (e.g. equivalent to line 1 -or line 2 on a telephone). If the app needs to define its slots out of band (e.g. mapping them to -widget IDs) then it can use those IDs as names. However, given a slot is the mechanism around which -sessions converge, it must have a predictable name. Unpredictable IDs such as session IDs should -never be used to name a slot. - #### Opening a MatrixRTC Slot A slot is opened by sending an `m.rtc.slot` state event with `state_key = slot_id`. This event @@ -188,9 +182,11 @@ Where * `application.type` is the `type` field in the application JSON object * The `#` character MUST NOT be used in either `application.type` or `application_slot_id` -* `application_slot_id` is the application-specific slot ID. Each application MSC defines its own +* `application_slot_id` is the application-specific slot ID. Each application defines its own schema (e.g., `ROOM`, `1`, `2`) to allow multiple parallel slots of the same type according to the - application requirements. + application requirements. If the application needs to define its slots out of band (e.g. mapping them to + widget IDs) then it can use those IDs as `application_slot_id`. However, given a slot is the mechanism + around which sessions converge, it MUST have a predictable `application_slot_id`. This grammar MUST never be used to parse `slot_ids`; it exists only to namespace the `state_key`. From 535138abc184cb4d26fc67479c209b548986add7 Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Tue, 9 Jun 2026 19:50:59 +0200 Subject: [PATCH 44/62] Avoid "slot name" term --- proposals/4143-matrix-rtc.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 8bd3ff69df9..4a3d5f7db18 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -162,8 +162,7 @@ state_key: "m.call#ROOM" // slot_id **Field description:** -* **state key**: The state key of the `m.rtc.slot` state event, referred to as the `slot_id`, serves - as the slot name. +* **state key**: The state key of the `m.rtc.slot` state event, referred to as the `slot_id`. * `application` An application JSON object, which **MUST** specify the application type and MAY include additional fields which **constrain** the application (e.g., restricting a call to be voice-only). **Note** those additional fields may be extended through fields in the `application` From 118c9dbe0c33427410badd5cf5c1e4c7837534ec Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Mon, 15 Jun 2026 15:24:56 +0200 Subject: [PATCH 45/62] Clarify common namespaced identifier requirements Signed-off-by: Johannes Marbach --- proposals/4143-matrix-rtc.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 4a3d5f7db18..fc80724ce92 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -113,8 +113,7 @@ The minimum Application definition consists of a simple JSON object } ``` -The `type` field MUST be a string that does not contain the `#` character. The type field SHOULD -follow the [*Common Namespaced Identifier +The `type` is a unique identifier for the application and MUST follow the [*Common Namespaced Identifier Grammar*](https://spec.matrix.org/v1.16/appendices/#common-namespaced-identifier-grammar). Each application type specifies the additional fields and defines how communication with the @@ -163,7 +162,7 @@ state_key: "m.call#ROOM" // slot_id **Field description:** * **state key**: The state key of the `m.rtc.slot` state event, referred to as the `slot_id`. -* `application` An application JSON object, which **MUST** specify the application type and MAY +* `application` An application JSON object, which **MUST** specify the application `type` and MAY include additional fields which **constrain** the application (e.g., restricting a call to be voice-only). **Note** those additional fields may be extended through fields in the `application` content block of `m.rtc.member` events. If a field occurs in both, the value from the `m.rtc.slot` event @@ -180,13 +179,17 @@ MatrixRTC applications never occupy the same slot according to: Where * `application.type` is the `type` field in the application JSON object -* The `#` character MUST NOT be used in either `application.type` or `application_slot_id` -* `application_slot_id` is the application-specific slot ID. Each application defines its own - schema (e.g., `ROOM`, `1`, `2`) to allow multiple parallel slots of the same type according to the +* `application_slot_id` is the application-specific slot ID. The value MUST follow the + [*Common Namespaced Identifier Grammar*](https://spec.matrix.org/v1.16/appendices/#common-namespaced-identifier-grammar) + but without the namespacing requirements. Each application defines its own + schema (e.g., `ROOM`, `line1`, `line2`) to allow multiple parallel slots of the same type according to the application requirements. If the application needs to define its slots out of band (e.g. mapping them to widget IDs) then it can use those IDs as `application_slot_id`. However, given a slot is the mechanism around which sessions converge, it MUST have a predictable `application_slot_id`. +Note that due the use of the [*Common Namespaced Identifier Grammar*](https://spec.matrix.org/v1.16/appendices/#common-namespaced-identifier-grammar), +neither `application.type` nor `application_slot_id` can contain the `#` character. + This grammar MUST never be used to parse `slot_ids`; it exists only to namespace the `state_key`. #### Closing a MatrixRTC Slot From 9852003412d1557d474503bc6f4b50bab5e47c4a Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Mon, 15 Jun 2026 15:34:36 +0200 Subject: [PATCH 46/62] Clarify why we require stickyness to consider a member connected Signed-off-by: Johannes Marbach --- proposals/4143-matrix-rtc.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index fc80724ce92..317f7c14a04 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -277,9 +277,13 @@ An `m.rtc.member` sticky event can be either **Connected** or **Disconnected** * **Slot open**: the `sticky_key` of the event needs to match the `member.id` AND the `slot_id` matches the state\_key of the slot event. * **Content:** MUST match the JSON content schema for connecting to a slot (see below). - * The sender is still a member of the room (not kicked / left) + * The sender is still a member of the room (not kicked / left). * **Event is sticky**: The sticky event needs not to be expired as described in [MSC4354 Sticky - Events](https://github.com/matrix-org/matrix-spec-proposals/pull/4354) + Events](https://github.com/matrix-org/matrix-spec-proposals/pull/4354). This is to ensure + that the membership view is as consistent as possible across all participants. When a member + event's stickyness expires, the associated delivery guarantee vanishes. As a result, some + participants might not have received the event while others did resulting in inconsistent + call memberships. * **Disconnected**, if * **Not Connected:** any of the required conditions to be connected are not met. * **Content:** SHOULD match the JSON content schema for disconnecting from a slot (see below). If From 3ccd90b57ab4a36b2cf80a27f577a46f5540be25 Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Mon, 15 Jun 2026 15:43:25 +0200 Subject: [PATCH 47/62] Remove vulnerable hashing example Signed-off-by: Johannes Marbach --- proposals/4143-matrix-rtc.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 317f7c14a04..2c0c068d0f0 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -339,10 +339,9 @@ A valid `m.rtc.member` event as a prerequisite for connecting to a slot has the accurate historical reconstruction, and allow deriving the participant’s start time using the `origin_server_ts` of the referenced event. * `member` — Uniquely identifies this participation instance; includes: - * `id` — UUID to distinguish multiple participations, even for the same user and same device. This - ID can also serve as a canonical identifier for certain MatrixRTC transports, helping to prevent - metadata leakage if used as additional entropy, e.g., `SHA-256(user_id|device_id|member.id)`. - The ID MUST be unique for each connect event. + * `id` — Identifier to distinguish multiple participations, even for the same user and same device + MUST be unique for each connect event. MatrixRTC transports MAY use `id` as the canonical identifier + to help prevent leaking metadata such as the user ID or device ID to external services. * `claimed_device_id` — Matrix device identifier. * `claimed_user_id` — Matrix user ID. * `rtc_transports` — List of objects describing how to access this participant’s media streams. See From 85872970992dcef3c7482a04dfae958594e7c722 Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Tue, 16 Jun 2026 10:33:17 +0200 Subject: [PATCH 48/62] Fix spelling Signed-off-by: Johannes Marbach --- proposals/4143-matrix-rtc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 2c0c068d0f0..5497d7b971f 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -281,7 +281,7 @@ An `m.rtc.member` sticky event can be either **Connected** or **Disconnected** * **Event is sticky**: The sticky event needs not to be expired as described in [MSC4354 Sticky Events](https://github.com/matrix-org/matrix-spec-proposals/pull/4354). This is to ensure that the membership view is as consistent as possible across all participants. When a member - event's stickyness expires, the associated delivery guarantee vanishes. As a result, some + event's stickiness expires, the associated delivery guarantee vanishes. As a result, some participants might not have received the event while others did resulting in inconsistent call memberships. * **Disconnected**, if From d597297b3b380b7eaa9bf17766600513a99ed05a Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Tue, 16 Jun 2026 10:54:10 +0200 Subject: [PATCH 49/62] Clarify origin of disconnect_reason design and remove examples as they belong into MSC4196 Signed-off-by: Johannes Marbach --- proposals/4143-matrix-rtc.md | 42 +++++++++++++++++------------------- 1 file changed, 20 insertions(+), 22 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 5497d7b971f..2e3a945e719 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -401,28 +401,26 @@ A valid `m.rtc.member` event as a prerequisite for disconnecting from a slot has `disconnect_reason` **Field explanations:** -| Field | Type | Required | Description | -| ----- | ----- | ----- | ----- | -| `class` | string | ✅ | High-level category of the disconnection or error. | -| `reason` | string | ✅ | Machine-readable identifier of the specific cause. | -| `description` | string | ⚪ | Optional human-readable explanation providing additional context. | - -**Class categories and examples:** - -| Class | Example Reason | Description / When Used | -| ----- | ----- | ----- | -| user\_action | `hangup` | Participant intentionally ended the call after joining. | -| | `switch_device` | User moved the session to another device mid-call. | -| client\_error | `media_error` | Failed to capture or transmit audio/video after joining. | -| | `transport_failure` | Local ICE/DTLS setup failed despite a successful `m.rtc.member` event. | -| | `encryption_error` | Failed to set up E2EE for the media channel after connecting. | -| server\_error | `ice_failed` | ICE negotiation could not complete due to network/server issues. | -| | `dtls_failed` | DTLS handshake failed. | -| | `network_error` | Temporary network outage caused the connection to drop. | -| redirection | `call_transferred` | Call was redirected to another slot, device, or user. | -| | `moved_temporarily` | Session temporarily moved (e.g., server migration). | -| permanent\_failure | `codec_mismatch` | Participant cannot decode/encode the call media. | -| | `unsupported_features` | Session requested unsupported capabilities. | +| Field | Type | Required | Description | +| ------------- | ------ | -------- | ----------- | +| `class` | string | Yes | High-level category of the disconnection or error. | +| `reason` | string | Yes | Machine-readable identifier of the specific cause. | +| `description` | string | No | Optional human-readable explanation providing additional context. | + +The following values are defined for `class`: + +- `user_action`: The disconnect was due to explicit user action (e.g. a hang up). +- `client_error`: The client experienced a failure. +- `server_error`: The server experienced a failure. +- `redirection`: The connection was moved somewhere else (e.g. to a different slot). +- `permanent_failure`: An unrecoverable failure occured. + +Values for `reason` are application-specific and are defined by each particular MatrixRTC +application type. + +The structured design of `disconnect_reason` allows representing complex error situations +such as found in e.g. [SIP](https://en.wikipedia.org/wiki/List_of_SIP_response_codes) in an +accessible way. **Note: (Pre-join)** In situations where a client never successfully connects to a call (for example, if the user is busy or declines a MatrixRTC session), a dedicated **sticky event** is From 37ded3e0eff197b09bbae55a50b64a3007e2dfcc Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Tue, 16 Jun 2026 11:00:44 +0200 Subject: [PATCH 50/62] Fix typo Signed-off-by: Johannes Marbach --- proposals/4143-matrix-rtc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 2e3a945e719..3a039b2cf90 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -413,7 +413,7 @@ The following values are defined for `class`: - `client_error`: The client experienced a failure. - `server_error`: The server experienced a failure. - `redirection`: The connection was moved somewhere else (e.g. to a different slot). -- `permanent_failure`: An unrecoverable failure occured. +- `permanent_failure`: An unrecoverable failure occurred. Values for `reason` are application-specific and are defined by each particular MatrixRTC application type. From e5e135a208b4e5fa60d0b55dbfe1a587ae8fcd4b Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Tue, 16 Jun 2026 11:32:24 +0200 Subject: [PATCH 51/62] Clarify that we're using unpadded base64 Signed-off-by: Johannes Marbach --- proposals/4143-matrix-rtc.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 3a039b2cf90..688e1c613b9 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -768,12 +768,12 @@ schema: `member.id` is globally unique per member instance, it is sufficient to disambiguate multiple key events for the same device, even if they use the same `media_key.index` value. * `media_key` The media key to use to decrypt the participant media: - * `key` required string: The base64 encoded key material. + * `key` required string: The key material encoded using unpadded base64. * `index` required int: The index of the key to distinguish it from other keys. This must be between 0 and 255 inclusive. In some implementations of MatrixRTC this may correspond to the `keyID` field of the WebRTC [SFrame](https://www.w3.org/TR/webrtc-encoded-transform/#sframe) header. -* `format` the key export format, `0` for the raw bytes base64 encoded. +* `format` the key export format, `0` for the raw bytes encoded using unpadded base64. * Depending on the RTC application, additional fields may be added to this event. Upon receipt, any `m.rtc.encryption_key` to-device event sent in cleartext SHOULD be discarded. The From 82353657faa632e12b3abc09f8a3340c4fae0104 Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Tue, 16 Jun 2026 11:46:07 +0200 Subject: [PATCH 52/62] Clarify delegation requirements Signed-off-by: Johannes Marbach --- proposals/4143-matrix-rtc.md | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 688e1c613b9..ab3f2e62357 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -486,6 +486,11 @@ timer expires due to a missing reset, the leave event is automatically emitted, participant as disconnected and ensuring accurate session state even in cases of sudden disconnection, crashes, or network failures. +For increased accuracy, some MatrixRTC transports may offer the ability to delegate management +of a user's delayed disconnect event to an external service with high visibility on the connection +state (such as a Selective Forwarding Unit / SFU). An example of this is the LiveKit transport +from [MSC4195](https://github.com/matrix-org/matrix-spec-proposals/pull/4195). + ### MatrixRTC Session A **MatrixRTC Session** is defined as the period of overlapping, **Connected** `m.rtc.member` events @@ -930,15 +935,6 @@ From this we conclude: both approaches are valid within MatrixRTC: managed context with potential state rollbacks that do not necessarily interfere with the session itself. -### MatrixRTC Membership Heartbeat Keep Alive - -If an RTC Transport is involved like a SFU then per definition the SFU has a very good understanding -of the connectivity of the individual participants. In that case it would be nice to be able to -delegate the ownership of the delayed leave event to that infrastructure component. However, -practical implementation has so far proven that the current implementation of delayed events are -also sufficient in adverse network conditions. So for now it's good enough and considered as an -optimisation for a future MSC. - ### Discovery and Negotiation of Application Types MatrixRTC does not currently define how clients should discover or negotiate which real-time From 6bc4ff09f95f05c2fc4e958534b37b80c1e35004 Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Tue, 16 Jun 2026 16:51:23 +0200 Subject: [PATCH 53/62] Add unstable version flag Signed-off-by: Johannes Marbach --- proposals/4143-matrix-rtc.md | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index ab3f2e62357..01d4288a2b4 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -1128,15 +1128,20 @@ that was sent in the three seconds after leaving the session. ## Unstable prefix -Use `org.matrix.msc4143.rtc.member` as the sticky event type in place of `m.rtc.member`. -Use `org.matrix.msc4143.rtc.slot` as the state event type in place of `m.rtc.slot`. - -For MatrixRTC Transport discovery via `GET` endpoint use -`/_matrix/client/unstable/org.matrix.msc4143/rtc/transports` instead of -`/_matrix/client/v1/rtc/transports` - -Use `org.matrix.msc4143.rtc.encryption_key` in place of the `m.rtc.encryption_key` room event and -to-device event types. +| Stable identifier | Purpose | Unstable identifier | +| ----------------- | ------- | --------------------| +| `m.rtc.slot` | Event type | `org.matrix.msc4143.rtc.slot` | +| `m.rtc.member` | Event type | `org.matrix.msc4143.rtc.member` | +| `m.rtc.shared_encryption_key` | Event type | `org.matrix.msc4143.rtc.shared_encryption_key` | +| `m.rtc.encryption_key` | To-device message event type | `org.matrix.msc4143.rtc.encryption_key` | +| `/_matrix/client/v1/rtc/transports` | Endpoint | `/_matrix/client/unstable/org.matrix.msc4143/rtc/transports` | + +Servers may advertise support for the feature by listing `org.matrix.msc4143` in the `unstable_features` +section of the response to [`GET /_matrix/client/versions`](https://spec.matrix.org/v1.18/client-server-api/#get_matrixclientversions). + +Once this proposal completes FCP, servers may advertise support for the _stable_ identifiers by listing +`org.matrix.msc4143.stable` in `unstable_features`. Clients may use this while they are waiting for the +server to adopt a version of the spec that includes it. ## Dependencies From 6bf733fb2773ba9de1f7323dd2942a15cabbc5c6 Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Tue, 16 Jun 2026 17:10:28 +0200 Subject: [PATCH 54/62] Cross-link MSC4075 Signed-off-by: Johannes Marbach --- proposals/4143-matrix-rtc.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 01d4288a2b4..ec8ba598fa1 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -49,6 +49,9 @@ This MSC focuses on the three aspects above, the other modules are defined by ac * What streams to connect to * What data to send over RTC channels and in which format * Which MatrixRTC transports are supported +* **MatrixRTC Notifications** + * How clients can inform each other about the existence of MatrixRTC sessions + * Covered in [MSC4075](https://github.com/matrix-org/matrix-spec-proposals/pull/4075/changes): MatrixRTC notifications & call ringing ## Proposal From eac10054304e7251af911ef4d683d8d379274d10 Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Wed, 17 Jun 2026 10:38:28 +0200 Subject: [PATCH 55/62] Add .well-known alternative Signed-off-by: Johannes Marbach --- proposals/4143-matrix-rtc.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index ec8ba598fa1..b35b406b0f4 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -1084,6 +1084,17 @@ The encrypted content of the `m.rtc.encryption_keys` event was as follows: } ``` +### Transport discovery via .well-known + +Rather than using a dedicated endpoint, homeservers could publish supported transports +via a `.well-known` document. This exposes transports to unauthenticated users, however, +which can be a security concern. Additionally, in enterprise deployments, `.well-known` +files are often not served by the homeserver itself and it can be bureaucratically complicated +to update entries under the top-level domain. + +`GET /_matrix/client/v1/rtc/transports` avoids these issues and offers more flexibility for +future extensions such as user-specific transports. + ## Extensibility considerations This MSC introduces a completely new concept to Matrix. The proposal is designed to be as abstract From 57c8cbe2a3a0b14163b46aa0dc1894e1c7f14267 Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Wed, 17 Jun 2026 10:57:56 +0200 Subject: [PATCH 56/62] Explain why we use the claimed_ prefixes Signed-off-by: Johannes Marbach --- proposals/4143-matrix-rtc.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index b35b406b0f4..52d931a86b3 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -345,8 +345,10 @@ A valid `m.rtc.member` event as a prerequisite for connecting to a slot has the * `id` — Identifier to distinguish multiple participations, even for the same user and same device MUST be unique for each connect event. MatrixRTC transports MAY use `id` as the canonical identifier to help prevent leaking metadata such as the user ID or device ID to external services. - * `claimed_device_id` — Matrix device identifier. - * `claimed_user_id` — Matrix user ID. + * `claimed_device_id` — Matrix device identifier. This is "claimed" because without the encryption + envelope, a receiving device has no way to tell if the event actually came from this device ID. + * `claimed_user_id` — Matrix user ID. This is "claimed" because without the encryption envelope, + a receiving device has no way to tell if the event actually came from this user ID. * `rtc_transports` — List of objects describing how to access this participant’s media streams. See [MatrixRTC Transport](#matrixrtc-transport) for the correct object format. * `sticky_key` — A unique key used to track this membership across updates. The key persists for the From 5bcd4f0bcbcaf9a2602164ea732d08b43066085f Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Wed, 17 Jun 2026 13:41:43 +0200 Subject: [PATCH 57/62] Some clarifications around encryption Signed-off-by: Johannes Marbach --- proposals/4143-matrix-rtc.md | 44 ++++++++++++++++++++++++------------ 1 file changed, 30 insertions(+), 14 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 52d931a86b3..2f23a802113 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -798,14 +798,23 @@ then keys received from such devices MUST also be excluded. The corresponding `m.rtc.member` is determined by matching its `member.id` with the one from `m.rtc.encryption_key`. +The receiving MUST apply the following checks: + +- The `sender` property from the decryption result must match the `sender` of the `m.rtc.member` + event. +- The `device_id` property from the decryption result must match the `device_id` of the `member.device_id` + of `m.rtc.member` event. + +Any `m.rtc.encryption_key` event that does not comply with these checks MUST be discarded. + Clients SHOULD rotate their keys to ensure **confidentiality** whenever a participant joins or leaves the slot. The **key rotation** process is as follows: -* The sending application generates the new key material for the local participant. -* The sending application sends the new key material to all other participants with a new `index` value. -* The receiving application stores the new key material for the specified `index`. -* The sending application continues to use the old/current key to encrypt media. -* After a short delay `delayBeforeUse,` default: 5 seconds), the sending application switches to the +* The sending client generates the new key material for the local participant. +* The sending client sends the new key material to all other participants with a new `index` value. +* The receiving client stores the new key material for the specified `index`. +* The sending client continues to use the old/current key to encrypt media. +* After a short delay `delayBeforeUse,` default: 5 seconds), the sending client switches to the new key. * It is possible to overwrite the default delay on a per application basis in case an application has specific requirements on security or wants to minimize missed stream data @@ -816,7 +825,7 @@ this is expensive. Clients SHOULD minimize key exchange traffic for rapid joiner For rapid new joiners: A key rotation grace period (`keyRotationGracePeriod`) is used, if a new member joins during the grace period, then the same key can be used and shared just to that new -member.. +member. For rapid leavers: The `delayBeforeUse` period is used to coalesce any membership change occurring in that period and then a single key rotation is scheduled afterward. @@ -1061,9 +1070,6 @@ MatrixRTC transport remains the most natural direction. Earlier iterations of this MSC used an encrypted `m.rtc.encryption_keys` room event to distribute the per-participant sender keys. -Whilst reducing traffic by only needing to send one event per participant to the homeserver, this -approach does not allow for perfect forward secrecy as the keys are persisted in the room history. - The encrypted content of the `m.rtc.encryption_keys` event was as follows: ```json5 @@ -1086,6 +1092,21 @@ The encrypted content of the `m.rtc.encryption_keys` event was as follows: } ``` +#### Issues Encountered + +1. **Scalability Problems** + - Generated high volumes of message traffic in rooms + - Frequently hit rate limiting thresholds + +2. **Timeline Pollution** + - Introduced invisible events into the room timeline + - Created notification noise depending on user settings + - Negatively impacted backpagination experience + +3. **Security Concerns** + - Over-exposed call keys by sharing them with all room participants + - Failed to limit key distribution to active call participants only (impossibility to rotate key on leaver) + ### Transport discovery via .well-known Rather than using a dedicated endpoint, homeservers could publish supported transports @@ -1132,11 +1153,6 @@ infrastructure type defines its own authentication mechanisms, as detailed in it These mechanisms may involve a service interacting with the homeserver to determine whether a user is authorized to utilize the infrastructure. -### Forward secrecy for end-to-end encryption of media streams - -The considerations to ensure forward secrecy are described in the [Key -Distribution](#key-distribution) section above. - ### End-to-end media encryption key rotation lag The proposed key rotation semantics does mean that a participant could continue to decrypt media From 0229f899e3e041f5995e61d16da44d226278c99a Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Wed, 17 Jun 2026 14:05:07 +0200 Subject: [PATCH 58/62] Move future ratcheting extension to potential issues Signed-off-by: Johannes Marbach --- proposals/4143-matrix-rtc.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 2f23a802113..b08fd8ee4d8 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -833,9 +833,6 @@ in that period and then a single key rotation is scheduled afterward. `keyRotationGracePeriod` must be greater than `delayBeforeUse` or it will have no effects (default 10s and 5s) -A future version of key exchange could introduce ratcheting, this would reduce the key traffic for -new joiners (only sends the ratcheted key to the new joiners instead of rotating to all). - #### Shared Key alternative For big calls in a room, it might be interesting to use a shared key system instead of a per-sender @@ -996,6 +993,17 @@ important, however, is that each `m.rtc.slot` event, at its position in the DAG, applicable authorisation rules** at that point in time to ensure the reconstructed session history is consistent with valid room state transitions. +### Excessive key traffic + +When using per-user encryption keys, keys are rotated and distributed to _all_ participants +whenever a member joins or leaves the session. This could result in a large amount of to-device +messages being exchanged. This is deemed acceptable for now given that it should usually only +occur during session setup. + +To mitigate this, a future version of the key exchange mechanism could introduce ratcheting. Rather +than rotating the key for all members, this would allow to ratchet the key and send it to the new +joiner only. + ## Alternatives ### Represent MatrixRTC Members as regular Matrix State From 0a78a66a6ad60ca4c0dc97b591adfd0cae221c8c Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Thu, 18 Jun 2026 12:01:38 +0200 Subject: [PATCH 59/62] Merge content 'Scope and Responsibilities of MatrixRTC Applications' into the introduction to prevent duplication Signed-off-by: Johannes Marbach --- proposals/4143-matrix-rtc.md | 121 ++++++++++------------------------- 1 file changed, 34 insertions(+), 87 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index b08fd8ee4d8..73144f0b181 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -1,57 +1,37 @@ -# MSC4143: MatrixRTC - -## Overview - -This MSC defines MatrixRTC: how to set up real-time communication sessions over Matrix. -This is the base layer to build real-time systems on top of Matrix. - -MatrixRTC specifies how a real-time session is described in a room and how Matrix users can connect -to a session. - -The MatrixRTC specification is separated into different modules: - -* **The MatrixRTC State** — defines the state of a real-time session and serves as the source of - truth for: - * Which participants are part of a session - * Which RTC technology each participant is connected through - * Which kind and how many sessions can take place in this room. - * Per-device metadata used by other participants to decide whether streams from that source are of - interest and should be subscribed to. -* **Discovery of RTC Transports** - * The actual transport for real-time data provided by the Matrix deployment (e.g. Selective - Forwarding Units (SFUs) or assisted by STUN/TURN) - * Supports real-time communication exchange, with or without backend infrastructure - * Works across topologies such as SFU, P2P, or MCU -* **The E2EE key Sharing for RTC Data** - * Each participant requires a secret in order to publish encrypted real-time data - * Every participant must hold a valid key for every other participant at all times during the - session to decrypt their media - * Sessions may use multiple keys or a single shared key across all participants; keys may rotate - during the session - * This MSC also specifies how E2EE keys are distributed - -This MSC focuses on the three aspects above, the other modules are defined by accompanying MSCs: - -* **MatrixRTC Transports** - * Supports multiple transport implementations - * Defines how to connect to peers or servers, update connections, and subscribe to streams - * A LiveKit-based transport is the current standard proposal - ([MSC4195](https://github.com/matrix-org/matrix-spec-proposals/pull/4195): MatrixRTC Transport using - LiveKit backend) - * Another planned transport is a full-mesh WebRTC implementation based on - [MSC3401](https://github.com/matrix-org/matrix-spec-proposals/pull/3401) -* **MatrixRTC Applications** - * Each application type can have its own specification - * Example: voice and video conferencing via an application of type `m.call` - * Defining finer-grained signalling UX, e.g., ringing, declining, stopping ringing - * Specifies the full RTC experience, including: - * How to interpret member event metadata - * What streams to connect to - * What data to send over RTC channels and in which format - * Which MatrixRTC transports are supported -* **MatrixRTC Notifications** - * How clients can inform each other about the existence of MatrixRTC sessions - * Covered in [MSC4075](https://github.com/matrix-org/matrix-spec-proposals/pull/4075/changes): MatrixRTC notifications & call ringing +# MSC4143: MatrixRTC – Real-time communication over Matrix + +Matrix is a generalised protocol for decentralised communication. This includes chatting but also +real-time communication (RTC) such as VoIP. While Matrix supports VoIP signalling for [1-to-1 calls], +and [MSC3401] attempts to extend it to group calling, a unified system for RTC applications is +currently missing. + +[1-to-1 calls]: https://spec.matrix.org/v1.18/client-server-api/#voice-over-ip +[MSC3401]: https://github.com/matrix-org/matrix-spec-proposals/pull/3401 + +The present proposal aims to close this gap by introducing "MatrixRTC", a generalised framework for +reliably establishing and maintaining RTC sessions over Matrix. This MSC is concerned with the +foundational protocol and covers: + +- RTC session state (including session configuration, membership and publishment of RTC media streams) +- Discovering available modes for the actual RTC media transport (e.g. Selective Forwarding Units + (SFUs) or peer-to-peer assisted by STUN/TURN servers) +- End-to-end encryption of RTC media + +This proposal is entirely agnostic of RTC applications or transport technologies and treats both +as generic building blocks. As first concrete instances, a voice- and video conferencing application +type and a transport mode based on the [LiveKit SFU] are proposed in +[MSC4196: MatrixRTC voice and video calling application `m.call`] and +[MSC4195: MatrixRTC Transport using LiveKit Backend]. Further applications and transports may be +added in the future. + +[LiveKit SFU]: https://docs.livekit.io/reference/internals/livekit-sfu/ +[MSC4196: MatrixRTC voice and video calling application `m.call`]: https://github.com/matrix-org/matrix-spec-proposals/pull/4196 +[MSC4195: MatrixRTC Transport using LiveKit Backend]: https://github.com/matrix-org/matrix-spec-proposals/pull/4195/changes + +This proposal also doesn't cover notifications for RTC sessions. These are considered outside the core +protocol and are described in [MSC4075: MatrixRTC notifications & call ringing]. + +[MSC4075: MatrixRTC notifications & call ringing]: https://github.com/matrix-org/matrix-spec-proposals/pull/4075 ## Proposal @@ -876,39 +856,6 @@ keys call. Clients should detect when the slot is configured for a shared key, and then use the shared key instead of generating and sharing sender keys. -### Scope and Responsibilities of MatrixRTC Applications - -This MSC defines the **foundational protocol for real-time communication (RTC)** in Matrix — -establishing the primitives and data structures needed for RTC functionality. It deliberately **does -not define a complete calling experience**, leaving that to specific MatrixRTC applications that -build on top of this foundation. Each such application can define its own UX expectations, -signalling behaviour, and feature set. - -Implementations, especially chat clients, are expected to support one or more of these applications -as they are defined by future MSCs. -The first of these is expected to be the -**`m.call` application ([MSC4196](https://github.com/matrix-org/matrix-spec-proposals/pull/4196))**. -Clients SHOULD treat this application as a special case and present a familiar call experience that -abstracts away the underlying slot and membership model. - -Common UX patterns include: - -* A “Call” button in the room header. -* Ongoing call information displayed in the room header. -* Moderation-style wording for slot management, e.g. *“Allow users to start a call.”* -* Permission-related messaging, e.g. *“You are not allowed to start a call in this room.”* - -This MSC’s goal is to define the **core RTC protocol**, not to specify every user-facing behaviour. -It delegates **user experience, call control, and signalling flows** to individual RTC applications -layered on top. For example, pre-call signalling (ringing, declining, etc.) is handled outside the -RTC session itself, using membership and slot application data and other events (TODO: link related -MSCs). - -Importantly, **MatrixRTC can be used by custom or experimental RTC applications** without those -applications being merged into the Matrix specification. The only difference is that a *merged* RTC -application may impose additional interoperability requirements on clients, whereas *non-merged* -applications can still operate using the same MatrixRTC primitives. - ## Potential issues ### Shared State for a MatrixRTC Session using Matrix Primitives From a5bef33503fb91ffd038e6258a9cac4b364c1a9e Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Mon, 6 Jul 2026 12:56:07 +0200 Subject: [PATCH 60/62] Remove relations --- proposals/4143-matrix-rtc.md | 48 +++++++++++++++++++----------------- 1 file changed, 25 insertions(+), 23 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 73144f0b181..109748c45fb 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -290,10 +290,6 @@ A valid `m.rtc.member` event as a prerequisite for connecting to a slot has the "claimed_device_id": "DEVICEID" "claimed_user_id": "@user:matrix.domain" }, - "m.relates_to":{ // an updated m.rtc.member event MUST reference the first m.rtc.member - rel_type: "m.reference", // event you sent for this call. You should omit if this is the first event. - event_id: "$join_event_id" - }, "rtc_transports": [ {...TRANSPORT_1}, {...TRANSPORT_2} @@ -316,11 +312,6 @@ A valid `m.rtc.member` event as a prerequisite for connecting to a slot has the allowing clients to avoid connecting to participants outside their area of interest. **Note** application-specific metadata as part of the MatrixRTC slot `application` object have precedence over the MatrixRTC member ones. -* `m.relates_to` — The `m.relates_to` field optionally references the initial connect event, - distinguishing connect events from updates. Clients SHOULD include this field when sending updates - to an existing `m.rtc.member` event to ensure continuity in membership lifecycle tracking, enable - accurate historical reconstruction, and allow deriving the participant’s start time using the - `origin_server_ts` of the referenced event. * `member` — Uniquely identifies this participation instance; includes: * `id` — Identifier to distinguish multiple participations, even for the same user and same device MUST be unique for each connect event. MatrixRTC transports MAY use `id` as the canonical identifier @@ -353,10 +344,6 @@ A valid `m.rtc.member` event as a prerequisite for disconnecting from a slot has // event type: "m.rtc.member" { "slot_id": "m.call#ROOM", // MUST - "m.relates_to":{ // SHOULD - rel_type: "m.reference", - event_id: "$join_event_id" - }, "disconnect_reason": { // SHOULD "class": "server_error", "reason": "ice_failed", @@ -373,10 +360,6 @@ A valid `m.rtc.member` event as a prerequisite for disconnecting from a slot has **Field explanations:** - `slot_id`: The slot this member belongs to. -- `m.relates_to`: Clients SHOULD include the `m.relates_to` field referencing the original - `m.rtc.member` join event when disconnecting. This facilitates accurate session history and - retrospective computations by fetching the relation to determine the participant’s original join - time or associated metadata. - `sticky_key:` The sticky key from the disconnecting MatrixRTC member. - `disconnect_reason`: The `disconnect_reason` object is **optional** and provides additional context when a participant disconnects from a call. It is only meaningful if the user has @@ -431,12 +414,6 @@ The MatrixRTC membership is a collection of linked `m.rtc.member` events. With t (Connect) (Update) (Disconnect) m.rtc.member ───────► m.rtc.member ──── ... ───► m.rtc.member - ^ | | - ├─────────────────────┘ m.relation | - | | - └────────────────────────────────────────────────┘ m.relation - References original join event - [---------------- Connected ------- ... --------][--- Disconnected ... Time ───────────────────────────────────────────────────────────────────────► @@ -1097,6 +1074,31 @@ follow-up proposal. However, this MSC has been designed to remain compatible wit Future constraint definitions would likely exist alongside the `"application"` key within the `m.rtc.slot` event structure. +### Chaining member events with relations + +An earlier version of this proposal used `m.reference` relations to link updated `m.rtc.member` +events to the initial connect event. + +``` +(Connect) (Update) (Disconnect) (Reconnect) (Update) + +m.rtc.member ──► m.rtc.member ─ ... ─► m.rtc.member m.rtc.member ──► m.rtc.member ─ ... + ^ │ │ ^ │ + ├────────────────┘ │ └────────────────┘ + │ m.reference │ m.reference + └──────────────────────────────────────┘ + m.reference + +Time ─────────────────────────────────────────────────────────────────────────────────────► +``` + +This was meant to assist in reconstructing historical sessions accurately. However, the relations +turned out to not be helpful because finding the slot as well as other participants' member events +still required manual history traversal while employing timestamp overlap logic. + +A future proposal may tackle the problem of performant session history reconstruction, possibly +by using relations. This proposal does not add relations in order not to preclude such later attempts. + ## Security considerations ### Discoverability of RTC Infrastructure From 98bd92d87f7ace9b693e9ad69d3570d701ae41a9 Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Mon, 6 Jul 2026 14:28:17 +0200 Subject: [PATCH 61/62] Blend part of the beginning of the proposal into the introduction to reduce repetition --- proposals/4143-matrix-rtc.md | 73 ++++++++++++++---------------------- 1 file changed, 28 insertions(+), 45 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index 109748c45fb..c140a94faf5 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -9,20 +9,29 @@ currently missing. [MSC3401]: https://github.com/matrix-org/matrix-spec-proposals/pull/3401 The present proposal aims to close this gap by introducing "MatrixRTC", a generalised framework for -reliably establishing and maintaining RTC sessions over Matrix. This MSC is concerned with the -foundational protocol and covers: +building RTC experiences on top of Matrix. At a high level, MatrixRTC consists of the following parts: -- RTC session state (including session configuration, membership and publishment of RTC media streams) -- Discovering available modes for the actual RTC media transport (e.g. Selective Forwarding Units - (SFUs) or peer-to-peer assisted by STUN/TURN servers) -- End-to-end encryption of RTC media - -This proposal is entirely agnostic of RTC applications or transport technologies and treats both -as generic building blocks. As first concrete instances, a voice- and video conferencing application -type and a transport mode based on the [LiveKit SFU] are proposed in -[MSC4196: MatrixRTC voice and video calling application `m.call`] and -[MSC4195: MatrixRTC Transport using LiveKit Backend]. Further applications and transports may be -added in the future. +* **Applications** describe the type of RTC activity (e.g. a call, a shared document, or a real-time + game) and may define additional metadata and constraints. +* **Slots** are represented in room state and govern what kind of applications may run, along with + any needed configuration. +* **Transports** define how participants actually exchange media streams. This can, for instance, happen + peer-to-peer or through Selective Forwarding Unit (SFUs). +* **Membership** is expressed via room events and provides a record of who is participating in a slot, + and under which transports. +* **Sessions** are formed only indirectly through the temporal overlap of connected members within + a slot. +* **End-to-end encryption** provides the basis for encrypted media exchange and reuses existing + Matrix primitives such as encrypted room and to-device messages. + +This MSC is concerned with the foundational MatrixRTC protocol and covers slots, membership, sessions and +end-to-end encryption. Applications and transports are treated as generic building blocks only. The proposal +defines how these components are plugged into the system while leaving the introduction of concrete applications +and transports to other proposals. + +As first concrete instances, a voice- and video conferencing application and a transport based on the +[LiveKit SFU] are proposed in [MSC4196: MatrixRTC voice and video calling application `m.call`] and +[MSC4195: MatrixRTC Transport using LiveKit Backend], respectively. [LiveKit SFU]: https://docs.livekit.io/reference/internals/livekit-sfu/ [MSC4196: MatrixRTC voice and video calling application `m.call`]: https://github.com/matrix-org/matrix-spec-proposals/pull/4196 @@ -35,38 +44,6 @@ protocol and are described in [MSC4075: MatrixRTC notifications & call ringing]. ## Proposal -MatrixRTC provides a framework for building real-time communication on top of Matrix. At its core, -it introduces the concept of **applications**, which define the semantics of a real-time experience, -and **slots**, which act as containers where those experiences take place within a room. -Participants join as **members** within slots, and their overlap defines a **session**. - -This proposal defines how MatrixRTC slots are opened and closed, how MatrixRTC members join and -leave, and how these interactions together form the lifecycle of a MatrixRTC session. - -The proposal also specifies how: - -* **Applications** describe the type of RTC activity (e.g. a call, a shared document, or a real-time - game) and may define additional metadata and constraints. -* **Slots** are represented in room state (`m.rtc.slot` identified by a unique `slot_id`) and govern - what kind of application may run, along with permissions and configuration. -* **Membership** (`m.rtc.member` [MSC4354 Sticky - Events](https://github.com/matrix-org/matrix-spec-proposals/pull/4354)) provides a precise record - of who is actively participating, and under which transports and devices. -* **Sessions** emerge from the overlap of active members within an open slot, with clear start and - end conditions. -* **Transports** define how participants exchange media, supporting multiple backends such as SFUs, - MCUs, or peer-to-peer. -* **End-to-end encryption** ensures secure media exchange, including mechanisms for key sharing and - rotation. - -This design deliberately separates **slot management** (opening/closing slots, requiring higher -power levels) from **slot participation** (joining as a member, requiring lower power levels). This -separation gives applications a robust surface for conflict resolution, clear ownership of events, -and flexibility across use cases such as always-on spaces, scheduled meetings, or ephemeral -conversations. - -The following sections specify these components in detail. - ### MatrixRTC Application Types An **application type** defines the semantics of a MatrixRTC session, such as a real-time game, a @@ -123,6 +100,12 @@ defined by the Matrix room state at all times. The ability to open and close slo patterns enables a wide variety of use cases, from always-on shared spaces to short lived scheduled meetings. +This design deliberately separates **slot management** (opening/closing slots, requiring higher +power levels) from **slot participation** (joining as a member, requiring lower power levels). This +separation gives applications a robust surface for conflict resolution, clear ownership of events, +and flexibility across use cases such as always-on spaces, scheduled meetings, or ephemeral +conversations. + #### Opening a MatrixRTC Slot A slot is opened by sending an `m.rtc.slot` state event with `state_key = slot_id`. This event From 01abcd682b2a798dd9a7dfa843de90950c11af39 Mon Sep 17 00:00:00 2001 From: Johannes Marbach Date: Tue, 7 Jul 2026 16:18:14 +0200 Subject: [PATCH 62/62] Update slot description and include encryption setting --- proposals/4143-matrix-rtc.md | 244 +++++++++++++---------------------- 1 file changed, 87 insertions(+), 157 deletions(-) diff --git a/proposals/4143-matrix-rtc.md b/proposals/4143-matrix-rtc.md index c140a94faf5..fc3ef6c5938 100644 --- a/proposals/4143-matrix-rtc.md +++ b/proposals/4143-matrix-rtc.md @@ -11,18 +11,19 @@ currently missing. The present proposal aims to close this gap by introducing "MatrixRTC", a generalised framework for building RTC experiences on top of Matrix. At a high level, MatrixRTC consists of the following parts: -* **Applications** describe the type of RTC activity (e.g. a call, a shared document, or a real-time - game) and may define additional metadata and constraints. +* **End-to-end encryption** provides a generic basis for encrypted media exchange and reuses existing + Matrix primitives such as encrypted room and to-device messages. +* **Transports** define how participants exchange media streams. This can, for instance, happen + peer-to-peer or through Selective Forwarding Unit (SFUs). Transports also determine how the generic + end-to-end encryption is used in transport-specific encryption. +* **Applications** describe the type of RTC activity such as a call, a shared document, or a real-time + game. Applications also define what types of transports they can work with and how media streams are used. * **Slots** are represented in room state and govern what kind of applications may run, along with any needed configuration. -* **Transports** define how participants actually exchange media streams. This can, for instance, happen - peer-to-peer or through Selective Forwarding Unit (SFUs). * **Membership** is expressed via room events and provides a record of who is participating in a slot, and under which transports. * **Sessions** are formed only indirectly through the temporal overlap of connected members within a slot. -* **End-to-end encryption** provides the basis for encrypted media exchange and reuses existing - Matrix primitives such as encrypted room and to-device messages. This MSC is concerned with the foundational MatrixRTC protocol and covers slots, membership, sessions and end-to-end encryption. Applications and transports are treated as generic building blocks only. The proposal @@ -44,188 +45,112 @@ protocol and are described in [MSC4075: MatrixRTC notifications & call ringing]. ## Proposal -### MatrixRTC Application Types - -An **application type** defines the semantics of a MatrixRTC session, such as a real-time game, a -voice/video call, or a shared document. Each application type has its own specification describing: +### Slots -* How different RTC streams are interpreted. -* Which MatrixRTC transport types are supported or required. +MatrixRTC slots act as containers for MatrixRTC applications to run in. Slots are represented by +state events of type `m.rtc.slot` which means that they can only be created or modified by users +with sufficient power level. As described [below], participants can connect to slots by sending +`m.rtc.member` room events. -For example, a [Third Room](https://thirdroom.io) like experience could include information about -the virtual scene, e.g., a place, available objects or weather conditions. +[below]: #slot-membership -This modular design makes MatrixRTC flexible. For example, a Jitsi-based conference could be -supported by defining a new application type and a corresponding RTC transport. While such a session -would not be compatible with Matrix clients that do not implement the Jitsi transport, those clients -would still be able to detect the presence of an unsupported application and transport type, and -handle it with a sensible user interface. +This design deliberately separates slot management, requiring higher power level, from slot +participation, requiring lower power level. This separation gives applications a robust surface +for conflict resolution, clear ownership of events, and flexibility across use cases such as +always-on spaces, scheduled meetings, or ephemeral conversations. -The minimum Application definition consists of a simple JSON object +Slots are always tied to specific applications through their slot ID which acts as the `state_key`. +The slot ID is constructed as: ```json5 -{ - "application": { - "type": "m.call", // The # character MUST NOT appear in a valid type string. - // Optional: application-specific metadata - "m.call.spatial_audio": true - } -} +slot_id = {application_type}#{application_slot_id} (= state_key) ``` -The `type` is a unique identifier for the application and MUST follow the [*Common Namespaced Identifier -Grammar*](https://spec.matrix.org/v1.16/appendices/#common-namespaced-identifier-grammar). - -Each application type specifies the additional fields and defines how communication with the -corresponding transport is handled. Concrete applications types are introduced in separate proposals -such as: - -* `m.call` — voice and video calling, as defined in - [MSC4196](https://github.com/matrix-org/matrix-spec-proposals/pull/4196). +`application_type` is the application's globally unique identifier. This identifier is defined +by the application's specification and MUST follow the [Common Namespaced Identifier Grammar][^nohash]. -### MatrixRTC Slot and Constraining Slots +`application_slot_id` is the application-specific slot ID and enables applications to support +multiple parallel application instances per room. Again, the allowed values are defined by +the application's specification and MUST follow the [Common Namespaced Identifier Grammar] +but this time without the namespacing requirements[^nohash]. Additionally, the values should +be predictable for clients given that slots act like a virtual addresses where participants +are allowed to meet. -A **MatrixRTC slot** is the container for MatrixRTC members and temporal overlapping MatrixRTC -members form a MatrixRTC session. Each slot is identified by a unique `slot_id`, tied to a specific -`application` type, and represented by the `m.rtc.slot` state event with the `slot_id` as the state -key. +As an example, the default slot ID for the calling applications from [MSC4196: MatrixRTC voice and video calling application `m.call`] +is `m.call#ROOM`. -> [!NOTE] -> Conceptually, a slot is like a virtual meeting room: MatrixRTC sessions occur within -> slots, and multiple sessions may occur consecutively without changing the slot’s configuration. +The grammar for forming slot IDs MUST NOT be used to parse the components out of a slot ID. +It exists only to namespace the `state_key`. -Slots are part of the shared room state and can only be created or modified by authorized users with -sufficient power level. Clients MUST react on and respect latest `m.rtc.slot` definitions as -defined by the Matrix room state at all times. The ability to open and close slots with different -patterns enables a wide variety of use cases, from always-on shared spaces to short lived scheduled -meetings. +[Common Namespaced Identifier Grammar]: https://spec.matrix.org/v1.16/appendices/#common-namespaced-identifier-grammar -This design deliberately separates **slot management** (opening/closing slots, requiring higher -power levels) from **slot participation** (joining as a member, requiring lower power levels). This -separation gives applications a robust surface for conflict resolution, clear ownership of events, -and flexibility across use cases such as always-on spaces, scheduled meetings, or ephemeral -conversations. +[^nohash]: Note that due the use of the [Common Namespaced Identifier Grammar](https://spec.matrix.org/v1.16/appendices/#common-namespaced-identifier-grammar), +neither `application_type` nor `application_slot_id` can contain the `#` character. -#### Opening a MatrixRTC Slot +#### Opening a slot -A slot is opened by sending an `m.rtc.slot` state event with `state_key = slot_id`. This event -authorises MatrixRTC members that intend to participate in the slot and follows the schema below: +A slot is opened by sending an `m.rtc.slot` state event with `state_key = slot_id` and the +following schema: ```json5 -// Example: an open slot with application-specific metadata { - "application": { - "type": "m.call", - // optional: app specific slot metadata - "m.call.id": UUID, // Note your application must handle rollback due to state resolution - "m.call.voice_only": true - } + "type": "m.rtc.slot", + "state_key": "{application_type}#{application_slot_id}", // = slot_id + "content": { + "application": { + "type": "{application_type}", + ... // Further application-specific properties (if required) + }, + "encryption": { + "type": "{encryption_type}", + ... // Further encryption-specific properties (if required) + } + }, + ... } - -state_key: "m.call#ROOM" // slot_id ``` -**Field description:** - -* **state key**: The state key of the `m.rtc.slot` state event, referred to as the `slot_id`. -* `application` An application JSON object, which **MUST** specify the application `type` and MAY - include additional fields which **constrain** the application (e.g., restricting a call to be - voice-only). **Note** those additional fields may be extended through fields in the `application` - content block of `m.rtc.member` events. If a field occurs in both, the value from the `m.rtc.slot` event - takes precedence. - -The `slot_id` of an open slot acts like a virtual address where participants are allowed to meet. -The grammar for the `slot_id` is a well formed state key confined such that members of different -MatrixRTC applications never occupy the same slot according to: - -```json5 -{application.type}#{application_slot_id} -``` +- `application` (required, object): Describes the application that can run in this slot. + - `type` (required, string): The globally unique application identifier. MUST follow the + [Common Namespaced Identifier Grammar]. + - Optionally includes further properties for settings that are specific to the application + `type`. The concrete properties are defined by the application's specification. A calling + application, for instance, could include properties for constraining the call to be voice-only. +- `encryption` (optional, object): Describes the encryption mechanism to use in this slot. Further, + details on the available mechanisms can be found in the [encryption section] below. + - `type` (required, string): The globally unique identifier of the encryption mechanism. + - Optionally includes further properties for settings that are specific to the encryption + `type`. -Where +[encryption section]: #end-to-end-encryption -* `application.type` is the `type` field in the application JSON object -* `application_slot_id` is the application-specific slot ID. The value MUST follow the - [*Common Namespaced Identifier Grammar*](https://spec.matrix.org/v1.16/appendices/#common-namespaced-identifier-grammar) - but without the namespacing requirements. Each application defines its own - schema (e.g., `ROOM`, `line1`, `line2`) to allow multiple parallel slots of the same type according to the - application requirements. If the application needs to define its slots out of band (e.g. mapping them to - widget IDs) then it can use those IDs as `application_slot_id`. However, given a slot is the mechanism - around which sessions converge, it MUST have a predictable `application_slot_id`. +#### Closing a slot -Note that due the use of the [*Common Namespaced Identifier Grammar*](https://spec.matrix.org/v1.16/appendices/#common-namespaced-identifier-grammar), -neither `application.type` nor `application_slot_id` can contain the `#` character. - -This grammar MUST never be used to parse `slot_ids`; it exists only to namespace the `state_key`. - -#### Closing a MatrixRTC Slot - -To close a slot, the corresponding `m.rtc.slot` state event is updated with empty content which -removes all remaining MatrixRTC members, for example: +To close a slot, the corresponding `m.rtc.slot` state event is updated with empty content. ```json5 -// Empty content represents a closed slot -{} - -state_key: "m.call#ROOM" // slot_id -``` - -#### Examples of MatrixRTC Slot Usage - -In the following example three different slot use-cases are depicted - -``` -m.rtc.slot[id_1] {content_1} ...████████████████████████████████████████████... -m.rtc.slot[id_2] {content_2} ███████████████████ - -m.rtc.slot[id_state_res] {content_ABC} ████XXXXXXXXXXXXXXXXXXXXXXXXXXXXX -m.rtc.slot[id_state_res] {content_XYZ} █████████████████████████████ - | - | - Rollback due to Matrix state resolution - - -Time ────────────────────────────────────────────────────► -``` - -* `id_1` comprises a long-lived slot suited for a Discord-style experience or a shared whiteboard - where people can hop on and hop off as desired. -* `id_2` suited for a scheduled conference meeting, where the state event might be managed using - cancellable delayed events (MSC4140) or a bot. -* `id_state_res` This `m.rtc.slot` state event is subject to Matrix state resolution. Hence, it is - divided potentially into two intervals. If a rollback occurs during an active MatrixRTC session, - application logic MUST handle conflicts (e.g., a rolled-back UUID). - -#### Recommended User Experience - -The lifecycle of a slot follows three distinct states: - -``` -Closed -Open - ├─ Active --- at least one m.rtc.member is connected - └─ Inactive --- no member is connected +{ + "type": "m.rtc.slot", + "state_key": "{application_type}#{application_slot_id}", // = slot_id + "content": {}, + ... +} ``` -Slots may transition between these states as a slot is opened or closed, or as users connect or disconnect. +The semantics of open and closed slots for slot participation are described in the membership +event section [below]. -While **participation** in a MatrixRTC session is possible if a **slot is open**, **management** of -slots depends on the user’s **power level**. Clients should be aware of this distinction when constructing -their user interfaces. +#### Slot lifecycle -For **open** and **active** slots, for instance, clients could display the participant count, an icon, -a list entry, or join button. For **open** but **inactive** slots, in turn, clients could indicate that the -slot is available and provide a way for a user to join as the first member (e.g., via a “Call” button or -placeholder icon). Finally, for users who have sufficient power level to manage slots, clients could -display a slot management UI with controls to open, close or modify slots. +Slots may follow different lifecycles depending on the use case. For instance, a long-lived slot +that is kept open continually could power a Discord-style experience where participants can hop on +and hop off as desired. Scheduled conference meetings, in turn, could benefit from a time-bounded +slot that is only opened when the meeting starts and closed again afterwards. -Multiple active slots may exist in the same room if the application type supports them. +For the time being, slots will have to be created manually. A future proposal may change the +defaults for newly created rooms to provide slots for standard RTC applications. -Clients MAY present an option to room administrators to enable specific applications in the room by -adding a slot and a type to room state. Future MSCs may change the defaults for new rooms to enable -RTC applications by default. - -### MatrixRTC Membership +### Slot membership A MatrixRTC membership is represented by a sticky `m.rtc.member` event ([MSC4354](https://github.com/matrix-org/matrix-spec-proposals/pull/4354) Sticky Events). These @@ -255,6 +180,11 @@ An `m.rtc.member` sticky event can be either **Connected** or **Disconnected** * **Content:** SHOULD match the JSON content schema for disconnecting from a slot (see below). If it does not match, clients MUST handle the lack of information gracefully. +Since `m.rtc.slot` events are subject to state resolution and may generally be changed at any +time, clients MUST react to and respect the latest known `m.rtc.slot` event at all times. +If a rollback occurs during an active MatrixRTC session, application logic MUST handle conflicts +(e.g., a rolled-back UUID). + #### Connecting to a MatrixRTC Slot A valid `m.rtc.member` event as a prerequisite for connecting to a slot has the following schema: @@ -662,7 +592,7 @@ Clients SHOULD use this list to determine which RTC transports to connect to and selected transports according to the respective MSC in the `rtc_transports` field of their `m.rtc.member` events. -### End-to-end Encryption of RTC Data +### End-to-end encryption This section defines how key material is shared between participants in MatrixRTC to enable end-to-end encryption of RTC data.