Limit outgoing to_device EDU size to 65536 by MatMaul · Pull Request #18416 · element-hq/synapse

MatMaul · 2025-05-09T14:15:27Z

If a set of messages exceeds this limit, the messages are split across several EDUs.

Fix #17035 (should)

There is currently no official specced limit for EDUs, but the consensus seems to be that it would be useful to have one to avoid this bug by bounding the transaction size.

As a side effect it also limits the size of a single to-device message to a bit less than 65536.

This should probably be added to the spec similarly to the message size limit..

Spec PR: matrix-org/matrix-spec#2340

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file.
Code style is correct

From my review of #18416

Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>

mcalinghee · 2025-08-20T21:17:08Z

Looks like the CI tests are flaky as mentioned here : #18537

mcalinghee · 2025-08-22T07:07:15Z

Not able to reproduce the issue here by running COMPLEMENT_DIR=../complement POSTGRES=1 WORKERS=1 ./scripts-dev/complement.sh
Probably missing something here.

MadLittleMods · 2026-03-23T20:20:57Z

Relevant thread is here: #18416 (comment) (press 'load more' when viewing the conversations to get everything visible)

We should link your spec PR in the code where we raise EventSizeError

MatMaul · 2026-03-23T21:36:15Z

Thank you stupid GH for this great view of a comment with no context and no way to easily reach the context :)

Thanks for the direct link, MadLittleMods >> GitHub.

MadLittleMods

Looks good. I'm unable to apply the last edits

Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>

MadLittleMods · 2026-03-24T16:23:19Z

Thanks for the fix @MatMaul 🐏

MatMaul · 2026-03-24T16:25:18Z

We did it! Thanks for the reviews 🙏

Spawning from #18416 (comment)

…clock) (#19602) Spawning from #18416 (comment)

This reverts commit 7fad50f.

@MadLittleMods

Reverts #18416 Unfortunately, this causes failures on `/sendToDevice` endpoint in normal circumstances. If a single user has, say, a hundred devices then we easily go over the limit. This blocks message sending entirely in encrypted rooms. cc @MadLittleMods @MatMaul

If a set of messages exceeds this limit, the messages are split across several EDUs. Fix #17035 (should) There is currently [no official specced limit for EDUs](matrix-org/matrix-spec#807), but the consensus seems to be that it would be useful to have one to avoid this bug by bounding the transaction size. As a side effect it also limits the size of a single to-device message to a bit less than 65536. This should probably be added to the spec similarly to the [message size limit.](https://spec.matrix.org/v1.14/client-server-api/#size-limits) Spec PR: matrix-org/matrix-spec#2340 --------- Co-authored-by: mcalinghee <mcalinghee.dev@gmail.com> Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>

This is based on #18416, which got reverted due to it rejecting to-device messages to users with many devices. The main change here is that if a to-device EDU for a single user is too large, then we split it up into multiple EDUs.

@user

This is based on #18416, which got reverted (#19614) due to it incorrectly rejecting to-device messages to users with many devices (and thus breaking message sending). Fix #17035 A to-device message content looks like: ```jsonc { "@user:domain": {"device1": {...}, "device2": {...}}, ... } ``` The previous PR would split up into multiple EDUs, each with a subset of the users. However, if one user's entry was too large it would not further split it up and then error out. The main change in this PR is to allow splitting up a single user into multiple EDUs. Other changes: 1. Rename to `SOFT_MAX_EDU_SIZE` to indicate that we sometimes send EDUs with larger size than that, and its more a target than a hard limit. 2. Check early if any to-device message (to a specific device) is too large to send, even if we're not going to send it over federation. This ensures that we catch issues where clients try to send too large to-device. This still means that if a client send a large individual to-device message it will fail, but I don't believe we ever send such large to-device messages (normally they're in the range of a few KB). --- I ended up changing the implementation a bunch to make it easy to reuse the code to split up dictionaries. Instead of repeatedly splitting up the EDU until each bit fits into the size, we instead record the size of each entry in the dict and instead split up based on cumulative size. This means we call `encode_canonical_json` on each entry rather than once on the entire struct, but its not significantly slower to do so. -- cc @MatMaul @MadLittleMods --------- Co-authored-by: Mathieu Velten <matmaul@gmail.com> Co-authored-by: mcalinghee <mcalinghee.dev@gmail.com> Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>

If a set of messages exceeds this limit, the messages are split across several EDUs. Fix element-hq#17035 (should) There is currently [no official specced limit for EDUs](matrix-org/matrix-spec#807), but the consensus seems to be that it would be useful to have one to avoid this bug by bounding the transaction size. As a side effect it also limits the size of a single to-device message to a bit less than 65536. This should probably be added to the spec similarly to the [message size limit.](https://spec.matrix.org/v1.14/client-server-api/#size-limits) Spec PR: matrix-org/matrix-spec#2340 --------- Co-authored-by: mcalinghee <mcalinghee.dev@gmail.com> Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>

…clock) (element-hq#19602) Spawning from element-hq#18416 (comment)

@MadLittleMods

Reverts element-hq#18416 Unfortunately, this causes failures on `/sendToDevice` endpoint in normal circumstances. If a single user has, say, a hundred devices then we easily go over the limit. This blocks message sending entirely in encrypted rooms. cc @MadLittleMods @MatMaul

@user

This is based on element-hq#18416, which got reverted (element-hq#19614) due to it incorrectly rejecting to-device messages to users with many devices (and thus breaking message sending). Fix element-hq#17035 A to-device message content looks like: ```jsonc { "@user:domain": {"device1": {...}, "device2": {...}}, ... } ``` The previous PR would split up into multiple EDUs, each with a subset of the users. However, if one user's entry was too large it would not further split it up and then error out. The main change in this PR is to allow splitting up a single user into multiple EDUs. Other changes: 1. Rename to `SOFT_MAX_EDU_SIZE` to indicate that we sometimes send EDUs with larger size than that, and its more a target than a hard limit. 2. Check early if any to-device message (to a specific device) is too large to send, even if we're not going to send it over federation. This ensures that we catch issues where clients try to send too large to-device. This still means that if a client send a large individual to-device message it will fail, but I don't believe we ever send such large to-device messages (normally they're in the range of a few KB). --- I ended up changing the implementation a bunch to make it easy to reuse the code to split up dictionaries. Instead of repeatedly splitting up the EDU until each bit fits into the size, we instead record the size of each entry in the dict and instead split up based on cumulative size. This means we call `encode_canonical_json` on each entry rather than once on the entire struct, but its not significantly slower to do so. -- cc @MatMaul @MadLittleMods --------- Co-authored-by: Mathieu Velten <matmaul@gmail.com> Co-authored-by: mcalinghee <mcalinghee.dev@gmail.com> Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>

@user

This is based on element-hq/synapse#18416, which got reverted (#19614) due to it incorrectly rejecting to-device messages to users with many devices (and thus breaking message sending). Fix element-hq/synapse#17035 A to-device message content looks like: ```jsonc { "@user:domain": {"device1": {...}, "device2": {...}}, ... } ``` The previous PR would split up into multiple EDUs, each with a subset of the users. However, if one user's entry was too large it would not further split it up and then error out. The main change in this PR is to allow splitting up a single user into multiple EDUs. Other changes: 1. Rename to `SOFT_MAX_EDU_SIZE` to indicate that we sometimes send EDUs with larger size than that, and its more a target than a hard limit. 2. Check early if any to-device message (to a specific device) is too large to send, even if we're not going to send it over federation. This ensures that we catch issues where clients try to send too large to-device. This still means that if a client send a large individual to-device message it will fail, but I don't believe we ever send such large to-device messages (normally they're in the range of a few KB). --- I ended up changing the implementation a bunch to make it easy to reuse the code to split up dictionaries. Instead of repeatedly splitting up the EDU until each bit fits into the size, we instead record the size of each entry in the dict and instead split up based on cumulative size. This means we call `encode_canonical_json` on each entry rather than once on the entire struct, but its not significantly slower to do so. -- cc @MatMaul @MadLittleMods --------- Co-authored-by: Mathieu Velten <matmaul@gmail.com> Co-authored-by: mcalinghee <mcalinghee.dev@gmail.com> Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>

Limit to_device EDU size to 65536

61fa1b9

MatMaul force-pushed the edu-limit-size branch from 8add186 to 61fa1b9 Compare May 9, 2025 14:19

MatMaul marked this pull request as ready for review May 9, 2025 14:38

MatMaul requested a review from a team as a code owner May 9, 2025 14:38

MatMaul and others added 10 commits May 12, 2025 01:43

Increment to_device stream for each EDU otherwise we loose some

c80f24d

Simplify

eda00e1

Add comment

6627bed

Cosmetic

c6bc691

Cosmetic

57ab541

Add logs

a0e6dc3

Improve logs

9ce1488

fix bug

5e86c59

Improve logs

35d98b6

Merge remote-tracking branch 'origin/develop' into edu-limit-size

6be7bcc

MadLittleMods reviewed May 20, 2025

View reviewed changes

erikjohnston reviewed May 23, 2025

View reviewed changes

Comment thread synapse/handlers/devicemessage.py Outdated

MadLittleMods changed the title ~~Limit to_device EDU size to 65536~~ Limit outgoing to_device EDU size to 65536 Jun 3, 2025

MadLittleMods added a commit that referenced this pull request Jun 19, 2025

Remove unrelated comment from reviewing another PR

ec16224

From my review of #18416

mcalinghee and others added 8 commits August 20, 2025 15:53

Apply suggestion from @MadLittleMods

a15a585

Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>

Apply suggestion from @MadLittleMods

e81e737

Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>

Apply suggestion from @MadLittleMods

1f9d243

Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>

Apply suggestion from @MadLittleMods

b503f97

Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>

add comment and add helper create_new_to_device_edu_content()

eef8132

Merge branch 'develop' into edu-limit-size

0e74f3e

add helper method and refactor EDUs constant

cf7441a

fix trial tests due to twisted lib issue

71a26a4

MadLittleMods reviewed Aug 20, 2025

View reviewed changes

Comment thread tests/rest/client/test_sendtodevice.py Outdated

try to fix test flaky issue

55afc88

MatMaul requested a review from MadLittleMods March 23, 2026 08:57

MadLittleMods removed their request for review March 23, 2026 20:23

Add spec link

6c93d88

MatMaul requested a review from MadLittleMods March 23, 2026 21:36

MadLittleMods reviewed Mar 23, 2026

View reviewed changes

Comment thread synapse/handlers/devicemessage.py Outdated

MadLittleMods approved these changes Mar 23, 2026

View reviewed changes

Update synapse/handlers/devicemessage.py

4b7da00

Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>

MatMaul requested a review from MadLittleMods March 24, 2026 09:00

MadLittleMods merged commit 7fad50f into element-hq:develop Mar 24, 2026
74 of 76 checks passed

MadLittleMods added a commit that referenced this pull request Mar 24, 2026

Demystify HomeserverTestCase.pump()

45d5867

Spawning from #18416 (comment)

MadLittleMods mentioned this pull request Mar 24, 2026

Demystify and deprecate HomeserverTestCase.pump() (Twisted reactor/clock) #19602

Merged

3 tasks

MadLittleMods added a commit that referenced this pull request Mar 25, 2026

Demystify and deprecate HomeserverTestCase.pump() (Twisted reactor/…

f2b325f

…clock) (#19602) Spawning from #18416 (comment)

erikjohnston added a commit that referenced this pull request Mar 27, 2026

Revert "Limit outgoing to_device EDU size to 65536 (#18416)"

cffcba0

This reverts commit 7fad50f.

erikjohnston mentioned this pull request Mar 27, 2026

Revert "Limit outgoing to_device EDU size to 65536" #19614

Merged

erikjohnston mentioned this pull request Mar 27, 2026

Limit to-device EDU sizes #19617

Merged

MadLittleMods mentioned this pull request May 11, 2026

A long queue of to-device messages can prevent outgoing federation working #17035

Closed

FrenchGithubUser pushed a commit to famedly/synapse-upstreaming that referenced this pull request Jun 12, 2026

Demystify and deprecate HomeserverTestCase.pump() (Twisted reactor/…

2196f96

…clock) (element-hq#19602) Spawning from element-hq#18416 (comment)

Uh oh!

Conversation

MatMaul commented May 9, 2025 • edited by MadLittleMods Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mcalinghee commented Aug 20, 2025

Uh oh!

Uh oh!

mcalinghee commented Aug 22, 2025

Uh oh!

MadLittleMods commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MatMaul commented Mar 23, 2026

Uh oh!

Uh oh!

MadLittleMods left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MadLittleMods commented Mar 24, 2026

Uh oh!

MatMaul commented Mar 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

MatMaul commented May 9, 2025 •

edited by MadLittleMods

Loading

MadLittleMods commented Mar 23, 2026 •

edited

Loading