Synapse 1.152.0 spins at 100% CPU after ~7 minutes under E2E keys/upload load (regression vs 1.151.0)

[evgn-k]

Description

After upgrading from Synapse v1.151.0 to v1.152.0, the homeserver process
consistently hangs after ~7 minutes of normal operation. Reverting to
v1.151.0 restores stability.

Symptoms

• CPU pegs at 100% for one Python thread
• All HTTP endpoints (including /health) stop responding (curl from
  inside container times out at 30s)
• Logging completely stops; last entry is always
  synapse.handlers.e2e_keys - 955 - INFO - Adding one_time_keys ... for device 'XXXX' for user '@user:server'
• Container is not OOM-killed; RSS stays modest (~115-130 MiB)
• Healthcheck curl -fSs http://localhost:8008/health starts failing
  with timeout

Reproduction

Reproduced twice in the same session, with realistic load (~5
concurrent E2E-enabled clients, federated server):

• First start after redeploy 22:03 → hang 22:10:28 (7 min 28 sec uptime)
• After docker restart 22:40:30 → healthy 22:43:25 → second hang
  22:48:25 (5 min after healthy, 7 min 55 sec after start)

Workaround

Reverted to v1.151.0 — 9 min stable, no regression observed.

Related context (might or might not share a root cause)

On 1.151.0 we previously hit a different issue where
synapse.handlers.worker_lock:280 logged
WARNING - Lock timeout is getting excessive: <stream_id> with
stream_id reaching thousands of decimal digits (~19 warnings/sec),
exhausting host RAM via stdout buffering. Worked around with a
log_config override silencing synapse.handlers.worker_lock to ERROR.

The 1.152.0 hang occurs with this override active — the WARNINGs are
suppressed, but the hang's last log line is consistently from
synapse.handlers.e2e_keys (Adding one_time_keys), which is the same
code path that previously triggered the worker_lock WARNINGs. This may
suggest both issues share a root cause around stream_id growth in
worker locks during E2E key upload.

Environment

• Synapse: v1.152.0 (regression) / v1.151.0 (works)
• Python: 3.13.12 (Docker image default)
• DB: PostgreSQL 18.3 (postgres:18-bookworm)
• Deployment: single-process (no separate workers), Docker on Ubuntu
  24.04 host
• Federation enabled, ~5 concurrent E2E-enabled clients
  (Element Desktop, FluffyChat, Cinny)

What I haven't captured (open to suggestions)

• Python-level stack trace from the hung process (py-spy/faulthandler)
• strace/perf
• DB lock state at hang time

Happy to collect any of these on the next reproduction if pointed at
specific commands.

[reivilibre]

As a note, I would normally expect the word 'hangs' to mean not being spinning at 100% CPU.

But I think this is a duplicate of #19315 / #19588 which was fixed by #19394 and should be in the next release