Skip to content

fix(MHPL-615): cap QUIC UDP payload size for Cloudflare Spectrum#3

Open
alepaez wants to merge 1 commit into
alexandresequeira/mhg-1132-update-protocolfrom
alexandresequeira/mhpl-615-fix-hytale-proxy-disconnection-issues-behind-cloudflare
Open

fix(MHPL-615): cap QUIC UDP payload size for Cloudflare Spectrum#3
alepaez wants to merge 1 commit into
alexandresequeira/mhg-1132-update-protocolfrom
alexandresequeira/mhpl-615-fix-hytale-proxy-disconnection-issues-behind-cloudflare

Conversation

@alepaez

@alepaez alepaez commented Jul 14, 2026

Copy link
Copy Markdown

Problem

Players intermittently disconnect while playing on the Hytale server when it sits behind Cloudflare Spectrum. Confirmed: connecting directly to the origin is stable; through Spectrum it drops. (MHPL-615)

Root cause

Hytale runs on QUIC over UDP (Netty + quiche, ALPN hytale/2, UDP 5520). Two facts collide:

  1. Cloudflare Spectrum does not fragment UDP — it drops any datagram too large to forward at its edge (Spectrum limitations). The edge→origin leg, minus QUIC/UDP/IP overhead on an anycast path with ICMP "Packet Too Big" filtered, has a lower effective MTU than a normal path.
  2. The client-facing QUIC codec invited large datagrams. It never set maxRecvUdpPayloadSize, so it advertised quiche's default max_udp_payload_size = 65527 to clients (letting the client grow its send size toward its local ~1450 MTU), and ran discoverPmtu(true) (probing its own sends upward). IP_DONTFRAGMENT=true on the sockets removes any fallback.

When a grown datagram exceeds the Spectrum path MTU — most often during large-data bursts (voxel chunks / world-map / voice) or after an anycast/route change — it is silently blackholed. QUIC must detect the blackhole and fall back; during that loss window an active session stalls, and with a 60 s idle timeout and no keepalive a bad-enough stall becomes a full disconnect. Small steady-state packets fit under the limit, which is why it's intermittent rather than constant.

Fix

Cap the QUIC datagram size so packets stay within the Spectrum-forwardable size, and stop PMTU probing — both driven by new config keys so the value can be tuned per deployment without a rebuild:

Key Default Effect
max-udp-payload-size 1200 Sets maxRecvUdpPayloadSize (advertised to the client → caps client→proxy, the direction Spectrum blackholes) and maxSendUdpPayloadSize (caps proxy→client). Must be ≥ 1200.
discover-pmtu false Drives discoverPmtu(...). Off behind Spectrum; set true only for direct (non-Spectrum) UDP exposure.

Default ships at 1200 — QUIC's universal floor, guaranteed deliverable on any path — so the first test deploy is maximally safe.

How to test

  1. Build server.jar from this branch and deploy it into hytale-proxy (the deployed jar is committed there; this branch is what it's built from).
  2. Play through Spectrum with the default max-udp-payload-size = 1200. Disconnects should stop.
  3. To recover throughput, bump max-udp-payload-size in hytale-proxy/config.toml (no rebuild — it's a config key) to 12521350, re-testing stability at each step. Back off one step at the first sign of drops.
  4. -Dhyproxy.debugBytes=true can be used to watch frame sizes during testing.

Verification

  • Compiles cleanly against Netty QUIC 4.2.9.Final with JDK 25 (:common + :proxy compileJava SUCCESSFUL). No unit tests exist in this repo; live Spectrum testing is the real verification.

Notes / scope

  • Client-facing codec only. The proxy→backend leg (InboundForwardingPacketHandler) is intra-cluster and never crosses Spectrum, so it's untouched.
  • Backwards compatible. Existing config.toml files without the new keys fall back to the defaults (1200 / false).
  • Base branch: targets alexandresequeira/mhg-1132-update-protocol because main is frozen (identical to upstream SantioMC/hyproxy, no Hytale 0.5.5 support) and the deployed proxy is built from the MHG-1132 branch.
  • Follow-up (not in this PR): the 60 s idle timeout with no keepalive and activeMigration(false) are secondary Spectrum-robustness levers worth revisiting if any drops remain after the MTU cap.

Players intermittently disconnect when the Hytale proxy runs behind
Cloudflare Spectrum. Spectrum does not fragment UDP and drops any datagram
too large to forward. The client-facing QUIC codec advertised the quiche
default max_udp_payload_size (65527) and ran DPLPMTUD (discoverPmtu=true),
so datagrams grew past the Spectrum-forwardable size and were blackholed,
stalling sessions until the 60s idle timeout dropped the player.

Cap the datagram size in both directions and disable PMTU probing, both
driven by new config keys so the value can be tuned per deployment without
a rebuild:

- max-udp-payload-size (default 1200, QUIC's universal floor) sets both
  maxRecvUdpPayloadSize (advertised to the client, caps client->proxy, the
  direction Spectrum blackholes) and maxSendUdpPayloadSize (caps proxy->client)
- discover-pmtu (default false) drives discoverPmtu(...)

Behind Spectrum start at 1200 and raise toward 1350 once stability is
confirmed. For direct (non-Spectrum) UDP exposure set discover-pmtu=true.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant