fix(MHPL-615): cap QUIC UDP payload size for Cloudflare Spectrum#3
Open
alepaez wants to merge 1 commit into
Conversation
Players intermittently disconnect when the Hytale proxy runs behind Cloudflare Spectrum. Spectrum does not fragment UDP and drops any datagram too large to forward. The client-facing QUIC codec advertised the quiche default max_udp_payload_size (65527) and ran DPLPMTUD (discoverPmtu=true), so datagrams grew past the Spectrum-forwardable size and were blackholed, stalling sessions until the 60s idle timeout dropped the player. Cap the datagram size in both directions and disable PMTU probing, both driven by new config keys so the value can be tuned per deployment without a rebuild: - max-udp-payload-size (default 1200, QUIC's universal floor) sets both maxRecvUdpPayloadSize (advertised to the client, caps client->proxy, the direction Spectrum blackholes) and maxSendUdpPayloadSize (caps proxy->client) - discover-pmtu (default false) drives discoverPmtu(...) Behind Spectrum start at 1200 and raise toward 1350 once stability is confirmed. For direct (non-Spectrum) UDP exposure set discover-pmtu=true.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Players intermittently disconnect while playing on the Hytale server when it sits behind Cloudflare Spectrum. Confirmed: connecting directly to the origin is stable; through Spectrum it drops. (MHPL-615)
Root cause
Hytale runs on QUIC over UDP (Netty +
quiche, ALPNhytale/2, UDP5520). Two facts collide:maxRecvUdpPayloadSize, so it advertised quiche's defaultmax_udp_payload_size = 65527to clients (letting the client grow its send size toward its local ~1450 MTU), and randiscoverPmtu(true)(probing its own sends upward).IP_DONTFRAGMENT=trueon the sockets removes any fallback.When a grown datagram exceeds the Spectrum path MTU — most often during large-data bursts (voxel chunks / world-map / voice) or after an anycast/route change — it is silently blackholed. QUIC must detect the blackhole and fall back; during that loss window an active session stalls, and with a 60 s idle timeout and no keepalive a bad-enough stall becomes a full disconnect. Small steady-state packets fit under the limit, which is why it's intermittent rather than constant.
Fix
Cap the QUIC datagram size so packets stay within the Spectrum-forwardable size, and stop PMTU probing — both driven by new config keys so the value can be tuned per deployment without a rebuild:
max-udp-payload-size1200maxRecvUdpPayloadSize(advertised to the client → caps client→proxy, the direction Spectrum blackholes) andmaxSendUdpPayloadSize(caps proxy→client). Must be ≥ 1200.discover-pmtufalsediscoverPmtu(...). Off behind Spectrum; settrueonly for direct (non-Spectrum) UDP exposure.Default ships at 1200 — QUIC's universal floor, guaranteed deliverable on any path — so the first test deploy is maximally safe.
How to test
server.jarfrom this branch and deploy it intohytale-proxy(the deployed jar is committed there; this branch is what it's built from).max-udp-payload-size = 1200. Disconnects should stop.max-udp-payload-sizeinhytale-proxy/config.toml(no rebuild — it's a config key) to1252→1350, re-testing stability at each step. Back off one step at the first sign of drops.-Dhyproxy.debugBytes=truecan be used to watch frame sizes during testing.Verification
4.2.9.Finalwith JDK 25 (:common+:proxycompileJavaSUCCESSFUL). No unit tests exist in this repo; live Spectrum testing is the real verification.Notes / scope
InboundForwardingPacketHandler) is intra-cluster and never crosses Spectrum, so it's untouched.config.tomlfiles without the new keys fall back to the defaults (1200 / false).alexandresequeira/mhg-1132-update-protocolbecausemainis frozen (identical to upstreamSantioMC/hyproxy, no Hytale 0.5.5 support) and the deployed proxy is built from the MHG-1132 branch.activeMigration(false)are secondary Spectrum-robustness levers worth revisiting if any drops remain after the MTU cap.