Replication-apply / full-copy re-encode mints shared structures off the saveStructures CAS (root cause behind #1506)

[kriszyp]

Summary

The replication-receive thread builds its own in-memory classic shared-structures dictionary (its replication-stream / full-copy encounter order) without reconciling through RecordEncoder.saveStructures. Under concurrent v4→v5 upgrade write load this order diverges from the canonical durable order the local HTTP-write thread commits, and because the divergent dictionary is never persisted (and never reloaded — see #1506), every subsequent decode on that thread hits a present-but-wrong structure → Unexpected end of MessagePack data / Data read, but end of buffer not reached → reads return null and the node wedges.

This is the creation path for the divergence that #1506 recovers from. #1506 makes the condition self-healing (durable-wins reload on decode); this issue is the prevention.

Evidence

4-node 4.7.34 → 5.1.14 in-place rolling upgrade @ 100 w/s. On the wedged node, structure-level instrumentation showed:

• The forking thread is the replication-receive thread ([replication] Connected to wss://…, Requesting full copy of database data, received and broadcasting committed update from …).
• That thread never appears in saveStructures (no CAS commit for the table) yet holds a complete in-memory named dictionary that is a permutation of durable (same shapes, different ids) — e.g. in-memory id0 = [tags,id,name,…] vs durable id0 = [level,items,child].
• The local HTTP-write thread does all the saveStructures commits, monotonic and faithful → durable is canonical.

Proposed fix

Make the replication-apply / full-copy re-encode path reconcile structures through the same CAS every other writer uses — i.e. adopt the durable dictionary before minting, and persist any new structure via saveStructures (so all workers converge on one order) — rather than building a private in-memory order. Pairs with the #1455 seed fix (separate issue) so workers start from one canonical order.

Refs #1506, #1453, #1445

— filed by KrAIs (Claude) on Kris's behalf

[kriszyp]

Update: root-cause refinement after a code-level audit (CAS is sound; narrowing the off-CAS source)

Went back through this with the encoder/CAS code in hand (against #1506's kris/struct-reload-recovery branch) plus the msgpackr and rocksdb-js internals. The instrumented finding in the issue body holds up, but a couple of the mechanisms in the surrounding analysis need correcting, and the fix can be narrowed.

What the instrumentation got right

• The wedge is an in-memory divergence on the replication-receive thread: it holds a complete named dictionary that is a permutation of durable (same shapes, different ids), and it never appears in saveStructures.
• Durable is canonical — the HTTP-write thread does all the faithful, monotonic saveStructures commits.
• msgpackr only auto-reloads structures on a missing id, never on a present-but-wrong one, so the permutation never self-heals → Data read, but end of buffer not reached on read-back → null reads until restart.

What needs correcting

The classic shared-structures encode CAS is sound — it does not clobber, and it is not the thing to harden.

• RecordEncoder.saveStructures runs in a rocksdb-js OptimisticTransactionDB transaction. The txn.putSync on the structures key is a write-write conflict point: a concurrent peer's commit makes the later committer fail with ERR_BUSY, and retryOnBusy re-runs the callback against a fresh getBinarySync, where isCompatible then reconciles. Two workers cannot both commit divergent shapes at the same id.
• msgpackr _mergeStructures on the save/reload paths is durable-wins (the one-arg form). The in-memory-wins (two-arg) form is decode-only and is reverted by restoreStructures — including in the catch on an over-read throw.

So the earlier "length-only CAS clobber" framing (and the idea that isFaithfulExtension is load-bearing against a real durable clobber) doesn't hold: durable stays canonical, exactly as the instrumentation shows. isFaithfulExtension and the replayLogs length-only structuresWouldShrink guard are still worth keeping as defense-in-depth for invariant uniformity, but neither is the production mechanism (the unguarded replayLogs case 'structures' path is effectively dormant — nothing currently emits a STRUCTURES-type audit entry; regular put/patch replay goes through the guarded path).

Where the off-CAS permutation is not born

I traced the v5↔v5 full-copy / replication apply path end to end:

• Receive builds event.value = auditRecord.getValue(tableDecoder) — a decoded JS object (using the sender's shipped TABLE_FIXED_STRUCTURE).
• Core Table.ts writeCommit → updateRecord(recordToStore, …) with recordToStore = recordUpdate (that same JS object) → re-encoded through the primary guarded encoder.
• isCopyApply only changes durability/audit/resequencing (WAL-off snapshot, no audit entry, regress guard); it does not bypass the re-encode, and there's no raw-foreign-bytes store path.

Because that re-encode mints through the guarded saveStructures, the v5↔v5 path reconciles to durable by construction — it can't produce the off-CAS permutation. So the divergence is born elsewhere.

Narrowed open question

The remaining suspects for where the receive thread acquires an off-CAS, never-committed, permuted in-memory dictionary:

1. Cross-version apply — a 4.7.34 sender's structures/protocol interpreted by a 5.1.14 receiver during the rolling-upgrade window (prime suspect; this is the only window the repro exercises that the v5↔v5 trace doesn't cover).
2. A runtime-concurrency subtlety in msgpackr's in-memory structure assignment under simultaneous copy-apply + HTTP-write, not visible from static reading of the v5 path.

I'm pinning this with the live repro now (4-node 4.7.34 → 5.1.14 @ 100 w/s), dumping and diffing the durable [Symbol.for('structures'), <table>] across nodes and over time and tying a failing record to a diverged id + its arrival path.

Net for the fix

• fix(encoder): heal a diverged shared-structures dictionary on decode (durable-wins reload) #1506's decode-time durable-wins reload is the correct, sufficient recovery (durable is canonical, so reloading it is exactly right). Not a band-aid.
• Replication-apply / full-copy re-encode mints shared structures off the saveStructures CAS (root cause behind #1506) #1508 (prevention) should make the receive / full-copy re-encode adopt the durable dictionary order before minting, rather than building a private encounter-order dictionary. It is not about hardening the saveStructures CAS (which is sound). Final shape pending the repro result on whether the entry point is cross-version-specific.

— KrAIs (Claude Opus 4.8), on Kris's behalf

[kriszyp]

Repro results — refutes the cross-version hypothesis; confirms in-memory + restart-recoverable

Ran the in-place upgrade repro (incidents/inplace-upgrade-2026-06-11, 4.7.34 → 5.1.14, RATE=100, SEED=20000). Harness note: 5.1.14 rejects http.securePort == operationsApi.network.securePort (both default 9926), so I split http to 9927 — unrelated to this bug, just needed to boot 5.1.14.

The wedge reproduced. Per-node Unexpected end of MessagePack data / Data read, but end of buffer not reached: node1 = 53,469, node2 = 1,357,687, node3 = 40,006, node4 = 1,588,181. node2 + node4 wedged (unhealthy / failed to stabilize); node1 + node3 burst-then-recovered. Ledger parity failed (divergent counts).

Decisive findings

1. Not cross-version-gated. node4 flipped last — all its peers were already on 5.1.14 — and it wedged the hardest (1.59M errors). So a v4 peer is not required. (My earlier comment leaned toward cross-version; the repro refutes it.)

2. Not the copyDb migration. node4's migration completed cleanly (copied 87694 entries, zero errors in the migration log) at 00:14:56. The error storm starts 00:14:59, after migration, in the post-migration replication-setup phase, on [http/N] workers — Error decoding record while setting up subscriptions to v5 peers. This is exactly the replication-receive / full-copy catch-up + read-back path described in the issue.

3. In-memory, restart-recoverable — durable is not corrupted. Restarting a wedged node (node2) brought it healthy with 0 decode errors and it served queries again. This validates fix(encoder): heal a diverged shared-structures dictionary on decode (durable-wins reload) #1506's premise verbatim ("no durable data loss; restart recovers"). fix(encoder): heal a diverged shared-structures dictionary on decode (durable-wins reload) #1506's decode-time durable-wins reload does in-process exactly what the restart does → it's the correct, sufficient recovery.

4. Real divergence while wedged. Post-restart counts stay short and divergent (127301 / 117539 / 127290 / 107311 vs expected ~127306) — records missed during the wedge don't backfill on restart. So this is an outage and data-divergence class issue, not just log noise.

5. Non-deterministic, matching the original report: 2 of 4 nodes recovered on their own; which nodes wedge is load/timing dependent.

What this means for the fix

• The off-CAS in-memory permutation is born in a runtime-concurrency window during the post-migration full-copy / catch-up re-encode (multiple [http] workers minting structures in different encounter orders, with the durable-wins reconcile lagging a read-back) — not cross-version, and not a durable CAS clobber. A static trace of the v5↔v5 apply path shows it re-encodes through the guarded encoder and reconciles by construction, so this only manifests under concurrent load, which is why it's intermittent.
• fix(encoder): heal a diverged shared-structures dictionary on decode (durable-wins reload) #1506 (recovery): ship it — the decode-heal is the right reconciliation and it's what makes a wedged node self-recover without a restart.
• Replication-apply / full-copy re-encode mints shared structures off the saveStructures CAS (root cause behind #1506) #1508 (prevention): have the receive / full-copy re-encode adopt and pin the durable dictionary order before the workers mint, eliminating the per-worker encounter-order window. It is not about hardening the saveStructures CAS (sound) and not about a v4 peer.

(Couldn't capture the stranded in-memory structures diff directly — the restart healed it before I could dump — but the restart test gives the stronger durable-vs-in-memory answer.)

— KrAIs (Claude Opus 4.8), on Kris's behalf

[kriszyp]

Exact mechanism pinned (instrumented capture) + #1506 validated end-to-end

Built a harper-pro:wip from the #1506 core branch with a one-shot-per-table dump in RecordEncoder.decode that, on the first structure-mismatch, logs the worker's in-memory this.structures vs durable getStructures() and the failing record bytes — before the heal runs. Re-ran the same 4.7.34 → 5.1.14 @ 100 w/s repro. Here's the captured divergence (node1, events):

Durable is correctly seeded (so #1455 works) — a Map{named: 19, typed: 36}:

• durable named id0 = [level, items, child] (a 3-field nested sub-object shape)
• durable named id2… = top-level record shapes, field order [tags, id, name, nested, num, flag, created, attr_N, …] (tags-first)

The worker's in-memory dict has the same 19 shapes but assigned to different ids:

• worker id0 = [id, name, tags, num, flag, nested, created, attr_9, …] (a 10-field top-level record, id-first) — a shape durable never has at id0.

So the worker built its named dictionary in a different encounter order than the durable seed. The failing record references named id 0 (0x40); it was encoded against durable id0 (nested, 3 fields), and the worker decodes it against its id0 (top-level, 10 fields) → reads past end → Unexpected end of MessagePack data. msgpackr only auto-reloads on a missing id, never present-but-wrong, so it never self-heals.

Root of the divergence

Shared-structure ids are assigned by per-builder encounter order. The migration seed-builder (copyDb's observer encoder) encountered the nested sub-object shapes first (durable id0/id1) and the top-level records later; the runtime worker encountered a top-level record first (id0). Same shapes, different ids. The worker minted id0 from an unadopted/empty dictionary — i.e. it did not adopt the durable seed order before its first mint (a timing/seed-adoption window; the from-empty mint is proven by the worker's id0 being a brand-new shape rather than an append at id ≥ 19). A secondary contributor: the same logical top-level record appears in two field orders (migration tags-first vs runtime id-first), which creates distinct shapes — but the wedge itself is the divergent id0 assignment, not the field-order count.

So this confirms the issue's framing (off-CAS encounter-order in-memory dictionary on the receive path) with the concrete dict diff, and corrects the earlier "durable clobber / cross-version" theories: durable is canonical and seeded; the fault is the worker's in-memory id assignment diverging from it.

#1506 validated end-to-end

Same repro, the #1506-heal image vs stock 5.1.14:

	stock 5.1.14	#1506 (heal)
flips	all 4 fail to stabilize	all 4 pass (~36s each)
decode errors	~1.5M, nodes wedge	0 surfaced (1 silent dump+heal)
ledger parity	divergent / FAIL	exact (36936 on all 4 nodes)
replication health	FAIL	PASS

The durable-wins reload-on-present-but-wrong realigns the worker to durable and the record decodes — a wedged, diverging upgrade becomes a clean, fully-converged one.

Net

• fix(encoder): heal a diverged shared-structures dictionary on decode (durable-wins reload) #1506 (recovery): ship it — proven to convert the wedge into a clean upgrade with exact convergence.
• Replication-apply / full-copy re-encode mints shared structures off the saveStructures CAS (root cause behind #1506) #1508 (prevention): make the receive / full-copy re-encode adopt and pin the durable dictionary order before any worker mints, so a worker can never assign id0 (or any id) to a shape that disagrees with the seeded durable order. (Normalizing record field order so the same logical record always yields one shape would additionally shrink the dict, but is secondary to the id-assignment pin.) This is the typed/named seed-adoption family — related to fix(migration): persist canonical classic structures so v5 workers don't fork (#1453) #1455 (seed) and Migration canonical-structures seed (#1455) is discarded by the typed-table runtime (format mismatch) -> workers mint from empty #1509 (typed seed).

— KrAIs (Claude Opus 4.8), on Kris's behalf

[kriszyp]

Exact trigger captured — and a correction to my earlier "CAS is sound" claim

I instrumented a stock-5.1.14 image (compiled-dist logging only: at copyDb's persist-guard, at every saveStructures, and at the decode error), migrated a node, and drove writes. This captures the birth directly.

The sequence (captured)

1. copyDb never writes the canonical seed. BIRTH_COPYDB table=events/ canonicalStructures=undefined WILL_PERSIST=false. The observer encoder's saveStructures is never called during the migration re-encode — the migrated records are own/inline (self-describing 0xd4), so nothing mints shared structures, so canonicalStructures stays undefined and the if (isPrimary && canonicalStructures?.length) write is skipped. Durable is left as the raw v4 structures copy. So fix(migration): persist canonical classic structures so v5 workers don't fork (#1453) #1455's seed is inert here — not because of a Map.length skip, but because the observer never mints.

2. Workers mint concurrently from the empty dict in divergent encounter orders. BIRTH_SAVE shows main/0 + http/5 committing id0 = [level,items,child] (nested-first), then http/1 committing id0 = [id,name,tags,…] (top-level-first). Different worker threads, different id0, all committed=true (4 nested-id0 commits, then 26 top-level-id0).

3. The length-only CAS admits the divergent commit → durable clobber. http/1 gets one committed=false (durable already len 6), re-packs, then committed=true at len 7 — writing its whole top-level-first dictionary, overwriting durable's nested-first prefix. The length check (lastNamedStructuresLength === durable.length) passes; content is never compared.

4. BIRTH_MISMATCH proves it's a durable clobber. At the failing read, in-memory == durable (both id0 = top-level), yet the record references id0 and fails — because the record was written when durable id0 = nested, then durable id0 was overwritten to top-level. Reload-durable can't heal it; durable itself is the clobbered version.

Correction to my earlier comments

I previously said the encode CAS is sound (OptimisticTransactionDB write-write conflict + durable-wins) and that isFaithfulExtension wasn't load-bearing. That was wrong for the multi-worker-mint-from-empty case. The write-write check only catches concurrent same-key writes; it does not catch a worker committing a divergent-content dictionary of a coincidentally-equal length after a clean read — which is exactly what http/1 does. So isFaithfulExtension is the load-bearing fix: it rejects the divergent prefix, forces a reload+reconcile, and prevents the durable clobber. (This is why the #1506 image runs all-green with exact convergence while stock 5.1.14 clobbers and wedges.)

Final root cause — two compounding defects

• (A) Missing canonical seed (fix(migration): persist canonical classic structures so v5 workers don't fork (#1453) #1455 inert / Migration canonical-structures seed (#1455) is discarded by the typed-table runtime (format mismatch) -> workers mint from empty #1509 family): the classic observer never mints during migration → canonicalStructures undefined → seed never written → durable empty → workers mint from empty concurrently in divergent orders. Fix: have copyDb write an adoptable canonical seed (mirror the runtime's typed mode; the Migration canonical-structures seed (#1455) is discarded by the typed-table runtime (format mismatch) -> workers mint from empty #1509 fix removes this precondition).
• (B) Length-only saveStructures CAS admits divergent-content commits → durable append-only violation (id0 shape changes). Fix: isFaithfulExtension (fix(encoder): heal a diverged shared-structures dictionary on decode (durable-wins reload) #1506) — the load-bearing correctness guard.

On correctness (re: an earlier point)

The decode-heal alone is not a correctness fix: it only fires on a thrown mismatch, so same-field-count-but-reordered shapes decode silently to wrong values; and once durable is clobbered, reload can't recover. The repro's ledger_parity/acked_spot_check gates only check counts/presence, never values, so "green" doesn't rule out silent corruption. The correctness fixes are (A) the seed + (B) isFaithfulExtension; the heal is the in-memory-recovery layer that works once durable is kept consistent.

So: ship #1506 (isFaithfulExtension + heal) and the #1509 seed fix together — A removes the precondition, B prevents the clobber, and the heal recovers any residual in-memory divergence.

— KrAIs (Claude Opus 4.8), on Kris's behalf