fix(replication): reconcile connected bit UP from shared-memory truth; log corrections (#431)

[kriszyp]

What

Completes the W1 (#431) truth-reconcile consumer that #445 started. The 5s reconcileWorkers pass corrected the edge-triggered connected bit only downward (shared-memory truth says down → flip to false, feeding the wedge re-drive). This PR adds:

1. Up-direction correction — a connected:false entry whose connect edge was lost or never processed (Replication: ~9/11 outgoing peers show connected:false in connectionReplicationMap after restart despite being reachable #289, the founding W1 bug) previously stayed false until findWedgedNodeUrls force-reconnected a perfectly healthy link 30s later. Now truth CONNECTED + fresh liveness corrects the bit in place — no disruptive reconnect. Never-connected (undefined) entries are deliberately not up-corrected: they're mid-connect and owned by the connect/retry path.
2. Correction telemetry — both directions now log the correction (previously the down-correction was silent). Edge-bit-vs-truth disagreement is exactly the desync class W1 exists to retire, so its frequency in production should be observable before the deeper watchdog demotion leans on the truth.

How

The correction rules move into an exported pure helper, reconcileEntryWithTruth(entry, truth, now), following the file's established pure-decision-helper pattern (isReceiveStalled, findWedgedNodeUrls, …). The reconcile loop calls it per (db, peer) entry and owns the logging. Down-correction semantics are unchanged: liveness-reported-at-least-once gate, disconnectedAt stamped only if absent (doesn't restart the wedge clock).

Safety notes

• Up-correction requires state === CONNECTED and fresh liveness (LIVENESS_STALE_MS), so a worker that died without writing DOWN can't cause a false up-correction beyond the staleness window — and the findStaleNodeUrls exited-worker net operates independently of the connected bit.
• A racing up-correction just before the real connect edge arrives is idempotent (the edge sets the same state).

Tests

unitTests/replication/reconcileEntryWithTruth.test.mjs — 8 cases: both corrections, the lastLiveness > 0 gate, wedge-clock preservation, the undefined-entry up-exclusion, and agreement no-ops. Full unitTests/replication/ dir: 266 passing.

Part of #430 (epic) · W1 #431 · follows #445.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces the reconcileEntryWithTruth function in subscriptionManager.ts to reconcile the main thread's connection state with the authoritative shared-memory truth in both directions, along with comprehensive unit tests. Feedback suggests adding an explicit return undefined; at the end of the new function to prevent potential compilation errors under strict TypeScript configurations.

[gemini-code-assist]

To prevent potential compilation errors under strict TypeScript configurations (such as noImplicitReturns: true), it is recommended to add an explicit return undefined; at the end of the function.

Suggested change

	if (entry.connected === false && truth.connected) {
	entry.connected = true;
	entry.disconnectedAt = undefined;
	return 'up';
	}
	}
	if (entry.connected === false && truth.connected) {
	entry.connected = true;
	entry.disconnectedAt = undefined;
	return 'up';
	}
	return undefined;
	}

[claude]

Reviewed; no blockers found.

[kriszyp]

Cross-model review (thorough) found and fixed a blocker — pushed

The Gemini + Codex + Harper-domain adjudication surfaced a real correctness blocker that this PR's up-correction would have introduced, now fixed in 636d825.

The hazard: the per-(db, peer) shared status buffer is keyed only by (db, peer), but its connection-truth slots (CONNECTION_STATE / LAST_LIVENESS / LAST_ERROR, from #445) were written by any connection resolving that buffer — a cache-miss retrieval connection (getRetrievalConnectionByName, which connect()s with connection: this → fires the pong write) and, via the ungated handshake/receive writes, an inbound server connection from the same peer. In the common full-mesh case, an active inbound B→A stream kept (db,B) stamped CONNECTED+fresh-liveness even after the outbound A→B subscription genuinely died — and the new up-correction would resurrect that dead entry to connected:true, so findWedgedNodeUrls skips it → silent one-directional replication halt / divergence. (#445's down-only correction failed safe; the up-correction is what made the contamination consequential, so the fix lands here.)

The fix (root-cause, not defensive): enforce the ownership invariant — only the connection that owns the (db,peer) subscription may write slots 9–12 — at every truth-slot write (close, forceReconnect teardown, handshake, receive, pong), via connectionOwnsTruth(c) = c?.nodeSubscriptions !== undefined. nodeSubscriptions is set by subscribe() and never cleared, and is never set on a retrieval (connect() only) or inbound (no options.connection) connection — a reliable ownership marker checked at write time, not the frozen open-time isSubscriptionConnection snapshot (which the review flagged as incidental ordering). Receive-progress and latency slots are untouched; only the authoritative connection-state slots are owner-gated. This also de-contaminates the cluster_status view (same buffer).

Adjudicated non-issues: Gemini's "up-correction flaps against a fresh disconnect edge" is bounded by the 120s staleness net and subsumed by the ownership fix; the undefined-entry down-correction and the correction && truth narrowing guard are both by-design/load-bearing (the guard is needed for TS narrowing of truth). No changes there.

Full unitTests/replication/ suite green (270 passing), including new connectionOwnsTruth coverage. Keeping this draft until the fix gets a re-review pass over the newly-touched write sites.

[kriszyp]

Focused re-review of the ownership-gate fix — complete & correct

Ran a scoped re-review (Codex + Harper-domain adjudication) over the newly-gated write sites.

Independent completeness check caught one more site the first Codex pass missed and reported clean: the back-pressure-pause liveness write (replicationConnection.ts ~1592, pausedStatus[LAST_LIVENESS_TIME] = Date.now()) was still ungated — a paused inbound/retrieval connection would refresh a dead subscription's liveness, the same masking. Fixed in ac5f177. Grep-verified all six truth-slot (9–12) write sites are now owner-gated: close, forceReconnect teardown, pause, handshake, receive, pong.

The domain pass independently re-derived completeness (not trusting the Codex verdict, which had confabulated) and confirmed:

• No false-negative on the owner — this.nodeSubscriptions is assigned once (subscribe()) and never cleared; options.connection === this; subscribe() runs synchronously before any network handler fires, so a real subscription's DOWN (close/forceReconnect) and CONNECTED (handshake/receive/pong/pause) writes are never suppressed. The subscribe([], false) DB-removal path keeps nodeSubscriptions = [] (!== undefined), so its teardown DOWN still writes.
• Correct exclusions — retrieval (connect() only) and inbound (no options.connection) both fail the gate.
• Ungated slots are correct by design — deriveConnectionTruth reads only slots 9/10; the ungated receive-progress/latency slots feed stall-detection & cluster_status, not the up-correction.
• Up-correction consumer still sound — a non-owner can no longer stamp CONNECTED+fresh-liveness, so it can't drive the resurrection; down-correction only becomes more available.

Verdict: no blockers, no significant concerns. Full unitTests/replication/ suite green (270 passing), incl. new connectionOwnsTruth + reconcileEntryWithTruth coverage.

One noted follow-up (pre-existing, out of scope, NOT this PR): a retrieval connection actively receiving sets RECEIVING_STATUS+RECEIVED_TIME on the shared buffer, which can mask a subscription receive-stall in the looser isReceiveStalled net (#453/#463) — same ungated-slots property, different consumer. Worth folding into the watchdog-consolidation step later.

Marking Ready.