feat(replication): gracefully drain in-flight blob sends before worker shutdown

[kriszyp]

Summary

Fix (2) of the deploy-reload blob-divergence root cause (harper-pro#527 is the receiver-side fix (1)). A rolling http_workers restart (deploy reload) was tearing down a worker mid-stream while it was sending a replication blob to a peer; the peer then treated the blob as diverged until its source-idle watchdog fired and stamped a PENDING stub. This drains in-flight blob sends before the worker shuts down, so a healthy transfer finishes instead of being interrupted.

Requires the core drain hook — bumps the core submodule to HarperFast/harper#1621 (companion PR).

What changed

• replication/blobSendDrain.ts (new): per-worker tracker of in-flight blob sends + their per-chunk progress, and drainBlobSends(deadline) — resolves when every send has finished, stalled (no bytes for ~5s), or the absolute deadline passes.
• replication/replicationConnection.ts (sendBlobs): registers each send, marks progress as bytes are written (including around backpressure waits and the terminal finished:true frame flush), ends it in finally, and stops starting new sends once the worker is draining.
• replication/replicator.ts: registers the drain against core's registerShutdownDrain, gated on hasProgressingBlobSends() (only a progressing send extends the shutdown backstops).

Design

• Reset-on-bytes: a large transfer that keeps flowing is never treated as stalled and is allowed to finish, bounded by an absolute 10-minute ceiling (replication.blobSendDrainTimeout; 0 disables). Only a send that goes silent for the stall window is abandoned.
• Availability-safe: the #1417 pre-started replacement worker is already serving, so a lingering draining worker costs no capacity.
• Correctness backstop: any send that doesn't finish is still made correct by fix (1) — the receiver aborts + re-requests on reconnect (harper-pro#527). So this is a bounded optimization, not a correctness requirement.

Where to look / open items

• Deploy-rollout latency: because the core restart throttle now waits on a draining worker, a large progressing send can delay the next worker's restart up to the ceiling (bounded; capacity preserved). Surfaced in the core PR; a follow-up could stop counting a draining worker against maxWorkersDown.
• The terminal-frame flush is best-effort under no-backpressure (the frame may sit in the kernel TCP buffer); fix (1) covers the miss.

Unit tests cover the pure helpers (isSendProgressing, isDrainComplete, hasProgressingBlobSends) and the drain/quiesce behavior. Cross-model reviewed (Codex + Harper-domain adjudication; Codex's 1 blocker + 5 significant all addressed; the Gemini/agy leg hung out and was skipped).

Generated by Claude (Opus 4.8).

[kriszyp]

📖 Docs: HarperFast/documentation#569 documents the new replication.blobSendDrainTimeout option. Core companion: HarperFast/harper#1621. (comment by Claude/KrAIs)

[gemini-code-assist]

Code Review

This pull request introduces a per-worker tracker (blobSendDrain.ts) to gracefully drain in-flight replication blob sends before a worker shuts down, preventing data divergence on the receiver side. It registers this shutdown drain in replicator.ts and integrates tracking hooks in replicationConnection.ts, accompanied by comprehensive unit tests. The review feedback suggests using optional chaining when accessing ws._socket (e.g., ws._socket?.writableNeedDrain) to prevent potential TypeError crashes if the socket is closed or destroyed mid-stream.

[claude]

Reviewed; no blockers found.