raft §8: a sent forwarded proposal parks forever if the leader crashes (no leader-change reaping)

[keddie]

🤖 This issue was generated by Claude on behalf of @keddie.

Found by the exactly-once/deferred-leak audit during the ProposalForwarder extraction (PR #1237). Pre-existing (unchanged by that PR; carried verbatim from before) — same parked-forever shape as the §6.4 read-machine hang (#1235).

Severity: 🟠 liveness (caller hangs; the requestId dedup makes caller-driven retry exactly-once-safe, so this may be intended recovery — but nothing spontaneous completes the deferred, and the map entry leaks for the engine's lifetime).

Bug. Once a Forward has been sent (entry in forwardedProposals, id no longer in waitingForLeader), its deferred is completed only by (a) a matching ForwardResponse, or (b) actor teardown failAll. If the leader crashes (vs. stepping down, where onForward's watcher replies NotLeader), no response ever arrives, and leader-change/re-election touches only waitingForLeader (via flush) — never already-sent entries. The follower's readIndex()-style d.await() in proposeWithRequestId suspends indefinitely; the only escape is caller-side cancellation/timeout.

Fix options. On an observed leader change (or after a bounded no-response interval), either fail outstanding sent forwards with LeadershipLostException, or move their ids back into waitingForLeader for re-send to the new leader (safe under the dedupKey — the leader dedups a re-sent proposal). The former is simpler and matches the transfer/read machines' failAll-on-relinquish behavior.

Consider grouping with #1235 (both are "deferred parked forever on a leadership event" bugs) if a spec-fix sweep addresses them together.

[keddie]

🤖 This comment was generated by Claude on behalf of @keddie.

Reframed as needs-design — the reaping fix (PR #1241) breaks graceful leadership transfer, and the right disposition is an API-contract call.

Diagnosis (confirmed). Fixing this by reaping sent forwards on a leader-change is too eager: a graceful §3.10 transfer keeps the old leader alive, so the forward resolves normally (commit / NotLeader+retry / queue-flush). The reap preempts that with a spurious LeadershipLostException that escapes an unretried propose, breaking ClusterClientMultiClientHardeningE2ETest ("no double-apply"). A reap must fire only when the old leader is genuinely gone (crashed), not merely changed.

Options:

• (b) bounded no-response reaping — arm a grace timer when a forward is sent; reap only if no ForwardResponse within it. Separates crash (silent past bound → reaped) from transfer (resolves first → not reaped). Cost: a timer + injected dispatcher/clock in the currently timer-free, actor-confined ProposalForwarder.
• (d) caller-timeout is the contract — a bare propose() hanging on a crashed remote leader is the caller's responsibility to time out and retry (the universal distributed-propose pattern; every consumer already does it; requestId dedup makes retry exactly-once-safe). Drop the reap; document propose() requires caller timeout+retry. Close as working-as-intended.

Related: #1235 (§6.4 read hang, fixed) and #1243 are the sibling "deferred parks on a leadership event" cluster — but note #1238 differs: it's a REMOTE leader's crash (caller-timeout-shaped), not a local leadership loss.