Skip to content

feat(session): coalesce concurrent /history reconstructions per workstream#893

Merged
eous merged 1 commit into
mainfrom
fix/884-history-coalescing
Jul 22, 2026
Merged

feat(session): coalesce concurrent /history reconstructions per workstream#893
eous merged 1 commit into
mainfrom
fix/884-history-coalescing

Conversation

@eous

@eous eous commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Closes #884.

After a node restart every open pane resyncs via REST /history inside the same jitter window (post-#888, one per open coordinator tab as well). Client-side jitter spreads the peak, not the total: per restarted node it was one full reconstruction (load_messages -> verdict decoration -> reasoning extraction -> projection) per connected pane, against a cold process. Concurrent requests for the same (ws_id, limit) now share ONE reconstruction via a single-flight task map in the handler closure.

Design rulings (each carried by an at-site comment)

  • Single-flight only, no TTL cache. The payload depends on live-mutable inputs with no total, cheap invalidation signal (the surface_persisted_reasoning registry toggle emits no per-ws event; cold workstreams have no event counter at all). A joiner's worst-case staleness equals the flight duration — the same window a lone slow request already exposes.
  • Gates before join. Permission / tenant / kind / existence / limit all run per-request above the join; only the caller-independent reconstruction is shared. The payload carries no per-caller field.
  • Fan-out-wipe guard. A shared draw that hit a transient load_messages failure is NOT fanned out: joiners retry once, independently — both clients render a 200-empty as an authoritative pane wipe, and the seedless clear_ui path has no SSE redelivery to repair it. The owner keeps its failed draw; blast radius equals the un-coalesced baseline. ws.history.load_failed rises debug -> warning accordingly.
  • Flight lifecycle. Detached task; awaiters shield it (an owner disconnect cannot strand joiners); pop-in-finally inside the task, so the map only ever holds in-flight work — a single-flight, not a cache. Accepted cost, documented at the shield: a lone reader's mid-flight disconnect no longer aborts the bounded reconstruction (refcount-cancel reviewed and declined: it adds a join-vs-cancel race to a seam that is race-free because the flight is never cancelled).
  • Per-awaiter serialization kept (cost parity with main); sharing pre-rendered bytes declined at the return site (it splits the flight-result contract the retry path needs, for a herd-only micro-win).
  • Scope. One map per cfg mount per process. Interactive routing is owner-node-sticky, so the herd for one ws lands on one process; coordinator /history is served by the console process. Cross-kind collision is impossible (closure-scoped maps).
  • Stale-cursor joiners degrade gracefully to replay_truncated -> resync; no data loss.

Tests

TestHistoryCoalescing — a 7-case matrix: join/share, failed-draw-not-fanned-out, gates-before-join, key-includes-limit, no-cross-flight-reuse, decoration-failure-shared-not-retried, owner-disconnect-leaves-joiner-completing. Driven through httpx.ASGITransport with the real storage backend behind a gated wrapper; every synchronization point is a positive observable edge (the join waits on the handler's ws.history.coalesced record — no wall-clock beats).

Validation

  • Full suite: 9713 passed / 10 skipped.
  • SSE e2e recovery harness, both tiers, on this branch: 6/6 server-contract tests; browser scenarios RECOVERY-READY-STORM-5-nested-2, RECOVERY-READY-RESTART-rows1-trunc1, RECOVERY-READY-COORD-rows1-trunc1 — the restart scenarios drive the truncated-resync -> /history rebuild through the coalesced handler in a real browser, both clients.
  • Review: three pipeline rounds; rounds 2 and 3 unprimed full-surface, both zero-correctness. Declined findings carry their rulings at the site.

Related: #890 (interactive guard-before-wipe, separate PR) — complementary restart-window robustness, no file overlap.

…tream (#884)

After a node restart every open pane resyncs via REST /history inside
the same jitter window; client jitter spreads the peak but not the
total. Concurrent requests for the same (ws_id, limit) now share ONE
reconstruction (load_messages -> decoration -> projection) via a
single-flight task map in the handler closure.

Deliberately single-flight only, no TTL cache: the payload depends on
live-mutable inputs with no total cheap invalidation signal (the
surface_persisted_reasoning registry toggle emits no per-ws event;
cold workstreams have no event counter), so a cache could serve stale
reasoning/approval/cursor state for its whole TTL, while a joiner's
worst-case staleness equals the flight duration -- the window a lone
slow request already exposes.

All auth/tenant/kind/existence gates stay per-request ahead of the
join; only the caller-independent reconstruction is shared. A shared
draw that hit a transient load_messages failure is not fanned out:
joiners retry once, independently, so one storage blip cannot wipe
every coalesced pane (the 200-empty payload renders as an
authoritative empty pane in both clients, and the seedless clear_ui
path has no SSE redelivery to repair it). The flight is a detached
task (awaiters shield it) so an owner disconnect cannot strand
joiners, and each task pops its own key in a finally, so the map only
ever holds in-flight work. ws.history.load_failed rises to warning:
it now names the draw that triggers joiner retries and renders as a
pane wipe.
Copilot AI review requested due to automatic review settings July 22, 2026 05:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds server-side single-flight coalescing to the /history endpoint so that concurrent requests for the same (ws_id, limit) share a single reconstruction pipeline (load → decorate → reasoning extraction → projection), reducing restart-window herd amplification after node restarts.

Changes:

  • Added an in-handler in-flight task map in make_history_handler to coalesce concurrent /history reconstructions per (ws_id, limit), with per-request auth/tenant/kind/existence/limit gates kept above the join point.
  • Introduced a “failed shared draw not fanned out” safeguard: joiners retry once independently when the shared flight’s load_messages raised (to avoid broadcasting a 200-empty pane wipe).
  • Added a dedicated concurrency test matrix (TestHistoryCoalescing) using httpx.ASGITransport to drive truly concurrent requests and assert join/share, gating, keying, lifecycle (no cache), and cancellation semantics.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
turnstone/core/session_routes.py Implements per-(ws_id, limit) single-flight coalescing for /history reconstructions with explicit join/retry lifecycle rules.
tests/test_workstream_endpoints.py Adds deterministic concurrent-request coverage for /history single-flight behavior using an ASGI transport and gated storage wrapper.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@eous
eous merged commit 7f74e95 into main Jul 22, 2026
18 checks passed
@eous
eous deleted the fix/884-history-coalescing branch July 22, 2026 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Server-side /history coalescing for restart-herd truncated resyncs (escalation lever)

2 participants