feat(session): coalesce concurrent /history reconstructions per workstream#893
Merged
Conversation
…tream (#884) After a node restart every open pane resyncs via REST /history inside the same jitter window; client jitter spreads the peak but not the total. Concurrent requests for the same (ws_id, limit) now share ONE reconstruction (load_messages -> decoration -> projection) via a single-flight task map in the handler closure. Deliberately single-flight only, no TTL cache: the payload depends on live-mutable inputs with no total cheap invalidation signal (the surface_persisted_reasoning registry toggle emits no per-ws event; cold workstreams have no event counter), so a cache could serve stale reasoning/approval/cursor state for its whole TTL, while a joiner's worst-case staleness equals the flight duration -- the window a lone slow request already exposes. All auth/tenant/kind/existence gates stay per-request ahead of the join; only the caller-independent reconstruction is shared. A shared draw that hit a transient load_messages failure is not fanned out: joiners retry once, independently, so one storage blip cannot wipe every coalesced pane (the 200-empty payload renders as an authoritative empty pane in both clients, and the seedless clear_ui path has no SSE redelivery to repair it). The flight is a detached task (awaiters shield it) so an owner disconnect cannot strand joiners, and each task pops its own key in a finally, so the map only ever holds in-flight work. ws.history.load_failed rises to warning: it now names the draw that triggers joiner retries and renders as a pane wipe.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds server-side single-flight coalescing to the /history endpoint so that concurrent requests for the same (ws_id, limit) share a single reconstruction pipeline (load → decorate → reasoning extraction → projection), reducing restart-window herd amplification after node restarts.
Changes:
- Added an in-handler in-flight task map in
make_history_handlerto coalesce concurrent/historyreconstructions per(ws_id, limit), with per-request auth/tenant/kind/existence/limit gates kept above the join point. - Introduced a “failed shared draw not fanned out” safeguard: joiners retry once independently when the shared flight’s
load_messagesraised (to avoid broadcasting a 200-empty pane wipe). - Added a dedicated concurrency test matrix (
TestHistoryCoalescing) usinghttpx.ASGITransportto drive truly concurrent requests and assert join/share, gating, keying, lifecycle (no cache), and cancellation semantics.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| turnstone/core/session_routes.py | Implements per-(ws_id, limit) single-flight coalescing for /history reconstructions with explicit join/retry lifecycle rules. |
| tests/test_workstream_endpoints.py | Adds deterministic concurrent-request coverage for /history single-flight behavior using an ASGI transport and gated storage wrapper. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #884.
After a node restart every open pane resyncs via REST
/historyinside the same jitter window (post-#888, one per open coordinator tab as well). Client-side jitter spreads the peak, not the total: per restarted node it was one full reconstruction (load_messages-> verdict decoration -> reasoning extraction -> projection) per connected pane, against a cold process. Concurrent requests for the same(ws_id, limit)now share ONE reconstruction via a single-flight task map in the handler closure.Design rulings (each carried by an at-site comment)
surface_persisted_reasoningregistry toggle emits no per-ws event; cold workstreams have no event counter at all). A joiner's worst-case staleness equals the flight duration — the same window a lone slow request already exposes.load_messagesfailure is NOT fanned out: joiners retry once, independently — both clients render a 200-empty as an authoritative pane wipe, and the seedless clear_ui path has no SSE redelivery to repair it. The owner keeps its failed draw; blast radius equals the un-coalesced baseline.ws.history.load_failedrises debug -> warning accordingly.shieldit (an owner disconnect cannot strand joiners); pop-in-finallyinside the task, so the map only ever holds in-flight work — a single-flight, not a cache. Accepted cost, documented at the shield: a lone reader's mid-flight disconnect no longer aborts the bounded reconstruction (refcount-cancel reviewed and declined: it adds a join-vs-cancel race to a seam that is race-free because the flight is never cancelled)./historyis served by the console process. Cross-kind collision is impossible (closure-scoped maps).replay_truncated-> resync; no data loss.Tests
TestHistoryCoalescing— a 7-case matrix: join/share, failed-draw-not-fanned-out, gates-before-join, key-includes-limit, no-cross-flight-reuse, decoration-failure-shared-not-retried, owner-disconnect-leaves-joiner-completing. Driven throughhttpx.ASGITransportwith the real storage backend behind a gated wrapper; every synchronization point is a positive observable edge (the join waits on the handler'sws.history.coalescedrecord — no wall-clock beats).Validation
RECOVERY-READY-STORM-5-nested-2,RECOVERY-READY-RESTART-rows1-trunc1,RECOVERY-READY-COORD-rows1-trunc1— the restart scenarios drive the truncated-resync ->/historyrebuild through the coalesced handler in a real browser, both clients.Related: #890 (interactive guard-before-wipe, separate PR) — complementary restart-window robustness, no file overlap.