Restart-safe observation identity and turn-scoped events (FIG-615)#135
Merged
Conversation
Give each live replay store construction a fresh incarnation and include it in recoverable event identity. Persisted applied ids can now survive restarts without suppressing events at reused cursors, while gap replacement still clears a bounded deduplication window. Release-Notes: Changed: Recoverable chat event identities now include a replay-store incarnation, so hosts may safely persist applied identities across process restarts.
Carry optional turn identity on core and remote observation envelopes while leaving TurnActivity unchanged. Physical-turn activity and commits are attributed; queue, process, Agent Frame, and runtime-operation observations remain unscoped. Remote protocol v17 also carries replay incarnation identity and rejects v16 through exact-version validation. Release-Notes: Breaking: Remote protocol v17 adds required replay_incarnation_id and optional turn_id fields to session observation event envelopes; remote hosts must upgrade their exact-version contract.
Carry turn identity on product completion events and transient browser rows so late completion for one turn cannot retract a newer turn's output. Cover the production reducer race and document the scoped settlement behavior. Release-Notes: Changed: Agent Workbench terminal settlement now retracts only transient rows from the completed turn, preserving newer in-flight turn output.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The three recoverable-chat items FIG-610 left open. Each needed an upstream change — the observation stream simply did not carry what a host needed — so all three land as one protocol bump (16 → 17).
1. Event identity is now restart-safe
RecoverableChatEventIdwas(session_id, cursor). The in-memory replay position restarts at 0, so a cursor is reused after a restart and a host retaining applied identities could silently suppress a genuinely new event.Investigated first: the durable
SessionRevisiondoes not fix this — it advances only on commit, so a crash before a commit leaves it unchanged while the cursor resets. The fix is a replay-store incarnation id: a UUID generated when the store is constructed, neither persisted nor ordered, because it only needs to be distinct per instantiation. Identity becomes(session_id, replay_incarnation_id, cursor).Consequence worth noting: retaining applied ids across a restart is now safe — a fresh incarnation cannot match a pre-restart identity — so the old "never persist these" warning is removed rather than reworded. Gap-clearing stays as belt-and-braces and the applied set is bounded (4,096, FIFO).
2. Turn identity on the observation envelope
Nothing told a host which turn an event belonged to (
TurnActivityhas no turn id; neither did the envelope), so the workbench had to retract transient rows session-globally — turn A's terminal wiped turn B's in-flight prose.turn_idgoes on the envelope, not onTurnActivity: the envelope already carries session/revision/cursor, so this scopes every payload kind. Optional, becauseQueueChanged/ProcessChanged/AgentFrameSwitchedare genuinely not turn-owned. Turn activities andCommittedpopulate it.3.
Committedgains turn identity and nothing elseNo read view, no committed-delta. The host still refetches
/api/statefor content but can now fence that fetch precisely by turn. A full read view would grow unboundedly with session length and dominate stream bandwidth on long chats; the stale-snapshot race is already closed by FIG-610's generation fencing.Workbench
finishTransientRows(turn_id)retracts only state belonging to the terminating turn; browser dedup uses the three-part identity. All FIG-610 behaviour intact (generation fence, authoritative dedup replacement, per-session cursor scoping, canonical merge).Verification
Protocol v17 with exact-version validation — a v16 peer is rejected with a structured
UnsupportedProtocolVersion, no panic or silent mismatch. 2,489 workspace tests, strict clippy, docs lint, and the Restate E2E lane on both SQLite and PostgreSQL. Key regressions independently re-run (9/9), includingrecoverable_chat_restart_identity_does_not_depend_on_gap_clearing— which proves a retained pre-restart identity no longer suppresses a real event.Breaking: v17 envelopes require
replay_incarnation_id; remote hosts must adopt the new envelope and identity key.Closes FIG-615.