Skip to content

Restart-safe observation identity and turn-scoped events (FIG-615)#135

Merged
SamGalanakis merged 3 commits into
mainfrom
samuel-fig-615-observation-identity
Jul 25, 2026
Merged

Restart-safe observation identity and turn-scoped events (FIG-615)#135
SamGalanakis merged 3 commits into
mainfrom
samuel-fig-615-observation-identity

Conversation

@SamGalanakis

Copy link
Copy Markdown
Collaborator

The three recoverable-chat items FIG-610 left open. Each needed an upstream change — the observation stream simply did not carry what a host needed — so all three land as one protocol bump (16 → 17).

1. Event identity is now restart-safe

RecoverableChatEventId was (session_id, cursor). The in-memory replay position restarts at 0, so a cursor is reused after a restart and a host retaining applied identities could silently suppress a genuinely new event.

Investigated first: the durable SessionRevision does not fix this — it advances only on commit, so a crash before a commit leaves it unchanged while the cursor resets. The fix is a replay-store incarnation id: a UUID generated when the store is constructed, neither persisted nor ordered, because it only needs to be distinct per instantiation. Identity becomes (session_id, replay_incarnation_id, cursor).

Consequence worth noting: retaining applied ids across a restart is now safe — a fresh incarnation cannot match a pre-restart identity — so the old "never persist these" warning is removed rather than reworded. Gap-clearing stays as belt-and-braces and the applied set is bounded (4,096, FIFO).

2. Turn identity on the observation envelope

Nothing told a host which turn an event belonged to (TurnActivity has no turn id; neither did the envelope), so the workbench had to retract transient rows session-globally — turn A's terminal wiped turn B's in-flight prose.

turn_id goes on the envelope, not on TurnActivity: the envelope already carries session/revision/cursor, so this scopes every payload kind. Optional, because QueueChanged/ProcessChanged/AgentFrameSwitched are genuinely not turn-owned. Turn activities and Committed populate it.

3. Committed gains turn identity and nothing else

No read view, no committed-delta. The host still refetches /api/state for content but can now fence that fetch precisely by turn. A full read view would grow unboundedly with session length and dominate stream bandwidth on long chats; the stale-snapshot race is already closed by FIG-610's generation fencing.

Workbench

finishTransientRows(turn_id) retracts only state belonging to the terminating turn; browser dedup uses the three-part identity. All FIG-610 behaviour intact (generation fence, authoritative dedup replacement, per-session cursor scoping, canonical merge).

Verification

Protocol v17 with exact-version validation — a v16 peer is rejected with a structured UnsupportedProtocolVersion, no panic or silent mismatch. 2,489 workspace tests, strict clippy, docs lint, and the Restate E2E lane on both SQLite and PostgreSQL. Key regressions independently re-run (9/9), including recoverable_chat_restart_identity_does_not_depend_on_gap_clearing — which proves a retained pre-restart identity no longer suppresses a real event.

Breaking: v17 envelopes require replay_incarnation_id; remote hosts must adopt the new envelope and identity key.

Closes FIG-615.

Give each live replay store construction a fresh incarnation and include it in recoverable event identity. Persisted applied ids can now survive restarts without suppressing events at reused cursors, while gap replacement still clears a bounded deduplication window.

Release-Notes: Changed: Recoverable chat event identities now include a replay-store incarnation, so hosts may safely persist applied identities across process restarts.
Carry optional turn identity on core and remote observation envelopes while leaving TurnActivity unchanged. Physical-turn activity and commits are attributed; queue, process, Agent Frame, and runtime-operation observations remain unscoped. Remote protocol v17 also carries replay incarnation identity and rejects v16 through exact-version validation.

Release-Notes: Breaking: Remote protocol v17 adds required replay_incarnation_id and optional turn_id fields to session observation event envelopes; remote hosts must upgrade their exact-version contract.
Carry turn identity on product completion events and transient browser rows so late completion for one turn cannot retract a newer turn's output. Cover the production reducer race and document the scoped settlement behavior.

Release-Notes: Changed: Agent Workbench terminal settlement now retracts only transient rows from the completed turn, preserving newer in-flight turn output.
@SamGalanakis
SamGalanakis merged commit b826be9 into main Jul 25, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant