fix: make vector reconciliation sequence-monotonic

[rita-aga]

Summary

• make vector-index reconciliation sequence-monotonic with an exact SHA-256 declaration fingerprint
• co-commit journal and vector rows atomically across Sim, Turso, and Postgres stores
• persist replacement generations, empty-set fences, tombstones, and declaration authority so stale or restarted rebuilds cannot resurrect superseded rows
• make catalog replacement scoped and transactional, and make actor publication race-free during hot replacement

Root cause

Backfill rows were written without the journal sequence and declaration authority they represented. A rebuild that read sequence N could therefore overwrite a live vector co-committed at sequence N+1, then persist a completion watermark that made the stale result durable. Replacement also lacked durable omission tombstones, so restart could resurrect removed declarations.

Design

ADR-0181: Monotonic vector reconciliation

ADR-0181 was selected after auditing origin/main and every open PR; ADR-0180 is already claimed by ARN-242.

RED → GREEN

• standalone RED commit: 541f58b6 (test: reproduce stale vector reconciliation)
• root-cause implementation: bb50c0e6 (fix: make vector reconciliation durably monotonic)
• behavioral coverage includes stale/live-write races, exact-fingerprint replacement, empty-set fences, omission across restart, removal/re-add, malformed/truncated replay, injected atomic-write faults, hot actor replacement, and real Turso/Postgres persistence paths

Validation on bb50c0e6dbd32ded74625be159c09d9321de5674

• cargo fmt --all -- --check: PASS
• git diff --check HEAD: PASS
• strict clippy for all touched crates/all targets/all features: PASS
• CARGO_BUILD_JOBS=2 cargo test --workspace: PASS, including randomized simulation, all 656 temper-server tests, integration tests, and doc tests
• fresh code-quality review: PASS
• fresh DST review: PASS; code-path unity CLEAN
• live Turso server restart/hot-replacement E2E: PASS (exact evidence is posted in the PR timeline)

Scope note

No source under crates/temper-actor-runtime is changed.

Gate status

Independent GitHub-head review, Greptile review, final independent review, and CI are still required before ARENA SHIPPABLE. Do not merge.

Linear: https://linear.app/arni-build/issue/ARN-216/bug-vector-backfill-races-live-writes-and-can-permanently-mark-a-stale-index

Greptile Summary

This PR makes vector reconciliation sequence-monotonic and coordinates specification publication across runtime and storage layers.

• Adds declaration fingerprints, replacement generations, per-entity sequence fences, omission tombstones, and completion watermarks.
• Co-commits journal and vector-index updates atomically in Postgres, Turso, and Sim, including composite writes.
• Makes verified catalog replacement transactional and prevents actor publication races during hot replacement.
• Adds migrations and extensive race, restart, fault-injection, and backend-parity coverage.

Confidence Score: 5/5

The PR appears safe to merge because no concrete changed-code defect remained after checking reconciliation ordering, backend parity, atomic append behavior, and actor publication races.

Live writes and rebuilds consistently validate declaration authority and compare durable generations and journal sequences, while failed or stale work cannot publish completion; catalog replacement also updates existing actor table references in place and revalidates actor incarnations around publication.

Important Files Changed

Filename	Overview
crates/temper-server/src/state/projection_backfill/vector_index.rs	Reworks vector backfill into generation- and sequence-fenced reconciliation over all durable streams, including deleted entities and removed declarations.
crates/temper-runtime/src/persistence/mod.rs	Extends the event-store contract with declaration authority, reconciliation generations, retained sequence fences, and atomic vector metadata.
crates/temper-server/src/state/dispatch/composite.rs	Supplies post-transition vector rows and declaration fingerprints to atomic multi-journal appends.
crates/temper-store-postgres/src/store.rs	Implements transactional declaration validation and generation/sequence-monotonic vector replacement for live and backfill writes.
crates/temper-store-postgres/migrations/0013_monotonic_vector_reconciliation.sql	Adds durable reconciliation generations, entity sequence fences, declaration authority, tombstones, and authority-advancing triggers.
crates/temper-store-turso/src/store/event_store.rs	Mirrors atomic journal/vector co-commit and monotonic reconciliation semantics in Turso.
crates/temper-store-sim/src/lib.rs	Mirrors production reconciliation semantics and fault behavior in the deterministic simulation backend.
crates/temper-server/src/observe/specs/load_dir.rs	Serializes verified catalog publication with actor eviction, in-place table swaps, and post-publication incarnation revalidation.
crates/temper-runtime/src/actor/cell.rs	Adds actor readiness and incarnation tracking that clears on restart, cancellation, panic, and shutdown.
crates/temper-store-postgres/src/spec_catalog.rs	Publishes exact verified staging sets and replacement omissions atomically under a tenant-scoped catalog lock.

Sequence Diagram

sequenceDiagram
    participant Registry
    participant Coordinator as Reconciliation coordinator
    participant Store as Postgres/Turso/Sim
    participant Writer as Live entity writer
    Registry->>Store: Persist declaration fingerprint/revision
    Coordinator->>Store: Begin reconciliation
    Store-->>Coordinator: Current generation
    Coordinator->>Store: Read journal/snapshot at sequence N
    Writer->>Store: Atomically append event N+1 and vector rows
    Store->>Store: Advance entity sequence fence
    Coordinator->>Store: Replace rows at generation G, sequence N
    Store-->>Coordinator: Ignore stale sequence or reject stale generation
    Coordinator->>Store: Publish completion watermark
    Store->>Store: Verify generation and declaration remain current

Loading

_{Reviews (1): Last reviewed commit: "test: satisfy strict inline stream lint" | Re-trigger Greptile}

Context used (4)

• Knowledge Base — temper-server
• Knowledge Base — Temper Runtime and Actor Runtime
• Knowledge Base — Temper storage backends
• Knowledge Base — temper-cli

[rita-aga]

Live local E2E evidence — exact head bb50c0e6dbd32ded74625be159c09d9321de5674

No external telemetry was enabled. The database and HOME artifacts are retained at /tmp/temper-arn216-e2e-bb50c0e6.db and /tmp/temper-arn216-e2e-home-bb50c0e6; nothing was cleaned.

RED behavioral baseline (standalone commit 541f58b6)

Command:

$ cargo test -p temper-store-sim stale_vector_backfill_does_not_overwrite_newer_live_write -- --nocapture

Recorded failure:

test tests::stale_vector_backfill_does_not_overwrite_newer_live_write ... FAILED

assertion `left == right` failed: a stale rebuild observed at sequence 1 must not overwrite the vector co-committed at sequence 2
  left: [EntityVectorCandidate { entity_id: "item-race", vector: [1.0, 0.0] }]
 right: [EntityVectorCandidate { entity_id: "item-race", vector: [0.0, 1.0] }]

test result: FAILED. 0 passed; 1 failed

That is the durability regression: the stale sequence-1 backfill overwrote the sequence-2 live vector.

Build

$ env CARGO_BUILD_JOBS=2 cargo build -p temper-cli --bin temper
   Compiling temper-platform v0.1.0 (.../crates/temper-platform)
   Compiling temper-cli v0.1.0 (.../crates/temper-cli)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 52.12s

Start with v1 catalog and durable Turso store

$ env LOGFIRE_TOKEN= OTLP_ENDPOINT= OTEL_EXPORTER_OTLP_ENDPOINT= OTEL_EXPORTER_OTLP_HEADERS= OTEL_EXPORTER_OTLP_TRACES_ENDPOINT= OTEL_EXPORTER_OTLP_TRACES_HEADERS= DD_API_KEY= DD_LLMOBS_API_ENABLED=false DISCORD_BOT_TOKEN= TEMPER_PROFILING_ENABLED=false TEMPER_PROFILING_AUTO_UPLOAD=false OTEL_SDK_DISABLED=true TURSO_URL=file:/tmp/temper-arn216-e2e-bb50c0e6.db HOME=/tmp/temper-arn216-e2e-home-bb50c0e6 SANDBOX_URL=http://127.0.0.1:65534 target/debug/temper serve --port 3216 --storage turso --no-observe --app arn216=/Users/seshendranalla/Development/temper/agent-worktrees/arn165/codex-arn-216-vector-backfill-race/crates/temper-server/tests/fixtures/arn216/full_v1
OTEL export disabled: no endpoint configured.
Loaded 2 entity spec(s) for tenant 'arn216'
Temper server listening on http://127.0.0.1:3216

Create the durable entity:

$ curl -sS -i -X POST http://127.0.0.1:3216/tdata/Notes -H 'Content-Type: application/json' -H 'X-Tenant-Id: arn216' -H 'X-Temper-Principal-Id: operator' -H 'X-Temper-Principal-Kind: admin' --data-binary '{"Id":"hot","Body":""}'
HTTP/1.1 201 Created
...
{"entity_type":"Note","entity_id":"hot","status":"Draft",...,"sequence_nr":1,...}

Hot-replace with the exact v2 fingerprint:

$ curl -sS -i -X POST http://127.0.0.1:3216/api/specs/load-dir -H 'Content-Type: application/json' -H 'X-Tenant-Id: arn216' -H 'X-Temper-Principal-Id: operator' -H 'X-Temper-Principal-Kind: admin' --data-binary '{"tenant":"arn216","specs_dir":"/Users/seshendranalla/Development/temper/agent-worktrees/arn165/codex-arn-216-vector-backfill-race/crates/temper-server/tests/fixtures/arn216/full_v2","merge":false}'
HTTP/1.1 200 OK
content-type: application/x-ndjson

{"type":"specs_loaded","tenant":"arn216","entities":["Item","Note"]}
{"type":"verification_result","entity":"Item","all_passed":true,...}
{"type":"verification_result","entity":"Note","all_passed":true,...}
{"type":"summary","tenant":"arn216","all_passed":true,"entities":{"Item":true,"Note":true}}

Drive the already-live actor through the v2 transition:

$ curl -sS -i -X POST -H 'Content-Type: application/json' -H 'X-Tenant-Id: arn216' -H 'X-Temper-Principal-Id: operator' -H 'X-Temper-Principal-Kind: admin' --data-binary '{"Body":"survives hot swap"}' "http://127.0.0.1:3216/tdata/Notes('hot')/Temper.Arn216.Review"
HTTP/1.1 200 OK
...
{"entity_type":"Note","entity_id":"hot","status":"Reviewed",...,"fields":{"Id":"hot","Body":"survives hot swap","Status":"Reviewed"},...,"sequence_nr":2,...}

Replace the catalog with Item only and verify omission:

$ curl -sS -i -X POST http://127.0.0.1:3216/api/specs/load-dir -H 'Content-Type: application/json' -H 'X-Tenant-Id: arn216' -H 'X-Temper-Principal-Id: operator' -H 'X-Temper-Principal-Kind: admin' --data-binary '{"tenant":"arn216","specs_dir":"/Users/seshendranalla/Development/temper/agent-worktrees/arn165/codex-arn-216-vector-backfill-race/crates/temper-server/tests/fixtures/arn216/item_only","merge":false}'
HTTP/1.1 200 OK
content-type: application/x-ndjson

{"type":"specs_loaded","tenant":"arn216","entities":["Item"]}
{"type":"verification_result","entity":"Item","all_passed":true,...}
{"type":"summary","tenant":"arn216","all_passed":true,"entities":{"Item":true}}

$ curl -sS -i -H 'X-Tenant-Id: arn216' -H 'X-Temper-Principal-Id: operator' -H 'X-Temper-Principal-Kind: admin' http://127.0.0.1:3216/tdata/Notes
HTTP/1.1 404 Not Found
content-type: application/json;odata.metadata=minimal
...
{"error":{"code":"EntitySetNotFound","message":"Entity set 'Notes' not found"}}

The first server was stopped with SIGINT.

Restart with no app/catalog argument

$ env LOGFIRE_TOKEN= OTLP_ENDPOINT= OTEL_EXPORTER_OTLP_ENDPOINT= OTEL_EXPORTER_OTLP_HEADERS= OTEL_EXPORTER_OTLP_TRACES_ENDPOINT= OTEL_EXPORTER_OTLP_TRACES_HEADERS= DD_API_KEY= DD_LLMOBS_API_ENABLED=false DISCORD_BOT_TOKEN= TEMPER_PROFILING_ENABLED=false TEMPER_PROFILING_AUTO_UPLOAD=false OTEL_SDK_DISABLED=true TURSO_URL=file:/tmp/temper-arn216-e2e-bb50c0e6.db HOME=/tmp/temper-arn216-e2e-home-bb50c0e6 SANDBOX_URL=http://127.0.0.1:65534 target/debug/temper serve --port 3216 --storage turso --no-observe
OTEL export disabled: no endpoint configured.
Temper server listening on http://127.0.0.1:3216

Durable omission survives restart:

$ curl -sS -i -H 'X-Tenant-Id: arn216' -H 'X-Temper-Principal-Id: operator' -H 'X-Temper-Principal-Kind: admin' http://127.0.0.1:3216/tdata/Notes
HTTP/1.1 404 Not Found
content-type: application/json;odata.metadata=minimal
...
{"error":{"code":"EntitySetNotFound","message":"Entity set 'Notes' not found"}}

Re-add v2 (full verification output abbreviated only where shown by ...):

$ curl -sS -i -X POST http://127.0.0.1:3216/api/specs/load-dir -H 'Content-Type: application/json' -H 'X-Tenant-Id: arn216' -H 'X-Temper-Principal-Id: operator' -H 'X-Temper-Principal-Kind: admin' --data-binary '{"tenant":"arn216","specs_dir":"/Users/seshendranalla/Development/temper/agent-worktrees/arn165/codex-arn-216-vector-backfill-race/crates/temper-server/tests/fixtures/arn216/full_v2","merge":false}'
HTTP/1.1 200 OK
content-type: application/x-ndjson

{"type":"specs_loaded","tenant":"arn216","entities":["Item","Note"]}
{"type":"verification_result","entity":"Item","all_passed":true,"levels":[{"level":"Level 0: Symbolic Verification","passed":true,...},{"level":"Level 1: Model Check","passed":true,...},{"level":"Level 2: Deterministic Simulation","passed":true,...},{"level":"Level 3: Property Tests","passed":true,...}]}
{"type":"verification_result","entity":"Note","all_passed":true,"levels":[{"level":"Level 0: Symbolic Verification","passed":true,...},{"level":"Level 1: Model Check","passed":true,...},{"level":"Level 2: Deterministic Simulation","passed":true,...},{"level":"Level 3: Property Tests","passed":true,...}]}
{"type":"summary","tenant":"arn216","all_passed":true,"entities":{"Item":true,"Note":true}}

Recover the prior actor from its retained journal:

$ curl -sS -i -H 'X-Tenant-Id: arn216' -H 'X-Temper-Principal-Id: operator' -H 'X-Temper-Principal-Kind: admin' "http://127.0.0.1:3216/tdata/Notes('hot')"
HTTP/1.1 200 OK
content-type: application/json;odata.metadata=minimal
...
{"entity_type":"Note","entity_id":"hot","status":"Reviewed",...,"fields":{"Id":"hot","Body":"survives hot swap","Status":"Reviewed"},"events":[],"total_event_count":2,"events_since_snapshot":2,"last_snapshot_sequence_nr":0,"sequence_nr":2,...,"@odata.actions":[{"name":"Publish","target":"Notes('hot')/Temper.Publish","hint":null}],...}

Advance the recovered entity under v2:

$ curl -sS -i -X POST -H 'Content-Type: application/json' -H 'X-Tenant-Id: arn216' -H 'X-Temper-Principal-Id: operator' -H 'X-Temper-Principal-Kind: admin' --data-binary '{}' "http://127.0.0.1:3216/tdata/Notes('hot')/Temper.Publish"
HTTP/1.1 200 OK
content-type: application/json;odata.metadata=minimal
...
{"entity_type":"Note","entity_id":"hot","status":"Published",...,"fields":{"Id":"hot","Status":"Published","Body":"survives hot swap"},"events":[{"action":"Created","from_status":"","to_status":"Draft",...},{"action":"Review","from_status":"Draft","to_status":"Reviewed",...},{"action":"Publish","from_status":"Reviewed","to_status":"Published",...}],"total_event_count":3,"events_since_snapshot":3,"last_snapshot_sequence_nr":0,"sequence_nr":3,...}

The restarted server was stopped with SIGINT; pgrep -fl 'target/debug/temper serve --port 3216' returned no process.

Result: PASS. Exact-fingerprint hot replacement works, omission is durable across a process restart, re-add restores the retained sequence-2 journal, and the recovered actor accepts the current v2 action at sequence 3.

[rita-aga]

Independent GPT-5.6 pre-Greptile review

Reviewed the complete open PR diff (76 changed files) at exact head bb50c0e6dbd32ded74625be159c09d9321de5674. I rechecked immediately before posting that PR #388 is still open, unmerged, and at this head.

Findings

1. [P1] Staged, uncommitted specs are promoted to declaration authority before verification/commit. The PostgreSQL migration seeds every row in specs without filtering committed (migration lines 93–100), and its insert/update/delete triggers likewise ignore committed (lines 210–232). Turso has the same behavior in its seed and triggers (lines 20–27, lines 53–124). That conflicts with the existing two-phase catalog contract: upsert_spec deliberately writes changed specs as committed=false before verification (Postgres lines 1044–1067; Turso lines 27–78). Staging an update to a live type therefore immediately changes the durable fingerprint and fences the still-published table, so normal actor writes fail with a stale fingerprint while verification is running. If verification fails, delete_uncommitted_specs fires the delete trigger and leaves a retained absence tombstone for the still-live committed declaration. Also, simply gating the current triggers is insufficient: commit_specs changes only committed, while the update triggers watch only IOA/hash, so authority must advance on the false -> true commit transition and ignore/clean up uncommitted staging mutations. No added test exercises this existing staged-spec workflow.

2. [P1] A PostgreSQL app restart tombstones the built-in agent specs and Phase 8 never restores their authority. Disk app startup now calls persist_spec_catalog_update(..., replace=true) with only the directory's IOA sources (storage.rs lines 70–77). PostgreSQL replacement intentionally includes compatibility authority rows that have no catalog row (store.rs lines 53–67), so any built-in Agent/Task/etc. type whose first live write established compatibility authority is treated as an omission and tombstoned on the next boot. Phase 8 then merges those built-ins back into the in-memory registry, but persists them only through a Turso accessor (bootstrap.rs lines 455–462); the PostgreSQL StorageStack is not updated. The server therefore advertises the built-in entities after restart while every fingerprinted PostgreSQL write to them is rejected against absent:v1, and the retained tombstone prevents compatibility bootstrap from recovering. The startup replacement must include the built-in catalog or Phase 8 must publish the same declarations through the active PlatformStore/Postgres backend. No added restart fixture covers the PostgreSQL app-plus-built-ins path.

3. [P1] Supervised actor restarts reuse the snapshotted UID, so a publication-gap restart can be incorrectly preserved. Publication snapshots only actor.id().uid (entity_ops.rs lines 824–843) and preserves the actor when that UID is unchanged and readiness has returned true (lines 846–872). But the supervision loop runs pre_start repeatedly with the same ActorId/UID and merely toggles the boolean readiness flag (cell.rs lines 94–107, lines 159–181). Thus an actor snapshotted before durability can restart during the durability/publication gap, complete pre_start against the old table, become ready again with the same UID, and survive the eviction immediately before registry publication (load_dir.rs lines 225–243, lines 258–297). Its state was initialized/replayed under the superseded declaration even though its live table is then swapped. The identity needs a monotonically changing readiness/pre-start epoch (or a new UID per supervised incarnation), and the snapshot/recheck must include it. The new test covers unready actors and same-key replacements with a different UID, but not a supervised restart that retains the UID.

Verdict: FAIL

[rita-aga]

Verdict: PASS

Fresh independent GPT-5.6 review of the complete OPEN PR diff against the current base found zero actionable findings at exact head 07f2689f82a434544deeb2feba4286cf05890449.

Reviewed publication authority and atomicity, PostgreSQL/Turso parity, migration and cold-start/restart paths, vector backfill/live-write fencing and deletion reconciliation, determinism, and the full changed test surface. Independent validation included the affected library suites, targeted server integrations, formatting, strict Clippy, and diff hygiene.

[rita-aga]

@greptile review

[rita-aga]

Verdict: PASS

Fresh final independent GPT-5.6 review of the complete OPEN PR diff after Greptile found zero actionable findings at exact head 07f2689f82a434544deeb2feba4286cf05890449.

The final audit covered atomicity, PostgreSQL/Turso/Sim parity, verified-spec publication boundaries, migrations and restart/bootstrap paths, vector generation/sequence fencing and deletion reconciliation, actor incarnation handoff, determinism, and the changed test surface. Greptile Review is SUCCESS with zero review threads.

[rita-aga]

ARENA SHIPPABLE

ARN-216 is complete on OPEN, unmerged PR #388 at exact head 07f2689f82a434544deeb2feba4286cf05890449. The published history includes required commits 3a879bf7 and b1504f30; the branch is clean and synchronized.

Evidence:

• Unbypassed local pre-push pipeline: rustfmt, strict Clippy, readability ratchet, complete workspace tests/DST/doctests — PASS.
• Focused race/publication regressions and live PostgreSQL/Turso E2E coverage — PASS. Supplementary manual restart/hot-replacement evidence: fix: make vector reconciliation sequence-monotonic #388 (comment)
• Fresh pre-Greptile complete OPEN-PR-diff GPT-5.6 PASS: fix: make vector reconciliation sequence-monotonic #388 (comment)
• Greptile requested only after that PASS: fix: make vector reconciliation sequence-monotonic #388 (comment)
• Greptile Review — SUCCESS on the exact head, with zero review threads/actionable findings.
• Fresh final post-Greptile complete OPEN-PR-diff GPT-5.6 PASS: fix: make vector reconciliation sequence-monotonic #388 (comment)
• Current-head CI — all required jobs green: https://github.com/nerdsane/temper/actions/runs/30170875708

crates/temper-actor-runtime was not changed, so no outside-contributor sign-off flag applies. MERGE NOTHING.