From 6a72d09465b338f76e846c619439364931cfb521 Mon Sep 17 00:00:00 2001 From: macanderson Date: Sun, 26 Jul 2026 10:48:33 -0700 Subject: [PATCH] docs: the export ledger's three watermarks, and a page for `stella doctor` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #631. Closes #690. #631 — `enterprise_telemetry/export_ledger.rs` (the path in the issue is stale; the module moved) documented its lifecycle but never stated the three-watermark contract as such. Adds a `# The three watermarks` section naming each floor, what advances it, and what breaks if it moves early or late. The audit declined this on the grounds that the WHY can only be inferred. Nothing here is inferred: every claim is read off the SQL that enforces it. `enrolled_after_execution_id` and `compacted_through_execution_id` are the two `execution_id >` floors in `mark_enterprise_export_pending`'s eligibility check; `after_execution_id` is the keyset cursor in `pending_enterprise_export_page`. Why there are three falls out of their failure modes rather than out of anyone's intent: the two durable floors fail permanently and silently (a silent gap, a back-export, or a double-count once the row holding the nonce is reclaimed), while the cursor's failures are absorbed in both directions by nonce stability plus the spool's dedup — which is exactly why persisting it would be a downgrade, not a safeguard. Two consequences worth having in writing, both derived and neither previously stated: compaction carries a precondition (an execution still unfinished when a cutoff passes its id is refused forever), and the compaction floor gates re-admission only — pending rows survive compaction and still drain. #690 — `stella doctor` shipped with no docs-site page, so `--repair`, which rewrites a user's `store.db`, was discoverable only via `--help`. Adds `commands/doctor.mdx` and registers it in `index.mdx` (23 → 24 rows) and `meta.json`. Covers the single `store integrity` check and its binary verdict, the exit-code contract, and precisely what `--repair` does on disk: that it acts only on a corruption verdict and leaves an inconclusive one untouched, that the db and both sidecars are renamed (never deleted) under one timestamp, that the salvage is a separate copy, and that uncheckpointed WAL pages do not reach it. Verified: `RUSTDOCFLAGS="-D warnings" cargo doc -p stella-store` clean, clippy/fmt/file-size/doc-citations/no-scratch pass, `next build` renders /docs/commands/doctor with the index row and sidebar links wired. --- .../src/enterprise_telemetry/export_ledger.rs | 79 +++++++++++ website/content/docs/commands/doctor.mdx | 125 ++++++++++++++++++ website/content/docs/commands/index.mdx | 1 + website/content/docs/commands/meta.json | 2 +- 4 files changed, 206 insertions(+), 1 deletion(-) create mode 100644 website/content/docs/commands/doctor.mdx diff --git a/stella-store/src/enterprise_telemetry/export_ledger.rs b/stella-store/src/enterprise_telemetry/export_ledger.rs index ffe444bc..6fb85d87 100644 --- a/stella-store/src/enterprise_telemetry/export_ledger.rs +++ b/stella-store/src/enterprise_telemetry/export_ledger.rs @@ -34,6 +34,85 @@ //! //! Every mutation runs in an `IMMEDIATE` transaction: two processes sharing one //! workspace must not interleave "is it eligible?" with "mint its nonce". +//! +//! # The three watermarks +//! +//! Three `execution_id >` floors decide what this ledger admits and what it +//! hands out, and between them they are the whole of its correctness. They are +//! not three copies of one idea: each answers a different question, is advanced +//! by a different caller, and fails in a different direction. Two must be +//! durable; the third must deliberately not be. +//! +//! | Watermark | Question it answers | Lives | Advanced by | +//! | --- | --- | --- | --- | +//! | `enrolled_after_execution_id` | may this sink ever see it? | durable, per sink | [`Store::begin_enterprise_enrollment`](crate::Store::begin_enterprise_enrollment), once | +//! | `compacted_through_execution_id` | is it already settled and reclaimed? | durable, per sink | [`Store::compact_enterprise_export_ledger`](crate::Store::compact_enterprise_export_ledger), monotonically | +//! | `after_execution_id` | where did this drain get to? | transient, per pass | the caller, between pages | +//! +//! ## 1. `enrolled_after_execution_id` — consent +//! +//! `MAX(executions.id)` as it stood when the sink was first enrolled. +//! Everything at or below it predates the operator's decision to export and +//! must never leave the workspace. Stamped once by `INSERT OR IGNORE` and never +//! rewritten. +//! +//! Advance it **early** — a re-enrollment that re-stamped it with today's +//! `MAX(id)` — and every execution recorded since the original enrollment is +//! dropped without a trace: those rows never enter the ledger, so no skip +//! counter records them and no backlog shows them missing. Stamp it **low** — +//! `0` on a workspace that already has history — and the first drain +//! back-exports everything recorded before the operator opted in. Both +//! directions are unrecoverable: one loses data silently, the other discloses +//! it. (Dropping the enrollment row entirely is a third thing again, and not a +//! low watermark: the eligibility check joins against that row, so a sink with +//! no enrollment exports *nothing* rather than everything.) +//! +//! ## 2. `compacted_through_execution_id` — memory of deleted rows +//! +//! Every id at or below it has already settled *and* had its ledger row +//! reclaimed. It exists because the row that would otherwise prove "this was +//! exported" is gone, so the scalar has to remember in its place. Compaction is +//! its only writer, and writes it as `MAX(existing, cutoff)` so it can only +//! rise. +//! +//! Leave it **late** — reclaim rows without raising it — and the reclaimed ids +//! look un-exported to +//! [`Store::mark_enterprise_export_pending`](crate::Store::mark_enterprise_export_pending), +//! which re-admits them and mints *fresh* nonces. Nonce idempotence cannot save +//! this: the row that stored the original nonce is exactly what was deleted, so +//! one execution becomes two distinct event ids and is double-counted +//! downstream. Advance it **early** — past an execution that had not finished +//! when the cutoff was chosen — and that execution is refused forever when it +//! does finish, again with no counter. So compaction carries a precondition: +//! run it only when the backlog below the cutoff is settled. The cutoff is +//! drawn from settled rows alone, which makes an early advance narrow but not +//! impossible — an execution can finish after ids above it already have. +//! +//! Note what this watermark does *not* gate: pending rows are never deleted by +//! compaction, and +//! [`Store::pending_enterprise_export_page`](crate::Store::pending_enterprise_export_page) +//! does not consult it. An execution already admitted still drains normally. +//! The floor governs re-admission only. +//! +//! ## 3. `after_execution_id` — a position, not a promise +//! +//! The keyset cursor within one drain pass, supplied by the caller and stored +//! nowhere. A pass that ends mid-backlog simply starts again from the beginning +//! next time. +//! +//! Advance it **early**, skipping rows, and nothing is lost — the skipped rows +//! are still `pending`, and the next pass lists them again. Leave it **late**, +//! or reset it to `0`, and the same rows are re-listed and re-spooled — also +//! harmless, because their nonces are stable, so the event ids are byte-for-byte +//! the ones already enqueued and the spool dedups them. Both directions are +//! absorbed. +//! +//! That is precisely why it must stay transient. Persisting it would buy +//! nothing (a re-read is already free) and would convert its harmless failure +//! into watermark 2's: a durable cursor that advanced past undrained rows would +//! hide them permanently, with no counter to show for it. The two floors are +//! durable because their failures are permanent and silent; the cursor is not, +//! because its safety comes from being re-derivable. use rusqlite::{OptionalExtension, TransactionBehavior, params}; diff --git a/website/content/docs/commands/doctor.mdx b/website/content/docs/commands/doctor.mdx new file mode 100644 index 00000000..ad01500e --- /dev/null +++ b/website/content/docs/commands/doctor.mdx @@ -0,0 +1,125 @@ +--- +title: stella doctor +description: Check the local state stella owns, report each named check's verdict, and exit non-zero if any fails — with an opt-in repair that moves a corrupt session store aside without ever deleting it. +--- + +Answer "is my local state sound?" before you go looking for a bug that isn't yours. `stella doctor` runs each check it knows how to run, prints a verdict per check, and exits non-zero if any failed — so it can gate a script. It reads local state only: no provider, no API key, no network. + +## Synopsis + +```bash +stella doctor # diagnose, change nothing +stella doctor --repair # also quarantine a corrupt store +``` + +## What it does + +Today it runs exactly one check — **`store integrity`**, the soundness of this workspace's session store at `/.stella/private/store.db`, verified with SQLite's own `quick_check` and `integrity_check` pragmas. The list is built to hold more, but one check is what currently ships; the command is worth what it actually verifies. + +A verdict is deliberately binary. There is no "warn": a check that cannot say *this is fine* has failed, and a third maybe-state would only be an invitation to ignore it. + +| Verdict | When | +| --- | --- | +| ✓ pass | The store passed, **or** there is no store yet — `store.db` is created by your first session, and its absence is not a fault. | +| ✗ fail | SQLite reported damage, **or** the check could not be performed at all (a locked file, a permissions problem). | + +Diagnosis never writes. The check opens the database immutably — no locks, no `-shm`, zero bytes written — and escalates to a read-write open only when a first verdict already looks bad and a `-wal` sibling exists. Being asked a question is not a reason to create a `.stella/` that wasn't there. + +`stella doctor` runs *before* stella loads your configuration, on purpose: a workspace whose store is corrupt has to stay diagnosable without a working model config. One consequence is that `--output-format` does not apply to it — there is no JSON form of this output. + +## Flags + +| Flag | Meaning | +| --- | --- | +| `--repair` | Act on a store SQLite judged corrupt: move it aside to a timestamped name (renamed, **never** deleted) and copy out whatever is still readable. A healthy store and an inconclusive check are both left untouched. | + +## Exit codes + +`0` when every check passed — including a `--repair` that repaired. `1` when any check failed, with `stella: 1 of 1 doctor check failed` on stderr. That is the whole contract, which is what makes the command usable as a gate: + +```bash +stella doctor || echo "local state needs attention" +``` + +## What `--repair` does to the file on disk + +This is the one part of `stella doctor` that touches your disk, so it is worth stating exactly. + +It runs **only** on a verdict of genuine corruption — a database SQLite could read but found structurally damaged, or one it could not read as a database at all. A check that merely failed to *complete* is inconclusive, and an inconclusive diagnosis leaves the store exactly where it is; moving a database on a guess is how a repair tool loses somebody's data. On a healthy store, `--repair` reports that it had nothing to do. + +When it does act, in this order: + +1. **The database is renamed**, along with both sidecars if present, all sharing one timestamp (seconds since the Unix epoch): + + ```text + .stella/private/store.db → .stella/private/store.db.corrupt-1769472000 + .stella/private/store.db-wal → .stella/private/store.db-wal.corrupt-1769472000 + .stella/private/store.db-shm → .stella/private/store.db-shm.corrupt-1769472000 + ``` + + The sidecars must travel with the database. A stale WAL left beside the fresh store your next session creates is how a successful repair turns into new corruption. Renames come first because once they land the workspace is usable again even if everything after them fails. + +2. **Whatever is still readable is copied out** to a separate database beside the original — `store.db.salvaged-1769472000` — restricted to your user. If nothing can be salvaged, that is reported and the quarantine still stands; a failed salvage is never fatal. + +3. **Your next session starts a fresh store.** Nothing was deleted. + + +Uncheckpointed pages in the quarantined `-wal` do not reach the salvaged copy — SQLite only replays a WAL sitting at its database's exact name. If you need them, rename the quarantined pair back to `store.db` and `store.db-wal` and let SQLite's own recovery run. + + +Every quarantine is undoable with `mv`, and a repeat repair within the same second gets a `.2`, `.3`, … suffix rather than overwriting an earlier one — a quarantine that clobbered a quarantine would destroy exactly the bytes the whole path exists to preserve. + +The store holds local telemetry and session replay only. It never holds your source. + +## Examples + +A healthy workspace: + +```text +─── Doctor — local state checks + ✓ store integrity — .stella/private/store.db: ok (quick_check) + + 1 check: 1 ok, 0 failed +``` + +`quick_check` is the fast path; `integrity_check` runs only when `quick_check` flags something, and the headline names whichever produced the verdict. + +A corrupt store, diagnosed but untouched (exit `1`): + +```text +─── Doctor — local state checks + ✗ store integrity — .stella/private/store.db: not readable as a SQLite database + file is not a database + → `stella doctor --repair` moves it aside (renamed, never deleted) and copies out whatever is still readable + → by hand: sqlite3 .stella/private/store.db ".recover" | sqlite3 .stella/private/store.db.recovered + → the store holds local telemetry and session replay only — never your source + + 1 check: 0 ok, 1 failed +``` + +At most five problem rows are printed, followed by `… and N more`; the store itself records up to twenty, and the total is never truncated. + +The same workspace after `stella doctor --repair` (exit `0`): + +```text +─── Doctor — local state checks + ✓ store integrity — .stella/private/store.db: quarantined + was: not readable as a SQLite database + moved .stella/private/store.db → .stella/private/store.db.corrupt-1769472000 + moved .stella/private/store.db-wal → .stella/private/store.db-wal.corrupt-1769472000 + salvaged what was readable → .stella/private/store.db.salvaged-1769472000 + nothing was deleted; the next session starts a fresh store + + 1 check: 1 ok, 0 failed +``` + +A check that could not be performed — reported, and the store is not moved even with `--repair`: + +```text + ✗ store integrity — could not be checked: + → the store was NOT touched — resolve the error above and re-run `stella doctor` +``` + + +You will usually meet this command by being sent here: when a session cannot open `store.db`, the error names `stella doctor` directly. Running it without `--repair` first is always safe — it is a read-only diagnosis. + diff --git a/website/content/docs/commands/index.mdx b/website/content/docs/commands/index.mdx index b23c9e80..25d49984 100644 --- a/website/content/docs/commands/index.mdx +++ b/website/content/docs/commands/index.mdx @@ -39,6 +39,7 @@ stella chat | `stella auth ` | Manage BYOK provider keys in [`credentials.toml`](/docs/commands/auth) (never prints a secret). | | `stella tools` | List every tool available to the agent this session. | | `stella config` | Show the fully resolved configuration. | +| `stella doctor` | Check the local state stella owns — today the integrity of this workspace's [session store](/docs/commands/doctor) — and exit non-zero if any check fails. `--repair` moves a corrupt `store.db` aside (renamed, never deleted). | | `stella version` | Print the version and exit. | Each command page in this section documents its usage synopsis, behavior, relevant flags, and realistic examples. diff --git a/website/content/docs/commands/meta.json b/website/content/docs/commands/meta.json index 9cd17f36..7ec3b445 100644 --- a/website/content/docs/commands/meta.json +++ b/website/content/docs/commands/meta.json @@ -1,4 +1,4 @@ { "title": "Commands", - "pages": ["index", "run", "chat", "resume", "goal", "monitor", "init", "fleet", "graph", "storage", "scripts", "models", "stats", "inspect", "observe", "memory", "mcp", "connect", "auth", "tools", "config", "version"] + "pages": ["index", "run", "chat", "resume", "goal", "monitor", "init", "fleet", "graph", "storage", "scripts", "models", "stats", "inspect", "observe", "memory", "mcp", "connect", "auth", "tools", "config", "doctor", "version"] }