diff --git a/stella-store/src/enterprise_telemetry/export_ledger.rs b/stella-store/src/enterprise_telemetry/export_ledger.rs index ffe444bc..6fb85d87 100644 --- a/stella-store/src/enterprise_telemetry/export_ledger.rs +++ b/stella-store/src/enterprise_telemetry/export_ledger.rs @@ -34,6 +34,85 @@ //! //! Every mutation runs in an `IMMEDIATE` transaction: two processes sharing one //! workspace must not interleave "is it eligible?" with "mint its nonce". +//! +//! # The three watermarks +//! +//! Three `execution_id >` floors decide what this ledger admits and what it +//! hands out, and between them they are the whole of its correctness. They are +//! not three copies of one idea: each answers a different question, is advanced +//! by a different caller, and fails in a different direction. Two must be +//! durable; the third must deliberately not be. +//! +//! | Watermark | Question it answers | Lives | Advanced by | +//! | --- | --- | --- | --- | +//! | `enrolled_after_execution_id` | may this sink ever see it? | durable, per sink | [`Store::begin_enterprise_enrollment`](crate::Store::begin_enterprise_enrollment), once | +//! | `compacted_through_execution_id` | is it already settled and reclaimed? | durable, per sink | [`Store::compact_enterprise_export_ledger`](crate::Store::compact_enterprise_export_ledger), monotonically | +//! | `after_execution_id` | where did this drain get to? | transient, per pass | the caller, between pages | +//! +//! ## 1. `enrolled_after_execution_id` — consent +//! +//! `MAX(executions.id)` as it stood when the sink was first enrolled. +//! Everything at or below it predates the operator's decision to export and +//! must never leave the workspace. Stamped once by `INSERT OR IGNORE` and never +//! rewritten. +//! +//! Advance it **early** — a re-enrollment that re-stamped it with today's +//! `MAX(id)` — and every execution recorded since the original enrollment is +//! dropped without a trace: those rows never enter the ledger, so no skip +//! counter records them and no backlog shows them missing. Stamp it **low** — +//! `0` on a workspace that already has history — and the first drain +//! back-exports everything recorded before the operator opted in. Both +//! directions are unrecoverable: one loses data silently, the other discloses +//! it. (Dropping the enrollment row entirely is a third thing again, and not a +//! low watermark: the eligibility check joins against that row, so a sink with +//! no enrollment exports *nothing* rather than everything.) +//! +//! ## 2. `compacted_through_execution_id` — memory of deleted rows +//! +//! Every id at or below it has already settled *and* had its ledger row +//! reclaimed. It exists because the row that would otherwise prove "this was +//! exported" is gone, so the scalar has to remember in its place. Compaction is +//! its only writer, and writes it as `MAX(existing, cutoff)` so it can only +//! rise. +//! +//! Leave it **late** — reclaim rows without raising it — and the reclaimed ids +//! look un-exported to +//! [`Store::mark_enterprise_export_pending`](crate::Store::mark_enterprise_export_pending), +//! which re-admits them and mints *fresh* nonces. Nonce idempotence cannot save +//! this: the row that stored the original nonce is exactly what was deleted, so +//! one execution becomes two distinct event ids and is double-counted +//! downstream. Advance it **early** — past an execution that had not finished +//! when the cutoff was chosen — and that execution is refused forever when it +//! does finish, again with no counter. So compaction carries a precondition: +//! run it only when the backlog below the cutoff is settled. The cutoff is +//! drawn from settled rows alone, which makes an early advance narrow but not +//! impossible — an execution can finish after ids above it already have. +//! +//! Note what this watermark does *not* gate: pending rows are never deleted by +//! compaction, and +//! [`Store::pending_enterprise_export_page`](crate::Store::pending_enterprise_export_page) +//! does not consult it. An execution already admitted still drains normally. +//! The floor governs re-admission only. +//! +//! ## 3. `after_execution_id` — a position, not a promise +//! +//! The keyset cursor within one drain pass, supplied by the caller and stored +//! nowhere. A pass that ends mid-backlog simply starts again from the beginning +//! next time. +//! +//! Advance it **early**, skipping rows, and nothing is lost — the skipped rows +//! are still `pending`, and the next pass lists them again. Leave it **late**, +//! or reset it to `0`, and the same rows are re-listed and re-spooled — also +//! harmless, because their nonces are stable, so the event ids are byte-for-byte +//! the ones already enqueued and the spool dedups them. Both directions are +//! absorbed. +//! +//! That is precisely why it must stay transient. Persisting it would buy +//! nothing (a re-read is already free) and would convert its harmless failure +//! into watermark 2's: a durable cursor that advanced past undrained rows would +//! hide them permanently, with no counter to show for it. The two floors are +//! durable because their failures are permanent and silent; the cursor is not, +//! because its safety comes from being re-derivable. use rusqlite::{OptionalExtension, TransactionBehavior, params}; diff --git a/website/content/docs/commands/doctor.mdx b/website/content/docs/commands/doctor.mdx new file mode 100644 index 00000000..ad01500e --- /dev/null +++ b/website/content/docs/commands/doctor.mdx @@ -0,0 +1,125 @@ +--- +title: stella doctor +description: Check the local state stella owns, report each named check's verdict, and exit non-zero if any fails — with an opt-in repair that moves a corrupt session store aside without ever deleting it. +--- + +Answer "is my local state sound?" before you go looking for a bug that isn't yours. `stella doctor` runs each check it knows how to run, prints a verdict per check, and exits non-zero if any failed — so it can gate a script. It reads local state only: no provider, no API key, no network. + +## Synopsis + +```bash +stella doctor # diagnose, change nothing +stella doctor --repair # also quarantine a corrupt store +``` + +## What it does + +Today it runs exactly one check — **`store integrity`**, the soundness of this workspace's session store at `/.stella/private/store.db`, verified with SQLite's own `quick_check` and `integrity_check` pragmas. The list is built to hold more, but one check is what currently ships; the command is worth what it actually verifies. + +A verdict is deliberately binary. There is no "warn": a check that cannot say *this is fine* has failed, and a third maybe-state would only be an invitation to ignore it. + +| Verdict | When | +| --- | --- | +| ✓ pass | The store passed, **or** there is no store yet — `store.db` is created by your first session, and its absence is not a fault. | +| ✗ fail | SQLite reported damage, **or** the check could not be performed at all (a locked file, a permissions problem). | + +Diagnosis never writes. The check opens the database immutably — no locks, no `-shm`, zero bytes written — and escalates to a read-write open only when a first verdict already looks bad and a `-wal` sibling exists. Being asked a question is not a reason to create a `.stella/` that wasn't there. + +`stella doctor` runs *before* stella loads your configuration, on purpose: a workspace whose store is corrupt has to stay diagnosable without a working model config. One consequence is that `--output-format` does not apply to it — there is no JSON form of this output. + +## Flags + +| Flag | Meaning | +| --- | --- | +| `--repair` | Act on a store SQLite judged corrupt: move it aside to a timestamped name (renamed, **never** deleted) and copy out whatever is still readable. A healthy store and an inconclusive check are both left untouched. | + +## Exit codes + +`0` when every check passed — including a `--repair` that repaired. `1` when any check failed, with `stella: 1 of 1 doctor check failed` on stderr. That is the whole contract, which is what makes the command usable as a gate: + +```bash +stella doctor || echo "local state needs attention" +``` + +## What `--repair` does to the file on disk + +This is the one part of `stella doctor` that touches your disk, so it is worth stating exactly. + +It runs **only** on a verdict of genuine corruption — a database SQLite could read but found structurally damaged, or one it could not read as a database at all. A check that merely failed to *complete* is inconclusive, and an inconclusive diagnosis leaves the store exactly where it is; moving a database on a guess is how a repair tool loses somebody's data. On a healthy store, `--repair` reports that it had nothing to do. + +When it does act, in this order: + +1. **The database is renamed**, along with both sidecars if present, all sharing one timestamp (seconds since the Unix epoch): + + ```text + .stella/private/store.db → .stella/private/store.db.corrupt-1769472000 + .stella/private/store.db-wal → .stella/private/store.db-wal.corrupt-1769472000 + .stella/private/store.db-shm → .stella/private/store.db-shm.corrupt-1769472000 + ``` + + The sidecars must travel with the database. A stale WAL left beside the fresh store your next session creates is how a successful repair turns into new corruption. Renames come first because once they land the workspace is usable again even if everything after them fails. + +2. **Whatever is still readable is copied out** to a separate database beside the original — `store.db.salvaged-1769472000` — restricted to your user. If nothing can be salvaged, that is reported and the quarantine still stands; a failed salvage is never fatal. + +3. **Your next session starts a fresh store.** Nothing was deleted. + + +Uncheckpointed pages in the quarantined `-wal` do not reach the salvaged copy — SQLite only replays a WAL sitting at its database's exact name. If you need them, rename the quarantined pair back to `store.db` and `store.db-wal` and let SQLite's own recovery run. + + +Every quarantine is undoable with `mv`, and a repeat repair within the same second gets a `.2`, `.3`, … suffix rather than overwriting an earlier one — a quarantine that clobbered a quarantine would destroy exactly the bytes the whole path exists to preserve. + +The store holds local telemetry and session replay only. It never holds your source. + +## Examples + +A healthy workspace: + +```text +─── Doctor — local state checks + ✓ store integrity — .stella/private/store.db: ok (quick_check) + + 1 check: 1 ok, 0 failed +``` + +`quick_check` is the fast path; `integrity_check` runs only when `quick_check` flags something, and the headline names whichever produced the verdict. + +A corrupt store, diagnosed but untouched (exit `1`): + +```text +─── Doctor — local state checks + ✗ store integrity — .stella/private/store.db: not readable as a SQLite database + file is not a database + → `stella doctor --repair` moves it aside (renamed, never deleted) and copies out whatever is still readable + → by hand: sqlite3 .stella/private/store.db ".recover" | sqlite3 .stella/private/store.db.recovered + → the store holds local telemetry and session replay only — never your source + + 1 check: 0 ok, 1 failed +``` + +At most five problem rows are printed, followed by `… and N more`; the store itself records up to twenty, and the total is never truncated. + +The same workspace after `stella doctor --repair` (exit `0`): + +```text +─── Doctor — local state checks + ✓ store integrity — .stella/private/store.db: quarantined + was: not readable as a SQLite database + moved .stella/private/store.db → .stella/private/store.db.corrupt-1769472000 + moved .stella/private/store.db-wal → .stella/private/store.db-wal.corrupt-1769472000 + salvaged what was readable → .stella/private/store.db.salvaged-1769472000 + nothing was deleted; the next session starts a fresh store + + 1 check: 1 ok, 0 failed +``` + +A check that could not be performed — reported, and the store is not moved even with `--repair`: + +```text + ✗ store integrity — could not be checked: + → the store was NOT touched — resolve the error above and re-run `stella doctor` +``` + + +You will usually meet this command by being sent here: when a session cannot open `store.db`, the error names `stella doctor` directly. Running it without `--repair` first is always safe — it is a read-only diagnosis. + diff --git a/website/content/docs/commands/index.mdx b/website/content/docs/commands/index.mdx index b23c9e80..25d49984 100644 --- a/website/content/docs/commands/index.mdx +++ b/website/content/docs/commands/index.mdx @@ -39,6 +39,7 @@ stella chat | `stella auth ` | Manage BYOK provider keys in [`credentials.toml`](/docs/commands/auth) (never prints a secret). | | `stella tools` | List every tool available to the agent this session. | | `stella config` | Show the fully resolved configuration. | +| `stella doctor` | Check the local state stella owns — today the integrity of this workspace's [session store](/docs/commands/doctor) — and exit non-zero if any check fails. `--repair` moves a corrupt `store.db` aside (renamed, never deleted). | | `stella version` | Print the version and exit. | Each command page in this section documents its usage synopsis, behavior, relevant flags, and realistic examples. diff --git a/website/content/docs/commands/meta.json b/website/content/docs/commands/meta.json index 9cd17f36..7ec3b445 100644 --- a/website/content/docs/commands/meta.json +++ b/website/content/docs/commands/meta.json @@ -1,4 +1,4 @@ { "title": "Commands", - "pages": ["index", "run", "chat", "resume", "goal", "monitor", "init", "fleet", "graph", "storage", "scripts", "models", "stats", "inspect", "observe", "memory", "mcp", "connect", "auth", "tools", "config", "version"] + "pages": ["index", "run", "chat", "resume", "goal", "monitor", "init", "fleet", "graph", "storage", "scripts", "models", "stats", "inspect", "observe", "memory", "mcp", "connect", "auth", "tools", "config", "doctor", "version"] }