diff --git a/CHANGELOG.md b/CHANGELOG.md index 5dbb3084..25b9acdb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -57,8 +57,8 @@ normative. discover the surface pre-auth. - `GET /api/v1/files?path=&language=` — single-file identity resolution. Closed error envelope `{error, code}` with codes `INVALID_PATH`, - `PATH_OUTSIDE_PROJECT`, `NOT_FOUND`, `BRIEFING_BLOCKED`, `UNAUTHORIZED`, - `STORAGE_ERROR`, `INTERNAL`. ETag / `If-None-Match` supported. + `PATH_OUTSIDE_PROJECT`, `NOT_FOUND`, `BRIEFING_BLOCKED`, `UNAUTHENTICATED`, + `BATCH_TOO_LARGE`, `STORAGE_ERROR`, `INTERNAL`. ETag / `If-None-Match` supported. - `POST /api/v1/files/batch` — bulk resolution, up to 256 queries per request, single pooled `ReaderPool` checkout per batch. Four-way partitioning: `resolved` / `not_found` / `briefing_blocked` / `errors`. @@ -105,9 +105,9 @@ normative. ### Known v1.0 limitations - **Python only.** Other-language plugins (`NG-15`) are v2.0+ scope. -- **Filigree finding emission deferred to v0.2.** Cross-product POSTing of - Clarion-generated findings into Filigree's intake (WP9-B) is deferred per - the [Sprint 2 scope amendment](docs/implementation/sprint-2/scope-amendment-2026-05.md). +- **Filigree finding emission deferred to a future release (tracked under + `release:v1.1`).** Cross-product POSTing of Clarion-generated findings into + Filigree's intake (WP9-B) is deferred per the [Sprint 2 scope amendment](docs/implementation/sprint-2/scope-amendment-2026-05.md). `issues_for(id)` (the WP9-A binding for reading from Filigree) ships in 1.0. - **HTTP file language inference** uses persisted plugin manifest language when available, with a narrow core-extension fallback for files that predate @@ -115,13 +115,28 @@ normative. - **Cooperative HMAC inbound auth** ships for the HTTP read API via `serve.http.identity_token_env` and `X-Loom-Component: clarion:`. The older bearer-token path remains available for compatibility. +- **Python plugin imports `wardline.core.registry.REGISTRY` at startup** + (loom.md §5 asterisk 2). Initialization coupling scoped to the + Wardline-aware plugin only; Clarion core and non-Wardline-aware plugins are + unaffected. *Retirement condition*: Wardline ships a stable runtime probe + API. +- **Pre-WP5 catalogue upgrade requirement.** Briefing-blocked annotations + are stored as a JSON property on file entities at v1.0 (v1.1 promotes + the field to a typed column). A v1.0 binary opening a `.clarion/clarion.db` + produced by a pre-WP5 binary finds no `briefing_blocked` properties — + pre-WP5 analyzers never wrote them — and will serve the entire catalogue + without refusal. Operators upgrading from a pre-WP5 install MUST run + `clarion analyze` (scanner active by default) against the project root + before exposing the HTTP read API or calling the `summary` MCP tool. See + [`docs/operator/secret-scanning.md`](docs/operator/secret-scanning.md#pre-wp5-catalogue-upgrade-requirement). ### Documentation - Design ladder under [`docs/clarion/1.0/`](docs/clarion/1.0/) — `requirements.md`, `system-design.md`, `detailed-design.md`. - ADRs under [`docs/clarion/adr/`](docs/clarion/adr/) — 28 Accepted at 1.0 - (through ADR-034). ADR-012 is superseded by ADR-014, whose Security + (ADR-001…ADR-034 with the documented Backlog/Superseded subset excluded). + ADR-012 is superseded by ADR-014, whose Security Posture and Error Envelope are in turn partially extended by ADR-034 for the Sprint 3 federation hardening. Four ADRs (ADR-009, ADR-010, ADR-019, ADR-020) remain Backlog and are tracked inside diff --git a/README.md b/README.md index fcca2a56..d872f828 100644 --- a/README.md +++ b/README.md @@ -101,6 +101,14 @@ is at [docs/clarion/adr/README.md](docs/clarion/adr/README.md). The Loom federation doctrine that anchors every cross-product decision is in [docs/suite/loom.md](docs/suite/loom.md). +## Storage and operations + +Clarion v1.0 keeps all state in a project-local `.clarion/` directory. +The local-first storage model, the no-NFS constraint, the no-double-analyze +constraint (fs2 advisory lock), and the v1.0 backup/restore procedure are +documented in +[docs/clarion/1.0/operations.md](docs/clarion/1.0/operations.md). + ## Contributing Read [CLAUDE.md](CLAUDE.md) for repository conventions, work-package diff --git a/docs/clarion/1.0/operations.md b/docs/clarion/1.0/operations.md new file mode 100644 index 00000000..4079dab9 --- /dev/null +++ b/docs/clarion/1.0/operations.md @@ -0,0 +1,138 @@ +# Clarion v1.0 Storage Operations + +Operator-facing reference for the constraints around Clarion's local-state +directory. v1.0 is local-first; the storage subsystem is plain SQLite under +`.clarion/`. The constraints below come straight out of +[ADR-011](../adr/ADR-011-storage-architecture.md) (writer-actor + reader-pool +over SQLite) and the v1.0 tag-cut gap-register entries DOC-11, STO-01, +STO-04, and STO-05. + +## 1. Local-first storage layout + +Per ADR-011, every Clarion project keeps its state in a `.clarion/` +directory at the project root: + +``` +.clarion/ +├── clarion.db SQLite database (entities, edges, runs, findings, summary_cache) +├── clarion.db-wal SQLite WAL companion file +├── clarion.db-shm SQLite shared-memory file +├── clarion.lock fs2 advisory lock file (writer claim) +├── instance_id Stable per-project instance ID +└── ... plugin caches, scanner baseline, etc. +``` + +There is no central server, no shared registry, no networked state. +`clarion install --path` creates `.clarion/` on a new project root; +`clarion analyze` and `clarion serve` both read and (in `analyze`'s case) +write into it. + +## 2. NFS is prohibited + +**`.clarion/` MUST live on a local filesystem.** Do not place a project root +on NFS, SMB, sshfs, or any other network filesystem and expect `clarion +analyze` or `clarion serve` to behave correctly. + +The two specific failure modes: + +- **POSIX advisory locks over NFS are unreliable.** The `fs2` exclusive + lock that protects against concurrent analyzers (§3) is silently + no-op or partially honoured on most NFS configurations. Two analyzers + on two clients can both believe they hold the lock. +- **SQLite WAL mode is not safe over network filesystems.** The SQLite + manual explicitly warns that WAL requires shared-memory primitives the + kernel does not provide across NFS. A WAL-mode database opened over + NFS can lose committed writes or corrupt the database file. + +If your only available storage is networked, do not use Clarion against +that workspace. Clone the project to local disk first. + +## 3. No double-analyze + +Only one `clarion analyze` may run against a given project root at a time. +The v1.0 binary enforces this with an exclusive `fs2` advisory lock on +`.clarion/clarion.lock`, acquired at the start of `analyze` and held for the +writer-actor lifetime. A second `clarion analyze` against the same +`.clarion/` fails fast with a clear "another clarion analyze is in progress +against this project" error rather than racing the first analyzer's run +state. (See STO-01 in the v1.0 tag-cut gap register for the originating +finding.) + +The lock is process-scoped, not user-scoped: the same user starting two +analyzers in two terminals is the typical mistake the lock prevents. + +`clarion serve` is read-only against the database and does not contend with +`analyze` on the writer lock, but it shares the same database file via the +reader pool and will observe whichever state `analyze` last committed. + +**Same-process restart caveat (STO-05):** the fs2 lock is released when +the analyzer process exits, including on crash. If the previous analyzer +exited abnormally, the next `clarion analyze` will sweep +`runs.status='running'` rows to `'failed'` (see +`recover_preexisting_running_runs`). There is no PID column or heartbeat +on `runs` at v1.0 — a same-host restart cannot distinguish "previous +analyzer crashed" from "previous analyzer is still alive but unlocked +during a brief teardown window". v1.0 mitigates this with the fs2 lock; +the `runs.owner_pid` + `heartbeat_at` schema additions are a v1.1 +follow-up. + +## 4. Supported backup procedure + +The v1.0 supported backup procedure is a four-step shutdown-and-copy. There +is no live `clarion db backup` subcommand at v1.0 (deferred to v1.1, §6). + +1. **Ensure no `clarion analyze` is running** against the project root. + The fs2 advisory lock on `.clarion/clarion.lock` will be released; a + subsequent `clarion analyze` from a backup script would otherwise race + with the backup. + +2. **Stop `clarion serve` or wait for it to be idle.** `serve` only reads, + so a torn copy from a running `serve` is less catastrophic than from + `analyze`, but a clean shutdown is required for the WAL checkpoint to + complete. + +3. **Force the WAL into the main database file** so a plain file copy + captures all committed state: + + ```bash + sqlite3 .clarion/clarion.db "PRAGMA wal_checkpoint(TRUNCATE);" + ``` + + `TRUNCATE` mode is the strongest checkpoint — it flushes the WAL into + `clarion.db` and resets `clarion.db-wal` to zero length. + +4. **Copy `.clarion/` to the backup location** with any standard tool + (`cp -a`, `rsync -a`, `tar`). All three of `clarion.db`, + `clarion.db-wal`, and `clarion.db-shm` should be present in the copy; + after a successful TRUNCATE the WAL is empty but the file should still + be copied. + +## 5. Restore + +To restore a backup: + +1. Stop any running `clarion analyze` or `clarion serve` against the + project root. +2. Replace the project's `.clarion/` directory with the backup copy. The + `instance_id` file inside `.clarion/` is part of the backup; restoring + it preserves the project's federation identity (`/api/v1/_capabilities` + `instance_id` stays stable across the restore). +3. Run `clarion analyze` to validate the restored database. A fresh analyze + re-applies any pending migrations and exercises the integrity-check path + on the e2e test surface. + +## 6. v1.1 follow-up: `clarion db backup` subcommand + +A live-safe `clarion db backup ` subcommand backed by SQLite's online +backup API is tracked for v1.1. It will replace steps 1–4 of §4 with a +single command that takes a snapshot without requiring `analyze` and +`serve` to be quiesced. Until then, follow §4. + +## References + +- [ADR-011 — Storage Architecture](../adr/ADR-011-storage-architecture.md) + — writer-actor and reader-pool design, normative source for §1 and §3. +- [`docs/implementation/v1.0-tag-cut/gap-register.md`](../../implementation/v1.0-tag-cut/gap-register.md) + — DOC-11, STO-01, STO-04, STO-05 gap entries that drove this document. +- [SQLite WAL mode caveats](https://www.sqlite.org/wal.html#noshm) — the + upstream warning that underlies the NFS prohibition in §2. diff --git a/docs/operator/clarion-http-read-api.md b/docs/operator/clarion-http-read-api.md index 0e699a08..101dfbf1 100644 --- a/docs/operator/clarion-http-read-api.md +++ b/docs/operator/clarion-http-read-api.md @@ -54,3 +54,31 @@ is available. to the entity ID and project-relative canonical path Filigree should store. It fails closed when the path is invalid, outside the project, missing from the catalog, or unavailable because of storage errors. + +## Trust assumption: loopback-no-token mode + +When both `serve.http.token_env` (legacy bearer) and +`serve.http.identity_token_env` (HMAC, preferred per +[ADR-034](../clarion/adr/ADR-034-federation-hardening.md)) are unset and the +bind is loopback (default: `127.0.0.1:9111`), the HTTP read API serves +unauthenticated. This is the intended single-user developer-workstation +trust model — the loopback socket is reachable only from processes on the +same host, and Clarion's catalogue is no more sensitive than the project +source those processes can already read. + +**On a multi-tenant developer host or shared CI runner this trust model +does not hold.** Any local process — any UID with read access to the +loopback bind socket — can read the entire non-blocked catalogue, +including every file's `entity_id`, `canonical_path`, `language`, and +`content_hash`. This is the documented v1.0 trust matrix and is not a +defect, but operators on multi-tenant hosts must configure authentication +before binding. + +Multi-tenant operators MUST set `identity_token_env` (HMAC, preferred) or +`token_env` (bearer, legacy) before running `clarion serve`. The HMAC +configuration shape is documented in the [Trust Model](#trust-model) +section above. + +The Clarion `serve` startup banner emits a `[TRUST]` line warning when +loopback-no-token mode is active (forward-reference: the banner code is a +SEC-02 follow-up PR; the trust assumption itself is current as of v1.0). diff --git a/docs/operator/secret-scanning.md b/docs/operator/secret-scanning.md index da7a2077..94be9f1d 100644 --- a/docs/operator/secret-scanning.md +++ b/docs/operator/secret-scanning.md @@ -80,6 +80,63 @@ Filigree integration for scanner findings is planned for v0.2. Until then, the l The scanner is pattern-based. It can miss novel internal key formats and it can flag high-entropy test data. Use a justified baseline for reviewed false positives, and disable LLM dispatch entirely for repos where any source disclosure would be unacceptable — set `llm.allow_live_provider: false` in `clarion.yaml` (or leave `CLARION_LLM_LIVE` unset) so the recording provider is the only path Clarion will take. -Contextual credential suppression only recognises shell/Python `#` comments in v0.1. It does not recognise `//` or `/* */` comments; use a justified baseline entry for reviewed non-Python test fixtures. +Contextual credential suppression currently recognises shell/Python `#` comments only. It does not recognise `//` or `/* */` comments; use a justified baseline entry for reviewed non-Python test fixtures. See [ADR-013](../clarion/adr/ADR-013-pre-ingest-secret-scanner.md) for design rationale. + +## Trust assumption: loopback-no-token mode + +The pre-ingest scanner's briefing-blocked annotations are only effective if +the HTTP read API also refuses to surface blocked entities to unauthorised +callers. The v1.0 HTTP API has one mode where it serves any local caller +without authentication: **loopback bind with no token configured.** + +When both `serve.http.token_env` (legacy bearer) and `serve.http.identity_token_env` +(HMAC, preferred per [ADR-034](../clarion/adr/ADR-034-federation-hardening.md)) +are unset and the bind is loopback (default: `127.0.0.1:9111`), the HTTP read +API serves unauthenticated. On a single-user developer workstation this is +the intended trust model: the loopback socket is reachable only from +processes on that host, and Clarion's catalogue is no more sensitive than +the project source those processes can already read. + +**On a multi-tenant developer host or shared CI runner the trust model is +different.** Any local process — any UID with read access to the loopback +bind socket — can read the entire non-blocked catalogue, including every +file's `entity_id`, `canonical_path`, `language`, and `content_hash`. This +is the documented v1.0 trust matrix; it is not a defect, but it is a +constraint operators must understand. + +Multi-tenant operators MUST set `identity_token_env` (HMAC, preferred) or +`token_env` (bearer, legacy) before running `clarion serve`. See +[`clarion-http-read-api.md`](./clarion-http-read-api.md) for the +configuration shape. + +The Clarion `serve` startup banner emits a `[TRUST]` line warning when +loopback-no-token mode is active (forward-reference: SEC-02 banner code +ships in a follow-up PR; the trust assumption itself is current). + +## Pre-WP5 catalogue upgrade requirement + +The WP5 pre-ingest secret scanner ships in v1.0. Briefing-blocked entities +are marked by writing `briefing_blocked: ` into the file entity's +`properties` JSON column. v1.1 will promote `briefing_blocked` to a typed +column on `entities`; v1.0 carries it as a JSON property. + +**A v1.0 binary opening a `.clarion/clarion.db` produced by a pre-WP5 +Clarion binary will find no `briefing_blocked` properties on any row.** +Pre-WP5 binaries never ran the scanner and never wrote the property; the +1.0 binary cannot retroactively discover which files contained secrets at +that earlier scan time. The HTTP read API will serve the entire catalogue +without refusal because every row's `briefing_blocked` is structurally +absent. + +**Required upgrade procedure:** after installing the v1.0 binary against +a project root that was previously analyzed by a pre-WP5 binary, run +`clarion analyze` (with the secret scanner active, which is the default) +against the project root **before** exposing the HTTP read API or calling +the `summary` MCP tool. The re-analyze produces a fresh briefing-blocked +annotation pass over all current file entities. + +This applies only to upgrades from a Clarion binary built before WP5 +landed. A v1.0 installation that has never been opened by a pre-WP5 +binary is unaffected. diff --git a/docs/operator/v1.0-release-governance.md b/docs/operator/v1.0-release-governance.md index 6ec770dc..e75baa28 100644 --- a/docs/operator/v1.0-release-governance.md +++ b/docs/operator/v1.0-release-governance.md @@ -177,6 +177,12 @@ evaluated. Python sdist from the Release and repeat the [getting started walkthrough](./getting-started.md) on a clean machine or VM. +## See also + +- [`v1.0-release-rollback.md`](./v1.0-release-rollback.md) — post-publish + incident runbook (yanking a bad release, supersession via v1.0.1, + Rekor non-revocability, downstream notification, postmortem). + ## References - GitHub REST API: repository rulesets: diff --git a/docs/operator/v1.0-release-rollback.md b/docs/operator/v1.0-release-rollback.md new file mode 100644 index 00000000..d8ec72cb --- /dev/null +++ b/docs/operator/v1.0-release-rollback.md @@ -0,0 +1,141 @@ +# v1.0 Release Rollback / Yank Runbook + +This runbook covers the h+30min "we shipped a bad release" scenario for +`v1.0.0` (or any later 1.x tag) once the GitHub Release is public. It is the +sibling of [`v1.0-release-governance.md`](./v1.0-release-governance.md), which +covers the *pre-publish* controls; this page covers the *post-publish* +incident path. + +Read this page top to bottom before taking any action. The order of steps +matters: removing the "latest" pointer is reversible; deleting assets is not, +and Sigstore Rekor entries cannot be deleted at all. + +## 1. Immediate first action — un-mark "latest" + +Within minutes of confirming a bad release, demote the release from `latest` +to `prerelease`. This removes it from the GitHub UI's "Latest" pointer and +from `gh release view --json isLatest` queries, but leaves the assets and +download URLs intact: + +```bash +gh release edit v1.0.0 --prerelease +``` + +This is the **only** action you take while still triaging. Do not delete +assets, do not delete the tag, do not delete the release. + +Verify: + +```bash +gh release view v1.0.0 --json tagName,isLatest,isPrerelease +``` + +Expect `isLatest: false`, `isPrerelease: true`. + +## 2. Asset handling policy — keep the URLs working + +**Do NOT delete release assets unless they contain sensitive data +(leaked credentials, embedded PII, or a binary the legal team has ruled must +be withdrawn).** A broken download URL is materially worse than a stale file: + +- Operator docs and the Sprint 1 walkthrough pin specific release-asset URLs + by tag. A `curl -L` against a deleted asset returns 404; a stale-but-known + binary at least produces a reproducible failure mode. +- Filigree CI and any downstream consumers that pin Clarion plugin URLs (e.g. + `pipx install https://.../clarion-plugin-python-1.0.0.tar.gz`) break loudly + rather than silently rolling forward. +- The cosign signatures and SLSA provenance attestations are bound to the + exact asset bytes; deleting and re-uploading does not restore the original + signature chain. + +If the release contains sensitive data, document the deletion in the +postmortem (§6) and notify downstream consumers (§5) before deleting. +Otherwise, leave the assets in place and rely on the `--prerelease` +demotion in §1 plus the v1.0.1 supersession in §4. + +## 3. Sigstore Rekor non-revocability + +Cosign keyless signatures recorded in the public Sigstore Rekor transparency +log **cannot be deleted, revoked, or hidden**. This is by design — Rekor is +an append-only log, and that immutability is the property that lets a +verifier trust a signature retrospectively. + +What this means for an operator yanking a release: + +- Every cosign-signed artifact the release-workflow uploaded will continue + to verify against its Rekor entry, indefinitely. `cosign verify-blob` + against the yanked assets will return success. +- The SLSA provenance attestations attached to the workflow run will also + persist in Rekor and continue to verify. +- The yank is therefore a **supersession**, not a revocation. The mechanism + is publishing a v1.0.1 (§4) that explicitly references the yanked + v1.0.0 entries in its release body, and updating downstream consumers + (§5) so they pin v1.0.1. + +If a yanked release shipped with a vulnerability that operators must not +run, the supersession note in §4 is the primary user-visible signal. Do +not assume the Rekor entry can be removed — it cannot. + +## 4. v1.0.1 publication — the supersession mechanism + +The standard tag-cut procedure from +[`v1.0-release-governance.md`](./v1.0-release-governance.md) applies, with +one addition: the v1.0.1 release body **must** carry a supersession note. +Suggested body opening: + +```markdown +**Supersedes v1.0.0 (yanked) — see .** + +v1.0.0 was demoted to prerelease and superseded by this release because +. Operators who installed v1.0.0 should +upgrade to v1.0.1; the v1.0.0 cosign signatures and Rekor entries remain +verifiable but should not be used to gate new installs. +``` + +Run the full governance guard before tagging v1.0.1 — none of the gates from +the governance doc are waived for an incident-driven release. + +## 5. Downstream notification + +Any known Filigree integrators and other downstream consumers (e.g. +operators running `pipx install` of the Clarion plugin against a pinned +v1.0.0 URL) must be informed out-of-band that v1.0.0 has been yanked. + +**Current notification path (informal):** the maintainer notifies known +operators by direct message and posts to whichever issue tracker they own. +There is no broadcast list at v1.0. + +**Flagged as a v1.1 hardening target:** a maintained downstream-consumers +list and a documented broadcast path (mailing list, security advisory feed, +or equivalent). Until then, the maintainer is the source of truth for who +needs to be told. + +If you are the operator running the yank, write down everyone you notified +and add the list to the postmortem (§6). + +## 6. Postmortem requirement + +Create a `filigree` incident issue at the moment the yank begins: + +```bash +filigree create \ + --title "Incident: v1.0.0 yanked — " \ + --priority P0 \ + --label incident \ + --actor +``` + +The issue captures the incident timeline, defect summary, fix landed in +v1.0.1, downstream notifications sent (§5), and any asset-deletion decisions +(§2). Link the issue URL from the v1.0.1 release body's supersession note. + +The incident issue stays `release:1.x` (the release that fixed it) and uses +the `incident` label so the post-incident review pass can find it. + +## See also + +- [`v1.0-release-governance.md`](./v1.0-release-governance.md) — pre-publish + controls and the tag-cut procedure that v1.0.1 also runs. +- [`getting-started.md`](./getting-started.md) — the public install path that + the supersession is protecting.