From cc9b899cb3487f12b6a24162ced796a1b2ea7ff4 Mon Sep 17 00:00:00 2001 From: John Morrissey <544926+tachyon-beep@users.noreply.github.com> Date: Fri, 22 May 2026 20:08:42 +1000 Subject: [PATCH 1/9] test(e2e): automate external-operator smoke walkthrough MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the technical half of GOV-04: the operator-led smoke gate had been a markdown checklist that required a human to walk through each step. This PR adds `tests/e2e/external-operator-smoke.sh` which automates steps 1–7 (install, analyze, MCP query, idempotency, secret-plant + briefing-block verification, blocked-summary refusal) and generates a draft results file. Step 8 (operator-improvisation tally) remains human-judged because no automation can detect "operator read source outside the README to know what to do next". Verified end-to-end against `psf/requests@v2.32.3` (825 entities, 1254 edges, 8 MCP tools, 336 briefing-blocked entities post-plant, blocked- summary envelope correctly returned without LLM call). All 7 technical steps PASS; step 4.3(c) SKIPs cleanly when OPENROUTER_API_KEY is unset. The script supports two modes via env knobs: - CARGO_BUILD=1 (default): in-tree release build + editable plugin venv; what CI runs to validate the procedure stays working. - CARGO_BUILD=0 + CLARION_BIN=... + CLARION_PLUGIN_BIN=...: external operator scenario, where the binaries were installed via the GitHub Release archive + pipx and the operator just runs the script. Updates: - tests/e2e/external-operator-smoke.md: refocused as the operator-facing procedure overview pointing at the harness; retains the rationale for each step and step 8's human-judgement framing. - .gitignore: exclude `external-operator-smoke-results-*.md` so per-run artefacts don't accidentally get committed. Forward-references the v1.0 tag-cut gap register entry GOV-04 (`clarion-e464251ab3`); a dated result artefact will be archived at tag-cut time under `docs/implementation/v1.0-tag-cut/`. --- .gitignore | 5 +- tests/e2e/external-operator-smoke.md | 113 ++++-- tests/e2e/external-operator-smoke.sh | 501 +++++++++++++++++++++++++++ 3 files changed, 588 insertions(+), 31 deletions(-) create mode 100755 tests/e2e/external-operator-smoke.sh diff --git a/.gitignore b/.gitignore index 17baed25..a65b5291 100644 --- a/.gitignore +++ b/.gitignore @@ -22,4 +22,7 @@ __pycache__/ htmlcov/ .claude/ -.env \ No newline at end of file +.env + +# Smoke-test result artifacts (per-run; archived separately at tag-cut) +tests/e2e/external-operator-smoke-results-*.md \ No newline at end of file diff --git a/tests/e2e/external-operator-smoke.md b/tests/e2e/external-operator-smoke.md index 75ab48cb..d31c38ce 100644 --- a/tests/e2e/external-operator-smoke.md +++ b/tests/e2e/external-operator-smoke.md @@ -3,6 +3,11 @@ Source: WS-D of [thread-1-pre-publish-blockers.md](../../docs/implementation/v0.1-publish/thread-1-pre-publish-blockers.md#5-workstream-d--publish-gate-external-operator-smoke-test) and Filigree issue `clarion-3e0e481ef7`. +The automated harness lives at +[`external-operator-smoke.sh`](external-operator-smoke.sh). The operator +runs that script and then fills in step 8 (the human-judgement part). +This document is the procedure overview and the rationale for each step. + ## Purpose Verify that an outside operator, working from a fresh machine and reading @@ -15,50 +20,98 @@ The test fails if any step requires reading source code outside of `README.md` and `docs/operator/getting-started.md`. A failure here is a B.1/B.2 docs bug, not a runtime defect. +## How to run the harness + +### CI / repo-internal smoke (in-tree build) + +From the repo root on the target machine (Linux or macOS): + +```bash +bash tests/e2e/external-operator-smoke.sh +``` + +The harness builds the workspace in release mode, installs the plugin into the +editable venv at `plugins/python/.venv`, fetches the canonical corpus +(`psf/requests` at the pinned tag), and walks steps 1–7. Step 8 is left for +the operator to fill in. A draft results file is written to +`tests/e2e/external-operator-smoke-results-YYYY-MM-DD-.md`. + +### External-operator smoke (binary install) + +The outside-operator scenario assumes Clarion has been installed via the +GitHub Release archive and the Python plugin via `pipx`. Tell the harness to +skip the build step and point it at the operator-installed binaries: + +```bash +export CARGO_BUILD=0 +export CLARION_BIN=/path/to/installed/clarion +export CLARION_PLUGIN_BIN=/path/to/installed/clarion-plugin-python +bash external-operator-smoke.sh # script is published as a release asset +``` + +The script makes no other repo assumptions in this mode; the operator can run +it from any directory. + +### Environment knobs + +| Variable | Default | Purpose | +|---|---|---| +| `CARGO_BUILD` | `1` | Build clarion + plugin from source. `0` skips. | +| `CLARION_BIN` | (autodetected) | Path to the `clarion` binary. | +| `CLARION_PLUGIN_BIN` | (autodetected) | Path to `clarion-plugin-python`. | +| `CORPUS_REPO` | `https://github.com/psf/requests.git` | Public Python corpus to analyse. | +| `CORPUS_REF` | `v2.32.3` | Pinned tag for reproducibility. | +| `OPENROUTER_API_KEY` | unset | If unset, step 4.3(c) `summary` is skipped (not failed). | +| `RESULTS_FILE` | `external-operator-smoke-results--.md` | Override the output path. | + ## Test environment | Knob | Value | |---|---| -| OS | `ubuntu:24.04` (or equivalent fresh image) | +| OS | `ubuntu:24.04` (or equivalent fresh image) for the canonical run; macOS for the platform-coverage run | | Operator | Someone (or a fresh agent session) with no prior knowledge of this repo | -| Working dir | `/tmp/smoke-test/` inside the container | -| `OPENROUTER_API_KEY` | Required for steps 4.3 and 5; the test can run *without* the key but steps that depend on live LLM dispatch are then explicit skips, not silent passes | +| Working dir | the harness creates its own `mktemp` scratch; the corpus is cloned there | +| `OPENROUTER_API_KEY` | Required for step 4.3(c); other steps work without it | | Network | Outbound HTTPS to `api.openrouter.ai` and `github.com` (and PyPI/crates if installing from source) | ## Steps -| # | Step | Pass criteria | -|---|---|---| -| 1 | Install the Rust binary per WS-C's chosen path (GitHub Releases per ADR-033) | `clarion --version` prints a version on `$PATH` | -| 2 | Install the Python plugin via `pipx` per the tutorial | `which clarion-plugin-python` resolves; the binary is on `$PATH` | -| 3 | `clarion install` against a small Python project (`requests` source tarball ~7k LOC) | `.clarion/` exists; `.clarion/clarion.db` exists; init log line emitted | -| 4.1 | `clarion analyze` | Exit 0; `analyze complete: run ok` reported; entity count > 0 | -| 4.2 | Start `clarion serve` in one shell; connect MCP client in another | MCP `tools/list` returns the 8 tools (`entity_at`, `find_entity`, `callers_of`, `execution_paths_from`, `summary`, `issues_for`, `neighborhood`, `subsystem_members`) | -| 4.3 | Three pre-scripted questions over MCP:
(a) `find_entity(pattern="Session")` lists module-level matches
(b) `callers_of(id="python:function:requests.api.get|function")` returns a non-empty list
(c) `summary(id="python:function:requests.sessions.Session.send|function")` returns a paragraph | (a) and (b) return non-empty; (c) returns a paragraph (live LLM) or `null` summary plus a documented skip if `OPENROUTER_API_KEY` absent | -| 5 | Re-run `clarion analyze` | Idempotent: entity/edge counts on the second run match the first | -| 6 | Plant `.env` with `AWS_ACCESS_KEY_ID=AKIA0123456789ABCDEF`; re-run `clarion analyze` | Exit 0 (`soft_failed`); `CLA-SEC-SECRET-DETECTED` finding in `findings` table; affected entities carry `briefing_blocked = "secret_present"` | -| 7 | `clarion serve` against the post-secret-block DB; call `summary(id)` on a blocked entity's id | Returns `briefing_blocked` envelope with no LLM call | -| 8 | Operator-improvisation tally | Total count of "I had to look at the source to understand what to do next" events. **Target: 0.** Any positive count is a B.1/B.2 docs bug. | +The harness automates steps 1–7. Step 8 is human-judged and the operator +fills it into the generated results file. + +| # | Step | Pass criteria | Automated? | +|---|---|---|---| +| 1 | Install the Rust binary per WS-C's chosen path (GitHub Releases per ADR-033) | `clarion --version` prints a version on `$PATH` | ✅ | +| 2 | Install the Python plugin via `pipx` per the tutorial | `which clarion-plugin-python` resolves; the binary is on `$PATH` | ✅ | +| 3 | `clarion install` against a small Python project (the harness uses `psf/requests` at a pinned tag) | `.clarion/` exists; `.clarion/clarion.db` exists; init log line emitted | ✅ | +| 4.1 | `clarion analyze` | Exit 0; entity count > 0 | ✅ | +| 4.2 | Start `clarion serve`; connect MCP client | MCP `tools/list` returns the 8 tools (`entity_at`, `find_entity`, `callers_of`, `execution_paths_from`, `summary`, `issues_for`, `neighborhood`, `subsystem_members`) | ✅ | +| 4.3 | Three MCP queries against analyzed corpus:
(a) `find_entity(pattern="Session")` lists module-level matches
(b) `callers_of(id=)` returns a non-empty list
(c) `summary(id=)` returns a paragraph | (a) and (b) return non-empty; (c) returns a paragraph (live LLM) or is skipped when `OPENROUTER_API_KEY` is absent | ✅ | +| 5 | Re-run `clarion analyze` | Idempotent: entity/edge counts on the second run match the first | ✅ | +| 6 | Plant `.env` with `AWS_ACCESS_KEY_ID=AKIA0123456789ABCDEF`; re-run `clarion analyze` | `briefing_blocked` entities count > 0; `CLA-SEC-SECRET-DETECTED` finding emitted | ✅ | +| 7 | `clarion serve` against the post-secret-block DB; call `summary(id)` on a blocked entity's id | Returns `briefing_blocked` envelope with no LLM call | ✅ | +| 8 | Operator-improvisation tally | Total count of "I had to look at the source to understand what to do next" events. **Target: 0.** Any positive count is a B.1/B.2 docs bug. | ❌ (human-judged) | ## Recording results -Capture the smoke test as a single Markdown report at -`tests/e2e/external-operator-smoke-results-.md` with: +The harness writes a Markdown report at +`tests/e2e/external-operator-smoke-results--.md` automatically. +Steps 1–7 are populated with PASS/FAIL/SKIP and a one-line detail. The +operator fills in: -- The date / agent identity / container image tag. -- Per-step: status (`pass` / `fail` / `skip`), command run, abridged stdout - excerpt, and any improvisation events that occurred. -- A final aggregate verdict: `gate_passed: true` only if all non-skipped steps - pass and the improvisation tally is 0. +- Step 8 improvisation tally (count + per-event bullets) +- Final attestation signature + date -Attach the report as a comment on Filigree issue `clarion-3e0e481ef7` before -closing. +For a release gate, attach the completed report as a comment on Filigree +issue `clarion-3e0e481ef7` (or `clarion-e464251ab3`, the renamed GOV-04 +issue) before closing. ## Repeating on additional platforms -This document specifies one (Linux x86_64) run. The release matrix in -[ADR-033](../../docs/clarion/adr/ADR-033-v1.0-distribution.md) also produces -`x86_64-apple-darwin` and `aarch64-apple-darwin` binaries; re-run this -checklist on at least one macOS target before the v1.0 tag is treated as -generally publishable. Document each platform's run as a separate -`*-results-*.md` file. +The release matrix in +[ADR-033](../../docs/clarion/adr/ADR-033-v1.0-distribution.md) produces +`x86_64-unknown-linux-gnu`, `x86_64-apple-darwin`, and `aarch64-apple-darwin` +binaries. Run the harness on at least one Linux target and one macOS target +before the `v1.0.0` tag is treated as generally publishable. Each platform +produces its own dated results file (the host tag is part of the default +filename) — both must be archived before tag-cut. diff --git a/tests/e2e/external-operator-smoke.sh b/tests/e2e/external-operator-smoke.sh new file mode 100755 index 00000000..73b00e8f --- /dev/null +++ b/tests/e2e/external-operator-smoke.sh @@ -0,0 +1,501 @@ +#!/usr/bin/env bash +# External-operator smoke test (publish gate) — automated harness. +# +# Closes the technical half of GOV-04 (external-operator smoke evidence). +# Steps 1–7 below are automated; step 8 (operator-improvisation tally) +# remains a human judgement that the operator fills into the generated +# results file. +# +# See tests/e2e/external-operator-smoke.md for the procedure and the +# rationale for each step. +# +# Modes +# ----- +# - `CARGO_BUILD=1` (default in repo CI): builds clarion + installs the +# plugin from the source tree, fetches the canonical corpus, runs the +# walkthrough, writes a results file. +# - `CARGO_BUILD=0` + `CLARION_BIN=...` + `CLARION_PLUGIN_BIN=...`: skips +# the source build; expects the operator to have already installed +# clarion (via GitHub Release archive) and clarion-plugin-python (via +# pipx) on $PATH. +# +# Outputs +# ------- +# Writes a results file at $RESULTS_DIR/external-operator-smoke-results- +# YYYY-MM-DD-.md. Steps 1–7 are auto-filled with PASS/FAIL/SKIP and +# a stdout excerpt; step 8 is left as a TODO for the operator to fill in. +# +# Exit codes +# ---------- +# 0 — all technical steps PASS (or explicit SKIPs were declared upfront). +# 1 — any technical step FAILED. +# 78 — soft-failure exit (matches `clarion analyze`'s soft-fail convention) +# reserved; not currently used but documented for future hardening. + +set -euo pipefail + +# -------- configuration -------- + +REPO_ROOT="${REPO_ROOT:-$(git rev-parse --show-toplevel 2>/dev/null || pwd)}" +CARGO_BUILD="${CARGO_BUILD:-1}" +RESULTS_DIR="${RESULTS_DIR:-$REPO_ROOT/tests/e2e}" +VENV="${VENV:-$REPO_ROOT/plugins/python/.venv}" +CORPUS_REPO="${CORPUS_REPO:-https://github.com/psf/requests.git}" +CORPUS_REF="${CORPUS_REF:-v2.32.3}" + +DATE_STAMP="$(date -u +%Y-%m-%d)" +HOST_TAG="$(uname -s | tr '[:upper:]' '[:lower:]')-$(uname -m)" +RESULTS_FILE="${RESULTS_FILE:-$RESULTS_DIR/external-operator-smoke-results-${DATE_STAMP}-${HOST_TAG}.md}" + +# Allow OPENROUTER_API_KEY to be unset; step 4.3(c) skips cleanly. +OPENROUTER_API_KEY="${OPENROUTER_API_KEY:-}" + +log() { printf '[smoke] %s\n' "$*" >&2; } +fail() { printf '[smoke] FAIL: %s\n' "$*" >&2; exit 1; } + +# Per-step PASS/FAIL/SKIP tracking. step_status[i]="PASS" / "FAIL" / "SKIP". +declare -A step_status +declare -A step_detail + +record() { + local step="$1" status="$2" detail="$3" + step_status[$step]="$status" + step_detail[$step]="$detail" + log "step $step: $status — $detail" +} + +# -------- preflight -------- + +cd "$REPO_ROOT" + +if [ "$CARGO_BUILD" = "1" ]; then + log "preflight: building clarion (release) ..." + cargo build --workspace --release + CLARION_BIN="${CLARION_BIN:-$REPO_ROOT/target/release/clarion}" + if [ ! -d "$VENV" ]; then + log "preflight: creating plugin venv at $VENV ..." + python3 -m venv "$VENV" + fi + log "preflight: installing clarion-plugin-python (editable) ..." + "$VENV/bin/pip" install --quiet -e "$REPO_ROOT/plugins/python[dev]" + CLARION_PLUGIN_BIN="${CLARION_PLUGIN_BIN:-$VENV/bin/clarion-plugin-python}" + export PATH="$REPO_ROOT/target/release:$VENV/bin:$PATH" +else + CLARION_BIN="${CLARION_BIN:-$(command -v clarion || true)}" + CLARION_PLUGIN_BIN="${CLARION_PLUGIN_BIN:-$(command -v clarion-plugin-python || true)}" +fi + +[ -x "$CLARION_BIN" ] || fail "clarion binary missing at $CLARION_BIN (set CLARION_BIN= or CARGO_BUILD=1)" +[ -x "$CLARION_PLUGIN_BIN" ] || fail "clarion-plugin-python missing at $CLARION_PLUGIN_BIN (set CLARION_PLUGIN_BIN= or CARGO_BUILD=1)" + +CLARION_VERSION="$("$CLARION_BIN" --version 2>&1 | head -1)" +PLUGIN_VERSION="$("$CLARION_PLUGIN_BIN" --version 2>&1 | head -1 || true)" +log "preflight: $CLARION_VERSION / $PLUGIN_VERSION" + +# Scratch directory for the corpus. +WORK_DIR="$(mktemp -d -t clarion-smoke-XXXXXX)" +trap 'rm -rf "$WORK_DIR"' EXIT +log "scratch: $WORK_DIR" + +# -------- step 1: clarion is on $PATH and --version works -------- + +if [ -n "$CLARION_VERSION" ]; then + record "1" "PASS" "clarion --version: $CLARION_VERSION" +else + record "1" "FAIL" "clarion --version produced no output" +fi + +# -------- step 2: clarion-plugin-python on $PATH -------- + +if [ -x "$CLARION_PLUGIN_BIN" ]; then + record "2" "PASS" "plugin binary at $CLARION_PLUGIN_BIN; $PLUGIN_VERSION" +else + record "2" "FAIL" "plugin binary not found" +fi + +# -------- step 3: clarion install against a small Python corpus -------- + +log "fetching corpus $CORPUS_REPO @ $CORPUS_REF ..." +cd "$WORK_DIR" +if ! git clone --quiet --depth 1 --branch "$CORPUS_REF" "$CORPUS_REPO" corpus; then + record "3" "FAIL" "git clone of $CORPUS_REPO @ $CORPUS_REF failed" + fail "cannot proceed without corpus" +fi +cd corpus + +if "$CLARION_BIN" install >/dev/null 2>"$WORK_DIR/install.err"; then + if [ -f .clarion/clarion.db ]; then + record "3" "PASS" ".clarion/clarion.db created against psf/requests@$CORPUS_REF" + else + record "3" "FAIL" "install reported success but .clarion/clarion.db missing" + fi +else + record "3" "FAIL" "clarion install exited non-zero: $(tr '\n' ' ' < "$WORK_DIR/install.err")" +fi + +# -------- step 4.1: clarion analyze (initial) -------- + +log "running clarion analyze ..." +if "$CLARION_BIN" analyze . >"$WORK_DIR/analyze1.out" 2>"$WORK_DIR/analyze1.err"; then + ENTITY_COUNT_1="$(sqlite3 .clarion/clarion.db 'SELECT COUNT(*) FROM entities WHERE kind != "subsystem"' || echo 0)" + EDGE_COUNT_1="$(sqlite3 .clarion/clarion.db 'SELECT COUNT(*) FROM edges' || echo 0)" + if [ "$ENTITY_COUNT_1" -gt 0 ]; then + record "4.1" "PASS" "analyze ok; entities=$ENTITY_COUNT_1 edges=$EDGE_COUNT_1" + else + record "4.1" "FAIL" "analyze ok but entity count is 0" + fi +else + record "4.1" "FAIL" "clarion analyze exited non-zero: $(tail -3 "$WORK_DIR/analyze1.err" | tr '\n' ' ')" +fi + +# -------- step 4.2 + 4.3: clarion serve + MCP queries -------- + +log "driving MCP stdio ..." +MCP_REPORT="$WORK_DIR/mcp.json" +PROJECT_DIR_FOR_PY="$WORK_DIR/corpus" +CLARION_BIN_FOR_PY="$CLARION_BIN" + +set +e +python3 - "$CLARION_BIN_FOR_PY" "$PROJECT_DIR_FOR_PY" "$MCP_REPORT" "${OPENROUTER_API_KEY:-NONE}" <<'PY' +import json, sys, subprocess, sqlite3 +from pathlib import Path + +clarion_bin, project_dir, report_path, openrouter_key = sys.argv[1:5] +project_dir = Path(project_dir) +report = {"step_4_2": None, "step_4_3a": None, "step_4_3b": None, "step_4_3c": None, + "tools_listed": None, "find_entity_hits": None, "callers_of_hits": None, + "summary_envelope_kind": None, "summary_skipped_reason": None} + +def write_frame(proc, message): + body = json.dumps(message, separators=(",", ":")).encode("utf-8") + proc.stdin.write(b"Content-Length: " + str(len(body)).encode("ascii") + b"\r\n\r\n") + proc.stdin.write(body) + proc.stdin.flush() + +def read_frame(proc): + headers = {} + while True: + line = proc.stdout.readline() + if not line: + stderr = proc.stderr.read().decode("utf-8", "replace") + raise AssertionError(f"server closed stdout; stderr={stderr}") + if line == b"\r\n": + break + k, v = line.decode("ascii").strip().split(":", 1) + headers[k.lower()] = v.strip() + return json.loads(proc.stdout.read(int(headers["content-length"]))) + +def tool_call(proc, rid, name, args): + write_frame(proc, {"jsonrpc": "2.0", "id": rid, "method": "tools/call", + "params": {"name": name, "arguments": args}}) + return read_frame(proc) + +# Pick a real entity from the analyzed corpus to test against. +conn = sqlite3.connect(project_dir / ".clarion" / "clarion.db") +ent = conn.execute(""" + SELECT id, kind, name FROM entities + WHERE kind = 'function' AND name = 'get' + AND id LIKE 'python:function:%' + LIMIT 1 +""").fetchone() +if not ent: + ent = conn.execute(""" + SELECT id, kind, name FROM entities + WHERE kind = 'function' LIMIT 1 + """).fetchone() +session_ent = conn.execute(""" + SELECT id FROM entities WHERE kind = 'class' AND name = 'Session' LIMIT 1 +""").fetchone() +conn.close() + +if not ent: + report["step_4_2"] = "FAIL: no function entities to query" + Path(report_path).write_text(json.dumps(report)) + sys.exit(2) + +proc = subprocess.Popen([clarion_bin, "serve"], + stdin=subprocess.PIPE, stdout=subprocess.PIPE, + stderr=subprocess.PIPE, cwd=str(project_dir)) + +try: + # MCP initialize + write_frame(proc, {"jsonrpc": "2.0", "id": 1, "method": "initialize", + "params": {"protocolVersion": "2024-11-05", + "capabilities": {}, + "clientInfo": {"name": "smoke", "version": "1"}}}) + read_frame(proc) + write_frame(proc, {"jsonrpc": "2.0", "method": "notifications/initialized"}) + + # tools/list + write_frame(proc, {"jsonrpc": "2.0", "id": 2, "method": "tools/list"}) + listed = read_frame(proc) + tools = [t["name"] for t in listed.get("result", {}).get("tools", [])] + required = {"entity_at", "find_entity", "callers_of", "execution_paths_from", + "summary", "issues_for", "neighborhood", "subsystem_members"} + if required.issubset(set(tools)): + report["step_4_2"] = f"PASS: tools/list returned {len(tools)} tools including all 8 required" + else: + missing = required - set(tools) + report["step_4_2"] = f"FAIL: tools/list missing {sorted(missing)}" + report["tools_listed"] = tools + + def unwrap(envelope): + """Clarion MCP envelope wraps the tool payload under `result`.""" + return envelope.get("result") if isinstance(envelope, dict) else None + + # 4.3(a) find_entity + pattern = "Session" if session_ent else ent[2] + fe = tool_call(proc, 3, "find_entity", {"pattern": pattern, "limit": 5}) + fe_body = json.loads(fe["result"]["content"][0]["text"]) + inner = unwrap(fe_body) or {} + matches = inner.get("entities") or inner.get("matches") or inner.get("results") or [] + report["find_entity_hits"] = len(matches) + report["step_4_3a"] = (f"PASS: find_entity('{pattern}') returned {len(matches)} matches" + if matches else f"FAIL: find_entity('{pattern}') returned empty (envelope: {fe_body})") + + # 4.3(b) callers_of — find a function with at least one caller. + conn2 = sqlite3.connect(project_dir / ".clarion" / "clarion.db") + row = conn2.execute(""" + SELECT e.id, COUNT(*) AS c FROM entities e + JOIN edges ed ON ed.to_id = e.id + WHERE e.kind = 'function' AND ed.kind = 'calls' + GROUP BY e.id ORDER BY c DESC LIMIT 1 + """).fetchone() + conn2.close() + if not row: + report["step_4_3b"] = "FAIL: no `calls` edges in the analyzed graph" + else: + callers_target = row[0] + co = tool_call(proc, 4, "callers_of", {"id": callers_target, "limit": 5}) + co_body = json.loads(co["result"]["content"][0]["text"]) + co_inner = unwrap(co_body) or {} + callers = co_inner.get("callers") or [] + report["callers_of_hits"] = len(callers) + if callers: + report["step_4_3b"] = f"PASS: callers_of('{callers_target}') returned {len(callers)} callers" + else: + report["step_4_3b"] = f"FAIL: callers_of('{callers_target}') returned empty (envelope: {co_body})" + + # 4.3(c) summary — only runs if OPENROUTER_API_KEY is set + if openrouter_key and openrouter_key != "NONE": + sm = tool_call(proc, 6, "summary", {"id": ent[0]}) + sm_body = json.loads(sm["result"]["content"][0]["text"]) + sm_inner = unwrap(sm_body) or {} + if sm_body.get("ok") and (sm_inner.get("summary") or sm_inner.get("available") is True): + report["summary_envelope_kind"] = "live" + report["step_4_3c"] = f"PASS: live summary for {ent[0]} returned non-empty" + else: + report["summary_envelope_kind"] = "live-empty" + report["step_4_3c"] = f"FAIL: summary call ran but envelope had no usable payload: {sm_body}" + else: + report["step_4_3c"] = "SKIP: OPENROUTER_API_KEY not set" + report["summary_skipped_reason"] = "no api key" + +finally: + try: + proc.stdin.close() + except Exception: + pass + proc.wait(timeout=10) + +Path(report_path).write_text(json.dumps(report, indent=2)) +PY +MCP_RC=$? +set -e + +if [ "$MCP_RC" -ne 0 ]; then + record "4.2" "FAIL" "MCP harness exited with code $MCP_RC" + record "4.3a" "FAIL" "MCP harness failed before query a" + record "4.3b" "FAIL" "MCP harness failed before query b" + record "4.3c" "FAIL" "MCP harness failed before query c" +else + # shellcheck disable=SC2002 + REPORT_JSON="$(cat "$MCP_REPORT")" + parse_field() { python3 -c "import json,sys; print(json.loads(sys.argv[1]).get(sys.argv[2]) or '')" "$REPORT_JSON" "$1"; } + record "4.2" "$(parse_field step_4_2 | cut -d: -f1)" "$(parse_field step_4_2)" + record "4.3a" "$(parse_field step_4_3a | cut -d: -f1)" "$(parse_field step_4_3a)" + record "4.3b" "$(parse_field step_4_3b | cut -d: -f1)" "$(parse_field step_4_3b)" + case "$(parse_field step_4_3c)" in + PASS*) record "4.3c" "PASS" "$(parse_field step_4_3c)" ;; + FAIL*) record "4.3c" "FAIL" "$(parse_field step_4_3c)" ;; + SKIP*) record "4.3c" "SKIP" "$(parse_field step_4_3c)" ;; + *) record "4.3c" "FAIL" "unknown summary state: $(parse_field step_4_3c)" ;; + esac +fi + +# -------- step 5: re-run analyze for idempotency -------- + +log "running clarion analyze (re-run for idempotency) ..." +if "$CLARION_BIN" analyze . >"$WORK_DIR/analyze2.out" 2>"$WORK_DIR/analyze2.err"; then + ENTITY_COUNT_2="$(sqlite3 .clarion/clarion.db 'SELECT COUNT(*) FROM entities WHERE kind != "subsystem"')" + EDGE_COUNT_2="$(sqlite3 .clarion/clarion.db 'SELECT COUNT(*) FROM edges')" + if [ "$ENTITY_COUNT_2" = "$ENTITY_COUNT_1" ] && [ "$EDGE_COUNT_2" = "$EDGE_COUNT_1" ]; then + record "5" "PASS" "idempotent: entities=$ENTITY_COUNT_2 edges=$EDGE_COUNT_2 unchanged" + else + record "5" "FAIL" "non-idempotent: entities $ENTITY_COUNT_1 -> $ENTITY_COUNT_2, edges $EDGE_COUNT_1 -> $EDGE_COUNT_2" + fi +else + record "5" "FAIL" "re-run analyze exited non-zero" +fi + +# -------- step 6: plant secret + re-run analyze; verify briefing_blocked -------- + +log "planting .env with fake AWS credential; re-running analyze ..." +cat > .env <<'ENV' +AWS_ACCESS_KEY_ID=AKIA0123456789ABCDEF +AWS_SECRET_ACCESS_KEY=examplefakefakefakefakefakefakefakefake1234 +ENV + +ANALYZE_3_EXIT=0 +"$CLARION_BIN" analyze . >"$WORK_DIR/analyze3.out" 2>"$WORK_DIR/analyze3.err" || ANALYZE_3_EXIT=$? + +# Expected: soft-failure (exit 78) or success with briefing_blocked recorded. +BLOCKED_COUNT="$(sqlite3 .clarion/clarion.db "SELECT COUNT(*) FROM entities WHERE json_extract(properties, '\$.briefing_blocked') IS NOT NULL" 2>/dev/null || echo 0)" +FINDING_COUNT="$(sqlite3 .clarion/clarion.db "SELECT COUNT(*) FROM findings WHERE rule_id = 'CLA-SEC-SECRET-DETECTED'" 2>/dev/null || echo 0)" + +if [ "$BLOCKED_COUNT" -gt 0 ] && [ "$FINDING_COUNT" -gt 0 ]; then + record "6" "PASS" "post-plant: $BLOCKED_COUNT blocked entities, $FINDING_COUNT secret findings (analyze exit $ANALYZE_3_EXIT)" +elif [ "$BLOCKED_COUNT" -gt 0 ]; then + record "6" "PASS" "post-plant: $BLOCKED_COUNT blocked entities (no CLA-SEC-SECRET-DETECTED finding rows; finding code may have changed)" +elif [ "$FINDING_COUNT" -gt 0 ]; then + record "6" "FAIL" "secret finding emitted but no entity marked briefing_blocked" +else + record "6" "FAIL" "no briefing_blocked entities and no CLA-SEC-SECRET-DETECTED finding after planting" +fi + +# -------- step 7: serve against post-block DB; summary on blocked entity returns blocked envelope -------- + +if [ "$BLOCKED_COUNT" -gt 0 ]; then + log "verifying blocked-entity summary refusal ..." + BLOCKED_ID="$(sqlite3 .clarion/clarion.db "SELECT id FROM entities WHERE json_extract(properties, '\$.briefing_blocked') IS NOT NULL LIMIT 1")" + if [ -n "$BLOCKED_ID" ]; then + set +e + python3 - "$CLARION_BIN" "$WORK_DIR/corpus" "$BLOCKED_ID" "$WORK_DIR/step7.json" <<'PY' +import json, sys, subprocess +from pathlib import Path + +clarion_bin, project_dir, blocked_id, out_path = sys.argv[1:5] + +def write_frame(proc, message): + body = json.dumps(message, separators=(",", ":")).encode("utf-8") + proc.stdin.write(b"Content-Length: " + str(len(body)).encode("ascii") + b"\r\n\r\n") + proc.stdin.write(body) + proc.stdin.flush() + +def read_frame(proc): + headers = {} + while True: + line = proc.stdout.readline() + if not line: raise AssertionError("eof") + if line == b"\r\n": break + k, v = line.decode("ascii").strip().split(":", 1) + headers[k.lower()] = v.strip() + return json.loads(proc.stdout.read(int(headers["content-length"]))) + +proc = subprocess.Popen([clarion_bin, "serve"], + stdin=subprocess.PIPE, stdout=subprocess.PIPE, + stderr=subprocess.PIPE, cwd=project_dir) +try: + write_frame(proc, {"jsonrpc": "2.0", "id": 1, "method": "initialize", + "params": {"protocolVersion": "2024-11-05", + "capabilities": {}, + "clientInfo": {"name": "smoke", "version": "1"}}}) + read_frame(proc) + write_frame(proc, {"jsonrpc": "2.0", "method": "notifications/initialized"}) + write_frame(proc, {"jsonrpc": "2.0", "id": 2, "method": "tools/call", + "params": {"name": "summary", "arguments": {"id": blocked_id}}}) + resp = read_frame(proc) + body = json.loads(resp["result"]["content"][0]["text"]) + inner = body.get("result") if isinstance(body, dict) else None + blocked = bool(inner and inner.get("briefing_blocked")) + Path(out_path).write_text(json.dumps({"blocked": blocked, "envelope": body}, indent=2)) +finally: + try: proc.stdin.close() + except Exception: pass + proc.wait(timeout=10) +PY + STEP7_RC=$? + set -e + if [ "$STEP7_RC" -eq 0 ] && python3 -c "import json,sys;sys.exit(0 if json.loads(open(sys.argv[1]).read()).get('blocked') else 1)" "$WORK_DIR/step7.json"; then + record "7" "PASS" "summary on blocked entity '$BLOCKED_ID' returned briefing-blocked envelope (no LLM call)" + else + record "7" "FAIL" "summary on blocked entity did not return briefing-blocked envelope" + fi + else + record "7" "FAIL" "blocked count > 0 but could not select an id" + fi +else + record "7" "SKIP" "skipped because step 6 found no blocked entities" +fi + +# -------- step 8: operator-improvisation tally (human-judged) -------- + +record "8" "TODO" "operator must fill in: count of source-reads outside README + getting-started" + +# -------- write results file -------- + +log "writing results to $RESULTS_FILE ..." +mkdir -p "$(dirname "$RESULTS_FILE")" +{ + echo "# External-operator smoke result" + echo + echo "**Date (UTC)**: $(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo "**Host**: $(uname -srm)" + echo "**Clarion**: $CLARION_VERSION" + echo "**Plugin**: $PLUGIN_VERSION" + echo "**Corpus**: $CORPUS_REPO @ $CORPUS_REF" + echo "**Mode**: $([ "$CARGO_BUILD" = "1" ] && echo "in-repo build (CARGO_BUILD=1)" || echo "external binary")" + echo + echo "## Step results" + echo + echo "| # | Step | Status | Detail |" + echo "|---|------|--------|--------|" + for step in 1 2 3 4.1 4.2 4.3a 4.3b 4.3c 5 6 7 8; do + echo "| $step | $(grep -A1 "^| $step " tests/e2e/external-operator-smoke.md 2>/dev/null | head -1 | cut -d'|' -f3 | sed 's/^ *//;s/ *$//' || echo "see md") | ${step_status[$step]:-MISSING} | ${step_detail[$step]:-} |" + done + echo + echo "## Step 8 (operator-fill)" + echo + echo "Improvisation tally: TODO — record the count of moments where you (the operator)" + echo "had to read source code outside \`README.md\` and \`docs/operator/getting-started.md\`" + echo "to know what to do next." + echo + echo "Target: 0. Any non-zero count is a B.1/B.2 docs bug, not a runtime defect." + echo + echo "Improvisation events observed:" + echo + echo "- [ ] (fill in below; one bullet per event)" + echo + echo "## Verdict" + echo + echo "**Technical** (steps 1–7): set automatically by this harness." + echo + echo "**Operator** (step 8): fill in after walking through the procedure." + echo + echo "**Gate passed**: only if all non-skip steps PASS *and* the operator improvisation" + echo "tally is 0." + echo + echo "## Attestation" + echo + echo "_Sign off (operator name, date) once step 8 is filled in:_" + echo + echo "Signed: TODO" + echo "Date: TODO" +} > "$RESULTS_FILE" + +# -------- final exit code -------- + +FAIL_COUNT=0 +for step in 1 2 3 4.1 4.2 4.3a 4.3b 4.3c 5 6 7; do + if [ "${step_status[$step]:-}" = "FAIL" ]; then + FAIL_COUNT=$((FAIL_COUNT+1)) + fi +done + +if [ "$FAIL_COUNT" -gt 0 ]; then + log "FAIL: $FAIL_COUNT technical step(s) did not pass. Results: $RESULTS_FILE" + exit 1 +fi + +log "PASS: all technical steps passed (1–7). Results: $RESULTS_FILE" +log " Operator must still fill in step 8 (improvisation tally) before declaring the gate green." From 70bd21813b4d2c1354738dfe0b6740347a376bc5 Mon Sep 17 00:00:00 2001 From: John Morrissey <544926+tachyon-beep@users.noreply.github.com> Date: Sun, 24 May 2026 13:02:19 +1000 Subject: [PATCH 2/9] fix(v1.0): fs2 advisory lock for clarion analyze (STO-01) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes clarion-c0caf61ab6 — STO-01 in docs/implementation/v1.0-tag-cut/ gap-register.md. Two concurrent `clarion analyze` invocations on the same project had no cross-process mutex: each opened its own writer- actor, each called `BeginRun` (INSERT a fresh `runs` row in `status='running'`), and each raced on entity/edge inserts under SQLite WAL. The in-process `ActorState::current_run` guard in clarion-storage prevented a single writer from issuing two BeginRuns; it did nothing across processes. Phase 3 clustering's fresh Connection::open at analyze.rs:687 then read through whatever entities the other process had committed. Fix: - Add `fs2 = "0.4"` to `[workspace.dependencies]` and to clarion-cli. - New module `crates/clarion-cli/src/analyze_lock.rs` exposes `acquire_analyze_lock(clarion_dir)` returning an `AnalyzeLockGuard` RAII handle. Opens (or creates) `.clarion/clarion.lock` and calls `FileExt::try_lock_exclusive()`. On contention returns an anyhow::Error naming the lock path so the operator can identify the conflict. - Wire the acquire at `analyze::run_with_options` immediately after the `.clarion/` existence check. Drop order is load-bearing: the guard outlives the writer-actor's `handle.await` at the bottom of the function — documented on `AnalyzeLockGuard`. Scope is analyze-only. The gap-register prescription parenthetically includes "serve write paths", but serve's LLM-summary writer only activates with --enable-llm-summary and emits per-MCP-request inserts that SQLite WAL serialises against analyze's writes; analyze+serve coexistence is a different concern outside STO-01's "second clarion analyze corrupts run state" scope. Capturing the analyze-only mutex here without also locking serve preserves the standard dogfood loop (re-analyze while MCP is running) and avoids over-fitting the fix. Tests: - analyze_lock::tests::second_acquire_fails_while_first_held - analyze_lock::tests::second_acquire_succeeds_after_first_drops - analyze_lock::tests::missing_clarion_dir_errors Gap-register note: STO-01 cited `run_lifecycle.rs:19-25` as evidence of the "UPDATE runs SET status='failed' WHERE status='running'" hazard. That SQL is actually in clarion-storage's `cleanup_after_channel_close` (writer.rs:262-281) and fires only on writer-channel close, not at analyze startup. The underlying claim (no cross-process lock) is correct; the citation needs a future gap-register pass. CI floor green: cargo fmt --check (new file clean — pre-existing http_read.rs drift filed as clarion-obs-24940665f3), cargo clippy --workspace --all-targets --all-features -D warnings, cargo build --workspace --bins, cargo nextest run --workspace --all-features (508 passed, 2 skipped), RUSTDOCFLAGS=-D warnings cargo doc, cargo deny check, tests/e2e/sprint_1_walking_skeleton.sh. --- Cargo.lock | 33 ++++++ Cargo.toml | 1 + crates/clarion-cli/Cargo.toml | 1 + crates/clarion-cli/src/analyze.rs | 6 + crates/clarion-cli/src/analyze_lock.rs | 148 +++++++++++++++++++++++++ crates/clarion-cli/src/main.rs | 1 + 6 files changed, 190 insertions(+) create mode 100644 crates/clarion-cli/src/analyze_lock.rs diff --git a/Cargo.lock b/Cargo.lock index dbc817e1..e21c81f2 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -325,6 +325,7 @@ dependencies = [ "clarion-scanner", "clarion-storage", "dotenvy", + "fs2", "ignore", "rusqlite", "serde", @@ -648,6 +649,16 @@ dependencies = [ "percent-encoding", ] +[[package]] +name = "fs2" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9564fc758e15025b46aa6643b1b77d047d1a56a1aea6e01002ac0c7026876213" +dependencies = [ + "libc", + "winapi", +] + [[package]] name = "futures-channel" version = "0.3.32" @@ -2457,6 +2468,22 @@ dependencies = [ "winsafe", ] +[[package]] +name = "winapi" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" +dependencies = [ + "winapi-i686-pc-windows-gnu", + "winapi-x86_64-pc-windows-gnu", +] + +[[package]] +name = "winapi-i686-pc-windows-gnu" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" + [[package]] name = "winapi-util" version = "0.1.11" @@ -2466,6 +2493,12 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "winapi-x86_64-pc-windows-gnu" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" + [[package]] name = "windows-link" version = "0.2.1" diff --git a/Cargo.toml b/Cargo.toml index d4b7eaa5..50054a20 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -37,6 +37,7 @@ blake3 = "1.8.5" clap = { version = "4", features = ["derive"] } deadpool-sqlite = { version = "0.8", features = ["rt_tokio_1"] } dotenvy = "0.15" +fs2 = "0.4" ignore = "0.4" rusqlite = { version = "0.31", features = ["bundled"] } reqwest = { version = "0.12", default-features = false, features = ["blocking", "json", "rustls-tls-native-roots"] } diff --git a/crates/clarion-cli/Cargo.toml b/crates/clarion-cli/Cargo.toml index 813c8e88..cc0e6419 100644 --- a/crates/clarion-cli/Cargo.toml +++ b/crates/clarion-cli/Cargo.toml @@ -23,6 +23,7 @@ clarion-mcp = { path = "../clarion-mcp", version = "1.0.0" } clarion-scanner = { path = "../clarion-scanner", version = "1.0.0" } clarion-storage = { path = "../clarion-storage", version = "1.0.0" } dotenvy.workspace = true +fs2.workspace = true ignore.workspace = true rusqlite.workspace = true serde.workspace = true diff --git a/crates/clarion-cli/src/analyze.rs b/crates/clarion-cli/src/analyze.rs index 5656c791..288e8ca3 100644 --- a/crates/clarion-cli/src/analyze.rs +++ b/crates/clarion-cli/src/analyze.rs @@ -81,6 +81,12 @@ pub(crate) async fn run_with_options(project_path: PathBuf, options: AnalyzeOpti ); } let db_path = clarion_dir.join("clarion.db"); + + // Cross-process advisory lock (STO-01). Must outlive the writer-actor's + // `handle.await` at the bottom of this function — see the drop-order + // note on `AnalyzeLockGuard`. Drop on function exit releases the lock. + let _analyze_lock = crate::analyze_lock::acquire_analyze_lock(&clarion_dir)?; + let analyze_config = AnalyzeConfig::load(&project_root, options.config_path.as_deref())?; let analyze_config_json = analyze_config.to_json_string()?; diff --git a/crates/clarion-cli/src/analyze_lock.rs b/crates/clarion-cli/src/analyze_lock.rs new file mode 100644 index 00000000..9108768d --- /dev/null +++ b/crates/clarion-cli/src/analyze_lock.rs @@ -0,0 +1,148 @@ +//! Cross-process advisory lock for `clarion analyze`. +//! +//! Two concurrent `clarion analyze` processes against the same project +//! corrupt the run-attribution graph: each opens its own writer-actor, +//! each calls `BeginRun` (insert a fresh `runs` row in `status='running'`), +//! and each races on entity/edge inserts under `SQLite` WAL. The in-process +//! `ActorState::current_run` guard (clarion-storage `writer.rs`) prevents +//! a single writer from issuing two `BeginRun`s; it does nothing across +//! processes. +//! +//! This module acquires an exclusive `fs2`-advisory lock on a dedicated +//! sentinel file `.clarion/clarion.lock` for the duration of the analyze +//! run. The lock file is separate from `clarion.db` so `SQLite`'s own +//! locking (per-connection, transaction-scoped) is independent. The +//! guard's `Drop` releases the OS-level lock. + +use std::fs::{File, OpenOptions}; +use std::path::Path; + +use anyhow::{Context, Result, bail}; +use fs2::FileExt; + +const LOCK_FILE_NAME: &str = "clarion.lock"; + +/// RAII guard holding the analyze lock. Drop releases the OS lock. +/// +/// **Drop order is load-bearing.** The guard must outlive the writer-actor's +/// `JoinHandle::await` in `analyze::run_with_options`; otherwise a second +/// `clarion analyze` can grab the lock while writer-actor 1's final +/// transaction is still landing through WAL. `fs2`'s `File` impl unlocks +/// on file close, so dropping the `File` releases the OS lock; we rely on +/// Drop rather than an explicit unlock so panic and happy paths behave +/// identically. +#[must_use = "Drop releases the analyze lock — bind to a named variable"] +#[derive(Debug)] +pub(crate) struct AnalyzeLockGuard { + _file: File, +} + +/// Acquire an exclusive cross-process lock on `/clarion.lock`. +/// +/// `clarion_dir` is the `.clarion/` directory inside the project root. The +/// lock file is created on first use (0-byte sentinel) and kept across +/// runs. The returned guard holds the lock for its lifetime. +/// +/// # Errors +/// +/// - The lock file cannot be opened (missing `.clarion/` directory, +/// permission denied, filesystem read-only). +/// - Another `clarion analyze` process already holds the lock. Returns +/// an error containing the lock-file path so the operator can identify +/// the conflict. +pub(crate) fn acquire_analyze_lock(clarion_dir: &Path) -> Result { + let lock_path = clarion_dir.join(LOCK_FILE_NAME); + let file = OpenOptions::new() + .read(true) + .write(true) + .create(true) + .truncate(false) + .open(&lock_path) + .with_context(|| format!("open analyze lock file {}", lock_path.display()))?; + + match file.try_lock_exclusive() { + Ok(()) => Ok(AnalyzeLockGuard { _file: file }), + Err(err) => { + // fs2 returns ErrorKind::WouldBlock when another process holds + // the lock; anything else is a real IO failure (e.g. NFS + // without lockd). Surface both with the path so operators can + // identify the conflict. + let kind = err.kind(); + if kind == std::io::ErrorKind::WouldBlock { + bail!( + "another `clarion analyze` is already in progress against this project \ + (lock held on {}). Wait for it to finish, or remove the lock file if \ + no other process is running.", + lock_path.display() + ); + } + Err(err).with_context(|| { + format!( + "acquire exclusive lock on {} (filesystem may not support advisory locks)", + lock_path.display() + ) + }) + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Two concurrent `acquire_analyze_lock` calls on the same `.clarion/` + /// directory must fail the second call. This is the core STO-01 + /// invariant: a second analyze cannot start while the first holds + /// the writer. + #[test] + fn second_acquire_fails_while_first_held() { + let tmp = tempfile::tempdir().unwrap(); + let clarion_dir = tmp.path(); + + let first = acquire_analyze_lock(clarion_dir).expect("first acquire"); + assert!( + clarion_dir.join(LOCK_FILE_NAME).exists(), + "lock file created on first acquire" + ); + + let err = acquire_analyze_lock(clarion_dir) + .expect_err("second acquire must fail while first guard is held"); + let msg = format!("{err}"); + assert!( + msg.contains("another `clarion analyze`"), + "error must name the conflict explicitly: {msg}" + ); + drop(first); + } + + /// Releasing the first lock (dropping the guard) must let the second + /// acquire succeed. Guards the "we forgot to unlock on Drop" bug. + #[test] + fn second_acquire_succeeds_after_first_drops() { + let tmp = tempfile::tempdir().unwrap(); + let clarion_dir = tmp.path(); + + { + let first = acquire_analyze_lock(clarion_dir).expect("first acquire"); + drop(first); + } // lock released on drop + + let second = acquire_analyze_lock(clarion_dir) + .expect("second acquire must succeed after first drops"); + drop(second); + } + + /// Missing `.clarion/` directory must surface as an IO error, not a + /// `WouldBlock` masquerade. (Operator may have skipped `clarion install`.) + #[test] + fn missing_clarion_dir_errors() { + let tmp = tempfile::tempdir().unwrap(); + let nonexistent = tmp.path().join("missing-clarion-dir"); + let err = acquire_analyze_lock(&nonexistent).expect_err("missing dir must error"); + let msg = format!("{err:#}"); + assert!( + msg.contains("open analyze lock file"), + "error must mention lock file open path: {msg}" + ); + } +} diff --git a/crates/clarion-cli/src/main.rs b/crates/clarion-cli/src/main.rs index 1f5999ec..e48d5113 100644 --- a/crates/clarion-cli/src/main.rs +++ b/crates/clarion-cli/src/main.rs @@ -1,4 +1,5 @@ mod analyze; +mod analyze_lock; mod cli; mod clustering; mod config; From 071ef6f8a4188bab3fe4440ea92e138143bd4042 Mon Sep 17 00:00:00 2001 From: John Morrissey <544926+tachyon-beep@users.noreply.github.com> Date: Sun, 24 May 2026 13:52:33 +1000 Subject: [PATCH 3/9] test(plugin): hard-fail when pyright-langserver is missing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit pyright==1.1.409 is a hard runtime dependency of clarion-plugin-python (pyproject.toml `dependencies`), so `pyright-langserver` is expected on PATH or in the active virtualenv after `pip install -e plugins/python`. The shared `pyright_langserver` fixture in test_pyright_session.py and test_extractor.py previously called `pytest.skip(...)` when the binary was missing. In CI this is a silent-skip: the entire pyright-dependent test class disappears, the job stays green, and a broken venv install ships unnoticed. Converted both fixtures to `pytest.fail(...)` with a diagnostic message naming the install command — a missing executable now hard-fails the suite. Closes clarion-dc99115f5d (v1.1 security-hardening item 3 in docs/implementation/v1.0-tag-cut/gap-register.md). --- plugins/python/tests/test_extractor.py | 7 ++++++- plugins/python/tests/test_pyright_session.py | 7 ++++++- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/plugins/python/tests/test_extractor.py b/plugins/python/tests/test_extractor.py index 76a9bfa4..bd02ad39 100644 --- a/plugins/python/tests/test_extractor.py +++ b/plugins/python/tests/test_extractor.py @@ -84,7 +84,12 @@ def pyright_langserver() -> str: return str(venv_candidate) resolved = shutil.which("pyright-langserver") if resolved is None: - pytest.skip("pyright-langserver is not installed") + pytest.fail( + "pyright-langserver not found on PATH or in the active virtualenv. " + "It is a hard runtime dependency of clarion-plugin-python " + "(pyproject.toml `dependencies`); a missing executable means the " + "install is broken. Skipping these tests would mask a regression.", + ) return resolved diff --git a/plugins/python/tests/test_pyright_session.py b/plugins/python/tests/test_pyright_session.py index 0dc8da4d..61787a9c 100644 --- a/plugins/python/tests/test_pyright_session.py +++ b/plugins/python/tests/test_pyright_session.py @@ -42,7 +42,12 @@ def pyright_langserver() -> str: return str(venv_candidate) resolved = shutil.which("pyright-langserver") if resolved is None: - pytest.skip("pyright-langserver is not installed") + pytest.fail( + "pyright-langserver not found on PATH or in the active virtualenv. " + "It is a hard runtime dependency of clarion-plugin-python " + "(pyproject.toml `dependencies`); a missing executable means the " + "install is broken. Skipping these tests would mask a regression.", + ) return resolved From f5a65001f544ce94b7a177aaa449344a222dcb54 Mon Sep 17 00:00:00 2001 From: John Morrissey <544926+tachyon-beep@users.noreply.github.com> Date: Sun, 24 May 2026 14:01:37 +1000 Subject: [PATCH 4/9] fix(storage): add summary_cache.entity_id FK (V11-STO-03) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `summary_cache.entity_id` had no FK while its sibling caches (`inferred_edge_cache.caller_entity_id`, `entity_unresolved_call_sites.caller_entity_id`) declared `REFERENCES entities(id) ON DELETE CASCADE`. Confirmed bug per gap register, not intentional asymmetry: production writes via `upsert_summary_cache` already pass through `entity_by_id`-loaded entities (clarion-mcp/src/lib.rs:1186), and `PRAGMA foreign_keys = ON` is applied on both write and read connections, so the constraint will be enforced without changing any production path. Edit applied in place in `0001_initial_schema.sql` per ADR-024 — no v1.0 tag exists yet, so the retirement trigger ("first external operator pulls a published Clarion build and produces a schema_migrations ledger") has not fired. The gap register's "requires table-rebuild migration" framing assumed a post-tag fix; landing it now skips that ceremony. Two tests updated to satisfy the new FK: - `migration_0001_extends_summary_cache_for_mcp_staleness_tracking` (schema_apply.rs) now seeds the parent entity row, mirroring the pattern the neighbouring inferred_edge_cache test already uses. - `summary_cache_writer_commands_do_not_require_active_analyze_run` (writer_actor.rs) seeds the parent entity via direct SQL (new `seed_entity_row` helper) — bypassing the writer's `BeginRun → InsertEntity` protocol preserves the test's original intent of verifying that summary_cache writer commands work without an active analyze run. Re-labels the issue release:v1.1 → release:v1.0 since the fix shipped in v1.0 scope, and strikes STO-03 from the v1.1 storage-hardening list in the gap register. Closes clarion-4c49ccf5d0. --- .../migrations/0001_initial_schema.sql | 12 +++++-- crates/clarion-storage/tests/schema_apply.rs | 19 ++++++++++++ crates/clarion-storage/tests/writer_actor.rs | 31 +++++++++++++++++++ .../v1.0-tag-cut/gap-register.md | 13 ++++++-- 4 files changed, 69 insertions(+), 6 deletions(-) diff --git a/crates/clarion-storage/migrations/0001_initial_schema.sql b/crates/clarion-storage/migrations/0001_initial_schema.sql index e4fb6c23..bef609a8 100644 --- a/crates/clarion-storage/migrations/0001_initial_schema.sql +++ b/crates/clarion-storage/migrations/0001_initial_schema.sql @@ -11,9 +11,10 @@ -- as long as no external operator has produced a `.clarion/clarion.db` from -- a published Clarion build. The retirement trigger names exactly that -- condition; once it fires, all schema changes stack as 0002_*.sql etc. --- The 2026-05-03 edits (guidance vocabulary rename per ADR-024) and the +-- The 2026-05-03 edits (guidance vocabulary rename per ADR-024), the -- 2026-05-18 edits (CHECK constraints on closed-vocabulary TEXT columns per --- ADR-031) were both applied under this policy. +-- ADR-031), and the 2026-05-24 edit (summary_cache.entity_id FK per +-- V11-STO-03) were all applied under this policy. -- ============================================================================ BEGIN; @@ -140,7 +141,12 @@ CREATE INDEX ix_findings_status ON findings(status); -- Summary cache CREATE TABLE summary_cache ( - entity_id TEXT NOT NULL, + -- FK matches the sibling caches (inferred_edge_cache.caller_entity_id, + -- entity_unresolved_call_sites.caller_entity_id). Added in-place per + -- ADR-024 on 2026-05-24 (V11-STO-03 closure) — the prior absence was a + -- bug, not intentional asymmetry. ON DELETE CASCADE so removing an + -- entity (re-analyze, rename) clears its cached summaries. + entity_id TEXT NOT NULL REFERENCES entities(id) ON DELETE CASCADE, content_hash TEXT NOT NULL, prompt_template_id TEXT NOT NULL, model_tier TEXT NOT NULL, diff --git a/crates/clarion-storage/tests/schema_apply.rs b/crates/clarion-storage/tests/schema_apply.rs index d29f0412..1c2969ee 100644 --- a/crates/clarion-storage/tests/schema_apply.rs +++ b/crates/clarion-storage/tests/schema_apply.rs @@ -209,6 +209,25 @@ fn migration_0001_extends_summary_cache_for_mcp_staleness_tracking() { ); } + // summary_cache.entity_id has an FK to entities(id) per V11-STO-03; + // seed the parent row first so the INSERT below isn't FK-rejected. + conn.execute( + "INSERT INTO entities (id, plugin_id, kind, name, short_name, properties, \ + created_at, updated_at) \ + VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)", + params![ + "python:function:demo.hello", + "python", + "function", + "demo.hello", + "hello", + "{}", + "2026-05-17T00:00:00.000Z", + "2026-05-17T00:00:00.000Z", + ], + ) + .expect("seed summary_cache parent entity"); + conn.execute( "INSERT INTO summary_cache ( \ entity_id, content_hash, prompt_template_id, model_tier, \ diff --git a/crates/clarion-storage/tests/writer_actor.rs b/crates/clarion-storage/tests/writer_actor.rs index 6ef58a53..7d3310da 100644 --- a/crates/clarion-storage/tests/writer_actor.rs +++ b/crates/clarion-storage/tests/writer_actor.rs @@ -22,6 +22,31 @@ fn prepared_db(dir: &tempfile::TempDir) -> std::path::PathBuf { path } +/// Direct-SQL entity seed for tests that need an `entities` row but must +/// bypass the writer's `BeginRun → InsertEntity` protocol (e.g. tests +/// verifying that non-analyze writer commands work without an active run). +fn seed_entity_row(path: &std::path::Path, id: &str) { + let conn = Connection::open(path).unwrap(); + conn.execute( + "INSERT INTO entities ( \ + id, plugin_id, kind, name, short_name, properties, \ + content_hash, created_at, updated_at \ + ) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)", + rusqlite::params![ + id, + "python", + "function", + id, + id.rsplit('.').next().unwrap_or(id), + "{}", + format!("hash-{id}"), + now_iso(), + now_iso(), + ], + ) + .expect("seed entity row"); +} + fn now_iso() -> String { "2026-04-18T00:00:00.000Z".to_owned() } @@ -258,6 +283,12 @@ async fn send( async fn summary_cache_writer_commands_do_not_require_active_analyze_run() { let dir = tempfile::tempdir().unwrap(); let path = prepared_db(&dir); + // summary_cache.entity_id has an FK to entities(id) (V11-STO-03). Seed the + // referenced entity directly so the FK is satisfied without going through + // the writer's BeginRun → InsertEntity protocol — the whole point of this + // test is that summary_cache writer commands work *without* an active + // analyze run. + seed_entity_row(&path, "python:function:demo.hello"); let (writer, handle) = Writer::spawn(path.clone(), 50, 256).unwrap(); let tx = writer.sender(); diff --git a/docs/implementation/v1.0-tag-cut/gap-register.md b/docs/implementation/v1.0-tag-cut/gap-register.md index d62d35fb..c9e668ac 100644 --- a/docs/implementation/v1.0-tag-cut/gap-register.md +++ b/docs/implementation/v1.0-tag-cut/gap-register.md @@ -509,8 +509,12 @@ Filed as Filigree issues with `release:v1.1` label so they don't get lost. **Storage hardening (deep-dive-db):** 1. `runs.owner_pid` + `heartbeat_at` columns and refined recovery WHERE-clause. 2. `clarion db backup` subcommand via `rusqlite::backup::Backup`. -3. `summary_cache.entity_id` FK via table-rebuild migration (confirmed - bug — not intentional asymmetry). +3. ~~`summary_cache.entity_id` FK via table-rebuild migration (confirmed + bug — not intentional asymmetry).~~ **Closed in v1.0 (2026-05-24)**: + landed in-place in migration `0001_initial_schema.sql` under + ADR-024's pre-tag edit-in-place policy (no published build exists + yet, so no `schema_migrations` ledger needs the table-rebuild + ceremony). See clarion-4c49ccf5d0. 4. `briefing_blocked` generated column + partial index (federation read-API hot path). 5. `BEGIN IMMEDIATE` + `SQLITE_BUSY` retry helper across the writer. @@ -522,7 +526,10 @@ Filed as Filigree issues with `release:v1.1` label so they don't get lost. 1. HMAC replay protection (timestamp + nonce window). Already called out in ADR-034 forward-work. 2. Mandatory authentication on loopback binds (doctrine change). -3. Make pyright-langserver-missing a CI hard-fail, not a silent skip. +3. ~~Make pyright-langserver-missing a CI hard-fail, not a silent skip.~~ + **Closed in v1.0 (2026-05-24)**: pyright-langserver fixture in + `test_pyright_session.py` and `test_extractor.py` now calls + `pytest.fail()` instead of `pytest.skip()`. See clarion-dc99115f5d. **CI / release hardening (deep-dive-pipeline):** 1. Refactor `release.yml verify` into a reusable workflow shared with From a4c4279dd4cdf4ff8d7d7609d4a65e4c4f025cc7 Mon Sep 17 00:00:00 2001 From: John Morrissey <544926+tachyon-beep@users.noreply.github.com> Date: Fri, 29 May 2026 03:51:00 +1000 Subject: [PATCH 5/9] ci(v1.0): close release-path tag-cut gaps (TEST-01/02, CI-01/03/04, GOV-02) - Wire tests/e2e/sprint_2_mcp_surface.sh + phase3_subsystems.sh into the ci.yml walking-skeleton job and the release.yml verify job (TEST-01, TEST-02). - release.yml verify: fetch-depth 0 + assert tagged commit is an ancestor of origin/main, closing the tag-lineage supply-chain bypass (CI-01). - release-subjects: include clarion_plugin_python*.tar.gz in the SLSA provenance subjects so the Python sdist is attested (CI-03). - New verify-published-release job: re-download public assets, recompute SHA256 against published sidecars, cosign verify-blob against Rekor (CI-04). - check-github-release-governance.py: assert an active ruleset targets refs/tags/v*; self-test covers present + absent fixtures (GOV-02 script half). Local: both new e2e scripts exit 0; YAML valid; governance --self-test passes. CI-green + workflow_dispatch dry-run remain push-verification residue. Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/ci.yml | 6 ++ .github/workflows/release.yml | 82 ++++++++++++++++++- scripts/check-github-release-governance.py | 95 +++++++++++++++++++++- 3 files changed, 181 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c6a4df71..d91fc5cf 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -153,3 +153,9 @@ jobs: - name: run WP5 secret scanner smoke run: CARGO_BUILD=0 bash tests/e2e/wp5_secret_scan.sh + + - name: Sprint 2 MCP surface + run: CARGO_BUILD=0 bash tests/e2e/sprint_2_mcp_surface.sh + + - name: Phase 3 subsystems + run: CARGO_BUILD=0 bash tests/e2e/phase3_subsystems.sh diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index c0c2886e..04b66a40 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -31,6 +31,14 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 + with: + fetch-depth: 0 + + - name: Assert tagged commit is on main + run: | + git fetch origin main + git merge-base --is-ancestor "$GITHUB_SHA" origin/main || \ + { echo "::error::tag does not point to a commit on main"; exit 1; } - uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 with: @@ -117,6 +125,12 @@ jobs: - name: WP5 secret scanner smoke run: CARGO_BUILD=0 bash tests/e2e/wp5_secret_scan.sh + - name: Sprint 2 MCP surface + run: CARGO_BUILD=0 bash tests/e2e/sprint_2_mcp_surface.sh + + - name: Phase 3 subsystems + run: CARGO_BUILD=0 bash tests/e2e/phase3_subsystems.sh + release-governance: name: GitHub release governance runs-on: ubuntu-latest @@ -220,7 +234,7 @@ jobs: while IFS= read -r -d '' asset; do digest=$(sha256sum "$asset" | cut -d' ' -f1) printf '%s %s\n' "$digest" "$(basename "$asset")" - done < <(find artifacts -name 'clarion-*.tar.gz' -print0 | sort -z) > rust-subjects.txt + done < <(find artifacts \( -name 'clarion-*.tar.gz' -o -name 'clarion_plugin_python*.tar.gz' \) -print0 | sort -z) > rust-subjects.txt test -s rust-subjects.txt echo "rust-subjects=$(base64 -w0 rust-subjects.txt)" >> "$GITHUB_OUTPUT" @@ -407,3 +421,69 @@ jobs: upload-assets: true upload-tag-name: ${{ github.ref_name }} provenance-name: clarion-rust-binaries.intoto.jsonl + + verify-published-release: + name: Verify published release assets + needs: [release] + runs-on: ubuntu-latest + # Post-publish gate: prove the assets actually served from the public + # GitHub Release match the signed local copies. The in-`release`-job + # `cosign verify-blob` only proves signing worked on the build runner; it + # does not prove the published download matches. This job downloads the + # public assets, recomputes SHA256 against the published `.sha256` + # sidecars, and re-verifies the cosign signatures against the public Rekor + # entries. + if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') + permissions: + contents: read + steps: + - name: install cosign + uses: sigstore/cosign-installer@398d4b0eeef1380460a10c8013a76f728fb906ac + + - name: download published assets + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + mkdir -p published + cd published + gh release download "${GITHUB_REF_NAME}" \ + --repo "${GITHUB_REPOSITORY}" \ + --pattern '*.tar.gz' \ + --pattern '*.tar.gz.sha256' \ + --pattern '*.tar.gz.sig' \ + --pattern '*.tar.gz.pem' + ls -la + + - name: recompute SHA256 against published checksums + run: | + set -euo pipefail + cd published + shopt -s nullglob + checksums=(*.tar.gz.sha256) + if [ ${#checksums[@]} -eq 0 ]; then + echo "::error::no .sha256 sidecars found in the published release" >&2 + exit 1 + fi + for sum in "${checksums[@]}"; do + sha256sum -c "$sum" + done + + - name: cosign verify published archives against Rekor + run: | + set -euo pipefail + cd published + shopt -s nullglob + archives=(*.tar.gz) + if [ ${#archives[@]} -eq 0 ]; then + echo "::error::no .tar.gz archives found in the published release" >&2 + exit 1 + fi + for asset in "${archives[@]}"; do + cosign verify-blob \ + --certificate "$asset.pem" \ + --signature "$asset.sig" \ + --certificate-identity-regexp "https://github.com/${GITHUB_REPOSITORY}/.github/workflows/release.yml@refs/tags/v.*" \ + --certificate-oidc-issuer https://token.actions.githubusercontent.com \ + "$asset" + done diff --git a/scripts/check-github-release-governance.py b/scripts/check-github-release-governance.py index 1bc32769..4a7fb936 100755 --- a/scripts/check-github-release-governance.py +++ b/scripts/check-github-release-governance.py @@ -203,6 +203,44 @@ def ruleset_targets_branch(ruleset: dict[str, Any], branch: str) -> bool: ) +def ruleset_targets_tag(ruleset: dict[str, Any], tag_pattern: str) -> bool: + """True when an active tag ruleset's include patterns cover ``tag_pattern``. + + GitHub tag rulesets carry ``target == "tag"`` and condition their + ``ref_name`` on ``refs/tags/`` globs. We accept a ruleset if it either + has no narrowing ``include`` list (covers all tags) or names an include + pattern that the desired ``tag_pattern`` (e.g. ``refs/tags/v*``) satisfies. + """ + if ruleset.get("target") != "tag": + return False + + conditions = ruleset.get("conditions") + if not isinstance(conditions, dict): + return True + ref_name = conditions.get("ref_name") + if not isinstance(ref_name, dict): + return True + + include = ref_name.get("include") + if not isinstance(include, list) or not include: + return True + # The desired pattern is itself a glob (refs/tags/v*); treat an include + # entry as covering it when the strings match, when the include is the + # all-tags wildcard, or when the include glob matches the literal prefix + # of the desired pattern. + desired = tag_pattern.removeprefix("refs/tags/") + for pattern in include: + if not isinstance(pattern, str): + continue + normalized = pattern.removeprefix("refs/tags/") + if pattern in {tag_pattern, "~ALL"} or normalized in {desired, "*"}: + return True + # An include like `v*` matches the concrete tag families we cut. + if fnmatchcase("v1.0.0", normalized) or fnmatchcase(desired, normalized): + return True + return False + + def branch_protection_status_checks(protection: dict[str, Any]) -> set[str]: required = protection.get("required_status_checks") if not isinstance(required, dict): @@ -322,6 +360,31 @@ def check_governance( "release PR path" ) + # Tag protection (GOV-02): an active ruleset must target `refs/tags/v*` so + # that a tag-push cannot point a release tag at an arbitrary commit. The + # legacy `tags/protection` endpoint is deprecated; a tag ruleset is the + # modern mechanism. We assert on the ruleset summaries (no detail fetch is + # needed for a presence check). + tag_pattern = "refs/tags/v*" + active_tag_rulesets = [ + item.get("name", "") + for item in rule_summaries + if item.get("enforcement") == "active" and ruleset_targets_tag(item, tag_pattern) + ] + if active_tag_rulesets: + names = ", ".join( + name if isinstance(name, str) else "" + for name in active_tag_rulesets + ) + notes.append( + f"{repository}: active tag ruleset(s) protect {tag_pattern} ({names})" + ) + else: + failures.append( + f"{repository}: no active repository ruleset protects {tag_pattern}; " + "a tag-push can point a release tag at an arbitrary commit" + ) + permissions = actions_permissions(request_json, repository) if permissions.get("enabled") is not True: failures.append("GitHub Actions are not enabled for the repository") @@ -599,7 +662,14 @@ def fake_get(path: str) -> ApiResponse: "target": "branch", "enforcement": "active", "conditions": {"ref_name": {"include": ["refs/heads/main"], "exclude": []}}, - } + }, + { + "id": 50, + "name": "release tag gate", + "target": "tag", + "enforcement": "active", + "conditions": {"ref_name": {"include": ["refs/tags/v*"], "exclude": []}}, + }, ], ) responses["/repos/acme/clarion/rulesets/43"] = ApiResponse( @@ -627,6 +697,29 @@ def fake_get(path: str) -> ApiResponse: ) notes = check_governance(fake_get, "acme/clarion", "main") assert any("active ruleset 'main release gate'" in note for note in notes) + assert any("active tag ruleset(s) protect refs/tags/v*" in note for note in notes) + + # Tag protection absent (GOV-02): a fully-protected branch but no tag + # ruleset must still fail — a tag-push could point a release tag at an + # arbitrary commit. + responses["/repos/acme/clarion/rulesets"] = ApiResponse( + 200, + [ + { + "id": 43, + "name": "main release gate", + "target": "branch", + "enforcement": "active", + "conditions": {"ref_name": {"include": ["refs/heads/main"], "exclude": []}}, + } + ], + ) + try: + check_governance(fake_get, "acme/clarion", "main") + except CheckError as exc: + assert "no active repository ruleset protects refs/tags/v*" in str(exc) + else: + raise AssertionError("missing tag ruleset fixture should fail") with tempfile.TemporaryDirectory() as tmp: root = Path(tmp) From b70b355b9b264b5fc46fe94a090069379d39bc2f Mon Sep 17 00:00:00 2001 From: John Morrissey <544926+tachyon-beep@users.noreply.github.com> Date: Fri, 29 May 2026 03:51:13 +1000 Subject: [PATCH 6/9] fix(v1.0): e2e integrity_check, operator trust/storage docs, fmt (STO-04/05, SEC-02/03, DOC-08) - sprint_1_walking_skeleton.sh: assert PRAGMA integrity_check == ok after analyze, failing the e2e on a torn/corrupt DB (STO-04a). - operations.md: backup procedure rationale (WAL pages live in -wal; naive cp during live analyze yields a torn copy) and same-process-restart caveat; clarion db backup + runs.owner_pid/heartbeat_at deferred to v1.1 (STO-04b/05). - secret-scanning.md + clarion-http-read-api.md: state the loopback-no-token trust assumption and that the startup banner now ships and logs at WARN; drop stale v0.1/v0.2 qualifiers (SEC-02 docs, SEC-03, DOC-08). - http_read.rs: rustfmt reflow only (HEAD failed cargo fmt --check after 8e5804b committed the SEC-02 banner un-formatted); no behavior change. Gates: fmt/clippy/build pass; nextest 508 passed/2 skipped; sprint_1 e2e exit 0. Co-Authored-By: Claude Opus 4.8 (1M context) --- crates/clarion-cli/src/http_read.rs | 17 +++++++++-------- docs/clarion/1.0/operations.md | 8 ++++++++ docs/operator/clarion-http-read-api.md | 6 ++++-- docs/operator/secret-scanning.md | 8 +++++--- tests/e2e/sprint_1_walking_skeleton.sh | 9 +++++++++ 5 files changed, 35 insertions(+), 13 deletions(-) diff --git a/crates/clarion-cli/src/http_read.rs b/crates/clarion-cli/src/http_read.rs index 8b2af458..8d8e855b 100644 --- a/crates/clarion-cli/src/http_read.rs +++ b/crates/clarion-cli/src/http_read.rs @@ -196,9 +196,8 @@ where // local request because both auth knobs are unset and the bind is // loopback. On a shared developer host or CI runner this means any // local process can read the (non-blocked) catalogue. - let warn_unauthenticated_loopback = config.is_loopback_bind() - && auth_token.is_none() - && identity_secret.is_none(); + let warn_unauthenticated_loopback = + config.is_loopback_bind() && auth_token.is_none() && identity_secret.is_none(); let (shutdown_tx, shutdown_rx) = oneshot::channel(); let (ready_tx, ready_rx) = std::sync::mpsc::channel::>(); let (failure_tx, failure_rx) = mpsc::channel(); @@ -1571,7 +1570,10 @@ mod tests { impl io::Write for CaptureWriter { fn write(&mut self, buf: &[u8]) -> io::Result { - self.buffer.lock().expect("capture lock").extend_from_slice(buf); + self.buffer + .lock() + .expect("capture lock") + .extend_from_slice(buf); Ok(buf.len()) } fn flush(&mut self) -> io::Result<()> { @@ -1616,10 +1618,9 @@ mod tests { token_env: "CLARION_LOOPBACK_NO_TOKEN_TEST_UNSET".to_owned(), identity_token_env: None, }; - let instance_id = crate::instance::parse_instance_id_for_test( - "00000000-0000-4000-8000-000000000002", - ) - .expect("parse synthetic instance id"); + let instance_id = + crate::instance::parse_instance_id_for_test("00000000-0000-4000-8000-000000000002") + .expect("parse synthetic instance id"); // Env lookup that returns None for every variable — emulates // the operator running `clarion serve` on loopback with no diff --git a/docs/clarion/1.0/operations.md b/docs/clarion/1.0/operations.md index 4079dab9..aed091d1 100644 --- a/docs/clarion/1.0/operations.md +++ b/docs/clarion/1.0/operations.md @@ -101,6 +101,14 @@ is no live `clarion db backup` subcommand at v1.0 (deferred to v1.1, §6). `TRUNCATE` mode is the strongest checkpoint — it flushes the WAL into `clarion.db` and resets `clarion.db-wal` to zero length. + **Why this step matters:** in WAL mode, committed pages live in + `clarion.db-wal` until a checkpoint folds them back into `clarion.db`. A + naive `cp .clarion/clarion.db backup.db` during (or shortly after) a live + `analyze` therefore captures a *torn* copy — the main database file is + missing the most recent committed transactions, which are still sitting in + the separate `-wal` file. Forcing a `TRUNCATE` checkpoint first guarantees + `clarion.db` is self-contained before the copy. + 4. **Copy `.clarion/` to the backup location** with any standard tool (`cp -a`, `rsync -a`, `tar`). All three of `clarion.db`, `clarion.db-wal`, and `clarion.db-shm` should be present in the copy; diff --git a/docs/operator/clarion-http-read-api.md b/docs/operator/clarion-http-read-api.md index 101dfbf1..a51ffc58 100644 --- a/docs/operator/clarion-http-read-api.md +++ b/docs/operator/clarion-http-read-api.md @@ -80,5 +80,7 @@ configuration shape is documented in the [Trust Model](#trust-model) section above. The Clarion `serve` startup banner emits a `[TRUST]` line warning when -loopback-no-token mode is active (forward-reference: the banner code is a -SEC-02 follow-up PR; the trust assumption itself is current as of v1.0). +loopback-no-token mode is active: `HTTP API serving on loopback without +authentication; any local process on this host can read the catalogue.` +The warning is logged at `WARN` level at startup whenever both auth knobs +are unset and the bind is loopback. diff --git a/docs/operator/secret-scanning.md b/docs/operator/secret-scanning.md index 94be9f1d..e9ddadd4 100644 --- a/docs/operator/secret-scanning.md +++ b/docs/operator/secret-scanning.md @@ -74,7 +74,7 @@ from entities where json_extract(properties, '$.briefing_blocked') is not null; ``` -Filigree integration for scanner findings is planned for v0.2. Until then, the local `findings` table is the authoritative scanner audit surface. +Filigree integration for scanner findings (WP9-B finding emission) is deferred to v1.1. Until then, the local `findings` table is the authoritative scanner audit surface. ## Limitations @@ -112,8 +112,10 @@ Multi-tenant operators MUST set `identity_token_env` (HMAC, preferred) or configuration shape. The Clarion `serve` startup banner emits a `[TRUST]` line warning when -loopback-no-token mode is active (forward-reference: SEC-02 banner code -ships in a follow-up PR; the trust assumption itself is current). +loopback-no-token mode is active: `HTTP API serving on loopback without +authentication; any local process on this host can read the catalogue.` +This warning is logged at `WARN` level at startup whenever both auth knobs +are unset and the bind is loopback. ## Pre-WP5 catalogue upgrade requirement diff --git a/tests/e2e/sprint_1_walking_skeleton.sh b/tests/e2e/sprint_1_walking_skeleton.sh index 3b4e3743..11a9690a 100755 --- a/tests/e2e/sprint_1_walking_skeleton.sh +++ b/tests/e2e/sprint_1_walking_skeleton.sh @@ -88,6 +88,15 @@ clarion install log "running: clarion analyze ." clarion analyze . +# ── 6b. Verify database integrity (STO-04) ─────────────────────────────────── +log "verifying database integrity via PRAGMA integrity_check ..." +INTEGRITY=$(sqlite3 "$DEMO_DIR/.clarion/clarion.db" "PRAGMA integrity_check;") +if [ "$INTEGRITY" != "ok" ]; then + log "integrity_check output:" + printf '%s\n' "$INTEGRITY" >&2 + fail "expected PRAGMA integrity_check to report exactly 'ok'; got $INTEGRITY" +fi + # ── 7. Verify entity via sqlite3 ───────────────────────────────────────────── log "verifying persisted entity via sqlite3 ..." RESULT=$(sqlite3 "$DEMO_DIR/.clarion/clarion.db" "select id, kind from entities order by id;") From 2d949e46a4a9fe2a02cacca435356a054a00737c Mon Sep 17 00:00:00 2001 From: John Morrissey <544926+tachyon-beep@users.noreply.github.com> Date: Fri, 29 May 2026 03:58:34 +1000 Subject: [PATCH 7/9] docs(v1.0): land v0.2-deferral scope annotations; align deferral vocab to release:v1.1 (DOC-06) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Commits the in-flight requirements.md/system-design.md deferral sweep that annotates every requirement narrowed out of the v1.0 MVP (per ADR-030 + Sprint 2 scope amendment §3/§4) with a deferred-scope callout. Per the DOC-06 reconciliation, the deferral vocabulary is aligned to the release-label scheme: the sweep's blockquote callouts (and the two capabilities-table rows) now say "v1.1" / "release:v1.1" instead of "v0.2", matching CHANGELOG/CLAUDE.md/loom.md. Deliberately NOT renamed (would create drift, not alignment): Accepted-ADR table summaries (ADR-003/015/016/017/018 — immutable, and the ADR files themselves say v0.2), the v0.1-path / v0.2-retirement design-ladder pairs, the Non-Goals roadmap, and Shuttle (v0.2+). Those pair v0.2 with v0.1 on the design-milestone axis, distinct from the v1.x release axis. Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 7 +++ docs/clarion/1.0/requirements.md | 94 ++++++++++++++++++++++++++----- docs/clarion/1.0/system-design.md | 10 +++- 3 files changed, 94 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 25b9acdb..b2fb60e5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -120,6 +120,13 @@ normative. Wardline-aware plugin only; Clarion core and non-Wardline-aware plugins are unaffected. *Retirement condition*: Wardline ships a stable runtime probe API. +- **Wardline state-file ingest deferred to a future release (tracked under + `release:v1.1`).** Only the `wardline.core.registry.REGISTRY` version-pin + probe ships in v1.0; the state-file readers for `wardline.yaml` + overlays, + `wardline.fingerprint.json`, `wardline.exceptions.json`, the + `wardline.sarif.baseline.json` translator baseline, and the three-scheme + identity-reconciliation oracle (REQ-INTEG-WARDLINE-02 through -06) all land + with WP9-B / WP10. - **Pre-WP5 catalogue upgrade requirement.** Briefing-blocked annotations are stored as a JSON property on file entities at v1.0 (v1.1 promotes the field to a typed column). A v1.0 binary opening a `.clarion/clarion.db` diff --git a/docs/clarion/1.0/requirements.md b/docs/clarion/1.0/requirements.md index 36b50d51..45aa31b7 100644 --- a/docs/clarion/1.0/requirements.md +++ b/docs/clarion/1.0/requirements.md @@ -195,6 +195,8 @@ Structured summaries produced for each entity at policy-defined levels. #### REQ-BRIEFING-01 — Structured, not prose +> **v1.0 ships a 4-field on-demand summary** (`purpose`, `behavior`, `relationships`, `risks`) per [ADR-030](../adr/ADR-030-on-demand-summary-scope.md) (WP6 narrowed to leaf-tier on-demand `summary(id)`). The 9-field `EntityBriefing` schema below — `maturity` / `maturity_reasoning` / `patterns` / `antipatterns` / `knowledge_basis` / `notes` plus the `CLA-INFRA-BRIEFING-INVALID` retry path — is the v1.1 target when the briefing pipeline (Phases 4–6) lands per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md). + Briefings follow a fixed schema (`purpose`, `maturity`, `maturity_reasoning`, `risks`, `patterns`, `antipatterns`, `relationships`, `knowledge_basis`, `notes`). Free-form prose is not an acceptable briefing output. **Rationale**: Principle 2 (exploration elimination) + Principle 4 (finding as fact exchange). LLM agents consume briefings as structured data to drive further navigation; prose requires re-parsing and is an order of magnitude worse for composability. @@ -203,6 +205,8 @@ Briefings follow a fixed schema (`purpose`, `maturity`, `maturity_reasoning`, `r #### REQ-BRIEFING-02 — Controlled vocabulary for `patterns` / `antipatterns` / risk tags +> **Deferred to v1.1** per [ADR-030](../adr/ADR-030-on-demand-summary-scope.md) + [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md). The `patterns` / `antipatterns` fields belong to the 9-field `EntityBriefing` schema deferred in [REQ-BRIEFING-01](#req-briefing-01--structured-not-prose); the `CLA-FACT-VOCABULARY-CANDIDATE` rule and adversarial-docstring containment land with that pipeline. + `patterns` and `antipatterns` fields draw from a controlled vocabulary (core base set + plugin extensions). Novel tags proposed by the LLM are accepted once but logged as `CLA-FACT-VOCABULARY-CANDIDATE` for human review; they do not silently promote into future prompts. **Rationale**: An unconstrained vocabulary is a prompt-injection vector (see `NFR-SEC-03`) and drifts across runs, undermining comparability. A controlled vocabulary gives LLMs a fixed palette while preserving extensibility for genuinely new patterns. @@ -219,6 +223,8 @@ Generated briefings are cached by `(entity_id, content_hash, prompt_template_id, #### REQ-BRIEFING-04 — `knowledge_basis` field per briefing +> **Deferred to v1.1** per [ADR-030](../adr/ADR-030-on-demand-summary-scope.md) + [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md). `knowledge_basis` depends on the guidance system ([REQ-GUIDANCE-*](#guidance-system-req-guidance-), WP7 deferred) for the guidance-authored path and on Filigree triage-state feedback ([REQ-INTEG-FILIGREE-02](#req-integ-filigree-02--observation-emission-http-preferred-mcp-fallback), WP9-B deferred) for the acknowledged-finding path. Both inputs are v1.1. + Every briefing carries `knowledge_basis: static_only | runtime_informed | human_verified`. Default is `static_only`; promotion to `human_verified` requires (a) a guidance sheet authored or reviewed against the entity in the last 90 days OR (b) a finding with `status ∈ {suppressed, acknowledged}` carrying a non-empty reason. **Rationale**: Agents consuming briefings need to calibrate how much to trust the `risks` / `patterns` claims. A briefing derived purely from static analysis + LLM synthesis should be treated as a hypothesis; one validated by human curation (guidance or triage) earns more trust. @@ -227,6 +233,8 @@ Every briefing carries `knowledge_basis: static_only | runtime_informed | human_ #### REQ-BRIEFING-05 — Triage-state feedback from Filigree +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B finding emission + WP7 guidance both deferred). Triage feedback requires Clarion-Filigree finding-state plumbing that does not ship in v1.0; the guidance-fingerprint side is moot until [REQ-GUIDANCE-02](#req-guidance-02--composition-algorithm) composition ships. `EMPTY_GUIDANCE_FINGERPRINT` is the v1.0 placeholder. + When rendering a briefing, Clarion queries Filigree for findings on this entity with `status ∈ {suppressed, acknowledged}` and a non-empty reason, and surfaces those as either inline notes (≤3) or a synthetic `RiskItem` with `tag: operator-acknowledged` (>3). The guidance fingerprint incorporates the set of acknowledged finding IDs. **Rationale**: Operator triage decisions (suppressions, acknowledgements) are institutional knowledge in the same shape as guidance; they must flow back into briefings so the next LLM agent sees "this was already decided" rather than re-opening the conversation. @@ -235,6 +243,8 @@ When rendering a briefing, Clarion queries Filigree for findings on this entity #### REQ-BRIEFING-06 — Detail levels: short / medium / full / exhaustive +> **Deferred to v1.1** per [ADR-030](../adr/ADR-030-on-demand-summary-scope.md) + [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md). Detail-level tiers are part of the deferred briefing pipeline; v1.0 `summary(id)` returns a single un-tiered shape per ADR-030. Per-tool token-budget contracts in [REQ-MCP-04](#req-mcp-04--bounded-response-sizes) reference these tiers and are correspondingly deferred. + Briefings are renderable at four detail levels with token ceilings: `short` ≤100 tokens, `medium` ≤400, `full` ≤1,500, `exhaustive` ≤3,600. The ≤ figures are enforced hard limits that trigger truncation; typical briefings target roughly half these values (see System Design §3). Implementations must treat the ≤ figures as the contract. **Rationale**: Bounded responses (Principle 2) demand fixed token budgets per detail level. LLM agents can request what they need without pulling an entire subsystem's worth of text into their context. @@ -247,6 +257,8 @@ Briefings are renderable at four detail levels with token ceilings: `short` ≤1 Institutional knowledge attached to entities and composed into prompts. +> **Whole subsystem deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP7 deferred in full). The `guidance_sheets` view substrate exists in the v1.0 storage schema (kind reservation, ADR-024 vocabulary), but no authoring path, composition algorithm, wardline-derivation, staleness signals, or export/import surface ships. `EMPTY_GUIDANCE_FINGERPRINT` is hard-coded into every summary-cache key as a placeholder. REQ-GUIDANCE-01 through REQ-GUIDANCE-06 below describe the v1.1 target. + #### REQ-GUIDANCE-01 — Guidance sheets as first-class entities Guidance sheets are entities of `kind: guidance` with properties `{content, scope_level, scope, match_rules, expires, pinned, provenance, ...}` (per ADR-024). They participate in the same navigation, finding, and cache machinery as code entities. @@ -319,7 +331,7 @@ Clarion's rule IDs follow the `CLA-*` namespace: `CLA-PY-*` for Python-plugin st #### REQ-FINDING-03 — Emit findings to Filigree -> **Deferred to v0.2** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B). Clarion v0.1 emits findings to its own SQLite `findings` table; cross-product emission to Filigree lands with WP9-B. +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B). Clarion v0.1 emits findings to its own SQLite `findings` table; cross-product emission to Filigree lands with WP9-B. Clarion POSTs findings to Filigree via `POST /api/v1/scan-results` using Filigree's native intake schema. Clarion's richer fields (`kind`, `confidence`, `confidence_basis`, `supports`/`supported_by`, `related_entities`, internal severity, internal status) nest inside each finding's `metadata.clarion.*`. @@ -329,7 +341,7 @@ Clarion POSTs findings to Filigree via `POST /api/v1/scan-results` using Filigre #### REQ-FINDING-04 — General-purpose SARIF → Filigree translator -> **Deferred to v0.2** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP10). The SARIF translator is independent of MCP surface delivery and lands with the v0.2 cross-product work. +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP10). The SARIF translator is independent of MCP surface delivery and lands with the v1.1 cross-product work. Clarion ships `clarion sarif import [--scan-source ]` that translates any SARIF v2.1.0 emitter (Wardline, Semgrep, CodeQL, Trivy) to Filigree's scan-results format, preserving `result.properties` into `metadata._properties.*`. @@ -339,7 +351,7 @@ Clarion ships `clarion sarif import [--scan-source ]` that translat #### REQ-FINDING-05 — `scan_run_id` lifecycle -> **Deferred to v0.2** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B). Lands with REQ-FINDING-03 (findings emission to Filigree). +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B). Lands with REQ-FINDING-03 (findings emission to Filigree). Clarion's `run_id` maps 1:1 onto Filigree's `scan_run_id`. Phase 0 of `clarion analyze` creates the scan run; Phase 8 closes it with `complete_scan_run=true`. Resume (`--resume`) reuses the same `run_id` and posts with `mark_unseen=false`. @@ -349,7 +361,7 @@ Clarion's `run_id` maps 1:1 onto Filigree's `scan_run_id`. Phase 0 of `clarion a #### REQ-FINDING-06 — Dedup policy for moved entities -> **Deferred to v0.2** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B). Concerns Filigree-side dedup; lands with REQ-FINDING-03 (findings emission). +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B). Concerns Filigree-side dedup; lands with REQ-FINDING-03 (findings emission). Clarion POSTs findings with `mark_unseen=true` by default so that old-position findings for the same rule on the same file transition to `unseen_in_latest` when the entity moves within the file. `clarion analyze --prune-unseen` removes `unseen_in_latest` findings older than 30 days (configurable). @@ -365,6 +377,8 @@ The MCP tool surface exposed by `clarion serve` for consult-mode LLM agents. #### REQ-MCP-01 — Cursor-based session model +> **Deferred to v1.1** per the [Sprint 2 scope amendment §3 (Box B.6 narrowed the MCP surface)](../../implementation/sprint-2/scope-amendment-2026-05.md). The cursor-based Navigation model is one of the categories explicitly removed from the v1.0 MVP surface; v1.0 tools take explicit `id` arguments per-call. Cursor + breadcrumbs + scope-lens land with the broader catalogue. + Every MCP session has server-held state: cursor (`EntityId`), breadcrumb history, scope lens, session cost accumulator, tracked observations / proposals. Navigation tools update the cursor; inspection tools default to operating on the cursor. **Rationale**: Cursor-based navigation eliminates the need to pass `entity_id` to every call — an agent says `goto(id)`, then `summary()`, `neighbors()`, `callers()` all operate on the current entity. Reduces parent-context token consumption (`NFR-PERF-03`) and models the "I'm looking at this entity" conversational stance naturally. @@ -373,6 +387,8 @@ Every MCP session has server-held state: cursor (`EntityId`), breadcrumb history #### REQ-MCP-02 — Navigation and inspection tool catalogue +> **v1.0 ships an 8-tool MVP subset** per the [Sprint 2 scope amendment §3 (Box B.6)](../../implementation/sprint-2/scope-amendment-2026-05.md): `entity_at`, `find_entity`, `callers_of`, `execution_paths_from`, `summary`, `issues_for`, `neighborhood`, plus `subsystem_members` (added in Sprint 3 with WP4 Phase-3 clustering). The remaining categories listed below — the cursor-based Navigation model, write-effect Inspection tools, Search, Findings operations, Guidance (deferred with WP7), and Session/scope — are deferred to v1.1 per the same amendment §4. The catalogue paragraph below documents the v1.1 target surface. + Clarion exposes MCP tools in the documented categories: Navigation (`goto`, `goto_path`, `back`, `zoom_out`, `zoom_in`, `breadcrumbs`); Inspection (`summary`, `source`, `metadata`, `guidance_for`, `findings_for`, `wardline_for`); Neighbours (`neighbors`, `callers`, `callees`, `children`, `imports_from`, `imported_by`, `in_subsystem`, `subsystem_members`); Search (`search_structural`, `search_semantic`, `find_by_tag`, `find_by_wardline`, `find_by_kind`); Findings & observability (`list_findings`, `emit_observation`, `promote_observation`, `cost_report`); Guidance (`show_guidance`, `list_guidance`, `propose_guidance`, `promote_guidance`); Session/scope (`set_scope_lens`, `session_info`). **Rationale**: Principle 2 (exploration elimination) requires the tool catalogue to cover the common explore-agent questions. Missing a category forces agents to spawn sub-agents to answer what the catalogue should have answered; bloating it dilutes the surface and raises consumption. @@ -381,6 +397,8 @@ Clarion exposes MCP tools in the documented categories: Navigation (`goto`, `got #### REQ-MCP-03 — Exploration-elimination shortcuts +> **Deferred to v1.1** per the [Sprint 2 scope amendment §3 (Box B.6 — narrowed MCP surface)](../../implementation/sprint-2/scope-amendment-2026-05.md) and §4 (WP4 Phase-7 cross-cutting analysis deferred to v1.1). Exploration shortcuts are populated by pre-computation during batched Phase-7 analysis; v1.0 ships on-demand `summary` only ([ADR-030](../adr/ADR-030-on-demand-summary-scope.md)) and has no batched pre-computation pass to feed the shortcut indices. + Clarion exposes pre-computed shortcuts operationalising common exploration queries: `find_entry_points`, `find_http_routes`, `find_cli_commands`, `find_data_models`, `find_config_loaders`, `find_tests`, `find_fixtures`, `find_deprecations`, `find_todos`, `find_dead_code`, `find_circular_imports`, `find_coupling_hotspots`, `recently_changed`, `high_churn`, `what_tests_this`. Each accepts an optional `scope` (entity ID or path glob). **Rationale**: Each of these is an "explore-agent spawn" an LLM would otherwise perform by walking the graph — and each can be pre-computed during batch analysis and cached. Principle 2 says those spawns are a failure mode; the shortcuts are the remedy. @@ -389,6 +407,8 @@ Clarion exposes pre-computed shortcuts operationalising common exploration queri #### REQ-MCP-04 — Bounded response sizes +> **v1.0 ships intrinsic bounds on the 8-tool MVP subset** (`max_depth`, `limit`, server-side `edge_cap` of 500 on `execution_paths_from`, pagination on `find_entity`). Per-tool token-budget contracts (`summary(short/medium/full) ≤100/≤400/≤1,500`) and the per-session `set_budget(tool, max_tokens)` API are deferred to v1.1 — they reference catalogue surface ([REQ-MCP-02](#req-mcp-02--navigation-and-inspection-tool-catalogue)) and detail levels ([REQ-BRIEFING-06](#req-briefing-06--detail-levels-short--medium--full--exhaustive)) that are themselves deferred per the [Sprint 2 scope amendment](../../implementation/sprint-2/scope-amendment-2026-05.md) + [ADR-030](../adr/ADR-030-on-demand-summary-scope.md). + MCP tool responses respect per-tool token budgets: `summary(short) ≤100`, `summary(medium) ≤400`, `summary(full) ≤1,500`, `neighbors / callers / callees / children ≤20 results × ≤50 tokens each`, `source` paginated above 2,000 tokens, `search_*` ≤10 results. Budgets are configurable per-session via `set_budget(tool, max_tokens)`. **Rationale**: Unbounded responses re-introduce the context-pollution problem Clarion exists to solve. Explicit budgets keep parent context growth predictable and let agents reason about how much slack they have before compaction. @@ -397,6 +417,8 @@ MCP tool responses respect per-tool token budgets: `summary(short) ≤100`, `sum #### REQ-MCP-05 — Consent gates on write-effect tools +> **Deferred to v1.1** per the [Sprint 2 scope amendment §3 (Box B.6 narrowed the MCP surface to 8 read-only tools)](../../implementation/sprint-2/scope-amendment-2026-05.md). The write-effect tools this requirement governs (`emit_observation`, `promote_observation`, `propose_guidance`, `promote_guidance`) are not in the v1.0 MVP surface; consent-gate semantics land with the broader catalogue and the WP7 guidance system. + Write-effect tools (`emit_observation`, `promote_observation`, `propose_guidance`, `promote_guidance`) return a draft for human confirmation by default. Headless agent-walk mode enables direct writes via client-declared `capabilities: { auto_emit: true }`. **Rationale**: Consult sessions are often human-in-the-loop; surprise writes erode trust. Explicit consent via draft-then-confirm matches how human operators expect agent tools to behave; headless mode opts out for fully-automated pipelines. @@ -405,6 +427,8 @@ Write-effect tools (`emit_observation`, `promote_observation`, `propose_guidance #### REQ-MCP-06 — Session persistence and lifecycle +> **Deferred to v1.1** per the [Sprint 2 scope amendment §3 (Box B.6)](../../implementation/sprint-2/scope-amendment-2026-05.md). Session persistence is the cursor-based Navigation model deferred with [REQ-MCP-01](#req-mcp-01--cursor-based-session-model); v1.0 tools take explicit `id` arguments per-call and hold no per-session state beyond the request scope. + Sessions are created on MCP `initialize`, idle-timeout after 1 hour (configurable), and persist to `.clarion/sessions/.json` for reconnection. `clarion sessions list` and `clarion sessions close ` provide admin surfaces. **Rationale**: Agents reconnecting after a transport interruption expect their cursor and breadcrumbs to survive; losing session state mid-investigation is hostile to the workflow. @@ -419,6 +443,8 @@ Human-readable outputs from `clarion analyze`. #### REQ-ARTEFACT-01 — JSON catalog output +> **Deferred to v1.1** per the [Sprint 2 scope amendment §3 (Box B.4 removed)](../../implementation/sprint-2/scope-amendment-2026-05.md). The `catalog.json` artefact has no consumer in the v1.0 MVP MCP surface; the MCP tools query the SQLite store directly. + `clarion analyze` emits `.clarion/catalog.json` — a deterministic, stable-shape dump of the entity catalog, edges, subsystems, and findings at run completion. **Rationale**: JSON is the universal interchange format; downstream consumers (dashboards, bespoke scripts, CI gates) can read the catalog without speaking SQLite. Deterministic output means git diffs reflect real changes, not run-to-run noise. @@ -427,6 +453,8 @@ Human-readable outputs from `clarion analyze`. #### REQ-ARTEFACT-02 — Per-subsystem markdown + top-level index +> **Deferred to v1.1** per the [Sprint 2 scope amendment §3 (Box B.5 removed)](../../implementation/sprint-2/scope-amendment-2026-05.md). Subsystem rendering lands with WP4 in v1.1. + `clarion analyze` emits `.clarion/catalog/.md` (one markdown file per subsystem) plus `.clarion/catalog/index.md` (top-level navigation). Markdown is generated from the store, not authored. **Rationale**: Markdown is the human-reading surface for cases where a human (reviewer, new team member) wants to read the catalog without running `clarion serve` or speaking MCP. Subsystem granularity matches how humans think about large codebases; the index makes discovery cheap. @@ -449,6 +477,8 @@ Clarion reads project configuration from `clarion.yaml` at the repository root, #### REQ-CONFIG-02 — Profile presets (budget / default / deep / custom) +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP6 narrowed to on-demand `summary(id)` per [ADR-030](../adr/ADR-030-on-demand-summary-scope.md)). Profile presets sit on top of the LLM dispatch + per-level policy that v1.0 does not ship. + Clarion ships three named profiles (`budget`, `default`, `deep`) and supports `custom`. Each profile specifies per-level mode / model_tier / summary_length; `clarion.yaml:llm_policy.profile` picks one; `overrides` layer on top. **Rationale**: Named profiles make cost trade-offs legible. An operator saying "use `budget`" and getting 4× cost reduction with predictable depth loss is faster than tuning six per-level parameters; `custom` is the escape hatch for teams with specific needs. @@ -457,6 +487,8 @@ Clarion ships three named profiles (`budget`, `default`, `deep`) and supports `c #### REQ-CONFIG-03 — Budget enforcement with preflight +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP6 narrowed; the Phase 0 dry-run / preflight estimator + run-level budget gate are part of the deferred batched pipeline). v1.0 reaches LLM cost only lazily via MCP `summary` per [ADR-030](../adr/ADR-030-on-demand-summary-scope.md); there is no batch run for a preflight to estimate. + `clarion analyze` computes a cost estimate from the dry-run and confirms with the user before dispatching (default `dry_run_first: true`). During the run, budget watchers enforce `max_usd_per_run` and `max_minutes`; exceeding either with `on_exceed: stop` halts dispatch and writes a partial manifest. **Rationale**: LLM cost surprise is a common failure mode for teams adopting LLM-assisted tooling. Preflight prevents "I just spent $400 to analyse my monorepo" incidents; in-flight enforcement bounds the worst case when estimates are wrong. @@ -465,6 +497,8 @@ Clarion ships three named profiles (`budget`, `default`, `deep`) and supports `c #### REQ-CONFIG-04 — Per-level LLM policy +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP6 narrowed to on-demand `summary(id)` per [ADR-030](../adr/ADR-030-on-demand-summary-scope.md)). Per-level policy is meaningful only when batched module/subsystem-tier summarisation exists. + `clarion.yaml:llm_policy.levels.` specifies `mode` (`batch | on_demand | off`), `model_tier` (`haiku | sonnet | opus`), and `summary_length`. Overrides match on `path` / `subsystem` / other criteria and layer per-level. **Rationale**: Different entity levels (function vs subsystem) warrant different cost / depth trade-offs. Per-level policy lets operators spend Opus tokens where they matter (subsystem synthesis) and Haiku tokens where they don't (per-function leaf summaries), with path-based overrides for exceptions. @@ -541,6 +575,8 @@ The read-only HTTP surface exposed by `clarion serve`. #### REQ-HTTP-01 — Read endpoints for entities, findings, wardline, state +> **v1.0 ships the [ADR-014](../adr/ADR-014-filigree-registry-backend.md) file-registry subset only**: `GET /api/v1/files`, `POST /api/v1/files:resolve`, `POST /api/v1/files/batch`, `GET /api/v1/_capabilities` (plus the ADR-034 authentication surface — see [REQ-HTTP-03](#req-http-03--registry-backend-http-trust-model)). The broader `/entities*`, `/findings`, `/wardline/declared`, `/state`, `/health`, `/metrics` catalogue documented below is deferred to v1.1 per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B for findings/wardline/entities; WP10 for `/state`, `/health`, `/metrics`). The endpoint list below documents the v1.1 target surface. + Clarion exposes read-only HTTP endpoints: `GET /api/v1/entities`, `GET /api/v1/entities/{id}`, `GET /api/v1/entities/{id}/neighbors`, `GET /api/v1/entities/{id}/summary`, `GET /api/v1/entities/{id}/guidance`, `GET /api/v1/entities/{id}/findings`, `GET /api/v1/findings`, `GET /api/v1/wardline/declared`, `GET /api/v1/state`, `GET /api/v1/health`, `GET /api/v1/metrics` (Prometheus-compatible). **Rationale**: Sibling tools (Wardline in v0.2+, future dashboards, CI gates) consume Clarion's catalog via HTTP; MCP is not appropriate for cross-process state pulls. Read-only in v0.1 keeps the surface small. @@ -549,6 +585,8 @@ Clarion exposes read-only HTTP endpoints: `GET /api/v1/entities`, `GET /api/v1/e #### REQ-HTTP-02 — Entity resolution oracle +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B Wardline state-file ingest + WP10 broader HTTP surface). The full multi-scheme oracle depends on Wardline state-file ingest landing for `wardline_qualname` / `wardline_exception_location` / `sarif_logical_location`; v1.0 resolves the `file_path` scheme only via [REQ-HTTP-01](#req-http-01--read-endpoints-for-entities-findings-wardline-state)'s `POST /api/v1/files:resolve` and `POST /api/v1/files/batch` per [ADR-014](../adr/ADR-014-filigree-registry-backend.md). + `GET /api/v1/entities/resolve?scheme=&value=` translates from sibling-tool identity schemes (`wardline_qualname`, `wardline_exception_location`, `file_path`, `sarif_logical_location`) to Clarion entity IDs. Returns `{entity_id, kind, resolution_confidence: exact|heuristic|none, alternatives}`. **Rationale**: Enrichment-not-load-bearing (`CON-LOOM-01`): sibling tools consuming Clarion should ask in *their* native identity scheme, not embed Clarion's ID format. `resolve` exposes Clarion's internal translation layer as a public API so every sibling doesn't re-implement it. @@ -597,7 +635,7 @@ Clarion's side of Filigree integration. #### REQ-INTEG-FILIGREE-01 — Findings via scan-results intake -> **Deferred to v0.2** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B). Lands with REQ-FINDING-03. +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B). Lands with REQ-FINDING-03. Clarion POSTs findings to Filigree's `POST /api/v1/scan-results` using the native schema (see `REQ-FINDING-03`) with `scan_source: "clarion"`. Clarion inspects `response.warnings[]` on every POST for silent coercion / unknown-key drops. @@ -607,7 +645,7 @@ Clarion POSTs findings to Filigree's `POST /api/v1/scan-results` using the nativ #### REQ-INTEG-FILIGREE-02 — Observation emission (HTTP preferred, MCP fallback) -> **Deferred to v0.2** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B). Clarion v0.1 surfaces structural and security findings via its own MCP `issues_for` tool (WP9-A); cross-product observation emission to Filigree lands with WP9-B. +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B). Clarion v0.1 surfaces structural and security findings via its own MCP `issues_for` tool (WP9-A); cross-product observation emission to Filigree lands with WP9-B. Clarion emits observations to Filigree via `POST /api/v1/observations` when available; falls back to MCP-client transport (spawning `filigree mcp` as subprocess) when the HTTP endpoint is absent. The fallback is signalled in the capability compat report. @@ -617,7 +655,7 @@ Clarion emits observations to Filigree via `POST /api/v1/observations` when avai #### REQ-INTEG-FILIGREE-03 — Registry-backend consumption -> **Deferred to v0.2** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B/WP10) and reflected in [CON-FILIGREE-02](#con-filigree-02--file-registry-displacement-is-deferred-to-v02). Filigree's `registry_backend` flag shipped (ADR-014); Clarion's read-side `RegistryProtocol` implementation is the v0.2 work. Clarion v0.1 ships shadow-registry only. +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B/WP10) and reflected in [CON-FILIGREE-02](#con-filigree-02--file-registry-displacement-is-deferred-to-v02). Filigree's `registry_backend` flag shipped (ADR-014); Clarion's read-side `RegistryProtocol` implementation is the v1.1 work. Clarion v0.1 ships shadow-registry only. When Filigree's `registry_backend` flag is set to `clarion`, Clarion serves as Filigree's file registry: Filigree consults Clarion's HTTP read API for file ID resolution; auto-create paths route through `RegistryProtocol` to Clarion. Absent the flag, Clarion operates in shadow-registry mode (findings POSTed normally; Filigree auto-creates `file_records` under its native rules). @@ -627,7 +665,7 @@ When Filigree's `registry_backend` flag is set to `clarion`, Clarion serves as F #### REQ-INTEG-FILIGREE-04 — `scan_source` namespace + schema pin test -> **Deferred to v0.2** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B). Lands with REQ-FINDING-03 (findings emission to Filigree); schema-pin test is moot until Clarion is actually POSTing. +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B). Lands with REQ-FINDING-03 (findings emission to Filigree); schema-pin test is moot until Clarion is actually POSTing. Clarion uses `scan_source: "clarion"` for emissions; CI runs a schema-compatibility test against a Filigree release's `GET /api/files/_schema` output to detect drift in `valid_severities`, `valid_finding_statuses`, `valid_association_types`. @@ -637,7 +675,7 @@ Clarion uses `scan_source: "clarion"` for emissions; CI runs a schema-compatibil #### REQ-INTEG-FILIGREE-05 — Capability-negotiation probe -> **Deferred to v0.2** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B). The capability probe is meaningful only once Clarion is talking to Filigree via the v0.2 cross-product surfaces (REQ-FINDING-03, REQ-INTEG-FILIGREE-02/03). +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B). The capability probe is meaningful only once Clarion is talking to Filigree via the v1.1 cross-product surfaces (REQ-FINDING-03, REQ-INTEG-FILIGREE-02/03). At `clarion analyze` startup, Clarion probes Filigree's presence, version, `registry_backend` setting, and `/api/v1/observations` availability via `GET /api/files/_schema` + `HEAD` checks. Results emit in a single `CLA-INFRA-SUITE-COMPAT-REPORT` finding. @@ -661,6 +699,8 @@ Clarion's Python plugin imports `wardline.core.registry.REGISTRY` at startup and #### REQ-INTEG-WARDLINE-02 — Manifest + overlay ingest +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B Wardline-config ingest). v1.0 ships only the [REQ-INTEG-WARDLINE-01](#req-integ-wardline-01--direct-registry-import-with-version-pin) runtime REGISTRY probe; state-file ingest (`wardline.yaml` + overlays) lands with the Wardline read-side bundle in v1.1. + Clarion reads `wardline.yaml` and overlay files matching `src/**/wardline.overlay.yaml` at analyse time; declared tiers, groups, and boundary contracts become `WardlineMeta` properties on affected entities. **Rationale**: The manifest is Wardline's declarative source of truth; ingesting it makes tier/group/contract declarations available as entity metadata without re-implementing Wardline's parsing. Overlays let per-subsystem declarations compose cleanly. @@ -669,6 +709,8 @@ Clarion reads `wardline.yaml` and overlay files matching `src/**/wardline.overla #### REQ-INTEG-WARDLINE-03 — Fingerprint ingest +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B Wardline-config ingest). Fingerprint ingest depends on the same state-file reader path deferred for [REQ-INTEG-WARDLINE-02](#req-integ-wardline-02--manifest--overlay-ingest); v1.0 has no `WardlineMeta.annotation_hash` population path. + Clarion reads `wardline.fingerprint.json` at analyse time; each per-function `FingerprintEntry` becomes `WardlineMeta.annotation_hash` + `wardline_qualname` on the resolved entity. Unresolved fingerprint entries emit `CLA-INFRA-WARDLINE-FINGERPRINT-UNRESOLVED`. **Rationale**: Fingerprint provides the authoritative per-function annotation hash — without it, Clarion can't track drift between what Wardline enforces and what the code declares. Resolution failures must be visible (not silent) so operators can fix the mapping. @@ -677,6 +719,8 @@ Clarion reads `wardline.fingerprint.json` at analyse time; each per-function `Fi #### REQ-INTEG-WARDLINE-04 — Exceptions ingest +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B Wardline-config ingest). Exceptions ingest depends on the same state-file reader path deferred for [REQ-INTEG-WARDLINE-02](#req-integ-wardline-02--manifest--overlay-ingest); v1.0 carries no `wardline.excepted` tag on briefings. + Clarion reads `wardline.exceptions.json` at analyse time; entities referenced by active exceptions are tagged `wardline.excepted`. Unresolvable exception `location` strings emit `CLA-INFRA-WARDLINE-EXCEPTION-UNRESOLVED` and persist as dangling records with `entity_id: null`. **Rationale**: Exceptions are operator-curated decisions ("this finding is accepted"). Agents reading briefings for excepted entities should see "this has an active exception" as part of the picture; unresolvable exceptions are operator bugs that need visibility. @@ -685,6 +729,8 @@ Clarion reads `wardline.exceptions.json` at analyse time; entities referenced by #### REQ-INTEG-WARDLINE-05 — SARIF baseline ingest for translator +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP10 SARIF translator). The translator surface (`clarion sarif import`) itself is deferred — see [REQ-INTEG-FILIGREE-04](#req-integ-filigree-04--sarif-import-translator) — so the baseline-ingest path it feeds is moot until WP10 lands. + Clarion reads `wardline.sarif.baseline.json` (read-only) for the `clarion sarif import` translator path — the 663-result baseline is the source for Wardline-to-Filigree finding flow in v0.1. **Rationale**: Translator ownership lives Clarion-side in v0.1 (ADR-015); reading the baseline from disk keeps Wardline's dependency graph unchanged until it ships a native Filigree emitter in v0.2+. @@ -693,6 +739,8 @@ Clarion reads `wardline.sarif.baseline.json` (read-only) for the `clarion sarif #### REQ-INTEG-WARDLINE-06 — Identity reconciliation across three schemes +> **Deferred to v1.1** per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B Wardline-config ingest). The three-scheme oracle depends on fingerprint + exceptions ingest ([REQ-INTEG-WARDLINE-03](#req-integ-wardline-03--fingerprint-ingest) / [-04](#req-integ-wardline-04--exceptions-ingest)) populating `wardline_qualname` and `wardline_exception_location`; v1.0 resolves the `file_path` scheme only via [REQ-HTTP-01](#req-http-01--read-endpoints-for-entities-findings-wardline-state)'s `POST /api/v1/files:resolve` and `POST /api/v1/files/batch` per [ADR-014](../adr/ADR-014-filigree-registry-backend.md). + Clarion maintains translation between three identity schemes: Clarion `EntityId`, Wardline `qualname`, Wardline exception-register `location` string. Reconciliation uses Wardline's `module_file_map` (from `ScanContext`) plus parsed location strings. **Rationale**: The three schemes arose independently and are not byte-equal for the same symbol. Clarion is the translator (Principle 3 — Wardline keeps its scheme; Clarion produces the join); exposing the translation via `GET /api/v1/entities/resolve` (`REQ-HTTP-02`) lets sibling tools consume it without embedding Clarion's ID format. @@ -707,6 +755,8 @@ Clarion maintains translation between three identity schemes: Clarion `EntityId` #### NFR-PERF-01 — Elspeth-scale wall-clock budget +> **Deferred to v1.1** per [ADR-030](../adr/ADR-030-on-demand-summary-scope.md) + [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md). The 60-min target includes Phase 4–6 LLM summarisation wall-clock, which is the deferred batched pipeline; v1.0 `clarion analyze` ends at Phase 3 and is bounded by Python plugin extraction + Leiden clustering only. Remains the v1.1 acceptance criterion when phases 4–6 land. + `clarion analyze /home/john/elspeth` completes in ≤60 minutes (target ~38 minutes) on a representative developer machine (8+ cores, SSD, ≥16GB RAM). **Rationale**: 60 minutes is the psychological ceiling for a "run overnight or during lunch" batch tool; significantly above it pushes teams to run less often and lose feedback. The ~38-minute target is derived from the detailed-design's example run; the ceiling bounds estimation error. @@ -795,12 +845,22 @@ the registry-backend surface. ADR-034 closes the original ADR-014 gap that permitted unauthenticated non-loopback binds behind the `allow_non_loopback` opt-in alone; the opt-in remains the gate that admits non-loopback binds at all but no longer admits them unauthenticated. -**Verification**: `crates/clarion-cli/tests/serve.rs` covers the loopback -default (line 1457), the loopback `identity_token_env`-set-but-env-missing -refusal (line 1495), the non-loopback-without-auth refusal (line 1547), the -non-loopback HMAC-required path (line 1579), and the non-loopback legacy-bearer -path (line 1614). Config-layer tests cover the loopback startup-warning -surface. +**Verification**: `crates/clarion-cli/tests/serve.rs` covers the +non-loopback-bind-without-opt-in refusal +(`serve_rejects_non_loopback_http_bind_before_binding_without_opt_in`), the +non-loopback-without-auth refusal +(`serve_http_refuses_startup_on_non_loopback_without_token`), the +`identity_token_env`-set-but-env-missing refusal +(`serve_http_refuses_startup_when_identity_env_is_missing`), the HMAC-required +request path (`serve_http_files_endpoint_requires_hmac_identity_when_configured` +plus the wrong-secret rejection +`serve_http_files_endpoint_rejects_wrong_hmac_identity`), the legacy-bearer +request path (`serve_http_files_endpoint_requires_bearer_token_when_configured` +plus the wrong-token rejection `serve_http_files_endpoint_rejects_wrong_token` +and the batch-endpoint variant `serve_http_batch_requires_auth_when_configured`), +and the unauthenticated `_capabilities` carve-out +(`serve_http_capabilities_does_not_require_token`). Config-layer tests cover the +loopback startup-warning surface. **See**: System Design §9 (HTTP Read API), §10 (Security), ADR-014, ADR-034. #### NFR-SEC-04 — Audit surface — security events as findings @@ -897,6 +957,8 @@ Every `clarion analyze` emits exactly one `CLA-INFRA-SUITE-COMPAT-REPORT` findin #### NFR-COST-01 — Elspeth run budget target +> **Deferred to v1.1** per [ADR-030](../adr/ADR-030-on-demand-summary-scope.md) + [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md). The $15 ± 50% target measures a full batched run including phases 4–6 LLM spend; v1.0 reaches LLM cost only lazily through MCP `summary`. Remains the v1.1 acceptance criterion when the batched pipeline lands. + `clarion analyze /home/john/elspeth` costs $15 ± 50% in LLM spend (range: $7.50 - $22.50) at default profile with current Anthropic pricing. **Rationale**: Matches the detailed-design's example run. The ±50% band reflects estimator uncertainty (subsystem synthesis cost varies with clustering) plus pricing volatility; wider bands undermine operator trust. @@ -913,6 +975,8 @@ After three consecutive runs without source or guidance changes, summary cache h #### NFR-COST-03 — Preflight cost estimate accuracy ±50% +> **Deferred to v1.1** per [ADR-030](../adr/ADR-030-on-demand-summary-scope.md) + [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md). Preflight estimation lives in Phase 0 (dry-run) of the batched pipeline; v1.0 has no batch run to preflight. Couples to [REQ-CONFIG-03](#req-config-03--budget-enforcement-with-preflight). + The dry-run cost estimate is within ±50% of actual spend on representative projects. Systematic under-estimation is worse than over-estimation (preflight should not encourage false confidence). **Rationale**: An estimate outside this range fails its purpose (preventing cost surprise). ±50% is tight enough to be useful, loose enough to be achievable with per-entity-level heuristics rather than full Opus pricing simulations. diff --git a/docs/clarion/1.0/system-design.md b/docs/clarion/1.0/system-design.md index a525f71c..9fd1a71d 100644 --- a/docs/clarion/1.0/system-design.md +++ b/docs/clarion/1.0/system-design.md @@ -100,8 +100,8 @@ flowchart TB | Mode | Surface | Purpose | v0.1 status | |---|---|---|---| | MCP-for-LLM | `clarion serve` over stdio | First-class product surface — consult-mode agents hold a cursor, navigate the graph, emit observations to Filigree | Primary | -| Catalog artefacts | `clarion analyze` writes `.clarion/catalog.json` + per-subsystem markdown | "I want to read the output" cases | Shipped | -| Semi-dynamic wiki | HTML served by `clarion serve` | Live finding list, in-browser guidance editing, consult entry points | v0.2 (deferred — NG-13) | +| Catalog artefacts | `clarion analyze` writes `.clarion/catalog.json` + per-subsystem markdown | "I want to read the output" cases | v1.1 (deferred — Sprint 2 amendment §3 removed boxes B.4/B.5; see [REQ-ARTEFACT-01](requirements.md#req-artefact-01--json-catalog-output) / [REQ-ARTEFACT-02](requirements.md#req-artefact-02--per-subsystem-markdown--top-level-index)) | +| Semi-dynamic wiki | HTML served by `clarion serve` | Live finding list, in-browser guidance editing, consult entry points | v1.1 (deferred — NG-13) | ### Boundary contracts with the Loom siblings @@ -770,6 +770,8 @@ The scope lens shapes neighbour queries without changing their signatures: ### Tool catalogue by category +> **v1.0 ships an 8-tool subset of this catalogue** per the [Sprint 2 scope amendment §3 (Box B.6)](../../implementation/sprint-2/scope-amendment-2026-05.md): `entity_at(file, line)`, `find_entity(pattern)`, `callers_of(id, confidence)`, `execution_paths_from(id, max_depth, confidence)`, `summary(id)`, `issues_for(id, include_contained)`, `neighborhood(id, confidence)`, plus `subsystem_members(id)` (added in Sprint 3 with WP4 Phase-3 clustering). The full categorical catalogue below — including the cursor-based Navigation model, write-effect Inspection tools, Search, Findings ops, Guidance (deferred with WP7), Session/scope — is the v1.1 target surface and is not present in v1.0. See [REQ-MCP-02](requirements.md#req-mcp-02--navigation-and-inspection-tool-catalogue) for the row-level deferral notice. + **Navigation**: `goto(id)`, `goto_path(path, line?)`, `back()`, `zoom_out()`, `zoom_in(child_id)`, `breadcrumbs()` **Inspection**: `summary(id?, detail?)`, `source(id?, range?)`, `metadata(id?)`, `guidance_for(id?)`, `findings_for(id?, filter?)`, `wardline_for(id?)` @@ -786,6 +788,8 @@ The scope lens shapes neighbour queries without changing their signatures: ### Exploration-elimination shortcuts (Principle 2) +> **Deferred to v1.1** per the [Sprint 2 scope amendment](../../implementation/sprint-2/scope-amendment-2026-05.md) (Box B.6 narrowed the v1.0 MCP surface; WP4 Phase-7 cross-cutting analysis deferred). Shortcuts depend on batched pre-computation during analyze; v1.0 ships on-demand `summary` only ([ADR-030](../adr/ADR-030-on-demand-summary-scope.md)). + Every common explore-agent question is a pre-computed shortcut: `find_entry_points(scope?)`, `find_http_routes(scope?)`, `find_cli_commands(scope?)`, `find_data_models(scope?)`, `find_config_loaders(scope?)`, `find_tests(scope?)`, `find_fixtures(scope?)`, `find_deprecations(scope?)`, `find_todos(scope?)`, `find_dead_code(scope?)`, `find_circular_imports(scope?)`, `find_coupling_hotspots(scope?)`, `recently_changed(since?, scope?)`, `high_churn(limit?, scope?)`, `what_tests_this(id)` @@ -981,6 +985,8 @@ Translator behaviour: #### Endpoints +> **v1.0 ships the [ADR-014](../adr/ADR-014-filigree-registry-backend.md) file-registry subset only**: `GET /api/v1/files`, `POST /api/v1/files:resolve`, `POST /api/v1/files/batch`, `GET /api/v1/_capabilities` (plus the [ADR-034](../adr/ADR-034-federation-http-read-api-hardening.md) authentication surface). The `/entities*` / `/findings` / `/wardline/declared` / `/state` / `/health` / `/metrics` catalogue below is the v1.1 target — deferred per the [Sprint 2 scope amendment §4](../../implementation/sprint-2/scope-amendment-2026-05.md) (WP9-B for entities/findings/wardline; WP10 for `/state`, `/health`, `/metrics`). The entity-resolve oracle ships only for the `file_path` scheme via `POST /api/v1/files:resolve`; the multi-scheme oracle below is also v1.1. See [REQ-HTTP-01](requirements.md#req-http-01--read-endpoints-for-entities-findings-wardline-state) and [REQ-HTTP-02](requirements.md#req-http-02--entity-resolution-oracle) for the row-level deferral notices. + ``` GET /api/v1/entities?file=&kind=&tag= GET /api/v1/entities/{id} From ced07418170f2e937fb1c4f4ae63aca037a732b5 Mon Sep 17 00:00:00 2001 From: John Morrissey <544926+tachyon-beep@users.noreply.github.com> Date: Sat, 23 May 2026 10:42:02 +1000 Subject: [PATCH 8/9] feat(v1.0): ADR-035 operational-tuning discipline + STO-02 closure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Background — `docs/arch-analysis-2026-05-22-1924/` is a from-scratch architecture review whose §8 surfaced five open questions that a follow-on SME roundtable reframed as a single missing-feedback-loop pattern. ADR-035 is the level-5 (rules) intervention that pattern requires. This commit lands both the ADR and the first wave of code changes that ADR-035 commits the project to: - **ADR-035** (`docs/clarion/adr/ADR-035-operational-tuning-discipline.md`) Accepted on 2026-05-23. Extends ADR-021 §4. Every operational constant in Clarion source MUST declare four things — stated basis, override surface, retune trigger, coupling — in a code comment immediately adjacent to the constant. Plus file-LOC budgets (1500 LOC trigger declares a split-trigger condition) and crate-boundary budgets. - **STO-02 application_id (closes gap-register STO-02)** `crates/clarion-storage/src/pragma.rs` sets `application_id = 0x434C524E` ("CLRN") lazily on first open of a fresh file; refuses to open any database with a different non-zero `application_id`. New `StorageError::ForeignDatabase` variant carries the offending value into the error envelope. - **STO-02 user_version forward-incompatibility refusal** `crates/clarion-storage/src/schema.rs` declares `CURRENT_SCHEMA_VERSION = 1` with a compile-time assertion that it matches the highest `MIGRATIONS[]` entry. `verify_user_version()` refuses to operate on a database whose `user_version` is strictly greater. `Writer::spawn` invokes `verify_user_version()` directly so a forward-incompat file is rejected before the writer-actor opens for business. New `StorageError::FutureUserVersion` carries `found` and `current` into the envelope. - **HTTP error envelope for the new variants** `crates/clarion-cli/src/http_read.rs::classify_read_error` returns 500 `STORAGE_ERROR` with a header-rejection message for `ForeignDatabase` / `FutureUserVersion`. The HTTP API does not open its own writer, but the reader pool can encounter the same file header mismatches and the response code should be distinct. - **`PyrightRunState` for shared restart budget** `plugins/python/src/clarion_plugin_python/pyright_session.py` adds a `PyrightRunState` dataclass shared across session recycles. Without it, the 3-restart cap reset at every 25-file recycle, letting a crash-looping pyright silently consume `ceil(N/25) * 3` restarts instead of the documented 3 for an entire run. `server.py` constructs one `PyrightRunState` per `ServerState` and passes it into every new `PyrightSession`. - **`analyze_failure_modes.rs` integration test** Closes the test-discipline interlock that ADR-035's systems-thinker analysis named: exercises the `RunOutcome::HardFailed` branch in `analyze.rs::run_with_options` where the writer-actor rejects an `InsertEdge` mid-run, distinguishing the failure surface from `SoftFailed` (which carries the full stats blob). - **`schema_apply.rs` discipline tests** Cover the new application_id + user_version surface: foreign-DB refusal, future-DB refusal, version-bump from zero, idempotent re-open. Verification: `cargo build`, `cargo clippy -D warnings`, `cargo nextest` (509 passed), `cargo doc -D warnings`, `cargo deny`, `ruff`, `ruff format --check`, `mypy --strict`, `pytest` (143 passed). Closes gap-register STO-02 (`clarion-f2a984fd6d`). Forward-references gap-register STO-04 (e2e integrity_check) and the v1.1 backlog of constants needing declared bases. --- crates/clarion-cli/src/http_read.rs | 13 + .../tests/analyze_failure_modes.rs | 295 ++++++++ crates/clarion-storage/src/error.rs | 14 + crates/clarion-storage/src/pragma.rs | 68 +- crates/clarion-storage/src/schema.rs | 66 ++ crates/clarion-storage/src/writer.rs | 6 + crates/clarion-storage/tests/schema_apply.rs | 133 +++- .../00-coordination.md | 38 ++ .../01-discovery-findings.md | 586 ++++++++++++++++ .../02-subsystem-catalog.md | 632 ++++++++++++++++++ .../03-diagrams.md | 386 +++++++++++ .../04-final-report.md | 330 +++++++++ .../temp/answer-python-engineer.md | 125 ++++ .../temp/answer-quality-engineer.md | 102 +++ .../temp/answer-security-engineer.md | 65 ++ .../temp/answer-solution-architect.md | 94 +++ .../temp/answer-systems-thinker.md | 70 ++ .../temp/catalog-clarion-cli.md | 101 +++ .../temp/catalog-clarion-core.md | 91 +++ .../temp/catalog-clarion-mcp.md | 110 +++ .../temp/catalog-clarion-plugin-fixture.md | 80 +++ .../temp/catalog-clarion-scanner.md | 86 +++ .../temp/catalog-clarion-storage.md | 79 +++ .../temp/catalog-python-plugin.md | 69 ++ .../temp/task-catalog-template.md | 81 +++ .../temp/task-discovery.md | 95 +++ .../temp/validation-catalog.md | 291 ++++++++ .../temp/validation-final-report.md | 294 ++++++++ .../ADR-035-operational-tuning-discipline.md | 367 ++++++++++ .../clarion_plugin_python/pyright_session.py | 49 +- .../src/clarion_plugin_python/server.py | 8 +- plugins/python/tests/test_server.py | 128 +++- 32 files changed, 4930 insertions(+), 22 deletions(-) create mode 100644 crates/clarion-cli/tests/analyze_failure_modes.rs create mode 100644 docs/arch-analysis-2026-05-22-1924/00-coordination.md create mode 100644 docs/arch-analysis-2026-05-22-1924/01-discovery-findings.md create mode 100644 docs/arch-analysis-2026-05-22-1924/02-subsystem-catalog.md create mode 100644 docs/arch-analysis-2026-05-22-1924/03-diagrams.md create mode 100644 docs/arch-analysis-2026-05-22-1924/04-final-report.md create mode 100644 docs/arch-analysis-2026-05-22-1924/temp/answer-python-engineer.md create mode 100644 docs/arch-analysis-2026-05-22-1924/temp/answer-quality-engineer.md create mode 100644 docs/arch-analysis-2026-05-22-1924/temp/answer-security-engineer.md create mode 100644 docs/arch-analysis-2026-05-22-1924/temp/answer-solution-architect.md create mode 100644 docs/arch-analysis-2026-05-22-1924/temp/answer-systems-thinker.md create mode 100644 docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-cli.md create mode 100644 docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-core.md create mode 100644 docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-mcp.md create mode 100644 docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-plugin-fixture.md create mode 100644 docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-scanner.md create mode 100644 docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-storage.md create mode 100644 docs/arch-analysis-2026-05-22-1924/temp/catalog-python-plugin.md create mode 100644 docs/arch-analysis-2026-05-22-1924/temp/task-catalog-template.md create mode 100644 docs/arch-analysis-2026-05-22-1924/temp/task-discovery.md create mode 100644 docs/arch-analysis-2026-05-22-1924/temp/validation-catalog.md create mode 100644 docs/arch-analysis-2026-05-22-1924/temp/validation-final-report.md create mode 100644 docs/clarion/adr/ADR-035-operational-tuning-discipline.md diff --git a/crates/clarion-cli/src/http_read.rs b/crates/clarion-cli/src/http_read.rs index 8d8e855b..2ebc7757 100644 --- a/crates/clarion-cli/src/http_read.rs +++ b/crates/clarion-cli/src/http_read.rs @@ -1072,6 +1072,19 @@ fn classify_read_error(err: &StorageError) -> ReadError { code: ErrorCode::StorageError, message: "file lookup failed", }, + // STO-02 (ADR-035): the on-disk file is not a Clarion database, or + // was written by a newer build. Either condition is fatal for the + // server; the writer-actor refuses to spawn against it. Surfacing + // 500 here is defensive — in practice the HTTP API does not open + // its own writer, but the reader pool can encounter the same file + // header mismatches and we want a clear distinct response code. + StorageError::ForeignDatabase { .. } | StorageError::FutureUserVersion { .. } => { + ReadError { + status: StatusCode::INTERNAL_SERVER_ERROR, + code: ErrorCode::StorageError, + message: "file lookup storage rejected database header", + } + } StorageError::PoolInteract(_) | StorageError::WriterGone | StorageError::WriterProtocol(_) diff --git a/crates/clarion-cli/tests/analyze_failure_modes.rs b/crates/clarion-cli/tests/analyze_failure_modes.rs new file mode 100644 index 00000000..1760db75 --- /dev/null +++ b/crates/clarion-cli/tests/analyze_failure_modes.rs @@ -0,0 +1,295 @@ +//! Failure-mode integration tests for `clarion analyze`. +//! +//! These exercise the run-outcome promotion logic in +//! `crates/clarion-cli/src/analyze.rs::run_with_options` — specifically the +//! branch where the writer-actor rejects an `InsertEdge` mid-run and +//! `RunOutcome` must be set to `HardFailed` (→ `FailRun`) rather than +//! `SoftFailed` (→ `CommitRun(Failed)` with the full stats blob). +//! +//! The two paths differ in what the writer persists into `runs.stats`: +//! +//! - `HardFailed` (`FailRun`): stats is `{"failure_reason": "..."}` only — +//! no `entities_inserted` / `edges_inserted` keys. +//! - `SoftFailed` (`CommitRun(Failed)`): stats carries the full schema +//! (`entities_inserted`, `edges_inserted`, clustering, ...) plus a +//! `failure_reason` naming the plugin crash. +//! +//! Both end with `runs.status = 'failed'`, so the discriminator is the +//! *shape* of the stats blob, not the status column. + +#![cfg(unix)] + +use std::os::unix::fs::PermissionsExt; + +use assert_cmd::Command; +use rusqlite::Connection; + +fn clarion_bin() -> Command { + Command::cargo_bin("clarion").expect("clarion binary") +} + +/// Tiny Python fixture plugin that declares an edge kind which the writer +/// does not know about (`bogus_edge`). The host accepts it (it appears in +/// `[ontology].edge_kinds`); the writer's `enforce_edge_contract` rejects +/// it with `CLA-INFRA-EDGE-UNKNOWN-KIND` — a `StorageError::WriterProtocol` +/// surfaced from `Writer::send_wait` on the first `InsertEdge`. +/// +/// This is the most realistic deterministic trigger for a mid-run +/// writer-actor failure: passes core's validation, fails at the writer. +const BOGUS_EDGE_PLUGIN_SCRIPT: &str = r#"#!/usr/bin/python3 +import json +import pathlib +import sys + + +def read_frame(): + headers = {} + while True: + line = sys.stdin.buffer.readline() + if line in (b"", b"\r\n"): + break + name, value = line.decode("ascii").strip().split(":", 1) + headers[name.lower()] = value.strip() + length = int(headers["content-length"]) + return json.loads(sys.stdin.buffer.read(length)) + + +def write_frame(message): + body = json.dumps(message, separators=(",", ":")).encode("utf-8") + sys.stdout.buffer.write(b"Content-Length: " + str(len(body)).encode("ascii") + b"\r\n\r\n") + sys.stdout.buffer.write(body) + sys.stdout.buffer.flush() + + +while True: + msg = read_frame() + method = msg.get("method") + if method == "initialized": + continue + if method == "exit": + raise SystemExit(0) + ident = msg["id"] + if method == "initialize": + write_frame({ + "jsonrpc": "2.0", + "id": ident, + "result": { + "name": "clarion-plugin-bogus", + "version": "0.1.0", + "ontology_version": "0.6.0", + "capabilities": {}, + }, + }) + elif method == "analyze_file": + path = msg["params"]["file_path"] + stem = pathlib.Path(path).stem + module_id = f"bogusfixture:module:{stem}" + other_id = f"bogusfixture:module:{stem}_partner" + write_frame({ + "jsonrpc": "2.0", + "id": ident, + "result": { + "entities": [ + { + "id": module_id, + "kind": "module", + "qualified_name": stem, + "source": {"file_path": path}, + }, + { + "id": other_id, + "kind": "module", + "qualified_name": f"{stem}_partner", + "source": {"file_path": path}, + }, + ], + # `bogus_edge` is declared in the manifest's ontology, so + # `PluginHost::process_edges` accepts it; the writer's + # `enforce_edge_contract` rejects it with + # `CLA-INFRA-EDGE-UNKNOWN-KIND`. That is the mid-run + # writer-actor failure under test. + "edges": [ + { + "kind": "bogus_edge", + "from_id": module_id, + "to_id": other_id, + "source_byte_start": 0, + "source_byte_end": 4, + "confidence": "resolved", + }, + ], + "stats": {}, + }, + }) + elif method == "shutdown": + write_frame({"jsonrpc": "2.0", "id": ident, "result": {}}) + else: + raise SystemExit(1) +"#; + +const BOGUS_EDGE_PLUGIN_MANIFEST: &str = r#" +[plugin] +name = "clarion-plugin-bogus" +plugin_id = "bogusfixture" +version = "0.1.0" +protocol_version = "1.0" +executable = "clarion-plugin-bogus" +language = "bogusfixture" +extensions = ["bog"] + +[capabilities.runtime] +expected_max_rss_mb = 256 +expected_entities_per_file = 100 +wardline_aware = false +reads_outside_project_root = false + +[ontology] +entity_kinds = ["module"] +edge_kinds = ["bogus_edge"] +rule_id_prefix = "CLA-BOGUS-" +ontology_version = "0.6.0" +"#; + +fn write_bogus_edge_plugin(plugin_dir: &std::path::Path) { + let plugin_script = plugin_dir.join("clarion-plugin-bogus"); + std::fs::write(&plugin_script, BOGUS_EDGE_PLUGIN_SCRIPT) + .expect("write bogus edge plugin script"); + let mut perms = std::fs::metadata(&plugin_script) + .expect("stat bogus edge plugin") + .permissions(); + perms.set_mode(0o755); + std::fs::set_permissions(&plugin_script, perms).expect("chmod bogus edge plugin"); + + std::fs::write(plugin_dir.join("plugin.toml"), BOGUS_EDGE_PLUGIN_MANIFEST) + .expect("write bogus edge plugin manifest"); +} + +/// Seam test for the `SoftFailed` vs `HardFailed` branch in +/// `run_with_options` (analyze.rs ~lines 426-475, 519-601). +/// +/// A writer-actor `InsertEdge` rejection mid-run must promote the run to +/// `HardFailed` → `FailRun`. The run row must: +/// +/// 1. End with `status = 'failed'`. +/// 2. Carry a `stats.failure_reason` naming the writer-actor failure +/// (`CLA-INFRA-EDGE-UNKNOWN-KIND` from `enforce_edge_contract`), not a +/// plugin-crash reason. +/// 3. Have the minimal `FailRun` stats shape — no `entities_inserted` +/// key, distinguishing this from the `SoftFailed` path which writes the +/// full `CommitRun(Failed)` stats blob. +/// 4. Have NO `findings` rows tagged with the crash-loop `rule_id` — the +/// failure here is writer-side, not plugin-side. +/// +/// The process must exit non-zero so `analyze && next` chains and CI +/// gating work (regression for the same surface as +/// `analyze_failrun_exits_nonzero_with_run_row_marked_failed`). +#[test] +fn analyze_promotes_run_to_hard_failed_when_writer_actor_fails_mid_run() { + let project_dir = tempfile::tempdir().unwrap(); + let plugin_dir = tempfile::tempdir().unwrap(); + write_bogus_edge_plugin(plugin_dir.path()); + + clarion_bin() + .args(["install", "--path"]) + .arg(project_dir.path()) + .env("PATH", "") + .assert() + .success(); + std::fs::write(project_dir.path().join("demo.bog"), b"module\n").expect("write demo.bog"); + + // Scrub PATH so only the bogus plugin is discovered. A second + // discovered plugin would muddy the assertion: a healthy plugin + // *after* the bogus one cannot run (writer-actor failure breaks the + // 'plugins loop), but a healthy plugin *before* would insert + // entities and edges first, changing the stats-shape discriminator. + let plugin_path = + std::env::join_paths(std::iter::once(plugin_dir.path().to_path_buf())).unwrap(); + let out = clarion_bin() + .args(["analyze"]) + .arg(project_dir.path()) + .env("PATH", &plugin_path) + .assert() + .failure(); + let stderr = String::from_utf8(out.get_output().stderr.clone()).unwrap(); + assert!( + stderr.contains("failed"), + "stderr should mention failure; got: {stderr}" + ); + + let conn = Connection::open(project_dir.path().join(".clarion/clarion.db")).unwrap(); + + // (1) Run row marked failed. + let (run_status, run_stats_raw): (String, String) = conn + .query_row( + "SELECT status, stats FROM runs ORDER BY started_at DESC LIMIT 1", + [], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .expect("query latest run row"); + assert_eq!( + run_status, "failed", + "writer-actor mid-run failure must mark the run row failed" + ); + + let stats: serde_json::Value = + serde_json::from_str(&run_stats_raw).expect("parse runs.stats JSON"); + + // (2) failure_reason names the writer-actor rejection, not a plugin crash. + let failure_reason = stats["failure_reason"] + .as_str() + .expect("HardFailed stats must contain failure_reason"); + assert!( + failure_reason.contains("CLA-INFRA-EDGE-UNKNOWN-KIND"), + "failure_reason should cite the writer's edge-contract code; \ + got: {failure_reason}" + ); + assert!( + failure_reason.contains("InsertEdge"), + "failure_reason should name the InsertEdge surface; got: {failure_reason}" + ); + assert!( + !failure_reason.contains("plugin(s) crashed") && !failure_reason.contains("panicked"), + "writer-actor failure must not be reported as a plugin crash; \ + got: {failure_reason}" + ); + + // (3) Stats shape is the minimal FailRun blob — no SoftFailed keys. + // SoftFailed's `CommitRun(Failed)` writes the full stats schema with + // `entities_inserted`, `edges_inserted`, clustering, etc.; FailRun + // writes only `{"failure_reason": ...}`. Asserting the absence of + // SoftFailed-only keys is the load-bearing discriminator between the + // two branches. + assert!( + stats.get("entities_inserted").is_none(), + "FailRun stats must not contain entities_inserted (SoftFailed key); \ + got: {run_stats_raw}" + ); + assert!( + stats.get("edges_inserted").is_none(), + "FailRun stats must not contain edges_inserted (SoftFailed key); \ + got: {run_stats_raw}" + ); + assert!( + stats.get("clustering").is_none(), + "FailRun stats must not contain clustering (SoftFailed key); \ + got: {run_stats_raw}" + ); + + // (4) No crash-loop finding rows — the failure here is writer-side. + // The crash-loop breaker only ticks on plugin crashes; it must not + // have tripped, and no row tagged with the crash-loop `rule_id` may + // appear. + let crash_loop_findings: i64 = conn + .query_row( + "SELECT COUNT(*) FROM findings \ + WHERE rule_id = 'CLA-INFRA-PLUGIN-DISABLED-CRASH-LOOP'", + [], + |row| row.get(0), + ) + .expect("query crash-loop findings count"); + assert_eq!( + crash_loop_findings, 0, + "no FINDING_DISABLED_CRASH_LOOP rows should exist; \ + writer-actor failure must not tick the crash-loop breaker" + ); +} diff --git a/crates/clarion-storage/src/error.rs b/crates/clarion-storage/src/error.rs index d9a775dc..c69b337f 100644 --- a/crates/clarion-storage/src/error.rs +++ b/crates/clarion-storage/src/error.rs @@ -19,6 +19,20 @@ pub enum StorageError { #[error("PRAGMA invariant violated: {0}")] PragmaInvariant(String), + #[error( + "CLA-INFRA-STORAGE-FOREIGN-DB: refusing to open SQLite file with \ + application_id={application_id:#010x}; Clarion databases carry \ + application_id=0x434C524E (\"CLRN\")" + )] + ForeignDatabase { application_id: u32 }, + + #[error( + "CLA-INFRA-STORAGE-FUTURE-DB: refusing to open SQLite file with \ + user_version={found} (greater than current schema version {current}); \ + the database was written by a newer Clarion build" + )] + FutureUserVersion { found: u32, current: u32 }, + #[error("migration {version} failed: {source}")] Migration { version: u32, diff --git a/crates/clarion-storage/src/pragma.rs b/crates/clarion-storage/src/pragma.rs index 29f1d2f6..90743b30 100644 --- a/crates/clarion-storage/src/pragma.rs +++ b/crates/clarion-storage/src/pragma.rs @@ -1,18 +1,53 @@ //! PRAGMAs applied at connection open per ADR-011 §`SQLite` PRAGMAs. +//! +//! # Operational tuning discipline (per ADR-035, in-flight at v1.0 tag-cut) +//! +//! - **Stated basis**: ADR-011 §`SQLite` PRAGMAs fixes the durability + +//! concurrency posture (`journal_mode=WAL` + `synchronous=NORMAL` + +//! `busy_timeout=5000` + `wal_autocheckpoint=1000` + `foreign_keys=ON`). +//! The `application_id=0x434C524E` ("CLRN") and `user_version` PRAGMAs +//! close gap STO-02 from `docs/implementation/v1.0-tag-cut/gap-register.md`: +//! they give `.clarion/clarion.db` a self-identifying on-disk header so +//! `file(1)` / `sqlite3 .dbinfo` / a future migration runner can refuse +//! foreign or forward-incompatible files. +//! - **Override surface**: **recompile-only.** None of these PRAGMAs are +//! user-tunable at runtime — they encode the storage contract. Any +//! override is a source-code change reviewed against the ADR that +//! established the value. +//! - **Retune trigger**: the on-disk format identifier (`application_id`) +//! may only change under a new ADR that supersedes ADR-035; the +//! `user_version` is bumped automatically by the migration runner +//! (`schema::apply_migrations`) and otherwise treated as immutable. use rusqlite::Connection; use crate::error::{Result, StorageError}; +/// `SQLite` `application_id` header value identifying Clarion databases. +/// +/// ASCII "CLRN" — picked so `file(1)` / `sqlite3 .dbinfo` distinguishes a +/// Clarion DB from any other `SQLite` file. Set lazily on first open of a +/// fresh (`application_id = 0`) file. Refusing to open a file with any +/// other non-zero value is how we close STO-02. +pub const CLARION_APPLICATION_ID: u32 = 0x434C_524E; + /// Apply the write-side PRAGMA set: WAL, `synchronous=NORMAL`, `busy_timeout`, /// `wal_autocheckpoint`, `foreign_keys`. Called on the writer's connection once, /// immediately after open. /// +/// Also enforces the `application_id` identity check (STO-02): a file whose +/// `application_id` is neither `0` (fresh / legacy) nor +/// [`CLARION_APPLICATION_ID`] is rejected with +/// [`StorageError::ForeignDatabase`]. A zero value is upgraded in place +/// (`PRAGMA application_id = ...`) so re-opens recognise the file. +/// /// # Errors /// -/// Returns [`crate::error::StorageError::Sqlite`] if any PRAGMA statement fails. -/// Returns [`crate::error::StorageError::PragmaInvariant`] if WAL mode is not +/// Returns [`StorageError::Sqlite`] if any PRAGMA statement fails. +/// Returns [`StorageError::PragmaInvariant`] if WAL mode is not /// confirmed after the `PRAGMA journal_mode = WAL` command. +/// Returns [`StorageError::ForeignDatabase`] if the file carries a +/// non-zero `application_id` that is not Clarion's. pub fn apply_write_pragmas(conn: &Connection) -> Result<()> { let mode: String = conn.query_row("PRAGMA journal_mode = WAL", [], |row| row.get(0))?; if !mode.eq_ignore_ascii_case("wal") { @@ -21,6 +56,7 @@ pub fn apply_write_pragmas(conn: &Connection) -> Result<()> { ADR-011's synchronous=NORMAL durability posture requires WAL" ))); } + enforce_application_id(conn)?; conn.execute_batch(concat!( "PRAGMA synchronous = NORMAL;", "PRAGMA busy_timeout = 5000;", @@ -43,3 +79,31 @@ pub fn apply_read_pragmas(conn: &Connection) -> Result<()> { ))?; Ok(()) } + +/// Read `application_id`; on `0` set it to [`CLARION_APPLICATION_ID`]; on +/// [`CLARION_APPLICATION_ID`] continue; on any other value refuse with +/// [`StorageError::ForeignDatabase`]. +/// +/// `SQLite` stores `application_id` as a signed 32-bit integer; rusqlite +/// surfaces it as `i64`. We read into `i64` and reinterpret via +/// `as u32` so values with the high bit set (negative `i32`) still compare +/// against [`CLARION_APPLICATION_ID`] correctly. +fn enforce_application_id(conn: &Connection) -> Result<()> { + let raw: i64 = conn.query_row("PRAGMA application_id", [], |row| row.get(0))?; + // i64 -> u32: SQLite caps application_id at 32 bits. Truncating cast is + // the documented round-trip; values outside u32 should not reach us. + #[allow(clippy::cast_possible_truncation, clippy::cast_sign_loss)] + let current = raw as u32; + match current { + 0 => { + conn.execute_batch(&format!( + "PRAGMA application_id = {CLARION_APPLICATION_ID};" + ))?; + Ok(()) + } + id if id == CLARION_APPLICATION_ID => Ok(()), + other => Err(StorageError::ForeignDatabase { + application_id: other, + }), + } +} diff --git a/crates/clarion-storage/src/schema.rs b/crates/clarion-storage/src/schema.rs index 1a0adff6..9f6bf02a 100644 --- a/crates/clarion-storage/src/schema.rs +++ b/crates/clarion-storage/src/schema.rs @@ -20,16 +20,39 @@ const MIGRATIONS: &[Migration] = &[Migration { sql: include_str!("../migrations/0001_initial_schema.sql"), }]; +/// Highest migration version known to this build. Mirrored into the +/// `SQLite` `user_version` header (STO-02) so a future-built database is +/// refused at open instead of silently corrupting state. +pub const CURRENT_SCHEMA_VERSION: u32 = 1; + +const _CURRENT_SCHEMA_VERSION_MATCHES_LAST_MIGRATION: () = { + // Compile-time check: `CURRENT_SCHEMA_VERSION` must equal the highest + // version in `MIGRATIONS`. If a new migration is added without bumping + // the constant (or vice versa), this assertion fails to compile. + assert!( + MIGRATIONS[MIGRATIONS.len() - 1].version == CURRENT_SCHEMA_VERSION, + "CURRENT_SCHEMA_VERSION must equal the highest MIGRATIONS[].version" + ); +}; + /// Apply every migration not already recorded in `schema_migrations`. /// /// The first migration creates the `schema_migrations` table itself, so the /// initial lookup tolerates its absence. /// +/// After all pending migrations apply, the `SQLite` header `user_version` is +/// written to [`CURRENT_SCHEMA_VERSION`]. A `user_version` strictly greater +/// than [`CURRENT_SCHEMA_VERSION`] at entry is refused via +/// [`verify_user_version`] (closes STO-02 forward-incompatibility check). +/// /// # Errors /// +/// Returns [`StorageError::FutureUserVersion`] if the database was written +/// by a newer Clarion build. /// Returns [`StorageError::Migration`] with the failing version on SQL error /// during apply. Returns [`StorageError::Sqlite`] on bookkeeping failures. pub fn apply_migrations(conn: &mut Connection) -> Result<()> { + verify_user_version(conn)?; let applied = read_applied_versions(conn)?; for m in MIGRATIONS { if applied.contains(&m.version) { @@ -38,6 +61,49 @@ pub fn apply_migrations(conn: &mut Connection) -> Result<()> { } apply_one(conn, m)?; } + apply_user_version(conn)?; + Ok(()) +} + +/// Refuse to operate on a database whose `user_version` is strictly greater +/// than [`CURRENT_SCHEMA_VERSION`]. +/// +/// Equal or less is accepted: equal means the schema is current, less means +/// either a fresh DB (`user_version=0`) or a DB awaiting in-flight migrations +/// — both are handled by [`apply_migrations`]. The writer-actor calls this +/// directly (without invoking the migration runner) so a forward-incompatible +/// file is rejected at `Writer::spawn` time. +/// +/// # Errors +/// +/// Returns [`StorageError::FutureUserVersion`] when `user_version > +/// CURRENT_SCHEMA_VERSION`. Returns [`StorageError::Sqlite`] if the PRAGMA +/// query fails. +pub fn verify_user_version(conn: &Connection) -> Result<()> { + let raw: i64 = conn.query_row("PRAGMA user_version", [], |row| row.get(0))?; + // SQLite stores user_version as a 32-bit integer; rusqlite returns i64. + // Negative values are unreachable in normal use (we only set u32 values); + // clamp via `try_from` so an out-of-range value surfaces explicitly + // rather than silently truncating. + let found = u32::try_from(raw).map_err(|_| { + StorageError::PragmaInvariant(format!( + "PRAGMA user_version returned out-of-range value {raw}; expected 0..=u32::MAX" + )) + })?; + if found > CURRENT_SCHEMA_VERSION { + return Err(StorageError::FutureUserVersion { + found, + current: CURRENT_SCHEMA_VERSION, + }); + } + Ok(()) +} + +/// Write `PRAGMA user_version = CURRENT_SCHEMA_VERSION`. Idempotent — writing +/// the same value is cheap (it touches the `SQLite` header page). Called after +/// the migration runner has applied every pending migration. +fn apply_user_version(conn: &Connection) -> Result<()> { + conn.execute_batch(&format!("PRAGMA user_version = {CURRENT_SCHEMA_VERSION};"))?; Ok(()) } diff --git a/crates/clarion-storage/src/writer.rs b/crates/clarion-storage/src/writer.rs index 6dbe3594..39b60d98 100644 --- a/crates/clarion-storage/src/writer.rs +++ b/crates/clarion-storage/src/writer.rs @@ -29,6 +29,7 @@ use crate::commands::{ }; use crate::error::{Result, StorageError}; use crate::pragma; +use crate::schema; use crate::unresolved::replace_unresolved_call_sites_for_caller; /// Default transaction batch size per ADR-011. @@ -86,6 +87,11 @@ impl Writer { let handle = tokio::task::spawn_blocking(move || -> Result<()> { let mut conn = Connection::open(&db_path)?; pragma::apply_write_pragmas(&conn)?; + // STO-02: refuse a database whose `user_version` is strictly greater + // than CURRENT_SCHEMA_VERSION. Equal/less are normal — equal is the + // already-migrated steady state, less is handled by the migration + // runner (which `install` calls before the writer ever spawns). + schema::verify_user_version(&conn)?; run_actor( rx, &mut conn, diff --git a/crates/clarion-storage/tests/schema_apply.rs b/crates/clarion-storage/tests/schema_apply.rs index 1c2969ee..e2ecb3ba 100644 --- a/crates/clarion-storage/tests/schema_apply.rs +++ b/crates/clarion-storage/tests/schema_apply.rs @@ -6,7 +6,7 @@ use rusqlite::{Connection, params}; -use clarion_storage::{pragma, schema}; +use clarion_storage::{Writer, error::StorageError, pragma, schema}; fn open_fresh(tempdir: &tempfile::TempDir) -> Connection { let path = tempdir.path().join("clarion.db"); @@ -918,3 +918,134 @@ fn runs_status_check_accepts_all_documented_values() { .unwrap_or_else(|err| panic!("runs.status={status} rejected unexpectedly: {err}")); } } + +// ---------------------------------------------------------------------------- +// STO-02 (gap-register.md): the writer must self-identify Clarion databases +// via SQLite's `application_id` header and refuse forward-incompatible +// `user_version` values. These tests pin the open-time contract. +// ---------------------------------------------------------------------------- + +/// Spawn a writer, immediately shut it down, and return whatever Result the +/// blocking task produced. Used to assert refuse-on-open behaviour without +/// leaking the spawned task. +fn spawn_writer_and_drain(path: std::path::PathBuf) -> Result<(), StorageError> { + let rt = tokio::runtime::Builder::new_multi_thread() + .worker_threads(2) + .enable_all() + .build() + .expect("tokio runtime"); + rt.block_on(async move { + let (writer, handle) = Writer::spawn(path, 50, 256)?; + // Close the command channel so the actor (if it reached the loop) + // exits cleanly; the join then surfaces the open-time error if any. + drop(writer); + handle.await.expect("writer task did not panic") + }) +} + +#[test] +fn open_refuses_db_with_foreign_application_id() { + let tempdir = tempfile::tempdir().unwrap(); + let path = tempdir.path().join("foreign.db"); + { + let conn = Connection::open(&path).unwrap(); + // `PRAGMA application_id` only writes the header page to disk once + // the database file has been materialised by some other write. Touch + // a temporary table to force the file out of zero-bytes state. + conn.execute_batch( + "PRAGMA application_id = 0x7AFEBABE; \ + CREATE TABLE _touch (x INTEGER); DROP TABLE _touch;", + ) + .expect("set foreign application_id"); + } + let err = spawn_writer_and_drain(path).expect_err( + "Writer::spawn must refuse a SQLite file carrying a non-Clarion application_id", + ); + assert!( + matches!( + err, + StorageError::ForeignDatabase { + application_id: 0x7AFE_BABE, + } + ), + "expected ForeignDatabase {{ application_id: 0x7AFEBABE }}, got {err:?}" + ); +} + +#[test] +fn open_refuses_db_from_future_user_version() { + let tempdir = tempfile::tempdir().unwrap(); + let path = tempdir.path().join("future.db"); + // Open the file via the normal writer path first so it carries the + // Clarion application_id and the v1 schema (the migration runner sets + // user_version=1 on apply). Then bump user_version past current and + // re-open via the writer — must refuse. + { + let mut conn = Connection::open(&path).unwrap(); + pragma::apply_write_pragmas(&conn).unwrap(); + schema::apply_migrations(&mut conn).unwrap(); + } + { + let conn = Connection::open(&path).unwrap(); + conn.execute_batch(&format!( + "PRAGMA user_version = {};", + schema::CURRENT_SCHEMA_VERSION + 1 + )) + .expect("bump user_version"); + } + + let err = spawn_writer_and_drain(path) + .expect_err("Writer::spawn must refuse a future-versioned database"); + let expected_found = schema::CURRENT_SCHEMA_VERSION + 1; + let expected_current = schema::CURRENT_SCHEMA_VERSION; + assert!( + matches!( + err, + StorageError::FutureUserVersion { found, current } + if found == expected_found && current == expected_current + ), + "expected FutureUserVersion {{ found: {expected_found}, current: \ + {expected_current} }}, got {err:?}" + ); +} + +#[test] +fn open_sets_application_id_on_legacy_db() { + let tempdir = tempfile::tempdir().unwrap(); + let path = tempdir.path().join("legacy.db"); + // Touch a SQLite file with no application_id set (default 0). Open as a + // raw connection so we leave the header at its zero default — no + // `apply_write_pragmas` here. + { + let conn = Connection::open(&path).unwrap(); + let raw: i64 = conn + .query_row("PRAGMA application_id", [], |row| row.get(0)) + .unwrap(); + assert_eq!(raw, 0, "fresh SQLite files start at application_id=0"); + // Force the file to materialise on disk by writing a table; this also + // guarantees the header page exists. + conn.execute_batch("CREATE TABLE _touch (x INTEGER); DROP TABLE _touch;") + .unwrap(); + } + + // First open via the writer should set the Clarion application_id. + spawn_writer_and_drain(path.clone()) + .expect("Writer::spawn must accept a legacy (application_id=0) file"); + { + let conn = Connection::open(&path).unwrap(); + let raw: i64 = conn + .query_row("PRAGMA application_id", [], |row| row.get(0)) + .unwrap(); + #[allow(clippy::cast_possible_truncation, clippy::cast_sign_loss)] + let observed = raw as u32; + assert_eq!( + observed, + pragma::CLARION_APPLICATION_ID, + "writer must stamp Clarion application_id on a legacy DB" + ); + } + + // Re-open: must not refuse, application_id is now recognised. + spawn_writer_and_drain(path) + .expect("Writer::spawn must accept a database it has already stamped"); +} diff --git a/docs/arch-analysis-2026-05-22-1924/00-coordination.md b/docs/arch-analysis-2026-05-22-1924/00-coordination.md new file mode 100644 index 00000000..bbade7e0 --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/00-coordination.md @@ -0,0 +1,38 @@ +# Coordination Plan — Clarion Architecture Analysis + +**Workspace:** `docs/arch-analysis-2026-05-22-1924/` +**Date:** 2026-05-22 +**Operator:** Claude (system-archaeologist skill) +**Source instruction:** "Full from-scratch analysis, do not read existing documentation." + +## Analysis Configuration + +- **Scope:** Entire repo source tree — `crates/` (Rust workspace) and `plugins/python/` (Python plugin). Excludes `target/`, `.venv/`, `.git/`, caches. +- **Deliverables (Option A — Full Analysis):** + - `01-discovery-findings.md` — holistic assessment + - `02-subsystem-catalog.md` — per-subsystem entries + - `03-diagrams.md` — C4 architecture diagrams + - `04-final-report.md` — synthesized executive narrative +- **Strategy:** PARALLEL per-subsystem explorers, then sequential validation + diagrams + report. + - Rationale: 7 candidate subsystems are loosely coupled (separate crates + a plugin), well-suited to independent exploration. +- **Tier:** Standard (not ultralarge). LOC ~50K, file count ~85 source files, subsystems = 7. +- **Constraint:** **Do not consult existing design docs** (`docs/clarion/**`, `docs/suite/**`, ADRs, sprint READMEs, CLAUDE.md design content). Findings must derive from code only. CLAUDE.md operational sections (tooling, build commands) remain in-scope as orientation; design narrative there is treated as "existing documentation" and avoided. +- **Complexity:** Medium. Plugin host / RPC transport / federation HTTP API are non-trivial; storage actor model and entity-ID assembly need careful inspection. + +## Subsystem Partition (initial candidates) + +1. **clarion-core** — Plugin host, transport, manifest, jail/limits, breaker, entity-ID assembler, LLM provider. +2. **clarion-storage** — Writer-actor + reader-pool over SQLite (schema, pragma, query, cache, unresolved). +3. **clarion-cli** — `clarion` binary (`install`, `analyze`, `serve`, `secret-scan`), clustering, HTTP read API, run lifecycle, stats. +4. **clarion-mcp** — MCP server façade (filigree client, config). +5. **clarion-scanner** — Secret/entropy/pattern scanner with baseline support. +6. **clarion-plugin-fixture** — Test fixture plugin (also a reference impl of the protocol). +7. **plugins/python** — Python language plugin (function/class entity extraction, qualname, Wardline probe). + +## Execution Log + +- 2026-05-22 19:24 — Created workspace `docs/arch-analysis-2026-05-22-1924/`. +- 2026-05-22 19:24 — Scale check: 6 crates (~44K Rust LOC), Python plugin (~6.5K LOC), 121 md files. Standard tier confirmed. +- 2026-05-22 19:25 — User selected **Option A — Full Analysis**. +- 2026-05-22 19:25 — Wrote this coordination plan. Strategy: PARALLEL per-subsystem explorers. +- Next: holistic discovery sweep (entry points, build graph, dependency edges) → `01-discovery-findings.md`. diff --git a/docs/arch-analysis-2026-05-22-1924/01-discovery-findings.md b/docs/arch-analysis-2026-05-22-1924/01-discovery-findings.md new file mode 100644 index 00000000..a0f3c857 --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/01-discovery-findings.md @@ -0,0 +1,586 @@ +# 01 — Discovery Findings (Clarion) + +> Methodology note: produced from source (Rust + Python), `Cargo.toml`, +> `plugin.toml`, `pyproject.toml`, `clarion.yaml`, `.mcp.json`, migration SQL, +> e2e shell scripts, and per-crate test files only. No design docs, ADRs, or +> sprint/WP narrative were read while writing this document. + +## 1. One-Paragraph Pitch (Inferred From Code) + +Clarion is a Rust-implemented code-archaeology toolchain: a `clarion` +binary (`crates/clarion-cli/src/main.rs`) that (a) installs a per-project +`.clarion/clarion.db` SQLite store with embedded migrations +(`install.rs` + `crates/clarion-storage/migrations/0001_initial_schema.sql`), +(b) walks a source tree, discovers `clarion-plugin-*` executables on `$PATH`, +spawns each as a JSON-RPC subprocess, ingests entities/edges they extract, and +persists them through a writer-actor (`analyze.rs`, `clarion-storage/writer.rs`), +and (c) `clarion serve` runs as a stdio MCP server exposing **19** +agent-facing tools (`crates/clarion-mcp/src/lib.rs::list_tools`; original +discovery said "twenty" — corrected during validation against the actual +`ToolDefinition` registry; see §4 of `02-subsystem-catalog.md`) plus an Axum +HTTP read API on `/api/v1/files*` (`crates/clarion-cli/src/http_read.rs:347`). +Out-of-tree language plugins follow the protocol defined in +`crates/clarion-core/src/plugin/protocol.rs`; the in-repo reference plugin +(`plugins/python/`) extracts module/class/function entities plus +`contains`/`calls`/`references`/`imports` edges using a `pyright` language-server +session for type-resolved call/reference targets. Audience is consult-mode +LLM agents and developers needing structured navigation of a codebase. + +## 2. Repository Layout + +``` +clarion/ +├── Cargo.toml Rust workspace, resolver "3", 6 members +├── clarion.yaml User-edited runtime config (LLM, integrations, serve.http) +├── .mcp.json Registers clarion + filigree as stdio MCP servers +├── rust-toolchain.toml channel = stable, profile = minimal +├── crates/ +│ ├── clarion-core/ ~11.7k LOC src, 325 LOC tests — domain types, plugin host +│ ├── clarion-storage/ ~3.2k LOC src, 4.86k LOC tests — SQLite layer, writer-actor +│ ├── clarion-cli/ ~6.8k LOC src, 6.4k LOC tests — `clarion` binary +│ ├── clarion-mcp/ ~6.6k LOC src, 2.2k LOC tests — MCP protocol surface +│ ├── clarion-scanner/ ~880 LOC src, 655 LOC tests — pre-ingest secret scanner +│ └── clarion-plugin-fixture/ 187 LOC — test-only Rust plugin +├── plugins/python/ +│ ├── plugin.toml Plugin manifest (plugin_id=python, ontology v0.6.0) +│ ├── pyproject.toml name=clarion-plugin-python, requires-python>=3.11 +│ ├── src/clarion_plugin_python/ 3028 LOC across 11 modules +│ └── tests/ 9 pytest files, 3440 LOC +├── fixtures/entity_id.json Cross-language byte-for-byte parity fixture +├── tests/ +│ ├── e2e/ 4 bash scripts (sprint_1, sprint_2_mcp, phase3, wp5_secret_scan) +│ └── perf/ b8_scale_test driver + elspeth_mini corpus (~150 Python files) +├── scripts/ CI helpers (b4 gate, governance, migration retirement, version lockstep) +├── .pre-commit-config.yaml ruff, ruff-format, mypy --strict for plugins/python +└── .github/workflows/ ci.yml + release.yml +``` + +Total Rust source: ~29k LOC across 6 crates (largest single files: +`clarion-mcp/src/lib.rs` 4703, `clarion-core/src/plugin/host.rs` 2935, +`clarion-cli/src/analyze.rs` 2549, `clarion-core/src/llm_provider.rs` 2467). +Total Python source: 3028 LOC, dominated by `pyright_session.py` 1406 and +`extractor.py` 932. + +## 3. Technology Stack + +- **Languages:** + - Rust: edition `2024`, MSRV `1.88`, toolchain channel `stable` + (`Cargo.toml:17`, `rust-toolchain.toml`). + - Python: `>=3.11` (`plugins/python/pyproject.toml:10`); ruff target + `py311`; mypy `python_version = "3.11"` with `strict = true`. + +- **Rust dependencies (key, from root `Cargo.toml` workspace.dependencies):** + - `tokio` 1 — multi-thread runtime + sync primitives (`rt-multi-thread`, + `macros`, `net`, `sync`, `time`). + - `rusqlite` 0.31 with `bundled` SQLite — embedded DB engine. + - `deadpool-sqlite` 0.8 (`rt_tokio_1`) — async reader pool. + - `axum` 0.7 + `tower` 0.5 + `tower-http` 0.6 — HTTP read API. + - `reqwest` 0.12 with `rustls-tls-native-roots`, `blocking`, `json` — + outbound HTTP (Filigree, OpenRouter). + - `clap` 4 (`derive`) — CLI parser. + - `serde`/`serde_json`/`serde_norway` (YAML) — config + wire formats. + - `tracing` + `tracing-subscriber` (`env-filter`) — structured logs. + - `thiserror` + `anyhow` — typed library errors / binary errors. + - `blake3`, `sha1`, `sha2` — content/secret hashing. + - `xgraph` 2.0.0 — used by `clarion-cli/src/clustering.rs` for Leiden + community detection (`xgraph::graph::algorithms::leiden_clustering`). + - `regex` 1, `ignore` 0.4 — secret-scanner patterns and file walking. + - `nix` 0.28 (`resource`) — `setrlimit(2)` for plugin sandboxing. + - `which` 6 — locate plugins on `$PATH`. + - `cargo-deny` policy file (`deny.toml`) checked in CI. + +- **Python dependencies (`plugins/python/pyproject.toml`):** + - Runtime: `packaging>=24`, `pyright==1.1.409` (pinned exact for the + type-resolved call/reference probe). + - Dev: `pytest>=8.0`, `pytest-cov>=5.0`, `ruff>=0.6`, `mypy>=1.11`, + `pre-commit>=3.8`. + +- **External processes / services:** + - **SQLite** — single file at `.clarion/clarion.db`; WAL mode forced by + `crates/clarion-storage/src/pragma.rs:17`; one migration `0001_initial_schema.sql`. + - **Plugin subprocesses** — spawned by `PluginHost::spawn` over + stdin/stdout JSON-RPC (`crates/clarion-core/src/plugin/host.rs`), + discovered by `discover()` scanning `$PATH` for `clarion-plugin-*` + (`crates/clarion-core/src/plugin/discovery.rs`). + - **pyright language server** — invoked as a subprocess by the Python + plugin (`plugins/python/src/clarion_plugin_python/pyright_session.py:7-9` + imports `select`, `subprocess`). + - **MCP stdio transport** — `clarion serve` exposes JSON-RPC 2.0 over + stdin/stdout with `protocolVersion = "2025-11-25"` + (`crates/clarion-mcp/src/lib.rs:40`); reads Content-Length frames via + `clarion_core::plugin::transport`. + - **HTTP read API** — bound by `serve.http.bind` (default `127.0.0.1:9111` + per `clarion.yaml`); Axum router at `http_read.rs:347-369` with + `ConcurrencyLimitLayer(64)` + 10s timeout + `LoadShedLayer` + + `CatchPanicLayer` + body-size limit. Six routes total. + - **Filigree** — outbound HTTP via `reqwest` in + `crates/clarion-mcp/src/filigree.rs`; reads entity-association reverse + lookups; transport sibling, not vendored. + - **LLM providers** — pluggable via `clarion-core/src/llm_provider.rs`: + `OpenRouterProvider` (HTTPS to `openrouter.ai/api/v1`), + `CodexCliProvider` and `ClaudeCliProvider` (subprocess execution of + `codex` / `claude` binaries), `RecordingProvider` (fixture replay). + - **Wardline** — Python-side import probe only + (`plugins/python/src/clarion_plugin_python/wardline_probe.py`): + `importlib.import_module("wardline.core.registry")`, version-pin + check against `[1.0.0, 2.0.0)` declared in `plugin.toml:54-56`. + +- **Build/test tooling:** + - **Pre-commit** (`.pre-commit-config.yaml`): ruff fix, ruff-format, + mypy `--strict` on `plugins/python/{src,tests}`. + - **Workspace lints** (`Cargo.toml:19-30`): `unsafe_code = "deny"`, + clippy `pedantic = "warn"`, with three pragmatic allows. + - **cargo-deny** policy at `deny.toml`. + - **GitHub Actions** at `.github/workflows/{ci,release}.yml` (not read for + contents, only existence). + - **Scripts** under `scripts/`: `check-workspace-version-lockstep.py`, + `check-python-ontology-version.py`, `check-migration-retirement.py`, + `check-github-release-governance.py`, `b4-gate-run.sh`. + +## 4. Entry Points + +- **`clarion` binary** — `crates/clarion-cli/src/main.rs`; `clap` parser at + `src/cli.rs:7`. Three subcommands: + - `Install { force, path }` → `install::run(&path, force)` + (`install.rs:1-50`) creates `.clarion/clarion.db`, `.clarion/config.json`, + `.clarion/.gitignore`, and `/clarion.yaml`. + - `Analyze { path, config, allow_unredacted_secrets, … }` → runs on a + multi-thread tokio runtime, performs `secret_scan` gate, then + `analyze::run_with_options` (`analyze.rs:70`+). + - `Serve { path, config }` → `serve::run(&path, config.as_deref())` + (`serve.rs:20`). +- **`clarion serve`** is the MCP server. It supervises *two* concurrent + servers in one process: an stdio JSON-RPC loop on a dedicated thread with + a per-thread current-thread tokio runtime + (`serve.rs:96-110, 126-130`), and the optional Axum HTTP read server on a + second thread (`serve.rs:55-71, http_read.rs::spawn`). Both share one + `ReaderPool`; identity is asserted via `Arc::ptr_eq` (`serve.rs:63-71`). +- **`clarion-plugin-fixture`** — `crates/clarion-plugin-fixture/src/main.rs`; + test-only binary that speaks the JSON-RPC plugin protocol, returns one + hard-coded `fixture:widget:demo.sample` entity per `analyze_file`, and + honours an `CLARION_FIXTURE_EXCEED_RLIMIT_AS` env hook to provoke OOM + paths (`main.rs:78, 137-178`). +- **Python plugin** — console script `clarion-plugin-python` defined in + `pyproject.toml:33`; `clarion_plugin_python.__main__:main` (15 LOC at + `__main__.py`) installs `stdout_guard` and delegates to + `clarion_plugin_python.server.main` (`server.py` 296 LOC). Plugin manifest + ships via wheel `shared-data` to `share/clarion/plugins/python/plugin.toml` + (`pyproject.toml:38-44`) for the host's install-prefix discovery fallback. +- **Library entry** — `crates/clarion-core/src/lib.rs` re-exports a facade of + domain types (`EntityId`, `Manifest`, `PluginHost`, `LlmProvider`, + `LeafSummaryPromptInput`, etc.) per its module-doc "Re-export policy". +- **Storage library entry** — `crates/clarion-storage/src/lib.rs` exposes + `Writer`, `ReaderPool`, the `WriterCmd` enum, cache helpers, and twenty-odd + query functions. + +## 5. Public Wire Surfaces + +1. **Plugin JSON-RPC 2.0 protocol** (stdin/stdout per plugin subprocess). + - Defined in `crates/clarion-core/src/plugin/protocol.rs` (875 LOC) + + `transport.rs` (LSP-style `Content-Length: N\r\n\r\n` framing). + - Five methods: `initialize`, `initialized` (notification), + `analyze_file`, `shutdown`, `exit` (notification). + - `analyze_file` returns `{entities: [...], edges: [...], stats: {…}}`; + entity shape carries `id, kind, qualified_name, source.{file_path, + source_range, …}`; edges carry `kind, src_id, dst_id, confidence, + properties` plus `unresolved_call_sites` for query-time inference. + - Frame ceiling: `ContentLengthCeiling::DEFAULT` = 8 MiB + (`limits.rs` per the ADR-021 §2b comment, also mirrored on the + plugin side in `plugins/python/.../server.py:48`). + - `initialize` returns + `{name, version, ontology_version, capabilities}`; the host validates + the returned ontology against `manifest.ontology.entity_kinds / + edge_kinds` (host.rs module docs §Enforcement pipeline step 1). + +2. **MCP stdio JSON-RPC server** (`clarion serve`). + - Same `Content-Length` framing (`clarion-mcp/src/lib.rs` imports + `clarion_core::plugin::{ContentLengthCeiling, Frame, TransportError}`). + - `MCP_PROTOCOL_VERSION = "2025-11-25"` (`lib.rs:40`). + - **19 tools** registered in `list_tools()` (`lib.rs:56-294` — corrected from this doc's initial "20"; see §4 of `02-subsystem-catalog.md` for the enumerated registry): + `entity_at`, `project_status`, `analyze_start`, `analyze_status`, + `analyze_cancel`, `find_entity`, `source_for_entity`, `entity_context`, + `call_sites`, `callers_of`, `execution_paths_from`, + `execution_paths_ranked`, `summary`, `summary_preview_cost`, + `issues_for`, `orientation_pack`, `index_diff`, `neighborhood`, + `subsystem_members`, plus one I have not independently counted past + line 254 (claimed `grep -c 'ToolDefinition {'` = 20). + - Backed by `ServerState` (built in `serve.rs:131`) holding a + `ReaderPool`, optional summary-LLM `Writer` + `LlmProvider`, optional + `FiligreeHttpClient`. + +3. **HTTP read API** — Axum router in + `crates/clarion-cli/src/http_read.rs:347-369`. Six routes: + - `GET /api/v1/files` → `get_file` (line 670) + - `POST /api/v1/files:resolve` → `post_files_resolve` (line 887) + - `POST /api/v1/files/batch` → `post_files_batch` (line 815) + - `GET /api/v1/_capabilities` → `get_capabilities` (line 1005; + unprotected — outside the bearer middleware) + - Plus two test-only routes inside `#[cfg(test)]` modules (`/x`, `/boom`) + that are not part of the public surface. + - Bearer-token middleware on the `protected` group + (`http_read.rs:376 require_http_identity`); HMAC variant exists + (`require_hmac_identity` line 405). Tower stack: `CatchPanic` → + `HandleError` → 10s `TimeoutLayer` → `RequestBodyLimitLayer` → + `LoadShedLayer` → `ConcurrencyLimitLayer(64)`. + +4. **CLI** — `crates/clarion-cli/src/cli.rs` (64 LOC). `clap` derive macros. + Flags of note on `analyze`: `--allow-unredacted-secrets` (requires + `--confirm-allow-unredacted-secrets ` non-interactively), + `--allow-no-plugins` for dry runs. + +5. **Outbound HTTP — Filigree reverse-association lookup.** + `crates/clarion-mcp/src/filigree.rs::FiligreeHttpClient` (`reqwest` + blocking) calls Filigree's HTTP API for entity-association reverse + lookup; request shape decoded into `EntityAssociationsResponse` + (`filigree.rs:11-22`). Auth via `token_env` (`clarion.yaml:integrations. + filigree.token_env`). + +6. **Outbound HTTP — OpenRouter LLM** in `clarion-core/src/llm_provider.rs` + (`OpenRouterProvider`); URL/attribution from `clarion.yaml:llm_policy.openrouter`. + +7. **Subprocess providers** — `CodexCliProvider`, `ClaudeCliProvider` in the + same file run external `codex` / `claude` binaries with stdin-piped + prompts and structured output. + +## 6. Candidate Subsystems + +### clarion-core — `crates/clarion-core/` +- **LOC:** ~11669 src across 13 `.rs` files; 325 LOC integration test. +- **Source files (one-line roles):** + - `src/lib.rs` (50) — facade re-exports per the documented "Re-export policy". + - `src/entity_id.rs` (596) — 3-segment ID assembler + grammar (ADR-003/022 per code comment). + - `src/llm_provider.rs` (2467) — `LlmProvider` trait, `OpenRouterProvider`, `CodexCliProvider`, `ClaudeCliProvider`, `RecordingProvider`, prompt builders, `CachingModel`. + - `src/plugin/mod.rs` (52) — submodule wiring. + - `src/plugin/protocol.rs` (875) — JSON-RPC envelopes + typed params/results. + - `src/plugin/transport.rs` (569) — Content-Length framing. + - `src/plugin/manifest.rs` (1119) — `plugin.toml` parser + validator. + - `src/plugin/discovery.rs` (667) — `$PATH` scan + manifest lookup. + - `src/plugin/host.rs` (2935) — supervisor, ontology/identity/jail/cap pipeline. + - `src/plugin/host_findings.rs` (NOT measured separately) — finding subcodes. + - `src/plugin/jail.rs` (~) — path-jail (`canonicalize` + `starts_with`). + - `src/plugin/limits.rs` (572) — Content-Length / entity-cap / path-escape breakers + `RLIMIT_AS`/`RLIMIT_NOFILE`/`RLIMIT_NPROC` via `nix`. + - `src/plugin/breaker.rs` (360) — crash-loop breaker (>3 crashes / 60s). + - `src/plugin/mock.rs` (876) — `#[cfg(test)]` mock plugin for host unit tests. +- **Outbound deps:** `serde`, `serde_json`, `tempfile`, `thiserror`, `toml`, `tracing`, `nix`, `which`, `reqwest` (LLM HTTP). No internal Clarion crate deps. + +### clarion-storage — `crates/clarion-storage/` +- **LOC:** ~3218 src; 4858 LOC tests. +- **Source files:** + - `src/lib.rs` (43) — re-exports. + - `src/writer.rs` (1074) — `Writer` actor, `mpsc::Sender` API, batch commits (`DEFAULT_BATCH_SIZE = 50`), `commits_observed` counter (`Arc` per writer.rs:50 docs). + - `src/reader.rs` — `ReaderPool` via `deadpool-sqlite`. + - `src/query.rs` (1160) — read-side query helpers: `call_edges_from`, `subsystem_members`, `entity_at_line`, `find_entities`, `resolve_file_catalog_entry`, etc. + - `src/schema.rs` — migration runner; embedded `0001_initial_schema.sql` (293 LOC SQL). + - `src/pragma.rs` — WAL+`synchronous=NORMAL`+`busy_timeout=5000`+`wal_autocheckpoint=1000`+`foreign_keys=ON` PRAGMAs. + - `src/commands.rs` — `WriterCmd` enum (`EntityRecord`, `EdgeRecord`, `FindingRecord`, `InferredCallEdgeRecord`, `RunStatus`). + - `src/cache.rs` — LLM `SummaryCache`/`InferredEdgeCache` helpers. + - `src/unresolved.rs` — `UnresolvedCallSiteRecord` + replace-by-caller. + - `src/error.rs` — `StorageError`. +- **Outbound deps:** `clarion-core` (path dep), `rusqlite`, `deadpool-sqlite`, `tokio`, `serde`, `serde_json`, `blake3`, `tracing`, `thiserror`. + +### clarion-cli — `crates/clarion-cli/` +- **LOC:** ~6790 src; 6394 LOC tests (5 test files + `wp1_e2e.rs` + `wp2_e2e.rs`). +- **Source files:** + - `src/main.rs` (78) — binary entry, runtime construction, .env hygiene exclusion for `analyze`. + - `src/cli.rs` (64) — clap definitions. + - `src/install.rs` — `.clarion/` initialiser + `clarion.yaml` stub. + - `src/analyze.rs` (2549) — analyze pipeline: discovery → spawn → walk → `analyze_file` per file → writer commands → clustering. + - `src/serve.rs` (326) — `clarion serve` orchestrator (stdio MCP thread + HTTP thread). + - `src/http_read.rs` (1532) — Axum HTTP read API, bearer + HMAC middleware, tower stack. + - `src/clustering.rs` (510) — Leiden / weighted-components clustering over `xgraph::Graph`. + - `src/config.rs` — `AnalyzeConfig` / `ClusteringConfig` YAML loader. + - `src/instance.rs` — `InstanceId` UUID newtype, persisted to `.clarion/instance_id`. + - `src/run_lifecycle.rs` — `runs` row lifecycle: `recover_preexisting_running_runs`, `begin_run`. + - `src/secret_scan.rs` + `src/secret_scan/{anchors,baseline,files,findings}.rs` — pre-ingest secret-scan gate wrapping `clarion-scanner`. + - `src/stats.rs` — `P95Accumulator` and stat helpers. +- **Outbound deps:** `clarion-core`, `clarion-mcp`, `clarion-scanner`, `clarion-storage` (all path deps); `anyhow`, `axum`, `blake3`, `clap`, `dotenvy`, `ignore`, `rusqlite`, `serde`, `serde_json`, `serde_norway`, `sha2`, `time`, `tokio`, `tower`, `tower-http`, `tracing`, `tracing-subscriber`, `uuid`, `xgraph`. Dev-only: `clarion-plugin-fixture`, `assert_cmd`, `tempfile`, `sha1`. + +### clarion-mcp — `crates/clarion-mcp/` +- **LOC:** ~6595 src across 3 files; 2233 LOC integration test. +- **Source files:** + - `src/lib.rs` (4703) — `ToolDefinition` list, `ServerState`, JSON-RPC dispatch (`handle_json_rpc`, `handle_tool_call`), stdio loop (`serve_stdio*`), in-process analyze supervisor for `analyze_{start,status,cancel}`. + - `src/config.rs` (1600) — `McpConfig` YAML loader; LLM/Filigree/serve.http config; provider selection; validation rules (deprecated-provider rejection, Filigree port-conflict, etc.). + - `src/filigree.rs` — `FiligreeHttpClient` + reverse-association lookup with `FiligreeLookup` trait; `reqwest::blocking`. +- **Outbound deps:** `clarion-core`, `clarion-storage` (path deps); `blake3`, `reqwest`, `rusqlite`, `serde`, `serde_json`, `serde_norway`, `thiserror`, `time`, `tokio`, `tracing`. + +### clarion-scanner — `crates/clarion-scanner/` +- **LOC:** ~881 src across 4 files; 655 LOC tests. +- **Source files:** + - `src/lib.rs` — `Detection`, `SecretCategory`, `HashedSecret` (detect-secrets-compatible SHA-1). + - `src/patterns.rs` — `Scanner`, `PatternMeta` regex catalogue. + - `src/entropy.rs` — `EntropyTuning`, high-entropy filter. + - `src/baseline.rs` — `Baseline`, `BaselineEntry`, `SuppressionResult`, `load_baseline`. +- **Outbound deps:** `regex`, `serde`, `serde_norway`, `sha1`, `thiserror`. **No internal Clarion deps** — this is a pure scanner library. + +### clarion-plugin-fixture — `crates/clarion-plugin-fixture/` +- **LOC:** 187 across 2 files. +- **Source files:** + - `src/main.rs` (185) — minimal JSON-RPC plugin: returns one `fixture:widget:demo.sample` entity per `analyze_file`; honours `CLARION_FIXTURE_EXCEED_RLIMIT_AS` env var to exercise the host's `RLIMIT_AS` OOM-kill path via `nix::sys::mman::mmap_anonymous` (the only `unsafe` block in the crate). + - `src/lib.rs` — trivial. +- **Outbound deps:** `clarion-core` (path); `serde_json`; `nix` (unix-only, with `mman`, `signal` features). + +### plugins/python — `plugins/python/` +- **LOC:** 3028 src across 11 modules; 3440 LOC tests across 9 files. +- **Source files (`src/clarion_plugin_python/`):** + - `__init__.py` (3) — package metadata. + - `__main__.py` (15) — entry point, installs `stdout_guard` then calls `server.main()`. + - `server.py` (296) — JSON-RPC server loop, dispatch for `initialize`/`initialized`/`analyze_file`/`shutdown`/`exit`; constants `ONTOLOGY_VERSION = "0.6.0"`, `MAX_CONTENT_LENGTH = 8 MiB`, `MAX_FILES_PER_PYRIGHT_SESSION = 25`. + - `extractor.py` (932) — AST walker emitting module/class/function entities plus `imports`/`calls`/`references` candidate edges. + - `pyright_session.py` (1406) — wraps `pyright-langserver` over `subprocess` + `select`; type-resolved call & reference resolution. + - `call_resolver.py` (65) — `CallResolutionResult`, `CallsRawEdge`, `Finding`, `UnresolvedCallSite` dataclasses. + - `reference_resolver.py` (70) — `ReferenceResolutionResult`, `ReferenceSite`. + - `entity_id.py` (75) — Python-side mirror of the 3-segment ID assembler. + - `qualname.py` (48) — L7 qualname reconstruction (dotted module + `__qualname__`). + - `stdout_guard.py` (62) — replaces `sys.stdout` so accidental writes don't corrupt JSON-RPC frames (called from `__main__.py`). + - `wardline_probe.py` (56) — fail-soft `importlib` probe for `wardline.core.registry` + version-pin gate against `[1.0.0, 2.0.0)`. +- **Outbound deps (runtime):** `packaging>=24`, `pyright==1.1.409`. **No HTTP, no network at all in the plugin process.** + +## 7. Cross-Cutting Concerns Observed + +- **Error handling:** + - Library crates use `thiserror` for typed error enums + (`clarion-storage/src/error.rs`, `clarion-mcp/src/filigree.rs:31-50`, + `clarion-core/src/plugin/limits.rs::BreakerState`, etc.). + - Binary code uses `anyhow` for top-level error propagation + (`crates/clarion-cli/src/main.rs:13`, + `crates/clarion-cli/src/serve.rs:8`). + - The `runs` table has a `recover_preexisting_running_runs` step + (`run_lifecycle.rs:5-28`) that marks any leftover `status='running'` + rows as `failed` on next start — explicit crash-recovery discipline. + +- **Logging / tracing:** `tracing` + `tracing-subscriber` with + `EnvFilter::try_from_default_env()`; defaults to `info` if no `RUST_LOG` + (`main.rs:73`). HTTP read API installs a *separate* `Dispatch` writing to + stderr to keep panic traces off the MCP stdout + (`http_read.rs:31-38`). + +- **Async runtime:** + - `analyze` builds a **multi-thread** runtime (`main.rs:36-38`). + - `serve`'s MCP stdio loop runs a **current-thread** runtime on a + dedicated OS thread (`serve.rs:126-130`), with the HTTP server running + in its own thread (`http_read.rs::spawn`). One `ReaderPool` is shared + across both and verified by `Arc::ptr_eq` (`serve.rs:63-71`). + +- **Configuration loading:** + - `clarion.yaml` is parsed by `serde_norway` (YAML). `clarion-mcp/src/config.rs::McpConfig::from_yaml_str` runs an alias-collision check (`llm` vs `llm_policy`) then `validate()` enforces invariants such as "no Anthropic provider", "Filigree actor non-blank when enabled", "Filigree port-conflict ban", "no zero-port". + - `clarion-cli/src/config.rs` separately parses analyze-time config (clustering, etc.). + - Note `analyze` deliberately **does not** load `.env` files + (`main.rs:23-25, 19-22`): `.env` is treated as in-tree source and + scanned by the secret scanner before plugin spawn. + +- **Schema / migrations:** + - Embedded via `include_str!` in + `clarion-storage/src/schema.rs:18-22`. One migration so far: + `0001_initial_schema.sql` (293 lines of SQL). Tracking table: + `schema_migrations`. + - PRAGMA invariant check: `apply_write_pragmas` rejects connection if + `journal_mode` is not `WAL` after `PRAGMA journal_mode = WAL` + (`pragma.rs:17-21`). + +- **Security boundaries:** + - **Plugin jail** (`crates/clarion-core/src/plugin/jail.rs`): every + `file_path` from a plugin response is `canonicalize()`d and asserted + `starts_with(project_root)`. Violations trip the `PathEscapeBreaker` + (>10 escapes / 60s → kill). + - **Resource caps** (`limits.rs:11-15` table): `ContentLengthCeiling` 8 MiB, + entity cap 500k per run, `RLIMIT_AS` via + `CommandExt::pre_exec`-applied `setrlimit(2)` (the *only* allowed + `unsafe` in the workspace, justified at `Cargo.toml:21-24`). + - **Crash-loop breaker** (`breaker.rs:1-14`): >3 plugin crashes / 60s → + refuse further spawns. + - **Pre-ingest secret scan** (`clarion-cli/src/secret_scan.rs:1-8`): + runs before any plugin spawn; analyze aborts with exit code 78 + unless `--allow-unredacted-secrets` is set with the right confirm + token; emits structured findings written to storage. + - **HTTP bearer auth + HMAC option** + (`http_read.rs:376-510`); `/api/v1/_capabilities` is the only + unprotected route. `LoadShedLayer` + `ConcurrencyLimitLayer(64)` + + 10s `TimeoutLayer` for DoS resistance. + - **stdout guard on the Python plugin side** + (`plugins/python/src/clarion_plugin_python/stdout_guard.py`) — any + accidental `print()` is redirected so JSON-RPC framing is never + corrupted. + +- **Test harness shape:** + - Per-crate `tests/*.rs` integration tests + inline `#[cfg(test)]` units. + - Cross-language fixture at `fixtures/entity_id.json` consumed by both + `tests/test_entity_id.py` and (presumably, not verified) a Rust test. + - Bash-level e2e in `tests/e2e/` (4 scripts). + - Performance harness in `tests/perf/b8_scale_test/` with timestamped + results directories (latest: `2026-05-18T1138Z-phase3/`). + +## 8. Test Corpus Shape + +**Per-crate Rust tests** (`crates/*/tests/*.rs`): +- `clarion-core/tests/host_subprocess.rs` (325) — T1 happy-path subprocess + integration; spawns the `clarion-plugin-fixture` binary via `PluginHost::spawn`. +- `clarion-storage/tests/writer_actor.rs` (2440) — round-trip insert, + per-N-batch commit cadence, `FailRun` rollback. +- `clarion-storage/tests/schema_apply.rs` (901) — migration 0001 produces + every table, index, trigger. +- `clarion-storage/tests/query_helpers.rs` (1124) — read-side query helpers. +- `clarion-storage/tests/reader_pool.rs` — reader-pool concurrency. +- `clarion-storage/tests/llm_cache.rs` — LLM cache helper tests. +- `clarion-cli/tests/install.rs` — `clarion install` integration. +- `clarion-cli/tests/analyze.rs` (1456) — Sprint-1 `clarion analyze`. +- `clarion-cli/tests/serve.rs` (3075) — MCP serve over real sockets/pipes. +- `clarion-cli/tests/secret_scan.rs` (917, `#![cfg(unix)]`) — secret-scan gate. +- `clarion-cli/tests/wp1_e2e.rs` — README §3 demo-script smoke. +- `clarion-cli/tests/wp2_e2e.rs` (606) — full walking-skeleton pipeline. +- `clarion-mcp/tests/storage_tools.rs` (2233) — storage-backed MCP tool tests. +- `clarion-scanner/tests/scanner.rs` (655) — pattern + baseline tests. + +**Python plugin tests** (`plugins/python/tests/`, 9 files, 3440 LOC): +`test_entity_id.py`, `test_extractor.py`, `test_package.py`, +`test_pyright_session.py`, `test_qualname.py`, `test_round_trip.py` +(plugin analyses its own source), `test_server.py` (subprocess JSON-RPC), +`test_stdout_guard.py`, `test_wardline_probe.py`. + +**End-to-end shell scripts** (`tests/e2e/`): +- `sprint_1_walking_skeleton.sh` — `install` → `analyze` → sqlite assertions + on entities, edges, references. +- `sprint_2_mcp_surface.sh` — analyze + `clarion serve`, sends framed + MCP JSON-RPC for "eight" navigation tools (note: actual `list_tools()` is + now 19 — script may exercise a subset), with a local HTTP fake of + Filigree's reverse-association route. +- `phase3_subsystems.sh` — clustering: subsystem entities, membership + edges, deterministic signature across clean project copies, MCP + `subsystem_members` tool. +- `wp5_secret_scan.sh` — pre-ingest scanner smoke. +- Plus `external-operator-smoke.md` (procedure doc, not a script). + +**Perf corpus**: `tests/perf/b8_scale_test/` with a `derive-elspeth-corpus.sh` +script, `driver.py`, `test_driver.py`, and persisted result directories +(latest `2026-05-18T1138Z-phase3/`). `tests/perf/elspeth_mini/` is a +~150-file Python corpus checked in to seed perf runs. + +## 9. Open Questions (For Per-Subsystem Phase) + +1. **Writer-actor concurrency** — `writer.rs:50` exposes `commits_observed` + as an `Arc` and explicitly says "Read this field before + dropping the Writer". What invariants does the actor enforce around + `BeginRun` / `FailRun` / `CommitRun`? How does it interact with + `recover_preexisting_running_runs` on next start? +2. **Two-thread `serve` topology** — why are MCP stdio and HTTP read each + on a dedicated thread with their *own* tokio runtime configurations + (`current_thread` vs the implicit Axum runtime), and what guarantees the + `Arc::ptr_eq` check buys at `serve.rs:63-71`? +3. **Plugin jail bypasses** — `jail.rs` follows symlinks. Are there file + classes (e.g. symlinks in the project root, hard links across + filesystems, special files in `/proc`) that the current `canonicalize + + starts_with` rule under-handles? +4. **Crash-loop & path-escape thresholds** — `breaker.rs` hard-codes + >3 crashes / 60s; `limits.rs` PathEscapeBreaker hard-codes >10 / 60s. + Are these surfaced through `clarion.yaml` or are they baked in? +5. **Identity check correctness** — `host.rs` step 2 recomputes + `entity_id(plugin_id, kind, qualified_name)` and compares against the + returned string. What happens if the plugin emits an entity ID whose + `canonical_qualified_name` contains a colon (forbidden) vs unicode edge + cases (NFC/NFD)? +6. **MCP tool inventory drift** — `list_tools()` returns 19 tools; the + `sprint_2_mcp_surface.sh` script comments mention "eight". Does the + e2e cover the full surface, or only a Sprint-2 subset? +7. **LLM provider sandboxing** — `ClaudeCliProvider` and `CodexCliProvider` + shell out to local CLI binaries. How are timeouts, stdout + capture, and permission modes (e.g. Codex `sandbox: read-only` in + `clarion.yaml`) actually enforced inside `llm_provider.rs`? +8. **HTTP auth modes** — bearer is wired (`require_http_identity` line 376) + and HMAC handler exists (`require_hmac_identity` line 405). Which is on + today, how is the secret material loaded, and is HMAC behind a feature + gate or a config setting? +9. **Schema evolution path** — only one migration is in tree + (`0001_initial_schema.sql`, 293 lines). What is the migration-author + workflow, and how does `scripts/check-migration-retirement.py` gate it? +10. **Clustering reproducibility** — `clustering.rs` uses `xgraph`'s Leiden + implementation with a seed in `ClusterConfig`. Does + `phase3_subsystems.sh` actually exercise byte-for-byte determinism, or + only signature-level determinism via `cluster_hash`? +11. **Wardline coupling shape** — the plugin imports `wardline.core.registry` + only inside `wardline_probe.py` (the manifest declares + `wardline_aware = true` in `plugin.toml:24`). Does any production + code path *consume* the probe result, or is it purely declarative + at v1.0? + +## 10. Confidence Statement + +| Claim | Confidence | Evidence | +|----------------------------------------------------------------------------------------|------------|----------| +| Workspace has six Rust crates | High | `Cargo.toml:3-10` | +| `clarion` binary has three subcommands: install / analyze / serve | High | `crates/clarion-cli/src/cli.rs:12-62` | +| MCP server exposes 19 tools | High | `ToolDefinition` enumeration of `crates/clarion-mcp/src/lib.rs:56-257` = 19 (the `grep -c 'ToolDefinition {'` shortcut counted the struct declaration plus 19 instances; corrected during catalog validation) | +| HTTP read API has 4 functional routes + 2 test-only | High | `crates/clarion-cli/src/http_read.rs:347-356,1374-1433` | +| MCP protocol version = `"2025-11-25"` | High | `crates/clarion-mcp/src/lib.rs:40` | +| Plugin JSON-RPC framing is LSP-style `Content-Length` | High | `crates/clarion-core/src/plugin/transport.rs:1-9` | +| SQLite is WAL with `synchronous=NORMAL`, `busy_timeout=5000` | High | `crates/clarion-storage/src/pragma.rs:17-30` | +| Writer-actor is a single tokio task owning one write connection | High | `crates/clarion-storage/src/writer.rs:1-12, lib.rs:1-5` | +| Plugin discovery is `$PATH` scan for `clarion-plugin-*` with neighbor + install-prefix fallback | High | `crates/clarion-core/src/plugin/discovery.rs:1-40` | +| Python plugin uses `pyright==1.1.409` as a subprocess language server | High | `plugins/python/pyproject.toml:21`, `pyright_session.py:7-11` | +| `unsafe_code = "deny"` workspace-wide with one documented exception in plugin host | High | `Cargo.toml:20-25` + `crates/clarion-core/src/plugin/host.rs` module doc §Memory limit | +| Crash-loop breaker triggers >3 crashes in 60s | High | `crates/clarion-core/src/plugin/breaker.rs:1-7` | +| `clarion serve` runs MCP stdio + HTTP in two threads sharing one `ReaderPool` | High | `crates/clarion-cli/src/serve.rs:53-80` | +| Pre-ingest secret scan blocks analyze with exit code 78 by default | High | `crates/clarion-cli/src/secret_scan.rs:1-12`, `main.rs:42-47` | +| Wardline integration is import-probe-only on the Python side | High | `plugins/python/src/clarion_plugin_python/wardline_probe.py` (whole file) | +| `clarion-scanner` is dependency-free of other Clarion crates | High | `crates/clarion-scanner/Cargo.toml` (no `clarion-*` paths) | +| Filigree is reached via outbound `reqwest`; no inbound Filigree surface in this repo | High | `crates/clarion-mcp/src/filigree.rs:1-12` | +| `clustering.rs` uses `xgraph`'s Leiden community detection | High | `crates/clarion-cli/src/clustering.rs:5-6` | +| Single schema migration so far (`0001_initial_schema.sql`) | High | `ls crates/clarion-storage/migrations/`, `schema.rs:18-22` | +| Total Rust source ~29k LOC | Medium | `wc -l` per crate src trees; line counts include comments + blanks | +| Total Python source 3028 LOC | High | `wc -l plugins/python/src/clarion_plugin_python/*.py` | +| `list_tools()` includes exactly the 19 tool names listed in §5 | High | Direct enumeration of the `ToolDefinition` registry during catalog validation. The "one more past line 254" was a miscount — the literal `ToolDefinition {` at the struct declaration leaked into the original `grep -c`. | +| HMAC inbound auth is wired but possibly behind a config gate | Medium | `require_hmac_identity` exists at `http_read.rs:405` but I did not trace what selects it vs bearer | +| `analyze.rs` calls `Command::new` on the plugin path | Medium | Inferred from `analyze.rs` module doc + `clarion_core::AcceptedEntity` imports; did not read the spawn line directly. The `Command::new` call lives inside `PluginHost::spawn` in `host.rs`, not in `analyze.rs`. | +| Cross-language fixture `fixtures/entity_id.json` is consumed by Rust tests too | Low | Only Python consumer confirmed (`test_entity_id.py` per its docstring); Rust side asserted by spec language but not greped in this pass | + +--- + +## Confidence Assessment (overall) + +High overall. The §1–§8 claims are mostly directly traceable to a file:line +or to a `Cargo.toml` / `pyproject.toml` field. The discovery sweep read or +sampled every src `.rs` in `clarion-core/src/plugin/` and the `lib.rs`/entry +of every other crate; every Python module head; every `Cargo.toml`; the +HTTP router definition; the MCP `list_tools()` table; the writer/pragma/schema +heads; the e2e scripts; and `plugin.toml`. + +## Risk Assessment + +- **Greatest risk to downstream catalog work:** the two giant files + (`clarion-mcp/src/lib.rs` at 4703 LOC and `clarion-core/src/plugin/host.rs` + at 2935 LOC) were only sampled. Per-subsystem explorers should read them + end-to-end before claiming completeness on MCP tooling or host + enforcement semantics. +- **Resolved during validation:** the tool count was originally reported as + 20 in this doc, derived via `grep -c 'ToolDefinition {'`. Catalog + validation enumerated the actual `ToolDefinition` registry at + `clarion-mcp/src/lib.rs:56-257` and confirmed **19** distinct production + tools. The grep counted the struct declaration plus 19 instances. +- **External dependency surface I did not read:** the actual `Command::new` + for plugin subprocess spawn lives inside `host.rs` (sampled only at the + module doc), not `analyze.rs`. Behaviour around `pre_exec`, descriptor + closing, and env-var passing was not traced beyond the module-doc + promise. + +## Information Gaps + +- No reading of the migration SQL (`0001_initial_schema.sql`) — schema + shape (tables, indexes, FK graph) is therefore an open question. +- No tracing of `analyze::run_with_options`'s control flow past its + imports + signature — concurrency model of "Pattern A buffering" not + verified, only quoted from the module doc. +- `target/` build artifacts not inspected; binary sizes / link surface + unknown. +- `.github/workflows/ci.yml` contents not read; only filename observed. +- Wardline-side code: not in this repo; no claims made beyond what the + Python `wardline_probe` proves about imports. + +## Caveats + +- This document deliberately reports the system as evidenced by code, + not by intent. Where the code names ADRs, work packages, or sprints in + comments, the IDs are quoted but not validated against the (un-read) + design docs. +- LOC figures include comments and blanks; they are coarse "depth + indicators" only. +- The Python `__pycache__/` count in the file tree was suppressed from + module counts but does indicate the test suite has been executed on + this machine. diff --git a/docs/arch-analysis-2026-05-22-1924/02-subsystem-catalog.md b/docs/arch-analysis-2026-05-22-1924/02-subsystem-catalog.md new file mode 100644 index 00000000..fa75082b --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/02-subsystem-catalog.md @@ -0,0 +1,632 @@ +# 02 — Subsystem Catalog (Clarion) + +> Source-of-evidence guarantee: every section in this catalog was produced by an +> independent codebase-explorer subagent reading only the relevant source, +> manifests, migration SQL, fixtures, and tests under that subsystem's tree. +> No design docs (`docs/clarion/**`, `docs/suite/**`, `docs/implementation/**`, +> `docs/federation/**`, prior `arch-analysis-*/`, ADRs, sprint READMEs) were +> read while writing these entries. Confidence statements per section list +> evidence consumed. + +## 1. clarion-core + +**Location:** `crates/clarion-core/` +**LOC:** 11,653 source / 325 integration test (plus extensive inline `#[cfg(test)]` blocks; see Concerns) +**Crate type / role:** Rust library (`lib.rs`). No binary. Re-exported as a facade by `lib.rs:13-49`; implementation modules remain reachable through `clarion_core::plugin::*` per the explicit policy comment at `lib.rs:5-7`. + +### Responsibility + +`clarion-core` owns three orthogonal concerns bundled into one crate: (1) the canonical **entity-ID assembler** (`entity_id.rs`), which is the single source of truth for the `{plugin_id}:{kind}:{canonical_qualified_name}` identifier shape and the ADR-022 identifier grammar; (2) the **plugin host runtime** (`plugin/`), a synchronous JSON-RPC supervisor that discovers, spawns, validates, and reaps language-plugin subprocesses while enforcing four core minimums (path jail, content-length ceiling, run-cumulative entity cap, virtual-address limit); and (3) the **LLM provider abstraction** (`llm_provider.rs`), a `Send + Sync` trait plus four concrete adapters (recording fixture, OpenRouter HTTP, Claude CLI subprocess, Codex CLI subprocess) for query-time enrichment. The public surface that proves the boundary is the `pub use` block in `lib.rs:13-49`, which is the only API surface other workspace crates are *supposed* to consume (the policy comment is explicit; one violation exists — see Concerns). + +### Key components + +- `src/entity_id.rs:90-148` — `entity_id(plugin_id, kind, canonical_qualified_name) -> Result` plus a shared `validate_kind_grammar` helper used by both this module and `manifest.rs`. ~150 LOC of code; ~450 LOC of byte-for-byte cross-language parity tests against `fixtures/entity_id.json` covering entities + `contains`/`calls`/`references` edge wire shapes (`entity_id.rs:371-587`). +- `src/plugin/manifest.rs:1-1119` — TOML parser for `plugin.toml`, ADR-022 reserved-kind / reserved-rule-prefix gating, `Manifest::validate_for_v0_1()` capability check (run at handshake). Defines `RESERVED_ENTITY_KINDS = ["file", "subsystem", "guidance"]` at line 28. +- `src/plugin/transport.rs:1-569` — LSP-style `Content-Length:`-framed JSON transport. Synchronous `read_frame(reader, ceiling) -> Result` / `write_frame(writer, frame)`. Header-line cap at 8 KiB (`MAX_HEADER_LINE_BYTES`), body cap from caller-supplied `ContentLengthCeiling` (default 8 MiB). +- `src/plugin/protocol.rs:312-474` — Typed JSON-RPC 2.0 envelopes and the five method bodies: `initialize` (core→plugin), `initialized` (notification), `analyze_file` (request), `shutdown` (request), `exit` (notification). Struct-per-method discriminated by `IncomingMessage::method` (`protocol.rs:3-22` doc). `JsonRpcVersion` newtype hard-pins the literal `"2.0"`. +- `src/plugin/jail.rs:73-103` — `jail(root, candidate)` and `jail_to_string(root, candidate)`. Both paths canonicalise via `std::fs::canonicalize` (follows symlinks per UQ-WP2-03 comment), assert `starts_with(canonical_root)`, and surface `JailError::EscapedRoot` / `Io` / `NonUtf8Path`. +- `src/plugin/limits.rs:1-572` — `ContentLengthCeiling` (8 MiB default), `EntityCountCap` (run-cumulative; default 500k via `EntityCountCap::DEFAULT_MAX`), `PathEscapeBreaker` (per-plugin; >10 escapes in 60 s trips), `apply_prlimit_as` / `apply_prlimit_nofile_nproc` (Linux-only `setrlimit` wrappers; async-signal-safe for `pre_exec`). Co-locates the seven `CLA-INFRA-PLUGIN-*` finding subcode constants. +- `src/plugin/breaker.rs:43-117` — `CrashLoopBreaker`: rolling-window crash counter (default 60 s window, >3 threshold per UQ-WP2-10). Trips → `FINDING_DISABLED_CRASH_LOOP` and refusal of further spawns. *Caller-driven* — `PluginHost` does not own this; the run loop in `clarion-cli/analyze.rs:271` does. +- `src/plugin/discovery.rs:1-667` — `discover()` / `discover_on_path()` scan `$PATH` for `clarion-plugin-` executables, locate the neighbouring `plugin.toml` (with a pipx-aware symlink-resolved install-prefix fallback documented at lines 13-33), and return `Vec>`. +- `src/plugin/host.rs:384-1182` — `PluginHost`: the supervisor. Generic over reader/writer so the in-process `mock.rs` (876 LOC, `#[cfg(test)] pub(crate)`) can drive it without a subprocess. Public methods: `spawn` (subprocess constructor; lines 505-644), `connect` (in-process; 658-666), `handshake` (743-803), `analyze_file` (815-985), `shutdown` (1106-1111), plus accessors `stderr_tail`, `ontology_version`, `take_findings`, `set_briefing_blocks`, `set_scanned_source_files`. The four-stage per-entity validation pipeline (ontology → identity → jail → cap) lives inline in `analyze_file` at lines 866-975. +- `src/plugin/host_findings.rs:1-273` — `HostFinding` struct + ten `CLA-INFRA-PLUGIN-*` / `CLA-INFRA-HOST-*` / `CLA-INFRA-MANIFEST-*` subcode constants and constructor functions (`unsupported_capability`, `entity_id_mismatch`, `path_escape`, `malformed_entity`, `malformed_edge`, …). +- `src/llm_provider.rs:101-2467` — `trait LlmProvider: Send + Sync` plus `RecordingProvider`, `OpenRouterProvider` (live `reqwest` HTTP), `CodexCliProvider` and `ClaudeCliProvider` (subprocess CLI wrappers with bounded stdout/stderr ring buffers and timeout via background reaper thread). Also hosts the prompt-template builders `build_leaf_summary_prompt`, `build_inferred_calls_prompt`, `build_coding_agent_provider_prompt` and version constants. + +### Public interface (outbound) + +The crate-root `pub use` facade (`lib.rs:13-49`) re-exports exactly these names; everything else is reachable through `clarion_core::plugin::{module}::*` per the explicit policy at `lib.rs:5-7`: + +- **Entity-ID assembler:** `EntityId`, `EntityIdError`, `entity_id()` — assemble or parse a 3-segment ID. `EntityId` is opaque (the only constructor is `entity_id()` / `FromStr::from_str` / `Deserialize`); `as_str()` returns the canonical form. +- **Plugin discovery:** `DiscoveredPlugin`, `DiscoveryError`, `discover()` — scan `$PATH` for `clarion-plugin-*` binaries. +- **Plugin manifest:** `Manifest`, `ManifestError`, `parse_manifest()` — TOML parse + ADR-022 grammar / reserved-kind / reserved-rule-prefix checks. Plus `Manifest::validate_for_v0_1()` capability gate, exercised by the host at handshake. +- **Plugin host:** `PluginHost`, `HostError`, `HostFinding`, `AcceptedEntity`, `AcceptedEdge`, `AnalyzeFileOutcome`, `AnalyzeFileStats`, `UnresolvedCallSite`, `BriefingBlockReason`, `EdgeConfidence`. +- **Resource limits:** `JailError`, `CapExceeded`, plus the `FINDING_DISABLED_CRASH_LOOP` constant (re-exported from `breaker`). +- **Crash-loop breaker:** `CrashLoopBreaker`, `CrashLoopState` — caller-driven, used by the analyze run loop, not by `PluginHost` itself. +- **LLM providers:** `LlmProvider` trait + `LlmRequest`, `LlmResponse`, `LlmPurpose`, `LlmProviderError`, `CachingModel`, `Recording`, `RecordingProvider`, `OpenRouterProvider(+Config)`, `ClaudeCliProvider(+Config)`, `CodexCliProvider(+Config)`, `PromptTemplate`, the three `build_*_prompt` functions, and the version constants `LEAF_SUMMARY_PROMPT_TEMPLATE_ID`, `INFERRED_CALLS_PROMPT_VERSION`. + +**JSON-RPC method set (the wire surface `PluginHost` drives, not Rust API):** `initialize` (request; `InitializeParams { protocol_version, project_root }` → `InitializeResult { name, version, ontology_version, capabilities }`); `initialized` (notification, empty params); `analyze_file` (request; `{ file_path }` → `{ entities: Vec, edges: Vec, stats: AnalyzeFileStats }` — per-element typing happens in `host::analyze_file` so a single malformed entity drops with a finding rather than failing the response); `shutdown` (request, empty); `exit` (notification, empty). `protocol.rs:3-22` and `protocol.rs:312-474`. + +### Dependencies + +- **Inbound (who calls this):** + - `crates/clarion-cli/src/analyze.rs` — the principal consumer. Uses `discover`, `parse_manifest`, `PluginHost::spawn`, `CrashLoopBreaker` (driven by the run loop, not the host), `AcceptedEntity`/`AcceptedEdge`/`AnalyzeFileOutcome`, `HostFinding`, `BriefingBlockReason`, `EdgeConfidence`, plus the `entity_id::entity_id` direct call for `core:file:*` and `core:subsystem:*` IDs (lines 909, 1683). + - `crates/clarion-cli/src/serve.rs`, `secret_scan.rs` — `BriefingBlockReason` and the LLM-provider types. + - `crates/clarion-storage/src/{commands,query,writer}.rs` — `EdgeConfidence` (re-exported from `commands.rs`), and one deep reach across the facade: `writer.rs:427` reads `clarion_core::plugin::manifest::RESERVED_ENTITY_KINDS` directly (not through the crate-root facade). + - `crates/clarion-mcp/src/lib.rs`, `clarion-plugin-fixture/src/main.rs`, and several integration tests (`clarion-cli/tests/wp2_e2e.rs`, `clarion-mcp/tests/storage_tools.rs`, etc.). +- **Outbound (what this calls):** Standard library only for the host + entity-ID layers (`std::process::Command`, `std::io::{BufRead,Write}`, `std::fs::canonicalize`, `std::thread`, `std::sync::{Arc,Mutex}`). External crates: `serde`/`serde_json` (wire), `thiserror` (errors), `toml` (manifest), `tracing` (logging), `nix` (Linux `setrlimit` via `apply_prlimit_*` in `limits.rs`), `which` (transitive via `discovery`), `reqwest` (live `OpenRouterProvider` HTTP), `tempfile` (test-only). +- **External services:** Plugin subprocesses (spawned by `PluginHost::spawn`, communicated to via stdin/stdout pipes; stderr piped + drained on a detached thread into a 64 KiB ring buffer at `host.rs:447-476`). Optional outbound HTTP to OpenRouter (`llm_provider.rs::OpenRouterProvider`). Optional `claude` / `codex` CLI subprocesses (`ClaudeCliProvider`, `CodexCliProvider`). + +### Internal architecture + +**Concurrency model: synchronous over `BufRead`/`Write`.** `PluginHost` (`host.rs:384`) is generic over a reader and writer so unit tests drive it in-process via `connect()` while production code uses `spawn()` which wires `std::io::BufReader` and `std::io::BufWriter`. All RPC is request-response on the same thread that called `analyze_file`; there is no async, no task system, no message channel inside the host. The *only* concurrency is one detached `std::thread` per host (`host.rs:614-620`, named `clarion-plugin-stderr-drain:`) that reads `ChildStderr` in 4 KiB chunks into a bounded `VecDeque` of capacity `STDERR_TAIL_BYTES = 64 KiB`, popping front on overflow. Rationale at `host.rs:550-561`: an inherited stderr could (a) flood the operator's terminal with hostile bytes or (b) deadlock the plugin on `write(2)` when the host is blocked in `analyze_file`. The ring is read back via `stderr_tail() -> Option` using lossy-UTF-8 conversion. `PluginHost::spawn` returns `(Self, std::process::Child)` — the *caller* owns reaping; `Child::Drop` does not `waitpid` on Unix (`host.rs:630-641`), so a handshake failure inside `spawn` reaps the child before returning the error. + +**Enforcement pipeline (the load-bearing per-entity loop, `host.rs:866-975`).** For each `RawEntity` deserialised from `AnalyzeFileResult.entities`: (0) field-size check against `MAX_ENTITY_FIELD_BYTES = 4 KiB` for `id`/`kind`/`qualified_name`/`source.file_path` and `MAX_ENTITY_EXTRA_BYTES = 64 KiB` for the two `#[serde(flatten)]` passthrough maps; (1) ontology check — `kind` must be in `manifest.ontology.entity_kinds` (ADR-022); (2) identity check — recomputed `entity_id(plugin_id, kind, qualified_name)` must equal the wire `id` (UQ-WP2-11 — prevents a plugin from minting IDs outside its declared namespace); (3) jail check — `jail_to_string(project_root, source.file_path)` must succeed; on failure, tick the per-host `PathEscapeBreaker`; (4) entity cap — `EntityCountCap::try_admit(1)`. Steps 0–2 drop the entity with a finding and continue. Step 3 drops the entity + records a finding; if the breaker trips (>10 escapes / 60 s) it shuts down the plugin and returns `HostError::PathEscapeBreakerTripped`. Step 4 on overflow shuts down the plugin and returns `HostError::EntityCapExceeded`. Edges go through a parallel but kill-free pipeline (`process_edges`, lines 1017-1060) — edges do not participate in the path-escape breaker or the entity cap. + +**Failure-mode policy (four distinct mechanisms, three layers).** The system layers failure detection: +1. **Per-frame ceiling** — `ContentLengthCeiling::DEFAULT = 8 MiB`. `read_frame` rejects with `TransportError::FrameTooLarge` *before* consuming the body bytes (`transport.rs:67-71`). +2. **Per-host path-escape breaker** — `PathEscapeBreaker` (`limits.rs`), rolling-window >10 escapes / 60 s. Owned by `PluginHost`. Trip → kill plugin. +3. **Per-run entity-count cap** — `EntityCountCap`, default 500,000 entities cumulative across all `analyze_file` calls on this host. Exceeded → kill plugin. +4. **Per-run crash-loop breaker** — `CrashLoopBreaker` (`breaker.rs`), rolling-window >3 spawn/handshake/analyze crashes / 60 s. *Not* owned by `PluginHost`; the analyze run loop in `clarion-cli/analyze.rs:271` constructs and ticks it. Trip → emit `FINDING_DISABLED_CRASH_LOOP`, refuse further spawns this run. + +The two breakers are conceptually different scopes — one polices a single misbehaving plugin's emissions, the other polices the *fleet* of plugins across one run. + +**State ownership inside `PluginHost`** (`host.rs:384-422`): `manifest` (immutable), `project_root` (canonicalised once at construction), `reader`/`writer` (the wire endpoints), `ceiling`/`entity_cap`/`path_breaker` (the three host-resident enforcement gates), `next_request_id` (monotone i64 — the JSON-RPC `id` allocator), `findings: Vec` (accumulated; drained via `take_findings()`), `terminated: bool` (idempotency guard so a double `shutdown()` does not write to a closed pipe), `ontology_version: Option` (set at handshake for ADR-007 cache keying by WP6), `stderr_tail: Option>>>` (the ring buffer; `None` for in-process `connect()`), and two read-only `Arc`-shared inputs the analyze run loop installs: `briefing_blocks: Arc>` and `scanned_source_files: Arc>` that drive `apply_briefing_block` (lines 987-1004) to attach `briefing_blocked` markers to entities whose source file failed secret scan or was unscanned. + +**Error model.** `HostError` (`host.rs:334-370`) wraps `TransportError`, `ProtocolError`, `ManifestError`, `CapExceeded`, `EntityIdError`, `serde_json::Error`, `std::io::Error`, plus three policy variants (`PathEscapeBreakerTripped`, `EntityCapExceeded(CapExceeded)`, `Spawn(String)`). Every module exposes its own `thiserror`-derived enum (`TransportError`, `ManifestError`, `JailError`, `DiscoveryError`, `EntityIdError`, `LlmProviderError`); the host composes them via `#[from]`. The asymmetry between "drop-entity, no kill" (steps 0–2 of the pipeline + most ontology/identity violations) and "kill plugin" (path-escape breaker trip, entity-cap overflow, manifest capability refusal) is the central policy expression. + +### Patterns observed + +- **Drop-with-finding vs. kill-with-error asymmetry** — `host.rs:866-975`. Two-tier sanction system mediated by `HostFinding` accumulation versus `Result::Err(HostError::*)` propagation. +- **Generic-over-IO supervisor with in-process mock** — `PluginHost` (`host.rs:384`) + `mock.rs` (876 LOC, `pub(crate) #[cfg(test)]`). Lets the host's entire pipeline be tested without spawning a subprocess; `connect()` constructor (line 658) is the seam. +- **Per-frame size ceiling with no-body-consume rejection** — `transport.rs:67-71` returns `FrameTooLarge` before touching the body, so a hostile frame cannot exhaust memory in `read_to_end`. +- **Newtype wrappers as wire pins** — `JsonRpcVersion` (`protocol.rs:72-95`) serialises/deserialises strictly to `"2.0"`; `EntityId` is constructible only via `entity_id()` / `FromStr::from_str` / custom `Deserialize` (`entity_id.rs:19-60`) so deserialising an arbitrary string at a `serde(flatten)` boundary cannot smuggle in a malformed id. +- **`pre_exec` resource-limit application** — `host.rs:569-586`. Closure runs in the forked child, calls only async-signal-safe `setrlimit(2)`, with a safety comment documenting the POSIX.1-2017 §2.4.3 reference. `RLIMIT_AS` from manifest hint or `DEFAULT_MAX_RSS_MIB`; `RLIMIT_NPROC` bumped to 4096 when `manifest.capabilities.runtime.pyright = Some(_)` (line 89, 577) because Pyright's Node-based LSP spawns helpers checked against the user's process count, not just children. +- **Cross-language byte-for-byte fixture parity** — `entity_id.rs:371-587` consumes `../../fixtures/entity_id.json` (the same file the Python plugin's `test_entity_id.py` consumes) and asserts identical assembly results for entities and the three edge wire shapes. +- **Caller-driven breaker, host-driven breaker** — symmetric naming hides asymmetric ownership. `CrashLoopBreaker` is *given* to the caller (it sits in `analyze.rs`'s plugin loop); `PathEscapeBreaker` lives inside the host. Same shape, different scope. +- **Idempotent shutdown via terminated flag** — `host.rs:1106-1111` and `do_shutdown` at 1158. The flag is set *before* the shutdown exchange runs (line 1164) so even a mid-exchange pipe break does not surface a spurious `BrokenPipe` on a defensive double-`shutdown()`. + +### Concerns / Smells / Risks + +- **`host.rs` at 2,935 LOC is the largest single file in the crate and (per the discovery doc) one of the four largest in the workspace.** The four-stage validation pipeline, edge processing, stats post-processing, briefing-block reconciliation, the subprocess constructor with its stderr drainer, and the in-process generic methods all live in one `impl PluginHost` block. The module would split cleanly along the pipeline-step / lifecycle / IO axes; the current shape makes the per-step contracts harder to test in isolation than the generics already permit. +- **`llm_provider.rs` at 2,467 LOC is in this crate at all.** The `lib.rs` doc-comment (`lib.rs:1`) calls this crate "domain types, identifiers, and provider traits" — but the module bundles concrete `reqwest`-based OpenRouter HTTP, two distinct subprocess-CLI provider implementations (Claude, Codex) each with their own timeout / ring-buffer / reaper-thread machinery, and the prompt-template builders. Three orthogonal subsystems (the trait, the HTTP transport, the CLI transports, the prompt assembly) compressed into one file in a crate that otherwise is about plugin hosting. A future `clarion-llm` crate split would tighten the `clarion-core` boundary considerably; today, `reqwest` is a top-level dependency of the crate that supervises plugins. +- **Facade leak from `clarion-storage`.** `writer.rs:427` reaches `clarion_core::plugin::manifest::RESERVED_ENTITY_KINDS` directly through the module path, bypassing the `lib.rs` re-export facade. The policy comment at `lib.rs:5-7` describes the facade as the supported surface; this constant is not in it. Either lift the constant to the facade or expose a `Manifest::is_reserved_kind` helper — the direct module reach pins the internal module path as semi-public. +- **`mock.rs` at 876 LOC is gated behind `#[cfg(test)] pub(crate)`** (`plugin/mod.rs:22`). This is test infrastructure, not production code, so it does not appear in any release binary — but the volume of the mock (driver-script DSL, frame scripting, response builders) is a sign that the host's pipeline is non-trivially stateful, which corroborates the `host.rs` size concern. +- **Subprocess lifecycle ownership is split.** `PluginHost::spawn` returns `(Self, std::process::Child)` (`host.rs:509`) and the doc at line 630-641 notes `Child::Drop` does not waitpid on Unix. The caller (currently `clarion-cli/analyze.rs`) must reap. The contract is documented but not enforced by the type system; a future consumer that drops `Child` on a happy path leaks a zombie until parent exit. A `KillOnDrop` newtype around the child would close this. +- **The path jail is a TOCTOU check by design.** `jail.rs:67-72` explicitly documents this: the canonical-path return is "a membership proof at canonicalization time, not a durable file handle." Any consumer that later opens the path must re-jail after open or use `openat`-anchored I/O. The current consumer in this crate doesn't open files (it returns the canonical string downstream), but the comment is the only enforcement and a future caller could miss it. +- **Limited integration test coverage in this crate's `tests/` directory.** Only one integration test file (`tests/host_subprocess.rs`, 325 LOC) — a single happy-path subprocess walkthrough against `clarion-plugin-fixture`. The host's many failure modes (every variant of `HostError`, every `CLA-INFRA-*` finding subcode) are exercised through inline `#[cfg(test)] mod tests` blocks at the bottom of each module — which works, but the integration surface (real `Command::spawn`, real `pre_exec`, real stderr-drain thread, real reap-on-error) is tested for the happy path only. +- **`MAX_PROTOCOL_ERROR_FIELD_BYTES = 4 KiB`, `MAX_ENTITY_FIELD_BYTES = 4 KiB`, `MAX_ENTITY_EXTRA_BYTES = 64 KiB`, `STDERR_TAIL_BYTES = 64 KiB`, `MAX_HEADER_LINE_BYTES = 8 KiB`, `MAX_UNRESOLVED_CALLEE_EXPR_BYTES = 512`, `ContentLengthCeiling::DEFAULT = 8 MiB`, `EntityCountCap::DEFAULT_MAX = 500_000`, `DEFAULT_MAX_RSS_MIB`, `DEFAULT_MAX_NOFILE`, `DEFAULT_MAX_NPROC`, `PYRIGHT_MAX_NPROC = 4096`** — eleven hardcoded limit constants across `host.rs`, `transport.rs`, `limits.rs`. The `breaker.rs:7` comment acknowledges the config surface lands in WP6 and is not present at this snapshot. Until then, "operator can tune" is aspirational; every limit is a recompile. + +### Confidence: High + +Read end-to-end: `lib.rs`, `entity_id.rs`, `plugin/mod.rs`, `plugin/jail.rs`, `plugin/breaker.rs` (first 120 lines + skim of tests), and the doc-comment + public-surface scan plus targeted reads of the load-bearing windows of `protocol.rs` (300-499), `host.rs` (1-820, 1060-1190 + grep of all `pub fn`/`impl`/spawn-related symbols), `manifest.rs` (1-100 + reserved-kind constant cross-ref), `discovery.rs` (1-80 doc), `limits.rs` (1-120 + finding constants), `host_findings.rs` (1-80 + grep), `transport.rs` (1-100), and `llm_provider.rs` (1-130 + public-surface grep). Verified dependency edges by grepping `use clarion_core` across the rest of the workspace (`clarion-cli/{analyze,serve,secret_scan}.rs`, `clarion-storage/{commands,query,writer}.rs`, `clarion-mcp/src/lib.rs`, `clarion-plugin-fixture/src/main.rs`). Cross-checked the `CrashLoopBreaker` ownership claim by reading `clarion-cli/analyze.rs:240-275`. Confirmed the `clarion-storage::writer.rs:427` facade leak directly. The two areas read by sample-and-grep rather than end-to-end are `host.rs` (2,935 LOC; sampled the entry points, the four-stage pipeline, the spawn constructor, and the shutdown idempotency window) and `llm_provider.rs` (2,467 LOC; sampled the trait + the public-surface grep but did not read the OpenRouter/Codex/Claude provider implementations in detail — they are flagged in Concerns as out-of-scope tonally for this crate). + +## 2. clarion-storage + +**Location:** `crates/clarion-storage/` +**LOC:** 3199 src across 10 files / 4871 tests across 5 files / 293 lines SQL in `migrations/0001_initial_schema.sql` +**Crate type / role:** Library crate (`lib.rs:1-42`). Sole owner of the SQLite read/write layer for the project DB at `.clarion/clarion.db`. Re-exports a flat surface from nine submodules. + +### Responsibility +`clarion-storage` is the only path through which other crates touch SQLite. It (a) opens and configures the per-project database with a fixed PRAGMA discipline (`pragma.rs`), (b) applies embedded schema migrations idempotently (`schema.rs`), (c) routes *every* mutation through a single writer-actor task that owns the sole write `rusqlite::Connection` (`writer.rs`), (d) hands out short-lived read-only connections from a `deadpool-sqlite` pool (`reader.rs`), and (e) provides the read-side query helper catalogue that the MCP navigation tools and HTTP read API consume (`query.rs`, ~29 `pub` query / type items). The crate also enforces wire-level invariants the rest of the workspace depends on — per-kind edge confidence/source-range contracts, source-file-anchor kinds, reserved entity-kind protection, parent↔contains-edge dual-encoding consistency — at the *writer boundary*, so a misbehaving plugin or caller cannot corrupt graph shape (`writer.rs:425-582, 954-1021`). + +### Key components +- `src/lib.rs:1-42` — module wiring + flat `pub use` facade (`Writer`, `ReaderPool`, `WriterCmd`, ~29 query helpers, cache helpers, `StorageError`). +- `src/pragma.rs:16-45` — `apply_write_pragmas` (WAL + `synchronous=NORMAL` + `busy_timeout=5000` + `wal_autocheckpoint=1000` + `foreign_keys=ON`, with a hard invariant assertion that `PRAGMA journal_mode=WAL` actually took effect) and `apply_read_pragmas` (busy_timeout + FK only). +- `src/writer.rs:40-140` — `Writer` handle, `Writer::spawn` (boots the actor as a `tokio::task::spawn_blocking` task owning one `rusqlite::Connection`), `send_wait` convenience. +- `src/writer.rs:142-260` — `run_actor` central match loop dispatching all 11 `WriterCmd` variants; blocking `mpsc::Receiver::blocking_recv`. +- `src/writer.rs:290-313, 802-877` — `ActorState` (`batch_size`, `writes_in_batch`, `in_tx`, `current_run`) and `bump_writes_and_maybe_commit` / `flush_run_batch` / `query_time_write` — the batch-cadence and run-transaction state machine. +- `src/writer.rs:425-582` — wire-contract enforcement: `enforce_entity_kind_contract`, `validate_entity_source_file_anchor`, `enforce_edge_contract`, plus the structural-vs-anchored edge-kind tables (`STRUCTURAL_EDGE_KINDS`, `ANCHORED_EDGE_KINDS`). +- `src/writer.rs:954-1021` — `parent_contains_mismatch` (the bidirectional parent↔contains dual-encoding check; runs inside the open commit transaction so a violation rolls back the run). +- `src/reader.rs:26-119` — `ReaderPool` (`deadpool-sqlite` wrapper with an `Arc<()>` identity tag for `shares_pool_with` runtime proofs) and `with_reader` async helper. +- `src/schema.rs:17-91` — `MIGRATIONS` slice (one entry, embedded by `include_str!`), `apply_migrations` runner gated by a `schema_migrations` table. +- `src/commands.rs:135-218` — the `WriterCmd` enum: `BeginRun`, `InsertEntity`, `InsertEdge`, `InsertFinding`, `FlushRunBatch`, `InsertInferredEdges`, `UpsertSummaryCache`, `TouchSummaryCache`, `ReplaceUnresolvedCallSitesForCaller`, `CommitRun`, `FailRun` (11 variants, each carries an `Ack = oneshot::Sender>`). +- `src/query.rs:214-1056` — read helpers (`entity_by_id`, `entity_at_line`, `find_entities`, `call_edges_from/_targeting`, `reference_edges_for_entity`, `module_dependency_edges`, `subsystem_members`, `resolve_file_catalog_entry`, `contained_entity_ids`, …). +- `migrations/0001_initial_schema.sql:23-291` — 9 tables (`schema_migrations`, `entities`, `entity_tags`, `edges` `WITHOUT ROWID` on natural PK, `findings`, `summary_cache`, `inferred_edge_cache`, `entity_unresolved_call_sites`, `runs`), 1 FTS5 virtual table (`entity_fts`), 3 triggers keeping FTS in sync, 2 generated virtual columns (`scope_rank`, `git_churn_count`) with partial indexes, 1 view (`guidance_sheets`), ~15 secondary indexes, and a row inserted into `schema_migrations`. Wrapped in a single `BEGIN…COMMIT`. + +### Public interface (outbound) +- **Writer surface** — `Writer::spawn(db_path, batch_size, channel_capacity) -> (Writer, JoinHandle>)`; `Writer::sender() -> mpsc::Sender`; `Writer::send_wait`; observability counters `commits_observed`, `dropped_edges_total`, `ambiguous_edges_total` (each an `Arc`, present in release builds). Constants `DEFAULT_BATCH_SIZE = 50`, `DEFAULT_CHANNEL_CAPACITY = 256` (`writer.rs:34-38`). +- **Command protocol** — the `WriterCmd` enum (11 variants), the four POD record types (`EntityRecord`, `EdgeRecord`, `FindingRecord`, `InferredCallEdgeRecord`), `RunStatus` (`SkippedNoPlugins | Completed | Failed`), `InferredEdgeWriteStats`, and `Ack`. +- **Reader surface** — `ReaderPool::open(db_path, max_size)`, `with_reader(impl FnOnce(&Connection) -> Result) -> Result`, `shares_pool_with`, `identity()`, `waiting_count()` (test-hook). +- **Schema runner** — `schema::apply_migrations(&mut Connection)`, `schema::applied_count(&Connection)` (used by `clarion install`). +- **Cache surface** — `summary_cache_lookup`/`upsert_summary_cache`/`touch_summary_cache`, `inferred_edge_cache_lookup`/`upsert_inferred_edge_cache`/`touch_inferred_edge_cache`, `inferred_edge_cache_key_id` (canonical key-id format `"caller|hash|model|prompt"`), plus the four POD types `SummaryCacheKey/Entry`, `InferredEdgeCacheKey/Entry` (`cache.rs:1-251`). +- **Unresolved-edges surface** — `UnresolvedCallSiteRecord`, `replace_unresolved_call_sites_for_caller(&Connection, …)` (atomic DELETE-then-INSERT per caller_entity_id; `unresolved.rs:20-49`). +- **Query helpers** — ~29 `pub` items in `query.rs`: row structs (`EntityRow`, `CallEdgeMatch`, `ContainedEntities`, `ResolvedFile`, `ResolvedFileCatalogEntry`, `ModuleDependencyEdge`, `SubsystemMember`, `UnresolvedCallSiteRow`, `ReferenceEdgeMatch`, `ReferenceDirection`, `CanonicalProjectPath`), and lookup functions (`entity_by_id`, `entity_at_line`, `find_entities`, `existing_entity_ids`, `call_edges_from`, `call_edges_targeting`, `reference_edges_for_entity`, `module_dependency_edges`, `subsystem_members`, `subsystem_for_member`, `unresolved_call_sites_for_caller`, `unresolved_callers_for_target`, `candidate_entities_for_unresolved_sites`, `contained_entity_ids`, `child_entity_ids`, `resolve_file`, `resolve_file_catalog_entry`, `entity_briefing_block_reason`, `normalize_source_path`). +- **Error type** — `StorageError` (`error.rs:5-46`) with `is_foreign_key_violation()` classifier for callers (notably the MCP envelope) that need `retryable=false` on FK breaches. +- **Helper inventory** — `known_scan_time_edge_kinds()` iterator (`writer.rs:506-511`) exposes the 9-kind ontology to callers that need to validate before sending. + +### Dependencies +- **Inbound (who calls this):** `clarion-cli` (`analyze.rs`, `serve.rs`, `install.rs`, `http_read.rs`, `run_lifecycle.rs`, `secret_scan.rs` + submodules `anchors.rs`/`findings.rs`); `clarion-mcp` (`lib.rs`, `tests/storage_tools.rs`). No other Clarion crate touches it. +- **Outbound (what this calls):** `clarion-core` for `EdgeConfidence` (`commands.rs:14`) and `manifest::RESERVED_ENTITY_KINDS` (`writer.rs:427`). Cargo-level: `rusqlite` 0.31 (`bundled`), `deadpool-sqlite` 0.8 (`rt_tokio_1`), `tokio` (`sync`, `task::spawn_blocking`), `serde`/`serde_json` (validate `properties_json` shape on inferred edges), `blake3` (declared dep but not imported in `src/`; likely intended for path/hash helpers), `tracing`, `thiserror`. +- **External services:** SQLite only — one file at `.clarion/clarion.db`, accessed via two distinct connection populations: one writer connection owned by the actor task, and `max_size` reader connections in `deadpool-sqlite`. No network. WAL is the cross-process coordination mechanism between writer and readers. + +### Internal architecture + +**Concurrency model — single-writer actor + reader pool (ADR-011, repeated verbatim in `lib.rs:1-5`).** All persistent mutations are funnelled through one `tokio::task::spawn_blocking` task that owns the sole write `rusqlite::Connection` (`writer.rs:86-98`). Producers communicate via a bounded `tokio::sync::mpsc::Sender` (default capacity 256); each variant carries a `oneshot::Sender` for the per-command ack so callers can fan out N producers (`Writer::sender().clone()`) and still observe each command's outcome (`commands.rs:20`, `writer.rs:998-1060` test `cloned_senders_accept_concurrent_entity_producers`). The actor loop is `rx.blocking_recv()` on the spawn_blocking thread (`writer.rs:152`), not `await` — this is correct because the loop is blocking-thread-resident and must call synchronous `rusqlite` APIs. + +**Transaction discipline — per-N-writes batches inside a per-run super-transaction.** `BeginRun` issues `INSERT INTO runs … status='running'` then `BEGIN`. Every successful `InsertEntity`/`InsertEdge`/`InsertFinding`/`ReplaceUnresolvedCallSitesForCaller` increments `state.writes_in_batch`; when it hits `batch_size` (default 50), `bump_writes_and_maybe_commit` issues `COMMIT` then `BEGIN` immediately to re-open. `CommitRun` runs the parent↔contains-edge consistency check inside the still-open transaction (`writer.rs:894-918`), then atomically folds the `UPDATE runs SET status=… completed_at=… stats=…` into the same `COMMIT`. `FailRun` issues `ROLLBACK` then a separate single-statement run-row UPDATE. The actor also has a `cleanup_after_channel_close` (`writer.rs:262-281`) that rolls back any open tx and marks the run failed if the channel is dropped mid-run. `FlushRunBatch` exists to let readers on separate SQLite connections observe in-flight graph rows mid-run by committing and re-opening the batch. The query-time MCP writes (`InsertInferredEdges`, `UpsertSummaryCache`, `TouchSummaryCache`) commit the surrounding run-batch first (if any), execute outside the run transaction, then re-open the run batch — they explicitly do not require an active run (`writer.rs:855-877`). + +**Write-side wire contracts (enforced before any SQL).** `enforce_entity_kind_contract` rejects non-`core` plugins emitting `RESERVED_ENTITY_KINDS` (`writer.rs:425-438`). `enforce_edge_contract` (`writer.rs:521-582`) splits the 9-kind ontology into `STRUCTURAL_EDGE_KINDS` (`contains`, `in_subsystem`, `guides`, `emits_finding` — must be `confidence=resolved`, must have NULL byte ranges) and `ANCHORED_EDGE_KINDS` (`calls`, `references`, `imports`, `decorates`, `inherits_from` — must have both byte-start and byte-end, must NOT be `inferred` at scan time); any unknown kind raises `CLA-INFRA-EDGE-UNKNOWN-KIND`. `validate_source_file_anchor` (`writer.rs:444-493`) checks that any `source_file_id` reference points to an entity whose kind is `file` or `module` (the comment at line 442 calls this transitional until core-minted `file` entities land). `parent_contains_mismatch` (`writer.rs:958-1021`) runs two SQL queries — one in each direction — to assert that `entities.parent_id` and `edges WHERE kind='contains'` are bijective; failure aborts the entire run with code `CLA-INFRA-PARENT-CONTAINS-MISMATCH`. + +**Idempotence and dedupe semantics.** `InsertEntity` uses `ON CONFLICT(id) DO UPDATE` that preserves `created_at` + `first_seen_commit` while refreshing `updated_at` + `last_seen_commit` (`writer.rs:362-420`); this is what makes `clarion analyze` re-run-safe. `InsertEdge` uses `INSERT OR IGNORE` on natural PK `(kind, from_id, to_id)`; dedupes increment `dropped_edges_total`. `InsertInferredEdges` first DELETEs the caller's existing inferred-tier `calls` edges that don't match the current cache key, then inserts new ones while *also* short-circuiting any pair that already has a `resolved`/`ambiguous` edge (`static_call_edge_exists`, lines 758-771) — that's how query-time LLM inference cohabits with scan-time static facts without double-counting. + +**PRAGMA discipline (`pragma.rs:16-45`).** Writer connection: `journal_mode=WAL` (and aborts with `PragmaInvariant` if SQLite returns anything other than `wal`), `synchronous=NORMAL`, `busy_timeout=5000` (5s — but the writer is single-threaded so this only matters against external `sqlite3` shell access), `wal_autocheckpoint=1000` (auto-checkpoint every 1000 frames), `foreign_keys=ON`. Reader connections: only `busy_timeout=5000` + `foreign_keys=ON`, re-applied on every `with_reader` acquisition as a belt-and-suspenders measure since `deadpool-sqlite` has no post-create hook. No `mmap_size`, no `temp_store`, no `cache_size`, no `application_id`, no `user_version` — schema versioning is done in the application-level `schema_migrations` table only. + +**Schema migrations (`schema.rs:17-91`).** Single compile-time-embedded migration via `include_str!`. The runner reads applied versions from `schema_migrations`, tolerating the table's absence via `OptionalExtension::optional()` (with an explicit comment at lines 45-49 warning that `.ok()` would silently mask `DatabaseLocked` / `CorruptDb`). Each migration runs in its own `execute_batch` and then an `INSERT OR IGNORE INTO schema_migrations` belt-and-suspenders insert. The migration SQL itself wraps everything in `BEGIN…COMMIT` and inserts its own row. No `application_id` / `user_version` is set — drift / cross-tool collisions on the SQLite file are not detected at the DB level. + +**Error model (`error.rs:5-65`).** One `thiserror`-derived enum, ten variants, three derived from external crates (`rusqlite::Error`, three `deadpool_sqlite::*Error` types, `std::io::Error`); six application-shaped (`PragmaInvariant`, `Migration{version, source}`, `InvalidQuery`, `InvalidSourcePath`, `WriterGone`, `WriterProtocol`, `WriterNoResponse`). `is_foreign_key_violation` is a public classifier on SQLite extended code 787, documented as feeding the MCP envelope's `retryable=false` decision. + +### Patterns observed +- **Actor + bounded mpsc channel + oneshot ack-per-command** (`writer.rs:74-140`, `commands.rs:20`). Producers can be cloned freely; back-pressure is the channel's `send().await`. +- **Per-connection PRAGMA reapply on every reader-pool acquisition** as a workaround for `deadpool-sqlite` having no post-create hook (`reader.rs:96-107`, comment at lines 76-83). +- **`Arc<()>` identity tag** for proving two `ReaderPool` clones share the same in-process pool, distinct from "same file path" (`reader.rs:25-70`, used by `clarion-cli/src/serve.rs:63-71`'s `Arc::ptr_eq` assertion). +- **Plain-old-data records at the wire** (`commands.rs:48-133`): `EntityRecord` / `EdgeRecord` / `FindingRecord` / `InferredCallEdgeRecord`. Caller is responsible for timestamps, content_hash, JSON-string encoding; writer inserts verbatim. No serde derivation on these. +- **Codified error subcodes embedded in error messages** (`CLA-INFRA-RESERVED-ENTITY-KIND`, `CLA-INFRA-SOURCE-FILE-MISSING`, `CLA-INFRA-SOURCE-FILE-KIND-CONTRACT`, `CLA-INFRA-EDGE-CONFIDENCE-CONTRACT`, `CLA-INFRA-EDGE-SOURCE-RANGE-CONTRACT`, `CLA-INFRA-EDGE-UNKNOWN-KIND`, `CLA-INFRA-PARENT-CONTAINS-MISMATCH`). Strings are load-bearing for downstream `runs.stats.failure_reason` parsing. +- **`WITHOUT ROWID` on the natural-key `edges` table** (migration line 96) and `WITHOUT ROWID`-style PKs on the three cache/unresolved tables — composite PKs are the access path. +- **FTS5 virtual table kept in sync by AFTER {INSERT,UPDATE,DELETE} triggers** (`migrations:209-238`); summary text is extracted from `summary.briefing.purpose` JSON path. +- **Generated VIRTUAL columns over JSON properties + partial indexes** (`scope_rank`, `git_churn_count`, lines 245-262) — the CASE-mapped `scope_rank` exists specifically because the project→subsystem→…→function ordering is not lexicographic. +- **Embedded migrations + edit-in-place policy until first external operator** (migration header lines 10-16) — comment names the retirement trigger for the policy. + +### Concerns / Smells / Risks +- **`query.rs` is 1161 LOC of mostly hand-written SQL strings**. No prepared-statement caching at the module level, no query builder — every helper builds its own SQL with `params!`/`params_from_iter`. Risk: easy to drift between similar helpers; partially mitigated by the 1137-LOC `tests/query_helpers.rs` (close to 1:1 LOC with the production file). Reviewers must check each new query for FK/composite-PK access path. +- **No `application_id` / `user_version` discipline**. The migration runner is the *only* schema-identity signal — a non-Clarion SQLite file opened at `.clarion/clarion.db` would pass `apply_migrations` if it happened to lack a `schema_migrations` table, and would *fail confusingly* if it has unrelated tables that collide. Cross-tool DB-file drift is not detectable at open time. +- **`blake3` is a declared dep but unused inside `src/`** (verified by `grep -rn blake3 src/` — no hits). Likely vestigial from an earlier hashing plan; not load-bearing but is build-time waste. +- **`busy_timeout=5000` is set on *both* connections, but the writer is single-thread**. The 5s only matters if an external `sqlite3` shell or a second Clarion process opens the file. There is no in-process advisory lock (no `cross-process claim-lease` table) — two `clarion analyze` runs against the same `.clarion/clarion.db` would race on `runs` rows and the SQLite writer mutex; the `recover_preexisting_running_runs` step in `clarion-cli/src/run_lifecycle.rs` is the only mitigation, and it's post-hoc. +- **`writer.rs` is 1074 LOC in one file with ~25 free functions plus the `ActorState` struct**. The match-and-dispatch loop is large; adding a 12th command means touching the `WriterCmd` enum, the match in `run_actor`, *and* the per-command handler — three places. The pattern is clean but the file is approaching "split me" territory. +- **`reader_pool` test (line 91) uses `max_size = 1`** as the *exhaustion scenario* — fine for the test, but worth flagging that the real production pool sizes in callers are 16 (`serve.rs`) and 4 (`http_read.rs`'s test seam), so the worst-case queue depth under MCP load is `64` (from `ConcurrencyLimitLayer`) waiting on `16` connections plus one writer holding the FS lock — the math hasn't been load-tested in this crate's tests. +- **`InsertEntity`/`InsertEdge` validate source-file anchors with a `SELECT kind FROM entities WHERE id = ?` per write** (`writer.rs:451-458`). At 500k entities this is 500k extra round-trips during analyze. The query is single-row PK lookup so it's cheap, but it's per-write overhead that the test corpus may not exercise to scale. +- **Tests are well above 1:1 LOC ratio with production code** (4871 test : 3199 src). Behavioural coverage looks strong (round-trip, idempotence, FK propagation, edge contracts, FailRun rollback, channel-close cleanup, query helpers, cache lifecycle, reader pool concurrency / queueing / panic recovery, schema-apply re-run safety). Concern: there is no fuzz/property-test seam for `WriterCmd` interleavings — only scripted scenarios. The `commits_observed` counter is the canonical batch-cadence oracle and is asserted in `writer_actor.rs:943` (`batch_size_fifty_commits_every_fifty_inserts`), but the channel-close-mid-run cleanup path (`cleanup_after_channel_close`, `writer.rs:262-281`) does not appear to have a dedicated test asserting the run-row is left in `failed` rather than `running`. +- **`FlushRunBatch` exists for cross-connection visibility but its consumers are external to this crate**. Within `clarion-storage` no test exercises a reader observing a flushed-but-uncommitted-run batch — that contract is asserted from `clarion-cli` / `clarion-mcp` tests instead, so a regression in `FlushRunBatch` semantics would surface as a downstream test failure rather than a unit-level one here. + +### Confidence: High +Read all 10 source files in `src/` end-to-end, plus `migrations/0001_initial_schema.sql` end-to-end, plus skimmed test-function headers in all 5 test files (notably `writer_actor.rs` 2440 LOC). Cross-checked inbound callers via `grep -rn "use clarion_storage"` — only `clarion-cli` (7 files) and `clarion-mcp` (lib + one test) import it. Confirmed `Writer::spawn` and `ReaderPool::open` call sites in `clarion-cli/src/{analyze,serve,http_read}.rs`. The only items I deliberately did not exhaust are the bodies of the read-side query helpers in `query.rs` past line 220 — I read the public surface and signatures, sampled `entity_at_line`/`find_entities`/`call_edges_from` lightly, and trusted the test file (1137 LOC of `query_helpers.rs`) as evidence of behavioural coverage rather than re-deriving each SQL statement. + +## 3. clarion-cli + +**Location:** `crates/clarion-cli/` +**LOC:** ~6981 src across 13 `.rs` files (4 of them in `src/secret_scan/`); 5894 LOC tests across 6 integration test files. +**Crate type / role:** binary crate; produces the `clarion` executable (`[[bin]] name = "clarion"`, `Cargo.toml:12-14`). No `lib.rs`, no inbound Rust callers — this crate is a pure orchestrator that wires `clarion-core`, `clarion-mcp`, `clarion-scanner`, and `clarion-storage` into three subcommands. + +### Responsibility + +`clarion-cli` owns the operator-facing entry surface: the `clarion` binary, its three subcommands (`install`, `analyze`, `serve`), and all orchestration logic that sits *between* the user invoking the command and the lower-level libraries doing the work. It owns the `.clarion/` filesystem layout (`install.rs:20-100`, `instance.rs:10-86`), the `analyze` pipeline that walks the tree, gates on secrets, fans out to discovered plugins, persists results, and clusters modules into subsystems (`analyze.rs:75-645`), the `serve` supervisor that runs MCP stdio and the Axum HTTP read API as two threads sharing one `ReaderPool` (`serve.rs:20-227`), and the federation HTTP read surface itself (`http_read.rs:363-387` router; `:392-461` bearer + HMAC middleware). The federation-visible `/api/v1/files*` endpoints live in this crate, not in `clarion-mcp` — a noteworthy split. + +### Key components + +- `src/main.rs:1-78` — binary entry. Three-arm `match` on `cli::Command`; builds a multi-thread tokio runtime *only* for `analyze`. `dotenvy::dotenv()` is loaded for `install`/`serve` but deliberately **skipped for `analyze`** (`main.rs:23-25`) so `.env` contents flow through the secret scanner instead of being imported into plugin subprocess envs. `analyze` exits with `EX_CONFIG=78` on a misconfigured `--allow-unredacted-secrets`. +- `src/cli.rs:1-63` — clap derive definitions; three subcommands, five `analyze`-time flags. +- `src/install.rs:109-194` — initialises `.clarion/{clarion.db, config.json, .gitignore}` plus a `clarion.yaml` stub at project root. `populate_after_mkdir` is wrapped in a cleanup guard (`:142-153`) that `rm -rf`s `.clarion/` if any post-mkdir step fails so retry isn't blocked by "already exists". +- `src/analyze.rs:75-645` — `run_with_options`, the central analyze pipeline. Single async function (~570 LOC) that runs the entire flow described in §"Internal architecture" below. +- `src/serve.rs:20-227` — `serve::run` + `supervise_stdio_with_http`. Builds the LLM provider (`build_llm_provider:229-291`), opens a 16-conn `ReaderPool`, spawns the HTTP server thread, spawns the MCP stdio thread, then polls the stdio result channel with a 100 ms timeout while periodically checking HTTP health (`:176-202`). On `Arc::ptr_eq` mismatch between the HTTP thread's reported `ReaderPool` identity and the one held in `serve.rs`, `debug_assert!` fires (`:63-71`) — this catches a refactor that re-opens the pool inside `http_read::spawn`. +- `src/http_read.rs:146-260` — `spawn` / `spawn_with_env`; `:262-311` `run_http_read_server` (binds TCP, sends back ready signal with `ReaderPool` identity captured *after* the move into the runtime thread); `:363-387` `router`; `:392-461` `require_http_identity` + `require_hmac_identity`; `:692-1034` six request handlers (`get_file`, `post_files_batch`, `post_files_resolve`, `get_capabilities`, plus two cfg(test)-only fixtures referenced in `:329-353`). +- `src/clustering.rs:53-101` — `cluster_modules` entry point + `cluster_modules_with_algorithms` test seam; `:117-145` `leiden_communities` calling `xgraph::graph::algorithms::leiden_clustering`; `:170-224` `local_weighted_components` fallback; `:247-296` directed modularity score; `:103-115` `cluster_hash` = SHA-256 of sorted member IDs truncated to 12 hex chars. +- `src/secret_scan.rs:202-325` — `pre_ingest`; submodules `anchors.rs` (links scanner detections to plugin-emitted module/file entities), `baseline.rs` (loads `.clarion/secrets-baseline.yaml`), `files.rs` (extension/skip-dir walk + sidecar matcher for `.env`/`.env.*`/`*.env`), `findings.rs` (`PendingFinding` + `FindingRecord` shaping with `CLA-SEC-SECRET-DETECTED` / `CLA-SEC-UNREDACTED-SECRETS-ALLOWED` rule IDs). +- `src/run_lifecycle.rs:6-44` — two helpers: `recover_preexisting_running_runs` (raw `UPDATE runs SET status='failed' WHERE status='running'` on next start; not routed through the writer-actor because the writer isn't running yet) and `begin_run` (writer `BeginRun` wrapper). +- `src/instance.rs:44-86` — `InstanceId(Uuid)` newtype persisted to `.clarion/instance_id` with `O_CREAT|O_EXCL` (`mode = 0o600`) → `fsync` → `hard_link` for atomic publish, and `EEXIST`-on-link → read-existing-file race resolution. Federation contract requires a stable per-project ID; this is it. + +### Public interface (outbound) + +`clarion-cli` exposes no `pub` library API — it has no `lib.rs`. Its public surface is three things: + +- **The `clarion` binary CLI** — `clarion install [--force] [--path P]`, `clarion analyze [P] [--config C] [--allow-unredacted-secrets [--confirm-allow-unredacted-secrets TOKEN]] [--allow-no-plugins]`, `clarion serve [--path P] [--config C]` (`cli.rs:13-62`). Confirmation token literal is `"yes-i-understand"` (`secret_scan.rs:38`). +- **The HTTP read API** — Axum router at `http_read.rs:363-387`. Four production routes (one unprotected): + - `GET /api/v1/files?path=&language=` → `get_file`. Returns `{entity_id, content_hash, canonical_path, language}`; honours `If-None-Match` ETag (= `\"\"`) returning 304. 403 with `BRIEFING_BLOCKED` if entity has `briefing_blocked` property set by secret scan. Protected (bearer or HMAC). + - `POST /api/v1/files/batch` body `{queries:[{path,language},…]}` → returns `{resolved[], not_found[], briefing_blocked[], errors[]}`. Hard cap **256** queries/batch (`http_read.rs:608`), runs the whole batch in a single `with_reader` checkout. Protected. + - `POST /api/v1/files:resolve` body `{paths:[…]}` → returns `{results:[{path,response:{status,body}}]}` where `status ∈ {resolved,not_found,blocked,error}`. Hard cap **1000** paths (`http_read.rs:609`). Protected. + - `GET /api/v1/_capabilities` → `{registry_backend, file_registry, api_version, instance_id}`. **Unprotected by design** so siblings can probe pre-auth. + - Body cap **16 KiB** (`HTTP_BODY_LIMIT_BYTES`, `http_read.rs:610`). Tower stack (`:373-386`): `CatchPanicLayer` (panics → 500 envelope) → `HandleErrorLayer` (panics on unenumerated middleware errors; `:547-556`) → `TraceLayer` → 10 s `TimeoutLayer` → `RequestBodyLimitLayer` → `LoadShedLayer` → `ConcurrencyLimitLayer(64)`. Errors carry an envelope `{error, code}` with `code ∈ {INVALID_PATH, PATH_OUTSIDE_PROJECT, NOT_FOUND, BRIEFING_BLOCKED, UNAUTHENTICATED, STORAGE_ERROR, BATCH_TOO_LARGE, INTERNAL}` (`http_read.rs:590-601`). +- **`clarion serve`'s stdio MCP server** — owned by `clarion-mcp`. `clarion-cli` calls `clarion_mcp::serve_stdio_with_state_on_runtime` (`serve.rs:150`) and constructs the `ServerState` with the reader pool, optional summary-LLM writer + `LlmProvider`, and optional `FiligreeHttpClient` (`serve.rs:131-147`). + +### Dependencies + +- **Inbound (who calls this):** No Rust callers — verified by `grep -rn "clarion_cli" crates/` returning empty. This crate is invoked by humans/CI through the `clarion` binary and by `tests/e2e/*.sh` shell scripts. +- **Outbound (other Clarion crates):** `clarion-core` (`AcceptedEntity`/`AcceptedEdge`, `discover`, `PluginHost`, `CrashLoopBreaker`, `BriefingBlockReason`, `HostError`, `HostFinding`, LLM provider types), `clarion-mcp` (`ServerState`, `config::McpConfig`, `config::HttpReadConfig`, `filigree::FiligreeHttpClient`, `serve_stdio_with_state_on_runtime`, `select_provider_with_env`, `resolve_filigree_http_target`), `clarion-storage` (`Writer`, `WriterCmd`, `ReaderPool`, `module_dependency_edges`, `resolve_file_catalog_entry`, `CanonicalProjectPath`, `StorageError`, `pragma`, `schema`), `clarion-scanner` (`Scanner`, `Detection`, `Baseline`, `SuppressionResult`). +- **External crates of note:** `axum` 0.7 (HTTP), `tower`/`tower-http` (middleware), `tokio` (runtimes — multi-thread for analyze + HTTP, current-thread for MCP stdio), `clap` (CLI), `xgraph` (Leiden clustering), `ignore::WalkBuilder` (`.gitignore`-honouring tree walk in both `analyze.rs:2016-2026` and `secret_scan/files.rs`), `rusqlite` (one direct `Connection::open` in `analyze.rs:711` to read module IDs/edges during phase 3, bypassing the reader pool), `uuid`, `dotenvy`, `sha2` (HMAC, `cluster_hash`). +- **External services / processes:** + - **SQLite** at `/.clarion/clarion.db`, opened both via `ReaderPool::open` (`serve.rs:53`) and direct `Connection::open` (`run_lifecycle.rs:10`, `analyze.rs:711`, `install.rs:189`). + - **Plugin subprocesses** — spawned via `clarion_core::PluginHost::spawn` from `run_plugin_blocking` (`analyze.rs:1322`) on `tokio::task::spawn_blocking` workers. + - **TCP listener** for HTTP read API on `config.serve.http.bind` (default `127.0.0.1:9111` per `install.rs:74`); bound by `tokio::net::TcpListener::bind` inside the HTTP thread's runtime (`http_read.rs:279`). + - **LLM CLIs and OpenRouter** — wired via `build_llm_provider` (`serve.rs:229-291`); reaches `codex`/`claude` binaries and OpenRouter HTTPS. + +### Internal architecture + +**Three subcommands, three concurrency shapes.** `install` is fully synchronous (`install.rs:109`). `analyze` builds a multi-thread tokio runtime in `main.rs:36-38` and runs one big `async fn run_with_options` (`analyze.rs:75-645`); per-plugin work happens inside `tokio::task::spawn_blocking` (`analyze.rs:325-339`) using a **Pattern A buffering** strategy named in the module doc (`analyze.rs:6-8`): collect all entities + edges into in-memory `Vec`s inside the blocking task, return them to the async side, then drain into the writer-actor via `WriterCmd::InsertEntity`/`InsertEdge` one-at-a-time with `send_wait` ack-and-block (`analyze.rs:399-447`). `serve` is the most complex (`serve.rs`): a foreground supervisor thread (the process's original thread) polls a `std::sync::mpsc::Receiver` from the spawned `"clarion-mcp-stdio"` thread (current-thread tokio runtime, `:126-130`) while also `check_running`-polling the `"clarion-http-read"` thread (multi-thread tokio runtime, `http_read.rs:355-360`). Identity of the single shared `ReaderPool` is asserted by `Arc::ptr_eq` on the inner `Arc<()>` tag (`serve.rs:63-71`); the HTTP thread captures its identity **after** the pool has been moved into its runtime (`http_read.rs:272-296`), so a refactor that opened a fresh pool inside the HTTP thread would surface immediately rather than silently double-opening WAL. + +**Analyze pipeline ordering (`analyze.rs:75-645`, single function):** +1. Path canonicalisation + `.clarion/` existence check (`:82-91`). +2. Load `clarion.yaml`-derived `AnalyzeConfig` (clustering knobs only; LLM/serve config is `serve`-only) — `config.rs:16-29`. +3. **Run-row crash recovery** — `recover_preexisting_running_runs` flips any prior `status='running'` rows to `failed` (`:95-102`, `run_lifecycle.rs:6-26`). +4. **Spawn writer-actor** + mint run_id (`:105-113`). +5. **Plugin discovery** — `clarion_core::discover()` scans `$PATH` for `clarion-plugin-*` (`:116-135`). Empty-with-errors → FailRun + non-zero exit (`:142-174`); empty-without-errors → `RunStatus::SkippedNoPlugins` + non-zero exit unless `--allow-no-plugins` (`:176-223`). +6. **Build the wanted-extensions union** across plugin manifests (`:226-231`). +7. **Walk the tree** — `collect_source_files` with `ignore::WalkBuilder` honouring `.gitignore` family + a hard-coded skip-dir list (`:234`, `:2013-2065`). +8. **Pre-ingest secret scan** — `secret_scan::collect_scan_files` (broader walk that includes `.env` sidecars) → `secret_scan::pre_ingest` runs `clarion-scanner` patterns in **parallel via `thread::scope` with `available_parallelism()` workers** (`secret_scan.rs:333-382`), applies the baseline, classifies each file as `Clean`/`Blocked`/`Overridden` (`:237-243`). **The scan happens before `BeginRun` and before any plugin spawn**, so plugin processes never receive `.env` contents in their analyze stream — they receive `briefing_blocked` flags instead. +9. **`BeginRun`** (`:244`). +10. **Per-plugin loop** (`:275-470`): filter source files to the plugin's extensions; `spawn_blocking` `run_plugin_blocking` (`analyze.rs:1309-1456`) which (a) calls `PluginHost::spawn` for the JSON-RPC handshake, (b) iterates files calling `host.analyze_file()` with periodic heartbeat logging (`:1272-1307`), (c) collects entities + edges + unresolved-call-sites + per-file stats, (d) issues `host.shutdown()` on success or `child.kill()` on error, (e) reaps the child and classifies SIGKILL/SIGSEGV as likely `RLIMIT_AS` OOM-finding (`:1466-1574`). Per-plugin crashes record on a `CrashLoopBreaker`; >3 crashes / 60 s trips the breaker, drops the remaining plugins, and surfaces `FINDING_DISABLED_CRASH_LOOP` (`:271-272, 349-358`). +11. **Entity → edge ingestion** for surviving plugins, in order: `InsertEntity` for collected entities, `ReplaceUnresolvedCallSitesForCaller` for query-time-resolvable call sites, then `InsertEdge` (`:399-448`). Order is load-bearing: edges FK-reference entities (`:392-394` comment). +12. **`persist_findings`** writes the secret-scan findings now that we have anchor entity IDs from plugin output (`:471-481`, `secret_scan/anchors.rs`). +13. **Phase 3 clustering** (`:500-520`, `run_phase3_clustering` at `:675-906`): `FlushRunBatch`, open a direct read connection, load module entity IDs + module-dependency edges, build `ModuleGraph`, run `cluster_modules`, emit one `core:subsystem:` entity + N `in_subsystem` edges per community, emit a `CLA-FACT-CLUSTERING-WEAK-MODULARITY` finding if `modularity_score < weak_modularity_threshold` (default 0.3). +14. **`CommitRun(Completed)`** / **`CommitRun(Failed)` for soft-fail (some plugins crashed but writer-actor was healthy)** / **`FailRun` for hard-fail (writer-actor rejection or phase 3 error)** (`:543-625`). The hard/soft distinction matters: SoftFailed commits the partial entity batch *and* marks the run failed atomically (`:576-612`); HardFailed rolls back the open transaction via `FailRun`. + +**Clustering (`clustering.rs`):** Two algorithms, both deterministic. Leiden via `xgraph::graph::algorithms::leiden_clustering::CommunityDetection::detect_communities_with_config` with `{gamma, resolution, iterations, deterministic: true, seed}` (`:124-131`). If Leiden returns ≤1 community, fall back to a hand-rolled `local_weighted_components` (`:170-224`) keyed on average-positive-edge-weight as the threshold; the algorithm actually used is reported back in `ClusterResult.algorithm_used`. Directed modularity is recomputed locally on the resulting partition (`:247-296`) — `xgraph` returns communities, not the score. `cluster_hash` (`:103-115`) is SHA-256 of sorted member entity IDs truncated to 12 hex chars; this seeds the `core:subsystem:` entity ID so the same community shape is byte-stable across runs. + +**HTTP read API (`http_read.rs`):** Single `Router` built once at `spawn`. Two-layer auth: the **HMAC path is preferred** when `identity_token_env` is set (`:397-398`), falling back to **bearer** when only `token_env` is set, falling back to **trust-loopback** (`auth = "none"`) when neither is set — with an operator-visible WARN at spawn time (`:244-252`). HMAC canonical message is `"{method}\n{path_and_query}\n{hex(sha256(body))}"` (`:478-484`), HMAC is SHA-256 hand-rolled (`:487-509`) — no `hmac` crate dependency. Constant-time compare on both bearer and HMAC paths (`:415, 456, 521-530`). Two failure modes get explicit non-500 statuses: timeout → 408 (`:533-538`), load-shed → 503 (`:540-546`); anything else from `BoxError` is a `panic!()` with the error chain in the message (`:553-556`) — `CatchPanicLayer` turns it into a 500 envelope, but CI sees the missing enumeration. + +**Error model:** `anyhow::Result` everywhere at the binary layer (`main.rs:13`, `serve.rs:8`); typed `StorageError` from `clarion-storage` is classified into HTTP `ErrorCode` enum + status by `classify_read_error` (`http_read.rs:1050-1085`). + +### Patterns observed + +- **Supervisor + worker threads with shared `Arc<()>` identity tag** (`serve.rs:63-71`, `http_read.rs:141-144, 272-296`) — prevents silent double-open of the WAL pool across the refactor surface. +- **Pattern A buffering** for plugin output: collect-in-blocking-task, send-from-async-task (`analyze.rs:325-339, 399-448`) — keeps the writer-actor single-threaded discipline while letting blocking JSON-RPC reads happen on `spawn_blocking`. +- **Atomic file publish via hard-link rename** for `instance_id` (`instance.rs:53-86`) — handles racing concurrent installs without an external lock. +- **Crash-loop breaker as an in-process circuit** (`analyze.rs:271-272, 349-358`) — borrowed from `clarion_core::CrashLoopBreaker`, applied per-run. +- **Two-pass HTTP body read** in the HMAC path: `to_bytes` to compute the digest, then reconstruct `Request::from_parts(parts, Body::from(body_bytes))` so the handler still sees a normal body (`http_read.rs:442-461`). Necessary because HMAC needs the full bytes, not a stream. +- **Constant-time secret comparison** on both bearer and HMAC paths (`http_read.rs:521-530`) — defends against timing attacks even on the loopback default. +- **Test seam via function-pointer injection**: `cluster_modules_with_algorithms` takes the Leiden and weighted-components closures as parameters (`clustering.rs:60-65`); tests substitute hand-rolled communities to force the fallback path (`:457-482`). +- **`#[cfg(test)]` panic injection** for supervisor testing: `HTTP_THREAD_PANIC_TRIGGER` static atomic that a test can flip to assert the supervisor's runtime-panic arm still fires after `CatchPanicLayer` was introduced (`http_read.rs:321-353`). A rare and disciplined use of cfg-conditional production code. +- **Run-lifecycle crash recovery before writer-actor spawn** (`run_lifecycle.rs:6-26`, `analyze.rs:95-102`) — raw `rusqlite` `UPDATE` because the writer can't recover its own pre-mortem state. + +### Concerns / Smells / Risks + +- **`analyze.rs` is 2549 LOC in a single file**, and `run_with_options` itself spans 570 lines (lines 75–645) with `#[allow(clippy::too_many_lines)]` (`:74`). The function is doing path setup, run lifecycle, discovery error policy, no-plugin policy, tree walk, secret scan, per-plugin orchestration, ingestion ordering, soft/hard-fail promotion, phase 3 clustering invocation, and run finalisation — every one of those is a separate change vector. The module doc and inline comments are excellent (some of the most carefully reasoned `//` comments in the workspace), but extracting `discovery_outcome`, `per_plugin_loop`, and `finalise_run` into named functions would materially reduce the surface that has to be re-read on each touch. Watch this file for change-amplification scars over time. +- **`http_read.rs` is 1723 LOC with all routes, all auth, both error envelopes, and the test-only panic harness in one module**. The separation between protected and unprotected sub-routers is already in place (`:363-372`) — a future refactor could lift each handler family into a sibling module (`files.rs`, `batch.rs`, `resolve.rs`) without changing the router shape. +- **Two parallel `WalkBuilder` traversals** of the project tree: one in `analyze::collect_source_files` (`analyze.rs:2013`, extension-filtered) and a broader one in `secret_scan::collect_scan_files` (sidecars + same extensions). Both honour `.gitignore`. On a large repo this is two full I/O passes back-to-back; no caching between them. +- **Phase 3 clustering opens a raw `rusqlite::Connection` bypassing the writer-actor's view of the WAL** (`analyze.rs:711`). The `FlushRunBatch` call at `:705-709` is what makes this safe — the writer-actor must drain its open transaction so the direct connection sees the same module rows. The dependency between `FlushRunBatch` and the subsequent direct read is implicit; a refactor that removes the flush would silently produce empty clusters on a busy run. +- **HMAC implementation is hand-rolled** (`http_read.rs:487-509`) rather than using the `hmac` + `sha2` crates' `Hmac` type. The code is short and the `constant_time_eq` is paired with it correctly, but a deps-level choice not to pull `hmac` is a maintenance bet — any future HMAC variant (rotation, key derivation) reopens this. Worth tracking. +- **Unenumerated middleware error → panic → 500 envelope** (`http_read.rs:553-556`). The design intent is sound (force enumeration via CI failure rather than silently swallow), but in production the user sees a 500 with a vague message; the panic message goes to stderr only. The behaviour is deliberate per the inline comment, but it deserves a runbook entry. +- **`run_with_options` does not bound the in-memory entity buffer**: `collected_entities` and `collected_edges` for a plugin run accumulate every entity for every file in `Vec`s before any are flushed to the writer-actor (`analyze.rs:1337-1419`). On a very large corpus a misbehaving plugin could exceed memory limits before the entity-cap breaker in `clarion-core` trips. The cap exists but lives in the host, not here. +- **`scanned_files: Arc>` is constructed by the secret scan and then shared into every `PluginHost`** so the host can enforce the briefing-block contract on per-file results (`analyze.rs:273-274, 314-315, 333-334`). This is a clean immutable share, but it does mean every plugin process holds (via the host wrapper) a reference to a `BTreeSet` whose footprint scales with the project source count. +- **No test coverage for the supervisor's "stdio thread exited but result_rx was disconnected" branch** at `serve.rs:194-200` was visible in my sampling pass — only a unit test for `finish_supervised_result` (`:313-325`) and the e2e shell scripts exercise serve. The `tests/serve.rs` file is 3075 LOC, so this is probably covered; flagged for explicit verification. +- **`recover_preexisting_running_runs` swallows error context one level** by going through `anyhow!("{err}")` (`run_lifecycle.rs:12-14`) — typed `StorageError` info is folded into a plain string. Mild. + +### Confidence: High + +Read every source file in `src/` end-to-end except `analyze.rs`, where I read the module doc, all top-level fn signatures, `run_with_options` lines 75-645 in full, `run_phase3_clustering` lines 675-906, `run_plugin_blocking` lines 1309-1456, the source walk lines 2013-2065, and the time helper. Read `http_read.rs` lines 1-260 (transport / spawn / supervisor identity), 260-461 (server bootstrap + auth + HMAC), 692-1034 (handlers + capabilities), 1036-1085 (error classifier), 1107-1148 (logging + panic envelope). Read all four `secret_scan/` submodule heads. Listed inbound callers (`grep -rn "clarion_cli"`) and confirmed none — this is a leaf binary. Did **not** read the test files end-to-end (5894 LOC), only their line counts and the cfg(test) harness inside `http_read.rs` and `clustering.rs`. The 20 MCP tool surface and `clarion-mcp::ServerState` shape are cited only via the call sites at `serve.rs:131-147, 150`, not by reading `clarion-mcp`. + +## 4. clarion-mcp + +**Location:** `crates/clarion-mcp/` +**LOC:** 6,595 source (`lib.rs` 4,703 + `config.rs` 1,600 + `filigree.rs` 292) / 2,233 test (`tests/storage_tools.rs`) +**Crate type / role:** Library crate. Implements the MCP (Model Context Protocol) JSON-RPC tool surface that consult-mode LLM agents call to query Clarion's index; not a binary — the `clarion serve` binary in `clarion-cli` consumes this library and drives the stdio loop. + +### Responsibility + +`clarion-mcp` owns the JSON-RPC tool dispatch layer and the stdio transport that fronts Clarion's index for external agents. It defines the twenty MCP tools (`list_tools` at `src/lib.rs:56`), translates MCP `tools/call` requests into bounded `clarion-storage` reader queries, mediates LLM-dispatch for on-demand summaries and inferred call edges with token-budget reservation (`BudgetLedger` at `lib.rs:2244`, `reserve_budget` at `lib.rs:2066`), and brokers Filigree enrichment lookups over HTTP. It also owns the YAML configuration schema (`McpConfig` at `config.rs:9`) consumed by `clarion serve` for LLM provider selection, Filigree integration, and the (separate) HTTP read-API bind/auth posture. The crate exposes its surface via `ServerState`, `serve_stdio*`, `handle_json_rpc`, and `list_tools`, and is consumed exclusively by `clarion-cli` (see Inbound below). + +### Key components + +- `src/lib.rs:56-258` — `list_tools()`: the canonical 20-tool registry with names, descriptions, and JSON schemas. Single source of truth; `handle_tool_call` validates names against it (`lib.rs:401`). +- `src/lib.rs:316-2210` — `ServerState`: the stateful dispatcher. Holds `ReaderPool`, optional `SummaryLlmState` (writer mpsc + provider), optional `FiligreeLookup`, an `AnalyzeProcess` slot, a clock, an `InferredInflight` coalescer (`lib.rs:43`), and a `BudgetLedger`. Per-tool handlers (`tool_entity_at` at 493 … `tool_subsystem_members` at 1229) sit on `ServerState`. +- `src/lib.rs:2704-2876` — Transport: `handle_frame*`, `read_stdio_frame`, `serve_stdio`, `serve_stdio_with_state`, `serve_stdio_with_state_on_runtime`. Implements dual framing (LSP-style `Content-Length` and JSON-line) auto-detected from the first non-whitespace byte (`peek_stdio_frame_start` at 2774). +- `src/lib.rs:1560-2030` — Inferred-edge dispatch pipeline: cache lookup (`read_inferred_inputs` 1624), in-flight coalescing via `tokio::sync::broadcast` (`InferredInflightGuard` 2438; `coalesced_inferred_dispatch`), budget reservation, LLM provider invocation via `spawn_blocking` (`invoke_llm` 2218), and writer-channel persistence (`WriterCmd::InsertInferredEdges` at 1682/1824). +- `src/lib.rs:2237-2290` — `AnalyzeProcess` + `BudgetLedger` + `BudgetReservation`: spawns `clarion analyze` as a detached child for `analyze_start`/`analyze_status`/`analyze_cancel`; tracks reserved-vs-spent LLM tokens with a `Mutex`-guarded ledger. +- `src/config.rs:9-742` — `McpConfig` (YAML, `serde_norway`): `LlmConfig` (provider kind, model id, session token ceiling, per-caller inferred-edge cap, cache TTL), provider-specific subsections (`OpenRouterConfig`, `CodexCliConfig`, `ClaudeCliConfig`), `FiligreeConfig` (base URL, project key, token env, actor, timeout), `HttpReadConfig` (bind, loopback trust, bearer + identity HMAC env vars), and `select_provider_with_env` (677) which builds an `LlmProvider` honoring `recording_fixture_path` and `allow_live_provider`. +- `src/filigree.rs:1-292` — Filigree HTTP client: `FiligreeLookup` trait (52), blocking-reqwest implementation `FiligreeHttpClient` (64), URL builder `entity_associations_url` (148), and response shape `EntityAssociationsResponse` (11). + +### Public interface (outbound) + +- **20 MCP tools** (the consult-agent contract, from `list_tools`): + 1. `entity_at` — innermost entity containing `(file, line)`; path-normalized against project root. + 2. `project_status` — orientation snapshot: index age, latest run, graph counts, git state, plugin discovery, LLM policy, Filigree routing, analyze lifecycle. + 3. `analyze_start` — spawn background `clarion analyze` child; supports `allow_no_plugins`. + 4. `analyze_status` — child process state + latest persisted `runs` row. + 5. `analyze_cancel` — kill the background analyze child if running. + 6. `find_entity` — paginated FTS-ranked search over entity id/name/short-name/summary; does *not* search `summary_cache`. + 7. `source_for_entity` — line-numbered source span with bounded context; includes decorator lines. + 8. `entity_context` — resolve by id or `(file,line)`; returns containing stack + source + diagnostics. + 9. `call_sites` — caller/callee evidence with source snippets at recorded byte offsets. + 10. `callers_of` — callers, default confidence `resolved`; opt-in to `ambiguous` (expand candidates) or `inferred` (may trigger LLM dispatch). + 11. `execution_paths_from` — bounded calls-only traversal; default `max_depth=3`, hard `execution_edge_cap` (default 500, settable via `with_edge_cap`). + 12. `execution_paths_ranked` — compact ranked path view; optional `exclude_tests`, `max_paths`. + 13. `summary` — on-demand cached *leaf* summary; module entities return scope-deferred policy envelope, not aggregation. + 14. `summary_preview_cost` — cache status, model/provider, token estimate, live-spend requirement; no dispatch. + 15. `issues_for` — Filigree associations for an entity, optionally `include_contained`; returns `unavailable` envelope if Filigree disabled rather than failing. + 16. `orientation_pack` — deterministic first-pass packet (status, source, callers, callees, issues, next reads); no LLM. + 17. `index_diff` — freshness signals: latest run, DB mtime, source files newer than index, changed entity hashes. + 18. `neighborhood` — one-hop graph (callers, callees, container, contained, references). + 19. `subsystem_members` — module entities for a subsystem entity. + 20. (twentieth slot) `summary_preview_cost` and `summary` count separately; full enumeration above totals 19 visible — the registry actually contains 19 entries, not 20. **Note:** the discovery doc cites "twenty tools"; the registry as of `lib.rs:56-257` contains 19 distinct `ToolDefinition` entries. Flagging in Concerns. +- **Rust API:** + - `pub fn list_tools() -> Vec` (`lib.rs:56`). + - `pub fn handle_json_rpc(&Value) -> Option` (`lib.rs:295`) — state-free metadata-only path (`initialize`, `tools/list`). + - `pub struct ServerState` with builder methods `new`, `with_edge_cap`, `with_summary_llm`, `with_clock`, `with_filigree_client` (`lib.rs:316-376`); `ServerState::handle_json_rpc(&Value) -> Option` (377). + - `pub fn serve_stdio`, `serve_stdio_with_state`, `serve_stdio_with_state_on_runtime` (`lib.rs:2831-2876`). + - `pub fn handle_frame`, `handle_frame_with_state` (`lib.rs:2704-2721`) — frame-level entry points for callers that own their own loop. + - `pub enum McpError` (`lib.rs:2690`), `pub const MCP_PROTOCOL_VERSION = "2025-11-25"` (`lib.rs:40`). +- **Config API (`config` module):** `McpConfig::from_path`, `McpConfig::from_yaml_str`, `select_provider_with_env`, `resolve_filigree_http_target`, `resolve_filigree_base_url`, `HttpReadConfig::{validate_loopback_trust, validate_auth_trust, is_loopback_bind}`. +- **Filigree API (`filigree` module):** `FiligreeLookup` trait, `FiligreeHttpClient::from_config`, `entity_associations_url`, `parse_entity_associations_response`. + +### Dependencies + +- **Inbound (who calls this):** + - `crates/clarion-cli/src/serve.rs:13-150` — sole production caller. Imports `config::{McpConfig, LlmConfig, select_provider_with_env, ...}`, `filigree::FiligreeHttpClient`, `ServerState::new`, `with_summary_llm`, `with_filigree_client`, `serve_stdio_with_state_on_runtime`. Spawns the stdio loop on a named thread `clarion-mcp-stdio`. + - `crates/clarion-cli/src/http_read.rs:18` — reuses `clarion_mcp::config::HttpReadConfig` to drive the *separate* HTTP read-API server (which is implemented in `clarion-cli`, not here). + - `crates/clarion-cli/src/install.rs:68` — references the literal string `"clarion-mcp"` in template output (the default Filigree actor). +- **Outbound (what this calls):** + - `clarion-core` (`plugin::{Frame, TransportError, ContentLengthCeiling, read_frame, write_frame}`; LLM primitives `LlmProvider`, `LlmRequest`, `LlmResponse`, `LlmProviderError`, `LlmPurpose`, `build_inferred_calls_prompt`, `build_leaf_summary_prompt`, `INFERRED_CALLS_PROMPT_VERSION`, `LEAF_SUMMARY_PROMPT_TEMPLATE_ID`, `InferredCallsPromptInput`, `LeafSummaryPromptInput`, `EdgeConfidence`). + - `clarion-storage` — read side via `ReaderPool::with_reader` (~30 call sites including `entity_at_line`, `entity_by_id`, `find_entities`, `call_edges_from`, `call_edges_targeting`, `reference_edges_for_entity`, `child_entity_ids`, `contained_entity_ids`, `existing_entity_ids`, `subsystem_members`, `summary_cache_lookup`, `inferred_edge_cache_lookup`, `inferred_edge_cache_key_id`, `unresolved_call_sites_for_caller`, `unresolved_callers_for_target`, `candidate_entities_for_unresolved_sites`, `normalize_source_path`); write side strictly via `WriterCmd` over `mpsc::Sender` — only three variants emitted: `InsertInferredEdges` (twice, `lib.rs:1682`/`1824`), `TouchSummaryCache` (`lib.rs:1924`), `UpsertSummaryCache` (`lib.rs:2024`). + - External crates: `tokio` (current-thread runtime, `AsyncMutex`, `mpsc`, `oneshot`, `broadcast`, `spawn_blocking`), `reqwest::blocking` (Filigree), `rusqlite` (re-exported types only — no direct `Connection::open`; all DB access flows through `ReaderPool`/`WriterCmd`), `serde_norway` (YAML config), `time`, `blake3`, `thiserror`, `tracing`. +- **External services:** + - **SQLite** via `clarion-storage::ReaderPool` — read concurrency; writes only via the writer-actor channel. + - **Subprocess: `clarion analyze`** — spawned by `tool_analyze_start` using `std::env::current_exe()` (`lib.rs:572-599`), stdin/stdout/stderr null'd. + - **Filigree HTTP** — blocking `reqwest` GET to `{base_url}/api[/p/{project_key}]/entity-associations?entity_id=...` with `x-filigree-actor` and bearer auth (`filigree.rs:98-127`). + - **LLM providers** — pluggable `Arc` invoked via `tokio::task::spawn_blocking` (`lib.rs:2218`); config chooses OpenRouter (HTTP), Codex CLI, Claude CLI, or a deterministic Recording fixture (`config.rs:677`). + +### Internal architecture + +The crate is a single stateful dispatcher (`ServerState`) plus a thin stdio I/O loop. `ServerState::handle_json_rpc` (`lib.rs:377`) routes `initialize` and `tools/list` to static helpers, and `tools/call` to `handle_tool_call` (394) which dispatches on tool name via a 19-arm `match` (`lib.rs:413-488`). Each `tool_*` handler is `async`, takes the argument map, parses required/optional params (`required_str`, `optional_bool`, …), and either returns a JSON envelope or a `ParamError` that becomes a JSON-RPC error response. Storage reads flow through `self.readers.with_reader(|conn| …)`, which offloads onto the reader pool's blocking workers; the handler awaits the result and wraps it in `envelope_from_storage_result`. + +**Concurrency model.** The crate is built around a borrowed Tokio current-thread runtime (`serve_stdio_with_state_on_runtime` accepts `&Runtime` from the caller; `serve_stdio_with_state` builds its own). The dispatch path is fully async, but the stdio I/O is synchronous: the loop does a blocking `read_stdio_frame` on `&mut impl std::io::BufRead`, then `runtime.block_on(handle_stdio_frame_with_state(...))`, then a blocking `write_stdio_response`. There is no parallel request handling — frames are processed strictly sequentially per stdio session. Three internal concurrency primitives matter: + 1. `BudgetLedger` behind a `std::sync::Mutex` (`lib.rs:322`) — gates LLM dispatch with a reserve/commit pattern and a `blocked` latch. + 2. `InferredInflight` — `Arc>>>` (`lib.rs:43`) — coalesces concurrent `callers_of`/inferred-edge requests for the same `(caller_id, content_hash, model_id, prompt_version)` tuple so only one LLM call fires; followers subscribe via `broadcast` and `RAII`-clean up via `InferredInflightGuard` (`lib.rs:2438`). + 3. `analyze_process` — a `std::sync::Mutex>` slot for the at-most-one in-flight `clarion analyze` child. + +**State ownership.** `ServerState` owns the `ReaderPool`, the optional writer `mpsc::Sender` (cloned out of `SummaryLlmState` per dispatch), the optional Filigree client behind `Arc`, the `clock` (injectable for tests), and the analyze-process slot. The writer-actor itself lives in `clarion-cli/src/serve.rs:135-144`, not here — `clarion-mcp` is a writer *client*, never a writer host. + +**Transport.** The stdio frame reader auto-detects between LSP-style `Content-Length` framing (delegated to `clarion_core::plugin::read_frame`/`write_frame`) and bare-JSON-line framing by peeking the first non-whitespace byte (`lib.rs:2757-2789`). The choice is per-frame, recorded in `StdioFrame::framing`, and reused on the response. JSON-RPC notifications (method present, id absent) are silently dropped (`is_json_rpc_notification` at `lib.rs:2740`). Test `serve_stdio_handles_multiple_content_length_frames` (`lib.rs:4309`) and `serve_stdio_with_state_uses_json_line_transport_for_json_line_requests` (4400) pin the dual-framing contract. + +**Error model.** Two layers: `McpError` (`lib.rs:2690`) — `Json | Transport | Runtime` — surfaces only from frame I/O and JSON decode failures and aborts the loop; per-tool errors stay inside the JSON-RPC response as either error responses (`ParamError`) or success envelopes with `tool_error_envelope(code, message, retryable)` (e.g. `analyze-already-running`, `analyze-start-failed`, `token-ceiling-exceeded`, `llm-disabled`, `invalid-path`). The retryability flag is on the wire, not just in logs. `ConfigError` (`config.rs:742`) uses stable string error codes (`CLA-CONFIG-*`) so the CLI can route them. + +### Patterns observed + +- **Tool registry as single source of truth** — `list_tools()` validates incoming names at dispatch (`lib.rs:401`) and is also served verbatim to `tools/list`; adding a tool requires both a registry entry and a `match` arm, but the unreachable arm (`lib.rs:487`) is documented by the prior validation. +- **Reader pool + writer-actor split** — every DB read goes through `ReaderPool::with_reader` (~30 sites); every DB write goes through `mpsc::Sender` with an `oneshot` ack (`send_writer` helper at `lib.rs:2043`). The crate never opens a `rusqlite::Connection` directly. +- **Coalescing + RAII guards for expensive async work** — `InferredInflightGuard` and `BudgetReservation` both use `Drop` to release in-flight slots / reserved tokens even if the awaiting future is cancelled (`lib.rs:2278-2290`, `2471-2486`). The `Drop` paths for guards spanning runtime boundaries use `tokio::runtime::Handle::try_current` to schedule async cleanup. +- **Dual-framing transport with byte-peek detection** — same `serve_stdio` loop handles LSP-style and JSON-line clients without configuration (`lib.rs:2757`). +- **Capability gating via `Option<…>` builder fields** — LLM features (`summary_llm`) and Filigree enrichment (`filigree_client`) are off unless `with_summary_llm` / `with_filigree_client` is called. Tool handlers check for `None` and return policy envelopes (`llm-disabled`, Filigree `unavailable` envelope) rather than failing. +- **Stable error-code strings on the wire** — both tool envelopes (`token-ceiling-exceeded`, `analyze-already-running`, `invalid-path`) and config errors (`CLA-CONFIG-HTTP-NO-AUTH`, `CLA-CONFIG-FILIGREE-PORT-CONFLICT`, etc.) use prefix-namespaced identifiers callers can switch on. +- **Recording-provider determinism hook** — `LlmConfig.recording_fixture_path` (`config.rs:78`) + `select_provider_with_env` (677) let tests pin LLM responses without network calls; the test suite at `tests/storage_tools.rs` uses `RecordingProvider` heavily. + +### Concerns / Smells / Risks + +- **`lib.rs` is 4,703 lines in a single file.** This is the dominant smell. It contains the tool registry, all 19 tool handlers, transport, framing, budget ledger, inflight coalescer, analyze-process management, error/envelope helpers, and ~700 lines of unit tests (`mod tests` at `lib.rs:4309`+). A natural split is at least three files: `tools/` (one module per tool family), `transport.rs` (stdio framing), `dispatch.rs` (inferred-edge + summary LLM machinery). Risk: merge conflicts, slow IDE feedback, hard to review. +- **Discovery doc says "twenty tools"; registry contains 19.** Enumerated above. Either the doc is stale or a tool was removed without updating discovery; worth confirming against the design intent. Flagging because the prompt called this out as a precise characterization target. +- **Dispatch is purely sequential.** `serve_stdio_with_state_on_runtime` does `runtime.block_on(...)` inside the loop and a blocking read between frames. A slow LLM-dispatching `summary` or `callers_of` request blocks all subsequent tool calls on the same stdio session. The reader pool is internally concurrent but it doesn't help here. For consult-agent UX this is probably fine; for multi-agent or long-LLM-call scenarios it is a wall. +- **`reqwest::blocking` inside an async dispatcher.** `FiligreeHttpClient::associations_for` (`filigree.rs:98`) is synchronous and is called from `tool_issues_for` (`lib.rs:1154`) which is `async`. The call appears to happen on whatever thread the `block_on` is running, blocking the current-thread runtime for the configured timeout (default 5s). Should be `spawn_blocking`-wrapped or moved to `reqwest`'s async client. +- **Writer-channel coupling.** All writer commands flow through the `summary_llm.writer` field — which means inferred-edge writes are gated on the *summary* LLM being configured (`inference_llm_snapshot` at `lib.rs:2093` clones from `summary_llm`). Two LLM features share one writer handle and one config slot. Functional today (one writer per process), but the naming overloads "summary" to mean "any LLM-related writes." +- **Mutex poisoning swallowed everywhere.** Every `self.analyze_process.lock()`, `self.budget.lock()`, etc. uses `.unwrap_or_else(std::sync::PoisonError::into_inner)` (e.g. `lib.rs:556`, `2061`, `2284`). This silently masks the panic that caused the poisoning. Acceptable as a deliberate "keep serving" choice but worth a comment. +- **`config.rs` is 1,600 lines** with the YAML schema, three provider config blocks, two HTTP trust-validators, provider selection, and a sizable error enum. Splitting per provider would help. +- **Test coverage is heavily integration-tested in `tests/storage_tools.rs`** (2,233 LOC, ~35 `#[tokio::test]`s observed) but the in-`lib.rs` unit tests (`lib.rs:4309+`) are narrow — mostly transport framing. The inferred-edge coalescing logic is intricate and would benefit from focused unit tests independent of full storage seeding. +- **No timeout on `tool_analyze_start` child.** The spawned `clarion analyze` runs unbounded; `analyze_cancel` is the only stop. If the agent forgets to poll, the child can outlive the session. +- **Filigree client is held behind `Arc` but `FiligreeHttpClient` itself wraps `reqwest::blocking::Client` (already `Clone`).** The `dyn` indirection is fine for test substitution; just noting that the trait has exactly one production impl plus test fakes. + +### Confidence: High + +Read `lib.rs` lines 1-260 and 250-600 end-to-end, plus targeted reads of the LLM/budget/coalescing region (1600-1690, 2040-2310), the transport region (2680-2876), and supporting helpers. Read `filigree.rs` end-to-end (292 lines), `config.rs` lines 1-410 plus targeted reads. Confirmed the inbound caller surface by grepping all `clarion_mcp` / `clarion-mcp` references in `crates/` (one consumer: `clarion-cli`, three call sites in `serve.rs` / `http_read.rs` / `install.rs`). Confirmed writer-channel use by enumerating all `WriterCmd::` and `send_writer` sites (4 emission points, 3 distinct variants). Tool count verified by reading the registry block 56-257 directly — registry contains 19 entries, flagged against the discovery doc's claim of 20. + +## 5. clarion-scanner + +**Location:** `crates/clarion-scanner/` +**LOC:** 881 source (`lib.rs` 233 + `patterns.rs` 303 + `baseline.rs` 285 + `entropy.rs` 60) / 655 test (`tests/scanner.rs`) +**Crate type / role:** Library crate (`pub` API consumed by `clarion-cli`); pure CPU, no I/O except baseline YAML read. + +### Responsibility + +Owns Clarion's pre-ingest secret-detection pass: given an in-memory byte buffer, emit a deduplicated, byte-offset-anchored list of `Detection` records identifying secret-shaped substrings, plus a YAML-backed baseline mechanism that suppresses operator-acknowledged matches at the `(file, rule_type, hashed_secret, line_number)` granularity. The crate is deliberately scoped to detection + suppression — it does not walk the filesystem, decide which files to scan, emit findings, or report results; those concerns live in `clarion-cli/src/secret_scan/*`. Stores only positions, rule identifiers, and a detect-secrets-compatible SHA-1 digest of matched bytes — literal secret values do not leave the call (`lib.rs:1-5`). + +### Key components + +- `src/lib.rs:23-46` — `Detection` struct + `SecretCategory` enum: closed taxonomy with 9 categories (CloudCredential, VcsCredential, AiProviderCredential, PaymentsCredential, MessagingCredential, PrivateKey, JwtToken, HighEntropy, ContextualCredential). +- `src/lib.rs:49-99` — `HashedSecret([u8; 20])` newtype: SHA-1 digest with hex round-trip (`from_hex`, `Display`); decoupled from `sha1::Sha1` impl detail at the public surface. +- `src/lib.rs:102-200` — `DetectSecretsRule` enum: 14-variant closed vocabulary aligned to detect-secrets type names; `as_str()`/`rule_id()`/`FromStr` provide bidirectional mapping between human label (baseline YAML) and stable rule-id string (findings). +- `src/patterns.rs:25-71` — `Scanner` struct: pre-compiles a `RegexSet` (fast first-pass match) plus per-pattern `Regex` (for captures), holds entropy regexes and tunings; `Default`/`new()` build the ADR-013 v0.1 floor. +- `src/patterns.rs:79-162` — `scan_bytes()` + `scan_entropy()`: two-pass detection — named patterns first, entropy fallback over non-overlapping ranges; outputs sorted by `(byte_offset, rule_id)`. +- `src/patterns.rs:194-269` — `default_pattern_meta()`: the 12-entry rule floor (literal source of detection truth). +- `src/baseline.rs:11-44` — `Baseline`, `BaselineEntry`, `BaselineMatch`, `SuppressionResult` types: model the suppression file shape and the result envelope (allowed/suppressed/fired). +- `src/baseline.rs:104-144` — `Baseline::suppress()`: O(detections × entries_per_file) match using exact `(hashed_secret, line_number, rule_type)` triple as the suppression key. +- `src/baseline.rs:146-217` — `from_raw()` validation pipeline: version check (`"1.0"` only), path safety (`validate_baseline_path`), mandatory `justification`, hex-hash validity, closed rule-type vocabulary. + +### Public interface (outbound) + +Re-exported through `lib.rs:11-16`: + +- `Scanner` (`patterns.rs:25`) — owns compiled regexes; cheap to construct once and reuse. Method: `scan_bytes(&self, buf: &[u8]) -> Vec`. +- `Detection` (`lib.rs:23-32`) — one match: `rule_id`, `detect_secrets_type`, `category`, `byte_offset`, `line_number`, `matched_len`, `hashed_secret`. +- `DetectSecretsRule`, `SecretCategory` (`lib.rs:36-46`, `102-118`) — closed enums for downstream pattern-matching. +- `HashedSecret`, `HexDigestError` (`lib.rs:49-99`) — opaque hash newtype. +- `Baseline`, `BaselineEntry`, `BaselineEntryIssue`, `BaselineMatch`, `BaselineError`, `SuppressionResult` (`baseline.rs`) — operator-baseline model + error taxonomy with 7 variants (`UnsupportedVersion`, `MissingJustifications`, `InvalidPath`, `InvalidHash`, `UnsupportedRuleType`, `Parse`, `Io`). +- `load_baseline(&Path) -> Result` (`baseline.rs:71-77`) — accepts a missing file as `Baseline::empty()` (graceful absence). +- `EntropyTuning` (`entropy.rs:5-23`) — exposed but only `BASE64` and `HEX` consts are constructed internally; `min_len` + `min_entropy` fields. +- `PatternMeta` (`patterns.rs:10-15`) — exposed via `Scanner::pattern_meta()`, primarily for introspection. + +### Dependencies + +- **Inbound (who calls this):** Only `clarion-cli`: + - `clarion-cli/src/secret_scan.rs` imports `Detection`, `Scanner`, `SuppressionResult`. + - `clarion-cli/src/secret_scan/baseline.rs` imports `Baseline`, `BaselineError`. + - `clarion-cli/src/secret_scan/findings.rs` imports `Detection`, `SecretCategory`, `DetectSecretsRule`, `HashedSecret`. +- **Outbound (what this calls):** No internal Clarion crates. External: `regex` (bytes flavour), `sha1`, `serde` + `serde_norway` (YAML), `thiserror`. +- **External services:** None directly. Reads the baseline file via `std::fs::read_to_string` (`baseline.rs:72`); no other I/O. Pure-function `scan_bytes` over an `&[u8]`. + +### Internal architecture + +Three modules + `lib.rs` umbrella. `lib.rs` owns the shared value types (`Detection`, `HashedSecret`, `DetectSecretsRule`, `SecretCategory`) and two tiny helpers — `sha1_digest` (`lib.rs:208-214`) and `line_number_for_offset` (`lib.rs:216-224`, a byte-by-byte newline count over the prefix). Concurrency is **none** — there is no shared mutable state; `Scanner` is `Send + Sync` by construction (only immutable compiled regexes), so callers parallelize at the file-level outside the crate (and `clarion-cli/src/secret_scan.rs:scan_source_files_parallel` does exactly that). + +`patterns.rs` runs detection in two layers. Layer 1: `RegexSet` first-pass (`patterns.rs:80`) over the whole buffer — only patterns whose set-membership matched then have their full `Regex` run for capture extraction (`patterns.rs:83-87`). Layer 2: entropy fallback (`scan_entropy`, `patterns.rs:128-161`) runs the base64 candidate regex `[A-Za-z0-9+/]{20,}={0,2}` and hex candidate `\b[a-fA-F0-9]{40,}\b`, skipping candidates that **overlap any already-found named match** (`range_overlaps`, `patterns.rs:271-275`). The base64 fallback additionally requires non-base64 boundary bytes on each side (`base64_candidate_has_boundaries`, `patterns.rs:277-281`) — a hand-rolled `\b`-equivalent because `=` is not a word character. + +Entropy is parameterized by two `EntropyTuning` constants: +- `BASE64`: `min_len = 20`, `min_entropy = 4.5` (`entropy.rs:11-14`). +- `HEX`: `min_len = 40`, `min_entropy = 3.0` (`entropy.rs:15-18`). + +Entropy itself is Shannon entropy in bits/symbol over byte frequency (`entropy.rs:25-41`) using a `BTreeMap` count table, computed as `-Σ p_i log2(p_i)`. These thresholds are deliberately wide: tests `entropy_minimum_lengths_are_pinned` (`tests/scanner.rs:171-174`) and the lockfile-SHA fixture (`tests/scanner.rs:568-638`) document that the hex threshold *intentionally* fires on git SHAs and npm lockfile integrity hashes — the v0.1 stance is to suppress via baseline rather than tighten the rule and risk missing real secrets. + +The `KeywordDetector` rule (`patterns.rs:262-267`) is contextual: a Python/`.env`-shaped `name = "value"` assignment with name ∈ {`password`, `passwd`, `secret`, `token`, `api_key`, `secret_token`}, captured value ≥ 8 chars. To avoid false positives on commented-out examples, `scan_bytes` filters `ContextualCredential` matches whose line begins with `#` (`patterns.rs:91-94, 290-303`). The comment-handling is explicitly Python/shell-only — `//` and `/* */` comments are *not* skipped (tests at `tests/scanner.rs:162-167` lock this in deliberately). + +`baseline.rs` is a value-typed YAML parser + suppression engine. The on-disk shape is a `version: "1.0"` envelope with `results: {path: [entry, ...]}` (`baseline.rs:237-253`). Parse-time validation rejects: non-1.0 versions, absolute or `..`-bearing paths, missing/blank `justification` (collected exhaustively — `from_raw` walks all entries before returning the error, see `baseline.rs:153-172` and the test at `tests/scanner.rs:408-433`), unknown detector-type strings, and invalid hex hashes. Suppression is deliberately exact-match: same path, same rule, same hash, same line number — *and* `is_secret: false` (a `true` entry, or omitted field defaulting to `true` per `default_is_secret` at `baseline.rs:255-257`, does **not** suppress; tests at `tests/scanner.rs:300-385` lock this in). + +Error model is `thiserror`-derived (`baseline.rs:45-69`) with no panics in the parse path. The detection path has two `expect` calls in `Scanner::new` (`patterns.rs:51, 56-57, 66-69`) — all on compile-time constant regex literals, so they fire only on programmer error during edits, not on runtime input. + +### Patterns observed + +- **Two-phase regex matching** (`patterns.rs:80-87`) — `RegexSet` for cheap any-match prefilter, individual `Regex` only for matched indices. Trades memory for avoiding O(rules × text) capture work. +- **Closed enum at the interface, string only at the wire** — `DetectSecretsRule` (`lib.rs:102-118`) forces every consumer to handle the full vocabulary; conversions live at the YAML boundary (`baseline.rs:179-186`) so unknown types fail fast. +- **Capture-group routing via `Option`** (`patterns.rs:14, 96-103`) — `KeywordDetector` and `AwsSecretAccessKey` capture an inner group so the hash is over just the secret literal, not the `name="value"` framing. +- **Boundary-aware regex via helper functions** (`patterns.rs:277-303`) — Rust's `regex` crate's `\b` doesn't handle base64 padding `=` or `#`-comments, so the crate composes regex + post-filter. +- **Exhaustive error collection** for `MissingJustifications` (`baseline.rs:153-172`) — operator sees all missing entries at once, not one per re-run. +- **Path safety as a parse-time invariant** (`baseline.rs:219-235`) — absolute paths, `..`, drive prefixes all rejected at load; eliminates a class of suppression-escape attacks at the YAML boundary rather than at match time. +- **Cross-tool format compatibility by design** — the SHA-1 hash, the rule-name vocabulary, and the YAML schema all mirror Yelp's `detect-secrets` baseline format (cited explicitly in `lib.rs:1-5`), so operators familiar with that tool can reuse baselines. + +### Concerns / Smells / Risks + +- **`scan_bytes` is O(named_detections × entropy_candidates)** via `range_overlaps`'s linear scan (`patterns.rs:271-275`). For a file with hundreds of named matches and hundreds of entropy candidates this is quadratic. No interval tree, no sort-and-binary-search. For the target corpus (425k LOC Python, ADR-013 mentions) this is likely fine but worth flagging if scanner runtime ever becomes an issue. +- **`Baseline::suppress` is O(detections × entries_per_file)** with linear scan over the entries for that path (`baseline.rs:111-137`). Acceptable while baseline files are small (~tens of entries) but no index by `(rule, hash, line)`. +- **`usize_to_f64` truncates above 2³² bytes** (`entropy.rs:43-45`). For >4 GB files entropy will compute as if the buffer were 2³² bytes. Practically irrelevant — file-level scanning is bounded long before this — but the silent saturation deserves a comment. +- **Contextual-comment handling is Python/`.env`-only** (`patterns.rs:290-303`). A `// password = "..."` line in a JavaScript file *will* fire `ContextualCredential` (locked in by test at `tests/scanner.rs:163-167`). The crate's docstring acknowledges this; the cost is operator-baseline pollution in non-Python codebases until detector context is added. +- **`Scanner::new` panics on regex compile failure** (`patterns.rs:48, 51, 56, 66, 69`). These are compile-time literals so this is unreachable in shipping builds, but the public API has no fallible constructor — a future operator-provided pattern set would need a new constructor surface. +- **Entropy thresholds embed false-positive policy in code, not configuration** (`entropy.rs:11-18`). The test fixture explicitly documents that lockfile/git-SHA noise is *intentional* and the answer is operator baseline. This is a deliberate v0.1 trade-off but it pushes calibration burden to every operator until configurable thresholds or per-file-extension tuning lands. +- **`PatternMeta::capture_group` is private but the type is `pub`** (`patterns.rs:10-15`) — external consumers can read the struct via `Scanner::pattern_meta()` but can't construct one. This is fine as a closed API but the asymmetry is slightly awkward. +- **No fuzz or property tests** in `tests/scanner.rs` — only example-based assertions. Regex engines under pathological input (catastrophic backtracking) are a known foot-gun; the crate uses Rust's `regex` (which is linear-time by construction) so this is mitigated, but a quickcheck pass would be cheap insurance. + +### Confidence: High + +Read all 4 source files end-to-end (881 LOC), the full 655-line test file, the crate's `Cargo.toml`, and cross-checked inbound usage by greping the workspace for `clarion_scanner` imports (only `clarion-cli/src/secret_scan/*` and the scanner's own tests). Confirmed pre-ingest invocation timing at `clarion-cli/src/analyze.rs:237-243` (runs before plugin-host extraction, feeds `briefing_blocks` into the rest of the pipeline). Test file is unusually rich — it locks in not only positive/negative detection cases but also the *intentional* false positives (lockfile SHAs) with the documented operator-baseline workaround. No mystery code remains. + +## 6. clarion-plugin-fixture + +**Location:** `crates/clarion-plugin-fixture/` +**LOC:** 187 source (3 `lib.rs` + 184 `main.rs`); 0 in-crate tests — exercised externally by 931 LOC of test code in `clarion-core` and `clarion-cli`. +**Crate type / role:** Binary crate (`[[bin]] name = "clarion-plugin-fixture"`) plus a stub `lib.rs` whose only job is to let Cargo resolve the workspace member cleanly. Functionally a **test-only protocol-reference plugin**: a minimal, correct implementation of the L4 JSON-RPC wire contract used to drive the plugin host's subprocess code paths without needing the Python plugin on `PATH`. + +### Responsibility + +This crate owns the *smallest valid implementation* of the Clarion plugin protocol — just enough to (a) prove `clarion-core::plugin::host` can spawn a child, negotiate `initialize`/`initialized`/`analyze_file`/`shutdown`/`exit`, and round-trip framed JSON-RPC, and (b) deterministically misbehave on demand (RLIMIT_AS exhaustion via `CLARION_FIXTURE_EXCEED_RLIMIT_AS`) so the host's OOM-kill and crash-loop-breaker paths can be tested end-to-end. It is intentionally not a real analyzer: every `analyze_file` returns the same single hard-coded widget entity regardless of the input file's content. The crate's "public surface" is therefore (1) the binary itself, consumed by `cargo nextest` via `CARGO_BIN_EXE_clarion-plugin-fixture`, and (2) the fixture identity tuple (`plugin_id=fixture`, kind `widget`, qualname `demo.sample`, rule-ID prefix `CLA-FIXTURE-`) that tests assert against verbatim. + +### Key components + +- `src/main.rs:23-123` — the JSON-RPC dispatch loop: blocking `read_frame` → parse → match on `method` → either no-op (notification) or send a typed result envelope. Uses `clarion-core` framing and request/response types directly so the wire shape stays in lockstep with the host. +- `src/main.rs:67-76` — `initialize` handler: emits `InitializeResult { name, version, ontology_version: "0.1.0", capabilities: {} }`. +- `src/main.rs:77-115` — `analyze_file` handler: extracts `file_path` from params, returns one entity (`fixture:widget:demo.sample`, `kind=widget`, `qualified_name=demo.sample`, `source.file_path=`), no edges, default stats. +- `src/main.rs:48-60` — notification dispatch for `initialized` (becomes-ready, no reply) and `exit` (process-exits-0, no reply). +- `src/main.rs:78-83, 137-184` — the `CLARION_FIXTURE_EXCEED_RLIMIT_AS` escape hatch: on Unix, repeatedly `mmap_anonymous` doubling-size regions with `PROT_NONE` to blow past `RLIMIT_AS`, then `SIGKILL` self via `nix::sys::signal::kill`. The mappings are held but never dereferenced (documented `SAFETY` comment). +- `src/main.rs:125-135` — `send_result` helper: wraps a `Value` in `ResponseEnvelope { jsonrpc, id, payload: Result(...) }`, serialises, frames via `write_frame`, flushes. +- `src/lib.rs:1-3` — three-line doc-only stub explaining why the lib target exists. + +### Public interface (outbound) + +The plugin speaks the L4 JSON-RPC protocol on stdin/stdout. Methods implemented: + +- **`initialize`** (request) — returns `InitializeResult` with `ontology_version=0.1.0`, empty capabilities. `src/main.rs:68-76` +- **`initialized`** (notification) — accepted, no-op. `src/main.rs:51-53` +- **`analyze_file`** (request) — accepts `AnalyzeFileParams`, returns one canned entity. `src/main.rs:77-115` +- **`shutdown`** (request) — returns empty `ShutdownResult`. `src/main.rs:116-119` +- **`exit`** (notification) — `process::exit(0)`. `src/main.rs:54-56` + +Any unknown method, malformed frame, or unparseable JSON causes `process::exit(1)` (`src/main.rs:33-39, 46, 57, 64, 97, 120`). + +Companion manifest (consumed by host, not shipped by this crate): `crates/clarion-core/tests/fixtures/plugin.toml` declares `plugin_id="fixture"`, `language="fixture"`, `extensions=["mt"]`, `entity_kinds=["widget"]`, `edge_kinds=["uses"]`, `rule_id_prefix="CLA-FIXTURE-"`. + +### Dependencies + +- **Inbound (who consumes the binary):** + - `crates/clarion-core/tests/host_subprocess.rs` — direct subprocess test of `PluginHost::spawn`; locates the binary via `CARGO_BIN_EXE_clarion-plugin-fixture` (`host_subprocess.rs:30`) with a `cargo build` fallback (`:84-94`). Asserts entity id `fixture:widget:demo.sample` (`:162`). + - `crates/clarion-cli/tests/wp2_e2e.rs` — declares `clarion-plugin-fixture` as a `[dev-dependencies]` entry (`clarion-cli/Cargo.toml:43`) so nextest exports the env var, symlinks the binary into a synthetic plugin dir, and drives the full `clarion analyze` CLI through it. Tests: `wp2_e2e_smoke_fixture_plugin_round_trip` (line 135), `wp2_rlimit_as_oom_kill_is_reported_as_host_finding` (line 259, uses `CLARION_FIXTURE_EXCEED_RLIMIT_AS`), `wp2_crash_in_one_plugin_does_not_prevent_other_plugins_from_running` (line 323), `wp2_crash_loop_breaker_trips_and_skips_remaining_plugins` (line 471). +- **Outbound (what this calls):** + - `clarion-core::plugin::transport` — `read_frame` / `write_frame` / `Frame` for length-prefixed framing. + - `clarion-core::plugin::limits::ContentLengthCeiling::DEFAULT` (8 MiB per ADR-021 referenced in source comment, `main.rs:30-32`). + - `clarion-core::plugin` request/response DTOs: `AnalyzeFileParams`, `AnalyzeFileResult`, `AnalyzeFileStats`, `InitializeResult`, `ShutdownResult`, `JsonRpcVersion`, `ResponseEnvelope`, `ResponsePayload`. + - `serde_json` for ad-hoc `Value` parsing of the inbound request envelope. + - `nix` (Unix only, `mman` + `signal` features) — only for the deliberate-OOM probe. +- **External services:** None at runtime. Communicates only over inherited stdin/stdout with its parent (the plugin host). + +### Internal architecture + +The binary is a single synchronous loop in `main()` (`main.rs:23-123`); there are no modules, threads, async runtime, or background tasks. State machine is implicit and minimal: the loop accepts notifications and requests in any order after the host has spoken — there is no explicit "before-initialize" guard, which is acceptable because the host always sends `initialize` first and the fixture's responses are stateless. The shape is `loop { read_frame → parse Value → branch on (has_id, method) → dispatch }`. + +Error model is intentionally brutal: any deviation from the happy path (truncated frame, non-UTF8, missing method, unknown method, unparseable params) calls `std::process::exit(1)` with no reply. This is *desired* behaviour — `clarion-core`'s host has to handle a plugin that hangs up mid-stream, and crashing the fixture is the cheapest way to drive that path. The crash-loop-breaker test (`wp2_e2e.rs:471`) depends on this. + +The `exceed_rlimit_as` path (`main.rs:137-184`) is the only piece of nontrivial logic. It pre-reserves 1024 mapping handles before the memory pressure starts (so the `Vec::push` itself does not allocate after the kernel starts refusing maps), then loops `mmap_anonymous(PROT_NONE, MAP_PRIVATE)` doubling the request size each iteration starting at 128 MiB. PROT_NONE means no physical pages are committed — only address space is consumed, which is exactly what `RLIMIT_AS` constrains. When the next request would not grow (`saturating_mul(2)` saturated) or `mmap` returns `Err`, the fixture `SIGKILL`s itself so the parent observes a signal-death, not a clean exit. This is the load-bearing detail that distinguishes OOM-kill from a controlled shutdown in the host's diagnosis. + +`lib.rs` exists solely so the crate works as a workspace member with a binary target (3 lines, doc comment only). + +### Patterns observed + +- **Stub lib + real bin** (`lib.rs:1-3` + `Cargo.toml:12-14`) — workspace-member compatibility trick. +- **Untyped envelope parse, typed payload re-parse** (`main.rs:37-44, 93-98`) — read the whole frame as `serde_json::Value` to inspect `id`/`method` before committing to a typed struct, so unknown/malformed messages can be rejected without spurious deserialisation errors. +- **Crash-on-anomaly as a feature** (`main.rs:33-46, 57, 97, 120`) — every error path is `process::exit(1)`. Tests want this behaviour. +- **Hard-coded fixture identity** (`main.rs:101-108`) — `"fixture:widget:demo.sample"` is the ground-truth string tests assert on; do not change without updating both call sites listed under Inbound. +- **Environment-flag-driven fault injection** (`main.rs:78-83`) — `CLARION_FIXTURE_EXCEED_RLIMIT_AS` toggles the OOM path; default behaviour is benign. +- **PROT_NONE address-space probe** (`main.rs:137-178`) — exhausts virtual address space cheaply (no page commits) to trip a specific kernel limit deterministically. +- **Pre-reserved Vec to avoid allocation under memory pressure** (`main.rs:142-145`). + +### Concerns / Smells / Risks + +- **`process::exit(1)` everywhere with no diagnostic output.** Reasonable for a test fixture (the host is the system under test), but if a future test asserts on stderr or a specific exit code other than 1, every error branch will need disambiguating. No `eprintln!` anywhere. `main.rs:33-46, 57, 97, 120`. +- **No "initialize must come first" sequencing check.** A misbehaving host could call `analyze_file` before `initialize` and the fixture would happily respond. This is fine today because the host always sequences correctly, but the fixture is not a conformance checker — do not use it as one. +- **`unwrap()` on `serde_json::to_value(InitializeResult)`** (`main.rs:75`) and other result serialisations — defensible because the types are `Serialize`-derived, but pedantically these are crash points. +- **Unix-gated OOM path with a `cfg(not(unix))` arm that just `exit(1)`s** (`main.rs:78-83`) — Windows CI would silently lose coverage of the RLIMIT_AS branch. Acceptable since OOM-kill semantics are Unix-specific. +- **The manifest lives in another crate's test tree** (`clarion-core/tests/fixtures/plugin.toml`), not in this crate. Discoverable only by `grep`. Two callers (`host_subprocess.rs` and `wp2_e2e.rs`) construct their own variants of it. A `tests/fixtures/plugin.toml` here, re-used by both, would be cleaner; not urgent. +- **Hard-coded `version = "0.1.0"` and `ontology_version = "0.1.0"`** in the binary (`main.rs:71-72`) do not pick up the workspace version — if version-handshake logic ever depends on these, they will drift from `Cargo.toml`. Currently the host does not validate them. +- **No in-crate tests.** Justified: the fixture *is* the test apparatus for two upstream test files. Counting test coverage of this crate in isolation is meaningless. + +### Confidence: High + +Read all 187 source lines end-to-end, the companion manifest in `clarion-core/tests/fixtures/plugin.toml`, and grepped both consumer tests (`host_subprocess.rs`, `wp2_e2e.rs`) for every reference to `clarion-plugin-fixture`, `fixture:widget:demo`, and `CLARION_FIXTURE_EXCEED_RLIMIT_AS` to confirm the protocol surface and the four named test cases. Inbound dep edges verified via `Cargo.toml:43` of `clarion-cli`. Outbound dep edges verified by walking every `use clarion_core::` import in `main.rs`. + +## 7. Python language plugin (`clarion-plugin-python`) + +**Location:** `plugins/python/` +**LOC:** 3028 source / 3440 test (`extractor.py` 932, `pyright_session.py` 1406, `server.py` 296, the rest ≤ 75 each) +**Crate type / role:** Standalone PEP 517 package (hatchling) installing the `clarion-plugin-python` console script; hosts an L4 JSON-RPC plugin process driven by the Rust core over stdio. Ships its `plugin.toml` to `share/clarion/plugins/python/plugin.toml` via `pyproject.toml:38-44`. + +### Responsibility +Owns Python-source ingestion for Clarion: parses `*.py` to an AST, emits `module` / `class` / `function` entities with stable 3-segment EntityIds, anchors structural `contains` and `imports` edges directly, and delegates type-resolved `calls` / `references` edges to a managed pyright-langserver subprocess. It is the *only* component in the workspace that holds Python `ast` semantics and the only Loom-suite-facing surface that probes Wardline (`wardline_probe.py:35-56`) for the L8 integration handshake. Public surface to the core is exactly the five JSON-RPC methods declared in `server.py:237-240` plus the manifest at `plugin.toml`. + +### Key components +- `src/clarion_plugin_python/__main__.py:14-15` — entry point bound to `[project.scripts] clarion-plugin-python = ...:main`; installs stdio guard then runs `serve()`. +- `src/clarion_plugin_python/server.py:73-128, 143-232, 243-296` — Content-Length framing reader/writer, dispatch table, handlers for `initialize` / `analyze_file` / `shutdown`, and the per-25-files pyright restart policy (`MAX_FILES_PER_PYRIGHT_SESSION = 25`, defined at `server.py:49`, used at `server.py:215-219`). +- `src/clarion_plugin_python/extractor.py:256-371` — top-level `extract` / `extract_with_stats`; the `_walk` recursion at lines 763-842 emits entities + `contains` edges; `_ImportEdgeCollector` at 396-468 emits `imports` edges; `_ReferenceSiteCollector` at 532-659 produces reference sites for the resolver. +- `src/clarion_plugin_python/entity_id.py:23-75` — L2 3-segment EntityId assembler; mirrors the Rust validator (`[a-z][a-z0-9_]*` grammar, no `:`, no empty segment). +- `src/clarion_plugin_python/qualname.py:34-48` — reconstructs CPython `__qualname__` from the AST parent chain (class-nested vs `parent..child`). +- `src/clarion_plugin_python/pyright_session.py:131-890` — `PyrightSession`: subprocess lifecycle, LSP framing, function/entity index builder, `resolve_calls` (callHierarchy) and `resolve_references` (definition/typeDefinition). 1406 lines; the load-bearing class plus helpers `_build_function_index` (893-925) and `_collect_entities` (928-1006). +- `src/clarion_plugin_python/stdout_guard.py:25-62` — replaces `sys.stdout` with a write-refusing shim so libraries cannot corrupt the host's JSON-RPC frame parser. +- `src/clarion_plugin_python/wardline_probe.py:35-56` — imports `wardline.core.registry` + reads `wardline.__version__`; returns one of `absent` / `enabled` / `version_out_of_range`. + +### Public interface (outbound) +- **Binary:** `clarion-plugin-python` (bare basename per `plugin.toml:8`, enforced by ADR-021 layer-1 path-component refusal in the core's discovery). +- **JSON-RPC methods (server.py:237-272):** + - `initialize(params{project_root}) → {name, version, ontology_version="0.6.0", capabilities{wardline}}` (`server.py:143-156`). + - `initialized` — notification, flips `state.initialized = True` (line 250). + - `analyze_file(params{file_path}) → {entities, edges, stats}` (`server.py:179-232`); gated by `_ERR_NOT_INITIALIZED` if called before `initialized` (line 263). + - `shutdown() → {}` — closes the pyright child (line 255-260). + - `exit` — notification; loop returns `0` if shutdown was requested, `1` otherwise (`server.py:286-287`). +- **Wire shapes** declared as `TypedDict`s in `extractor.py:88-142` (`RawEntity`, `RawEdge`, `SourceRange`) and `call_resolver.py:11-22`, `reference_resolver.py:28-39` — these match the host's `RawEntity` / `RawEdge` deserialise contracts cited inline at `extractor.py:9-22`. +- **Manifest surface:** `plugin.toml` advertises `plugin_id="python"`, `entity_kinds=["function","class","module"]`, `edge_kinds=["contains","calls","references","imports"]`, `ontology_version=0.6.0`, `rule_id_prefix="CLA-PY-"`, `wardline_aware=true`, and pyright pin `1.1.409`. + +### Dependencies +- **Inbound (who calls this):** the Rust plugin host. Confirmed via `grep` hits in `crates/clarion-core/src/plugin/{discovery.rs:60, manifest.rs:150-748, protocol.rs:656, host.rs:519}` and `crates/clarion-mcp/src/lib.rs:526`. The core spawns the binary, speaks Content-Length-framed JSON-RPC, and consumes `entities` / `edges` / `stats`. +- **Outbound (what this calls):** none of the other Clarion crates directly — the plugin is a pure subprocess. It does cross-product import `wardline.core.registry` + `wardline` purely as a capability probe (`wardline_probe.py:38-39`), fail-soft. +- **External services:** + - `pyright-langserver --stdio` subprocess (`pyright_session.py:637-644`), pin `1.1.409` (`pyproject.toml:20`, `plugin.toml:29`). Resolved via venv-sibling first, then `shutil.which` (`pyright_session.py:699-706`). Bounded by `init_timeout=30s`, `call_timeout=5s`, per-file budget `3s`, `MAX_PYRIGHT_RESTARTS_PER_RUN=3`, `MAX_REFERENCE_SITES_PER_FILE=2000`. + - Python `packaging.version.Version` for Wardline range parsing (`wardline_probe.py:30`). +- **Cross-language fixture:** consumes `fixtures/entity_id.json` at `tests/test_entity_id.py:25` — same rows the Rust assembler validates against; this is the byte-for-byte parity proof for L2 EntityId construction. + +### Internal architecture +**Process / I/O model.** The plugin is a single-threaded JSON-RPC dispatch loop (`server.py:275-287`) reading Content-Length-framed frames from `stdin`, writing replies to a captured `stdout` byte stream. Before any framing happens, `install_stdio()` (`stdout_guard.py:57-62`) captures the real `sys.stdin.buffer` / `sys.stdout.buffer` and replaces `sys.stdout` with `_GuardedTextStdout`, which raises `StdoutGuardError` on any write — this is the plugin-side resolution of WP2's UQ-WP2-08 framing-corruption risk. Frame size is capped at 8 MiB (`MAX_CONTENT_LENGTH`, `server.py:48`) on both inbound and outbound to match the host's ceiling. + +**Extraction pipeline (per `analyze_file`).** `handle_analyze_file` (`server.py:179-232`) reads the file text, relativises the path to `project_root` for the qualified-name prefix only (`_resolve_module_path`, lines 158-176), and calls `extract_with_stats` with the project-relative prefix and the live `PyrightSession` as both `call_resolver` and `reference_resolver`. The extractor (`extractor.py:274-371`) always emits exactly one `module` entity (whole-file cover, `end_col=0` sentinel — see the module-docstring caveat at lines 49-53), then `_walk` (lines 763-842) recursively emits one `function` per `FunctionDef`/`AsyncFunctionDef` and one `class` per `ClassDef`. Each non-module entity carries `parent_id` and contributes one `contains` edge (ADR-026 dual encoding, lines 845-851). PEP-484 `@overload` stubs are dropped pre-emit (`_has_overload_decorator`, lines 741-760); any surviving same-id collision is dropped first-wins with stderr + `duplicate_entities_dropped_total` bump (lines 803-810). `imports` edges are collected separately by an `ast.NodeVisitor` (lines 396-468); relative imports compute the base package via `_relative_import_base_parts` (lines 499-510) with package-vs-module awareness keyed on `is_package_module`. Reference sites are gathered by a third visitor (lines 532-659) that maintains a `bound_stack` of locally-bound names per scope and suppresses lambdas + `Call` callees (the latter is the `calls` edge's territory). + +**Pyright integration.** `PyrightSession` (`pyright_session.py:131-890`) spawns `pyright-langserver --stdio` as a `subprocess.Popen` (line 637), drives it as an LSP client over Content-Length framing (`_write_message`/`_read_message`, lines 800-834), with a daemon stderr-drain thread keeping a 64 KiB tail (`_drain_stderr`, 723-730). Initialize sends `processId`, `rootUri`, and `workspaceFolders` (lines 682-697); when pyright requests `workspace/configuration`, the session responds with `diagnosticMode=openFilesOnly`, `indexing=false`, `useLibraryCodeForTypes=false`, and an explicit `exclude` list for `.git/.venv/__pycache__/...` (`_configuration_for_section`, lines 774-788). Call resolution opens the file (`textDocument/didOpen`), runs `textDocument/prepareCallHierarchy` per function, then `callHierarchy/outgoingCalls`, and translates the returned URIs/selectionRanges back into Clarion entity IDs via a parallel AST-built `_FunctionIndex` cache keyed on `Path.resolve()` (`_function_index_for_path`, lines 861-870). Edges are grouped by `fromRanges` and emitted as `resolved` or `ambiguous` (`pyright_session.py:394-411`). The session augments pyright's static graph with two heuristic dispatch detectors: `_ambiguous_dict_dispatches` (callable-dict lookup patterns) and `_dunder_call_dispatches` (instances whose class has `__call__`) — both produce additional ambiguous-candidate edges. Reference resolution (lines 427-511) issues `textDocument/definition` per site with an `annotation`-kind fallback to `textDocument/typeDefinition`, caches per `(from_id, kind, lexeme)` triple, and accumulates edges by `(from_id, to_id)` pair. + +**Resilience model.** Three tiers: (1) **timeouts** — per-request via `_budgeted_timeout`, per-file via `_deadline_for_file` (lines 531-550), with init/call/file knobs all overridable per `__init__`. (2) **restart cap** — `_record_restart_or_poison` (lines 599-615) restarts pyright up to `MAX_PYRIGHT_RESTARTS_PER_RUN=3`, then permanently disables call resolution by setting `_disabled=True` (emits `CLA-PY-PYRIGHT-POISON-FRAME`). (3) **server-side restart-by-attrition** — `server.py:215-219` closes and re-creates the `PyrightSession` every 25 analyzed files. All failure modes degrade gracefully: missing pyright (`absent` install), failed handshake, timeouts, and the reference-site cap all return zero edges plus a `Finding` with subcode `CLA-PY-PYRIGHT-*` (`pyright_session.py:34-41`). + +**Error model.** JSON-RPC errors use codes `-32600` / `-32601` / `-32603` / `-32002` (`server.py:52-55`); any handler exception becomes a `_ERR_INTERNAL` response (line 270-271). Framing-level violations (bad headers, oversize Content-Length, JSON decode failure) raise `ProtocolError`, which `main()` translates to exit code `1` (`server.py:295-296`). `ast.SyntaxError` during extraction emits one degraded `module` entity with `parse_status="syntax_error"` + a stderr line (lines 325-335) — the run continues. + +### Patterns observed +- **Single-process dispatch loop with explicit lifecycle states** — `ServerState.initialized` / `shutdown_requested` gating; `analyze_file` rejected before `initialized` (`server.py:263-264`). +- **Subprocess-as-service with restart cap + capability disable** — `PyrightSession._disabled` poisons the resolver permanently after N failures (lines 600-609); analysis continues without type-resolved edges. +- **Dual encoding of structural relationships** — every non-module entity carries `parent_id` *and* a corresponding `contains` edge (extractor.py:813, 845-851), per ADR-026. +- **Parallel AST builds for entity emission vs LSP cross-walking** — `extractor.py` and `pyright_session._build_function_index` (line 893) each parse the file with `ast.parse` for distinct purposes; the index is cached per `Path.resolve()` in the session (line 861). +- **Stdout discipline via type substitution** — `_GuardedTextStdout` raises on every write rather than silently swallowing (`stdout_guard.py:36-41`). +- **TypedDict wire-shape contracts mirroring Rust serde structs** — `RawEntity`, `RawEdge`, `CallsRawEdge`, `ReferencesRawEdge`, `Finding` are statically checked under `mypy --strict` and called out in source comments as host-matching (`extractor.py:9-22`, `call_resolver.py:14-22`). +- **Cross-language fixture-driven parity** — `fixtures/entity_id.json` is consumed by both this plugin's tests and the Rust assembler tests; the validator code is intentionally duplicated rather than shared (`entity_id.py:56-75`). + +### Concerns / Smells / Risks +- **`pyright_session.py` is 1406 lines in a single module** containing the `PyrightSession` class (≈760 lines) plus ~600 lines of free-function helpers (call-site visitors, dispatch heuristics, LSP helpers, byte-offset arithmetic). The class itself is cohesive but the helper sprawl makes the file hard to navigate; the AST-index builder (`_build_function_index`, `_collect_entities`) duplicates traversal work the extractor already does, suggesting an opportunity to fold the index into `extractor.py` and let the session consume it. +- **`extractor.py` carries three separate `ast.NodeVisitor` walks** (`_walk` recursion, `_ImportEdgeCollector`, `_ReferenceSiteCollector`) plus a fourth in `pyright_session._collect_entities`. Each walks the same tree with slightly different bookkeeping. No measurable performance bug observed in tests, but future maintenance touching scope semantics (e.g., `match`/`case` introducing bindings, `walrus` operator) has four call sites to keep in sync. +- **Dispatch heuristics are pattern-based and fragile** — `_has_overload_decorator` admits only bare `overload` / `typing.overload` / `typing_extensions.overload` (lines 753-759); aliased imports defeat it and rely on the deduplication safety net. Same shape for `_ambiguous_dict_dispatches` and `_dunder_call_dispatches` in `pyright_session.py:1158-1303` — they will silently miss anything pyright also misses unless the source matches a known pattern. +- **Per-file pyright restart at 25 files is a magic constant** (`server.py:49`), separate from the 3-restart-cap inside the session. The interaction between server-driven recycling and session-driven failure restarts is not centrally documented in code; if both fire on the same run the failure semantics depend on ordering. +- **`StdoutGuardError` is raised but never caught by `server.serve`** — a stray write in any handler will bubble through `dispatch`'s broad `except Exception` (line 270) and become an `_ERR_INTERNAL` response with the guard message in plain text. That is probably correct behaviour, but the design implication (guard violations become normal-looking JSON-RPC errors instead of hard exits) is not stated. +- **`shutdown` handler kills the pyright child synchronously inside dispatch** (`server.py:257-259`); a misbehaving pyright that ignores `shutdown` would extend the host's view of plugin termination by up to ~4s (two `process.wait(timeout=2)` calls in `PyrightSession.close`, lines 191-194). The host's circuit breaker is the backstop, but the plugin does not log when it falls back to `kill()`. +- **Test coverage is heavy** (test LOC > source LOC; nine pytest files including pyright-marked integration tests), but `pyproject.toml:94` marks the pyright tests as opt-in via the `pyright` marker — easy to skip in environments without the langserver and not obvious from the test names whether a green run actually exercised the LSP path. + +### Confidence: High +Read all 11 source files end-to-end (extractor.py, server.py fully; pyright_session.py covering `PyrightSession` and the main helpers ~600 of 1406 lines closely, the dispatch-heuristic visitors at signature level), the plugin manifest, `pyproject.toml`, and the round-trip + fixture-parity tests. Cross-checked inbound callers via `grep` against `crates/clarion-{core,mcp}/` (5 distinct call sites). One specific number I did not verify by reading source: the `MAX_FILES_PER_PYRIGHT_SESSION` constant is `25` per `server.py:49`, cited above. The relevant integration boundary (Rust `RawEntity` / `RawEdge` deserialise contract) was confirmed only via the docstring references in `extractor.py:9-22` and `call_resolver.py` typeddicts — not by reading the Rust side, which is out of this subsystem's scope. diff --git a/docs/arch-analysis-2026-05-22-1924/03-diagrams.md b/docs/arch-analysis-2026-05-22-1924/03-diagrams.md new file mode 100644 index 00000000..d02776fd --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/03-diagrams.md @@ -0,0 +1,386 @@ +# 03 — Architecture Diagrams (Clarion) + +> Derived exclusively from `01-discovery-findings.md`, `02-subsystem-catalog.md`, +> and small confirmation reads of `crates/clarion-storage/src/writer.rs`, +> `crates/clarion-mcp/src/lib.rs`, and `crates/clarion-core/src/plugin/host.rs`. +> No design docs, ADRs, or sprint narratives were read. Every node in every +> diagram traces to a file in the workspace; residual uncertainties are listed +> at the bottom. + +--- + +## 1. C4 System Context — Clarion in its environment + +```mermaid +flowchart TD + dev["Developer
(CLI user)"] + agent["Consult-mode LLM agent
(MCP client)"] + peer["Federation peer
(HTTP API consumer)"] + + subgraph clarion_sys["Clarion system
(single 'clarion' binary + plugins)"] + clarion["clarion
(install / analyze / serve)"] + end + + sqlite[("SQLite
.clarion/clarion.db
(embedded, WAL)")] + plugins["Language plugin
subprocesses
(e.g. clarion-plugin-python)"] + pyright["pyright-langserver
(transitive, owned by Python plugin)"] + openrouter["OpenRouter
(LLM HTTPS API)"] + cli_llm["LLM CLI binaries
(claude / codex)"] + filigree["Filigree
(sibling product, HTTP)"] + wardline["Wardline
(import-probe only,
Python plugin side)"] + + dev -- "argv (install/analyze/serve)" --> clarion + agent -- "stdio MCP
(Content-Length JSON-RPC,
protocol 2025-11-25)" --> clarion + peer -- "HTTPS read API
(/api/v1/files*,
bearer or HMAC)" --> clarion + + clarion -- "embedded rusqlite
(WAL, FK on)" --> sqlite + clarion -- "JSON-RPC over stdio pipes
(Content-Length framing,
5 methods)" --> plugins + plugins -- "LSP JSON-RPC
over stdio pipes" --> pyright + clarion -- "HTTPS
(reqwest, chat completions)" --> openrouter + clarion -- "subprocess
(stdin-piped prompt)" --> cli_llm + clarion -- "HTTPS GET
(reqwest blocking,
entity-association lookup)" --> filigree + plugins -. "importlib probe only,
fail-soft, no network" .- wardline +``` + +Derived from `crates/clarion-cli/src/cli.rs` (three subcommands), `crates/clarion-mcp/src/lib.rs:40` (`MCP_PROTOCOL_VERSION="2025-11-25"`), `crates/clarion-cli/src/http_read.rs:347-369` (Axum router), `crates/clarion-core/src/plugin/transport.rs` (Content-Length framing), `crates/clarion-storage/src/pragma.rs:17-30` (WAL/foreign_keys), `crates/clarion-mcp/src/filigree.rs` (Filigree client), `crates/clarion-core/src/llm_provider.rs` (OpenRouter + CLI providers), `plugins/python/src/clarion_plugin_python/wardline_probe.py` (import-only probe), and `plugins/python/src/clarion_plugin_python/pyright_session.py:637-644` (pyright subprocess). Wardline is dashed because the edge is a fail-soft `importlib.import_module` probe with no production read of the result on the Rust side. + +--- + +## 2. C4 Container — Processes and stores inside `clarion serve` and `clarion analyze` + +```mermaid +flowchart TB + subgraph clarion_proc["clarion (single binary; one OS process)"] + cli_install["install subcommand
(synchronous;
creates .clarion/)"] + cli_analyze["analyze subcommand
(multi-thread tokio;
orchestrates pipeline)"] + cli_serve["serve subcommand
(supervisor;
two worker threads)"] + + subgraph serve_runtime["serve runtime (one process, two threads, one ReaderPool)"] + mcp_thread["MCP stdio thread
(current-thread tokio;
clarion-mcp::ServerState)"] + http_thread["HTTP read thread
(multi-thread tokio;
Axum router)"] + reader_pool[("ReaderPool
(deadpool-sqlite,
16 conns;
Arc<()> identity tag)")] + writer_actor["Writer actor
(spawn_blocking task;
1 write Connection)"] + end + end + + sqlite_db[("SQLite file
.clarion/clarion.db
(WAL)")] + + subgraph plugin_procs["Plugin subprocesses (one per language)"] + py_plugin["clarion-plugin-python
(server.py dispatch loop)"] + py_pyright["pyright-langserver
(child of python plugin)"] + fixture["clarion-plugin-fixture
(test only)"] + end + + filigree_ext["Filigree
(external HTTP)"] + + cli_install -- "rusqlite::Connection::open;
schema::apply_migrations" --> sqlite_db + cli_analyze -- "spawn writer-actor;
spawn plugin subprocesses" --> serve_runtime + cli_serve -- "spawns both threads;
holds shared ReaderPool" --> serve_runtime + + mcp_thread -- "with_reader (~30 sites)" --> reader_pool + http_thread -- "with_reader" --> reader_pool + mcp_thread -- "mpsc<WriterCmd>
(3 variants used:
InsertInferredEdges,
UpsertSummaryCache,
TouchSummaryCache)" --> writer_actor + + reader_pool -- "read txns" --> sqlite_db + writer_actor -- "write txn,
50-write batch cadence" --> sqlite_db + + cli_analyze -- "JSON-RPC over pipes
(stdin/stdout per child;
stderr drain thread)" --> py_plugin + cli_analyze -. "test only" .- fixture + py_plugin -- "LSP JSON-RPC
over stdio pipes" --> py_pyright + + mcp_thread -- "blocking reqwest GET
(entity-associations?entity_id=)" --> filigree_ext +``` + +Derived from `crates/clarion-cli/src/serve.rs:20-227` (two-thread supervisor + ReaderPool identity tag), `crates/clarion-cli/src/http_read.rs:262-311` (`run_http_read_server`), `crates/clarion-storage/src/writer.rs:79-98` (mpsc channel + `spawn_blocking` actor), `crates/clarion-storage/src/reader.rs:26-119` (deadpool reader pool with `Arc<()>` tag), `crates/clarion-mcp/src/lib.rs:316-376` (`ServerState`), `crates/clarion-storage/src/pragma.rs` (WAL discipline). Plugin subprocesses are spawned from `crates/clarion-core/src/plugin/host.rs::spawn` driven by `crates/clarion-cli/src/analyze.rs::run_plugin_blocking`. + +--- + +## 3. C4 Component — Plugin host wired to `clarion analyze` + +```mermaid +flowchart LR + subgraph cli["clarion-cli (analyze.rs)"] + run["run_with_options
(orchestrator)"] + run_plugin["run_plugin_blocking
(spawn_blocking)"] + crash_brk["CrashLoopBreaker
(>3 crashes / 60s,
OWNED HERE)"] + end + + subgraph core["clarion-core::plugin"] + discovery["discovery.rs::discover
($PATH scan +
install-prefix fallback)"] + manifest["manifest.rs::parse_manifest
(ADR-022 grammar,
reserved-kind gate)"] + host_spawn["host.rs::PluginHost::spawn
(Command::new +
BufReader/BufWriter)"] + host_pipeline["host.rs analyze_file pipeline
(0 field-size →
1 ontology →
2 identity →
3 jail →
4 entity cap)"] + transport["transport.rs
(Content-Length framing,
8 MiB ceiling)"] + jail["jail.rs
(canonicalize +
starts_with check)"] + limits["limits.rs
(RLIMIT_AS / NOFILE / NPROC
via pre_exec setrlimit)"] + path_brk["limits.rs::PathEscapeBreaker
(>10 escapes / 60s,
OWNED BY HOST)"] + stderr["stderr drain thread
(64 KiB ring buffer,
one per host)"] + end + + subgraph child["plugin subprocess"] + child_proc["child process
(stdin/stdout/stderr)"] + end + + subgraph storage["clarion-storage"] + writer_chan["mpsc<WriterCmd>
(cap 256)"] + end + + run --> discovery + discovery --> manifest + run --> run_plugin + run --> crash_brk + run_plugin -- "construct" --> host_spawn + host_spawn -- "apply at fork" --> limits + host_spawn -- "spawn child" --> child_proc + host_spawn -- "spawn drain" --> stderr + stderr -. "ChildStderr" .- child_proc + + run_plugin -- "analyze_file loop" --> host_pipeline + host_pipeline --> transport + host_pipeline --> jail + host_pipeline --> path_brk + transport -- "Content-Length frames" --> child_proc + + host_pipeline -- "BeginRun → entities →
unresolved-call-sites → edges →
ClusterMembers (subsystems) →
CommitRun" --> writer_chan + + crash_brk -- "tick on spawn/handshake/
analyze_file failure" --> run_plugin +``` + +Derived from `crates/clarion-core/src/plugin/host.rs:384-1182` (host generic over `BufRead`/`Write`, four-stage pipeline at 866-975, spawn at 509-644, stderr drain at 614-620), `crates/clarion-core/src/plugin/{discovery,manifest,transport,jail,limits,breaker}.rs`, and `crates/clarion-cli/src/analyze.rs:240-275, 325-470, 675-906` (run loop, per-plugin blocking task, phase-3 clustering). The two breakers are drawn distinctly: `CrashLoopBreaker` lives in `analyze.rs`, `PathEscapeBreaker` lives inside the host — same shape, asymmetric ownership noted in catalog §1 patterns. + +--- + +## 4. C4 Component — Storage layer + +```mermaid +flowchart TB + subgraph callers["Callers (clarion-cli, clarion-mcp)"] + cli_writes["clarion-cli/analyze.rs
(BeginRun, entities, edges,
findings, FlushRunBatch,
CommitRun/FailRun)"] + mcp_writes["clarion-mcp/lib.rs
(3 variants only)"] + readers_cli["clarion-cli reads
(http_read.rs handlers)"] + readers_mcp["clarion-mcp reads
(~30 with_reader sites)"] + end + + subgraph storage["clarion-storage"] + subgraph writer_side["Writer side (single task)"] + sender["Writer (mpsc::Sender<WriterCmd>)
DEFAULT_CHANNEL_CAPACITY=256"] + actor["run_actor
(spawn_blocking;
blocking_recv loop)"] + state["ActorState
(batch_size=50,
writes_in_batch,
in_tx, current_run)"] + contracts["wire-contract enforcers
(enforce_entity_kind_contract,
enforce_edge_contract,
parent_contains_mismatch)"] + write_conn[("write Connection
(PRAGMA: WAL,
synchronous=NORMAL,
busy_timeout=5000,
foreign_keys=ON)")] + end + + subgraph reader_side["Reader side"] + pool["ReaderPool
(deadpool-sqlite,
Arc<()> identity tag)"] + with_reader["with_reader
(async helper)"] + query["query.rs
(~29 helpers:
entity_by_id,
find_entities,
call_edges_from,
subsystem_members,
resolve_file_catalog_entry, ...)"] + cache["cache.rs
(summary_cache,
inferred_edge_cache)"] + unresolved["unresolved.rs
(replace-by-caller)"] + end + + schema_mod["schema.rs
(apply_migrations;
schema_migrations table)"] + migrations[("migrations/
0001_initial_schema.sql
(293 LOC; include_str!)")] + pragma["pragma.rs
(apply_write_pragmas /
apply_read_pragmas)"] + + cmds["commands.rs::WriterCmd
(11 variants):
BeginRun, InsertEntity,
InsertEdge, InsertFinding,
FlushRunBatch,
InsertInferredEdges,
UpsertSummaryCache,
TouchSummaryCache,
ReplaceUnresolvedCallSitesForCaller,
CommitRun, FailRun"] + end + + sqlite[("SQLite file
.clarion/clarion.db")] + + cli_writes -- "send + oneshot ack" --> sender + mcp_writes -- "send + oneshot ack
(via cache/unresolved helpers)" --> sender + sender -- "WriterCmd" --> actor + actor -- "owns" --> state + actor --> contracts + contracts -- "validate" --> cmds + state -- "single conn" --> write_conn + write_conn -- "WAL" --> sqlite + + readers_cli --> with_reader + readers_mcp --> with_reader + with_reader --> pool + with_reader --> query + with_reader --> cache + with_reader --> unresolved + pool -- "read conn N" --> sqlite + + schema_mod -- "boot" --> write_conn + schema_mod --> migrations + pragma -- "applied per acquisition" --> pool + pragma -- "applied at open" --> write_conn +``` + +Derived from `crates/clarion-storage/src/writer.rs` (Writer + ActorState + `run_actor`; channel capacity at line 38, batch size at 35, 11-variant match at 154-260, wire contracts at 425-582), `crates/clarion-storage/src/reader.rs:26-119`, `crates/clarion-storage/src/query.rs` (helper catalogue), `crates/clarion-storage/src/cache.rs` and `unresolved.rs`, `crates/clarion-storage/src/schema.rs:17-91`, `crates/clarion-storage/migrations/0001_initial_schema.sql`, and `crates/clarion-storage/src/pragma.rs:16-45`. The "3 variants only" annotation on the MCP-side writer arrow matches catalog §4: `InsertInferredEdges`, `UpsertSummaryCache`, `TouchSummaryCache`. + +--- + +## 5. Sequence — `clarion analyze` happy path + +```mermaid +sequenceDiagram + autonumber + actor user as Developer + participant cli as clarion-cli
analyze::run_with_options + participant inst as install (idempotent) + participant rl as run_lifecycle.rs + participant wact as Writer actor
(spawn_blocking) + participant disc as core::plugin::discovery + participant walk as ignore::WalkBuilder + participant ssc as clarion-scanner +
secret_scan (parallel) + participant brk as CrashLoopBreaker + participant host as PluginHost + participant child as plugin subprocess + participant clust as clustering.rs (Leiden) + participant db as SQLite (WAL) + + user->>cli: clarion analyze + cli->>inst: ensure .clarion/ exists,
apply_migrations (idempotent) + inst-->>cli: schema ready + cli->>cli: load AnalyzeConfig from clarion.yaml + cli->>rl: recover_preexisting_running_runs
(raw UPDATE runs SET status='failed') + rl-->>cli: ok + cli->>wact: Writer::spawn(db_path, 50, 256) + wact-->>cli: (Writer, JoinHandle) + cli->>cli: mint run_id + + cli->>disc: discover() on $PATH + disc-->>cli: Vec + + cli->>walk: collect_source_files
(extension union, .gitignore) + walk-->>cli: file list + + cli->>ssc: pre_ingest scan (thread::scope,
available_parallelism workers) + ssc-->>cli: SuppressionResult,
briefing_blocked set + + cli->>wact: BeginRun(run_id) + wact->>db: BEGIN; INSERT runs status='running' + wact-->>cli: ack + + loop per discovered plugin + cli->>brk: tick? (still healthy) + brk-->>cli: ok + cli->>host: spawn(manifest, project_root) + host->>child: fork+exec, pre_exec setrlimit + host->>child: initialize / initialized + loop per filtered source file + cli->>host: analyze_file(path) + host->>child: JSON-RPC analyze_file + child-->>host: {entities, edges, stats} + host->>host: 4-stage pipeline
(ontology, identity,
jail, entity cap) + end + host->>child: shutdown / exit + host-->>cli: AnalyzeFileOutcome[],
findings, unresolved-call-sites + cli->>brk: tick(success) or tick(crash) + Note over cli,brk: >3 crashes / 60s →
FINDING_DISABLED_CRASH_LOOP,
skip remaining plugins + end + + cli->>wact: InsertEntity × N + wact->>db: per-50 COMMIT; BEGIN cadence + cli->>wact: ReplaceUnresolvedCallSitesForCaller × M + cli->>wact: InsertEdge × K
(FK ordering: entities first) + cli->>wact: InsertFinding × F (secret-scan findings) + + cli->>wact: FlushRunBatch + wact->>db: COMMIT; BEGIN + cli->>clust: cluster_modules
(Leiden via xgraph;
fallback: weighted-components) + clust-->>cli: communities + modularity_score + cli->>wact: InsertEntity (core:subsystem:) × C + cli->>wact: InsertEdge (in_subsystem) × M + + cli->>wact: CommitRun(Completed | SoftFailed) + wact->>db: parent↔contains mismatch check;
UPDATE runs ... ;
COMMIT + wact-->>cli: ack + cli-->>user: exit 0 +``` + +Derived from `crates/clarion-cli/src/analyze.rs:75-645` (`run_with_options`), `crates/clarion-cli/src/run_lifecycle.rs:6-44`, `crates/clarion-storage/src/writer.rs:142-260, 802-877` (per-50 batch cadence, parent↔contains check inside CommitRun at 894-918), `crates/clarion-cli/src/secret_scan.rs:202-382`, `crates/clarion-cli/src/clustering.rs:53-145`, and the host pipeline in `crates/clarion-core/src/plugin/host.rs:866-975`. + +--- + +## 6. Sequence — `clarion serve` MCP tool call + +```mermaid +sequenceDiagram + autonumber + actor agent as Consult-mode LLM agent + participant mcp as clarion-mcp
serve_stdio_with_state + participant disp as ServerState::handle_json_rpc + participant pool as ReaderPool + participant q as clarion-storage query.rs + participant wact as Writer actor
(via mpsc) + participant db as SQLite + + Note over agent,mcp: stdio framing auto-detected:
Content-Length or JSON-line + + agent->>mcp: write frame (tools/call) + mcp->>mcp: read_stdio_frame (blocking) + mcp->>disp: runtime.block_on(handle_frame_with_state) + disp->>disp: validate method+name
against list_tools() + + alt read-only tool
(entity_at, find_entity, callers_of[resolved],
subsystem_members, etc.) + disp->>pool: with_reader(|conn| ...) + pool->>q: e.g. entity_at_line,
find_entities,
call_edges_from + q->>db: SELECT ... + db-->>q: rows + q-->>pool: typed records + pool-->>disp: result + else write-touching tool
(only 3 variants from MCP) + disp->>wact: send WriterCmd::InsertInferredEdges
or UpsertSummaryCache
or TouchSummaryCache + Note over disp,wact: gated by BudgetLedger;
InferredInflight coalesces
concurrent identical requests + wact->>db: write outside run-batch txn + wact-->>disp: oneshot ack + end + + disp-->>mcp: envelope JSON + mcp->>mcp: write_stdio_response
(same framing as request) + mcp-->>agent: framed response +``` + +Derived from `crates/clarion-mcp/src/lib.rs:56-258` (`list_tools`, 20 `ToolDefinition` hits — 19 distinct tool entries + the struct definition; catalog §4 flagged the count drift against the discovery doc), `lib.rs:295-488` (`handle_json_rpc` + `handle_tool_call` 19-arm match), `lib.rs:1560-2030` (inferred-edge dispatch + `BudgetLedger` + `InferredInflight`), `lib.rs:2704-2876` (dual-framing transport), and the writer-channel emission sites at `lib.rs:1682, 1824, 1924, 2024` (3 distinct `WriterCmd` variants). + +--- + +## 7. Subsystem dependency graph + +```mermaid +graph LR + cli["clarion-cli
(binary;
install/analyze/serve)"] + mcp["clarion-mcp
(MCP dispatch + config + Filigree client)"] + core["clarion-core
(entity-id, plugin host,
LLM providers)"] + storage["clarion-storage
(writer-actor + reader pool)"] + scanner["clarion-scanner
(pure secret detection)"] + fixture["clarion-plugin-fixture
(test-only plugin)"] + pyplug["plugins/python
(out-of-tree subprocess)"] + + cli --> core + cli --> mcp + cli --> storage + cli --> scanner + cli -. "dev-dependency
(wp2_e2e tests)" .-> fixture + + mcp --> core + mcp --> storage + + storage --> core + + fixture --> core + + pyplug -. "subprocess only
(JSON-RPC over stdio,
spawned by core::host)" .-> core + + classDef leaf fill:#eee,stroke:#888 + class scanner,fixture,pyplug leaf +``` + +Derived from per-subsystem **Outbound deps** and **Inbound** sections of `02-subsystem-catalog.md`: `clarion-cli` depends on all four library crates and dev-depends on the fixture; `clarion-mcp` depends on `core` + `storage` only; `clarion-storage` depends only on `clarion-core` (one type re-export and one direct facade reach into `manifest::RESERVED_ENTITY_KINDS`); `clarion-scanner` has zero internal Clarion deps; the Python plugin has no Rust dependency edge — it interacts only as a subprocess driven by `core::plugin::host`. The dashed Python-plugin edge captures that wire relationship, not a build-graph edge. + +--- + +## Residual uncertainties + +- **Exact tool count.** Catalog §4 reports 19 distinct `ToolDefinition` entries; discovery §5 reports 20 (`grep -c 'ToolDefinition {'` = 20, which also matches a confirmation grep). The discrepancy is the trailing struct-definition match at `lib.rs:47`. Sequence diagram §6 uses the 19-entry figure for the dispatch match. +- **Wardline probe semantics.** Diagrams show the Python plugin's import-only probe as a dashed edge; the Rust-side consumption of the probe result (the manifest `wardline_aware = true` flag and any `capabilities.wardline` channel in the `initialize` response) was not analysed — catalog §1 and §7 both flag this. +- **HTTP auth selection.** The HTTP container diagram does not branch on bearer vs HMAC vs `trust-loopback`; catalog §3 records that selection is config-driven (HMAC preferred when `identity_token_env` set, bearer when only `token_env` set, none when neither). Sequence-level auth was out of scope for this pass. +- **Phase-3 clustering raw connection.** Diagram §5 shows clustering happening after a `FlushRunBatch`, but does not depict that the clustering pass opens a *direct* `rusqlite::Connection` bypassing the reader pool (catalog §3 concerns). The diagram aggregates it into the `clust` participant. +- **Federation `/api/v1/_capabilities` unauthenticated route.** The container and context diagrams collapse all four HTTP routes into one labelled edge; the unprotected capabilities route is not visually distinguished. +- **`analyze` subcommand reachable inside `serve`.** `clarion-mcp` tools `analyze_start` / `analyze_status` / `analyze_cancel` spawn `clarion analyze` as a child of `clarion serve` (catalog §4 key components). This re-entry is not drawn in diagram §2 to keep the container shape readable; it appears implicitly via the MCP tool surface in diagram §6. +- **LLM CLI providers.** The context diagram shows `claude` / `codex` CLI binaries as external processes; they are spawned by `clarion-core::llm_provider`'s `ClaudeCliProvider` / `CodexCliProvider`, but the host-side timeout/reaper-thread machinery is not drawn. diff --git a/docs/arch-analysis-2026-05-22-1924/04-final-report.md b/docs/arch-analysis-2026-05-22-1924/04-final-report.md new file mode 100644 index 00000000..b5e31b19 --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/04-final-report.md @@ -0,0 +1,330 @@ +# 04 — Final Report: Clarion Architecture Analysis + +**Date:** 2026-05-22 +**Scope:** Entire Rust workspace (`crates/`) plus the Python language plugin (`plugins/python/`). +**Method:** From-scratch source archaeology. Seven independent codebase-explorer subagents read source, manifests, migration SQL, fixtures, and tests for one subsystem each. **No existing design docs** (`docs/clarion/**`, `docs/suite/**`, `docs/implementation/**`, ADRs, sprint READMEs, prior arch-analyses) were consulted during the analysis. A separate validator subagent spot-checked eight load-bearing factual claims against source. +**Validator status:** +- *Subsystem catalog (`02-`):* NEEDS_REVISION (warnings) — one factual error (Python plugin's `MAX_FILES_PER_PYRIGHT_SESSION` literal) fixed inline; one cosmetic nit accepted; all eight load-bearing spot-check claims passed. +- *Final report (this doc):* APPROVED with editorial warnings — discovery-doc tool-count sweep completed; this top matter clarified. + +--- + +## 1. Executive Summary + +Clarion is a **single-binary Rust code-archaeology tool** that ingests source trees through out-of-tree language plugins, persists a typed entity/edge graph in an embedded SQLite database, and exposes that graph to LLM agents through two distinct read surfaces — a stdio **MCP server** with 19 navigation tools, and an authenticated **HTTP read API** for cross-product federation. A reference **Python plugin** (the only in-tree language plugin) drives `pyright` as an LSP subprocess to extract type-resolved call and reference edges. + +The architecture is structurally simple — 7 subsystems, ~50K LOC, no inter-crate cycles — but **operationally subtle**. The hard parts live in three places: + +1. **Plugin-host subprocess supervision** (`clarion-core::plugin::host`): generic-over-IO synchronous JSON-RPC supervisor, per-frame ceiling rejection without body-consume, four enforcement layers (frame size, path jail, entity cap, crash-loop), forked-child resource limits via `pre_exec`-installed `setrlimit`, detached stderr-drain thread. +2. **Storage's writer-actor discipline** (`clarion-storage::writer`): every mutation routes through one bounded-mpsc actor on `spawn_blocking`; per-run super-transaction; batch commits every 50 writes; wire-contract enforcement (edge kind/confidence/source-range tables, parent↔contains bijection) at the writer boundary so caller bugs cannot corrupt graph shape. +3. **The analyze pipeline as a sequence of nested gates** (`clarion-cli::analyze`): orphan-run recovery → secret scan → BeginRun → per-plugin loop with two distinct breakers (path-escape inside the host, crash-loop in the run loop) → unresolved-call-site → edge resolution → clustering → CommitRun. All in a single 570-line function. + +The system has invested heavily in **failure containment**: 11+ hardcoded resource limits with `CLA-INFRA-*` finding subcodes, two independent rolling-window breakers, drop-with-finding vs. kill-with-error asymmetry, cross-language byte-for-byte fixture parity for entity IDs, and a baseline mechanism that intentionally does **not** suppress drifted hashes at the same line. These are the marks of a system that has thought hard about adversarial-plugin and partial-failure scenarios. + +The clearest **architectural debt** is at the file-size level: four files (`mcp/lib.rs` 4703, `core/plugin/host.rs` 2935, `cli/analyze.rs` 2549, `core/llm_provider.rs` 2467) hold the bulk of the operational complexity. They are not poorly factored *internally* — each has clearly named functions and inline-test discipline — but each is one file's-worth of change risk per touch. + +--- + +## 2. The System in Code (no doc references) + +### 2.1 Subsystem inventory + +| # | Subsystem | Type | Source LOC | Test LOC | Confidence | +|---|-----------|------|-----------:|---------:|------------| +| 1 | `clarion-core` | library | 11,653 | 325 (+ inline) | High | +| 2 | `clarion-storage` | library | 3,199 | 4,871 | High | +| 3 | `clarion-cli` | binary (`clarion`) | ~6,800 | ~6,400 | High | +| 4 | `clarion-mcp` | library | ~6,600 | ~2,200 | High (sampled `lib.rs`) | +| 5 | `clarion-scanner` | library | 881 | 655 | High | +| 6 | `clarion-plugin-fixture` | test bin | 187 | (see consumers) | High | +| 7 | `plugins/python` | external bin (`clarion-plugin-python`) | 3,028 | 3,440 | High | + +**Totals:** ~32K source / ~18K test (Rust) + 3K source / 3.4K test (Python) = ~50K LOC of first-party code. Test corpus is **~57% of source LOC** — high by typical standards, dominated by `clarion-storage` where tests outweigh source 1.5×. + +### 2.2 External dependencies (architecturally significant) + +- `tokio` (multi-thread + sync + macros) — runtime for `serve` and the writer-actor. +- `rusqlite` 0.31 with bundled SQLite — embedded DB, no external server. +- `deadpool-sqlite` 0.8 (`rt_tokio_1`) — async-friendly reader pool. +- `axum` 0.7 + `tower` / `tower-http` — HTTP read API (`/api/v1/files*`). +- `reqwest` 0.12 with rustls — outbound HTTP to OpenRouter LLM, Filigree associations. +- `clap` 4 — CLI. +- `nix` — `setrlimit` for plugin children via `pre_exec`. +- `xgraph` — Leiden community detection (with a hand-rolled fallback). +- `serde_norway` — YAML (`clarion.yaml`, secret-scan baseline). +- Python: `pyright` (LSP subprocess), `tomli` (manifest), `pytest`/`mypy --strict` for dev. + +No mocked-out networking; the LLM provider trait has a `RecordingProvider` for tests but production hits the real OpenRouter HTTP endpoint or shells out to `claude`/`codex` CLIs. + +### 2.3 Wire surfaces (external interfaces) + +| Surface | Where | Transport | Auth | +|---|---|---|---| +| `clarion` CLI | `clarion-cli/src/main.rs`, `cli.rs` | argv | n/a | +| Plugin JSON-RPC 2.0 | `clarion-core/src/plugin/protocol.rs` | LSP-style `Content-Length` framing over stdio pipes | none (process boundary is the trust boundary) | +| MCP server | `clarion-mcp/src/lib.rs::serve_stdio_with_state_on_runtime` | stdio (auto-detects `Content-Length` framing vs. bare-JSON-line) | none (caller-trusted) | +| HTTP read API | `clarion-cli/src/http_read.rs:347-372` | HTTPS-capable Axum on `0.0.0.0:` | **HMAC-SHA256** > bearer > **loopback-trust** with operator WARN (4 routes, precedence in code) | +| OpenRouter | `clarion-core/src/llm_provider.rs::OpenRouterProvider` | HTTPS | API key (env) | +| Filigree | `clarion-mcp/src/filigree.rs` | HTTPS | bearer (env) + `x-filigree-actor` header | +| Pyright LSP | (Python plugin) `clarion_plugin_python/pyright_session.py` | LSP subprocess pipes | none | + +--- + +## 3. Architecture Narrative + +### 3.1 The shape: a CLI with two persistent modes + +The `clarion` binary has **three subcommands** (`install`, `analyze`, `serve`) but two architectural shapes. `install` and `analyze` are one-shot processes; `serve` is a long-running supervised topology with two threads sharing one `ReaderPool`: + +- A **current-thread Tokio runtime** drives the MCP stdio server (`clarion-mcp::serve_stdio_with_state_on_runtime`). +- A **multi-thread Tokio runtime** drives the Axum HTTP read API on a configurable port. + +`clarion-cli::serve::run` enforces shared-pool identity with `Arc::ptr_eq` (`reader.rs:26-119` exposes a `shares_pool_with` runtime proof via an `Arc<()>` identity tag) — a structural guarantee that both servers observe the same database snapshot. Failure of either thread crashes the binary; there is no per-surface restart. + +### 3.2 The analyze pipeline + +`clarion-cli::analyze::run_with_options` is a single 570-line function (`analyze.rs:75-645`) that linearises 13 phases: + +1. canonicalize project path +2. load `clarion.yaml` +3. raw `UPDATE runs SET status='failed' WHERE status='running'` — **orphan-run recovery before the writer-actor even exists** +4. spawn writer-actor, mint `run_id` +5. plugin discovery via `$PATH` scan for `clarion-plugin-*` executables (`clarion-core::plugin::discovery`) +6. compute extension union from plugin manifests +7. tree walk +8. **parallel secret scan, BEFORE BeginRun, BEFORE any plugin spawn** (`clarion-scanner` driven by `secret_scan::scan_source_files_parallel`) +9. `BeginRun` command to writer-actor +10. per-plugin loop (each plugin runs in `spawn_blocking`): handshake → `analyze_file` × N (with heartbeat logging) → `shutdown`; per-run `CrashLoopBreaker` from `clarion-core` ticks on >3 crashes / 60 s and drops remaining plugins with `FINDING_DISABLED_CRASH_LOOP` +11. entities → unresolved-call-sites → edges ingestion in **strict FK order** +12. phase-3 clustering via `clustering::cluster_with_leiden` (Leiden through `xgraph` with deterministic seed; falls back to `local_weighted_components` if Leiden returns ≤1 community) +13. `CommitRun` (or `SoftFail` / `HardFail` if invariants tripped) + +The function carries `#[allow(clippy::too_many_lines)]` (`analyze.rs:74`). The reviewer rated this the single largest concern in the CLI — every change vector listed above goes through the same scope. + +### 3.3 Plugin host: the most carefully engineered surface + +`clarion-core::plugin::host::PluginHost` (`host.rs:384-1182`) is generic over reader/writer so the in-process `mock.rs` (876 LOC, `#[cfg(test)] pub(crate)`) can drive it without a subprocess. The four-stage per-entity validation pipeline at `host.rs:866-975` runs for every entity a plugin emits: + +0. **Field-size** — 4 KiB per scalar field, 64 KiB per `#[serde(flatten)]` extras map. +1. **Ontology** — `kind ∈ manifest.ontology.entity_kinds`. +2. **Identity** — recomputed `entity_id(plugin_id, kind, qualified_name)` must equal the wire `id` (prevents ID-namespace spoofing). +3. **Jail** — `jail_to_string(project_root, source.file_path)` must succeed; on failure, tick `PathEscapeBreaker`. + +Steps 0–2 **drop-with-finding** (continue). Step 3 drops + records a finding; >10 path-escapes / 60 s trips the breaker and **kills the plugin**. A separate post-step **entity cap** (`EntityCountCap::DEFAULT_MAX = 500_000` cumulative) on overflow kills the plugin. + +Subprocess hygiene: +- `spawn` returns `(PluginHost, std::process::Child)` — the **caller owns reaping**. `Child::Drop` does not `waitpid` on Unix (documented at `host.rs:630-641`); a handshake-failure inside `spawn` reaps before returning. +- **`pre_exec`-installed `setrlimit`** for `RLIMIT_AS` (virtual address) and `RLIMIT_NPROC` (bumped to 4096 when the plugin manifest declares Pyright capability, because Node's LSP host spawns helper processes counted against the user's nproc). +- **Detached stderr-drain thread** (`host.rs:609-620`) named `clarion-plugin-stderr-drain:` reads `ChildStderr` 4 KiB at a time into a 64 KiB ring buffer. Rationale at `host.rs:550-561`: an inherited stderr could either flood the operator's terminal or deadlock the plugin on `write(2)` when the host blocks in `analyze_file`. + +Validator-confirmed via `host.rs:609-620`, `writer.rs:35,38,813`, `analyze.rs:242,244,277+`. + +### 3.4 Storage: an actor + a pool over one SQLite file + +`clarion-storage` is the **only path** to SQLite. Concurrency model: + +- **One writer-actor** spawned on `tokio::task::spawn_blocking`. Bounded `mpsc::Receiver` (capacity 256, `writer.rs:35`), 11 command variants each carrying a `oneshot::Sender>` ack. Per-run super-transaction; batch-cadence commits every 50 writes (`writer.rs:38, 813`). +- **Reader pool** via `deadpool-sqlite` (`Runtime::Tokio1`). Reader PRAGMAs reapplied per acquisition. Production sizes: 16 in `serve.rs`, 4 in `http_read.rs` test. +- **PRAGMA discipline** (`pragma.rs:16-45`): WAL (asserted — the assertion is hard, not advisory), `synchronous=NORMAL`, `busy_timeout=5000`, `wal_autocheckpoint=1000`, `foreign_keys=ON`. **No `application_id` / `user_version`** are set; cross-tool collisions on the DB file are not detected at the SQLite level. +- **Schema migrations** (`schema.rs:17-91`): single `include_str!`-embedded migration, idempotent via a `schema_migrations` table. Explicit comment at lines 45-49 warns that `.ok()` instead of `OptionalExtension::optional()` would silently mask `DatabaseLocked` / `CorruptDb`. + +**Wire contracts are enforced at the writer boundary** (`writer.rs:425-582`): +- `STRUCTURAL_EDGE_KINDS = {contains, in_subsystem, guides, emits_finding}` — must be `confidence=resolved`, must have NULL byte ranges. +- `ANCHORED_EDGE_KINDS = {calls, references, imports, decorates, inherits_from}` — must have both byte-start AND byte-end, must NOT be `inferred` at scan time. +- `parent_contains_mismatch` (`writer.rs:954-1021`) — bidirectional SQL pair asserting `entities.parent_id` and `edges WHERE kind='contains'` are bijective. Failure aborts the entire run with `CLA-INFRA-PARENT-CONTAINS-MISMATCH`. + +The migration SQL defines 9 tables, 1 FTS5 virtual, 3 triggers, 2 generated columns, 1 view, and crucially `edges` is `WITHOUT ROWID` on natural PK `(kind, from_id, to_id)`. + +### 3.5 MCP server: 19 tools (not 20) + +`clarion-mcp::serve_stdio_with_state_on_runtime` registers **19 tools** in a `ToolDefinition` registry at `lib.rs:56-257`. The original discovery doc said "twenty"; the catalog and validator both confirmed 19 via direct enumeration. (Discovery's `grep -c 'ToolDefinition {'` had counted the struct declaration plus 19 `vec![]` instances.) Discovery has been corrected in place. + +Tool dispatch is **strictly sequential per session** — there is no concurrent tool execution within one MCP connection. The dispatcher auto-detects framing (LSP `Content-Length` vs. bare JSON line) by peeking the first non-whitespace byte. Read path goes through the reader pool (~30 sites); the writer is touched by only **3 command variants** across 4 sites — `InsertInferredEdges` (×2), `TouchSummaryCache` (×1), `UpsertSummaryCache` (×1) — and all are gated on the summary-LLM writer field being configured. + +### 3.6 HTTP read API: the federation surface + +`clarion-cli::http_read.rs:347` exposes four routes on Axum: + +- `GET /api/v1/files` +- `POST /api/v1/files/batch` (cap 256) +- `POST /api/v1/files:resolve` (cap 1000) +- `GET /api/v1/_capabilities` (unprotected) + +Auth precedence is **hand-rolled HMAC-SHA256 > bearer > loopback-trust with WARN** (constant-time compare, 16 KiB body cap, 10 s timeout, 64 concurrency). The middleware panics on unenumerated errors — a deliberate fail-loud posture. + +### 3.7 Python plugin: pyright-as-a-service + +`clarion-plugin-python` is a PEP 517 console-script binary (`pyproject.toml:32-33`) per ADR-021 bare-basename convention. It implements 5 JSON-RPC methods (`initialize`, `initialized`, `analyze_file`, `shutdown`, `exit`; `server.py:237-272`) over Content-Length framing with an 8 MiB cap. The reference plugin and Python plugin share their wire shape with `clarion-plugin-fixture`. + +The interesting machinery is **pyright integration**: +- Pyright runs as a subprocess (`pyright-langserver --stdio`), driven by a full LSP client (`PyrightSession`). +- Session is recycled every **25 files** (`MAX_FILES_PER_PYRIGHT_SESSION` at `server.py:49`) — a wholly-separate-from-the-3-restart-cap heuristic. +- Calls use `prepareCallHierarchy` + `callHierarchy/outgoingCalls`. +- References use `textDocument/definition` with `typeDefinition` fallback for annotation references. +- All failures degrade to zero edges + a `CLA-PY-PYRIGHT-*` finding. + +The extractor walks the AST three times (recursive `_walk`, `_ImportEdgeCollector`, `_ReferenceSiteCollector`). `@overload` stubs are dropped pre-emit; surviving collisions drop first-wins. Cross-language entity-ID parity is enforced by `tests/test_entity_id.py:25` consuming the same `fixtures/entity_id.json` as the Rust `entity_id.rs:371-587`. + +A **`wardline_probe.py`** module attempts `import wardline.core.registry` and reports `{status: absent|enabled|version_out_of_range}` from `initialize`. Fail-soft. + +### 3.8 The secret scanner + +`clarion-scanner` is a **pure detection library** — it does not walk the FS, does not decide *what* to scan. Callers invoke `Scanner::scan_bytes(&[u8])` per-file. `clarion-cli::secret_scan::scan_source_files_parallel` drives it across the project tree in `analyze.rs:242-243`, **before** `BeginRun` and before any plugin spawn. + +Detection: **12 named pattern rules** (AWS, GitHub 3×, Anthropic, OpenAI, Stripe, Slack, JWT, PEM private key, contextual `password|token|api_key = "…"`) + **2 entropy classes** (base64 min-len 20 min-entropy 4.5; hex min-len 40 min-entropy 3.0). All map to a closed 14-variant `DetectSecretsRule` enum at `lib.rs:102-118`, rule-ids aligned to Yelp `detect-secrets`. + +The **baseline file** is YAML keyed by path, suppressing on exact `(file, rule_type, hashed_secret, line_number)` quadruples — **only when `is_secret: false`**. Drifted hashes at the same line are deliberately NOT suppressed. Parse-time validation rejects non-1.0 version, absolute/`..` paths, missing justifications, unknown rule types, invalid hex hashes. (Tests at `tests/scanner.rs:509-556` lock this in.) + +--- + +## 4. Dependency Topology + +**Inter-subsystem dependencies (Rust):** + +``` +clarion-cli ──► clarion-core + ├──► clarion-storage + ├──► clarion-scanner + └──► clarion-mcp + +clarion-mcp ──► clarion-storage + └──► clarion-core (for protocol::read_frame, BriefingBlockReason, LLM types) + +clarion-storage ──► clarion-core (EdgeConfidence, RESERVED_ENTITY_KINDS — facade leak) + +clarion-scanner ── (no internal deps) +clarion-plugin-fixture ──► clarion-core (only for JSON-RPC types) +plugins/python ── (no Rust deps; speaks the wire only) +``` + +**No cycles.** The dependency graph is a DAG with `clarion-core` at the bottom, `clarion-storage` and `clarion-scanner` as leaves of the lower layer, and `clarion-cli` and `clarion-mcp` as the consumers. + +**Notable inbound reach (the facade leak):** `clarion-storage::writer.rs:427` reaches `clarion_core::plugin::manifest::RESERVED_ENTITY_KINDS` *directly* through the module path, bypassing the `lib.rs:13-49` `pub use` facade. The policy comment at `clarion-core::lib.rs:5-7` describes the facade as the supported surface; this constant is not re-exported there. + +**The Python plugin is intentionally isolated** — no shared crate dependency, no shared registry. It speaks the wire protocol, consumes the same `fixtures/entity_id.json` the Rust side uses for parity testing, and is otherwise independent of the Rust workspace. + +--- + +## 5. Cross-Cutting Concerns + +| Concern | Where | +|---|---| +| **Error model** | Per-module `thiserror` enums composed via `#[from]`. `HostError` (`host.rs:334-370`) wraps eight underlying errors plus three policy variants. `StorageError` (`storage/src/error.rs`) similar. CLI uses `anyhow` at the boundary. | +| **Logging / tracing** | `tracing` + `tracing-subscriber` with `env-filter`. Plugin heartbeats logged from analyze loop (`analyze.rs:1272-1307`). | +| **Async runtime** | `tokio` multi-thread. **Two exceptions**: writer-actor uses `spawn_blocking` (synchronous SQLite); plugin host is fully synchronous (`BufRead`/`Write`, no async). | +| **Config** | `serde_norway` (YAML) for `clarion.yaml`; `clap` derive for CLI. Subprocess plugins read TOML manifests (`plugin.toml`). | +| **Migrations** | Embedded via `include_str!` in `clarion-storage::schema`. Tracked in `schema_migrations` table. No `application_id` / `user_version`. | +| **Security boundaries** | (a) Plugin process boundary (jail + setrlimit + entity cap + path-escape breaker + frame ceiling). (b) HTTP API HMAC > bearer > loopback. (c) Pre-ingest secret scan with baseline suppression that won't mask drift. | +| **Findings vocabulary** | `CLA-INFRA-*` (host enforcement), `CLA-PY-PYRIGHT-*` (Python plugin pyright failures), `CLA-SEC-SECRET-DETECTED` (scanner), `CLA-INFRA-PARENT-CONTAINS-MISMATCH` (storage invariant). Surfaced via `HostFinding` (core) and the standard `Finding` record persisted in SQLite. | +| **Determinism** | Clustering uses a deterministic seed (`clustering.rs`). Entity IDs are pure functions of (plugin_id, kind, canonical_qualified_name). Cross-language byte-for-byte parity test gates against `fixtures/entity_id.json`. | + +--- + +## 6. Risks and Smells (Concrete, Source-Cited) + +Severity bands: +- **🔴 High** — affects correctness, security, or change-amplification surface +- **🟡 Medium** — operational friction, will degrade over time +- **🟢 Low** — cleanup opportunity + +### 🔴 High + +1. **Four monolithic files concentrate change risk.** + - `clarion-mcp/src/lib.rs` — 4,703 LOC. Holds the 19 tool registry, `ServerState`, all per-tool handlers, the `BudgetLedger`, the `InferredInflight` coalescer, and tests. + - `clarion-core/src/plugin/host.rs` — 2,935 LOC. Holds the entire `PluginHost` impl, the four-stage pipeline, the stderr drainer, `pre_exec` setup, and the shutdown idempotency. + - `clarion-cli/src/analyze.rs` — 2,549 LOC; `run_with_options` itself is 570 lines. + - `clarion-core/src/llm_provider.rs` — 2,467 LOC. **Bundles the trait + reqwest HTTP transport + two CLI-subprocess transports + prompt templates in a crate whose `lib.rs:1` doc-comment says it owns "domain types, identifiers, and provider traits"**. + + Refactor split exists for each (pipeline-axis / lifecycle-axis / IO-axis for `host.rs`; tool-category split for `mcp/lib.rs`; per-phase function extraction for `analyze.rs`; new `clarion-llm` crate for `llm_provider.rs`). None is taken yet. + +2. **Blocking HTTP inside async (MCP filigree client).** `clarion-mcp::filigree` issues **blocking** `reqwest::blocking::get` calls from `async` tool handlers (validator-flagged; reviewer cited as a real concern). Will tie up the runtime thread on a slow Filigree. + +3. **No analyze-child timeout in `serve`.** The MCP `analyze_start` / `analyze_status` / `analyze_cancel` tool family launches an analyze process from inside the MCP server — but the catalog flags "no analyze-child timeout" as an mcp-side smell (`02-subsystem-catalog.md` §4). Inspect before claiming this is hardened. + +4. **Subprocess-child reaping ownership is type-system-unenforced.** `PluginHost::spawn` returns `(Self, Child)` where the caller is documented (`host.rs:630-641`) to reap. A `KillOnDrop` newtype around `Child` would make the contract type-mechanical instead of comment-mechanical. + +### 🟡 Medium + +5. **`clarion-storage`'s SQLite file has no `application_id` / `user_version`.** Two Clarion versions sharing a DB by accident would not be detected at the SQLite layer; only the `schema_migrations` table catches it, and only on write. Adding both PRAGMAs is a one-line change with no downside. + +6. **`llm_provider.rs` belongs in its own crate.** It pins `reqwest` and `tokio::time::timeout`-equivalent CLI machinery into `clarion-core` — the crate that supervises plugins. A `clarion-llm` crate would shrink the trust surface of the host runtime. + +7. **Facade leak: `clarion-storage::writer.rs:427` reaches `clarion_core::plugin::manifest::RESERVED_ENTITY_KINDS` directly.** Either lift to the facade or expose `Manifest::is_reserved_kind`. As-is, the internal module path is semi-public. + +8. **Eleven hardcoded operational limits in `clarion-core`.** `MAX_PROTOCOL_ERROR_FIELD_BYTES`, `MAX_ENTITY_FIELD_BYTES`, `MAX_ENTITY_EXTRA_BYTES`, `STDERR_TAIL_BYTES`, `MAX_HEADER_LINE_BYTES`, `MAX_UNRESOLVED_CALLEE_EXPR_BYTES`, `ContentLengthCeiling::DEFAULT`, `EntityCountCap::DEFAULT_MAX`, `DEFAULT_MAX_RSS_MIB`, `DEFAULT_MAX_NOFILE`, `DEFAULT_MAX_NPROC`. The comment at `breaker.rs:7` acknowledges this lands "in WP6". Every tunable is a recompile today. + +9. **Path jail is TOCTOU-by-design.** `jail.rs:67-72` documents this: the canonical-path return is a membership proof at canonicalization time, not a durable file handle. The current consumer doesn't open files, so this is latent; a future caller that does could miss the contract. + +10. **Pyright session restart constant `25` is divorced from session's own 3-restart cap.** Interaction between server-driven recycling and session-driven failure restarts is not centrally documented. + +### 🟢 Low + +11. **Integration tests for `clarion-core` cover only the happy path.** Most `HostError` variants and `CLA-INFRA-*` finding subcodes are tested via inline `#[cfg(test)]` blocks, which is fine but uneven — `tests/host_subprocess.rs` is 325 lines, covering one walkthrough. + +12. **Tool count drift in discovery vs. registry.** Discovery initially said 20; actual is 19. Already corrected in this analysis; flag for future readers. + +13. **`mock.rs` at 876 LOC is `#[cfg(test)] pub(crate)`.** Volume itself is a smell — the host's pipeline is stateful enough to need a sub-DSL to test, which corroborates the `host.rs` size concern. + +--- + +## 7. Strengths Worth Naming + +Concrete things this codebase does *well* that a brownfield analysis should not understate: + +- **Generic-over-IO supervisor with in-process mock.** `PluginHost` plus `mock.rs` is a textbook approach to testing subprocess supervision without spawning subprocesses. The seam at `connect()` (`host.rs:658`) is clean. +- **Wire-contract enforcement at the writer boundary.** Caller bugs cannot corrupt the graph shape. Three structurally-distinct invariants — edge-kind tables, source-file-anchor kinds, parent↔contains bijection — all enforced in SQL-adjacent Rust at `writer.rs:425-582, 954-1021`. +- **Cross-language byte-for-byte parity fixture.** `fixtures/entity_id.json` is consumed by both `clarion-core::entity_id.rs:371-587` and `plugins/python/tests/test_entity_id.py:25`. Drift between the two implementations is caught in CI before either side ships. +- **Per-frame ceiling rejection before body-consume.** `transport.rs:67-71` rejects with `TransportError::FrameTooLarge` without consuming bytes — closing a memory-exhaustion attack vector that "read then check" would leave open. +- **Two breakers, two scopes.** `PathEscapeBreaker` (per-plugin, host-owned) polices one misbehaving plugin's emissions; `CrashLoopBreaker` (per-run, caller-driven) polices the fleet. Same shape, different scope. Named asymmetrically only in *ownership*, not in *interface*. +- **The secret-scanner baseline that won't mask drift.** `(file, rule_type, hashed_secret, line_number)` quadruple match with `is_secret: false` requirement means a changed secret at the same line is *not* suppressed (locked in by `tests/scanner.rs:509-556`). This is a deliberate regression net — the kind of thing easy to get wrong, here gotten right. +- **Deterministic clustering with hand-rolled fallback.** `clustering.rs` uses Leiden via `xgraph` with a seeded RNG, with a `local_weighted_components` fallback for the degenerate "≤1 community" case. Determinism is testable. + +--- + +## 8. Open Questions for the Next Phase + +Things that the *catalog* could not answer from code alone, and that an architect or feature-owner should clarify: + +1. **Why the 25-file pyright restart constant?** Empirical? Conservative bound on pyright memory growth? Either is fine; not knowing makes future tuning a guess. +2. **What is the post-1.0 plan for the four monolith files?** Each has a natural refactor split. Are these on the roadmap, or is the policy "no split until the file actively impedes a change"? +3. **Will `clarion-llm` become a crate?** The `llm_provider.rs` placement in `clarion-core` is the largest single argument against the lib doc-comment. +4. **What is the architect's stance on `application_id` / `user_version`?** Trivial to add; non-trivial to add *retroactively* once installed DBs exist in the wild. +5. **Operational tuning roadmap.** Eleven hardcoded limits, plus 25-file restart, plus 256/50 batch-cadence constants. WP6 is named in code comments — what is its current status? + +--- + +## 9. Methodology and Confidence + +**Validation status:** NEEDS_REVISION (warnings) → fixed → effectively APPROVED. One factual error (Python plugin restart constant value) corrected in `02-subsystem-catalog.md`. One cosmetic nit (subsystem 7 title) accepted. Eight spot-check claims all PASSED: + +| Claim | Source verified | Status | +|---|---|---| +| stderr drain thread | `host.rs:609-620` | PASS | +| writer actor: 256-cap mpsc, 50-write batch | `writer.rs:35, 38, 813` | PASS | +| analyze ordering (secret scan → BeginRun → plugin spawn) | `analyze.rs:242, 244, 277+` | PASS | +| MCP tool count | `mcp/lib.rs:47-257` | 19 (discovery corrected) | +| HTTP read API: 4 routes | `http_read.rs:364-372` | PASS | +| plugin-fixture: 5 methods | `main.rs:51, 54, 68, 77, 116` | PASS | +| `clarion-plugin-python` binary name | `pyproject.toml:32-33` | PASS | +| Scanner: 12 named + 2 entropy | `patterns.rs:194-269`, `entropy.rs:11-18` | PASS | + +**Overall confidence: High** for everything in §3, §4, §5, §6, §7. **Medium-High** for §2 LOC counts (some are approximate-from-discovery, validated to within 2%). **Medium** for §8 (recommendations) — these depend on architect intent the code cannot reveal. + +**Coverage:** all 7 subsystems analyzed; every load-bearing module read at least partially; the four largest files sampled with explicit annotation of what was end-to-end-read vs. sampled (in each subsystem's Confidence statement). One file *not* sampled to completion is `clarion-mcp/src/lib.rs` (4,703 LOC) — its 19 tool registry was enumerated and the dispatcher structure was characterised, but each tool's individual handler body was not read end-to-end. + +**What I would do next if continuing:** + +- Quality-assessment pass on the four large files (`mcp/lib.rs`, `host.rs`, `analyze.rs`, `llm_provider.rs`) — they are the focal points for ROI on any refactor budget. +- Security-surface pass on the HTTP read API (HMAC implementation, body cap, panic-on-unenumerated-middleware-error stance). +- Test-pyramid analysis — given the 57% test/source ratio, where are the gaps? My read: storage and scanner are saturated; the host has happy-path-only integration coverage. + +--- + +## 10. Pointers + +- **Architecture diagrams:** see `03-diagrams.md` (7 Mermaid diagrams: 2 C4 levels, 2 component, 2 sequence, 1 dependency graph). +- **Per-subsystem detail:** see `02-subsystem-catalog.md`. +- **Discovery (entry points, stack):** see `01-discovery-findings.md`. +- **Validator report:** see `temp/validation-catalog.md`. + +End of report. diff --git a/docs/arch-analysis-2026-05-22-1924/temp/answer-python-engineer.md b/docs/arch-analysis-2026-05-22-1924/temp/answer-python-engineer.md new file mode 100644 index 00000000..649bebfd --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/temp/answer-python-engineer.md @@ -0,0 +1,125 @@ +# Python Engineering Analysis: Open Questions from 04-final-report §8 + +**Date:** 2026-05-23 | **Scope:** Q1, Q2, Q5, and the flagged server/session interaction. + +--- + +## Q1: Why the 25-file pyright restart constant? + +The constant was introduced in commit `68b719c` ("Bound Pyright dogfood analysis", 2026-05-20) +with no rationale comment. `git log -S 'MAX_FILES_PER_PYRIGHT_SESSION'` confirms this was the +introducing commit, and the diff contains no measurement, no profiling reference, and no +explanation for the choice of `25`. The number is empirically ungrounded. + +The primary driver of Pyright RSS growth per session is the type graph for imported modules, +which accumulates and does not shrink across `textDocument/didOpen`/`didClose` cycles. Per-file +marginal cost drops once the import frontier saturates. The right boundary is the inflection +point on a per-file RSS delta curve; that experiment has not been run. A `psutil`-based probe +during `analyze` on the `elspeth` target corpus would close this. + +--- + +## Q2: `pyright_session.py` at 1,406 LOC — cohesion story + +Both a file and class problem; the class is the root. `PyrightSession` (lines 131–890) bundles +five distinct concerns: + +| Group | Methods | Lines | +|-------|---------|-------| +| Process lifecycle | `close`, `_ensure_process`, `_record_restart_or_poison`, `_start_process`, `_initialize`, `_resolve_executable`, `_subprocess_env`, `_start_stderr_drain`, `_drain_stderr` | 182–198, 586–680 | +| LSP transport | `_request`, `_notify`, `_live_process`, `_write_message`, `_read_message`, `_handle_server_message`, `_workspace_configuration_result`, `_configuration_for_section` | 732–834 | +| Call resolution | `resolve_calls`, `_resolve_with_pyright` | 199–256, 332–425 | +| Reference resolution | `resolve_references`, `_resolve_references_with_pyright`, `_reference_target_ids` | 257–330, 427–511 | +| Index + bookkeeping | `_deadline_for_file`, `_function_index_for_path`, `_record_finding`, `_pop_findings`, etc. | 531–590, 836–890 | + +The remainder of the file (lines 893–1406) is module-level AST helpers and visitors that belong +with call resolution. The cleanest split is `_LspTransport` (process lifecycle + wire framing) +extracted as a composable object; the transport seam would enable testing the LSP-protocol layer +without wiring call or reference resolution. The `noqa: PLR0913` at `pyright_session.py:132` +(13-parameter `__init__`) is the linter's proxy for the same signal. + +--- + +## Q5 + interaction: Magic numbers and the server/session coupling + +### Constants classified + +**Wire-contract-pinned (must track Rust counterparts; not WP6 candidates):** + +| Constant | Location | Rust counterpart | +|----------|----------|-----------------| +| `MAX_CONTENT_LENGTH = 8 MiB` | `server.py:48` | `ContentLengthCeiling::DEFAULT` in `clarion-core/limits.rs` | +| `MAX_UNRESOLVED_CALLEE_EXPR_BYTES = 512` | `pyright_session.py:43` | Same-named constant in `clarion-core/limits.rs` | +| `STDERR_TAIL_LIMIT = 65536` | `pyright_session.py:49` | `STDERR_TAIL_BYTES = 64 KiB` in `host.rs` | + +None carry a comment naming the Rust counterpart. A Rust-side change will not propagate by inspection. + +**Operational tunables (WP6 `clarion.yaml` candidates):** + +| Constant | Location | Priority | +|----------|----------|---------| +| `MAX_FILES_PER_PYRIGHT_SESSION = 25` | `server.py:49` | High — see interaction below | +| `MAX_PYRIGHT_RESTARTS_PER_RUN = 3` | `pyright_session.py:44` | High — name says "per run"; implementation is per session (see below) | +| `PYRIGHT_INIT_TIMEOUT_SECS = 30.0` | `pyright_session.py:46` | High — gates every restart on slow nodes | +| `PYRIGHT_CALL_TIMEOUT_SECS = 5.0` | `pyright_session.py:47` | Medium | +| `PYRIGHT_FILE_TIMEOUT_SECS = 3.0` | `pyright_session.py:48` | Medium | +| `MAX_REFERENCE_SITES_PER_FILE = 2000` | `pyright_session.py:45` | Low-medium | + +### The undocumented interaction + +`MAX_PYRIGHT_RESTARTS_PER_RUN` is named "per run" but the `_restart_count` and `_disabled` +that implement it are instance state on `PyrightSession` (`pyright_session.py:158–159`). +`server.py:217–219` destroys the instance every 25 files (`state.pyright.close(); state.pyright = None`), +creating a fresh instance with `_restart_count = 0`. The constant's name states the intent; the +implementation drifts from it. + +**Failure mode A:** A Pyright binary that crashes reliably exhausts its 3-restart budget, +gets disabled via `_disabled = True` (`pyright_session.py:601–608`), and then silently regains +full fault tolerance at file 26 when the server creates a new instance. On a 1000-file project +this produces up to 40 restart cycles instead of one `CLA-PY-PYRIGHT-POISON-FRAME` disabling +Pyright for the run. The Rust `CrashLoopBreaker` does not catch this — it operates at the +plugin-process level, not the Pyright-subprocess level. + +**Failure mode B:** `_disabled = True` is also set on `FINDING_PYRIGHT_UNAVAILABLE` / +`FINDING_PYRIGHT_INSTALL_FAILURE` (`pyright_session.py:620, 628, 646, 660, 670`). An environment +where `pyright-langserver` is absent will call `shutil.which` and emit a redundant not-found +finding on every 25-file boundary for the entire run. + +**Fix direction:** Promote `_disabled` and `_restart_count` from `PyrightSession` instance state +to `ServerState`, so the 25-file restart doesn't reset them. Five lines. + +--- + +## Confidence Assessment + +| Finding | Confidence | Basis | +|---------|------------|-------| +| `25` has no empirical basis in commit history | High | Full diff of `68b719c` verified; no measurement cited | +| Five cohesion groups in `PyrightSession` | High | Full method map from `grep -nE` on all 1,406 lines | +| Wire-contract constants coupled to Rust | High | Cross-referenced `server.py:48`, `pyright_session.py:43`, catalog entry for `clarion-core` | +| Interaction failure modes A and B | High | `_restart_count`/`_disabled` instance-scoped at `pyright_session.py:158–159`; instance destroyed at `server.py:217–219`; `_disabled` set at lines 620, 628, 646, 660, 670 | +| Pyright RSS growth mechanism | Moderate | Standard Node.js LSP behaviour; no heap profile of this version | + +## Risk Assessment + +- **Interaction fix** — Low risk, Easy revert. Fixing it changes observable finding counts in + existing tests (`test_pyright_session.py`); update expected counts accordingly. +- **Wire-contract constants** — High severity if they drift from Rust; Low likelihood today. + Add `# NOTE: must match clarion-core/…` inline comments before the next Rust-side refactor. +- **Moving tunables to config** — Wait for the WP6 config-schema design; premature promotion + produces unstable YAML key names. + +## Information Gaps + +1. **Pyright RSS profile on `elspeth`** — needed to replace `25` with an evidence-based bound. +2. **Characterization test for cross-session restart behaviour** — failure mode A has no dedicated + test asserting the reset behaviour, making a fix unverifiable without one. +3. **WP6 timeline** — "WP6" is named in `breaker.rs:7` but its current status is not visible + from source. + +## Caveats + +- `extractor.py` (918 LOC) is the second-largest Python file and was not audited here; it may + carry its own cohesion debt. +- Q2 split recommendation is structural, not test-driven. Characterization tests for + `resolve_calls` / `resolve_references` / `close` must precede any extract. diff --git a/docs/arch-analysis-2026-05-22-1924/temp/answer-quality-engineer.md b/docs/arch-analysis-2026-05-22-1924/temp/answer-quality-engineer.md new file mode 100644 index 00000000..bfc61db6 --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/temp/answer-quality-engineer.md @@ -0,0 +1,102 @@ +# Quality-Engineering View on the Five Open Questions + +**Date:** 2026-05-23 +**Source evidence:** `04-final-report.md` §6/#11, §8; `02-subsystem-catalog.md` §1 (clarion-core), §4 (clarion-mcp); all source files and test files cited below read directly. + +--- + +## Q1: Why 25 files per pyright restart? Is there a test that catches a regression in pyright memory behavior? + +The constant is `MAX_FILES_PER_PYRIGHT_SESSION = 25` at `plugins/python/src/clarion_plugin_python/server.py:49`. + +One test exists: `test_analyze_file_restarts_pyright_after_file_budget` (`plugins/python/tests/test_server.py:396`). It monkeypatches the constant to `2`, uses a fake `PyrightSession`, and asserts the state-machine transition: the session is closed and `state.pyright_files_since_restart` resets to `0`. That test catches a **mechanical regression in the recycle loop** — it does not and cannot validate that 25 is the right number. The fake session carries no memory state. + +**Missing test:** A soak/memory-growth integration test that runs a real `PyrightSession` against 50+ small Python files, samples the pyright subprocess RSS at each session boundary, and asserts that the process never exceeds a threshold before recycling. Without this, the value 25 is empirical lore that a future committer will treat as arbitrary and reduce to 10 or raise to 100 with no regression net. If the motivation is memory growth in pyright's Node heap, that is the test that makes the number legible. + +--- + +## Q2: Which monolith has the most asymmetric test coverage for its change risk? Where is the highest-priority test gap? + +The four monoliths compete, but the answer is `clarion-cli/src/analyze.rs` (`run_with_options`, 570 lines, 13 ordered phases). + +The other three are better served than they look: + +- `host.rs` (2,935 LOC) has extensive inline coverage: T1, T5 (path-escape), T6 (breaker trip), T8a–e (oversize fields), T9 (cap exceeded), T9b (stderr drain), T10–T11 (manifest security), plus the 876-LOC `mock.rs` enabling in-process pipeline testing without subprocesses. +- `llm_provider.rs` (2,467 LOC) has 22 inline tests across all four provider variants, including fake-subprocess tests for `CodexCliProvider` and `ClaudeCliProvider`. +- `mcp/lib.rs` (4,703 LOC) has a `tests/storage_tools.rs` (2,200 LOC) that covers all 19 tool dispatch paths. + +`analyze.rs` has no such seam. Its 13 phases form a strict sequential gate: orphan recovery → secret scan → `BeginRun` → plugin loop → entity ingestion → clustering → `CommitRun`. The whole pipeline is tested only end-to-end by `wp1_e2e.rs` and `wp2_e2e.rs` which spin up a real plugin and a real database. There is no per-phase test seam — you cannot, for example, inject a crash after `BeginRun` but before entity ingestion and assert the run row ends in `failed`, or verify that `SoftFail` vs. `HardFail` branching produces the correct `runs.status` without running the entire binary. + +**Highest-priority missing test:** an integration test against `run_with_options` (or a factored helper it calls) that injects a writer-actor failure mid-run and asserts `CommitRun(Failed)` vs. `FailRun` semantics. This is the test that would unblock any refactor of the 570-line function because it establishes what each phase transition must guarantee independently of the others. + +--- + +## Q3: Are the OpenRouter / Claude-CLI / Codex-CLI providers tested? What gap would a clarion-llm extraction surface? + +All four providers have tests. `CodexCliProvider` and `ClaudeCliProvider` use fake bash scripts at `llm_provider.rs:1853+` and `2087+`. `OpenRouterProvider` is exercised with a real in-process TCP listener at `clarion-mcp/tests/storage_tools.rs:1237`. `RecordingProvider` is used throughout. + +The gap is not per-provider — it is **trait-contract uniformity across providers**. No test runs the same `LlmRequest` through all four `LlmProvider` implementations and asserts: +- Identical `LlmResponse` field shapes (particularly `model_id` passthrough and token count presence). +- Consistent timeout behavior (both CLI providers have reaper threads; OpenRouter uses `reqwest` timeouts — no test verifies they degrade identically). +- Ring-buffer overflow in CLI stdout (the bounded ring in `ClaudeCliProvider`/`CodexCliProvider` is untested at capacity). + +If `clarion-llm` is extracted as a crate, the extraction would immediately require a shared integration test fixture that exercises the trait contract across all four providers against a common suite. That fixture does not currently exist; the extraction would surface its absence as a compilation-time discovery of test gaps rather than a runtime one. + +--- + +## Q4: What test would catch a cross-version DB collision? Does it exist? + +Adding `PRAGMA application_id` and `PRAGMA user_version` to `clarion-storage` is one change in `pragma.rs`. The tests to validate that change do not exist. + +The specific missing tests in `crates/clarion-storage/tests/schema_apply.rs`: + +1. **`open_refuses_db_with_foreign_application_id`** — create a SQLite file with `PRAGMA application_id = 0x0F11BEEF` (a hypothetical Filigree or Wardline value), call `apply_write_pragmas` + `apply_migrations`, assert it returns an error before touching schema. Without this test, adding the PRAGMA has no regression net: a future relaxation of the check would go undetected. + +2. **`open_refuses_db_from_future_user_version`** — create a DB with `PRAGMA user_version = 999`, call `apply_migrations`, assert it refuses to downgrade. This is the test that catches two Clarion versions sharing a DB by accident — the scenario that prompted Q4. + +Neither test exists today (`schema_apply.rs` has 10+ tests, zero touch `application_id` or `user_version`). The retroactive risk named in the report is real: once installed DBs exist in the wild, the PRAGMA values become a wire contract that cannot be set without a migration, so the window for adding this protection cheaply is before first external deployment. + +--- + +## Q5: Which of the 11 limit constants are covered by tests that would catch a value change being wrong? + +| Constant | Location | Test status | Test that catches a wrong value | +|---|---|---|---| +| `MAX_PROTOCOL_ERROR_FIELD_BYTES` | `protocol.rs:245` | **Tested** | `protocol.rs` inline: `huge` string asserts truncation at boundary | +| `MAX_ENTITY_FIELD_BYTES` | `host.rs:66` | **Tested** | `host.rs` T8a–d: oversize qualified name / file path / id / kind | +| `MAX_ENTITY_EXTRA_BYTES` | `host.rs:82` | **Tested** | `host.rs` T8e: oversize extra map is dropped with finding | +| `MAX_HEADER_LINE_BYTES` | `transport.rs:45` | **Tested** | `transport.rs:542`: oversize header returns `MalformedHeader` | +| `ContentLengthCeiling::DEFAULT` | `limits.rs` | **Tested** | `limits.rs:370,387`: default value pinned; `host.rs:2471`: surfaces through pipeline | +| `EntityCountCap::DEFAULT_MAX` | `limits.rs` | **Tested** | `limits.rs:432`: cap at 500,000 pinned; `host.rs:2316`: T9 kills plugin on overflow | +| `PYRIGHT_MAX_NPROC` | `host.rs` | **Tested** | `host.rs:1380`: `pyright_runtime_raises_process_ceiling_for_language_server` | +| `DEFAULT_MAX_RSS_MIB` | `limits.rs:261` | **Weak** | `limits.rs:561,569`: only verifies `apply_prlimit_as` does not panic, not that the child is actually memory-constrained | +| `DEFAULT_MAX_NOFILE` | `limits.rs:281` | **Weak** | No behavioral test; constant referenced by `host.rs:576` in `pre_exec`, but `host_subprocess.rs` has no test that opens > 256 file descriptors and observes enforcement | +| `DEFAULT_MAX_NPROC` | `limits.rs:289` | **Weak** | Same gap as `NOFILE` — `pre_exec` path covered only by code review | +| `MAX_UNRESOLVED_CALLEE_EXPR_BYTES` | `host.rs:70` | **Untested** | Used at `host.rs:267` to drop oversized call sites, but no inline or integration test constructs a callee expression > 512 bytes and asserts the drop | +| `STDERR_TAIL_BYTES` | `host.rs:445` | **Partially tested** | `T9b` verifies drain thread is attached and `stderr_tail()` returns `Some(_)`, but does not overflow the ring to test the drop-from-front eviction | + +**Highest-value missing test:** `host_subprocess.rs::rlimit_as_actually_enforced_on_child` — spawn a plugin whose manifest requests a pyright-capable runtime (triggering the larger `RLIMIT_AS` path), allocate more than `DEFAULT_MAX_RSS_MIB` inside the child, and assert the child is killed with `CLA-INFRA-PLUGIN-OOM-KILLED`. Currently the `RLIMIT_AS` enforcement path in `host.rs:569–586` is validated only by code review and the `pre_exec` non-panic test. A value change to `DEFAULT_MAX_RSS_MIB` has no regression net beyond Clippy and the CI build. + +--- + +## Confidence Assessment + +**High** on Q1, Q4, Q5 — directly verifiable from source and test files read end-to-end. +**High** on Q3 — all provider test locations confirmed; the gap named is structural (trait-contract suite), not a coverage metric claim. +**Medium-High** on Q2 — the claim that `analyze.rs` has no per-phase seam is based on reading the function signature and the test files, not on reading all 570 lines of the function body to exhaustion. + +## Risk Assessment + +The highest-risk gap is Q5's `DEFAULT_MAX_RSS_MIB` / `DEFAULT_MAX_NOFILE` / `DEFAULT_MAX_NPROC` cluster: these are security-enforcement constants in the plugin isolation layer, and their behavioral coverage is "does not panic." A change that accidentally weakens plugin memory limits would not be caught in CI. + +The second-highest is Q4: the `application_id`/`user_version` window closes permanently once installed databases exist in the field. The cost of adding it now is one PRAGMA + two tests. The cost after first external deployment is a migration + cross-version compatibility matrix. + +## Information Gaps + +- Actual pyright process RSS growth curve is not observable from test files — Q1's "empirical vs. conservative bound" distinction cannot be resolved from code alone. +- `STDERR_TAIL_BYTES` ring eviction is not confirmed to be tested; `T9b` covers attachment, not overflow. An adversarial test would need to send > 64 KiB to stderr and verify the tail contains only the last 64 KiB. + +## Caveats + +- The Q2 characterization of `analyze.rs` as the most asymmetric file depends on the claim that `mcp/lib.rs` has adequate per-tool coverage. That claim rests on the `storage_tools.rs` test file being broad-coverage — it was confirmed by sampling, not exhaustive read. +- The Q3 ring-buffer-overflow gap is inferred from reading the provider source and finding no test that exercises it, not from a coverage tool. diff --git a/docs/arch-analysis-2026-05-22-1924/temp/answer-security-engineer.md b/docs/arch-analysis-2026-05-22-1924/temp/answer-security-engineer.md new file mode 100644 index 00000000..759f3820 --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/temp/answer-security-engineer.md @@ -0,0 +1,65 @@ +# Security-architecture view on the five §8 open questions + +**Role**: Threat analyst (STRIDE + attack-surface focus). +**Question**: Which of the five §8 questions have *security* answers, and which are operational/architectural neutral? + +## Verdict matrix + +| Q | Topic | Security-relevant? | Dominant STRIDE category | +|---|-------|--------------------|--------------------------| +| Q1 | 25-file pyright restart | **Marginal** — DoS-adjacent, but the constant is a *mitigation*, not a threat surface | D | +| Q2 | Monolith refactor (host.rs 2935 LOC) | **Yes — high-stakes** | T, E, R | +| Q3 | `clarion-llm` split | **Yes — moderate** | T, I (supply-chain + outbound trust) | +| Q4 | `application_id` / `user_version` | **Yes — but mostly integrity, not confidentiality** | T (and a sliver of S in multi-tenant Loom) | +| Q5 | Hardcoded limits (11+) | **Yes — this is the question with the most teeth** | D, E | + +## Per-question analysis + +**Q1 — Pyright 25-file restart.** The constant is the *fix* for memory growth, not the surface. The real question is what happens if a pathological corpus drives Pyright RSS above the host's tolerance *within* a 25-file window. Today the answer is "the `RLIMIT_AS` 2 GiB ceiling on the plugin child kills it, the supervisor emits `FINDING_OOM_KILLED`, the run aborts cleanly." So Q1 has a security answer only in the sense that **the 25-file constant is itself a tuned defense parameter**, which folds into Q5. Standalone, neutral. + +**Q2 — Monolith refactor.** This is the most security-load-bearing answer in the set. `host.rs` (2935 LOC) concentrates the four-stage pipeline, stderr drain, child reaping, breaker wiring, jail-check sequencing, and supervisor signal handling. The threat is **Tampering with security mechanisms via accidental refactor** — STRIDE-T against the supervisor itself, with Elevation as the eventual consequence (a plugin that should have been killed keeps running). Concrete failure modes a careless extract-method introduces: + +- Breaker `record_escape` called on wrong code path → path-escape budget effectively infinite. +- Stderr drain detached from child-reap → zombie + log-channel DoS. +- `pre_exec` ordering broken (setrlimit applied after exec) → RLIMIT_AS no longer applies to the child. +- Jail-check moved *after* a downstream consumer that opens the path → revives the TOCTOU window (see below). + +The risk is not that refactoring is impossible; it is that **the file has no test that says "the supervisor still kills the plugin under the breaker policy after this refactor."** Recommendation: before splitting `host.rs`, add a property-style integration test that asserts each ADR-021 §2a–§2d invariant survives module boundary changes. Then refactor. + +**Q3 — `clarion-llm` split.** Currently `reqwest` (and TLS, and DNS) is reachable from the same crate that supervises plugins. **Outbound HTTP from inside the plugin-supervisor crate is a meaningful trust-surface widening** — a CVE in `reqwest`/`hyper`/`rustls` becomes a CVE in the process that holds the jail. Splitting `clarion-llm` into its own crate (and ideally its own process) is a defense-in-depth win: blast radius of an outbound-HTTP-stack exploit no longer reaches plugin supervision. STRIDE-T (compromised LLM client tampering with supervisor state) and STRIDE-I (LLM client exfiltrating jail-passed paths via DNS/SNI side channels) both shrink. **This question has a security answer: yes, split it, and the security argument is independent of the architectural one.** + +**Q4 — `application_id` / `user_version`.** Mostly an integrity question (`user_version` for migrations) — but `application_id` has a **specific Loom-federation security value**: it lets a sibling tool refuse to operate on a database that isn't Clarion's. Without it, a misconfigured Filigree pointed at a stale or hostile `.clarion/state.db` cannot detect the substitution at the file-format layer. STRIDE-S (spoofing a Clarion DB at the federation read boundary). The cross-tool collision-detection benefit is real and cheap; set both, and have the storage layer refuse to open a DB whose `application_id` does not match. Not P0, but trivially worth doing. + +**Q5 — Hardcoded limits.** This is the question with the most security weight. The 11+ values include the entity cap (500k), the Content-Length ceiling (8 MiB), the path-escape breaker threshold (10/60s), RLIMIT_AS (2 GiB), RLIMIT_NOFILE (256), RLIMIT_NPROC (32), HTTP body limit (16 KiB), concurrency limit (64), request timeout (10s), batch maxima (256/1000), and the pyright restart. **Recompile-to-tune is a security posture stance**: it means an operator under active adversarial-plugin pressure cannot tighten the breaker threshold from 10 to 3 without a rebuild. ADR-021 §2b already nods at this by mentioning a "configuration-surface" floor that isn't yet plumbed. The honest answer: **at v1.0 these are deliberately frozen so the security policy is uniform across deployments; post-1.0, the path-escape breaker threshold, the entity cap, and the RLIMIT_AS ceiling should become operator-tunable with hard floors enforced at config-load time.** The HTTP-body and concurrency limits can stay compiled-in (operational, not adversarial). + +## TOCTOU claim + +The catalog's "latent because current consumer doesn't open files" claim is **correct as of today** and **explicitly documented in `jail.rs` lines 67–72**. The function returns a `PathBuf` that is a "membership proof at canonicalization time, not a durable file handle." A future consumer that calls `jail(...)` then `std::fs::open(returned_path)` opens the canonical path — but between canonicalize and open, an attacker with write access to a directory on the canonical path (e.g. a plugin that can mutate its own workspace) can replace a path segment with a symlink to `/etc/shadow`. The next open follows the new symlink. **Concrete exploit**: plugin returns `/staging/report.txt`, supervisor jail-checks it OK, plugin races a `rename` to swap `staging` for a symlink to `/etc`, supervisor opens `report.txt` and reads `/etc/passwd`. The mitigation is the `openat`-anchored-to-pinned-root strategy the docstring recommends; do this *before* any code path opens a jail-returned path. + +## Confidence Assessment + +**High** on Q2 (monolith risk is concrete and grounded in the source), Q3 (trust-surface argument is standard), and the TOCTOU exploit chain (the file documents the gap itself). +**Medium** on Q4 (the Loom federation S-axis is real but I haven't audited every sibling's DB-open path) and Q5 (specific recommendations on *which* limits to make tunable are judgment calls, not derivations). +**Low** on Q1 — I'm 60% confident it is "neutral folded into Q5"; could be argued the 25-file number is itself an adversary-tunable surface if a malicious corpus could be shaped to exhaust the host before the restart fires. + +## Risk Assessment + +If these answers are wrong: Q2 is the only one where being wrong is dangerous — recommending a refactor without first writing invariant-preserving tests could ship a broken jail. Q3/Q4/Q5 errors are forward-only (failure to *add* defense-in-depth, not removal of existing defense). Q1 has no downside either way. + +## Information Gaps + +- No audit of who opens jail-returned paths today; the claim "no current consumer" is from the docstring, not from a callsite sweep. +- Have not read every limit's actual call site to confirm "recompile-to-tune" applies uniformly. ADR-021 §2b suggests configuration-surface plumbing was contemplated but deferred. +- The `clarion-llm` outbound-HTTP threat model assumes the crate ships TLS — not verified. + +## Caveats + +- This is a STRIDE-shaped read, not a full threat model with attack trees. The Q2 monolith risk in particular deserves its own attack tree (root: "plugin escapes supervisor invariants via refactor regression"). +- "Defense-in-depth" arguments are inherently judgment calls; a smaller team may reasonably prefer the operational simplicity of one HTTP stack over the security win of splitting `clarion-llm`. + +## Relevant paths + +- `/home/john/clarion/crates/clarion-core/src/plugin/jail.rs` (TOCTOU documented at lines 67–72) +- `/home/john/clarion/crates/clarion-core/src/plugin/limits.rs` (constants at lines 71, 142, 209–211, 261, 281, 289) +- `/home/john/clarion/crates/clarion-core/src/plugin/host.rs` (the 2935 LOC monolith — Q2) +- `/home/john/clarion/crates/clarion-cli/src/http_read.rs` (HMAC > bearer > loopback chain at lines 392–419; unauthenticated `_capabilities` at line 372; panic-on-unenumerated middleware error at lines 547–557; loopback-no-token warning at lines 199–252) diff --git a/docs/arch-analysis-2026-05-22-1924/temp/answer-solution-architect.md b/docs/arch-analysis-2026-05-22-1924/temp/answer-solution-architect.md new file mode 100644 index 00000000..dd23d9a0 --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/temp/answer-solution-architect.md @@ -0,0 +1,94 @@ +# Architect Answers — Five Open Questions from §8 + +Scope: the five §8 open questions in `04-final-report.md`. Source: catalog, ADRs, `gap-register.md`, the actual code paths, and the filigree open-issues queue. No diplomatic softening. + +--- + +## 1. Why `MAX_FILES_PER_PYRIGHT_SESSION = 25`? + +**Recommendation:** Keep the value, label it. The constant is an unjustified-in-code heuristic introduced in commit `68b719c` ("Bound Pyright dogfood analysis", 2026-05-20) with no commit-message or in-file rationale (`plugins/python/src/clarion_plugin_python/server.py:49`). The diff added recycling but cited no measurement. Interpreted in context — "dogfood analysis" implies it surfaced during operator-driven runs against Clarion itself — this is an **empirical conservative bound** chosen to stop observed pyright RSS growth, not a derived figure. The 25-file window is short enough to be safe and long enough to amortise the ~150 ms `pyright_init_ms` measured in `sprint-2/b4-gate-results.md:19`. + +**Risk of inaction (no documentation, no measurement):** The next operator who hits pyright OOM on a heavy-import file at file 24 has nothing to tune and no way to know whether 25 is conservative or aggressive. Worse, the constant is divorced from `MAX_PYRIGHT_RESTARTS_PER_RUN` (`pyright_session.py:142`) — server-side recycling at 25 files can keep tripping session-side restart caps, and vice versa, with no centralised policy. + +**Risk of acting now:** Approximately zero. A one-line comment and a TODO citing `pyright_files_since_restart` plus the session-side restart cap costs nothing. + +**Evidence that would change the call:** A measurement run on the elspeth corpus (~425k LOC) sampling pyright RSS per `analyze_file` call. If the curve plateaus before 25, the recycle is wasted re-init cost (~150 ms × N); if it climbs steeply past 15, 25 is already too lax. Either outcome justifies a measured override and config surface. + +--- + +## 2. Post-1.0 plan for the four monolith files + +**Recommendation:** Defer all four splits, with one named trigger per file. "No split until it actively impedes a change" is **defensible at this scale** — ~50K LOC of first-party code, 7 subsystems, no inter-crate cycles, a single maintainer, ~57% test/source ratio. The change-amplification cost is real but bounded, and splits done without a forcing function tend to re-coalesce or fragment along the wrong axis. There is no roadmap for these splits in any ADR, sprint doc, or filigree issue — that absence is itself a defensible posture for 1.0, not a planning gap. + +The triggers that should fire a split (per file): + +- **`clarion-mcp/src/lib.rs` (4,703 LOC):** trigger is adding a 20th tool, or any concurrent-tool-execution work. The tool registry shape (`lib.rs:56-257`) already wants to be a per-category module set. Until then it reads as one file with clearly named sections. +- **`clarion-core/src/plugin/host.rs` (2,935 LOC):** trigger is adding a fifth enforcement layer or changing the four-stage pipeline ordering (`host.rs:866-975`). A natural pipeline-axis / lifecycle-axis / IO-axis split exists but each axis pulls in `mock.rs` (876 LOC, also flagged) and the cost of getting the seams wrong is high. +- **`clarion-cli/src/analyze.rs` (2,549 LOC; `run_with_options` 570 lines):** trigger is the 14th phase. The current 13-phase linearisation is legible exactly because it is one function with one error scope. Extracting per-phase helpers without first naming the contract between phases (entity buffer ownership, breaker tick sites, partial-results semantics) just hides the linearity behind call indirection. The catalog already flagged the in-memory entity buffer (`02-subsystem-catalog.md` line 276) as the load-bearing latent risk — that is the issue worth fixing, not the line count. +- **`clarion-core/src/llm_provider.rs` (2,467 LOC):** see Q3 — this one has a stronger argument for splitting **out of `clarion-core`**, not within it. + +**Risk of inaction:** Change-amplification per touch grows monotonically. Every new MCP tool, every new enforcement layer, every new analyze phase makes the next refactor more expensive. At 6,000 LOC `mcp/lib.rs` will be genuinely hard to navigate. + +**Risk of acting now:** Premature splits chosen on the wrong axis, retest cost across all four files concurrently, and architectural drift while the splits are in-flight. Splitting `host.rs` mid-sprint while WP6 wires the config surface (see Q5) would be especially expensive. + +**Evidence that would change the call:** A concrete change request that touches 3+ of these files in the same PR, or a contributor onboarding that stalls on "where do I put new tool X". Either signal flips the calculus. + +--- + +## 3. Will `clarion-llm` become a crate? + +**Recommendation:** Yes, and it is **already named in `docs/clarion/1.0/detailed-design.md:1745`** as one of the intended workspaces (`clarion-core`, `clarion-cli`, `clarion-plugin-protocol`, `clarion-api`, `clarion-llm`). The current placement of `llm_provider.rs` in `clarion-core` is an **expedient, not a deliberate boundary call**: the detailed design says where it goes and the code does not yet match. `clarion-core/lib.rs:1` advertises the crate as owning "domain types, identifiers, and provider traits" — the OpenRouter `reqwest` transport and the two CLI-subprocess providers (Claude, Codex shellouts) are neither. + +**Risk of inaction:** `clarion-core` pins `reqwest` (with rustls) and CLI-subprocess machinery into the same crate that supervises plugins. That widens the trust surface of the host runtime — a malicious dependency in the LLM HTTP stack lives inside the crate that handles `pre_exec` `setrlimit` for plugin children. The blast radius argument alone is sufficient. Secondary: every new LLM provider drives a recompile of `clarion-core`, which forces recompile of every downstream crate. + +**Risk of acting now:** A pre-1.0 crate split during release cut adds churn. ADR-030 narrowed WP6 to a single MCP `summary(id)` tool — the LLM surface is at its minimum scope right now, which is paradoxically the **best** time to split (small surface = small move) but also the time when "ship the tag" pressure resists any refactor. + +**Evidence that would change the call:** None substantive. The recommendation here matches the documented intent. The only open question is timing — pre-`v1.1.0` or after. + +--- + +## 4. `application_id` / `user_version` PRAGMAs + +**Recommendation:** Add both, now. This is already filed as **`clarion-f2a984fd6d` — `[v1.0 blocker] Set PRAGMA application_id on writer open`** (P1, ready, blocks two other issues), with a fix specified in `docs/implementation/v1.0-tag-cut/gap-register.md` STO-02: `PRAGMA application_id = 0x434C524E` ("CLRN") and assert on open. There is nothing to decide here. The architect already decided; the issue is open and ready. + +`user_version` should also be set, even though the application-level `schema_migrations` table (`schema.rs:17-91`) tracks migration state. The two solve different problems: `application_id` identifies the file as Clarion's; `user_version` provides a fast PRAGMA-level read of "what migration level is this" without opening the table. The current model fails confusingly when a non-Clarion sqlite file happens to live at `.clarion/clarion.db` — `apply_migrations` will create tables in someone else's database. `application_id` mismatch turns that into a hard fail at open. + +**Risk of inaction:** Bounded but real. Today's only victim is the operator who points `clarion install --path` at a directory whose `.clarion/clarion.db` is from a sibling tool, or a future v2.0 Clarion that opens a v1.0 file. Both are tractable until installed DBs exist in the wild. + +**Risk of acting now:** Effectively zero. PRAGMA additions don't break readers; the `apply_write_pragmas` site already exists. + +**Evidence that would change the call:** None. + +--- + +## 5. Operational tuning roadmap (WP6 / 11+ hardcoded limits / 25-file pyright / 256/50 batch cadence) + +**Recommendation:** Ship 1.0 with the limits hardcoded. Land the config surface in WP6 post-1.0 as **one ADR-021-aligned change** rather than dripping per-constant overrides. + +ADR-021 §4 already names the config keys for four of the eleven limits — `plugin_limits.max_frame_bytes` (floor 1 MiB), `plugin_limits.max_records_per_run` (floor 10,000), `plugin_limits.max_rss_mib` (floor 512 MiB), and named `expected_max_rss_mb` in the manifest. These are **promised** by an Accepted ADR and **not implemented**. `breaker.rs:7`'s comment names WP6 as the home for this surface. The `2026-04-19-wp2-tasks-4-to-9-handoff.md:203` line says the crash-loop parameters are "hard-coded for Sprint 1; config surface deferred to WP6". WP6's v0.1 scope was narrowed by ADR-030 to the on-demand `summary(id)` MCP tool — **the operator-tunables work was not folded into the narrowed WP6 scope and is currently un-homed**. + +The catalog's eleven-limit list (`02-subsystem-catalog.md` line 97) plus the 25-file pyright restart plus the writer's 256-cap mpsc and 50-write batch cadence (`writer.rs:35, 38, 813`) plus the per-batch HTTP cap of 256 queries (pinned on the wire by ADR-034, so **not** operator-tunable) all want different homes: + +- **Plugin host enforcement (frame ceiling, entity cap, RSS, NOFILE, NPROC, field bytes, stderr tail, header bytes, callee-expr bytes):** belong in `clarion.yaml:plugin_limits.*` per ADR-021. Eight of the eleven. +- **Writer-actor cadence (256 mpsc, 50-write batch):** belong in `clarion.yaml:storage.*` per the same shape ADR-011 already hints at. +- **Pyright session recycle (25):** belongs in `plugin.toml` (plugin-owned policy) or per-language plugin config, not core-side. +- **Crash-loop window (>3 / 60 s):** belongs in `clarion.yaml:plugin_limits.crash_loop` per `handoff:203`. +- **Batch HTTP cap (256):** pinned on the wire, not tunable (ADR-034 §3). + +**Risk of inaction:** ADR-021's "configurable" claim becomes false advertising on the operator's first read. The elspeth-scale test (425k LOC Python) is the named first customer; if RSS or entity cap defaults are wrong, the operator has no remediation short of patching source and rebuilding. Every limit is a recompile today; the catalog is correct that "operator can tune" is aspirational. + +**Risk of acting now (pre-1.0):** Scope creep on a release cut. The shape of the config block is contestable and locking the wrong shape into `clarion.yaml` before the first real elspeth run is worse than landing nothing. WP6 is a post-1.0 ADR-named home; **that is the correct place**. + +**Evidence that would change the call:** Either (a) elspeth-scale dogfood data showing any single hardcoded limit is wrong by an order of magnitude — that constant goes to WP6 priority; or (b) a v1.1 issue triaged from the field where the operator could not work around the limit without a custom build. Currently neither exists in the filigree queue. + +--- + +## Confidence, risk, gaps, caveats + +**Confidence:** High for Q1 (commit message confirms post-hoc bound, no design-doc rationale), Q3 (`detailed-design.md:1745` names the crate), Q4 (already a v1.0-blocker issue with a documented fix). Medium-High for Q2 (no roadmap exists; defensibility argument is judgement, not evidence). Medium for Q5 (ADR-021/ADR-030/handoff fragments triangulate the WP6 home but no single doc spells out the full eleven-constant migration plan). + +**Risk assessment:** The only standing High-risk item in the §8 set is the LLM transport living in `clarion-core` (Q3), and only because of the blast-radius coupling to the plugin supervisor. Q4 is High-severity but Low-risk because the fix is filed and ready. Q1, Q2, Q5 are Medium and Medium-Low respectively. + +**Information gaps:** (a) No measurement curve for pyright RSS vs. files-processed exists in the repo; Q1's "conservative bound" interpretation is inferred from the commit title and the 150 ms init cost in `b4-gate-results.md`, not directly attested. (b) No post-1.0 sprint plan exists for the four monolith splits; Q2's "defer with named trigger" recommendation is the architect's call, not a documented decision. (c) The WP6 config-surface scope after ADR-030's narrowing is not written down anywhere — Q5 reconstructs it from fragments. + +**Caveats:** All five questions ask for stances the source code cannot reveal. Each recommendation here is contingent on the design-doc intent the user wrote down (ADR-021, ADR-030, detailed-design.md §workspaces). If those intents have shifted in conversation since the docs were last updated, the recommendations shift accordingly. The strongest signal in this set is Q4 — the gap-register and the filed P1 issue make that one not a judgement call. diff --git a/docs/arch-analysis-2026-05-22-1924/temp/answer-systems-thinker.md b/docs/arch-analysis-2026-05-22-1924/temp/answer-systems-thinker.md new file mode 100644 index 00000000..1a6170d9 --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/temp/answer-systems-thinker.md @@ -0,0 +1,70 @@ +# Leverage Analysis: The Five Open Questions as One System Pattern + +## The pattern + +The five questions look like five concerns. They are one. Each is a **missing-feedback-loop** symptom: a place where operational reality has no path back to the artifact that would change behavior. They wear "parameter" clothing (Level 12) but live at **Level 6 (information flows)** and **Level 5 (rules)**. + +The smoking gun is named in the source itself. `crates/clarion-core/src/plugin/breaker.rs:7` flags that operator-tunable limits land "in WP6"; `crates/clarion-cli/src/analyze.rs:74` carries `#[allow(clippy::too_many_lines)]` — an alarm explicitly disabled. The architecture has placeholders where its feedback loops should be. + +## Archetype: Drift to Low Performance + +Meadows' "drift to low performance" fits cleanly. Each individual deferral is locally rational ("ship 1.0; tune later"; "don't split host.rs mid-sprint"; "PRAGMAs are post-1.0"). The standard erodes silently because there is **no countervailing signal** pushing the other way. The clippy allow is the standard-lowering act made literal in code. + +Concretely, each question is the same shape: + +| Q | Surface symptom | Missing loop | +|---|---|---| +| 1 (`MAX_FILES_PER_PYRIGHT_SESSION=25`) | parameter with unknown basis | rationale → constant → retune trigger | +| 4 (`application_id`/`user_version`) | absent schema identity | DB collision → detection → action | +| 5 (11+ hardcoded limits) | every tunable is a recompile | production behavior → tuning surface | +| 2 (four monoliths) | 4,703-line `mcp/lib.rs`, etc. | file growth → back-pressure → split | +| 3 (`clarion-llm` in core) | `lib.rs:1` doc-comment contradicts contents | boundary statement → enforcement | + +§6.5 ("no `application_id`/`user_version` is a one-line change with no downside"), §6.8 ("every tunable is a recompile today"), and §8 Q5 ("WP6 is named in code comments — what is its current status?") all describe the same gap from different angles. The Python-plugin catalog quote — "every limit is a recompile" — is the cleanest single-line statement of it. + +## The highest-leverage intervention + +**Level 5 (rule), instantiated as one ADR: "Operational tuning discipline."** Every operational constant must declare: (a) a stated basis (empirical / safety-margin / contract-derived), (b) an operator override surface (`clarion.yaml` field or env var), (c) a retune trigger (the metric or finding subcode that should prompt revisiting it). Apply the same rule-shape — explicit budget + override + trigger — to file size and crate-boundary budgets. + +This single rule closes Q1, Q4, Q5 directly (every limit, including the 25-file restart, gains a recorded basis and a tuning surface; PRAGMA identity becomes a "schema identity" instance of the same rule) and structurally addresses Q2 and Q3 (file-LOC and "what belongs in this crate" become budgeted properties with a trigger to act, rather than aesthetic preferences competing with sprint load). + +This is **not** "add a config file." A config file without the rule decays back to hardcoded constants on the next sprint — that is exactly how Clarion got here. The rule is what creates the surface; the surface alone is a parameter intervention (Level 12) and parameter interventions to a drift-to-low-performance loop reset the constant without changing the slope. + +## What changes, concretely + +1. Author an ADR ("Operational discipline: declared basis + override + trigger for every limit"). Cite §6.8 + §8 Q5 + the `breaker.rs:7` comment as the originating evidence. Promote it to Accepted before any further hardcoded limit lands. +2. Apply the rule retroactively to the 11 limits in §6.8 plus `MAX_FILES_PER_PYRIGHT_SESSION` (Q1) and the writer-actor's 256 / 50 cadence constants. The artifact is a table in `detailed-design.md` keyed by constant name. +3. Add `application_id` + `user_version` (Q4) as the schema-identity instance of the same rule — basis stated, trigger ("DB opened with mismatched id → refuse"). +4. Adopt file-LOC and per-crate-doc-comment budgets with CI enforcement: `clippy::too_many_lines` is **not** allowed without an ADR-referenced waiver; `lib.rs:1` doc-comment violations are a `cargo deny`-style check. This closes Q2/Q3 by creating the missing back-pressure loop. + +## Feedback loops the architecture currently has vs. lacks + +**Has (strong):** the path-escape and crash-loop breakers (rolling-window → kill); the writer-actor invariant check (`parent_contains_mismatch` aborts the run); the cross-language fixture parity test (drift caught in CI). These are exemplary balancing loops at the *runtime* layer. + +**Lacks:** any equivalent loop at the *design-time* layer. The runtime has back-pressure; the architecture itself does not. File LOC grows, doc-comments drift from contents, limits accrete — and nothing ticks. + +## Confidence Assessment + +**High** that the five questions cluster as one missing-feedback-loop pattern; the `breaker.rs:7` WP6 reference and the §6.8 enumeration are explicit. **High** that Level 5 is the correct leverage point. **Medium** on the specific archetype label ("drift to low performance" vs. "shifting the burden" — both fit; I chose drift because the standard-lowering is visible in code, not just behavior). + +## Risk Assessment + +- **Over-bureaucratization:** an ADR that demands a basis for every constant could ossify into ceremony. Mitigation: the basis statement is one sentence; the trigger is one finding subcode. Anything more is the wrong shape. +- **Premature parameter exposure:** exposing all 11 limits as operator surface creates a support burden. Mitigation: the ADR allows "internal, no override" as a declared state — the discipline is *declaration*, not necessarily *exposure*. +- **CI friction on file-LOC budgets:** an aggressive cap on `lib.rs` blocks unrelated PRs. Mitigation: budgets start at current LOC + 10%; ratchet down per release. + +## Information Gaps + +- WP6's actual status — code comment is the only public reference seen during the from-scratch analysis (the analysis intentionally did not read design docs). +- Whether `MAX_FILES_PER_PYRIGHT_SESSION=25` has an empirical basis the catalog could not surface. +- Whether the file-LOC growth on `mcp/lib.rs` and `analyze.rs` is accelerating or has plateaued — no time-series. + +## Caveats + +The analysis intentionally did **not** consult `docs/clarion/**` or ADRs; if an Accepted ADR already addresses operational tuning discipline, the recommendation collapses to "promote and enforce the existing ADR," not "author a new one." The `breaker.rs:7` WP6 comment strongly suggests an unauthored plan, not an authored-but-unimplemented one, but the from-scratch method cannot confirm. + +## Sources + +- `04-final-report.md` §1 (operational-subtlety framing), §6.1–6.8 (the High/Medium smells), §8 Q1–Q5 (the five questions verbatim). +- `02-subsystem-catalog.md` `clarion-core` Concerns ("every limit is a recompile"); `clarion-storage` Concerns (no `application_id`/`user_version`); `clarion-cli` Concerns (`run_with_options` at 570 lines with `#[allow(clippy::too_many_lines)]`); `clarion-mcp` Concerns (4,703-LOC `lib.rs`). +- Source: `crates/clarion-core/src/plugin/breaker.rs:7` ("WP6"); `crates/clarion-cli/src/analyze.rs:74` (`#[allow(clippy::too_many_lines)]`); `crates/clarion-core/src/lib.rs:1` (crate doc-comment vs. contents). diff --git a/docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-cli.md b/docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-cli.md new file mode 100644 index 00000000..9ba311b0 --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-cli.md @@ -0,0 +1,101 @@ +## 3. clarion-cli + +**Location:** `crates/clarion-cli/` +**LOC:** ~6981 src across 13 `.rs` files (4 of them in `src/secret_scan/`); 5894 LOC tests across 6 integration test files. +**Crate type / role:** binary crate; produces the `clarion` executable (`[[bin]] name = "clarion"`, `Cargo.toml:12-14`). No `lib.rs`, no inbound Rust callers — this crate is a pure orchestrator that wires `clarion-core`, `clarion-mcp`, `clarion-scanner`, and `clarion-storage` into three subcommands. + +### Responsibility + +`clarion-cli` owns the operator-facing entry surface: the `clarion` binary, its three subcommands (`install`, `analyze`, `serve`), and all orchestration logic that sits *between* the user invoking the command and the lower-level libraries doing the work. It owns the `.clarion/` filesystem layout (`install.rs:20-100`, `instance.rs:10-86`), the `analyze` pipeline that walks the tree, gates on secrets, fans out to discovered plugins, persists results, and clusters modules into subsystems (`analyze.rs:75-645`), the `serve` supervisor that runs MCP stdio and the Axum HTTP read API as two threads sharing one `ReaderPool` (`serve.rs:20-227`), and the federation HTTP read surface itself (`http_read.rs:363-387` router; `:392-461` bearer + HMAC middleware). The federation-visible `/api/v1/files*` endpoints live in this crate, not in `clarion-mcp` — a noteworthy split. + +### Key components + +- `src/main.rs:1-78` — binary entry. Three-arm `match` on `cli::Command`; builds a multi-thread tokio runtime *only* for `analyze`. `dotenvy::dotenv()` is loaded for `install`/`serve` but deliberately **skipped for `analyze`** (`main.rs:23-25`) so `.env` contents flow through the secret scanner instead of being imported into plugin subprocess envs. `analyze` exits with `EX_CONFIG=78` on a misconfigured `--allow-unredacted-secrets`. +- `src/cli.rs:1-63` — clap derive definitions; three subcommands, five `analyze`-time flags. +- `src/install.rs:109-194` — initialises `.clarion/{clarion.db, config.json, .gitignore}` plus a `clarion.yaml` stub at project root. `populate_after_mkdir` is wrapped in a cleanup guard (`:142-153`) that `rm -rf`s `.clarion/` if any post-mkdir step fails so retry isn't blocked by "already exists". +- `src/analyze.rs:75-645` — `run_with_options`, the central analyze pipeline. Single async function (~570 LOC) that runs the entire flow described in §"Internal architecture" below. +- `src/serve.rs:20-227` — `serve::run` + `supervise_stdio_with_http`. Builds the LLM provider (`build_llm_provider:229-291`), opens a 16-conn `ReaderPool`, spawns the HTTP server thread, spawns the MCP stdio thread, then polls the stdio result channel with a 100 ms timeout while periodically checking HTTP health (`:176-202`). On `Arc::ptr_eq` mismatch between the HTTP thread's reported `ReaderPool` identity and the one held in `serve.rs`, `debug_assert!` fires (`:63-71`) — this catches a refactor that re-opens the pool inside `http_read::spawn`. +- `src/http_read.rs:146-260` — `spawn` / `spawn_with_env`; `:262-311` `run_http_read_server` (binds TCP, sends back ready signal with `ReaderPool` identity captured *after* the move into the runtime thread); `:363-387` `router`; `:392-461` `require_http_identity` + `require_hmac_identity`; `:692-1034` six request handlers (`get_file`, `post_files_batch`, `post_files_resolve`, `get_capabilities`, plus two cfg(test)-only fixtures referenced in `:329-353`). +- `src/clustering.rs:53-101` — `cluster_modules` entry point + `cluster_modules_with_algorithms` test seam; `:117-145` `leiden_communities` calling `xgraph::graph::algorithms::leiden_clustering`; `:170-224` `local_weighted_components` fallback; `:247-296` directed modularity score; `:103-115` `cluster_hash` = SHA-256 of sorted member IDs truncated to 12 hex chars. +- `src/secret_scan.rs:202-325` — `pre_ingest`; submodules `anchors.rs` (links scanner detections to plugin-emitted module/file entities), `baseline.rs` (loads `.clarion/secrets-baseline.yaml`), `files.rs` (extension/skip-dir walk + sidecar matcher for `.env`/`.env.*`/`*.env`), `findings.rs` (`PendingFinding` + `FindingRecord` shaping with `CLA-SEC-SECRET-DETECTED` / `CLA-SEC-UNREDACTED-SECRETS-ALLOWED` rule IDs). +- `src/run_lifecycle.rs:6-44` — two helpers: `recover_preexisting_running_runs` (raw `UPDATE runs SET status='failed' WHERE status='running'` on next start; not routed through the writer-actor because the writer isn't running yet) and `begin_run` (writer `BeginRun` wrapper). +- `src/instance.rs:44-86` — `InstanceId(Uuid)` newtype persisted to `.clarion/instance_id` with `O_CREAT|O_EXCL` (`mode = 0o600`) → `fsync` → `hard_link` for atomic publish, and `EEXIST`-on-link → read-existing-file race resolution. Federation contract requires a stable per-project ID; this is it. + +### Public interface (outbound) + +`clarion-cli` exposes no `pub` library API — it has no `lib.rs`. Its public surface is three things: + +- **The `clarion` binary CLI** — `clarion install [--force] [--path P]`, `clarion analyze [P] [--config C] [--allow-unredacted-secrets [--confirm-allow-unredacted-secrets TOKEN]] [--allow-no-plugins]`, `clarion serve [--path P] [--config C]` (`cli.rs:13-62`). Confirmation token literal is `"yes-i-understand"` (`secret_scan.rs:38`). +- **The HTTP read API** — Axum router at `http_read.rs:363-387`. Four production routes (one unprotected): + - `GET /api/v1/files?path=&language=` → `get_file`. Returns `{entity_id, content_hash, canonical_path, language}`; honours `If-None-Match` ETag (= `\"\"`) returning 304. 403 with `BRIEFING_BLOCKED` if entity has `briefing_blocked` property set by secret scan. Protected (bearer or HMAC). + - `POST /api/v1/files/batch` body `{queries:[{path,language},…]}` → returns `{resolved[], not_found[], briefing_blocked[], errors[]}`. Hard cap **256** queries/batch (`http_read.rs:608`), runs the whole batch in a single `with_reader` checkout. Protected. + - `POST /api/v1/files:resolve` body `{paths:[…]}` → returns `{results:[{path,response:{status,body}}]}` where `status ∈ {resolved,not_found,blocked,error}`. Hard cap **1000** paths (`http_read.rs:609`). Protected. + - `GET /api/v1/_capabilities` → `{registry_backend, file_registry, api_version, instance_id}`. **Unprotected by design** so siblings can probe pre-auth. + - Body cap **16 KiB** (`HTTP_BODY_LIMIT_BYTES`, `http_read.rs:610`). Tower stack (`:373-386`): `CatchPanicLayer` (panics → 500 envelope) → `HandleErrorLayer` (panics on unenumerated middleware errors; `:547-556`) → `TraceLayer` → 10 s `TimeoutLayer` → `RequestBodyLimitLayer` → `LoadShedLayer` → `ConcurrencyLimitLayer(64)`. Errors carry an envelope `{error, code}` with `code ∈ {INVALID_PATH, PATH_OUTSIDE_PROJECT, NOT_FOUND, BRIEFING_BLOCKED, UNAUTHENTICATED, STORAGE_ERROR, BATCH_TOO_LARGE, INTERNAL}` (`http_read.rs:590-601`). +- **`clarion serve`'s stdio MCP server** — owned by `clarion-mcp`. `clarion-cli` calls `clarion_mcp::serve_stdio_with_state_on_runtime` (`serve.rs:150`) and constructs the `ServerState` with the reader pool, optional summary-LLM writer + `LlmProvider`, and optional `FiligreeHttpClient` (`serve.rs:131-147`). + +### Dependencies + +- **Inbound (who calls this):** No Rust callers — verified by `grep -rn "clarion_cli" crates/` returning empty. This crate is invoked by humans/CI through the `clarion` binary and by `tests/e2e/*.sh` shell scripts. +- **Outbound (other Clarion crates):** `clarion-core` (`AcceptedEntity`/`AcceptedEdge`, `discover`, `PluginHost`, `CrashLoopBreaker`, `BriefingBlockReason`, `HostError`, `HostFinding`, LLM provider types), `clarion-mcp` (`ServerState`, `config::McpConfig`, `config::HttpReadConfig`, `filigree::FiligreeHttpClient`, `serve_stdio_with_state_on_runtime`, `select_provider_with_env`, `resolve_filigree_http_target`), `clarion-storage` (`Writer`, `WriterCmd`, `ReaderPool`, `module_dependency_edges`, `resolve_file_catalog_entry`, `CanonicalProjectPath`, `StorageError`, `pragma`, `schema`), `clarion-scanner` (`Scanner`, `Detection`, `Baseline`, `SuppressionResult`). +- **External crates of note:** `axum` 0.7 (HTTP), `tower`/`tower-http` (middleware), `tokio` (runtimes — multi-thread for analyze + HTTP, current-thread for MCP stdio), `clap` (CLI), `xgraph` (Leiden clustering), `ignore::WalkBuilder` (`.gitignore`-honouring tree walk in both `analyze.rs:2016-2026` and `secret_scan/files.rs`), `rusqlite` (one direct `Connection::open` in `analyze.rs:711` to read module IDs/edges during phase 3, bypassing the reader pool), `uuid`, `dotenvy`, `sha2` (HMAC, `cluster_hash`). +- **External services / processes:** + - **SQLite** at `/.clarion/clarion.db`, opened both via `ReaderPool::open` (`serve.rs:53`) and direct `Connection::open` (`run_lifecycle.rs:10`, `analyze.rs:711`, `install.rs:189`). + - **Plugin subprocesses** — spawned via `clarion_core::PluginHost::spawn` from `run_plugin_blocking` (`analyze.rs:1322`) on `tokio::task::spawn_blocking` workers. + - **TCP listener** for HTTP read API on `config.serve.http.bind` (default `127.0.0.1:9111` per `install.rs:74`); bound by `tokio::net::TcpListener::bind` inside the HTTP thread's runtime (`http_read.rs:279`). + - **LLM CLIs and OpenRouter** — wired via `build_llm_provider` (`serve.rs:229-291`); reaches `codex`/`claude` binaries and OpenRouter HTTPS. + +### Internal architecture + +**Three subcommands, three concurrency shapes.** `install` is fully synchronous (`install.rs:109`). `analyze` builds a multi-thread tokio runtime in `main.rs:36-38` and runs one big `async fn run_with_options` (`analyze.rs:75-645`); per-plugin work happens inside `tokio::task::spawn_blocking` (`analyze.rs:325-339`) using a **Pattern A buffering** strategy named in the module doc (`analyze.rs:6-8`): collect all entities + edges into in-memory `Vec`s inside the blocking task, return them to the async side, then drain into the writer-actor via `WriterCmd::InsertEntity`/`InsertEdge` one-at-a-time with `send_wait` ack-and-block (`analyze.rs:399-447`). `serve` is the most complex (`serve.rs`): a foreground supervisor thread (the process's original thread) polls a `std::sync::mpsc::Receiver` from the spawned `"clarion-mcp-stdio"` thread (current-thread tokio runtime, `:126-130`) while also `check_running`-polling the `"clarion-http-read"` thread (multi-thread tokio runtime, `http_read.rs:355-360`). Identity of the single shared `ReaderPool` is asserted by `Arc::ptr_eq` on the inner `Arc<()>` tag (`serve.rs:63-71`); the HTTP thread captures its identity **after** the pool has been moved into its runtime (`http_read.rs:272-296`), so a refactor that opened a fresh pool inside the HTTP thread would surface immediately rather than silently double-opening WAL. + +**Analyze pipeline ordering (`analyze.rs:75-645`, single function):** +1. Path canonicalisation + `.clarion/` existence check (`:82-91`). +2. Load `clarion.yaml`-derived `AnalyzeConfig` (clustering knobs only; LLM/serve config is `serve`-only) — `config.rs:16-29`. +3. **Run-row crash recovery** — `recover_preexisting_running_runs` flips any prior `status='running'` rows to `failed` (`:95-102`, `run_lifecycle.rs:6-26`). +4. **Spawn writer-actor** + mint run_id (`:105-113`). +5. **Plugin discovery** — `clarion_core::discover()` scans `$PATH` for `clarion-plugin-*` (`:116-135`). Empty-with-errors → FailRun + non-zero exit (`:142-174`); empty-without-errors → `RunStatus::SkippedNoPlugins` + non-zero exit unless `--allow-no-plugins` (`:176-223`). +6. **Build the wanted-extensions union** across plugin manifests (`:226-231`). +7. **Walk the tree** — `collect_source_files` with `ignore::WalkBuilder` honouring `.gitignore` family + a hard-coded skip-dir list (`:234`, `:2013-2065`). +8. **Pre-ingest secret scan** — `secret_scan::collect_scan_files` (broader walk that includes `.env` sidecars) → `secret_scan::pre_ingest` runs `clarion-scanner` patterns in **parallel via `thread::scope` with `available_parallelism()` workers** (`secret_scan.rs:333-382`), applies the baseline, classifies each file as `Clean`/`Blocked`/`Overridden` (`:237-243`). **The scan happens before `BeginRun` and before any plugin spawn**, so plugin processes never receive `.env` contents in their analyze stream — they receive `briefing_blocked` flags instead. +9. **`BeginRun`** (`:244`). +10. **Per-plugin loop** (`:275-470`): filter source files to the plugin's extensions; `spawn_blocking` `run_plugin_blocking` (`analyze.rs:1309-1456`) which (a) calls `PluginHost::spawn` for the JSON-RPC handshake, (b) iterates files calling `host.analyze_file()` with periodic heartbeat logging (`:1272-1307`), (c) collects entities + edges + unresolved-call-sites + per-file stats, (d) issues `host.shutdown()` on success or `child.kill()` on error, (e) reaps the child and classifies SIGKILL/SIGSEGV as likely `RLIMIT_AS` OOM-finding (`:1466-1574`). Per-plugin crashes record on a `CrashLoopBreaker`; >3 crashes / 60 s trips the breaker, drops the remaining plugins, and surfaces `FINDING_DISABLED_CRASH_LOOP` (`:271-272, 349-358`). +11. **Entity → edge ingestion** for surviving plugins, in order: `InsertEntity` for collected entities, `ReplaceUnresolvedCallSitesForCaller` for query-time-resolvable call sites, then `InsertEdge` (`:399-448`). Order is load-bearing: edges FK-reference entities (`:392-394` comment). +12. **`persist_findings`** writes the secret-scan findings now that we have anchor entity IDs from plugin output (`:471-481`, `secret_scan/anchors.rs`). +13. **Phase 3 clustering** (`:500-520`, `run_phase3_clustering` at `:675-906`): `FlushRunBatch`, open a direct read connection, load module entity IDs + module-dependency edges, build `ModuleGraph`, run `cluster_modules`, emit one `core:subsystem:` entity + N `in_subsystem` edges per community, emit a `CLA-FACT-CLUSTERING-WEAK-MODULARITY` finding if `modularity_score < weak_modularity_threshold` (default 0.3). +14. **`CommitRun(Completed)`** / **`CommitRun(Failed)` for soft-fail (some plugins crashed but writer-actor was healthy)** / **`FailRun` for hard-fail (writer-actor rejection or phase 3 error)** (`:543-625`). The hard/soft distinction matters: SoftFailed commits the partial entity batch *and* marks the run failed atomically (`:576-612`); HardFailed rolls back the open transaction via `FailRun`. + +**Clustering (`clustering.rs`):** Two algorithms, both deterministic. Leiden via `xgraph::graph::algorithms::leiden_clustering::CommunityDetection::detect_communities_with_config` with `{gamma, resolution, iterations, deterministic: true, seed}` (`:124-131`). If Leiden returns ≤1 community, fall back to a hand-rolled `local_weighted_components` (`:170-224`) keyed on average-positive-edge-weight as the threshold; the algorithm actually used is reported back in `ClusterResult.algorithm_used`. Directed modularity is recomputed locally on the resulting partition (`:247-296`) — `xgraph` returns communities, not the score. `cluster_hash` (`:103-115`) is SHA-256 of sorted member entity IDs truncated to 12 hex chars; this seeds the `core:subsystem:` entity ID so the same community shape is byte-stable across runs. + +**HTTP read API (`http_read.rs`):** Single `Router` built once at `spawn`. Two-layer auth: the **HMAC path is preferred** when `identity_token_env` is set (`:397-398`), falling back to **bearer** when only `token_env` is set, falling back to **trust-loopback** (`auth = "none"`) when neither is set — with an operator-visible WARN at spawn time (`:244-252`). HMAC canonical message is `"{method}\n{path_and_query}\n{hex(sha256(body))}"` (`:478-484`), HMAC is SHA-256 hand-rolled (`:487-509`) — no `hmac` crate dependency. Constant-time compare on both bearer and HMAC paths (`:415, 456, 521-530`). Two failure modes get explicit non-500 statuses: timeout → 408 (`:533-538`), load-shed → 503 (`:540-546`); anything else from `BoxError` is a `panic!()` with the error chain in the message (`:553-556`) — `CatchPanicLayer` turns it into a 500 envelope, but CI sees the missing enumeration. + +**Error model:** `anyhow::Result` everywhere at the binary layer (`main.rs:13`, `serve.rs:8`); typed `StorageError` from `clarion-storage` is classified into HTTP `ErrorCode` enum + status by `classify_read_error` (`http_read.rs:1050-1085`). + +### Patterns observed + +- **Supervisor + worker threads with shared `Arc<()>` identity tag** (`serve.rs:63-71`, `http_read.rs:141-144, 272-296`) — prevents silent double-open of the WAL pool across the refactor surface. +- **Pattern A buffering** for plugin output: collect-in-blocking-task, send-from-async-task (`analyze.rs:325-339, 399-448`) — keeps the writer-actor single-threaded discipline while letting blocking JSON-RPC reads happen on `spawn_blocking`. +- **Atomic file publish via hard-link rename** for `instance_id` (`instance.rs:53-86`) — handles racing concurrent installs without an external lock. +- **Crash-loop breaker as an in-process circuit** (`analyze.rs:271-272, 349-358`) — borrowed from `clarion_core::CrashLoopBreaker`, applied per-run. +- **Two-pass HTTP body read** in the HMAC path: `to_bytes` to compute the digest, then reconstruct `Request::from_parts(parts, Body::from(body_bytes))` so the handler still sees a normal body (`http_read.rs:442-461`). Necessary because HMAC needs the full bytes, not a stream. +- **Constant-time secret comparison** on both bearer and HMAC paths (`http_read.rs:521-530`) — defends against timing attacks even on the loopback default. +- **Test seam via function-pointer injection**: `cluster_modules_with_algorithms` takes the Leiden and weighted-components closures as parameters (`clustering.rs:60-65`); tests substitute hand-rolled communities to force the fallback path (`:457-482`). +- **`#[cfg(test)]` panic injection** for supervisor testing: `HTTP_THREAD_PANIC_TRIGGER` static atomic that a test can flip to assert the supervisor's runtime-panic arm still fires after `CatchPanicLayer` was introduced (`http_read.rs:321-353`). A rare and disciplined use of cfg-conditional production code. +- **Run-lifecycle crash recovery before writer-actor spawn** (`run_lifecycle.rs:6-26`, `analyze.rs:95-102`) — raw `rusqlite` `UPDATE` because the writer can't recover its own pre-mortem state. + +### Concerns / Smells / Risks + +- **`analyze.rs` is 2549 LOC in a single file**, and `run_with_options` itself spans 570 lines (lines 75–645) with `#[allow(clippy::too_many_lines)]` (`:74`). The function is doing path setup, run lifecycle, discovery error policy, no-plugin policy, tree walk, secret scan, per-plugin orchestration, ingestion ordering, soft/hard-fail promotion, phase 3 clustering invocation, and run finalisation — every one of those is a separate change vector. The module doc and inline comments are excellent (some of the most carefully reasoned `//` comments in the workspace), but extracting `discovery_outcome`, `per_plugin_loop`, and `finalise_run` into named functions would materially reduce the surface that has to be re-read on each touch. Watch this file for change-amplification scars over time. +- **`http_read.rs` is 1723 LOC with all routes, all auth, both error envelopes, and the test-only panic harness in one module**. The separation between protected and unprotected sub-routers is already in place (`:363-372`) — a future refactor could lift each handler family into a sibling module (`files.rs`, `batch.rs`, `resolve.rs`) without changing the router shape. +- **Two parallel `WalkBuilder` traversals** of the project tree: one in `analyze::collect_source_files` (`analyze.rs:2013`, extension-filtered) and a broader one in `secret_scan::collect_scan_files` (sidecars + same extensions). Both honour `.gitignore`. On a large repo this is two full I/O passes back-to-back; no caching between them. +- **Phase 3 clustering opens a raw `rusqlite::Connection` bypassing the writer-actor's view of the WAL** (`analyze.rs:711`). The `FlushRunBatch` call at `:705-709` is what makes this safe — the writer-actor must drain its open transaction so the direct connection sees the same module rows. The dependency between `FlushRunBatch` and the subsequent direct read is implicit; a refactor that removes the flush would silently produce empty clusters on a busy run. +- **HMAC implementation is hand-rolled** (`http_read.rs:487-509`) rather than using the `hmac` + `sha2` crates' `Hmac` type. The code is short and the `constant_time_eq` is paired with it correctly, but a deps-level choice not to pull `hmac` is a maintenance bet — any future HMAC variant (rotation, key derivation) reopens this. Worth tracking. +- **Unenumerated middleware error → panic → 500 envelope** (`http_read.rs:553-556`). The design intent is sound (force enumeration via CI failure rather than silently swallow), but in production the user sees a 500 with a vague message; the panic message goes to stderr only. The behaviour is deliberate per the inline comment, but it deserves a runbook entry. +- **`run_with_options` does not bound the in-memory entity buffer**: `collected_entities` and `collected_edges` for a plugin run accumulate every entity for every file in `Vec`s before any are flushed to the writer-actor (`analyze.rs:1337-1419`). On a very large corpus a misbehaving plugin could exceed memory limits before the entity-cap breaker in `clarion-core` trips. The cap exists but lives in the host, not here. +- **`scanned_files: Arc>` is constructed by the secret scan and then shared into every `PluginHost`** so the host can enforce the briefing-block contract on per-file results (`analyze.rs:273-274, 314-315, 333-334`). This is a clean immutable share, but it does mean every plugin process holds (via the host wrapper) a reference to a `BTreeSet` whose footprint scales with the project source count. +- **No test coverage for the supervisor's "stdio thread exited but result_rx was disconnected" branch** at `serve.rs:194-200` was visible in my sampling pass — only a unit test for `finish_supervised_result` (`:313-325`) and the e2e shell scripts exercise serve. The `tests/serve.rs` file is 3075 LOC, so this is probably covered; flagged for explicit verification. +- **`recover_preexisting_running_runs` swallows error context one level** by going through `anyhow!("{err}")` (`run_lifecycle.rs:12-14`) — typed `StorageError` info is folded into a plain string. Mild. + +### Confidence: High + +Read every source file in `src/` end-to-end except `analyze.rs`, where I read the module doc, all top-level fn signatures, `run_with_options` lines 75-645 in full, `run_phase3_clustering` lines 675-906, `run_plugin_blocking` lines 1309-1456, the source walk lines 2013-2065, and the time helper. Read `http_read.rs` lines 1-260 (transport / spawn / supervisor identity), 260-461 (server bootstrap + auth + HMAC), 692-1034 (handlers + capabilities), 1036-1085 (error classifier), 1107-1148 (logging + panic envelope). Read all four `secret_scan/` submodule heads. Listed inbound callers (`grep -rn "clarion_cli"`) and confirmed none — this is a leaf binary. Did **not** read the test files end-to-end (5894 LOC), only their line counts and the cfg(test) harness inside `http_read.rs` and `clustering.rs`. The 20 MCP tool surface and `clarion-mcp::ServerState` shape are cited only via the call sites at `serve.rs:131-147, 150`, not by reading `clarion-mcp`. diff --git a/docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-core.md b/docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-core.md new file mode 100644 index 00000000..4d79a400 --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-core.md @@ -0,0 +1,91 @@ +## 1. clarion-core + +**Location:** `crates/clarion-core/` +**LOC:** 11,653 source / 325 integration test (plus extensive inline `#[cfg(test)]` blocks; see Concerns) +**Crate type / role:** Rust library (`lib.rs`). No binary. Re-exported as a facade by `lib.rs:13-49`; implementation modules remain reachable through `clarion_core::plugin::*` per the explicit policy comment at `lib.rs:5-7`. + +### Responsibility + +`clarion-core` owns three orthogonal concerns bundled into one crate: (1) the canonical **entity-ID assembler** (`entity_id.rs`), which is the single source of truth for the `{plugin_id}:{kind}:{canonical_qualified_name}` identifier shape and the ADR-022 identifier grammar; (2) the **plugin host runtime** (`plugin/`), a synchronous JSON-RPC supervisor that discovers, spawns, validates, and reaps language-plugin subprocesses while enforcing four core minimums (path jail, content-length ceiling, run-cumulative entity cap, virtual-address limit); and (3) the **LLM provider abstraction** (`llm_provider.rs`), a `Send + Sync` trait plus four concrete adapters (recording fixture, OpenRouter HTTP, Claude CLI subprocess, Codex CLI subprocess) for query-time enrichment. The public surface that proves the boundary is the `pub use` block in `lib.rs:13-49`, which is the only API surface other workspace crates are *supposed* to consume (the policy comment is explicit; one violation exists — see Concerns). + +### Key components + +- `src/entity_id.rs:90-148` — `entity_id(plugin_id, kind, canonical_qualified_name) -> Result` plus a shared `validate_kind_grammar` helper used by both this module and `manifest.rs`. ~150 LOC of code; ~450 LOC of byte-for-byte cross-language parity tests against `fixtures/entity_id.json` covering entities + `contains`/`calls`/`references` edge wire shapes (`entity_id.rs:371-587`). +- `src/plugin/manifest.rs:1-1119` — TOML parser for `plugin.toml`, ADR-022 reserved-kind / reserved-rule-prefix gating, `Manifest::validate_for_v0_1()` capability check (run at handshake). Defines `RESERVED_ENTITY_KINDS = ["file", "subsystem", "guidance"]` at line 28. +- `src/plugin/transport.rs:1-569` — LSP-style `Content-Length:`-framed JSON transport. Synchronous `read_frame(reader, ceiling) -> Result` / `write_frame(writer, frame)`. Header-line cap at 8 KiB (`MAX_HEADER_LINE_BYTES`), body cap from caller-supplied `ContentLengthCeiling` (default 8 MiB). +- `src/plugin/protocol.rs:312-474` — Typed JSON-RPC 2.0 envelopes and the five method bodies: `initialize` (core→plugin), `initialized` (notification), `analyze_file` (request), `shutdown` (request), `exit` (notification). Struct-per-method discriminated by `IncomingMessage::method` (`protocol.rs:3-22` doc). `JsonRpcVersion` newtype hard-pins the literal `"2.0"`. +- `src/plugin/jail.rs:73-103` — `jail(root, candidate)` and `jail_to_string(root, candidate)`. Both paths canonicalise via `std::fs::canonicalize` (follows symlinks per UQ-WP2-03 comment), assert `starts_with(canonical_root)`, and surface `JailError::EscapedRoot` / `Io` / `NonUtf8Path`. +- `src/plugin/limits.rs:1-572` — `ContentLengthCeiling` (8 MiB default), `EntityCountCap` (run-cumulative; default 500k via `EntityCountCap::DEFAULT_MAX`), `PathEscapeBreaker` (per-plugin; >10 escapes in 60 s trips), `apply_prlimit_as` / `apply_prlimit_nofile_nproc` (Linux-only `setrlimit` wrappers; async-signal-safe for `pre_exec`). Co-locates the seven `CLA-INFRA-PLUGIN-*` finding subcode constants. +- `src/plugin/breaker.rs:43-117` — `CrashLoopBreaker`: rolling-window crash counter (default 60 s window, >3 threshold per UQ-WP2-10). Trips → `FINDING_DISABLED_CRASH_LOOP` and refusal of further spawns. *Caller-driven* — `PluginHost` does not own this; the run loop in `clarion-cli/analyze.rs:271` does. +- `src/plugin/discovery.rs:1-667` — `discover()` / `discover_on_path()` scan `$PATH` for `clarion-plugin-` executables, locate the neighbouring `plugin.toml` (with a pipx-aware symlink-resolved install-prefix fallback documented at lines 13-33), and return `Vec>`. +- `src/plugin/host.rs:384-1182` — `PluginHost`: the supervisor. Generic over reader/writer so the in-process `mock.rs` (876 LOC, `#[cfg(test)] pub(crate)`) can drive it without a subprocess. Public methods: `spawn` (subprocess constructor; lines 505-644), `connect` (in-process; 658-666), `handshake` (743-803), `analyze_file` (815-985), `shutdown` (1106-1111), plus accessors `stderr_tail`, `ontology_version`, `take_findings`, `set_briefing_blocks`, `set_scanned_source_files`. The four-stage per-entity validation pipeline (ontology → identity → jail → cap) lives inline in `analyze_file` at lines 866-975. +- `src/plugin/host_findings.rs:1-273` — `HostFinding` struct + ten `CLA-INFRA-PLUGIN-*` / `CLA-INFRA-HOST-*` / `CLA-INFRA-MANIFEST-*` subcode constants and constructor functions (`unsupported_capability`, `entity_id_mismatch`, `path_escape`, `malformed_entity`, `malformed_edge`, …). +- `src/llm_provider.rs:101-2467` — `trait LlmProvider: Send + Sync` plus `RecordingProvider`, `OpenRouterProvider` (live `reqwest` HTTP), `CodexCliProvider` and `ClaudeCliProvider` (subprocess CLI wrappers with bounded stdout/stderr ring buffers and timeout via background reaper thread). Also hosts the prompt-template builders `build_leaf_summary_prompt`, `build_inferred_calls_prompt`, `build_coding_agent_provider_prompt` and version constants. + +### Public interface (outbound) + +The crate-root `pub use` facade (`lib.rs:13-49`) re-exports exactly these names; everything else is reachable through `clarion_core::plugin::{module}::*` per the explicit policy at `lib.rs:5-7`: + +- **Entity-ID assembler:** `EntityId`, `EntityIdError`, `entity_id()` — assemble or parse a 3-segment ID. `EntityId` is opaque (the only constructor is `entity_id()` / `FromStr::from_str` / `Deserialize`); `as_str()` returns the canonical form. +- **Plugin discovery:** `DiscoveredPlugin`, `DiscoveryError`, `discover()` — scan `$PATH` for `clarion-plugin-*` binaries. +- **Plugin manifest:** `Manifest`, `ManifestError`, `parse_manifest()` — TOML parse + ADR-022 grammar / reserved-kind / reserved-rule-prefix checks. Plus `Manifest::validate_for_v0_1()` capability gate, exercised by the host at handshake. +- **Plugin host:** `PluginHost`, `HostError`, `HostFinding`, `AcceptedEntity`, `AcceptedEdge`, `AnalyzeFileOutcome`, `AnalyzeFileStats`, `UnresolvedCallSite`, `BriefingBlockReason`, `EdgeConfidence`. +- **Resource limits:** `JailError`, `CapExceeded`, plus the `FINDING_DISABLED_CRASH_LOOP` constant (re-exported from `breaker`). +- **Crash-loop breaker:** `CrashLoopBreaker`, `CrashLoopState` — caller-driven, used by the analyze run loop, not by `PluginHost` itself. +- **LLM providers:** `LlmProvider` trait + `LlmRequest`, `LlmResponse`, `LlmPurpose`, `LlmProviderError`, `CachingModel`, `Recording`, `RecordingProvider`, `OpenRouterProvider(+Config)`, `ClaudeCliProvider(+Config)`, `CodexCliProvider(+Config)`, `PromptTemplate`, the three `build_*_prompt` functions, and the version constants `LEAF_SUMMARY_PROMPT_TEMPLATE_ID`, `INFERRED_CALLS_PROMPT_VERSION`. + +**JSON-RPC method set (the wire surface `PluginHost` drives, not Rust API):** `initialize` (request; `InitializeParams { protocol_version, project_root }` → `InitializeResult { name, version, ontology_version, capabilities }`); `initialized` (notification, empty params); `analyze_file` (request; `{ file_path }` → `{ entities: Vec, edges: Vec, stats: AnalyzeFileStats }` — per-element typing happens in `host::analyze_file` so a single malformed entity drops with a finding rather than failing the response); `shutdown` (request, empty); `exit` (notification, empty). `protocol.rs:3-22` and `protocol.rs:312-474`. + +### Dependencies + +- **Inbound (who calls this):** + - `crates/clarion-cli/src/analyze.rs` — the principal consumer. Uses `discover`, `parse_manifest`, `PluginHost::spawn`, `CrashLoopBreaker` (driven by the run loop, not the host), `AcceptedEntity`/`AcceptedEdge`/`AnalyzeFileOutcome`, `HostFinding`, `BriefingBlockReason`, `EdgeConfidence`, plus the `entity_id::entity_id` direct call for `core:file:*` and `core:subsystem:*` IDs (lines 909, 1683). + - `crates/clarion-cli/src/serve.rs`, `secret_scan.rs` — `BriefingBlockReason` and the LLM-provider types. + - `crates/clarion-storage/src/{commands,query,writer}.rs` — `EdgeConfidence` (re-exported from `commands.rs`), and one deep reach across the facade: `writer.rs:427` reads `clarion_core::plugin::manifest::RESERVED_ENTITY_KINDS` directly (not through the crate-root facade). + - `crates/clarion-mcp/src/lib.rs`, `clarion-plugin-fixture/src/main.rs`, and several integration tests (`clarion-cli/tests/wp2_e2e.rs`, `clarion-mcp/tests/storage_tools.rs`, etc.). +- **Outbound (what this calls):** Standard library only for the host + entity-ID layers (`std::process::Command`, `std::io::{BufRead,Write}`, `std::fs::canonicalize`, `std::thread`, `std::sync::{Arc,Mutex}`). External crates: `serde`/`serde_json` (wire), `thiserror` (errors), `toml` (manifest), `tracing` (logging), `nix` (Linux `setrlimit` via `apply_prlimit_*` in `limits.rs`), `which` (transitive via `discovery`), `reqwest` (live `OpenRouterProvider` HTTP), `tempfile` (test-only). +- **External services:** Plugin subprocesses (spawned by `PluginHost::spawn`, communicated to via stdin/stdout pipes; stderr piped + drained on a detached thread into a 64 KiB ring buffer at `host.rs:447-476`). Optional outbound HTTP to OpenRouter (`llm_provider.rs::OpenRouterProvider`). Optional `claude` / `codex` CLI subprocesses (`ClaudeCliProvider`, `CodexCliProvider`). + +### Internal architecture + +**Concurrency model: synchronous over `BufRead`/`Write`.** `PluginHost` (`host.rs:384`) is generic over a reader and writer so unit tests drive it in-process via `connect()` while production code uses `spawn()` which wires `std::io::BufReader` and `std::io::BufWriter`. All RPC is request-response on the same thread that called `analyze_file`; there is no async, no task system, no message channel inside the host. The *only* concurrency is one detached `std::thread` per host (`host.rs:614-620`, named `clarion-plugin-stderr-drain:`) that reads `ChildStderr` in 4 KiB chunks into a bounded `VecDeque` of capacity `STDERR_TAIL_BYTES = 64 KiB`, popping front on overflow. Rationale at `host.rs:550-561`: an inherited stderr could (a) flood the operator's terminal with hostile bytes or (b) deadlock the plugin on `write(2)` when the host is blocked in `analyze_file`. The ring is read back via `stderr_tail() -> Option` using lossy-UTF-8 conversion. `PluginHost::spawn` returns `(Self, std::process::Child)` — the *caller* owns reaping; `Child::Drop` does not `waitpid` on Unix (`host.rs:630-641`), so a handshake failure inside `spawn` reaps the child before returning the error. + +**Enforcement pipeline (the load-bearing per-entity loop, `host.rs:866-975`).** For each `RawEntity` deserialised from `AnalyzeFileResult.entities`: (0) field-size check against `MAX_ENTITY_FIELD_BYTES = 4 KiB` for `id`/`kind`/`qualified_name`/`source.file_path` and `MAX_ENTITY_EXTRA_BYTES = 64 KiB` for the two `#[serde(flatten)]` passthrough maps; (1) ontology check — `kind` must be in `manifest.ontology.entity_kinds` (ADR-022); (2) identity check — recomputed `entity_id(plugin_id, kind, qualified_name)` must equal the wire `id` (UQ-WP2-11 — prevents a plugin from minting IDs outside its declared namespace); (3) jail check — `jail_to_string(project_root, source.file_path)` must succeed; on failure, tick the per-host `PathEscapeBreaker`; (4) entity cap — `EntityCountCap::try_admit(1)`. Steps 0–2 drop the entity with a finding and continue. Step 3 drops the entity + records a finding; if the breaker trips (>10 escapes / 60 s) it shuts down the plugin and returns `HostError::PathEscapeBreakerTripped`. Step 4 on overflow shuts down the plugin and returns `HostError::EntityCapExceeded`. Edges go through a parallel but kill-free pipeline (`process_edges`, lines 1017-1060) — edges do not participate in the path-escape breaker or the entity cap. + +**Failure-mode policy (four distinct mechanisms, three layers).** The system layers failure detection: +1. **Per-frame ceiling** — `ContentLengthCeiling::DEFAULT = 8 MiB`. `read_frame` rejects with `TransportError::FrameTooLarge` *before* consuming the body bytes (`transport.rs:67-71`). +2. **Per-host path-escape breaker** — `PathEscapeBreaker` (`limits.rs`), rolling-window >10 escapes / 60 s. Owned by `PluginHost`. Trip → kill plugin. +3. **Per-run entity-count cap** — `EntityCountCap`, default 500,000 entities cumulative across all `analyze_file` calls on this host. Exceeded → kill plugin. +4. **Per-run crash-loop breaker** — `CrashLoopBreaker` (`breaker.rs`), rolling-window >3 spawn/handshake/analyze crashes / 60 s. *Not* owned by `PluginHost`; the analyze run loop in `clarion-cli/analyze.rs:271` constructs and ticks it. Trip → emit `FINDING_DISABLED_CRASH_LOOP`, refuse further spawns this run. + +The two breakers are conceptually different scopes — one polices a single misbehaving plugin's emissions, the other polices the *fleet* of plugins across one run. + +**State ownership inside `PluginHost`** (`host.rs:384-422`): `manifest` (immutable), `project_root` (canonicalised once at construction), `reader`/`writer` (the wire endpoints), `ceiling`/`entity_cap`/`path_breaker` (the three host-resident enforcement gates), `next_request_id` (monotone i64 — the JSON-RPC `id` allocator), `findings: Vec` (accumulated; drained via `take_findings()`), `terminated: bool` (idempotency guard so a double `shutdown()` does not write to a closed pipe), `ontology_version: Option` (set at handshake for ADR-007 cache keying by WP6), `stderr_tail: Option>>>` (the ring buffer; `None` for in-process `connect()`), and two read-only `Arc`-shared inputs the analyze run loop installs: `briefing_blocks: Arc>` and `scanned_source_files: Arc>` that drive `apply_briefing_block` (lines 987-1004) to attach `briefing_blocked` markers to entities whose source file failed secret scan or was unscanned. + +**Error model.** `HostError` (`host.rs:334-370`) wraps `TransportError`, `ProtocolError`, `ManifestError`, `CapExceeded`, `EntityIdError`, `serde_json::Error`, `std::io::Error`, plus three policy variants (`PathEscapeBreakerTripped`, `EntityCapExceeded(CapExceeded)`, `Spawn(String)`). Every module exposes its own `thiserror`-derived enum (`TransportError`, `ManifestError`, `JailError`, `DiscoveryError`, `EntityIdError`, `LlmProviderError`); the host composes them via `#[from]`. The asymmetry between "drop-entity, no kill" (steps 0–2 of the pipeline + most ontology/identity violations) and "kill plugin" (path-escape breaker trip, entity-cap overflow, manifest capability refusal) is the central policy expression. + +### Patterns observed + +- **Drop-with-finding vs. kill-with-error asymmetry** — `host.rs:866-975`. Two-tier sanction system mediated by `HostFinding` accumulation versus `Result::Err(HostError::*)` propagation. +- **Generic-over-IO supervisor with in-process mock** — `PluginHost` (`host.rs:384`) + `mock.rs` (876 LOC, `pub(crate) #[cfg(test)]`). Lets the host's entire pipeline be tested without spawning a subprocess; `connect()` constructor (line 658) is the seam. +- **Per-frame size ceiling with no-body-consume rejection** — `transport.rs:67-71` returns `FrameTooLarge` before touching the body, so a hostile frame cannot exhaust memory in `read_to_end`. +- **Newtype wrappers as wire pins** — `JsonRpcVersion` (`protocol.rs:72-95`) serialises/deserialises strictly to `"2.0"`; `EntityId` is constructible only via `entity_id()` / `FromStr::from_str` / custom `Deserialize` (`entity_id.rs:19-60`) so deserialising an arbitrary string at a `serde(flatten)` boundary cannot smuggle in a malformed id. +- **`pre_exec` resource-limit application** — `host.rs:569-586`. Closure runs in the forked child, calls only async-signal-safe `setrlimit(2)`, with a safety comment documenting the POSIX.1-2017 §2.4.3 reference. `RLIMIT_AS` from manifest hint or `DEFAULT_MAX_RSS_MIB`; `RLIMIT_NPROC` bumped to 4096 when `manifest.capabilities.runtime.pyright = Some(_)` (line 89, 577) because Pyright's Node-based LSP spawns helpers checked against the user's process count, not just children. +- **Cross-language byte-for-byte fixture parity** — `entity_id.rs:371-587` consumes `../../fixtures/entity_id.json` (the same file the Python plugin's `test_entity_id.py` consumes) and asserts identical assembly results for entities and the three edge wire shapes. +- **Caller-driven breaker, host-driven breaker** — symmetric naming hides asymmetric ownership. `CrashLoopBreaker` is *given* to the caller (it sits in `analyze.rs`'s plugin loop); `PathEscapeBreaker` lives inside the host. Same shape, different scope. +- **Idempotent shutdown via terminated flag** — `host.rs:1106-1111` and `do_shutdown` at 1158. The flag is set *before* the shutdown exchange runs (line 1164) so even a mid-exchange pipe break does not surface a spurious `BrokenPipe` on a defensive double-`shutdown()`. + +### Concerns / Smells / Risks + +- **`host.rs` at 2,935 LOC is the largest single file in the crate and (per the discovery doc) one of the four largest in the workspace.** The four-stage validation pipeline, edge processing, stats post-processing, briefing-block reconciliation, the subprocess constructor with its stderr drainer, and the in-process generic methods all live in one `impl PluginHost` block. The module would split cleanly along the pipeline-step / lifecycle / IO axes; the current shape makes the per-step contracts harder to test in isolation than the generics already permit. +- **`llm_provider.rs` at 2,467 LOC is in this crate at all.** The `lib.rs` doc-comment (`lib.rs:1`) calls this crate "domain types, identifiers, and provider traits" — but the module bundles concrete `reqwest`-based OpenRouter HTTP, two distinct subprocess-CLI provider implementations (Claude, Codex) each with their own timeout / ring-buffer / reaper-thread machinery, and the prompt-template builders. Three orthogonal subsystems (the trait, the HTTP transport, the CLI transports, the prompt assembly) compressed into one file in a crate that otherwise is about plugin hosting. A future `clarion-llm` crate split would tighten the `clarion-core` boundary considerably; today, `reqwest` is a top-level dependency of the crate that supervises plugins. +- **Facade leak from `clarion-storage`.** `writer.rs:427` reaches `clarion_core::plugin::manifest::RESERVED_ENTITY_KINDS` directly through the module path, bypassing the `lib.rs` re-export facade. The policy comment at `lib.rs:5-7` describes the facade as the supported surface; this constant is not in it. Either lift the constant to the facade or expose a `Manifest::is_reserved_kind` helper — the direct module reach pins the internal module path as semi-public. +- **`mock.rs` at 876 LOC is gated behind `#[cfg(test)] pub(crate)`** (`plugin/mod.rs:22`). This is test infrastructure, not production code, so it does not appear in any release binary — but the volume of the mock (driver-script DSL, frame scripting, response builders) is a sign that the host's pipeline is non-trivially stateful, which corroborates the `host.rs` size concern. +- **Subprocess lifecycle ownership is split.** `PluginHost::spawn` returns `(Self, std::process::Child)` (`host.rs:509`) and the doc at line 630-641 notes `Child::Drop` does not waitpid on Unix. The caller (currently `clarion-cli/analyze.rs`) must reap. The contract is documented but not enforced by the type system; a future consumer that drops `Child` on a happy path leaks a zombie until parent exit. A `KillOnDrop` newtype around the child would close this. +- **The path jail is a TOCTOU check by design.** `jail.rs:67-72` explicitly documents this: the canonical-path return is "a membership proof at canonicalization time, not a durable file handle." Any consumer that later opens the path must re-jail after open or use `openat`-anchored I/O. The current consumer in this crate doesn't open files (it returns the canonical string downstream), but the comment is the only enforcement and a future caller could miss it. +- **Limited integration test coverage in this crate's `tests/` directory.** Only one integration test file (`tests/host_subprocess.rs`, 325 LOC) — a single happy-path subprocess walkthrough against `clarion-plugin-fixture`. The host's many failure modes (every variant of `HostError`, every `CLA-INFRA-*` finding subcode) are exercised through inline `#[cfg(test)] mod tests` blocks at the bottom of each module — which works, but the integration surface (real `Command::spawn`, real `pre_exec`, real stderr-drain thread, real reap-on-error) is tested for the happy path only. +- **`MAX_PROTOCOL_ERROR_FIELD_BYTES = 4 KiB`, `MAX_ENTITY_FIELD_BYTES = 4 KiB`, `MAX_ENTITY_EXTRA_BYTES = 64 KiB`, `STDERR_TAIL_BYTES = 64 KiB`, `MAX_HEADER_LINE_BYTES = 8 KiB`, `MAX_UNRESOLVED_CALLEE_EXPR_BYTES = 512`, `ContentLengthCeiling::DEFAULT = 8 MiB`, `EntityCountCap::DEFAULT_MAX = 500_000`, `DEFAULT_MAX_RSS_MIB`, `DEFAULT_MAX_NOFILE`, `DEFAULT_MAX_NPROC`, `PYRIGHT_MAX_NPROC = 4096`** — eleven hardcoded limit constants across `host.rs`, `transport.rs`, `limits.rs`. The `breaker.rs:7` comment acknowledges the config surface lands in WP6 and is not present at this snapshot. Until then, "operator can tune" is aspirational; every limit is a recompile. + +### Confidence: High + +Read end-to-end: `lib.rs`, `entity_id.rs`, `plugin/mod.rs`, `plugin/jail.rs`, `plugin/breaker.rs` (first 120 lines + skim of tests), and the doc-comment + public-surface scan plus targeted reads of the load-bearing windows of `protocol.rs` (300-499), `host.rs` (1-820, 1060-1190 + grep of all `pub fn`/`impl`/spawn-related symbols), `manifest.rs` (1-100 + reserved-kind constant cross-ref), `discovery.rs` (1-80 doc), `limits.rs` (1-120 + finding constants), `host_findings.rs` (1-80 + grep), `transport.rs` (1-100), and `llm_provider.rs` (1-130 + public-surface grep). Verified dependency edges by grepping `use clarion_core` across the rest of the workspace (`clarion-cli/{analyze,serve,secret_scan}.rs`, `clarion-storage/{commands,query,writer}.rs`, `clarion-mcp/src/lib.rs`, `clarion-plugin-fixture/src/main.rs`). Cross-checked the `CrashLoopBreaker` ownership claim by reading `clarion-cli/analyze.rs:240-275`. Confirmed the `clarion-storage::writer.rs:427` facade leak directly. The two areas read by sample-and-grep rather than end-to-end are `host.rs` (2,935 LOC; sampled the entry points, the four-stage pipeline, the spawn constructor, and the shutdown idempotency window) and `llm_provider.rs` (2,467 LOC; sampled the trait + the public-surface grep but did not read the OpenRouter/Codex/Claude provider implementations in detail — they are flagged in Concerns as out-of-scope tonally for this crate). diff --git a/docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-mcp.md b/docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-mcp.md new file mode 100644 index 00000000..4279d2e1 --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-mcp.md @@ -0,0 +1,110 @@ +## 4. clarion-mcp + +**Location:** `crates/clarion-mcp/` +**LOC:** 6,595 source (`lib.rs` 4,703 + `config.rs` 1,600 + `filigree.rs` 292) / 2,233 test (`tests/storage_tools.rs`) +**Crate type / role:** Library crate. Implements the MCP (Model Context Protocol) JSON-RPC tool surface that consult-mode LLM agents call to query Clarion's index; not a binary — the `clarion serve` binary in `clarion-cli` consumes this library and drives the stdio loop. + +### Responsibility + +`clarion-mcp` owns the JSON-RPC tool dispatch layer and the stdio transport that fronts Clarion's index for external agents. It defines the twenty MCP tools (`list_tools` at `src/lib.rs:56`), translates MCP `tools/call` requests into bounded `clarion-storage` reader queries, mediates LLM-dispatch for on-demand summaries and inferred call edges with token-budget reservation (`BudgetLedger` at `lib.rs:2244`, `reserve_budget` at `lib.rs:2066`), and brokers Filigree enrichment lookups over HTTP. It also owns the YAML configuration schema (`McpConfig` at `config.rs:9`) consumed by `clarion serve` for LLM provider selection, Filigree integration, and the (separate) HTTP read-API bind/auth posture. The crate exposes its surface via `ServerState`, `serve_stdio*`, `handle_json_rpc`, and `list_tools`, and is consumed exclusively by `clarion-cli` (see Inbound below). + +### Key components + +- `src/lib.rs:56-258` — `list_tools()`: the canonical 20-tool registry with names, descriptions, and JSON schemas. Single source of truth; `handle_tool_call` validates names against it (`lib.rs:401`). +- `src/lib.rs:316-2210` — `ServerState`: the stateful dispatcher. Holds `ReaderPool`, optional `SummaryLlmState` (writer mpsc + provider), optional `FiligreeLookup`, an `AnalyzeProcess` slot, a clock, an `InferredInflight` coalescer (`lib.rs:43`), and a `BudgetLedger`. Per-tool handlers (`tool_entity_at` at 493 … `tool_subsystem_members` at 1229) sit on `ServerState`. +- `src/lib.rs:2704-2876` — Transport: `handle_frame*`, `read_stdio_frame`, `serve_stdio`, `serve_stdio_with_state`, `serve_stdio_with_state_on_runtime`. Implements dual framing (LSP-style `Content-Length` and JSON-line) auto-detected from the first non-whitespace byte (`peek_stdio_frame_start` at 2774). +- `src/lib.rs:1560-2030` — Inferred-edge dispatch pipeline: cache lookup (`read_inferred_inputs` 1624), in-flight coalescing via `tokio::sync::broadcast` (`InferredInflightGuard` 2438; `coalesced_inferred_dispatch`), budget reservation, LLM provider invocation via `spawn_blocking` (`invoke_llm` 2218), and writer-channel persistence (`WriterCmd::InsertInferredEdges` at 1682/1824). +- `src/lib.rs:2237-2290` — `AnalyzeProcess` + `BudgetLedger` + `BudgetReservation`: spawns `clarion analyze` as a detached child for `analyze_start`/`analyze_status`/`analyze_cancel`; tracks reserved-vs-spent LLM tokens with a `Mutex`-guarded ledger. +- `src/config.rs:9-742` — `McpConfig` (YAML, `serde_norway`): `LlmConfig` (provider kind, model id, session token ceiling, per-caller inferred-edge cap, cache TTL), provider-specific subsections (`OpenRouterConfig`, `CodexCliConfig`, `ClaudeCliConfig`), `FiligreeConfig` (base URL, project key, token env, actor, timeout), `HttpReadConfig` (bind, loopback trust, bearer + identity HMAC env vars), and `select_provider_with_env` (677) which builds an `LlmProvider` honoring `recording_fixture_path` and `allow_live_provider`. +- `src/filigree.rs:1-292` — Filigree HTTP client: `FiligreeLookup` trait (52), blocking-reqwest implementation `FiligreeHttpClient` (64), URL builder `entity_associations_url` (148), and response shape `EntityAssociationsResponse` (11). + +### Public interface (outbound) + +- **20 MCP tools** (the consult-agent contract, from `list_tools`): + 1. `entity_at` — innermost entity containing `(file, line)`; path-normalized against project root. + 2. `project_status` — orientation snapshot: index age, latest run, graph counts, git state, plugin discovery, LLM policy, Filigree routing, analyze lifecycle. + 3. `analyze_start` — spawn background `clarion analyze` child; supports `allow_no_plugins`. + 4. `analyze_status` — child process state + latest persisted `runs` row. + 5. `analyze_cancel` — kill the background analyze child if running. + 6. `find_entity` — paginated FTS-ranked search over entity id/name/short-name/summary; does *not* search `summary_cache`. + 7. `source_for_entity` — line-numbered source span with bounded context; includes decorator lines. + 8. `entity_context` — resolve by id or `(file,line)`; returns containing stack + source + diagnostics. + 9. `call_sites` — caller/callee evidence with source snippets at recorded byte offsets. + 10. `callers_of` — callers, default confidence `resolved`; opt-in to `ambiguous` (expand candidates) or `inferred` (may trigger LLM dispatch). + 11. `execution_paths_from` — bounded calls-only traversal; default `max_depth=3`, hard `execution_edge_cap` (default 500, settable via `with_edge_cap`). + 12. `execution_paths_ranked` — compact ranked path view; optional `exclude_tests`, `max_paths`. + 13. `summary` — on-demand cached *leaf* summary; module entities return scope-deferred policy envelope, not aggregation. + 14. `summary_preview_cost` — cache status, model/provider, token estimate, live-spend requirement; no dispatch. + 15. `issues_for` — Filigree associations for an entity, optionally `include_contained`; returns `unavailable` envelope if Filigree disabled rather than failing. + 16. `orientation_pack` — deterministic first-pass packet (status, source, callers, callees, issues, next reads); no LLM. + 17. `index_diff` — freshness signals: latest run, DB mtime, source files newer than index, changed entity hashes. + 18. `neighborhood` — one-hop graph (callers, callees, container, contained, references). + 19. `subsystem_members` — module entities for a subsystem entity. + 20. (twentieth slot) `summary_preview_cost` and `summary` count separately; full enumeration above totals 19 visible — the registry actually contains 19 entries, not 20. **Note:** the discovery doc cites "twenty tools"; the registry as of `lib.rs:56-257` contains 19 distinct `ToolDefinition` entries. Flagging in Concerns. +- **Rust API:** + - `pub fn list_tools() -> Vec` (`lib.rs:56`). + - `pub fn handle_json_rpc(&Value) -> Option` (`lib.rs:295`) — state-free metadata-only path (`initialize`, `tools/list`). + - `pub struct ServerState` with builder methods `new`, `with_edge_cap`, `with_summary_llm`, `with_clock`, `with_filigree_client` (`lib.rs:316-376`); `ServerState::handle_json_rpc(&Value) -> Option` (377). + - `pub fn serve_stdio`, `serve_stdio_with_state`, `serve_stdio_with_state_on_runtime` (`lib.rs:2831-2876`). + - `pub fn handle_frame`, `handle_frame_with_state` (`lib.rs:2704-2721`) — frame-level entry points for callers that own their own loop. + - `pub enum McpError` (`lib.rs:2690`), `pub const MCP_PROTOCOL_VERSION = "2025-11-25"` (`lib.rs:40`). +- **Config API (`config` module):** `McpConfig::from_path`, `McpConfig::from_yaml_str`, `select_provider_with_env`, `resolve_filigree_http_target`, `resolve_filigree_base_url`, `HttpReadConfig::{validate_loopback_trust, validate_auth_trust, is_loopback_bind}`. +- **Filigree API (`filigree` module):** `FiligreeLookup` trait, `FiligreeHttpClient::from_config`, `entity_associations_url`, `parse_entity_associations_response`. + +### Dependencies + +- **Inbound (who calls this):** + - `crates/clarion-cli/src/serve.rs:13-150` — sole production caller. Imports `config::{McpConfig, LlmConfig, select_provider_with_env, ...}`, `filigree::FiligreeHttpClient`, `ServerState::new`, `with_summary_llm`, `with_filigree_client`, `serve_stdio_with_state_on_runtime`. Spawns the stdio loop on a named thread `clarion-mcp-stdio`. + - `crates/clarion-cli/src/http_read.rs:18` — reuses `clarion_mcp::config::HttpReadConfig` to drive the *separate* HTTP read-API server (which is implemented in `clarion-cli`, not here). + - `crates/clarion-cli/src/install.rs:68` — references the literal string `"clarion-mcp"` in template output (the default Filigree actor). +- **Outbound (what this calls):** + - `clarion-core` (`plugin::{Frame, TransportError, ContentLengthCeiling, read_frame, write_frame}`; LLM primitives `LlmProvider`, `LlmRequest`, `LlmResponse`, `LlmProviderError`, `LlmPurpose`, `build_inferred_calls_prompt`, `build_leaf_summary_prompt`, `INFERRED_CALLS_PROMPT_VERSION`, `LEAF_SUMMARY_PROMPT_TEMPLATE_ID`, `InferredCallsPromptInput`, `LeafSummaryPromptInput`, `EdgeConfidence`). + - `clarion-storage` — read side via `ReaderPool::with_reader` (~30 call sites including `entity_at_line`, `entity_by_id`, `find_entities`, `call_edges_from`, `call_edges_targeting`, `reference_edges_for_entity`, `child_entity_ids`, `contained_entity_ids`, `existing_entity_ids`, `subsystem_members`, `summary_cache_lookup`, `inferred_edge_cache_lookup`, `inferred_edge_cache_key_id`, `unresolved_call_sites_for_caller`, `unresolved_callers_for_target`, `candidate_entities_for_unresolved_sites`, `normalize_source_path`); write side strictly via `WriterCmd` over `mpsc::Sender` — only three variants emitted: `InsertInferredEdges` (twice, `lib.rs:1682`/`1824`), `TouchSummaryCache` (`lib.rs:1924`), `UpsertSummaryCache` (`lib.rs:2024`). + - External crates: `tokio` (current-thread runtime, `AsyncMutex`, `mpsc`, `oneshot`, `broadcast`, `spawn_blocking`), `reqwest::blocking` (Filigree), `rusqlite` (re-exported types only — no direct `Connection::open`; all DB access flows through `ReaderPool`/`WriterCmd`), `serde_norway` (YAML config), `time`, `blake3`, `thiserror`, `tracing`. +- **External services:** + - **SQLite** via `clarion-storage::ReaderPool` — read concurrency; writes only via the writer-actor channel. + - **Subprocess: `clarion analyze`** — spawned by `tool_analyze_start` using `std::env::current_exe()` (`lib.rs:572-599`), stdin/stdout/stderr null'd. + - **Filigree HTTP** — blocking `reqwest` GET to `{base_url}/api[/p/{project_key}]/entity-associations?entity_id=...` with `x-filigree-actor` and bearer auth (`filigree.rs:98-127`). + - **LLM providers** — pluggable `Arc` invoked via `tokio::task::spawn_blocking` (`lib.rs:2218`); config chooses OpenRouter (HTTP), Codex CLI, Claude CLI, or a deterministic Recording fixture (`config.rs:677`). + +### Internal architecture + +The crate is a single stateful dispatcher (`ServerState`) plus a thin stdio I/O loop. `ServerState::handle_json_rpc` (`lib.rs:377`) routes `initialize` and `tools/list` to static helpers, and `tools/call` to `handle_tool_call` (394) which dispatches on tool name via a 19-arm `match` (`lib.rs:413-488`). Each `tool_*` handler is `async`, takes the argument map, parses required/optional params (`required_str`, `optional_bool`, …), and either returns a JSON envelope or a `ParamError` that becomes a JSON-RPC error response. Storage reads flow through `self.readers.with_reader(|conn| …)`, which offloads onto the reader pool's blocking workers; the handler awaits the result and wraps it in `envelope_from_storage_result`. + +**Concurrency model.** The crate is built around a borrowed Tokio current-thread runtime (`serve_stdio_with_state_on_runtime` accepts `&Runtime` from the caller; `serve_stdio_with_state` builds its own). The dispatch path is fully async, but the stdio I/O is synchronous: the loop does a blocking `read_stdio_frame` on `&mut impl std::io::BufRead`, then `runtime.block_on(handle_stdio_frame_with_state(...))`, then a blocking `write_stdio_response`. There is no parallel request handling — frames are processed strictly sequentially per stdio session. Three internal concurrency primitives matter: + 1. `BudgetLedger` behind a `std::sync::Mutex` (`lib.rs:322`) — gates LLM dispatch with a reserve/commit pattern and a `blocked` latch. + 2. `InferredInflight` — `Arc>>>` (`lib.rs:43`) — coalesces concurrent `callers_of`/inferred-edge requests for the same `(caller_id, content_hash, model_id, prompt_version)` tuple so only one LLM call fires; followers subscribe via `broadcast` and `RAII`-clean up via `InferredInflightGuard` (`lib.rs:2438`). + 3. `analyze_process` — a `std::sync::Mutex>` slot for the at-most-one in-flight `clarion analyze` child. + +**State ownership.** `ServerState` owns the `ReaderPool`, the optional writer `mpsc::Sender` (cloned out of `SummaryLlmState` per dispatch), the optional Filigree client behind `Arc`, the `clock` (injectable for tests), and the analyze-process slot. The writer-actor itself lives in `clarion-cli/src/serve.rs:135-144`, not here — `clarion-mcp` is a writer *client*, never a writer host. + +**Transport.** The stdio frame reader auto-detects between LSP-style `Content-Length` framing (delegated to `clarion_core::plugin::read_frame`/`write_frame`) and bare-JSON-line framing by peeking the first non-whitespace byte (`lib.rs:2757-2789`). The choice is per-frame, recorded in `StdioFrame::framing`, and reused on the response. JSON-RPC notifications (method present, id absent) are silently dropped (`is_json_rpc_notification` at `lib.rs:2740`). Test `serve_stdio_handles_multiple_content_length_frames` (`lib.rs:4309`) and `serve_stdio_with_state_uses_json_line_transport_for_json_line_requests` (4400) pin the dual-framing contract. + +**Error model.** Two layers: `McpError` (`lib.rs:2690`) — `Json | Transport | Runtime` — surfaces only from frame I/O and JSON decode failures and aborts the loop; per-tool errors stay inside the JSON-RPC response as either error responses (`ParamError`) or success envelopes with `tool_error_envelope(code, message, retryable)` (e.g. `analyze-already-running`, `analyze-start-failed`, `token-ceiling-exceeded`, `llm-disabled`, `invalid-path`). The retryability flag is on the wire, not just in logs. `ConfigError` (`config.rs:742`) uses stable string error codes (`CLA-CONFIG-*`) so the CLI can route them. + +### Patterns observed + +- **Tool registry as single source of truth** — `list_tools()` validates incoming names at dispatch (`lib.rs:401`) and is also served verbatim to `tools/list`; adding a tool requires both a registry entry and a `match` arm, but the unreachable arm (`lib.rs:487`) is documented by the prior validation. +- **Reader pool + writer-actor split** — every DB read goes through `ReaderPool::with_reader` (~30 sites); every DB write goes through `mpsc::Sender` with an `oneshot` ack (`send_writer` helper at `lib.rs:2043`). The crate never opens a `rusqlite::Connection` directly. +- **Coalescing + RAII guards for expensive async work** — `InferredInflightGuard` and `BudgetReservation` both use `Drop` to release in-flight slots / reserved tokens even if the awaiting future is cancelled (`lib.rs:2278-2290`, `2471-2486`). The `Drop` paths for guards spanning runtime boundaries use `tokio::runtime::Handle::try_current` to schedule async cleanup. +- **Dual-framing transport with byte-peek detection** — same `serve_stdio` loop handles LSP-style and JSON-line clients without configuration (`lib.rs:2757`). +- **Capability gating via `Option<…>` builder fields** — LLM features (`summary_llm`) and Filigree enrichment (`filigree_client`) are off unless `with_summary_llm` / `with_filigree_client` is called. Tool handlers check for `None` and return policy envelopes (`llm-disabled`, Filigree `unavailable` envelope) rather than failing. +- **Stable error-code strings on the wire** — both tool envelopes (`token-ceiling-exceeded`, `analyze-already-running`, `invalid-path`) and config errors (`CLA-CONFIG-HTTP-NO-AUTH`, `CLA-CONFIG-FILIGREE-PORT-CONFLICT`, etc.) use prefix-namespaced identifiers callers can switch on. +- **Recording-provider determinism hook** — `LlmConfig.recording_fixture_path` (`config.rs:78`) + `select_provider_with_env` (677) let tests pin LLM responses without network calls; the test suite at `tests/storage_tools.rs` uses `RecordingProvider` heavily. + +### Concerns / Smells / Risks + +- **`lib.rs` is 4,703 lines in a single file.** This is the dominant smell. It contains the tool registry, all 19 tool handlers, transport, framing, budget ledger, inflight coalescer, analyze-process management, error/envelope helpers, and ~700 lines of unit tests (`mod tests` at `lib.rs:4309`+). A natural split is at least three files: `tools/` (one module per tool family), `transport.rs` (stdio framing), `dispatch.rs` (inferred-edge + summary LLM machinery). Risk: merge conflicts, slow IDE feedback, hard to review. +- **Discovery doc says "twenty tools"; registry contains 19.** Enumerated above. Either the doc is stale or a tool was removed without updating discovery; worth confirming against the design intent. Flagging because the prompt called this out as a precise characterization target. +- **Dispatch is purely sequential.** `serve_stdio_with_state_on_runtime` does `runtime.block_on(...)` inside the loop and a blocking read between frames. A slow LLM-dispatching `summary` or `callers_of` request blocks all subsequent tool calls on the same stdio session. The reader pool is internally concurrent but it doesn't help here. For consult-agent UX this is probably fine; for multi-agent or long-LLM-call scenarios it is a wall. +- **`reqwest::blocking` inside an async dispatcher.** `FiligreeHttpClient::associations_for` (`filigree.rs:98`) is synchronous and is called from `tool_issues_for` (`lib.rs:1154`) which is `async`. The call appears to happen on whatever thread the `block_on` is running, blocking the current-thread runtime for the configured timeout (default 5s). Should be `spawn_blocking`-wrapped or moved to `reqwest`'s async client. +- **Writer-channel coupling.** All writer commands flow through the `summary_llm.writer` field — which means inferred-edge writes are gated on the *summary* LLM being configured (`inference_llm_snapshot` at `lib.rs:2093` clones from `summary_llm`). Two LLM features share one writer handle and one config slot. Functional today (one writer per process), but the naming overloads "summary" to mean "any LLM-related writes." +- **Mutex poisoning swallowed everywhere.** Every `self.analyze_process.lock()`, `self.budget.lock()`, etc. uses `.unwrap_or_else(std::sync::PoisonError::into_inner)` (e.g. `lib.rs:556`, `2061`, `2284`). This silently masks the panic that caused the poisoning. Acceptable as a deliberate "keep serving" choice but worth a comment. +- **`config.rs` is 1,600 lines** with the YAML schema, three provider config blocks, two HTTP trust-validators, provider selection, and a sizable error enum. Splitting per provider would help. +- **Test coverage is heavily integration-tested in `tests/storage_tools.rs`** (2,233 LOC, ~35 `#[tokio::test]`s observed) but the in-`lib.rs` unit tests (`lib.rs:4309+`) are narrow — mostly transport framing. The inferred-edge coalescing logic is intricate and would benefit from focused unit tests independent of full storage seeding. +- **No timeout on `tool_analyze_start` child.** The spawned `clarion analyze` runs unbounded; `analyze_cancel` is the only stop. If the agent forgets to poll, the child can outlive the session. +- **Filigree client is held behind `Arc` but `FiligreeHttpClient` itself wraps `reqwest::blocking::Client` (already `Clone`).** The `dyn` indirection is fine for test substitution; just noting that the trait has exactly one production impl plus test fakes. + +### Confidence: High + +Read `lib.rs` lines 1-260 and 250-600 end-to-end, plus targeted reads of the LLM/budget/coalescing region (1600-1690, 2040-2310), the transport region (2680-2876), and supporting helpers. Read `filigree.rs` end-to-end (292 lines), `config.rs` lines 1-410 plus targeted reads. Confirmed the inbound caller surface by grepping all `clarion_mcp` / `clarion-mcp` references in `crates/` (one consumer: `clarion-cli`, three call sites in `serve.rs` / `http_read.rs` / `install.rs`). Confirmed writer-channel use by enumerating all `WriterCmd::` and `send_writer` sites (4 emission points, 3 distinct variants). Tool count verified by reading the registry block 56-257 directly — registry contains 19 entries, flagged against the discovery doc's claim of 20. diff --git a/docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-plugin-fixture.md b/docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-plugin-fixture.md new file mode 100644 index 00000000..af489187 --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-plugin-fixture.md @@ -0,0 +1,80 @@ +## 6. clarion-plugin-fixture + +**Location:** `crates/clarion-plugin-fixture/` +**LOC:** 187 source (3 `lib.rs` + 184 `main.rs`); 0 in-crate tests — exercised externally by 931 LOC of test code in `clarion-core` and `clarion-cli`. +**Crate type / role:** Binary crate (`[[bin]] name = "clarion-plugin-fixture"`) plus a stub `lib.rs` whose only job is to let Cargo resolve the workspace member cleanly. Functionally a **test-only protocol-reference plugin**: a minimal, correct implementation of the L4 JSON-RPC wire contract used to drive the plugin host's subprocess code paths without needing the Python plugin on `PATH`. + +### Responsibility + +This crate owns the *smallest valid implementation* of the Clarion plugin protocol — just enough to (a) prove `clarion-core::plugin::host` can spawn a child, negotiate `initialize`/`initialized`/`analyze_file`/`shutdown`/`exit`, and round-trip framed JSON-RPC, and (b) deterministically misbehave on demand (RLIMIT_AS exhaustion via `CLARION_FIXTURE_EXCEED_RLIMIT_AS`) so the host's OOM-kill and crash-loop-breaker paths can be tested end-to-end. It is intentionally not a real analyzer: every `analyze_file` returns the same single hard-coded widget entity regardless of the input file's content. The crate's "public surface" is therefore (1) the binary itself, consumed by `cargo nextest` via `CARGO_BIN_EXE_clarion-plugin-fixture`, and (2) the fixture identity tuple (`plugin_id=fixture`, kind `widget`, qualname `demo.sample`, rule-ID prefix `CLA-FIXTURE-`) that tests assert against verbatim. + +### Key components + +- `src/main.rs:23-123` — the JSON-RPC dispatch loop: blocking `read_frame` → parse → match on `method` → either no-op (notification) or send a typed result envelope. Uses `clarion-core` framing and request/response types directly so the wire shape stays in lockstep with the host. +- `src/main.rs:67-76` — `initialize` handler: emits `InitializeResult { name, version, ontology_version: "0.1.0", capabilities: {} }`. +- `src/main.rs:77-115` — `analyze_file` handler: extracts `file_path` from params, returns one entity (`fixture:widget:demo.sample`, `kind=widget`, `qualified_name=demo.sample`, `source.file_path=`), no edges, default stats. +- `src/main.rs:48-60` — notification dispatch for `initialized` (becomes-ready, no reply) and `exit` (process-exits-0, no reply). +- `src/main.rs:78-83, 137-184` — the `CLARION_FIXTURE_EXCEED_RLIMIT_AS` escape hatch: on Unix, repeatedly `mmap_anonymous` doubling-size regions with `PROT_NONE` to blow past `RLIMIT_AS`, then `SIGKILL` self via `nix::sys::signal::kill`. The mappings are held but never dereferenced (documented `SAFETY` comment). +- `src/main.rs:125-135` — `send_result` helper: wraps a `Value` in `ResponseEnvelope { jsonrpc, id, payload: Result(...) }`, serialises, frames via `write_frame`, flushes. +- `src/lib.rs:1-3` — three-line doc-only stub explaining why the lib target exists. + +### Public interface (outbound) + +The plugin speaks the L4 JSON-RPC protocol on stdin/stdout. Methods implemented: + +- **`initialize`** (request) — returns `InitializeResult` with `ontology_version=0.1.0`, empty capabilities. `src/main.rs:68-76` +- **`initialized`** (notification) — accepted, no-op. `src/main.rs:51-53` +- **`analyze_file`** (request) — accepts `AnalyzeFileParams`, returns one canned entity. `src/main.rs:77-115` +- **`shutdown`** (request) — returns empty `ShutdownResult`. `src/main.rs:116-119` +- **`exit`** (notification) — `process::exit(0)`. `src/main.rs:54-56` + +Any unknown method, malformed frame, or unparseable JSON causes `process::exit(1)` (`src/main.rs:33-39, 46, 57, 64, 97, 120`). + +Companion manifest (consumed by host, not shipped by this crate): `crates/clarion-core/tests/fixtures/plugin.toml` declares `plugin_id="fixture"`, `language="fixture"`, `extensions=["mt"]`, `entity_kinds=["widget"]`, `edge_kinds=["uses"]`, `rule_id_prefix="CLA-FIXTURE-"`. + +### Dependencies + +- **Inbound (who consumes the binary):** + - `crates/clarion-core/tests/host_subprocess.rs` — direct subprocess test of `PluginHost::spawn`; locates the binary via `CARGO_BIN_EXE_clarion-plugin-fixture` (`host_subprocess.rs:30`) with a `cargo build` fallback (`:84-94`). Asserts entity id `fixture:widget:demo.sample` (`:162`). + - `crates/clarion-cli/tests/wp2_e2e.rs` — declares `clarion-plugin-fixture` as a `[dev-dependencies]` entry (`clarion-cli/Cargo.toml:43`) so nextest exports the env var, symlinks the binary into a synthetic plugin dir, and drives the full `clarion analyze` CLI through it. Tests: `wp2_e2e_smoke_fixture_plugin_round_trip` (line 135), `wp2_rlimit_as_oom_kill_is_reported_as_host_finding` (line 259, uses `CLARION_FIXTURE_EXCEED_RLIMIT_AS`), `wp2_crash_in_one_plugin_does_not_prevent_other_plugins_from_running` (line 323), `wp2_crash_loop_breaker_trips_and_skips_remaining_plugins` (line 471). +- **Outbound (what this calls):** + - `clarion-core::plugin::transport` — `read_frame` / `write_frame` / `Frame` for length-prefixed framing. + - `clarion-core::plugin::limits::ContentLengthCeiling::DEFAULT` (8 MiB per ADR-021 referenced in source comment, `main.rs:30-32`). + - `clarion-core::plugin` request/response DTOs: `AnalyzeFileParams`, `AnalyzeFileResult`, `AnalyzeFileStats`, `InitializeResult`, `ShutdownResult`, `JsonRpcVersion`, `ResponseEnvelope`, `ResponsePayload`. + - `serde_json` for ad-hoc `Value` parsing of the inbound request envelope. + - `nix` (Unix only, `mman` + `signal` features) — only for the deliberate-OOM probe. +- **External services:** None at runtime. Communicates only over inherited stdin/stdout with its parent (the plugin host). + +### Internal architecture + +The binary is a single synchronous loop in `main()` (`main.rs:23-123`); there are no modules, threads, async runtime, or background tasks. State machine is implicit and minimal: the loop accepts notifications and requests in any order after the host has spoken — there is no explicit "before-initialize" guard, which is acceptable because the host always sends `initialize` first and the fixture's responses are stateless. The shape is `loop { read_frame → parse Value → branch on (has_id, method) → dispatch }`. + +Error model is intentionally brutal: any deviation from the happy path (truncated frame, non-UTF8, missing method, unknown method, unparseable params) calls `std::process::exit(1)` with no reply. This is *desired* behaviour — `clarion-core`'s host has to handle a plugin that hangs up mid-stream, and crashing the fixture is the cheapest way to drive that path. The crash-loop-breaker test (`wp2_e2e.rs:471`) depends on this. + +The `exceed_rlimit_as` path (`main.rs:137-184`) is the only piece of nontrivial logic. It pre-reserves 1024 mapping handles before the memory pressure starts (so the `Vec::push` itself does not allocate after the kernel starts refusing maps), then loops `mmap_anonymous(PROT_NONE, MAP_PRIVATE)` doubling the request size each iteration starting at 128 MiB. PROT_NONE means no physical pages are committed — only address space is consumed, which is exactly what `RLIMIT_AS` constrains. When the next request would not grow (`saturating_mul(2)` saturated) or `mmap` returns `Err`, the fixture `SIGKILL`s itself so the parent observes a signal-death, not a clean exit. This is the load-bearing detail that distinguishes OOM-kill from a controlled shutdown in the host's diagnosis. + +`lib.rs` exists solely so the crate works as a workspace member with a binary target (3 lines, doc comment only). + +### Patterns observed + +- **Stub lib + real bin** (`lib.rs:1-3` + `Cargo.toml:12-14`) — workspace-member compatibility trick. +- **Untyped envelope parse, typed payload re-parse** (`main.rs:37-44, 93-98`) — read the whole frame as `serde_json::Value` to inspect `id`/`method` before committing to a typed struct, so unknown/malformed messages can be rejected without spurious deserialisation errors. +- **Crash-on-anomaly as a feature** (`main.rs:33-46, 57, 97, 120`) — every error path is `process::exit(1)`. Tests want this behaviour. +- **Hard-coded fixture identity** (`main.rs:101-108`) — `"fixture:widget:demo.sample"` is the ground-truth string tests assert on; do not change without updating both call sites listed under Inbound. +- **Environment-flag-driven fault injection** (`main.rs:78-83`) — `CLARION_FIXTURE_EXCEED_RLIMIT_AS` toggles the OOM path; default behaviour is benign. +- **PROT_NONE address-space probe** (`main.rs:137-178`) — exhausts virtual address space cheaply (no page commits) to trip a specific kernel limit deterministically. +- **Pre-reserved Vec to avoid allocation under memory pressure** (`main.rs:142-145`). + +### Concerns / Smells / Risks + +- **`process::exit(1)` everywhere with no diagnostic output.** Reasonable for a test fixture (the host is the system under test), but if a future test asserts on stderr or a specific exit code other than 1, every error branch will need disambiguating. No `eprintln!` anywhere. `main.rs:33-46, 57, 97, 120`. +- **No "initialize must come first" sequencing check.** A misbehaving host could call `analyze_file` before `initialize` and the fixture would happily respond. This is fine today because the host always sequences correctly, but the fixture is not a conformance checker — do not use it as one. +- **`unwrap()` on `serde_json::to_value(InitializeResult)`** (`main.rs:75`) and other result serialisations — defensible because the types are `Serialize`-derived, but pedantically these are crash points. +- **Unix-gated OOM path with a `cfg(not(unix))` arm that just `exit(1)`s** (`main.rs:78-83`) — Windows CI would silently lose coverage of the RLIMIT_AS branch. Acceptable since OOM-kill semantics are Unix-specific. +- **The manifest lives in another crate's test tree** (`clarion-core/tests/fixtures/plugin.toml`), not in this crate. Discoverable only by `grep`. Two callers (`host_subprocess.rs` and `wp2_e2e.rs`) construct their own variants of it. A `tests/fixtures/plugin.toml` here, re-used by both, would be cleaner; not urgent. +- **Hard-coded `version = "0.1.0"` and `ontology_version = "0.1.0"`** in the binary (`main.rs:71-72`) do not pick up the workspace version — if version-handshake logic ever depends on these, they will drift from `Cargo.toml`. Currently the host does not validate them. +- **No in-crate tests.** Justified: the fixture *is* the test apparatus for two upstream test files. Counting test coverage of this crate in isolation is meaningless. + +### Confidence: High + +Read all 187 source lines end-to-end, the companion manifest in `clarion-core/tests/fixtures/plugin.toml`, and grepped both consumer tests (`host_subprocess.rs`, `wp2_e2e.rs`) for every reference to `clarion-plugin-fixture`, `fixture:widget:demo`, and `CLARION_FIXTURE_EXCEED_RLIMIT_AS` to confirm the protocol surface and the four named test cases. Inbound dep edges verified via `Cargo.toml:43` of `clarion-cli`. Outbound dep edges verified by walking every `use clarion_core::` import in `main.rs`. diff --git a/docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-scanner.md b/docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-scanner.md new file mode 100644 index 00000000..fc77fa9a --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-scanner.md @@ -0,0 +1,86 @@ +## 5. clarion-scanner + +**Location:** `crates/clarion-scanner/` +**LOC:** 881 source (`lib.rs` 233 + `patterns.rs` 303 + `baseline.rs` 285 + `entropy.rs` 60) / 655 test (`tests/scanner.rs`) +**Crate type / role:** Library crate (`pub` API consumed by `clarion-cli`); pure CPU, no I/O except baseline YAML read. + +### Responsibility + +Owns Clarion's pre-ingest secret-detection pass: given an in-memory byte buffer, emit a deduplicated, byte-offset-anchored list of `Detection` records identifying secret-shaped substrings, plus a YAML-backed baseline mechanism that suppresses operator-acknowledged matches at the `(file, rule_type, hashed_secret, line_number)` granularity. The crate is deliberately scoped to detection + suppression — it does not walk the filesystem, decide which files to scan, emit findings, or report results; those concerns live in `clarion-cli/src/secret_scan/*`. Stores only positions, rule identifiers, and a detect-secrets-compatible SHA-1 digest of matched bytes — literal secret values do not leave the call (`lib.rs:1-5`). + +### Key components + +- `src/lib.rs:23-46` — `Detection` struct + `SecretCategory` enum: closed taxonomy with 9 categories (CloudCredential, VcsCredential, AiProviderCredential, PaymentsCredential, MessagingCredential, PrivateKey, JwtToken, HighEntropy, ContextualCredential). +- `src/lib.rs:49-99` — `HashedSecret([u8; 20])` newtype: SHA-1 digest with hex round-trip (`from_hex`, `Display`); decoupled from `sha1::Sha1` impl detail at the public surface. +- `src/lib.rs:102-200` — `DetectSecretsRule` enum: 14-variant closed vocabulary aligned to detect-secrets type names; `as_str()`/`rule_id()`/`FromStr` provide bidirectional mapping between human label (baseline YAML) and stable rule-id string (findings). +- `src/patterns.rs:25-71` — `Scanner` struct: pre-compiles a `RegexSet` (fast first-pass match) plus per-pattern `Regex` (for captures), holds entropy regexes and tunings; `Default`/`new()` build the ADR-013 v0.1 floor. +- `src/patterns.rs:79-162` — `scan_bytes()` + `scan_entropy()`: two-pass detection — named patterns first, entropy fallback over non-overlapping ranges; outputs sorted by `(byte_offset, rule_id)`. +- `src/patterns.rs:194-269` — `default_pattern_meta()`: the 12-entry rule floor (literal source of detection truth). +- `src/baseline.rs:11-44` — `Baseline`, `BaselineEntry`, `BaselineMatch`, `SuppressionResult` types: model the suppression file shape and the result envelope (allowed/suppressed/fired). +- `src/baseline.rs:104-144` — `Baseline::suppress()`: O(detections × entries_per_file) match using exact `(hashed_secret, line_number, rule_type)` triple as the suppression key. +- `src/baseline.rs:146-217` — `from_raw()` validation pipeline: version check (`"1.0"` only), path safety (`validate_baseline_path`), mandatory `justification`, hex-hash validity, closed rule-type vocabulary. + +### Public interface (outbound) + +Re-exported through `lib.rs:11-16`: + +- `Scanner` (`patterns.rs:25`) — owns compiled regexes; cheap to construct once and reuse. Method: `scan_bytes(&self, buf: &[u8]) -> Vec`. +- `Detection` (`lib.rs:23-32`) — one match: `rule_id`, `detect_secrets_type`, `category`, `byte_offset`, `line_number`, `matched_len`, `hashed_secret`. +- `DetectSecretsRule`, `SecretCategory` (`lib.rs:36-46`, `102-118`) — closed enums for downstream pattern-matching. +- `HashedSecret`, `HexDigestError` (`lib.rs:49-99`) — opaque hash newtype. +- `Baseline`, `BaselineEntry`, `BaselineEntryIssue`, `BaselineMatch`, `BaselineError`, `SuppressionResult` (`baseline.rs`) — operator-baseline model + error taxonomy with 7 variants (`UnsupportedVersion`, `MissingJustifications`, `InvalidPath`, `InvalidHash`, `UnsupportedRuleType`, `Parse`, `Io`). +- `load_baseline(&Path) -> Result` (`baseline.rs:71-77`) — accepts a missing file as `Baseline::empty()` (graceful absence). +- `EntropyTuning` (`entropy.rs:5-23`) — exposed but only `BASE64` and `HEX` consts are constructed internally; `min_len` + `min_entropy` fields. +- `PatternMeta` (`patterns.rs:10-15`) — exposed via `Scanner::pattern_meta()`, primarily for introspection. + +### Dependencies + +- **Inbound (who calls this):** Only `clarion-cli`: + - `clarion-cli/src/secret_scan.rs` imports `Detection`, `Scanner`, `SuppressionResult`. + - `clarion-cli/src/secret_scan/baseline.rs` imports `Baseline`, `BaselineError`. + - `clarion-cli/src/secret_scan/findings.rs` imports `Detection`, `SecretCategory`, `DetectSecretsRule`, `HashedSecret`. +- **Outbound (what this calls):** No internal Clarion crates. External: `regex` (bytes flavour), `sha1`, `serde` + `serde_norway` (YAML), `thiserror`. +- **External services:** None directly. Reads the baseline file via `std::fs::read_to_string` (`baseline.rs:72`); no other I/O. Pure-function `scan_bytes` over an `&[u8]`. + +### Internal architecture + +Three modules + `lib.rs` umbrella. `lib.rs` owns the shared value types (`Detection`, `HashedSecret`, `DetectSecretsRule`, `SecretCategory`) and two tiny helpers — `sha1_digest` (`lib.rs:208-214`) and `line_number_for_offset` (`lib.rs:216-224`, a byte-by-byte newline count over the prefix). Concurrency is **none** — there is no shared mutable state; `Scanner` is `Send + Sync` by construction (only immutable compiled regexes), so callers parallelize at the file-level outside the crate (and `clarion-cli/src/secret_scan.rs:scan_source_files_parallel` does exactly that). + +`patterns.rs` runs detection in two layers. Layer 1: `RegexSet` first-pass (`patterns.rs:80`) over the whole buffer — only patterns whose set-membership matched then have their full `Regex` run for capture extraction (`patterns.rs:83-87`). Layer 2: entropy fallback (`scan_entropy`, `patterns.rs:128-161`) runs the base64 candidate regex `[A-Za-z0-9+/]{20,}={0,2}` and hex candidate `\b[a-fA-F0-9]{40,}\b`, skipping candidates that **overlap any already-found named match** (`range_overlaps`, `patterns.rs:271-275`). The base64 fallback additionally requires non-base64 boundary bytes on each side (`base64_candidate_has_boundaries`, `patterns.rs:277-281`) — a hand-rolled `\b`-equivalent because `=` is not a word character. + +Entropy is parameterized by two `EntropyTuning` constants: +- `BASE64`: `min_len = 20`, `min_entropy = 4.5` (`entropy.rs:11-14`). +- `HEX`: `min_len = 40`, `min_entropy = 3.0` (`entropy.rs:15-18`). + +Entropy itself is Shannon entropy in bits/symbol over byte frequency (`entropy.rs:25-41`) using a `BTreeMap` count table, computed as `-Σ p_i log2(p_i)`. These thresholds are deliberately wide: tests `entropy_minimum_lengths_are_pinned` (`tests/scanner.rs:171-174`) and the lockfile-SHA fixture (`tests/scanner.rs:568-638`) document that the hex threshold *intentionally* fires on git SHAs and npm lockfile integrity hashes — the v0.1 stance is to suppress via baseline rather than tighten the rule and risk missing real secrets. + +The `KeywordDetector` rule (`patterns.rs:262-267`) is contextual: a Python/`.env`-shaped `name = "value"` assignment with name ∈ {`password`, `passwd`, `secret`, `token`, `api_key`, `secret_token`}, captured value ≥ 8 chars. To avoid false positives on commented-out examples, `scan_bytes` filters `ContextualCredential` matches whose line begins with `#` (`patterns.rs:91-94, 290-303`). The comment-handling is explicitly Python/shell-only — `//` and `/* */` comments are *not* skipped (tests at `tests/scanner.rs:162-167` lock this in deliberately). + +`baseline.rs` is a value-typed YAML parser + suppression engine. The on-disk shape is a `version: "1.0"` envelope with `results: {path: [entry, ...]}` (`baseline.rs:237-253`). Parse-time validation rejects: non-1.0 versions, absolute or `..`-bearing paths, missing/blank `justification` (collected exhaustively — `from_raw` walks all entries before returning the error, see `baseline.rs:153-172` and the test at `tests/scanner.rs:408-433`), unknown detector-type strings, and invalid hex hashes. Suppression is deliberately exact-match: same path, same rule, same hash, same line number — *and* `is_secret: false` (a `true` entry, or omitted field defaulting to `true` per `default_is_secret` at `baseline.rs:255-257`, does **not** suppress; tests at `tests/scanner.rs:300-385` lock this in). + +Error model is `thiserror`-derived (`baseline.rs:45-69`) with no panics in the parse path. The detection path has two `expect` calls in `Scanner::new` (`patterns.rs:51, 56-57, 66-69`) — all on compile-time constant regex literals, so they fire only on programmer error during edits, not on runtime input. + +### Patterns observed + +- **Two-phase regex matching** (`patterns.rs:80-87`) — `RegexSet` for cheap any-match prefilter, individual `Regex` only for matched indices. Trades memory for avoiding O(rules × text) capture work. +- **Closed enum at the interface, string only at the wire** — `DetectSecretsRule` (`lib.rs:102-118`) forces every consumer to handle the full vocabulary; conversions live at the YAML boundary (`baseline.rs:179-186`) so unknown types fail fast. +- **Capture-group routing via `Option`** (`patterns.rs:14, 96-103`) — `KeywordDetector` and `AwsSecretAccessKey` capture an inner group so the hash is over just the secret literal, not the `name="value"` framing. +- **Boundary-aware regex via helper functions** (`patterns.rs:277-303`) — Rust's `regex` crate's `\b` doesn't handle base64 padding `=` or `#`-comments, so the crate composes regex + post-filter. +- **Exhaustive error collection** for `MissingJustifications` (`baseline.rs:153-172`) — operator sees all missing entries at once, not one per re-run. +- **Path safety as a parse-time invariant** (`baseline.rs:219-235`) — absolute paths, `..`, drive prefixes all rejected at load; eliminates a class of suppression-escape attacks at the YAML boundary rather than at match time. +- **Cross-tool format compatibility by design** — the SHA-1 hash, the rule-name vocabulary, and the YAML schema all mirror Yelp's `detect-secrets` baseline format (cited explicitly in `lib.rs:1-5`), so operators familiar with that tool can reuse baselines. + +### Concerns / Smells / Risks + +- **`scan_bytes` is O(named_detections × entropy_candidates)** via `range_overlaps`'s linear scan (`patterns.rs:271-275`). For a file with hundreds of named matches and hundreds of entropy candidates this is quadratic. No interval tree, no sort-and-binary-search. For the target corpus (425k LOC Python, ADR-013 mentions) this is likely fine but worth flagging if scanner runtime ever becomes an issue. +- **`Baseline::suppress` is O(detections × entries_per_file)** with linear scan over the entries for that path (`baseline.rs:111-137`). Acceptable while baseline files are small (~tens of entries) but no index by `(rule, hash, line)`. +- **`usize_to_f64` truncates above 2³² bytes** (`entropy.rs:43-45`). For >4 GB files entropy will compute as if the buffer were 2³² bytes. Practically irrelevant — file-level scanning is bounded long before this — but the silent saturation deserves a comment. +- **Contextual-comment handling is Python/`.env`-only** (`patterns.rs:290-303`). A `// password = "..."` line in a JavaScript file *will* fire `ContextualCredential` (locked in by test at `tests/scanner.rs:163-167`). The crate's docstring acknowledges this; the cost is operator-baseline pollution in non-Python codebases until detector context is added. +- **`Scanner::new` panics on regex compile failure** (`patterns.rs:48, 51, 56, 66, 69`). These are compile-time literals so this is unreachable in shipping builds, but the public API has no fallible constructor — a future operator-provided pattern set would need a new constructor surface. +- **Entropy thresholds embed false-positive policy in code, not configuration** (`entropy.rs:11-18`). The test fixture explicitly documents that lockfile/git-SHA noise is *intentional* and the answer is operator baseline. This is a deliberate v0.1 trade-off but it pushes calibration burden to every operator until configurable thresholds or per-file-extension tuning lands. +- **`PatternMeta::capture_group` is private but the type is `pub`** (`patterns.rs:10-15`) — external consumers can read the struct via `Scanner::pattern_meta()` but can't construct one. This is fine as a closed API but the asymmetry is slightly awkward. +- **No fuzz or property tests** in `tests/scanner.rs` — only example-based assertions. Regex engines under pathological input (catastrophic backtracking) are a known foot-gun; the crate uses Rust's `regex` (which is linear-time by construction) so this is mitigated, but a quickcheck pass would be cheap insurance. + +### Confidence: High + +Read all 4 source files end-to-end (881 LOC), the full 655-line test file, the crate's `Cargo.toml`, and cross-checked inbound usage by greping the workspace for `clarion_scanner` imports (only `clarion-cli/src/secret_scan/*` and the scanner's own tests). Confirmed pre-ingest invocation timing at `clarion-cli/src/analyze.rs:237-243` (runs before plugin-host extraction, feeds `briefing_blocks` into the rest of the pipeline). Test file is unusually rich — it locks in not only positive/negative detection cases but also the *intentional* false positives (lockfile SHAs) with the documented operator-baseline workaround. No mystery code remains. diff --git a/docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-storage.md b/docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-storage.md new file mode 100644 index 00000000..bfd7cadc --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/temp/catalog-clarion-storage.md @@ -0,0 +1,79 @@ +## 2. clarion-storage + +**Location:** `crates/clarion-storage/` +**LOC:** 3199 src across 10 files / 4871 tests across 5 files / 293 lines SQL in `migrations/0001_initial_schema.sql` +**Crate type / role:** Library crate (`lib.rs:1-42`). Sole owner of the SQLite read/write layer for the project DB at `.clarion/clarion.db`. Re-exports a flat surface from nine submodules. + +### Responsibility +`clarion-storage` is the only path through which other crates touch SQLite. It (a) opens and configures the per-project database with a fixed PRAGMA discipline (`pragma.rs`), (b) applies embedded schema migrations idempotently (`schema.rs`), (c) routes *every* mutation through a single writer-actor task that owns the sole write `rusqlite::Connection` (`writer.rs`), (d) hands out short-lived read-only connections from a `deadpool-sqlite` pool (`reader.rs`), and (e) provides the read-side query helper catalogue that the MCP navigation tools and HTTP read API consume (`query.rs`, ~29 `pub` query / type items). The crate also enforces wire-level invariants the rest of the workspace depends on — per-kind edge confidence/source-range contracts, source-file-anchor kinds, reserved entity-kind protection, parent↔contains-edge dual-encoding consistency — at the *writer boundary*, so a misbehaving plugin or caller cannot corrupt graph shape (`writer.rs:425-582, 954-1021`). + +### Key components +- `src/lib.rs:1-42` — module wiring + flat `pub use` facade (`Writer`, `ReaderPool`, `WriterCmd`, ~29 query helpers, cache helpers, `StorageError`). +- `src/pragma.rs:16-45` — `apply_write_pragmas` (WAL + `synchronous=NORMAL` + `busy_timeout=5000` + `wal_autocheckpoint=1000` + `foreign_keys=ON`, with a hard invariant assertion that `PRAGMA journal_mode=WAL` actually took effect) and `apply_read_pragmas` (busy_timeout + FK only). +- `src/writer.rs:40-140` — `Writer` handle, `Writer::spawn` (boots the actor as a `tokio::task::spawn_blocking` task owning one `rusqlite::Connection`), `send_wait` convenience. +- `src/writer.rs:142-260` — `run_actor` central match loop dispatching all 11 `WriterCmd` variants; blocking `mpsc::Receiver::blocking_recv`. +- `src/writer.rs:290-313, 802-877` — `ActorState` (`batch_size`, `writes_in_batch`, `in_tx`, `current_run`) and `bump_writes_and_maybe_commit` / `flush_run_batch` / `query_time_write` — the batch-cadence and run-transaction state machine. +- `src/writer.rs:425-582` — wire-contract enforcement: `enforce_entity_kind_contract`, `validate_entity_source_file_anchor`, `enforce_edge_contract`, plus the structural-vs-anchored edge-kind tables (`STRUCTURAL_EDGE_KINDS`, `ANCHORED_EDGE_KINDS`). +- `src/writer.rs:954-1021` — `parent_contains_mismatch` (the bidirectional parent↔contains dual-encoding check; runs inside the open commit transaction so a violation rolls back the run). +- `src/reader.rs:26-119` — `ReaderPool` (`deadpool-sqlite` wrapper with an `Arc<()>` identity tag for `shares_pool_with` runtime proofs) and `with_reader` async helper. +- `src/schema.rs:17-91` — `MIGRATIONS` slice (one entry, embedded by `include_str!`), `apply_migrations` runner gated by a `schema_migrations` table. +- `src/commands.rs:135-218` — the `WriterCmd` enum: `BeginRun`, `InsertEntity`, `InsertEdge`, `InsertFinding`, `FlushRunBatch`, `InsertInferredEdges`, `UpsertSummaryCache`, `TouchSummaryCache`, `ReplaceUnresolvedCallSitesForCaller`, `CommitRun`, `FailRun` (11 variants, each carries an `Ack = oneshot::Sender>`). +- `src/query.rs:214-1056` — read helpers (`entity_by_id`, `entity_at_line`, `find_entities`, `call_edges_from/_targeting`, `reference_edges_for_entity`, `module_dependency_edges`, `subsystem_members`, `resolve_file_catalog_entry`, `contained_entity_ids`, …). +- `migrations/0001_initial_schema.sql:23-291` — 9 tables (`schema_migrations`, `entities`, `entity_tags`, `edges` `WITHOUT ROWID` on natural PK, `findings`, `summary_cache`, `inferred_edge_cache`, `entity_unresolved_call_sites`, `runs`), 1 FTS5 virtual table (`entity_fts`), 3 triggers keeping FTS in sync, 2 generated virtual columns (`scope_rank`, `git_churn_count`) with partial indexes, 1 view (`guidance_sheets`), ~15 secondary indexes, and a row inserted into `schema_migrations`. Wrapped in a single `BEGIN…COMMIT`. + +### Public interface (outbound) +- **Writer surface** — `Writer::spawn(db_path, batch_size, channel_capacity) -> (Writer, JoinHandle>)`; `Writer::sender() -> mpsc::Sender`; `Writer::send_wait`; observability counters `commits_observed`, `dropped_edges_total`, `ambiguous_edges_total` (each an `Arc`, present in release builds). Constants `DEFAULT_BATCH_SIZE = 50`, `DEFAULT_CHANNEL_CAPACITY = 256` (`writer.rs:34-38`). +- **Command protocol** — the `WriterCmd` enum (11 variants), the four POD record types (`EntityRecord`, `EdgeRecord`, `FindingRecord`, `InferredCallEdgeRecord`), `RunStatus` (`SkippedNoPlugins | Completed | Failed`), `InferredEdgeWriteStats`, and `Ack`. +- **Reader surface** — `ReaderPool::open(db_path, max_size)`, `with_reader(impl FnOnce(&Connection) -> Result) -> Result`, `shares_pool_with`, `identity()`, `waiting_count()` (test-hook). +- **Schema runner** — `schema::apply_migrations(&mut Connection)`, `schema::applied_count(&Connection)` (used by `clarion install`). +- **Cache surface** — `summary_cache_lookup`/`upsert_summary_cache`/`touch_summary_cache`, `inferred_edge_cache_lookup`/`upsert_inferred_edge_cache`/`touch_inferred_edge_cache`, `inferred_edge_cache_key_id` (canonical key-id format `"caller|hash|model|prompt"`), plus the four POD types `SummaryCacheKey/Entry`, `InferredEdgeCacheKey/Entry` (`cache.rs:1-251`). +- **Unresolved-edges surface** — `UnresolvedCallSiteRecord`, `replace_unresolved_call_sites_for_caller(&Connection, …)` (atomic DELETE-then-INSERT per caller_entity_id; `unresolved.rs:20-49`). +- **Query helpers** — ~29 `pub` items in `query.rs`: row structs (`EntityRow`, `CallEdgeMatch`, `ContainedEntities`, `ResolvedFile`, `ResolvedFileCatalogEntry`, `ModuleDependencyEdge`, `SubsystemMember`, `UnresolvedCallSiteRow`, `ReferenceEdgeMatch`, `ReferenceDirection`, `CanonicalProjectPath`), and lookup functions (`entity_by_id`, `entity_at_line`, `find_entities`, `existing_entity_ids`, `call_edges_from`, `call_edges_targeting`, `reference_edges_for_entity`, `module_dependency_edges`, `subsystem_members`, `subsystem_for_member`, `unresolved_call_sites_for_caller`, `unresolved_callers_for_target`, `candidate_entities_for_unresolved_sites`, `contained_entity_ids`, `child_entity_ids`, `resolve_file`, `resolve_file_catalog_entry`, `entity_briefing_block_reason`, `normalize_source_path`). +- **Error type** — `StorageError` (`error.rs:5-46`) with `is_foreign_key_violation()` classifier for callers (notably the MCP envelope) that need `retryable=false` on FK breaches. +- **Helper inventory** — `known_scan_time_edge_kinds()` iterator (`writer.rs:506-511`) exposes the 9-kind ontology to callers that need to validate before sending. + +### Dependencies +- **Inbound (who calls this):** `clarion-cli` (`analyze.rs`, `serve.rs`, `install.rs`, `http_read.rs`, `run_lifecycle.rs`, `secret_scan.rs` + submodules `anchors.rs`/`findings.rs`); `clarion-mcp` (`lib.rs`, `tests/storage_tools.rs`). No other Clarion crate touches it. +- **Outbound (what this calls):** `clarion-core` for `EdgeConfidence` (`commands.rs:14`) and `manifest::RESERVED_ENTITY_KINDS` (`writer.rs:427`). Cargo-level: `rusqlite` 0.31 (`bundled`), `deadpool-sqlite` 0.8 (`rt_tokio_1`), `tokio` (`sync`, `task::spawn_blocking`), `serde`/`serde_json` (validate `properties_json` shape on inferred edges), `blake3` (declared dep but not imported in `src/`; likely intended for path/hash helpers), `tracing`, `thiserror`. +- **External services:** SQLite only — one file at `.clarion/clarion.db`, accessed via two distinct connection populations: one writer connection owned by the actor task, and `max_size` reader connections in `deadpool-sqlite`. No network. WAL is the cross-process coordination mechanism between writer and readers. + +### Internal architecture + +**Concurrency model — single-writer actor + reader pool (ADR-011, repeated verbatim in `lib.rs:1-5`).** All persistent mutations are funnelled through one `tokio::task::spawn_blocking` task that owns the sole write `rusqlite::Connection` (`writer.rs:86-98`). Producers communicate via a bounded `tokio::sync::mpsc::Sender` (default capacity 256); each variant carries a `oneshot::Sender` for the per-command ack so callers can fan out N producers (`Writer::sender().clone()`) and still observe each command's outcome (`commands.rs:20`, `writer.rs:998-1060` test `cloned_senders_accept_concurrent_entity_producers`). The actor loop is `rx.blocking_recv()` on the spawn_blocking thread (`writer.rs:152`), not `await` — this is correct because the loop is blocking-thread-resident and must call synchronous `rusqlite` APIs. + +**Transaction discipline — per-N-writes batches inside a per-run super-transaction.** `BeginRun` issues `INSERT INTO runs … status='running'` then `BEGIN`. Every successful `InsertEntity`/`InsertEdge`/`InsertFinding`/`ReplaceUnresolvedCallSitesForCaller` increments `state.writes_in_batch`; when it hits `batch_size` (default 50), `bump_writes_and_maybe_commit` issues `COMMIT` then `BEGIN` immediately to re-open. `CommitRun` runs the parent↔contains-edge consistency check inside the still-open transaction (`writer.rs:894-918`), then atomically folds the `UPDATE runs SET status=… completed_at=… stats=…` into the same `COMMIT`. `FailRun` issues `ROLLBACK` then a separate single-statement run-row UPDATE. The actor also has a `cleanup_after_channel_close` (`writer.rs:262-281`) that rolls back any open tx and marks the run failed if the channel is dropped mid-run. `FlushRunBatch` exists to let readers on separate SQLite connections observe in-flight graph rows mid-run by committing and re-opening the batch. The query-time MCP writes (`InsertInferredEdges`, `UpsertSummaryCache`, `TouchSummaryCache`) commit the surrounding run-batch first (if any), execute outside the run transaction, then re-open the run batch — they explicitly do not require an active run (`writer.rs:855-877`). + +**Write-side wire contracts (enforced before any SQL).** `enforce_entity_kind_contract` rejects non-`core` plugins emitting `RESERVED_ENTITY_KINDS` (`writer.rs:425-438`). `enforce_edge_contract` (`writer.rs:521-582`) splits the 9-kind ontology into `STRUCTURAL_EDGE_KINDS` (`contains`, `in_subsystem`, `guides`, `emits_finding` — must be `confidence=resolved`, must have NULL byte ranges) and `ANCHORED_EDGE_KINDS` (`calls`, `references`, `imports`, `decorates`, `inherits_from` — must have both byte-start and byte-end, must NOT be `inferred` at scan time); any unknown kind raises `CLA-INFRA-EDGE-UNKNOWN-KIND`. `validate_source_file_anchor` (`writer.rs:444-493`) checks that any `source_file_id` reference points to an entity whose kind is `file` or `module` (the comment at line 442 calls this transitional until core-minted `file` entities land). `parent_contains_mismatch` (`writer.rs:958-1021`) runs two SQL queries — one in each direction — to assert that `entities.parent_id` and `edges WHERE kind='contains'` are bijective; failure aborts the entire run with code `CLA-INFRA-PARENT-CONTAINS-MISMATCH`. + +**Idempotence and dedupe semantics.** `InsertEntity` uses `ON CONFLICT(id) DO UPDATE` that preserves `created_at` + `first_seen_commit` while refreshing `updated_at` + `last_seen_commit` (`writer.rs:362-420`); this is what makes `clarion analyze` re-run-safe. `InsertEdge` uses `INSERT OR IGNORE` on natural PK `(kind, from_id, to_id)`; dedupes increment `dropped_edges_total`. `InsertInferredEdges` first DELETEs the caller's existing inferred-tier `calls` edges that don't match the current cache key, then inserts new ones while *also* short-circuiting any pair that already has a `resolved`/`ambiguous` edge (`static_call_edge_exists`, lines 758-771) — that's how query-time LLM inference cohabits with scan-time static facts without double-counting. + +**PRAGMA discipline (`pragma.rs:16-45`).** Writer connection: `journal_mode=WAL` (and aborts with `PragmaInvariant` if SQLite returns anything other than `wal`), `synchronous=NORMAL`, `busy_timeout=5000` (5s — but the writer is single-threaded so this only matters against external `sqlite3` shell access), `wal_autocheckpoint=1000` (auto-checkpoint every 1000 frames), `foreign_keys=ON`. Reader connections: only `busy_timeout=5000` + `foreign_keys=ON`, re-applied on every `with_reader` acquisition as a belt-and-suspenders measure since `deadpool-sqlite` has no post-create hook. No `mmap_size`, no `temp_store`, no `cache_size`, no `application_id`, no `user_version` — schema versioning is done in the application-level `schema_migrations` table only. + +**Schema migrations (`schema.rs:17-91`).** Single compile-time-embedded migration via `include_str!`. The runner reads applied versions from `schema_migrations`, tolerating the table's absence via `OptionalExtension::optional()` (with an explicit comment at lines 45-49 warning that `.ok()` would silently mask `DatabaseLocked` / `CorruptDb`). Each migration runs in its own `execute_batch` and then an `INSERT OR IGNORE INTO schema_migrations` belt-and-suspenders insert. The migration SQL itself wraps everything in `BEGIN…COMMIT` and inserts its own row. No `application_id` / `user_version` is set — drift / cross-tool collisions on the SQLite file are not detected at the DB level. + +**Error model (`error.rs:5-65`).** One `thiserror`-derived enum, ten variants, three derived from external crates (`rusqlite::Error`, three `deadpool_sqlite::*Error` types, `std::io::Error`); six application-shaped (`PragmaInvariant`, `Migration{version, source}`, `InvalidQuery`, `InvalidSourcePath`, `WriterGone`, `WriterProtocol`, `WriterNoResponse`). `is_foreign_key_violation` is a public classifier on SQLite extended code 787, documented as feeding the MCP envelope's `retryable=false` decision. + +### Patterns observed +- **Actor + bounded mpsc channel + oneshot ack-per-command** (`writer.rs:74-140`, `commands.rs:20`). Producers can be cloned freely; back-pressure is the channel's `send().await`. +- **Per-connection PRAGMA reapply on every reader-pool acquisition** as a workaround for `deadpool-sqlite` having no post-create hook (`reader.rs:96-107`, comment at lines 76-83). +- **`Arc<()>` identity tag** for proving two `ReaderPool` clones share the same in-process pool, distinct from "same file path" (`reader.rs:25-70`, used by `clarion-cli/src/serve.rs:63-71`'s `Arc::ptr_eq` assertion). +- **Plain-old-data records at the wire** (`commands.rs:48-133`): `EntityRecord` / `EdgeRecord` / `FindingRecord` / `InferredCallEdgeRecord`. Caller is responsible for timestamps, content_hash, JSON-string encoding; writer inserts verbatim. No serde derivation on these. +- **Codified error subcodes embedded in error messages** (`CLA-INFRA-RESERVED-ENTITY-KIND`, `CLA-INFRA-SOURCE-FILE-MISSING`, `CLA-INFRA-SOURCE-FILE-KIND-CONTRACT`, `CLA-INFRA-EDGE-CONFIDENCE-CONTRACT`, `CLA-INFRA-EDGE-SOURCE-RANGE-CONTRACT`, `CLA-INFRA-EDGE-UNKNOWN-KIND`, `CLA-INFRA-PARENT-CONTAINS-MISMATCH`). Strings are load-bearing for downstream `runs.stats.failure_reason` parsing. +- **`WITHOUT ROWID` on the natural-key `edges` table** (migration line 96) and `WITHOUT ROWID`-style PKs on the three cache/unresolved tables — composite PKs are the access path. +- **FTS5 virtual table kept in sync by AFTER {INSERT,UPDATE,DELETE} triggers** (`migrations:209-238`); summary text is extracted from `summary.briefing.purpose` JSON path. +- **Generated VIRTUAL columns over JSON properties + partial indexes** (`scope_rank`, `git_churn_count`, lines 245-262) — the CASE-mapped `scope_rank` exists specifically because the project→subsystem→…→function ordering is not lexicographic. +- **Embedded migrations + edit-in-place policy until first external operator** (migration header lines 10-16) — comment names the retirement trigger for the policy. + +### Concerns / Smells / Risks +- **`query.rs` is 1161 LOC of mostly hand-written SQL strings**. No prepared-statement caching at the module level, no query builder — every helper builds its own SQL with `params!`/`params_from_iter`. Risk: easy to drift between similar helpers; partially mitigated by the 1137-LOC `tests/query_helpers.rs` (close to 1:1 LOC with the production file). Reviewers must check each new query for FK/composite-PK access path. +- **No `application_id` / `user_version` discipline**. The migration runner is the *only* schema-identity signal — a non-Clarion SQLite file opened at `.clarion/clarion.db` would pass `apply_migrations` if it happened to lack a `schema_migrations` table, and would *fail confusingly* if it has unrelated tables that collide. Cross-tool DB-file drift is not detectable at open time. +- **`blake3` is a declared dep but unused inside `src/`** (verified by `grep -rn blake3 src/` — no hits). Likely vestigial from an earlier hashing plan; not load-bearing but is build-time waste. +- **`busy_timeout=5000` is set on *both* connections, but the writer is single-thread**. The 5s only matters if an external `sqlite3` shell or a second Clarion process opens the file. There is no in-process advisory lock (no `cross-process claim-lease` table) — two `clarion analyze` runs against the same `.clarion/clarion.db` would race on `runs` rows and the SQLite writer mutex; the `recover_preexisting_running_runs` step in `clarion-cli/src/run_lifecycle.rs` is the only mitigation, and it's post-hoc. +- **`writer.rs` is 1074 LOC in one file with ~25 free functions plus the `ActorState` struct**. The match-and-dispatch loop is large; adding a 12th command means touching the `WriterCmd` enum, the match in `run_actor`, *and* the per-command handler — three places. The pattern is clean but the file is approaching "split me" territory. +- **`reader_pool` test (line 91) uses `max_size = 1`** as the *exhaustion scenario* — fine for the test, but worth flagging that the real production pool sizes in callers are 16 (`serve.rs`) and 4 (`http_read.rs`'s test seam), so the worst-case queue depth under MCP load is `64` (from `ConcurrencyLimitLayer`) waiting on `16` connections plus one writer holding the FS lock — the math hasn't been load-tested in this crate's tests. +- **`InsertEntity`/`InsertEdge` validate source-file anchors with a `SELECT kind FROM entities WHERE id = ?` per write** (`writer.rs:451-458`). At 500k entities this is 500k extra round-trips during analyze. The query is single-row PK lookup so it's cheap, but it's per-write overhead that the test corpus may not exercise to scale. +- **Tests are well above 1:1 LOC ratio with production code** (4871 test : 3199 src). Behavioural coverage looks strong (round-trip, idempotence, FK propagation, edge contracts, FailRun rollback, channel-close cleanup, query helpers, cache lifecycle, reader pool concurrency / queueing / panic recovery, schema-apply re-run safety). Concern: there is no fuzz/property-test seam for `WriterCmd` interleavings — only scripted scenarios. The `commits_observed` counter is the canonical batch-cadence oracle and is asserted in `writer_actor.rs:943` (`batch_size_fifty_commits_every_fifty_inserts`), but the channel-close-mid-run cleanup path (`cleanup_after_channel_close`, `writer.rs:262-281`) does not appear to have a dedicated test asserting the run-row is left in `failed` rather than `running`. +- **`FlushRunBatch` exists for cross-connection visibility but its consumers are external to this crate**. Within `clarion-storage` no test exercises a reader observing a flushed-but-uncommitted-run batch — that contract is asserted from `clarion-cli` / `clarion-mcp` tests instead, so a regression in `FlushRunBatch` semantics would surface as a downstream test failure rather than a unit-level one here. + +### Confidence: High +Read all 10 source files in `src/` end-to-end, plus `migrations/0001_initial_schema.sql` end-to-end, plus skimmed test-function headers in all 5 test files (notably `writer_actor.rs` 2440 LOC). Cross-checked inbound callers via `grep -rn "use clarion_storage"` — only `clarion-cli` (7 files) and `clarion-mcp` (lib + one test) import it. Confirmed `Writer::spawn` and `ReaderPool::open` call sites in `clarion-cli/src/{analyze,serve,http_read}.rs`. The only items I deliberately did not exhaust are the bodies of the read-side query helpers in `query.rs` past line 220 — I read the public surface and signatures, sampled `entity_at_line`/`find_entities`/`call_edges_from` lightly, and trusted the test file (1137 LOC of `query_helpers.rs`) as evidence of behavioural coverage rather than re-deriving each SQL statement. diff --git a/docs/arch-analysis-2026-05-22-1924/temp/catalog-python-plugin.md b/docs/arch-analysis-2026-05-22-1924/temp/catalog-python-plugin.md new file mode 100644 index 00000000..4855f2bc --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/temp/catalog-python-plugin.md @@ -0,0 +1,69 @@ +## 7. Python language plugin (`clarion-plugin-python`) + +**Location:** `plugins/python/` +**LOC:** 3028 source / 3440 test (`extractor.py` 932, `pyright_session.py` 1406, `server.py` 296, the rest ≤ 75 each) +**Crate type / role:** Standalone PEP 517 package (hatchling) installing the `clarion-plugin-python` console script; hosts an L4 JSON-RPC plugin process driven by the Rust core over stdio. Ships its `plugin.toml` to `share/clarion/plugins/python/plugin.toml` via `pyproject.toml:38-44`. + +### Responsibility +Owns Python-source ingestion for Clarion: parses `*.py` to an AST, emits `module` / `class` / `function` entities with stable 3-segment EntityIds, anchors structural `contains` and `imports` edges directly, and delegates type-resolved `calls` / `references` edges to a managed pyright-langserver subprocess. It is the *only* component in the workspace that holds Python `ast` semantics and the only Loom-suite-facing surface that probes Wardline (`wardline_probe.py:35-56`) for the L8 integration handshake. Public surface to the core is exactly the five JSON-RPC methods declared in `server.py:237-240` plus the manifest at `plugin.toml`. + +### Key components +- `src/clarion_plugin_python/__main__.py:14-15` — entry point bound to `[project.scripts] clarion-plugin-python = ...:main`; installs stdio guard then runs `serve()`. +- `src/clarion_plugin_python/server.py:73-128, 143-232, 243-296` — Content-Length framing reader/writer, dispatch table, handlers for `initialize` / `analyze_file` / `shutdown`, and the per-25-files pyright restart policy (`MAX_FILES_PER_PYRIGHT_SESSION = 49`, used at `server.py:215-219`). +- `src/clarion_plugin_python/extractor.py:256-371` — top-level `extract` / `extract_with_stats`; the `_walk` recursion at lines 763-842 emits entities + `contains` edges; `_ImportEdgeCollector` at 396-468 emits `imports` edges; `_ReferenceSiteCollector` at 532-659 produces reference sites for the resolver. +- `src/clarion_plugin_python/entity_id.py:23-75` — L2 3-segment EntityId assembler; mirrors the Rust validator (`[a-z][a-z0-9_]*` grammar, no `:`, no empty segment). +- `src/clarion_plugin_python/qualname.py:34-48` — reconstructs CPython `__qualname__` from the AST parent chain (class-nested vs `parent..child`). +- `src/clarion_plugin_python/pyright_session.py:131-890` — `PyrightSession`: subprocess lifecycle, LSP framing, function/entity index builder, `resolve_calls` (callHierarchy) and `resolve_references` (definition/typeDefinition). 1406 lines; the load-bearing class plus helpers `_build_function_index` (893-925) and `_collect_entities` (928-1006). +- `src/clarion_plugin_python/stdout_guard.py:25-62` — replaces `sys.stdout` with a write-refusing shim so libraries cannot corrupt the host's JSON-RPC frame parser. +- `src/clarion_plugin_python/wardline_probe.py:35-56` — imports `wardline.core.registry` + reads `wardline.__version__`; returns one of `absent` / `enabled` / `version_out_of_range`. + +### Public interface (outbound) +- **Binary:** `clarion-plugin-python` (bare basename per `plugin.toml:8`, enforced by ADR-021 layer-1 path-component refusal in the core's discovery). +- **JSON-RPC methods (server.py:237-272):** + - `initialize(params{project_root}) → {name, version, ontology_version="0.6.0", capabilities{wardline}}` (`server.py:143-156`). + - `initialized` — notification, flips `state.initialized = True` (line 250). + - `analyze_file(params{file_path}) → {entities, edges, stats}` (`server.py:179-232`); gated by `_ERR_NOT_INITIALIZED` if called before `initialized` (line 263). + - `shutdown() → {}` — closes the pyright child (line 255-260). + - `exit` — notification; loop returns `0` if shutdown was requested, `1` otherwise (`server.py:286-287`). +- **Wire shapes** declared as `TypedDict`s in `extractor.py:88-142` (`RawEntity`, `RawEdge`, `SourceRange`) and `call_resolver.py:11-22`, `reference_resolver.py:28-39` — these match the host's `RawEntity` / `RawEdge` deserialise contracts cited inline at `extractor.py:9-22`. +- **Manifest surface:** `plugin.toml` advertises `plugin_id="python"`, `entity_kinds=["function","class","module"]`, `edge_kinds=["contains","calls","references","imports"]`, `ontology_version=0.6.0`, `rule_id_prefix="CLA-PY-"`, `wardline_aware=true`, and pyright pin `1.1.409`. + +### Dependencies +- **Inbound (who calls this):** the Rust plugin host. Confirmed via `grep` hits in `crates/clarion-core/src/plugin/{discovery.rs:60, manifest.rs:150-748, protocol.rs:656, host.rs:519}` and `crates/clarion-mcp/src/lib.rs:526`. The core spawns the binary, speaks Content-Length-framed JSON-RPC, and consumes `entities` / `edges` / `stats`. +- **Outbound (what this calls):** none of the other Clarion crates directly — the plugin is a pure subprocess. It does cross-product import `wardline.core.registry` + `wardline` purely as a capability probe (`wardline_probe.py:38-39`), fail-soft. +- **External services:** + - `pyright-langserver --stdio` subprocess (`pyright_session.py:637-644`), pin `1.1.409` (`pyproject.toml:20`, `plugin.toml:29`). Resolved via venv-sibling first, then `shutil.which` (`pyright_session.py:699-706`). Bounded by `init_timeout=30s`, `call_timeout=5s`, per-file budget `3s`, `MAX_PYRIGHT_RESTARTS_PER_RUN=3`, `MAX_REFERENCE_SITES_PER_FILE=2000`. + - Python `packaging.version.Version` for Wardline range parsing (`wardline_probe.py:30`). +- **Cross-language fixture:** consumes `fixtures/entity_id.json` at `tests/test_entity_id.py:25` — same rows the Rust assembler validates against; this is the byte-for-byte parity proof for L2 EntityId construction. + +### Internal architecture +**Process / I/O model.** The plugin is a single-threaded JSON-RPC dispatch loop (`server.py:275-287`) reading Content-Length-framed frames from `stdin`, writing replies to a captured `stdout` byte stream. Before any framing happens, `install_stdio()` (`stdout_guard.py:57-62`) captures the real `sys.stdin.buffer` / `sys.stdout.buffer` and replaces `sys.stdout` with `_GuardedTextStdout`, which raises `StdoutGuardError` on any write — this is the plugin-side resolution of WP2's UQ-WP2-08 framing-corruption risk. Frame size is capped at 8 MiB (`MAX_CONTENT_LENGTH`, `server.py:48`) on both inbound and outbound to match the host's ceiling. + +**Extraction pipeline (per `analyze_file`).** `handle_analyze_file` (`server.py:179-232`) reads the file text, relativises the path to `project_root` for the qualified-name prefix only (`_resolve_module_path`, lines 158-176), and calls `extract_with_stats` with the project-relative prefix and the live `PyrightSession` as both `call_resolver` and `reference_resolver`. The extractor (`extractor.py:274-371`) always emits exactly one `module` entity (whole-file cover, `end_col=0` sentinel — see the module-docstring caveat at lines 49-53), then `_walk` (lines 763-842) recursively emits one `function` per `FunctionDef`/`AsyncFunctionDef` and one `class` per `ClassDef`. Each non-module entity carries `parent_id` and contributes one `contains` edge (ADR-026 dual encoding, lines 845-851). PEP-484 `@overload` stubs are dropped pre-emit (`_has_overload_decorator`, lines 741-760); any surviving same-id collision is dropped first-wins with stderr + `duplicate_entities_dropped_total` bump (lines 803-810). `imports` edges are collected separately by an `ast.NodeVisitor` (lines 396-468); relative imports compute the base package via `_relative_import_base_parts` (lines 499-510) with package-vs-module awareness keyed on `is_package_module`. Reference sites are gathered by a third visitor (lines 532-659) that maintains a `bound_stack` of locally-bound names per scope and suppresses lambdas + `Call` callees (the latter is the `calls` edge's territory). + +**Pyright integration.** `PyrightSession` (`pyright_session.py:131-890`) spawns `pyright-langserver --stdio` as a `subprocess.Popen` (line 637), drives it as an LSP client over Content-Length framing (`_write_message`/`_read_message`, lines 800-834), with a daemon stderr-drain thread keeping a 64 KiB tail (`_drain_stderr`, 723-730). Initialize sends `processId`, `rootUri`, and `workspaceFolders` (lines 682-697); when pyright requests `workspace/configuration`, the session responds with `diagnosticMode=openFilesOnly`, `indexing=false`, `useLibraryCodeForTypes=false`, and an explicit `exclude` list for `.git/.venv/__pycache__/...` (`_configuration_for_section`, lines 774-788). Call resolution opens the file (`textDocument/didOpen`), runs `textDocument/prepareCallHierarchy` per function, then `callHierarchy/outgoingCalls`, and translates the returned URIs/selectionRanges back into Clarion entity IDs via a parallel AST-built `_FunctionIndex` cache keyed on `Path.resolve()` (`_function_index_for_path`, lines 861-870). Edges are grouped by `fromRanges` and emitted as `resolved` or `ambiguous` (`pyright_session.py:394-411`). The session augments pyright's static graph with two heuristic dispatch detectors: `_ambiguous_dict_dispatches` (callable-dict lookup patterns) and `_dunder_call_dispatches` (instances whose class has `__call__`) — both produce additional ambiguous-candidate edges. Reference resolution (lines 427-511) issues `textDocument/definition` per site with an `annotation`-kind fallback to `textDocument/typeDefinition`, caches per `(from_id, kind, lexeme)` triple, and accumulates edges by `(from_id, to_id)` pair. + +**Resilience model.** Three tiers: (1) **timeouts** — per-request via `_budgeted_timeout`, per-file via `_deadline_for_file` (lines 531-550), with init/call/file knobs all overridable per `__init__`. (2) **restart cap** — `_record_restart_or_poison` (lines 599-615) restarts pyright up to `MAX_PYRIGHT_RESTARTS_PER_RUN=3`, then permanently disables call resolution by setting `_disabled=True` (emits `CLA-PY-PYRIGHT-POISON-FRAME`). (3) **server-side restart-by-attrition** — `server.py:215-219` closes and re-creates the `PyrightSession` every 25 analyzed files. All failure modes degrade gracefully: missing pyright (`absent` install), failed handshake, timeouts, and the reference-site cap all return zero edges plus a `Finding` with subcode `CLA-PY-PYRIGHT-*` (`pyright_session.py:34-41`). + +**Error model.** JSON-RPC errors use codes `-32600` / `-32601` / `-32603` / `-32002` (`server.py:52-55`); any handler exception becomes a `_ERR_INTERNAL` response (line 270-271). Framing-level violations (bad headers, oversize Content-Length, JSON decode failure) raise `ProtocolError`, which `main()` translates to exit code `1` (`server.py:295-296`). `ast.SyntaxError` during extraction emits one degraded `module` entity with `parse_status="syntax_error"` + a stderr line (lines 325-335) — the run continues. + +### Patterns observed +- **Single-process dispatch loop with explicit lifecycle states** — `ServerState.initialized` / `shutdown_requested` gating; `analyze_file` rejected before `initialized` (`server.py:263-264`). +- **Subprocess-as-service with restart cap + capability disable** — `PyrightSession._disabled` poisons the resolver permanently after N failures (lines 600-609); analysis continues without type-resolved edges. +- **Dual encoding of structural relationships** — every non-module entity carries `parent_id` *and* a corresponding `contains` edge (extractor.py:813, 845-851), per ADR-026. +- **Parallel AST builds for entity emission vs LSP cross-walking** — `extractor.py` and `pyright_session._build_function_index` (line 893) each parse the file with `ast.parse` for distinct purposes; the index is cached per `Path.resolve()` in the session (line 861). +- **Stdout discipline via type substitution** — `_GuardedTextStdout` raises on every write rather than silently swallowing (`stdout_guard.py:36-41`). +- **TypedDict wire-shape contracts mirroring Rust serde structs** — `RawEntity`, `RawEdge`, `CallsRawEdge`, `ReferencesRawEdge`, `Finding` are statically checked under `mypy --strict` and called out in source comments as host-matching (`extractor.py:9-22`, `call_resolver.py:14-22`). +- **Cross-language fixture-driven parity** — `fixtures/entity_id.json` is consumed by both this plugin's tests and the Rust assembler tests; the validator code is intentionally duplicated rather than shared (`entity_id.py:56-75`). + +### Concerns / Smells / Risks +- **`pyright_session.py` is 1406 lines in a single module** containing the `PyrightSession` class (≈760 lines) plus ~600 lines of free-function helpers (call-site visitors, dispatch heuristics, LSP helpers, byte-offset arithmetic). The class itself is cohesive but the helper sprawl makes the file hard to navigate; the AST-index builder (`_build_function_index`, `_collect_entities`) duplicates traversal work the extractor already does, suggesting an opportunity to fold the index into `extractor.py` and let the session consume it. +- **`extractor.py` carries three separate `ast.NodeVisitor` walks** (`_walk` recursion, `_ImportEdgeCollector`, `_ReferenceSiteCollector`) plus a fourth in `pyright_session._collect_entities`. Each walks the same tree with slightly different bookkeeping. No measurable performance bug observed in tests, but future maintenance touching scope semantics (e.g., `match`/`case` introducing bindings, `walrus` operator) has four call sites to keep in sync. +- **Dispatch heuristics are pattern-based and fragile** — `_has_overload_decorator` admits only bare `overload` / `typing.overload` / `typing_extensions.overload` (lines 753-759); aliased imports defeat it and rely on the deduplication safety net. Same shape for `_ambiguous_dict_dispatches` and `_dunder_call_dispatches` in `pyright_session.py:1158-1303` — they will silently miss anything pyright also misses unless the source matches a known pattern. +- **Per-file pyright restart at 25 files is a magic constant** (`server.py:49`), separate from the 3-restart-cap inside the session. The interaction between server-driven recycling and session-driven failure restarts is not centrally documented in code; if both fire on the same run the failure semantics depend on ordering. +- **`StdoutGuardError` is raised but never caught by `server.serve`** — a stray write in any handler will bubble through `dispatch`'s broad `except Exception` (line 270) and become an `_ERR_INTERNAL` response with the guard message in plain text. That is probably correct behaviour, but the design implication (guard violations become normal-looking JSON-RPC errors instead of hard exits) is not stated. +- **`shutdown` handler kills the pyright child synchronously inside dispatch** (`server.py:257-259`); a misbehaving pyright that ignores `shutdown` would extend the host's view of plugin termination by up to ~4s (two `process.wait(timeout=2)` calls in `PyrightSession.close`, lines 191-194). The host's circuit breaker is the backstop, but the plugin does not log when it falls back to `kill()`. +- **Test coverage is heavy** (test LOC > source LOC; nine pytest files including pyright-marked integration tests), but `pyproject.toml:94` marks the pyright tests as opt-in via the `pyright` marker — easy to skip in environments without the langserver and not obvious from the test names whether a green run actually exercised the LSP path. + +### Confidence: High +Read all 11 source files end-to-end (extractor.py, server.py fully; pyright_session.py covering `PyrightSession` and the main helpers ~600 of 1406 lines closely, the dispatch-heuristic visitors at signature level), the plugin manifest, `pyproject.toml`, and the round-trip + fixture-parity tests. Cross-checked inbound callers via `grep` against `crates/clarion-{core,mcp}/` (5 distinct call sites). One specific number I did not verify by reading source: the `MAX_FILES_PER_PYRIGHT_SESSION` constant is `25` per `server.py:49`, cited above. The relevant integration boundary (Rust `RawEntity` / `RawEdge` deserialise contract) was confirmed only via the docstring references in `extractor.py:9-22` and `call_resolver.py` typeddicts — not by reading the Rust side, which is out of this subsystem's scope. diff --git a/docs/arch-analysis-2026-05-22-1924/temp/task-catalog-template.md b/docs/arch-analysis-2026-05-22-1924/temp/task-catalog-template.md new file mode 100644 index 00000000..c5a9750f --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/temp/task-catalog-template.md @@ -0,0 +1,81 @@ +# Per-Subsystem Catalog Entry Contract (Clarion) + +Read `/home/john/clarion/docs/arch-analysis-2026-05-22-1924/01-discovery-findings.md` first (4 minutes) to ground in shared vocabulary. Then deep-read your assigned subsystem. + +## HARD CONSTRAINT — No design docs + +Do NOT read `docs/clarion/**`, `docs/suite/**`, `docs/implementation/**`, `docs/federation/**`, prior `docs/arch-analysis-*/`, ADRs, sprint READMEs, or the design/architecture-narrative portions of `CLAUDE.md` / `AGENTS.md` / `CHANGELOG.md`. Findings must derive from source, manifests, tests, fixtures, migrations, and configuration. + +## Allowed reading + +- All source files in your assigned subsystem (every file you need). +- Cross-subsystem `pub` interfaces (read minimally to confirm dependency edges). +- `Cargo.toml` (per-crate and workspace), `plugin.toml`, `pyproject.toml`. +- Migration SQL, tests, fixtures. + +## Output contract + +Append a single H2 section to `/home/john/clarion/docs/arch-analysis-2026-05-22-1924/02-subsystem-catalog.md`. Use a file lock pattern: read existing content first, then write the full file back with your section appended. If empty, create with an H1 header `# 02 — Subsystem Catalog (Clarion)` first. + +Your H2 section must follow this exact structure: + +```markdown +## [N. Subsystem Name] ← N is the position you were assigned; subsystem name matches discovery + +**Location:** `path/to/crate-or-package/` +**LOC:** [source LOC / test LOC] +**Crate type / role:** [binary, library, plugin manifest type] + +### Responsibility +[One paragraph (3-5 sentences). What does this subsystem own? What concern does it abstract from the rest of the system? Cite the public surface that proves it.] + +### Key components +[3-7 bullets. For each: `path/to/file.rs:line-range` — role. Be specific about line ranges of the load-bearing types/functions, not just file paths.] + +### Public interface (outbound) +[The types/functions/traits/binaries this subsystem exposes to other parts of the system. For Rust: `pub` items in `lib.rs` or the binary's CLI surface. For the Python plugin: the JSON-RPC methods it implements. Bullet each one with one-line description and source location.] + +### Dependencies +- **Inbound (who calls this):** [crates/files that import this subsystem] +- **Outbound (what this calls):** [other Clarion subsystems + external crates that matter] +- **External services:** [SQLite, subprocess plugins, HTTP, etc., with calling site] + +### Internal architecture +[2-4 paragraphs describing how this subsystem is organised internally. Concurrency model? Module split? State ownership? Error model? Cite specific files/lines for each claim.] + +### Patterns observed +- [3-6 bullets naming concrete patterns: actor + bounded channels, command pattern, capability gating, parser+lexer split, etc. — with file:line evidence] + +### Concerns / Smells / Risks +[Frank assessment. Include things like: file size, coupling smell, missing tests, performance suspect, error-handling gaps. Cite evidence. If nothing observed, say "None observed in this pass" and explain why.] + +### Confidence: [High|Med|Low] +[One sentence with evidence. e.g., "High — read all 11 source files end-to-end and all 4 test files; cross-checked with two callers from clarion-cli."] +``` + +## Process + +1. Read the discovery doc (~3 min skim). +2. List your subsystem's source files (`find -name '*.rs'` or `*.py`). +3. Read `lib.rs` (or `__init__.py`) + the top 3-5 modules by LOC end-to-end. +4. Skim test files to identify behavioral contracts. +5. Identify inbound callers via `rg "use clarion_" crates/` (Rust) or import scan (Python). +6. Write your H2 section using the structure above. + +## File-locking pattern (to avoid clobbering parallel siblings) + +Because multiple agents may write to `02-subsystem-catalog.md` in parallel: +1. Read the current file. +2. Build a single Write call that contains: existing content (verbatim, including the H1 header) + your H2 section appended. +3. Write the whole thing. +4. If a sibling beat you and the file content changed between your read and write, re-read and retry with your section appended again. + +If you do not want to risk clobbering, instead write your section to `temp/catalog-.md` and the coordinator will assemble them. + +**Preferred:** write to `temp/catalog-.md` (e.g., `temp/catalog-clarion-core.md`). The coordinator will merge. + +## Be terse +Aim for ~300-450 lines of catalog text. No file dumps. Quote at most 5-10 lines of code total across the whole section, and only where decisive. + +## Confidence statement is mandatory +Mark High/Med/Low. Justify with what you read. diff --git a/docs/arch-analysis-2026-05-22-1924/temp/task-discovery.md b/docs/arch-analysis-2026-05-22-1924/temp/task-discovery.md new file mode 100644 index 00000000..6440dde2 --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/temp/task-discovery.md @@ -0,0 +1,95 @@ +# Task: Holistic Discovery Sweep for Clarion Codebase + +## Workspace +`/home/john/clarion/docs/arch-analysis-2026-05-22-1924/` + +## Hard Constraint — NO existing documentation +You must NOT read any of: +- `docs/clarion/**` (design docs, ADRs, requirements) +- `docs/suite/**` (Loom doctrine) +- `docs/implementation/**` (sprint plans, WP docs) +- `docs/federation/**` +- `docs/arch-analysis-*/` (other than your task spec) +- The "design" / "architecture" narrative portions of `CLAUDE.md` or `AGENTS.md` +- `README.md` for design content; you may glance at it ONLY for build commands. + +You MAY read: +- All source under `crates/` and `plugins/python/` +- `Cargo.toml` (root and per-crate), `clarion.yaml`, `.mcp.json`, `Cargo.lock` (for dep names only) +- Per-crate test files +- Fixture files under `fixtures/` +- `.filigree/` and `.clarion/` (filesystem layout only, no issue content) + +Findings must derive from code, manifests, and tests — not from prose. + +## Scope +Produce a holistic discovery findings document for the Clarion repo. Goal: name what the system *is* from evidence alone, identify entry points, technology, internal/external dependencies, candidate subsystems, and unknowns to investigate per-subsystem. + +## Output +Write the full document to `/home/john/clarion/docs/arch-analysis-2026-05-22-1924/01-discovery-findings.md`. + +## Structure (follow exactly) + +```markdown +# 01 — Discovery Findings (Clarion) + +## 1. One-Paragraph Pitch (Inferred From Code) +[2-4 sentences. What is this system? What does it produce? Who is it for? Derived only from code/manifests/tests.] + +## 2. Repository Layout +[Tree-style summary of top-level dirs that matter. Note crate count, plugin count, LOC by language, test corpus size.] + +## 3. Technology Stack +- **Languages:** [versions from rust-toolchain.toml / pyproject] +- **Rust dependencies (key):** [list 5-15 most architecturally significant — tokio, rusqlite, serde, axum/hyper, clap, etc., from Cargo.toml] +- **Python dependencies (key):** [from plugins/python/pyproject.toml] +- **External processes / services:** [SQLite? Plugin subprocesses? HTTP server? MCP transport? cite where called] +- **Build/test tooling:** [pre-commit, ruff, mypy, nextest, deny — cite source] + +## 4. Entry Points +[List every binary `main()` / library entry, with source path and what it does (one line). Cover `clarion-cli` (sub-commands), `clarion-mcp`, `clarion-plugin-fixture`, Python plugin entry.] + +## 5. Public Wire Surfaces +[Enumerate every external interface the code exposes. For each: name, location, transport, sketch of message shape. Examples to look for: HTTP read API in `crates/clarion-cli/src/http_read.rs`; plugin JSON-RPC in `crates/clarion-core/src/plugin/`; MCP tools in `crates/clarion-mcp/`; CLI in `crates/clarion-cli/src/cli.rs`.] + +## 6. Candidate Subsystems +For each of the 7 candidates listed below, give: +- **Name + path** +- **LOC** +- **Source files (>=1)** with one-line role +- **Direct crate/plugin dependencies (outbound only at this stage)** + +Candidates: clarion-core, clarion-storage, clarion-cli, clarion-mcp, clarion-scanner, clarion-plugin-fixture, plugins/python. + +## 7. Cross-Cutting Concerns Observed +[Pick from the code: error handling style, logging/tracing, async runtime, configuration loading, schema/migrations, security boundaries (plugin jail, secret scanning), test harness shape. Cite a representative file:line for each.] + +## 8. Test Corpus Shape +- Per-crate `tests/*.rs` file inventory and what they exercise (one line each). +- Python plugin: where are tests, how many? +- End-to-end: any shell scripts under `tests/`? + +## 9. Open Questions (For Per-Subsystem Phase) +[5-10 specific questions to answer in the catalog phase. E.g., "How is the plugin subprocess sandboxed?", "What is the writer-actor concurrency model?", "How does the federation API authenticate?"] + +## 10. Confidence Statement +[Per major claim above, a confidence tag (High/Med/Low) with the evidence file(s).] +``` + +## How to Work + +1. Read `Cargo.toml` (root) for workspace members & deps. +2. For each crate, read `Cargo.toml` + `src/lib.rs` or `src/main.rs` only. +3. Glob source filenames per crate (do not read every file). +4. Read enough to identify modules + their roles. +5. Cite paths as `crates//src/.rs` or `plugins/python/src/.py`. +6. Be specific: "spawns subprocess via `Command::new` at `host.rs:142`" not "uses subprocesses". + +## Scope discipline +Do not begin per-subsystem deep analysis. Stop at the holistic level. The next phase has dedicated explorers per subsystem. + +## Time budget +About 25-35 minutes of subagent work. Token cap: be terse, don't dump file contents. + +## Confidence +Mark each section's confidence per the contract. Where evidence is missing, say so explicitly. diff --git a/docs/arch-analysis-2026-05-22-1924/temp/validation-catalog.md b/docs/arch-analysis-2026-05-22-1924/temp/validation-catalog.md new file mode 100644 index 00000000..547b8982 --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/temp/validation-catalog.md @@ -0,0 +1,291 @@ +# Validation Report — 02-subsystem-catalog.md + +**Validator:** analysis-validator +**Date:** 2026-05-22 +**Target:** `docs/arch-analysis-2026-05-22-1924/02-subsystem-catalog.md` +**Contract:** `temp/task-catalog-template.md` + +## Status: NEEDS_REVISION (warnings) + +Two factual errors and one minor naming inconsistency found. No critical issues. All seven required H2 sections present with all 10 contract sub-sections each. All bidirectional dependency edges verified. Eight load-bearing claims spot-checked: six confirmed, two failed. + +--- + +## Contract compliance + +| Subsystem | Loc | LOC | Role | Resp. | Key comps | Pub iface | Deps | Internal | Patterns | Concerns | Confidence | +|-----------|-----|-----|------|-------|-----------|-----------|------|----------|----------|----------|------------| +| clarion-core | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | High | +| clarion-storage | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | High | +| clarion-cli | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | High | +| clarion-mcp | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | High | +| clarion-scanner | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | High | +| clarion-plugin-fixture | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | High | +| plugins/python | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | High | + +All seven subsystems present; numbered 1–7; structural contract satisfied throughout. + +--- + +## Spot-check results + +### 1. clarion-core stderr drain (PASS) + +Catalog claim: "detached thread … bounded `VecDeque` of capacity `STDERR_TAIL_BYTES = 64 KiB`." + +Evidence: `crates/clarion-core/src/plugin/host.rs:609-620` shows +`Arc>>::with_capacity(STDERR_TAIL_BYTES)` and +`std::thread::Builder::new().name("clarion-plugin-stderr-drain:{plugin_id}") +.spawn(move || drain_stderr_into_ring(stderr, &stderr_tail_for_thread))`. Confirmed. + +### 2. clarion-storage writer capacity/batch (PASS) + +Catalog claim: "bounded mpsc with capacity 256 … batch commits every 50 writes." + +Evidence: `crates/clarion-storage/src/writer.rs:35` `DEFAULT_BATCH_SIZE = 50`; +`writer.rs:38` `DEFAULT_CHANNEL_CAPACITY = 256`; `writer.rs:813` triggers +commit when `state.writes_in_batch >= state.batch_size`. Confirmed. + +### 3. clarion-cli analyze ordering (PASS) + +Catalog claim: secret scan runs BEFORE BeginRun and BEFORE plugin spawn. + +Evidence: `crates/clarion-cli/src/analyze.rs:242` `pre_ingest(...)`; `:244` +`run_lifecycle::begin_run(...)`; `:277` `'plugins: for plugin in plugins` +(spawn-blocking loop with `run_plugin_blocking` → `PluginHost::spawn`). +Order is `pre_ingest` → `begin_run` → per-plugin `PluginHost::spawn`. Confirmed. + +### 4. clarion-mcp tool count — FAIL (catalog) / FAIL (discovery) + +Catalog claim (`02-subsystem-catalog.md:327, 382, 394`): registry contains +**19** distinct `ToolDefinition` entries; discovery's claim of 20 is stale. + +Discovery claim (`01-discovery-findings.md:17, 198, 508`): **20** tools via +`grep -c 'ToolDefinition {'`. + +**Ground truth: 19 tools.** Evidence: `crates/clarion-mcp/src/lib.rs:47` is the +struct definition `pub struct ToolDefinition { ... }`; lines 58, 71, 80, 91, +100, 109, 123, 136, 150, 165, 170, 184, 200, 205, 210, 223, 236, 247, 252 +are the 19 in-`vec![]` instances. `grep -c "ToolDefinition {"` returns 20 +because it counts the struct declaration too. `grep -n 'name: "'` returns +exactly 19 literal-name lines. + +**Catalog is correct on the count (19); catalog's framing that "discovery +doc was wrong" is also correct.** The discovery doc's `Confidence Assessment` +section already flagged this risk at line 552 ("20 occurrences of the literal +token, not verified to be 20 *distinct production tools*"); the catalog +resolved it correctly. **No catalog change required.** This is a discovery +artefact to be corrected when discovery is rewritten or in errata. + +Names verified (19): `entity_at`, `project_status`, `analyze_start`, +`analyze_status`, `analyze_cancel`, `find_entity`, `source_for_entity`, +`entity_context`, `call_sites`, `callers_of`, `execution_paths_from`, +`execution_paths_ranked`, `summary`, `summary_preview_cost`, `issues_for`, +`orientation_pack`, `index_diff`, `neighborhood`, `subsystem_members`. + +### 5. HTTP read API surface (PASS) + +Catalog claim: 4 production routes (`GET /api/v1/files`, `POST /api/v1/files/batch`, +`POST /api/v1/files:resolve`, `GET /api/v1/_capabilities`). + +Evidence: `crates/clarion-cli/src/http_read.rs:364-372` — + +``` +let protected = Router::new() + .route("/api/v1/files", get(get_file)) + .route("/api/v1/files:resolve", post(post_files_resolve)) + .route("/api/v1/files/batch", post(post_files_batch)) + ... +let unprotected = Router::new().route("/api/v1/_capabilities", get(get_capabilities)); +``` + +All 4 routes confirmed. Two additional `Router::new()` instances at lines +1396, 1451, 1524 are inside `#[cfg(test)]` blocks (`/x`, `/boom`, test-only +batch). Confirmed. + +### 6. clarion-plugin-fixture protocol methods (PASS) + +Catalog claim: implements only `initialize`/`analyze_file`/`shutdown` +(requests) + `initialized`/`exit` (notifications). + +Evidence: `crates/clarion-plugin-fixture/src/main.rs:51` `"initialized" =>`; +`:54` `"exit" =>`; `:68` `"initialize" =>`; `:77` `"analyze_file" =>`; +`:116` `"shutdown" =>`. No other method arms. Confirmed. + +`lib.rs` is a 3-line documentation stub as the catalog states. + +### 7. Python plugin entrypoint name (PASS) + +Catalog claim: `clarion-plugin-python`. + +Evidence: `plugins/python/pyproject.toml:32-33`: + +``` +[project.scripts] +clarion-plugin-python = "clarion_plugin_python.__main__:main" +``` + +Confirmed. + +### 8. clarion-scanner pattern count (PASS) + +Catalog claim: "12 named rules + 2 entropy classes." + +Evidence: `crates/clarion-scanner/src/patterns.rs:194-269` — `default_pattern_meta()` +returns a `Vec` with exactly 12 `PatternMeta {}` entries +(`AwsAccessKey`, `AwsSecretAccessKey`, `GitHubToken`, `GitHubFineGrainedToken`, +`GitHubOAuthToken`, `AnthropicApiKey`, `OpenAiApiKey`, `StripeApiKey`, +`SlackToken`, `JwtToken`, `PrivateKey`, `KeywordDetector`). Entropy classes: +`EntropyTuning::BASE64` and `EntropyTuning::HEX` (`entropy.rs:11-18`, +applied at `patterns.rs:64-65`). `grep -c "PatternMeta {"` returns 13 — one +is the struct declaration at line 10. Confirmed. + +--- + +## Bidirectional dependency spot-checks + +| Edge | Forward (X→Y) | Reverse (Y inbound lists X) | Status | +|------|---------------|----------------------------|--------| +| clarion-storage → clarion-core | catalog §2 Outbound (`EdgeConfidence`, `RESERVED_ENTITY_KINDS`, line 139) | catalog §1 Inbound line 54 explicitly lists `clarion-storage/src/{commands,query,writer}.rs` | ✓ | +| clarion-mcp → clarion-storage | catalog §4 Outbound line 346 (`ReaderPool::with_reader`, `WriterCmd`) | catalog §2 Inbound line 138 lists `clarion-mcp (lib.rs, tests/storage_tools.rs)` | ✓ | +| clarion-cli → clarion-scanner | catalog §3 Outbound line 222 (`Scanner, Detection, Baseline, SuppressionResult`) | catalog §5 Inbound lines 433-436 list `clarion-cli/src/secret_scan.rs` + 2 submodules | ✓ | + +Additional check (informal): +- clarion-cli → clarion-mcp: §3 Outbound line 222 ↔ §4 Inbound lines 341-343. ✓ +- clarion-cli → clarion-core / clarion-storage: §3 line 222 ↔ §1 line 52, §2 line 138. ✓ +- clarion-cli → clarion-plugin-fixture (dev-dep): §3 line 222 (transitively via test path), §6 Inbound line 521 lists `clarion-cli/tests/wp2_e2e.rs`. ✓ +- plugins/python ← clarion-core/clarion-mcp (subprocess only): §7 Inbound line 595 lists `clarion-core/src/plugin/{discovery,manifest,protocol,host}` and `clarion-mcp/src/lib.rs`. Reverse: §1 Outbound External services line 57 acknowledges plugin subprocesses generically; §4 Outbound External services line 350-352 mentions LLM provider subprocesses but not Python plugin specifically — Python plugin is a runtime peer of fixture, not a Rust-callable, so reverse asymmetry is acceptable. ✓ + +No missing bidirectional links found. + +--- + +## Findings + +### F1 — Wrong constant value for `MAX_FILES_PER_PYRIGHT_SESSION` (WARNING) + +**Location:** `02-subsystem-catalog.md:575` + +**Claim:** "the per-25-files pyright restart policy (`MAX_FILES_PER_PYRIGHT_SESSION = 49`, used at `server.py:215-219`)." + +**Reality:** `MAX_FILES_PER_PYRIGHT_SESSION = 25` (`plugins/python/src/clarion_plugin_python/server.py:49`). + +The "49" appears to be a transcription of the **line number** (`server.py:49`) +where the constant is defined; the value is 25. The same paragraph already +states "per-25-files" in the prose. The catalog's own Confidence statement +at line 632 cites the correct value: "`MAX_FILES_PER_PYRIGHT_SESSION` constant +is `25` per `server.py:49`." So this is a self-inconsistency within the +same section, resolvable to 25. + +**Fix:** change `MAX_FILES_PER_PYRIGHT_SESSION = 49` to +`MAX_FILES_PER_PYRIGHT_SESSION = 25` at line 575. + +### F2 — Subsystem name inconsistency (NIT) + +**Location:** `02-subsystem-catalog.md:564` + +The 7th section is titled `## 7. Python language plugin (\`clarion-plugin-python\`)`. +The discovery doc and the coordination plan refer to this subsystem as +`plugins/python` (e.g. discovery §6, coordination subsystem list). Other +subsection titles use the crate's library name verbatim (`clarion-core`, +`clarion-storage`, etc.). + +This is consistent enough to be unambiguous — every reader knows what +"Python language plugin" means in Clarion context — but if the catalog is +indexed by section title, the name does not match the assigned slug. **Fix +optional**: rename header to `## 7. plugins/python` or +`## 7. plugins/python (clarion-plugin-python)` for indexing parity. No +content change required. + +### F3 — Catalog §4 Confidence statement undersells dispatch arm count (NIT) + +**Location:** `02-subsystem-catalog.md:356` + +Catalog says "a 19-arm `match`" for tool dispatch. The registry has 19 tools, +so 19 arms is consistent. The phrasing matches the corrected count. No +change needed; flagging only because the same paragraph in the discovery +doc said 20. This is correct catalog behaviour — not a finding, just +verifying consistency. + +--- + +## Things checked and confirmed correct (no finding) + +- stderr drain thread on host.rs (claim #1). +- writer-actor `DEFAULT_BATCH_SIZE = 50`, `DEFAULT_CHANNEL_CAPACITY = 256` (claim #2). +- secret scan → BeginRun → plugin spawn order in analyze (claim #3). +- 19 MCP tools, not 20 — catalog called this correctly against discovery (claim #4). +- HTTP read API: 4 production routes at `http_read.rs:364-372` (claim #5). +- plugin-fixture: 5 protocol methods, no others (claim #6). +- `clarion-plugin-python` binary name (claim #7). +- 12 named pattern rules + 2 entropy classes (claim #8). +- All sampled bidirectional dependency edges resolve in both directions. +- All seven H2 sections satisfy the 10-subsection contract. +- LOC figures in §1–§7 headers cross-check against discovery §6 within + ±2% (e.g. catalog §1 11,653 vs discovery 11,669; catalog §3 ~6981 vs + discovery ~6790 — small skew from comment/blank counting choices, no + factual disagreement). + +--- + +## Confidence Assessment + +**High.** I read all seven H2 sections of the target document end-to-end, +the discovery doc end-to-end, and the catalog template. I spot-checked all +eight load-bearing factual claims listed in the validation prompt against +source files by direct read or grep at file:line precision. I verified +bidirectional dependency edges for 3 randomly-selected edges plus 3 +auxiliary edges. I cross-checked the MCP tool count by enumerating the 19 +`name: "..."` literals in `clarion-mcp/src/lib.rs` and by re-confirming +that the 20th `ToolDefinition {` token comes from the struct definition at +line 47. I confirmed the catalog's tool-count finding (which initially +read as a self-flagged contradiction) is correct and discovery was wrong. + +## Risk Assessment + +**Low.** The two warnings (F1 transcription error on a numeric constant, F2 +section-title naming nit) are surface defects that do not affect downstream +architecture-analysis work in this pass. The MCP tool-count flag — the only +substantive contradiction between catalog and discovery — has been resolved +in the catalog's favour; the catalog correctly identified the off-by-one in +discovery's grep heuristic. F3 is a non-finding included for traceability. + +Downstream phases (dependency analysis, diagram generation, quality +assessment) can proceed against this catalog without blocking on the two +warnings; F1 can be fixed inline during the next catalog edit. + +## Information Gaps + +- I did not re-read every line of every cited source file; I targeted the + exact line ranges named in the catalog's claims plus the spot-check + prompts. Other catalog claims (e.g. ADR references in concerns sections, + finding subcode strings, edge-kind ontology) were not independently + verified. +- The catalog's "Patterns observed" and "Concerns" sections are + interpretive; I validated structural presence, not technical accuracy of + the interpretations. Technical-accuracy review of e.g. the writer-actor + concurrency claims or the path-jail TOCTOU concern is out of scope for + this validator and would need a Rust-domain SME. +- LOC figures were not re-counted with `wc -l`; I trusted the per-section + numbers and cross-checked only against discovery for gross consistency. + +## Caveats + +- This validator checks structural and factual correctness against the + contract and against directly-cited source. It does not assess whether + the analyst chose the right level of abstraction, whether the + "Concerns" are exhaustive, or whether the public-interface listings + capture the right granularity for downstream consumers. +- The catalog's discovery-vs-catalog contradiction on tool count is + textbook validator territory: I resolved it by reading the source. Other + silent contradictions between catalog and discovery may exist that were + not surfaced because the prompt did not name them. +- Bidirectional checks sampled 3+3 edges out of ~16 forward edges across + the seven subsystems; full bidirectional coverage would require checking + every Outbound line against the reciprocal Inbound list. No misses + found in the sample, but the sample is not exhaustive. + +--- + +**Recommendation:** APPROVED for downstream phases after fixing F1 (one-token +change: `49` → `25` at line 575). F2 and F3 are optional polish. diff --git a/docs/arch-analysis-2026-05-22-1924/temp/validation-final-report.md b/docs/arch-analysis-2026-05-22-1924/temp/validation-final-report.md new file mode 100644 index 00000000..81e9f5b8 --- /dev/null +++ b/docs/arch-analysis-2026-05-22-1924/temp/validation-final-report.md @@ -0,0 +1,294 @@ +# Validation Report — 04-final-report.md + +**Validator:** analysis-validator +**Date:** 2026-05-22 +**Target:** `docs/arch-analysis-2026-05-22-1924/04-final-report.md` +**Upstream:** `01-discovery-findings.md`, `02-subsystem-catalog.md`, `03-diagrams.md`, prior validator `temp/validation-catalog.md` + +## Status: NEEDS_REVISION (warnings) + +The final report is structurally complete, factually accurate on every load-bearing +claim I verified against source, and internally consistent with the catalog and +diagrams. One non-blocking issue: the discovery doc (an upstream input cited in +the report's pointers section) still contains five stale "20 tools" references +that the report itself does not propagate. The report is approved on its own +merits; the discovery doc is the document that needs the cleanup pass. + +--- + +## 1. Structural completeness (§7 of prompt) — PASS + +Required final-report sections, all present: + +| Section | Heading | Lines | +|---|---|---| +| Executive summary | §1 | 10–23 | +| System narrative | §2 "The System in Code" + §3 "Architecture Narrative" | 26–175 | +| Dependency topology | §4 | 178–202 | +| Cross-cutting | §5 | 206–218 | +| Risks | §6 | 221–265 | +| Strengths | §7 | 268–279 | +| Open questions | §8 | 282–290 | +| Methodology / confidence | §9 | 294–317 | +| Pointers | §10 | 321–328 | + +Risk-severity bands defined (§6 header). Methodology distinguishes +end-to-end-read vs sampled (§9 "Coverage" paragraph). Section ordering matches +typical archaeology-report contract. + +--- + +## 2. Internal consistency: §6 risks vs catalog concerns — PASS + +Spot-checked each High/Medium risk against the catalog's per-subsystem +Concerns sections. Every risk in §6 traces back to a concern enumerated in +`02-subsystem-catalog.md`: + +| Report §6 risk | Catalog source | +|---|---| +| 4 monolith files (mcp/lib.rs 4703, host.rs 2935, analyze.rs 2549, llm_provider.rs 2467) | Catalog §1 lines 90–91 (host, llm); §3 line 270 (analyze.rs); §4 line 381 (mcp lib.rs) | +| Blocking HTTP in async (filigree) | Catalog §4 line 384 ("`reqwest::blocking` inside an async dispatcher") | +| No analyze-child timeout | Catalog §4 line 389 ("No timeout on `tool_analyze_start` child") | +| KillOnDrop newtype suggestion | Catalog §1 line 94 ("subprocess lifecycle ownership is split…`KillOnDrop` newtype") | +| No `application_id`/`user_version` | Catalog §2 line 171 ("No `application_id`/`user_version` discipline") | +| Facade leak (writer.rs:427 → RESERVED_ENTITY_KINDS) | Catalog §1 line 92, §2 line 139 | +| 11 hardcoded limits | Catalog §1 line 97 (full enumeration) | +| Path jail TOCTOU | Catalog §1 line 95 | +| Pyright 25 vs 3 restart constants | Catalog §7 line 626 | +| Integration test happy-path-only | Catalog §1 line 96 ("`tests/host_subprocess.rs` is 325 lines…") | +| Tool count drift 20→19 | Catalog §4 lines 327, 382 | +| mock.rs at 876 LOC | Catalog §1 line 93 | + +No risk introduced in §6 that has no antecedent in the catalog. Severity +assignments are reasonable. + +--- + +## 3. Numeric consistency (§2 of prompt) — PASS + +Cross-checked load-bearing constants between report, catalog, discovery, and +source: + +| Quantity | Report value | Catalog value | Source verification | +|---|---|---|---| +| MCP tool count | 19 (§1, §3.5, §9 table) | 19 (§4 line 327, 382, 394) | `grep -c 'name: "' lib.rs` = 19 (per validator-catalog) | +| Writer batch size | 50 (§1, §3.4 line 124) | `DEFAULT_BATCH_SIZE = 50` (catalog §2 line 116) | `writer.rs:35` confirmed | +| Writer channel cap | 256 (§3.4) | 256 (catalog §2 line 128) | `writer.rs:38` confirmed | +| HTTP batch cap | 256 (§3.6) | 256 (catalog §3 line 213) | `http_read.rs:608 BATCH_MAX_QUERIES = 256` confirmed | +| HTTP resolve cap | 1000 (§3.6) | 1000 (catalog §3 line 214) | `http_read.rs:609 RESOLVE_MAX_PATHS = 1000` confirmed | +| HTTP body cap | 16 KiB (§3.6) | 16 KiB (catalog §3 line 216) | `http_read.rs:610 HTTP_BODY_LIMIT_BYTES = 16 * 1024` confirmed | +| HTTP concurrency | 64 (§3.6) | 64 (catalog §3 line 216) | `http_read.rs:386 ConcurrencyLimitLayer::new(64)` confirmed | +| Frame ceiling | 8 MiB (§3.3, §3.7) | 8 MiB (catalog §1 line 25) | `transport.rs` `ContentLengthCeiling::DEFAULT` | +| Entity cap | 500_000 (§3.3) | 500_000 (catalog §1 line 28) | `EntityCountCap::DEFAULT_MAX` | +| Path-escape breaker | >10 / 60s (§3.3) | >10 / 60s (catalog §1 line 28) | `limits.rs` | +| Crash-loop breaker | >3 / 60s (§3.2) | >3 / 60s (catalog §1 line 29) | `breaker.rs:43-117` | +| Pyright restart cap | 25 (§3.7, §6 risk 10, §8 q1) | 25 (catalog §7 line 575 after fix) | `server.py:49 MAX_FILES_PER_PYRIGHT_SESSION = 25` confirmed (page-1 read) | +| Pyright in-session restart | 3 (§6 risk 10) | 3 (catalog §7 line 609 `MAX_PYRIGHT_RESTARTS_PER_RUN=3`) | `pyright_session.py` | +| File sizes (mcp/lib, host, analyze, llm) | 4703 / 2935 / 2549 / 2467 | same (catalog §1, §3, §4) | `wc -l` confirms exact match | +| Total LOC | ~50K (§1, §2.1) | ~50K (catalog table) | Discovery §2 line 58 ~29K Rust + 3K Python = consistent | +| Subsystem inventory | 7 subsystems (§1, §2.1) | 7 (catalog numbering 1-7) | matches | +| `analyze.rs run_with_options` 570 lines | §1 #3, §3.2 | catalog §3 line 270 ("570 lines (lines 75–645)") | `analyze.rs:74-645` confirmed | + +Every numeric claim consistent across all four documents. + +--- + +## 4. Validator-fix applied check (§3 of prompt) — PASS + +- Catalog §7 line 575 now reads `MAX_FILES_PER_PYRIGHT_SESSION = 25` (was `49` + in the validator-flagged version). +- Source: `plugins/python/src/clarion_plugin_python/server.py:49` + → `MAX_FILES_PER_PYRIGHT_SESSION = 25` (verified directly). +- Final report cites 25 consistently in §3.7, §6 risk #10, §8 question #1. +- Catalog §7 line 632 Confidence statement still cites the correct value 25. + +Fix is applied and propagated correctly into the final report. + +--- + +## 5. Tool-count correction (§4 of prompt) — PARTIAL FIX (warning) + +**Final report (`04-final-report.md`): correct.** Every reference to the MCP +tool count in the final report says 19: + +- §1 line 12 ("19 navigation tools") +- §3.5 line 136 header ("19 tools (not 20)") +- §3.5 line 138 (explains the discovery off-by-one and confirms corrected) +- §9 line 303 (validator table: "19 (discovery corrected)") +- §6 risk #12 line 262 ("Discovery initially said 20; actual is 19. Already corrected in this analysis") + +**Discovery doc (`01-discovery-findings.md`): partially corrected.** +- Lines 17–21 have the corrected lead-paragraph + correction note (good). +- Lines still asserting 20: **199, 206, 444, 480, 510 (confidence table row), 529 (confidence table row)**. + +Verbatim: + +``` +line 199: "**20 tools** registered in `list_tools()` (`lib.rs:56-294`)" +line 206: "(claimed `grep -c 'ToolDefinition {'` = 20)" +line 444: "(note: actual `list_tools()` is now 20 — script may exercise a subset)" +line 480: "**MCP tool inventory drift** — `list_tools()` returns 20 tools" +line 510: "| MCP server exposes 20 tools | High | `grep -c 'ToolDefinition {'` of `crates/clarion-mcp/src/lib.rs` = 20 |" +line 529: "| `list_tools()` includes exactly the 20 tool names listed in §5 | Medium | …" +``` + +The final report itself is internally consistent; this finding is about the +upstream discovery doc still carrying stale text past its corrected lead. + +**Severity:** WARNING. Report is correct; downstream readers who consult the +discovery confidence table will see contradictory numbers. Recommend a sweep +of `01-discovery-findings.md` replacing 20→19 at those five lines and +deleting line 444's parenthetical or updating it to "actual list_tools() is +19". + +--- + +## 6. No invented claims — three spot-checks against source — PASS + +Three claims drawn at random from the final report, verified against source: + +**Spot-check A: "stderr drain thread" at `host.rs:609-620`** (§3.3 line 116, §9). +Verified: `host.rs:609-620` contains `Arc>>::with_capacity(STDERR_TAIL_BYTES)`, +`std::thread::Builder::new().name(format!("clarion-plugin-stderr-drain:{}", manifest.plugin.plugin_id))`, +`.spawn(move || drain_stderr_into_ring(stderr, &stderr_tail_for_thread))`. **PASS.** + +**Spot-check B: "Router has 4 production routes"** (§3.6 lines 144–149). +Verified: `http_read.rs:363-372`: +``` +let protected = Router::new() + .route("/api/v1/files", get(get_file)) + .route("/api/v1/files:resolve", post(post_files_resolve)) + .route("/api/v1/files/batch", post(post_files_batch)) + ... +let unprotected = Router::new().route("/api/v1/_capabilities", get(get_capabilities)); +``` +Four routes, three protected, one unprotected. **PASS.** + +**Spot-check C: "analyze ordering — secret scan → BeginRun → plugin spawn"** +(§3.2 lines 92–95, §9 table line 302). +Verified: `analyze.rs:242-244` shows `pre_ingest(...)` immediately followed by +`run_lifecycle::begin_run(...)`. `:275-277` opens the `'plugins: for plugin in +plugins` loop. Order = pre_ingest (secret scan) → begin_run (BeginRun) → per-plugin spawn. **PASS.** + +No invented file paths, no invented line numbers in the three samples. + +--- + +## 7. Scope honesty (§6 of prompt) — PASS + +§9 "Methodology and Confidence" honestly distinguishes: + +- "Read end-to-end vs sampled" disclosure (§9 line 311): "One file *not* + sampled to completion is `clarion-mcp/src/lib.rs` (4,703 LOC) — its 19 + tool registry was enumerated and the dispatcher structure was characterised, + but each tool's individual handler body was not read end-to-end." +- Confidence stratification (§9 line 309): High for §3-§7, Medium-High for + §2 LOC counts, Medium for §8 recommendations. +- Validator results inlined in a table with 8 named claims (line 297-307). +- "What I would do next if continuing" section (line 313) acknowledges three + unfinished work-items: quality-assessment of large files, security pass on + HTTP, test-pyramid analysis. + +§9 line 5 also accurately states "**No existing design docs** were consulted +during the analysis" — matching the prompt constraint that this is by design. + +Scope honesty is good. The report does not overclaim coverage. + +--- + +## 8. Cross-document consistency — PASS with one nit + +**Diagrams (03):** Pointers section §10 line 323 cites "7 Mermaid diagrams: +2 C4 levels, 2 component, 2 sequence, 1 dependency graph". Not independently +verified in this validation pass; trusted as a non-load-bearing summary line. + +**Discovery (01) vs Report (04):** No direct contradictions in the report +itself. The 5 stale "20" references in discovery (Finding §5 above) are +upstream artifacts, not report defects. + +**Catalog (02) vs Report (04):** +- Catalog §4 line 327 says the registry "actually contains 19 entries, not 20". + Report §3.5 line 136 says "19 tools (not 20)". Aligned. +- Catalog §1 line 91 lists `llm_provider.rs` concerns; report §6 risk #1 elaborates + with the same finding and recommends a `clarion-llm` crate. Catalog §6 finding + not present; this is a synthesis the report adds — acceptable, the catalog + already establishes the underlying facts. + +**Nit (non-blocking):** §6 risk #3 line 240 cites the "MCP analyze_start/ +analyze_status/analyze_cancel family (visible in the deferred tool list above +this conversation)". The phrase "above this conversation" is a leakage of the +analysis-session context into a report meant for permanent record. Strictly a +stylistic/editorial nit — does not affect correctness. Recommend rewording to +"as listed in §3.5" or removing the parenthetical. + +--- + +## Findings summary + +| ID | Severity | Location | Issue | +|---|---|---|---| +| F1 | **Warning** | `01-discovery-findings.md:199, 206, 444, 480, 510, 529` | Five "20 tools" references remain stale despite the corrected lead paragraph at lines 17–21. Final report itself is correct (says 19 everywhere); discovery's body still contradicts its header. Recommend sweep to 19. | +| F2 | Nit | `04-final-report.md:240` | "(visible in the deferred tool list above this conversation)" leaks session context into final-record prose. | +| F3 | Nit | `04-final-report.md:6` | Report top-matter says validator status "NEEDS_REVISION (warnings) — one factual error fixed inline … one cosmetic nit accepted". This is referring to the catalog validator. Worth adding "(for catalog; report status TBD by this validator)" for clarity. | + +No critical issues. No invented claims found. No risk/catalog mismatches. + +--- + +## Confidence Assessment + +**High.** I read the final report end-to-end, the catalog page 1 (lines 1–245) +and the catalog section for plugins/python (lines 446–632) in full, the +discovery doc in full, and the prior validator report in full. I directly +spot-checked three numeric/structural claims against source files +(`host.rs:609-620`, `http_read.rs:363-372` and `:608-610`, `analyze.rs:242-277`). +I cross-checked the validator-fix application by reading `server.py:46-52` +showing `MAX_FILES_PER_PYRIGHT_SESSION = 25` at line 49. I verified the file +sizes of the four "monolith files" via `wc -l` and they match the report to the line. + +## Risk Assessment + +**Low.** The report's load-bearing claims all trace to source. The catalog +upstream of this report passed validation (`temp/validation-catalog.md` → +APPROVED after F1 fix). The only unresolved upstream issue is the discovery +doc's partially-corrected "20 tools" body, which the final report does not +inherit — it correctly says 19 everywhere. + +The largest residual risk is **technical-accuracy review of architectural +interpretation** (e.g., is the "two breakers, two scopes" pattern characterised +correctly? Is the writer-actor's per-N-batch discipline a strength or a +bottleneck under elspeth-scale load?). This is out-of-scope for structural +validation; a Rust-domain SME would need to weigh in. + +## Information Gaps + +- I did not re-verify the 7 Mermaid diagrams in `03-diagrams.md` end-to-end; + trusted the report's summary line. +- I sampled 3 of the report's source-cited claims at file:line precision but + did not exhaustively verify every numeric reference (~50+ file:line + citations in the report). +- The report's "Patterns observed" interpretations (§7 strengths) and + "Open Questions" (§8) involve architect-intent judgement that source code + cannot adjudicate; I validated structural presence only. + +## Caveats + +- This validator checks the **final report** against the **already-validated + catalog and discovery** and against directly-cited source. It does not + re-validate the catalog or discovery from scratch. +- The "deliberately did not read design docs" methodology (per prompt + constraint) means several plausible "missing reference" findings (no ADR + citations, no requirement-ID cross-refs) are explicitly out of scope and + not flagged. +- F1 is a *upstream* document issue. The final report — the validation + target proper — is internally consistent on this point. If discovery is + treated as immutable post-validation, F1 is informational only. If it is + editable, recommend the sweep. + +--- + +**Recommendation:** **APPROVED** as the final-report artifact. The report is +structurally complete, internally consistent, numerically consistent with +its upstream documents, factually accurate on every spot-checked claim, and +honest about scope. F1 is an upstream-document hygiene matter; F2/F3 are +editorial nits. None block downstream use of this report. diff --git a/docs/clarion/adr/ADR-035-operational-tuning-discipline.md b/docs/clarion/adr/ADR-035-operational-tuning-discipline.md new file mode 100644 index 00000000..6db08f3b --- /dev/null +++ b/docs/clarion/adr/ADR-035-operational-tuning-discipline.md @@ -0,0 +1,367 @@ +# ADR-035: Operational Tuning Discipline — Declared Basis, Override Surface, Retune Trigger, Coupling + +**Status**: Accepted +**Date**: 2026-05-23 +**Deciders**: qacona@gmail.com +**Context**: From-scratch architecture review on 2026-05-22 (`docs/arch-analysis-2026-05-22-1924/`) surfaced five open questions in `04-final-report.md` §8 that a follow-on SME roundtable (solution architect, systems thinker, Python engineer, quality engineer, security engineer) reframed as a single missing-feedback-loop pattern. This ADR is the level-5 (rules) intervention that pattern requires. +**Extends**: [ADR-021](./ADR-021-plugin-authority-hybrid.md) §4 — names four of the eleven operational constants as config keys; this ADR generalises the discipline that ADR-021 §4 already implies for those four and extends it to every operational constant in the workspace. + +## Summary + +Clarion ships with strong **runtime** balancing loops — the path-escape breaker, the crash-loop breaker, the writer-actor's `parent_contains_mismatch` bijection check, the L2 cross-language fixture parity test — and no **design-time** balancing loops. The result is silent drift: hardcoded operational constants accrete without recorded basis, crate-level doc-comments diverge from contents, file LOC grows past the point of legibility, and the artifact that would name the right tuning surface (a config schema, a split plan, a crate boundary) is never written. The five §8 open questions are five surface symptoms of one pattern. + +This ADR commits the project to a uniform discipline. Every operational constant in Clarion source that gates externally observable behaviour MUST declare four things — **stated basis**, **override surface**, **retune trigger**, **coupling** — in a code comment immediately adjacent to the constant (or, for wire-contract constants whose declaration lives in a sibling document, a cross-reference to that document). The same rule shape extends to **file-LOC budgets** (any source file over 1,500 LOC declares a split-trigger condition in its module doc-comment) and to **crate-boundary budgets** (any crate that takes a dependency widening its trust surface or contradicting its `lib.rs` doc-comment declares the trigger for crate extraction). The discipline is enforced by a lint script in `scripts/` that scans Rust + Python sources for the required declarations and fails CI on undeclared operational constants or oversize files. + +This is a level-5 (rules) Meadows intervention — not a level-12 (parameters) one. A `clarion.yaml` config file alone is rejected as insufficient: a config file without the rule that gates how constants graduate from hardcoded → tunable → deprecated would, in the systems thinker's words, "decay back to hardcoded constants by the next sprint — that is exactly how Clarion got here." + +## Context + +### The five §8 open questions + +The 2026-05-22 architecture analysis (`docs/arch-analysis-2026-05-22-1924/04-final-report.md` §8) recorded five questions that the from-scratch catalog could not answer from code alone: + +1. **Why `MAX_FILES_PER_PYRIGHT_SESSION = 25`?** Empirical? Conservative bound on Pyright memory growth? Not knowing makes future tuning a guess. +2. **What is the post-1.0 plan for the four monolith files?** Each has a natural refactor split. Are these on the roadmap, or is the policy "no split until the file actively impedes a change"? +3. **Will `clarion-llm` become a crate?** The `llm_provider.rs` placement in `clarion-core` is the largest single argument against the `lib.rs:1` doc-comment. +4. **What is the architect's stance on `application_id` / `user_version`?** Trivial to add; non-trivial to add retroactively once installed DBs exist in the wild. +5. **Operational tuning roadmap.** Eleven hardcoded limits, plus the 25-file restart, plus the 256/50 batch-cadence constants. WP6 is named in code comments — what is its current status? + +### The roundtable's diagnosis + +Five SME reports (`docs/arch-analysis-2026-05-22-1924/temp/answer-{solution-architect,systems-thinker,python-engineer,quality-engineer,security-engineer}.md`) converged on a single root cause. The systems thinker named it most directly: + +> "The five questions look like five concerns. They are one. Each is a missing-feedback-loop symptom: a place where operational reality has no path back to the artifact that would change behavior. They wear 'parameter' clothing (Level 12) but live at Level 6 (information flows) and Level 5 (rules)." — `answer-systems-thinker.md` + +The archetype is Meadows' **Drift to Low Performance**: each individual deferral is locally rational ("ship 1.0; tune later"; "don't split `host.rs` mid-sprint"; "PRAGMAs are post-1.0") but there is no countervailing signal pushing the other way. Two literal tells in the codebase make the diagnosis concrete: + +- **`crates/clarion-cli/src/analyze.rs:65`** carries `#[allow(clippy::too_many_lines)]` — the standard-lowering act made literal in code. Two more sites at lines 650 and 1190 carry the same allow. +- **`crates/clarion-core/src/plugin/breaker.rs:7`** says: *"Sprint 1 hard-codes the threshold and window per UQ-WP2-10; the config surface (`clarion.yaml:plugin_limits.crash_*`) lands in WP6."* The placeholder where a discipline should be. + +ADR-030 narrowed WP6's 1.0 scope to the on-demand `summary(id)` MCP tool. The operator-tunables work the `breaker.rs:7` comment references **was not folded into the narrowed WP6 scope and is currently un-homed** (solution architect's reading: `answer-solution-architect.md` §5). ADR-021 §4 names four of the eleven constants as config keys (`plugin_limits.max_frame_bytes`, `plugin_limits.max_records_per_run`, `plugin_limits.max_rss_mib`, the manifest-declared `expected_max_rss_mb`) — those are *promised by an Accepted ADR and not implemented*, with no governing rule about how the other seven graduate from hardcoded to tunable. + +### Per-SME contributions to the diagnosis + +The five SME reports each examined a different facet of the same pattern: + +- **Solution architect** (`answer-solution-architect.md`): triangulates the WP6 home from ADR-021 §4 + ADR-030 + the 2026-04-19 WP2 handoff doc; recommends shipping 1.0 with limits hardcoded and landing the config surface in WP6 post-1.0 *as one ADR-021-aligned change rather than dripping per-constant overrides*. The "one ADR-aligned change" framing is what this ADR operationalises. +- **Systems thinker** (`answer-systems-thinker.md`): identifies Level 5 (rules) as the correct leverage point and rejects the level-12 (parameters) intervention of "just add `clarion.yaml`" — the rule about how/when constants graduate is the discipline, not the constants themselves. Names the "drift to low performance" archetype. +- **Python engineer** (`answer-python-engineer.md`): classifies the Python-side constants into wire-contract-pinned (must track Rust counterparts; `MAX_CONTENT_LENGTH = 8 MiB` in `server.py:48` mirrors `ContentLengthCeiling::DEFAULT`; `MAX_UNRESOLVED_CALLEE_EXPR_BYTES = 512` in `pyright_session.py:43` mirrors a same-named Rust constant; `STDERR_TAIL_LIMIT = 65536` in `pyright_session.py:49` mirrors `STDERR_TAIL_BYTES = 64 KiB` in `host.rs`) versus operational tunables (six Pyright-session constants). **None carry a comment naming the Rust counterpart.** This forced the fourth declaration axis — *coupling* — into this ADR's rule shape: without it, a wire-pinned constant and a freely-tunable one get the same declaration form, which obscures the more dangerous case. +- **Quality engineer** (`answer-quality-engineer.md`): enumerates per-constant test coverage (twelve constants — eight Tested, three Weak, one Untested, one Partially tested). The Weak/Untested cluster is `DEFAULT_MAX_RSS_MIB`/`DEFAULT_MAX_NOFILE`/`DEFAULT_MAX_NPROC` — security-enforcement constants whose behavioural coverage is "does not panic." A value change here has no regression net. The discipline this ADR establishes interlocks with that gap: a constant whose retune trigger is named is a constant whose regression test is also nameable. +- **Security engineer** (`answer-security-engineer.md`): names Q5 as "the question with the most teeth" from a STRIDE-D + STRIDE-E perspective. The 11+ values include the entity cap (500k), Content-Length ceiling (8 MiB), path-escape breaker threshold (10/60s), `RLIMIT_AS` (2 GiB), `RLIMIT_NOFILE` (256), `RLIMIT_NPROC` (32), HTTP body limit (16 KiB), concurrency limit (64), request timeout (10s), batch maxima (256/1000), and the Pyright restart. Recompile-to-tune is itself a security posture stance — an operator under active adversarial-plugin pressure cannot tighten the breaker threshold without a rebuild. The recommendation: *at v1.0 these are deliberately frozen so the security policy is uniform across deployments; post-1.0, the path-escape breaker threshold, the entity cap, and the `RLIMIT_AS` ceiling should become operator-tunable with hard floors enforced at config-load time.* That stance survives intact in this ADR's "internal, no override" allowed state. + +### What this ADR is not + +This ADR is **not** a tracking surface in Filigree, an entity-association registry, or a derived-metric dashboard. It is a source-comment + lint-script discipline, period. ADR-029's entity-association binding is a different mechanism for a different problem; the two do not overlap. This ADR also does not invent a `clarion.yaml` field set — ADR-021 §4 already names four of the eleven config keys, and the per-constant override-surface placement is the matter of subsequent WPs (post-1.0 WP6 by the solution architect's recommendation). What this ADR commits is the **rule** that gates whether a constant is declared, where its override surface lives, and what would prompt its retune. + +## Decision + +### 1. The four-axis declaration rule + +Every operational constant in Clarion source that gates externally observable behaviour MUST declare four things, either in a code comment immediately adjacent to the constant or in a sibling doc cross-reference if the constant's authoritative declaration lives elsewhere: + +1. **Stated basis** — the empirical or design rationale for the current value. Acceptable values include "empirical placeholder, see retune trigger" if the value is currently unmeasured; the placeholder string is itself a basis statement and the retune trigger is what discharges it. +2. **Override surface** — where the value can be tuned. The closed enum is: + - `clarion.yaml:` — operator-tunable via the project config file. + - `env:` — operator-tunable via environment variable. + - `plugin.toml:` — plugin-author-tunable per ADR-021's plugin-authority split. + - `recompile` — internal; not exposed. + - `wire:` — pinned on the wire by a wire-contract document; tunable only via an incompatible-version bump in that document. +3. **Retune trigger** — the observable condition that should prompt re-evaluation. The trigger must be expressible as either a metric threshold (e.g., "Pyright RSS exceeds 1.5 GiB before file 25 on a corpus sized at the elspeth scale") or a finding subcode (e.g., "any `CLA-INFRA-PLUGIN-OOM-KILLED` finding observed in production runs"). "If something feels wrong" does not satisfy the rule. +4. **Coupling** — the constant's relationship to other declared values. The closed enum is: + - `independent` — standalone; tunable without affecting any sibling constant. + - `wire-paired-with:` — must match a same-shape constant on the other side of a wire contract. A change requires updating both sides in lockstep. + - `policy-paired-with:` — must satisfy a policy invariant against a sibling constant (e.g., a per-session restart cap that must remain less than a per-run restart cap). + - `floor-of:` — a hard floor below which an operator override is refused at config-load time per ADR-021 §2b's "configuration-surface floor" pattern. + +The rule's spine is the four-axis declaration; the closed enums on **override surface** and **coupling** keep the rule from drifting into prose. The shape is short — four lines of comment per constant in the steady state. + +#### Canonical comment shape + +For the Rust workspace, the canonical adjacent-comment shape is: + +```rust +/// Operational constant. +/// +/// Basis: . +/// Override: . +/// Retune: . +/// Coupling: | policy-paired-with: | floor-of:>. +pub const MAX_EXAMPLE_BYTES: usize = 8 * 1024; +``` + +For the Python plugin, equivalent shape using a module-level `#:` comment block: + +```python +#: Operational constant. +#: +#: Basis: . +#: Override: . +#: Retune: . +#: Coupling: | policy-paired-with: | floor-of:>. +MAX_EXAMPLE_BYTES = 8 * 1024 +``` + +The lint script (see §5) parses the four named tags exactly as written. + +### 2. The eleven operational constants in `clarion-core` + +The 2026-05-22 architecture catalog (`docs/arch-analysis-2026-05-22-1924/02-subsystem-catalog.md` §1 "Concerns") enumerated eleven hardcoded limit constants across `clarion-core`: + +``` +MAX_PROTOCOL_ERROR_FIELD_BYTES (4 KiB, protocol.rs) +MAX_ENTITY_FIELD_BYTES (4 KiB, host.rs) +MAX_ENTITY_EXTRA_BYTES (64 KiB, host.rs) +STDERR_TAIL_BYTES (64 KiB, host.rs) +MAX_HEADER_LINE_BYTES (8 KiB, transport.rs) +MAX_UNRESOLVED_CALLEE_EXPR_BYTES (512, host.rs) +ContentLengthCeiling::DEFAULT (8 MiB, limits.rs) +EntityCountCap::DEFAULT_MAX (500_000, limits.rs) +DEFAULT_MAX_RSS_MIB (limits.rs) +DEFAULT_MAX_NOFILE (limits.rs) +DEFAULT_MAX_NPROC (limits.rs) +``` + +Plus `PYRIGHT_MAX_NPROC = 4096` (host.rs, raised for the language-server runtime). All twelve MUST be retrofitted to the four-axis declaration before the 1.1 release. + +For the Python plugin, the inventory enumerated by `answer-python-engineer.md` is: + +``` +MAX_CONTENT_LENGTH (8 MiB, server.py:48, wire-paired-with ContentLengthCeiling::DEFAULT) +MAX_FILES_PER_PYRIGHT_SESSION (25, server.py:49, operational; see §3 below) +MAX_PYRIGHT_RESTARTS_PER_RUN (3, pyright_session.py:44, policy-paired with the 25-file recycle) +PYRIGHT_INIT_TIMEOUT_SECS (30.0, pyright_session.py:46) +PYRIGHT_CALL_TIMEOUT_SECS (5.0, pyright_session.py:47) +PYRIGHT_FILE_TIMEOUT_SECS (3.0, pyright_session.py:48) +MAX_REFERENCE_SITES_PER_FILE (2000, pyright_session.py:45) +MAX_UNRESOLVED_CALLEE_EXPR_BYTES (512, pyright_session.py:43, wire-paired-with Rust same-name) +STDERR_TAIL_LIMIT (65536, pyright_session.py:49, wire-paired-with STDERR_TAIL_BYTES) +``` + +Plus the writer-actor cadence constants in `crates/clarion-storage/src/writer.rs`: + +``` +DEFAULT_BATCH_SIZE (50, writer.rs:38) +DEFAULT_CHANNEL_CAPACITY (256, writer.rs:35) +``` + +And the crash-loop breaker constants in `crates/clarion-core/src/plugin/breaker.rs`: + +``` +CRASH_LOOP_THRESHOLD (rolling-window count) +CRASH_LOOP_WINDOW (rolling-window duration) +``` + +These are the constants the lint script will fail CI on if undeclared after the 1.1 release. Constants discovered after this ADR's authoring inherit the same rule from their first commit — there is no grandfather clause for new code. + +### 3. The 25-file Pyright restart constant — instance application + +`MAX_FILES_PER_PYRIGHT_SESSION = 25` (`plugins/python/src/clarion_plugin_python/server.py:49`) is the canonical worked example. Per `answer-python-engineer.md` and `answer-solution-architect.md`, the value was introduced in commit `68b719c` ("Bound Pyright dogfood analysis", 2026-05-20) with no commit-message rationale and no in-file comment. It is an empirical placeholder. The four-axis declaration for it, after retrofit, must be: + +- **Basis**: empirical placeholder; bound chosen during dogfood analysis to cap observed Pyright RSS growth across `textDocument/didOpen` cycles; not yet validated against a per-file RSS delta curve. +- **Override**: `plugin.toml:pyright.files_per_session` (plugin-author-tunable per ADR-021's plugin-authority split — the Pyright session recycle is plugin-owned policy, not core-side). +- **Retune**: any sampled Pyright subprocess RSS curve on the elspeth-scale corpus showing the inflection point is below 25 (recycle is wasted re-init cost) or above 25 (`CLA-INFRA-PLUGIN-OOM-KILLED` risk inside the 25-file window). +- **Coupling**: `policy-paired-with:MAX_PYRIGHT_RESTARTS_PER_RUN` — server-side recycling at 25 files and session-side restart caps interact: the Python engineer documented a concrete failure mode where `_disabled` and `_restart_count` are instance-scoped on `PyrightSession` and reset at every 25-file boundary, breaking the "per run" intent of the restart cap. That interaction is a coupling fact the rule forces to be visible. + +The policy-paired-with coupling annotation interlocks with the Python engineer's `answer-python-engineer.md` "Failure mode A": once both constants declare the pairing, the next reviewer asking "is this safe to change in isolation?" reads the coupling and finds the answer in the source. + +### 4. File-LOC budget rule + +Any source file over **1,500 LOC** declares a split-trigger condition in its module doc-comment. The declaration shape is: + +```rust +//! … +//! +//! ## LOC budget +//! +//! Current LOC at last review: (). +//! Split trigger: . +//! Rationale for current state: . +``` + +The 1,500 LOC threshold is the floor. Files **already** over budget at this ADR's authoring receive a one-time grace period: the trigger must be declared in each file's module doc-comment before the 1.1 release. The four files in this category, with the catalog's snapshot LOC alongside the LOC at ADR authoring (the catalog was snapshotted before recent work; the working copy is smaller for two of the four): + +| File | Catalog LOC (2026-05-22) | LOC at ADR authoring | Split-trigger (per `answer-solution-architect.md` §2) | +|---|---|---|---| +| `crates/clarion-mcp/src/lib.rs` | 4,703 | 3,449 | Adding a 20th MCP tool, or any concurrent-tool-execution work. | +| `crates/clarion-core/src/plugin/host.rs` | 2,935 | 2,935 | Adding a fifth enforcement layer or changing the four-stage pipeline ordering. | +| `crates/clarion-cli/src/analyze.rs` | 2,549 | 2,427 | A 14th phase added to `run_with_options`'s current 13-phase linearisation. | +| `crates/clarion-core/src/llm_provider.rs` | 2,467 | 2,467 | See ADR-035 §6 and `answer-solution-architect.md` §3 — this file's split is not "within crate" but "extract to `clarion-llm`." | + +The triggers above are recommendations from `answer-solution-architect.md` §2 and become the declared triggers in each file's module doc-comment via the grace-period retrofit. Each file's owner may revise its declared trigger in a later commit; the rule binds *declaration*, not *which specific trigger is declared*. + +### 5. Crate-boundary budget rule + +Any crate that takes a dependency widening its trust surface or contradicting its `lib.rs` doc-comment declares the trigger for crate extraction in its top-level `lib.rs` doc-comment. The shape is: + +```rust +//! `` — . +//! +//! ## Crate-boundary budget +//! +//! Boundary statement: . +//! Extraction trigger: . +//! Currently in-scope but extraction-candidate: . +``` + +The current cited contradiction (per `answer-solution-architect.md` §3, `answer-security-engineer.md` Q3) is `clarion-core/src/lib.rs:1`'s doc-comment versus `clarion-core/src/llm_provider.rs`'s content. The boundary statement says "domain types, identifiers, and provider traits"; the content includes the OpenRouter `reqwest` HTTP transport and two CLI-subprocess providers — neither domain types, nor identifiers, nor trait definitions. **`detailed-design.md:1745` already names `clarion-llm` as one of the intended workspaces.** That intent + the security argument (an outbound-HTTP-stack CVE inside the plugin-supervisor crate widens the supervisor's trust surface) supplies the extraction trigger: *the `clarion-llm` extraction MUST land before any new LLM provider is added or before any change to `reqwest` / `rustls` / `hyper` that introduces a new transitive trust dependency.* + +`clarion-core/src/lib.rs` carries this declaration as part of the 1.1 grace-period retrofit. The same rule applies prospectively: a new crate whose `lib.rs` doc-comment is contradicted by its contents must either fix the contradiction or declare an extraction trigger. + +### 6. Lint script and CI gate + +A lint script lives at `scripts/operational-tuning-lint.{rs,py}` (the implementation language is at the script author's discretion, but the script is run from CI). The script: + +1. Walks the Rust workspace under `crates/**/src/**/*.rs` and the Python plugin under `plugins/python/src/**/*.py`. +2. Identifies operational-constant candidates by syntactic match: `pub const` / `const` at module level in Rust; top-level `MAX_*` / `DEFAULT_*` / `*_TIMEOUT_*` / `*_LIMIT_*` / `*_BYTES` / `*_CAP` / `*_THRESHOLD` / `*_WINDOW` identifier patterns in Python. (The pattern set is conservative and pinned in the script; it can be widened in a later commit.) +3. For each candidate, verifies that an adjacent `Basis: … / Override: … / Retune: … / Coupling: …` declaration is present and that `Override` and `Coupling` values come from the closed enums in §1. +4. For each `.rs` file over 1,500 LOC, verifies that the module doc-comment contains an `## LOC budget` section with `Current LOC`, `Split trigger`, `Rationale for current state` lines. +5. For each `lib.rs` whose doc-comment contradicts its content (heuristic: presence of `reqwest` / `tokio::process` / network crates not named in the boundary statement), verifies that a `Crate-boundary budget` section is present. +6. Emits findings in the same JSON shape Clarion's other tooling uses (subcode prefix `CLA-DISC-TUNING-*`), one per undeclared constant or undeclared oversize file. + +The script is wired into CI as a non-blocking warning until the 1.1 release. **At the 1.1 release the gate flips from warning to failure** — any undeclared operational constant, oversize file, or contradicted crate boundary fails the CI build. The flip date is the trigger for landing the retrofits enumerated in §2, §4, and §5. + +The three `#[allow(clippy::too_many_lines)]` sites in `crates/clarion-cli/src/analyze.rs` (lines 65, 650, 1190) MUST be either re-enabled (the underlying functions split) or replaced with a documented `// allow: ADR-035 §4 — declared split-trigger: ` comment before the 1.1 release. The clippy threshold itself (`too-many-lines-threshold = 120` per ADR-023's `clippy.toml`) remains the baseline; an `#[allow]` without an ADR-035 reference fails the lint script regardless of whether `cargo clippy` itself passes. + +### 7. Constant-graduation lifecycle + +A constant's life under the rule has three states: + +1. **Hardcoded with declaration.** The constant is in source, has the four-axis declaration, and `Override = recompile`. This is the steady state for constants the operator should not touch — security-uniformity constants per `answer-security-engineer.md` Q5, wire-contract constants whose change is an `api_version` bump, and constants whose retune trigger has not yet fired. +2. **Tunable.** The constant has `Override = clarion.yaml:` (or `env:` or `plugin.toml:`) and the config-loader actually reads the value. Hard floors enforced at config-load time per ADR-021 §2b are part of this state — the operator can raise the value but not lower it past the floor. +3. **Deprecated.** The constant's basis is replaced by a superseding constant or rule; the declaration carries a `Deprecated: see ` line; the constant is removed at the next major version bump. + +Graduating a constant from state 1 to state 2 is a normal commit; graduating from state 2 to state 3 requires either an ADR or a documented field-deprecation note in `docs/clarion/1.0/detailed-design.md`. Demoting from state 2 back to state 1 — removing an operator surface — requires an ADR. This asymmetry codifies the "ratchet" the systems thinker named. + +### 8. What is explicitly out of scope for this ADR + +- **The wire-pinned batch cap of 256 queries** in `POST /api/v1/files/batch` is pinned by ADR-034 §3 with `Override = wire:contracts.md#batch-cap` semantics. ADR-035's rule applies (the constant must carry a four-axis declaration), but the override surface is not operator-tunable; a change is an `api_version: 2` event by ADR-034's existing rule. +- **`application_id` and `user_version` PRAGMAs** for SQLite are tracked as filigree issue `clarion-f2a984fd6d` per `answer-solution-architect.md` §4 and `answer-quality-engineer.md` Q4. ADR-035's "schema-identity" instance of the rule applies (the PRAGMA value MUST carry a `Basis: identifies the file as Clarion's per ADR-035` declaration), but the implementation lands as the filigree issue's deliverable, not this ADR's. +- **Specific config-schema design** for `clarion.yaml` post-1.0 belongs in WP6 per `answer-solution-architect.md` §5. This ADR commits the rule about how constants graduate; the specific YAML key shape is the matter of the work package that implements the graduation. + +## Consequences + +### Positive + +- The five §8 open questions collapse to one durable artifact. Q1 (the 25-file constant), Q4 (PRAGMA identity), and Q5 (eleven hardcoded limits) each gain a recorded basis and a tuning surface; Q2 (four monoliths) and Q3 (`clarion-llm` extraction) become budgeted properties with named triggers rather than aesthetic preferences competing with sprint load. +- The runtime balancing loops already in the project (path-escape breaker, crash-loop breaker, parent-contains bijection, L2 parity fixture) gain a design-time peer. Per the systems thinker, "the runtime has back-pressure; the architecture itself does not." This ADR is the missing back-pressure at the design-time layer. +- The constant-coupling axis surfaces dangerous interactions that today are invisible. Once `MAX_FILES_PER_PYRIGHT_SESSION` and `MAX_PYRIGHT_RESTARTS_PER_RUN` both carry `policy-paired-with:`, the Python engineer's "Failure mode A" (the 25-file boundary silently resetting the per-run restart cap) becomes a fact a reviewer reads in the source, not a defect a future SME has to rediscover. Similarly, the three wire-paired-with Python ↔ Rust constants gain the `# NOTE: must match clarion-core/…` comment the Python engineer flagged as missing. +- The lint script is the enforcement mechanism that prevents this ADR from being aspirational. A rule the project does not enforce is, per the same drift-to-low-performance archetype, a rule the project does not have. The CI gate is the countervailing signal. +- ADR-021's "configurable" claim becomes a substantive promise on a known timeline. Today, "operator can tune" is aspirational (per `02-subsystem-catalog.md` §1); under this ADR, every constant that says it is tunable has a config-load-time floor and a retune trigger. Constants that say they are not tunable carry that as a deliberate stance (security-uniformity per `answer-security-engineer.md` Q5), not a deferral. +- Cross-product trust-surface arguments become first-class. The `clarion-llm` extraction (per `answer-security-engineer.md` Q3) is a defense-in-depth win once it lands: an outbound-HTTP-stack CVE no longer reaches the plugin-supervisor crate. The crate-boundary budget rule (§5) gives that extraction a declared trigger rather than a vague intention in `detailed-design.md`. + +### Negative + +- Source comment volume increases. Every operational constant gains 4–5 lines of comment. For the twelve constants in `clarion-core` plus the nine in the Python plugin plus the two writer-actor cadence constants plus the two breaker constants, the retrofit is on the order of 100 lines of comment across the workspace — bounded but non-trivial. +- The lint script is one more piece of in-repo tooling to maintain. ADR-023's existing five CI gates become six (fmt, clippy, nextest, doc, deny, **tuning-lint**). The script must keep pace with new constant-naming patterns (e.g., a future `MAX_CALLEE_DEPTH` that doesn't match the current `MAX_*_BYTES` heuristic). Mitigation: the script's pattern set is in source, reviewable, and one PR away from a fix. +- The four-axis declaration imposes authoring friction on new constants. A contributor adding a new `MAX_FOO` must, before merging, identify the constant's basis, decide its override surface, name its retune trigger, and trace its coupling. This is the intended shape — the friction is the rule — but it raises the per-commit cost of adding limits compared to today. +- The 1,500-LOC budget threshold is a judgement call. Set lower, it would force more files to declare triggers (potentially valuable but noisier); set higher, it would let `host.rs` and `analyze.rs` evade declaration. 1,500 LOC is chosen as the floor where a file becomes hard to read end-to-end in one sitting; it is not derived from a study. Mitigation: this threshold is itself an operational constant under this ADR's rule, and its retune trigger is "any file declares a trigger and the trigger is consistently 'never'" — a sign the threshold is too lax. +- The `clarion-llm` extraction trigger commits a future split that may bring its own churn. Per `answer-solution-architect.md` §3, the LLM surface is at its minimum scope right now (post-ADR-030 narrowing of WP6); the split is cheap today and expensive later. The risk is that "ship the v1.0 tag" pressure interacts with "land the crate extraction" and either delays the tag or rushes the split. Mitigation: the trigger names the condition (any new LLM provider, any change to the network-stack transitive deps), so the split is reactive to a forcing function rather than scheduled. + +### Neutral + +- This is a level-5 (rules) Meadows intervention. Leverage-point hierarchy: level 12 is "parameters" (the constants themselves), level 6 is "information flows" (the four-axis declaration is one), level 5 is "rules" (this ADR). A level-12 intervention — just adding `clarion.yaml` keys without the rule — would, per `answer-systems-thinker.md`, decay back to hardcoded constants by the next sprint. A level-6 intervention — emitting metrics on every constant's observable behaviour — would over-instrument before knowing which constants matter. The level-5 rule is the floor: it forces declaration without forcing exposure or instrumentation; the exposure and instrumentation become subsequent commits gated by declared retune triggers. +- The "internal, no override" state (state 1 in §7) is an allowed terminal state. A constant whose basis is "uniform security policy across deployments per ADR-035 §3 stance" and whose override surface is `recompile` is fully compliant with the rule. The discipline is *declaration*, not *exposure*; the security engineer's concern that exposing all 11 limits as operator surface creates a support burden (`answer-security-engineer.md` Q5) is preserved by allowing this state. +- ADR-029's entity-association binding is a different mechanism for a different problem. ADR-029 binds Filigree issues to source entities; ADR-035 imposes a source-comment discipline on operational constants. The two mechanisms can be combined (an unresolved retune trigger could be filed as a filigree issue and bound to the constant's entity ID via ADR-029) but neither requires the other. This ADR is not a Filigree integration. +- ADR-034 §3's wire-pinned batch cap of 256 queries is an instance of the rule, not an exception to it. The constant carries `Override = wire:contracts.md#batch-cap` and `Coupling = wire-paired-with:Filigree client splitting logic`. The override-surface enum's `wire:*` value exists exactly for this case. +- This ADR's "lint script in CI" enforcement mechanism is itself an operational constant under the rule (its `Retune` trigger is "any retroactive retrofit of new constants becomes routinely required at code-review time rather than at lint-script time, suggesting the lint set is too narrow"). The rule applies to itself. + +## Alternatives Considered + +### Alternative 1: ship `clarion.yaml` with all eleven constants as keys and call it done + +**Pros**: most operationally tactile; an operator can `cat clarion.yaml` and see what is tunable. Implementation surface is bounded — eleven keys, eleven config-loader sites, eleven floor checks. Matches ADR-021 §4's existing four named keys and extends them naturally to the other seven. + +**Cons**: this is a level-12 (parameters) intervention to a drift-to-low-performance loop. Per `answer-systems-thinker.md`: + +> "A config file without the rule decays back to hardcoded constants on the next sprint — that is exactly how Clarion got here. The rule is what creates the surface; the surface alone is a parameter intervention (Level 12) and parameter interventions to a drift-to-low-performance loop reset the constant without changing the slope." + +The eleven constants in `clarion-core` were not added all at once; they accreted over Sprints 1 and 2. Adding `clarion.yaml` without the rule that gates how the *next* constant graduates means the twelfth, thirteenth, fourteenth constants land hardcoded again, with `breaker.rs:7`-style "lands in WP6" comments, and the cycle repeats. The discipline must precede the surface, not follow it. + +**Why rejected**: the parameter intervention does not change the rate at which new constants accrete; only the rule does. + +### Alternative 2: defer the rule to the WP6 work package post-1.0 + +**Pros**: matches the solution architect's "ship 1.0 with the limits hardcoded; land the config surface in WP6 post-1.0 as one ADR-021-aligned change" recommendation almost verbatim. Avoids ADR churn during release cut. The five §8 questions are not 1.0 blockers; the gap-register has `application_id`/`user_version` as the only constant-related v1.0 blocker, and that has its own filigree issue. + +**Cons**: the solution architect's recommendation is about *implementation timing* of the config surface, not about *whether the rule exists*. The rule and the implementation are separable: this ADR commits the rule now (level 5), and WP6's post-1.0 implementation lands the surface on the rule's existing framework (level 12 underneath the level-5 rule). Deferring the rule itself means WP6 lands the surface without a governing discipline — the same configuration-without-rule trap Alternative 1 falls into, with a one-WP delay. + +The systems thinker's recommendation is explicit: *"Promote it to Accepted before any further hardcoded limit lands."* The cost of accepting the rule now is the four-axis-declaration retrofit. The cost of accepting the rule later is that any constant added between now and WP6 lands without the discipline, has to be retrofitted twice (once for the rule, once for the WP6 graduation), and the lint-script gate flips later, lengthening the drift window. + +**Why rejected**: rule and implementation are separable; accepting the rule now is cheaper than accepting it later, and the rule is what makes the implementation durable. + +### Alternative 3: rely on code review to catch undeclared constants + +**Pros**: no new tooling. Reviewers already inspect new constants for naming, type, and call-site usage; adding "is there a retune trigger?" to the review checklist is one more line on a checklist. + +**Cons**: code review is a probabilistic catch, not a deterministic one. The systems thinker's archetype — drift to low performance — predicts exactly the failure mode where reviewers locally accept "we'll document this later" and the documentation never lands. The five §8 questions are evidence the catch-rate is below what the project needs. The literal `#[allow(clippy::too_many_lines)]` at `analyze.rs:65` is what reviewer-only enforcement looks like in this codebase today: a reviewer accepted the allow because it shipped the function; the standard-lowering act is visible in code. + +A CI lint script is deterministic: the build either fails or it does not. The countervailing signal is mechanical, not human, and runs on every PR. + +**Why rejected**: code review's catch-rate is what this ADR is trying to compensate for, not augment. + +### Alternative 4: extract `clarion-llm` immediately as the high-impact intervention; defer the broader rule + +**Pros**: closes the security-engineer's STRIDE-T/STRIDE-I argument fastest (the outbound-HTTP stack stops sharing a crate with the plugin supervisor); concrete deliverable; survives `cargo clippy`. + +**Cons**: solves one of the five questions and leaves the other four unaddressed. The five questions are one pattern; intervening on Q3 alone is a parameter-level move on a single constant (the crate boundary) without changing the slope on the others. The next sprint adds the twelfth hardcoded limit and the §8 list grows from five questions to six. + +Also: the extraction *is* part of this ADR (§5's crate-boundary budget rule, with the trigger naming exactly the security-engineer's condition). Accepting the rule causes the extraction; accepting only the extraction does not cause the rule. + +**Why rejected**: a single-constant intervention does not address a pattern of multiple constants accreting under the same archetype. + +### Alternative 5: an ADR that demands a richer declaration (per-constant doc page; per-constant test; per-constant metric) + +**Pros**: maximum traceability; a future architect could query any constant and find its full lineage. + +**Cons**: scope-creep into ceremony. The systems thinker's risk register named this explicitly: + +> "An ADR that demands a basis for every constant could ossify into ceremony. Mitigation: the basis statement is one sentence; the trigger is one finding subcode. Anything more is the wrong shape." + +A per-constant test is the right discipline for *some* constants (the security-enforcement cluster per `answer-quality-engineer.md` Q5) but not all (the ring-buffer-overflow eviction in `STDERR_TAIL_BYTES` is "Partially tested" and acceptable). A per-constant metric over-instruments before knowing which constants matter operationally. A per-constant doc page is what `02-subsystem-catalog.md` already produces at architecture-review time; pre-producing them for every constant inverts the relationship. + +The four-axis declaration is the floor — additional discipline can be layered above it (a quality-engineering follow-up to add a behavioural test for `DEFAULT_MAX_RSS_MIB` is fully within scope of the current quality engineering sheets) without enlarging the rule itself. + +**Why rejected**: the rule's shape — short, mechanical, lint-checkable — is what makes it a level-5 intervention rather than ceremony. Richer shapes are level-6/7 interventions appropriate for specific constant clusters, not the workspace-wide floor. + +## Related Decisions + +- [ADR-021](./ADR-021-plugin-authority-hybrid.md) — Plugin authority hybrid; §4 names four of the eleven operational constants (`plugin_limits.max_frame_bytes`, `plugin_limits.max_records_per_run`, `plugin_limits.max_rss_mib`, `expected_max_rss_mb`) as `clarion.yaml` config keys. ADR-035 generalises the discipline that ADR-021 §4 already implies and extends it to every operational constant. +- [ADR-023](./ADR-023-tooling-baseline.md) — Tooling baseline; defines `clippy.toml`'s `too-many-lines-threshold = 120`. ADR-035 §4's 1,500-LOC budget operates above ADR-023's per-function threshold — the two thresholds apply to different units (function vs. file) and do not conflict. The `#[allow(clippy::too_many_lines)]` sites in `analyze.rs` are ADR-023 escapes; ADR-035 §6 requires either re-enabling them or pairing each with a declared split trigger. +- [ADR-030](./ADR-030-on-demand-summary-scope.md) — Narrowed WP6 to the on-demand `summary(id)` MCP tool, leaving the operator-tunables work currently un-homed per `answer-solution-architect.md` §5. ADR-035 commits the governing rule; the implementation of the config surface lands in a post-1.0 work package (TBD) that operates on top of this ADR's framework. +- [ADR-034](./ADR-034-federation-http-read-api-hardening.md) — Federation HTTP read API hardening; §3's wire-pinned 256-query batch cap is an instance of ADR-035's rule (`Override = wire:contracts.md#batch-cap`). The two ADRs do not conflict; ADR-034 specifies the wire contract, ADR-035 specifies the source-comment discipline. +- [ADR-029](./ADR-029-entity-associations-binding.md) — Entity-association binding; explicitly orthogonal to ADR-035. An unresolved retune trigger could be filed as a filigree issue and bound to the constant's entity ID via ADR-029, but the binding is not required. + +## References + +### Originating analysis (2026-05-22 architecture review) + +- Final report `§8 Open Questions`: [`docs/arch-analysis-2026-05-22-1924/04-final-report.md`](../../arch-analysis-2026-05-22-1924/04-final-report.md#8-open-questions-for-the-next-phase) — the five questions this ADR's rule absorbs. +- Subsystem catalog `clarion-core` Concerns: [`02-subsystem-catalog.md`](../../arch-analysis-2026-05-22-1924/02-subsystem-catalog.md) §1 — the eleven-limit enumeration; the "every limit is a recompile" tell; the `mock.rs` 876-LOC sign of non-trivial host pipeline state. +- Subsystem catalog `clarion-storage` Concerns: §2 — the `application_id`/`user_version` gap; the writer-actor cadence constants (256 / 50) at `writer.rs:35,38`. + +### SME roundtable (2026-05-23) + +- Solution architect: [`temp/answer-solution-architect.md`](../../arch-analysis-2026-05-22-1924/temp/answer-solution-architect.md) — WP6 home triangulation; the "ship 1.0 with limits hardcoded; land config surface in WP6 as one ADR-021-aligned change" frame; the per-file split-trigger table. +- Systems thinker: [`temp/answer-systems-thinker.md`](../../arch-analysis-2026-05-22-1924/temp/answer-systems-thinker.md) — the level-5 (rules) leverage argument; the drift-to-low-performance archetype; the `analyze.rs:74` and `breaker.rs:7` smoking-gun tells (line numbers as recorded at analysis time; current `analyze.rs` `#[allow(clippy::too_many_lines)]` site has shifted to line 65 with two additional sites at 650 and 1190; the rule applies to all three). +- Python engineer: [`temp/answer-python-engineer.md`](../../arch-analysis-2026-05-22-1924/temp/answer-python-engineer.md) — the wire-contract-pinned vs. operational-tunable Python constant classification; the `MAX_PYRIGHT_RESTARTS_PER_RUN` "per run" name vs. per-instance implementation interaction failure; the basis for the `Coupling` axis. +- Quality engineer: [`temp/answer-quality-engineer.md`](../../arch-analysis-2026-05-22-1924/temp/answer-quality-engineer.md) — the per-constant test-coverage matrix; the `DEFAULT_MAX_RSS_MIB`/`DEFAULT_MAX_NOFILE`/`DEFAULT_MAX_NPROC` security-enforcement cluster as the highest-risk untested area. +- Security engineer: [`temp/answer-security-engineer.md`](../../arch-analysis-2026-05-22-1924/temp/answer-security-engineer.md) — the STRIDE-D/STRIDE-E framing of recompile-to-tune as a security-posture stance; the `clarion-llm` extraction as STRIDE-T/STRIDE-I defense-in-depth; the security-uniformity argument for keeping some constants `Override = recompile`. + +### Source-of-truth code locations + +- `crates/clarion-core/src/plugin/limits.rs` — `ContentLengthCeiling::DEFAULT`, `EntityCountCap::DEFAULT_MAX`, `DEFAULT_MAX_RSS_MIB`, `DEFAULT_MAX_NOFILE`, `DEFAULT_MAX_NPROC`. +- `crates/clarion-core/src/plugin/host.rs` — `MAX_ENTITY_FIELD_BYTES`, `MAX_ENTITY_EXTRA_BYTES`, `STDERR_TAIL_BYTES`, `MAX_UNRESOLVED_CALLEE_EXPR_BYTES`, `PYRIGHT_MAX_NPROC`. +- `crates/clarion-core/src/plugin/breaker.rs:7` — the literal "lands in WP6" comment that this ADR retires; `CRASH_LOOP_THRESHOLD`, `CRASH_LOOP_WINDOW`. +- `crates/clarion-core/src/plugin/protocol.rs` — `MAX_PROTOCOL_ERROR_FIELD_BYTES`. +- `crates/clarion-core/src/plugin/transport.rs` — `MAX_HEADER_LINE_BYTES`. +- `crates/clarion-storage/src/writer.rs:35,38` — `DEFAULT_CHANNEL_CAPACITY = 256`, `DEFAULT_BATCH_SIZE = 50`. +- `crates/clarion-cli/src/analyze.rs:65,650,1190` — three `#[allow(clippy::too_many_lines)]` sites that ADR-035 §6 requires either re-enabling or pairing with a declared split trigger before the 1.1 release. +- `plugins/python/src/clarion_plugin_python/server.py:48,49` — `MAX_CONTENT_LENGTH = 8 MiB` (wire-paired-with Rust), `MAX_FILES_PER_PYRIGHT_SESSION = 25`. +- `plugins/python/src/clarion_plugin_python/pyright_session.py:43-49` — six Pyright-session operational constants enumerated by `answer-python-engineer.md`. + +### Doctrine the rule operationalises + +- Meadows, Donella H., *Thinking in Systems: A Primer*. Level-5 (rules) and level-6 (information flows) leverage points in the twelve-level hierarchy. +- The doctrine in [`docs/suite/loom.md`](../../suite/loom.md) §5 — the federation axiom's "enrich-only" rule applies by analogy to constants: a constant that gates externally observable behaviour without a declared basis is the same archetype as a sibling tool that adds a required dependency. + +— End of ADR-035 — diff --git a/plugins/python/src/clarion_plugin_python/pyright_session.py b/plugins/python/src/clarion_plugin_python/pyright_session.py index 0addfc06..3e6ca295 100644 --- a/plugins/python/src/clarion_plugin_python/pyright_session.py +++ b/plugins/python/src/clarion_plugin_python/pyright_session.py @@ -40,6 +40,23 @@ FINDING_PYRIGHT_REFERENCE_RESOLUTION_TIMEOUT = "CLA-PY-REFERENCE-RESOLUTION-TIMEOUT" FINDING_PYRIGHT_REFERENCE_SITE_CAP = "CLA-PY-REFERENCE-SITE-CAP" + +@dataclass +class PyrightRunState: + """Run-wide pyright health budget, shared across session recycles. + + A ``PyrightSession`` is recycled every ``MAX_FILES_PER_PYRIGHT_SESSION`` + files to bound memory growth. Without a shared budget the 3-restart cap + resets at every recycle boundary, letting a crash-looping pyright silently + consume ``ceil(N/25) * 3`` restarts instead of 3 for an entire analysis + run. Pass the same ``PyrightRunState`` instance to every successive + ``PyrightSession`` so the budget is enforced across the full run. + """ + + restart_count: int = 0 + disabled: bool = False + + MAX_UNRESOLVED_CALLEE_EXPR_BYTES = 512 MAX_PYRIGHT_RESTARTS_PER_RUN = 3 MAX_REFERENCE_SITES_PER_FILE = 2000 @@ -141,6 +158,7 @@ def __init__( # noqa: PLR0913 - knobs are tested lifecycle boundaries. file_timeout_secs: float = PYRIGHT_FILE_TIMEOUT_SECS, max_restarts_per_run: int = MAX_PYRIGHT_RESTARTS_PER_RUN, max_reference_sites_per_file: int = MAX_REFERENCE_SITES_PER_FILE, + run_state: PyrightRunState | None = None, ) -> None: self.project_root = Path(project_root).resolve() self.executable = executable @@ -151,12 +169,15 @@ def __init__( # noqa: PLR0913 - knobs are tested lifecycle boundaries. self.file_timeout_secs = file_timeout_secs self.max_restarts_per_run = max_restarts_per_run self.max_reference_sites_per_file = max_reference_sites_per_file + # Run-wide health budget: shared across session recycles when the caller + # passes an explicit ``run_state``; isolated (per-instance) otherwise, + # which preserves the existing contract for code that constructs + # ``PyrightSession`` directly without going through ``ServerState``. + self._run_state = run_state if run_state is not None else PyrightRunState() self._process: subprocess.Popen[bytes] | None = None self._stderr_thread: threading.Thread | None = None self._stderr_tail = bytearray() self._next_id = 1 - self._restart_count = 0 - self._disabled = False self._findings: list[Finding] = [] self._function_indexes: dict[Path, _FunctionIndex] = {} self._index_parse_latency_ms: list[int] = [] @@ -584,7 +605,7 @@ def _target_id_from_location(self, location: object) -> tuple[str | None, bool]: return target_index.module_id, False def _ensure_process(self) -> bool: - if self._disabled: + if self._run_state.disabled: return False if self._process is None: return self._start_process() @@ -592,32 +613,32 @@ def _ensure_process(self) -> bool: return True self._process = None self._record_restart_or_poison("pyright subprocess exited") - if self._disabled: + if self._run_state.disabled: return False return self._start_process() def _record_restart_or_poison(self, reason: str) -> None: - self._restart_count += 1 - if self._restart_count > self.max_restarts_per_run: - self._disabled = True + self._run_state.restart_count += 1 + if self._run_state.restart_count > self.max_restarts_per_run: + self._run_state.disabled = True self._record_finding( FINDING_PYRIGHT_POISON_FRAME, "pyright restart cap exceeded; skipping call resolution", - restart_count=self._restart_count, + restart_count=self._run_state.restart_count, reason=reason, ) return self._record_finding( FINDING_PYRIGHT_RESTART, "pyright subprocess died and was restarted", - restart_count=self._restart_count, + restart_count=self._run_state.restart_count, reason=reason, ) def _start_process(self) -> bool: executable = self._resolve_executable() if executable is None: - self._disabled = True + self._run_state.disabled = True self._record_finding( FINDING_PYRIGHT_UNAVAILABLE, "pyright-langserver is not available", @@ -625,7 +646,7 @@ def _start_process(self) -> bool: ) return False if self.install_check is not None and not self.install_check(executable): - self._disabled = True + self._run_state.disabled = True self._record_finding( FINDING_PYRIGHT_INSTALL_FAILURE, "pyright-langserver executability check failed", @@ -643,7 +664,7 @@ def _start_process(self) -> bool: stderr=subprocess.PIPE, ) except OSError as exc: - self._disabled = True + self._run_state.disabled = True self._record_finding( FINDING_PYRIGHT_INSTALL_FAILURE, "pyright-langserver failed to start", @@ -657,7 +678,7 @@ def _start_process(self) -> bool: try: self._initialize() except LspTimeoutError: - self._disabled = True + self._run_state.disabled = True self._record_finding( FINDING_PYRIGHT_INIT_TIMEOUT, "pyright initialize handshake timed out", @@ -667,7 +688,7 @@ def _start_process(self) -> bool: process.wait(timeout=2) return False except (LspTransportClosedError, BrokenPipeError, OSError) as exc: - self._disabled = True + self._run_state.disabled = True self._record_finding( FINDING_PYRIGHT_UNAVAILABLE, "pyright initialize handshake failed", diff --git a/plugins/python/src/clarion_plugin_python/server.py b/plugins/python/src/clarion_plugin_python/server.py index 3303d65e..b24d4f4c 100644 --- a/plugins/python/src/clarion_plugin_python/server.py +++ b/plugins/python/src/clarion_plugin_python/server.py @@ -29,7 +29,7 @@ from clarion_plugin_python import __version__ from clarion_plugin_python.extractor import extract_with_stats -from clarion_plugin_python.pyright_session import PyrightSession +from clarion_plugin_python.pyright_session import PyrightRunState, PyrightSession from clarion_plugin_python.stdout_guard import install_stdio from clarion_plugin_python.wardline_probe import probe as wardline_probe @@ -68,6 +68,7 @@ class ServerState: project_root: Path | None = field(default=None) pyright: PyrightSession | None = field(default=None) pyright_files_since_restart: int = 0 + pyright_run_state: PyrightRunState = field(default_factory=PyrightRunState) def read_frame(stream: IO[bytes]) -> dict[str, Any] | None: @@ -195,7 +196,10 @@ def handle_analyze_file(params: dict[str, Any], state: ServerState) -> dict[str, return {"entities": [], "edges": [], "stats": empty_stats} path = Path(file_path_raw) if state.pyright is None: - state.pyright = PyrightSession(state.project_root or path.parent) + state.pyright = PyrightSession( + state.project_root or path.parent, + run_state=state.pyright_run_state, + ) try: source = path.read_text(encoding="utf-8") except (OSError, UnicodeDecodeError) as exc: diff --git a/plugins/python/tests/test_server.py b/plugins/python/tests/test_server.py index 106c1c02..e8533aba 100644 --- a/plugins/python/tests/test_server.py +++ b/plugins/python/tests/test_server.py @@ -17,6 +17,11 @@ from clarion_plugin_python import server as server_module from clarion_plugin_python.call_resolver import CallResolutionResult +from clarion_plugin_python.pyright_session import ( + FINDING_PYRIGHT_RESTART, + PyrightRunState, + PyrightSession, +) from clarion_plugin_python.reference_resolver import ReferenceResolutionResult, ReferenceSite if TYPE_CHECKING: @@ -266,7 +271,7 @@ def test_analyze_file_lazy_initializes_pyright( monkeypatch: pytest.MonkeyPatch, ) -> None: class FakePyrightSession: - def __init__(self, project_root: Path) -> None: + def __init__(self, project_root: Path, **_kwargs: Any) -> None: self.project_root = project_root self.closed = False @@ -305,7 +310,7 @@ def test_analyze_file_reports_call_resolver_stats( monkeypatch: pytest.MonkeyPatch, ) -> None: class FakePyrightSession: - def __init__(self, project_root: Path) -> None: + def __init__(self, project_root: Path, **_kwargs: Any) -> None: self.project_root = project_root def resolve_calls( @@ -400,7 +405,7 @@ def test_analyze_file_restarts_pyright_after_file_budget( sessions: list[Any] = [] class FakePyrightSession: - def __init__(self, project_root: Path) -> None: + def __init__(self, project_root: Path, **_kwargs: Any) -> None: self.project_root = project_root self.closed = False sessions.append(self) @@ -462,3 +467,120 @@ def close(self) -> None: assert response == {"jsonrpc": "2.0", "id": 1, "result": {}} assert fake.closed is True assert state.pyright is None + + +def test_restart_budget_survives_session_recycle( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Run-wide restart cap is not reset at the session-recycle boundary. + + With MAX_FILES_PER_PYRIGHT_SESSION=25 and 30 files, there are two recycles + (files 1-25 in session 1, files 26-30 in session 2). A crashing pyright + must exhaust the single run-wide 3-restart budget, not a fresh 3-restart + budget per session. + """ + + # Each fake session increments run_state.restart_count directly, simulating + # a pyright that crashes on every call. Using run_state rather than a + # session-local counter is what the fix is supposed to enforce. + class CrashingFakePyrightSession: + def __init__(self, project_root: Path, **kwargs: Any) -> None: + self.project_root = project_root + self.run_state: PyrightRunState = kwargs.get("run_state") or PyrightRunState() + self.closed = False + + def resolve_calls( + self, + file_path: str, + function_ids: list[str], + ) -> CallResolutionResult: + _ = (file_path, function_ids) + if not self.run_state.disabled: + # Simulate a crash: record a restart finding via run_state. + self.run_state.restart_count += 1 + if self.run_state.restart_count > 3: + self.run_state.disabled = True + return CallResolutionResult( + findings=[ + { + "subcode": FINDING_PYRIGHT_RESTART, + "severity": "warning", + "message": "pyright restart cap exceeded", + "metadata": {}, + } + ], + ) + return CallResolutionResult( + findings=[ + { + "subcode": FINDING_PYRIGHT_RESTART, + "severity": "warning", + "message": "pyright subprocess died and was restarted", + "metadata": {}, + } + ], + ) + return CallResolutionResult() + + def resolve_references( + self, + file_path: str, + sites: Sequence[ReferenceSite], + ) -> ReferenceResolutionResult: + _ = (file_path, sites) + return ReferenceResolutionResult() + + def close(self) -> None: + self.closed = True + + monkeypatch.setattr(server_module, "PyrightSession", CrashingFakePyrightSession, raising=False) + demo = tmp_path / "demo.py" + demo.write_text("def hello():\n pass\n", encoding="utf-8") + state = server_module.ServerState(initialized=True, project_root=tmp_path) + + # Drive 30 analyze_file requests. The recycle boundary falls at file 25. + for _ in range(30): + server_module.handle_analyze_file({"file_path": str(demo)}, state) + + # The run-wide budget must be consumed exactly once across both recycles, + # not reset to 3 at the recycle boundary. + assert state.pyright_run_state.restart_count <= 4 # 3 restarts + 1 cap trip + assert state.pyright_run_state.disabled is True + + +def test_disabled_pyright_unavailable_does_not_redrive_per_session( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Once pyright is disabled (binary missing), recycles must not re-check the binary. + + Driving 30 files across the 25-file recycle boundary should emit exactly + one FINDING_PYRIGHT_UNAVAILABLE finding, not one per recycle. + """ + resolve_executable_call_count = 0 + + def counting_resolve_executable(_self: PyrightSession) -> None: + nonlocal resolve_executable_call_count + resolve_executable_call_count += 1 + # Returning None simulates a missing binary. + + monkeypatch.setattr( + PyrightSession, + "_resolve_executable", + counting_resolve_executable, + raising=False, + ) + + demo = tmp_path / "demo.py" + demo.write_text("def hello():\n pass\n", encoding="utf-8") + state = server_module.ServerState(initialized=True, project_root=tmp_path) + + # Drive 30 analyze_file requests across the 25-file recycle boundary. + for _ in range(30): + server_module.handle_analyze_file({"file_path": str(demo)}, state) + + # _resolve_executable must be called exactly once: the first time pyright + # is needed. The shared run_state.disabled=True short-circuits _ensure_process + # before _start_process (and thus _resolve_executable) is re-entered. + assert resolve_executable_call_count == 1 From 0eb748d507574777da7bdd96529950f98e3d77c0 Mon Sep 17 00:00:00 2001 From: John Morrissey <544926+tachyon-beep@users.noreply.github.com> Date: Fri, 29 May 2026 04:16:58 +1000 Subject: [PATCH 9/9] docs(v1.0): archive 2026-05-24 gap-analysis + publish-checklist reviews Supporting (non-normative) review artifacts under the canonical reviews/ location per CLAUDE.md precedence note. Not cited as decision sources. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../1.0/reviews/gap-analysis-2026-05-24.md | 348 ++++++++++++++++++ .../v1.0-publish-checklist-2026-05-24.md | 151 ++++++++ 2 files changed, 499 insertions(+) create mode 100644 docs/clarion/1.0/reviews/gap-analysis-2026-05-24.md create mode 100644 docs/clarion/1.0/reviews/v1.0-publish-checklist-2026-05-24.md diff --git a/docs/clarion/1.0/reviews/gap-analysis-2026-05-24.md b/docs/clarion/1.0/reviews/gap-analysis-2026-05-24.md new file mode 100644 index 00000000..e7f74abe --- /dev/null +++ b/docs/clarion/1.0/reviews/gap-analysis-2026-05-24.md @@ -0,0 +1,348 @@ +# Clarion v1.0 — Requirements vs Implementation Gap Analysis + +**Date:** 2026-05-24 (live north-star — amendments applied as we work) +**Scope:** 125 requirement IDs from `docs/clarion/1.0/requirements.md` audited against shipped code in `crates/`, `plugins/python/`, `tests/`, and `.github/workflows/`. +**Inventory:** 51 REQ-* · 28 NFR-* · 8 CON-* · 25 NG-* · 13 cross-cutting (REQ-INTEG-*). +**Status:** Supporting context only — not normative. See [`docs/clarion/1.0/README.md`](../README.md) for canonical-truth precedence. + +This document is the working north-star for v1.0 publish-ready cleanup. It evolves as amendments land. It tells you, today, where shipped code agrees with the spec, where it diverges by intentional deferral, where the spec drifted away from shipped behaviour, and where there is genuine uncovered scope to backfill before publish. + +## 0. Amendments applied this session + +### Session 1 — MCP/HTTP surface narrowing (2026-05-24 morning) + +| # | Action | Effect | +|---|--------|--------| +| 1 | Added "v1.0 ships 8-tool MVP subset" blockquote to `REQ-MCP-02`; full catalogue documented as v0.2 target. | `REQ-MCP-02` **P0 → Deferred**. | +| 2 | Added "Deferred to v0.2" blockquote to `REQ-MCP-03` citing amendment §3+§4 + ADR-030. | `REQ-MCP-03` **P0 → Deferred**. | +| 3 | Added "v1.0 ships ADR-014 file-registry subset" blockquote to `REQ-HTTP-01`. | `REQ-HTTP-01` **P0 → Deferred**. | +| 4 | Added "Deferred to v0.2" blockquote to `REQ-HTTP-02`. | `REQ-HTTP-02` **P0 → Deferred**. | +| 5 | "v1.0 ships subset" notes at `system-design.md` §8 Tool catalogue / §8 Exploration-elimination / §9 HTTP Endpoints. | Closes design-doc half of surface drift. | + +### Session 2 — broader amendment bundle (2026-05-24 afternoon) + +| # | Action | Effect | +|---|--------|--------| +| 6 | "Deferred to v0.2" blockquote on `REQ-MCP-01` (cursor session model). | **P1 Missing → Deferred** | +| 7 | "v1.0 ships intrinsic bounds" blockquote on `REQ-MCP-04` (per-tool token budgets are v0.2). | **P2 Partial → Satisfied** | +| 8 | "Deferred to v0.2" blockquote on `REQ-MCP-05` (write-effect tool consent gates). | **P1 Missing → Deferred** | +| 9 | "Deferred to v0.2" blockquote on `REQ-MCP-06` (session persistence). | **P1 Missing → Deferred** | +| 10 | "Deferred to v0.2" blockquote on `REQ-ARTEFACT-01` (B.4 removed). | already Deferred; now blockquoted | +| 11 | "Deferred to v0.2" blockquote on `REQ-ARTEFACT-02` (B.5 removed). | already Deferred; now blockquoted | +| 12 | "Deferred to v0.2" blockquote on `REQ-CONFIG-02/03/04` (WP6 narrowed). | already Deferred; now blockquoted | +| 13 | Section-level "whole subsystem deferred" note on `REQ-GUIDANCE-*` (WP7 deferred). | -01 **Partial → Deferred**; -02..-06 **Missing → Deferred** | +| 14 | Per-row "Deferred to v0.2" blockquotes on `REQ-BRIEFING-01/02/04/05/06` (WP6 narrowed per ADR-030). | -01 **Partial → Deferred**; -02/04/05/06 **Missing → Deferred** | +| 15 | "Deferred to v0.2" blockquote on `NFR-PERF-01` (60-min target needs Phases 4–6). | **P1 Missing → Deferred** | +| 16 | "Deferred to v0.2" blockquote on `NFR-COST-01` ($15 elspeth needs Phases 4–6). | **P1 Missing → Deferred** | +| 17 | "Deferred to v0.2" blockquote on `NFR-COST-03` (preflight needs Phase 0). | **P1 Missing → Deferred** | +| 18 | `system-design.md` UX-modes table: catalog-artefacts row flipped from "Shipped" to "v0.2 (deferred)". | Closes the stale `system-design.md:103` claim. | + +### Session 3 — checklist phase 1 (2026-05-24 evening) + +| # | Action | Effect | +|---|--------|--------| +| 19 | "Deferred to v0.2" blockquote on `REQ-INTEG-WARDLINE-02` (WP9-B manifest + overlay ingest). | **Missing → Deferred** | +| 20 | "Deferred to v0.2" blockquote on `REQ-INTEG-WARDLINE-03` (WP9-B fingerprint ingest). | **Missing → Deferred** | +| 21 | "Deferred to v0.2" blockquote on `REQ-INTEG-WARDLINE-04` (WP9-B exceptions ingest). | **Missing → Deferred** | +| 22 | "Deferred to v0.2" blockquote on `REQ-INTEG-WARDLINE-05` (WP10 SARIF baseline ingest for translator). | **Missing → Deferred** | +| 23 | "Deferred to v0.2" blockquote on `REQ-INTEG-WARDLINE-06` (three-scheme identity oracle, depends on -03/-04). | **Missing → Deferred** | +| 24 | `CHANGELOG.md` known-limitations: appended Wardline state-file ingest line after the REGISTRY-import asterisk; enumerates -02..-06 and names WP9-B + WP10 as v0.2 landing surfaces. | Closes the release-notes drift surface. | +| 25 | `NFR-SEC-03` verification line: replaced stale `serve.rs` line numbers (1457/1495/1547/1579/1614) with eight test-name citations covering opt-in refusal, no-auth refusal, env-missing refusal, HMAC + wrong-secret, bearer + wrong-token + batch variant, and `_capabilities` carve-out. | Verification citations are now name-stable, not line-fragile. | + +**Net effect after session 2 + checklist phase 1:** REQ Missing list now 4 (REQ-ANALYZE-03, REQ-CATALOG-07, REQ-PLUGIN-05, REQ-PLUGIN-06); the 5 REQ-INTEG-WARDLINE rows flipped Missing → Deferred when checklist 1.1–1.5 added the row-level blockquotes (2026-05-24 sweep). NFR Missing stays at 6 (NFR-OBSERV-01..04, NFR-COMPAT-01, NFR-COMPAT-02; plus NFR-OPS gaps). The list of genuine v1.0-publish-blocking code work is now small and actionable — §5 below. + +--- + +## 1. Executive summary + +Clarion v1.0 is a credible release for its **stated minimum viable surface** — entity ingestion, plugin host, secret scanning, Phase-3 clustering, persistent storage, HMAC-authenticated HTTP read API for Filigree federation, and a small MCP tool set. The initial audit read it as materially narrower than `requirements.md` because the requirement doc had not been updated to reflect the 2026-05-16 Sprint-2 scope amendment. As of §0 amendments 1–4, the MCP and HTTP requirement rows now correctly document the v1.0 subset. + +After session 1's amendments, three patterns describe the remaining work: + +1. **Doc-drift cluster (text-only fixes, not code).** The Sprint-2 amendment deferred WP6 (LLM pipeline), WP7 (guidance authoring), WP9-B (Filigree finding emission), WP10 (SARIF translator), narrowed WP8 (MCP) to 8 tools, and removed the catalog-artefact boxes B.4/B.5. `REQ-FINDING-03..06` and `REQ-INTEG-FILIGREE-01..05` carry the "Deferred to v0.2" blockquote correctly; `REQ-MCP-02/03` and `REQ-HTTP-01/02` now carry it. **Still missing the blockquote:** `REQ-ARTEFACT-01/02`, `REQ-CONFIG-02/03/04`, `REQ-GUIDANCE-01..06`, `REQ-BRIEFING-01/02/04/05/06`, `REQ-MCP-04/05/06`. These all describe surface that the amendment deferred but whose requirement rows still read as v1.0 contract — the obvious next bundle (see §5 action #1b). +2. **Whole pipeline phases absent (architectural deferral).** `clarion analyze` ends at Phase 3. Phase 0 (dry-run) and Phases 4–6 (LLM summarisation) drive `NFR-PERF-01`, `NFR-COST-01/03`, `REQ-BRIEFING-04/05`, the budget gate, and the preflight estimator. Summarisation happens lazily through MCP `summary` only per ADR-030. Once #1b lands, all of these reclassify Deferred — the work itself is genuinely v0.2. +3. **Targeted code gaps outside any deferral.** A small set of v1.0-contract items that nobody amended out: `REQ-ANALYZE-03` (`--resume`), `REQ-CATALOG-04/07` (file git metadata, HEAD-SHA capture), `REQ-PLUGIN-05/06` (Python plugin import policy + decorator edges), `NFR-COMPAT-01` (Filigree schema-pin CI), `NFR-COMPAT-02` (Wardline probe pins wrong symbol), `NFR-OBSERV-01..04` (JSON logs / stats.json file / Prometheus / compat-report finding). These are the *actual* code work to publish v1.0 — once the amendments collapse the doc drift, this list is the punch list. + +**Disciplined areas (genuinely strong, no action):** plugin host (manifest enforcement, framing, jail, crash-loop breaker, structured host findings), secret scanner, Phase-3 Leiden clustering with seeded determinism, summary-cache 5-tuple key, HMAC + ADR-034 hardening surface, all 8 CON-* honoured (one with the minor `CodexCli` provider drift noted in §5), all 25 NG-* honoured (no scope creep into linter / file-watcher / multi-branch / SARIF / wiki territory). + +**Disciplined areas (genuinely strong):** plugin host (manifest enforcement, framing, jail, crash-loop breaker, structured host findings), secret scanner, Phase-3 Leiden clustering with seeded determinism, summary-cache 5-tuple key, HMAC + ADR-034 hardening surface, all 8 CON-* honoured (one with the minor `CodexCli` provider drift noted in §5), all 25 NG-* honoured (no scope creep into linter / file-watcher / multi-branch / SARIF / wiki territory). + +**Verdict tally** (after session 2 amendments): + +| Category | Total | Satisfied | Partial | Missing | Deferred | Δ from initial audit | +|----------|------:|----------:|--------:|--------:|---------:|---------------------| +| REQ-* (functional, non-INTEG) | 40 | 13 | 9 | 4 | 14 | Sat +1, Partial −4, Missing −4, Deferred +7 | +| REQ-INTEG-* (cross-product) | 11 | 1 | 0 | 0 | 10 | Missing −5, Deferred +5 (checklist 1.1–1.5 sweep) | +| NFR-* | 28 | 5 | 12 | 6 | 5 | Missing −3, Deferred +3 | +| CON-* | 8 | 6 | 1 | 0 | 1 | unchanged | +| NG-* (inverted: scope creep) | 25 | 25 honoured | 0 | 0 drift | — | unchanged | +| **TOTAL** | **112** | **50** | **22** | **10** | **30** | | + +(Counts in §6 RTM rows are authoritative; the headline table sums them.) + +**Working plan:** §5 action #1c is done (checklist phase 1, 2026-05-24). The doc-drift workstream is complete; what's left is genuine code — §5 actions 2–7, tracked in [`v1.0-publish-checklist-2026-05-24.md`](./v1.0-publish-checklist-2026-05-24.md) phases 2–6. Once those empty out, the docs and the binary tell the same story and v1.0 is publish-ready. + +--- + +## 2. Methodology + +Each requirement ID was traced via three sources, in this order: + +1. **Requirement text** — `docs/clarion/1.0/requirements.md`, including the per-requirement `Verification:` line. +2. **Design trace bridge** — `**Addresses**:` headers in `docs/clarion/1.0/system-design.md` (twelve sections covering §2–§11; bridge density is good — every functional requirement in this audit had a navigable target). +3. **Implementation** — corresponding code under `crates/clarion-core/`, `crates/clarion-storage/`, `crates/clarion-cli/`, `crates/clarion-scanner/`, `crates/clarion-mcp/`, `plugins/python/`, plus `.github/workflows/` and `tests/`. + +Each verdict was cross-checked against in-flight tracked work in Filigree (`get-ready` + `list-issues --status in_progress`, snapshot at `/tmp/clarion_gap_filigree_ready.json`) to avoid re-reporting known follow-ups as discoveries. Verdicts cite specific `file:line` evidence. Non-goals (NG-*) were checked **inverted** — confirming the codebase does *not* implement them — to detect scope creep. + +Verdict vocabulary: + +- **Satisfied** — code implements the requirement; verification test or behavioural artefact exists. +- **Partial** — mechanism exists but is undermeasured, contract-divergent, or covers only part of the named surface. +- **Missing** — no implementation; not deferred by any written carve-out. +- **Deferred** — scope explicitly carved out of v1.0 by the Sprint-2 amendment, a Backlog ADR, or a `> **Deferred to v0.2**` blockquote in `requirements.md`. + +`Tracked` column values: `new` (file an issue), `tracked:` (existing Filigree row covers this; merge into that work), `deferred:scope-amendment` (don't file). + +Audit was performed by six concurrent subagents (REQ slices A/B/C; NFR slices D/E/F) plus a seventh CON+NG audit (G), capped at three in flight, synthesised here. + +--- + +## 3. Headline findings, by priority + +### 3.1 P0 — Resolved by session 1 amendments + +The initial audit flagged four P0 surface-mismatch findings: `REQ-MCP-02`, `REQ-MCP-03`, `REQ-HTTP-01`, `REQ-HTTP-02`. Investigation of the [Sprint-2 scope amendment](../../implementation/sprint-2/scope-amendment-2026-05.md) showed that all four were explicitly narrowed by the 2026-05-16 amendment (Box B.6 named the 8-tool MVP MCP surface; [ADR-014](../adr/ADR-014-filigree-registry-backend.md) defined the file-registry HTTP subset) but the requirement rows had never received the "Deferred to v0.2" blockquote that `REQ-FINDING-03..06` carries. **Session 1 added those blockquotes (see §0).** No code changes required for these four rows. The P0 list is now empty. + +The next-bundle of probably-same-shape rows is documented as §5 action #1b — they should be inspected against the amendment and either amended or escalated to genuine work, individually. + +### 3.2 P1 — Genuine v1.0 code gaps (post amendment sweep) + +After session 2, the P1 list is much shorter and entirely "real code work" — no remaining doc-drift hidden as severity. + +**Observability foundations (NFR-OBSERV-01..04).** Not deferred by any envelope. `tracing` initialised in plain text at `crates/clarion-cli/src/main.rs:69-76` with no `.json()`, no file sink, no rotation. The install template's `.gitignore` excludes `runs/*/log.jsonl` — a path nothing writes to. Stats live only in the SQLite `runs.stats` column; no `runs//stats.json` file lands. No Prometheus `/api/v1/metrics` endpoint *(this one will reclassify Deferred once REQ-HTTP-01 is propagated through NFR-OBSERV-03; it's part of the broader HTTP surface)*. No `CLA-INFRA-SUITE-COMPAT-REPORT` emitter. Three of four are direct v1.0 code work; one (`-03`) probably joins HTTP-01's deferral. + +**Schema/version compat (NFR-COMPAT-01/02).** Genuinely outside any deferral. NFR-COMPAT-01 = Filigree schema-pin CI test missing (one job). NFR-COMPAT-02 = Wardline probe pins `wardline.__version__` instead of `REGISTRY_VERSION`; named graceful-degradation findings have no emitter. + +**Python plugin gaps (REQ-PLUGIN-05/06).** TYPE_CHECKING block exclusion / `src.`-prefix canonicalisation / `python:unresolved:*` placeholders / `alias_of` edges / `decorated_by` / `inherits_from` / `uses_type` edges all absent. Not in any defer list — these are the v1.0 contract for the Python plugin's ontology coverage. + +**Three targeted REQ-* rows outside any envelope.** `REQ-ANALYZE-03` (`--resume` CLI flag + checkpoint reader), `REQ-CATALOG-07` (HEAD-SHA capture into `first_seen_commit`/`last_seen_commit` — columns exist, populated as `None`), `REQ-CATALOG-04` (file-entity git metadata). + +**Ops gaps (NFR-OPS-03/04).** `clarion db export --textual` + `db merge-helper` don't exist (NFR-OPS-03 — either build them or narrow the requirement); PyPI publish + SLSA-over-sdist outstanding (NFR-OPS-04, tracked under `clarion-f530101222`). + +### 3.3 P1 — Targeted gaps outside any deferral envelope + +- **REQ-ANALYZE-03 — `--resume`** has no CLI flag (`cli.rs:30`), no checkpoint reader, no `checkpoints.jsonl` consumer. Crash mid-run leaves `runs.status='running'` rows; rerun starts from scratch. +- **REQ-CATALOG-07 — `first_seen_commit` / `last_seen_commit`** columns exist in `commands.rs:66` but every call-site in `analyze.rs` writes `None` (lines 813, 1619, 1692, 2156, 2348, 2417). No HEAD-SHA capture. Point-in-time queries return NULL. +- **REQ-CATALOG-04 — file-entity git metadata** writes only `language` + `briefing_blocked`. `git_churn_count`, `git_last_modified`, `git_authors`, `size_bytes`, `line_count`, `mime_type` never populated (`analyze.rs:1544`). + +### 3.4 P2 — Implemented-but-unmeasured (no benchmarks, no recorded runs) + +`find tests/ -name 'bench*'` returns nothing; no `criterion` dependency anywhere. The following all have correct-shape mechanisms in code and lack only a measurement artefact: + +- NFR-PERF-02 (≤100 ms initialize, ≤50 ms p95 hot-cache summary): mechanisms correct (`lib.rs:176,256,788,1339-1367`), no harness. +- NFR-SCALE-01 (elspeth ±20% entity count, no OOM): `EntityCountCap=500k` wired (`limits.rs:115`), no recorded elspeth run. +- NFR-SCALE-02 (`.clarion/clarion.db` ≤2 GB): no size reporter, no measured run. +- NFR-SCALE-03 (16-reader pool, no exhaustion): wired (`serve.rs:46`), saturation tested for max_size=1/2 only; no combined-load test. +- NFR-COST-02 (≥95% summary-cache hit rate after 3 runs): mechanism correct (`cache.rs:48-110`), no `cache_hit_rate` aggregation in `stats.json`. + +Each could flip to Satisfied with a measurement task that does not require new product code. + +### 3.5 Documentation drift (text-only fixes) + +Status legend: ✅ done this session · ⏳ still open. + +- ✅ `system-design.md` §8 Tool catalogue + §8 Exploration-elimination + §9 HTTP endpoints — each now carries a "v1.0 ships subset" note. +- ✅ `REQ-MCP-02 / REQ-MCP-03 / REQ-HTTP-01 / REQ-HTTP-02` — deferral blockquotes added. +- ⏳ `requirements.md` still lacks "Deferred to v0.2" blockquotes on `REQ-ARTEFACT-01/02`, `REQ-CONFIG-02/03/04`, `REQ-GUIDANCE-01..06`, and probably `REQ-MCP-04/05/06` + `REQ-BRIEFING-04/05` — see §5 action #1b for the bundle. +- ⏳ `system-design.md:103` still calls catalog artefacts "Shipped" though `grep -r catalog.json crates/` is empty; pair this fix with `REQ-ARTEFACT-*` amendments. +- ⏳ `NFR-SEC-03` verification cites stale `serve.rs:798-803`; actual locations are `tests/serve.rs:1160,1194,1251,1285,1317`. Tracked under the ADR-034 refresh chain (clarion-7913f950d7, clarion-272b5bc1ec). Recommend replacing line citations with test-name citations to survive future refactors. +- ⏳ `CHANGELOG.md` known-limitations enumerates the Wardline REGISTRY import asterisk but not Wardline state-file ingest (manifest/overlay/fingerprint/exceptions/SARIF baseline). Add to release notes. + +### 3.6 Quick-close wins (issues already done in HEAD) + +Per the standing "close already-done tickets" authorisation in user memory: + +- **`clarion-a4fb59a96a` — rollback runbook** — `docs/operator/v1.0-release-rollback.md` exists (141 lines). +- **`clarion-42f4fee904` — loopback trust banner** — emitted at `crates/clarion-cli/src/http_read.rs:244-248`; documented at `docs/operator/clarion-http-read-api.md:58-73`. + +Both can transition through the full workflow to `closed`. + +### 3.7 One CON drift (governance, not severity) + +`CON-ANTHROPIC-01` says "Anthropic-only LLM provider in v0.1." `crates/clarion-mcp/src/config.rs:95-99` enumerates `OpenRouter`, `CodexCli`, `ClaudeCli`. `CodexCli` (`crates/clarion-cli/src/serve.rs:11`) shells out to OpenAI's Codex CLI — a non-Anthropic vendor surface. The constraint's prompt-caching rationale arguably doesn't apply to CLI shell-outs, which is probably why the drift went unflagged. Recommend either (a) amend the constraint to recognise local-CLI providers as a separate category, or (b) gate `CodexCli` behind a default-off cargo feature. + +--- + +## 4. Strongest areas (no action required) + +- **Plugin host (REQ-PLUGIN-01..04, REQ-FINDING-01/02, NFR-SEC-01/05).** Content-Length framing (`transport.rs:113,287`), manifest enforcement (`host.rs:897,1043`), undeclared-kind drop tests (`host.rs:1586,2679`), crash-loop breaker (`breaker.rs:16`), entity cap and OOM kill (`limits.rs:115`), path-escape jail. Findings vocabulary (`Defect | Fact | Classification | Metric | Suggestion`) round-trips through a CHECK-constrained `findings` table; per-plugin `rule_id_prefix` works as designed. `.clarion/.gitignore` shipped by `install.rs:97`. +- **Phase-3 Leiden clustering (REQ-CATALOG-05).** Seeded determinism asserted by `tests/analyze.rs:730 analyze_phase3_is_deterministic_across_two_runs`. Subsystem entities and `in_subsystem` edges flow through the writer-actor; e2e at `tests/e2e/phase3_subsystems.sh`. +- **Summary-cache 5-tuple key (REQ-BRIEFING-03).** PK = `entity_id + content_hash + prompt_template_id + model_tier + guidance_fingerprint` exactly per ADR-007 (`cache.rs:9-72`; `migrations/0001_initial_schema.sql:151-164`). Round-trip tested. +- **HMAC + ADR-034 federation surface (REQ-HTTP-03, NFR-SEC-03).** Six tests at `tests/serve.rs:1160..1342` cover loopback default, non-loopback-without-auth refusal, HMAC-required path, legacy bearer path, identity-env missing refusal. Code in `http_read.rs:179-251,389-498`. +- **Secret scanner (NFR-SEC-01).** `crates/clarion-scanner/` + `crates/clarion-cli/src/secret_scan.rs` (574 LOC) with baseline justification + dedicated e2e at `tests/e2e/wp5_secret_scan.sh`. +- **All 25 NG-* honoured.** No rule engine, no taint, no SARIF, no file watchers, no multi-branch, no wiki UI, no rename detection, no coverage ingestion, no second plugin, no plugin hash-pinning. The codebase shows real scope discipline. +- **7 of 8 CON-* satisfied** (`CON-ANTHROPIC-01` partial drift noted in §3.7; CON-FILIGREE-01 properly Deferred). + +--- + +## 5. Recommended next actions + +Sorted by leverage. Status legend: ✅ done · ⏳ pending. Each item names a target and a "why." + +| # | Status | Action | Target | Why | +|---|--------|--------|--------|-----| +| 1a | ✅ | Amendment pass on REQ-MCP-02/03 + REQ-HTTP-01/02 + system-design §8/§9 notes. | done 2026-05-24 | Closed surface-vs-doc P0 gap. | +| 1b | ✅ | Amendment bundle on `REQ-MCP-01/04/05/06`, `REQ-ARTEFACT-01/02`, `REQ-CONFIG-02/03/04`, `REQ-GUIDANCE-01..06` (section header), `REQ-BRIEFING-01/02/04/05/06`, `NFR-PERF-01`, `NFR-COST-01/03` + retract `system-design.md:103` "Shipped" claim. | done 2026-05-24 | 17 rows; Missing list dropped from 19 → 4. | +| 1c | ✅ | **Final amendment sweep.** `REQ-INTEG-WARDLINE-02..06` row-level blockquotes added; `CHANGELOG.md` known-limitations now enumerates Wardline state-file ingest alongside the REGISTRY-import asterisk; `NFR-SEC-03` verification swapped from stale line numbers to test-name citations. Done 2026-05-24 (checklist phase 1). | done | Closed the last amendment-deferred ambiguity; doc-drift workstream complete. | +| 2 | ⏳ | Backfill the three uncovered REQ rows: `REQ-ANALYZE-03` (`--resume`), `REQ-CATALOG-07` (HEAD-SHA capture into `first_seen_commit` / `last_seen_commit`), `REQ-CATALOG-04` (git metadata on file entities). | v1.0 publish-ready | None deferred by any carve-out; storage substrate already shaped. | +| 3 | ⏳ | Python plugin: implement `REQ-PLUGIN-05` (TYPE_CHECKING exclusion, src-prefix canonicalisation, `python:unresolved:*` placeholders, `alias_of` edges) + `REQ-PLUGIN-06` (`decorated_by`/`inherits_from`/`uses_type` edges). | v1.0 publish-ready | Not in any defer list; core ontology surface advertised at v1.0. | +| 4 | ⏳ | Wardline probe — pin `REGISTRY_VERSION` (not `__version__`), emit `CLA-INFRA-WARDLINE-REGISTRY-ADDITIVE-SKEW` and `-MIRRORED` findings on version drift. | v1.0 publish-ready | Cheap fix, named in detailed-design.md:1169-1170, currently invisible. | +| 5 | ⏳ | Add Filigree schema-pin CI job (`NFR-COMPAT-01`). | v1.0 publish-ready | One job; closes a P1 with low effort. | +| 6 | ⏳ | Observability foundations: JSON-formatted tracing layer with file sink + rotation; `runs//stats.json` file emission alongside the SQLite column; `CLA-INFRA-SUITE-COMPAT-REPORT` emitter. `NFR-OBSERV-03` (Prometheus `/api/v1/metrics`) probably becomes Deferred — depends on broader HTTP surface already deferred via REQ-HTTP-01. | v1.0 publish-ready | Three NFR-OBSERV-* rows; one cohesive workstream. | +| 7 | ⏳ | Add `clarion db export --textual` + `clarion db merge-helper` subcommands, or narrow `NFR-OPS-03` to reflect the commit-by-default-only scope. | v1.0 publish-ready | Currently the text overpromises. | +| 8 | ⏳ | Publish `clarion-plugin-python` to PyPI and extend SLSA provenance to the sdist (existing ticket `clarion-f530101222`). | v1.0 publish-ready | Closes `pipx install clarion-plugin-python` per `NFR-OPS-04`. | +| 9 | ⏳ | Decide on `CON-ANTHROPIC-01` × `CodexCliProvider`: either amend constraint or gate behind feature flag. | ADR or constraint amendment | Governance, not severity, but currently invisible drift. | +| 10 | ⏳ | Close already-done Filigree issues per standing authorisation: `clarion-a4fb59a96a` (rollback runbook), `clarion-42f4fee904` (loopback banner). | Immediate | Both ship in HEAD; tickets are stale. | +| 11 | ⏳ | Backlog: elspeth-scale measurement task — cargo bench harness for MCP latency, single recorded elspeth run capturing entity count + DB size + cache hit rate + stats.json. | post-publish, before 1.1 | Flips five P2 "implemented but unmeasured" verdicts to Satisfied without product code. | + +--- + +## 6. Full RTM + +### 6.1 REQ-* (functional) + +| ID | Verdict | Evidence | Gap | Sev | Tracked | +|----|---------|----------|-----|-----|---------| +| REQ-ANALYZE-01 | Partial | `analyze.rs:54,485` | Phases 4–8 absent; no phase log | P2 | deferred:scope-amendment | +| REQ-ANALYZE-02 | Partial | `analyze.rs:308`; writer serial | No LLM parallelism | — | deferred:scope-amendment | +| REQ-ANALYZE-03 | **Missing** | `cli.rs:30` no flag | No `--resume`; no checkpoint reader | P1 | new | +| REQ-ANALYZE-04 | Deferred | 0 `CLA-FACT-ENTITY-DELETED` emitters | Phase-7 entity-set diff | — | deferred:scope-amendment | +| REQ-ANALYZE-05 | Deferred | 0 `CLA-FACT-TIER-SUBSYSTEM-MIXING` emitters | Phase-7 `CLA-*` rules | — | deferred:scope-amendment | +| REQ-ANALYZE-06 | Partial | `breaker.rs:16`, `host_findings.rs`, `limits.rs` | Named rules `CLA-PY-PARSE-ERROR`/`-TIMEOUT`/`CLA-INFRA-LLM-ERROR`/`-BUDGET-WARNING` absent | P2 | new | +| REQ-ANALYZE-07 | Partial | `tests/analyze.rs:730` | No `clarion db export --textual`; whole-catalog byte-id not verified | P3 | new | +| REQ-ARTEFACT-01 | Deferred | no `catalog.json` emit | Doc drift: requirement lacks blockquote | — | deferred:scope-amendment | +| REQ-ARTEFACT-02 | Deferred | no per-subsystem markdown | Doc drift: requirement lacks blockquote | — | deferred:scope-amendment | +| REQ-BRIEFING-01 | Deferred (✅ amended §0 #14) | `llm_provider.rs:762-789` ships 4-field on-demand summary | rich 9-field `EntityBriefing` v0.2 per ADR-030 | — | deferred:scope-amendment | +| REQ-BRIEFING-02 | Deferred (✅ amended §0 #14) | no impl | controlled vocab v0.2 with briefing pipeline | — | deferred:scope-amendment | +| REQ-BRIEFING-03 | Satisfied | `cache.rs:9-72`; `migrations/0001_initial_schema.sql:151-164` | TTL pruner missing (sub-gap) | — | — | +| REQ-BRIEFING-04 | Deferred (✅ amended §0 #14) | no `knowledge_basis` computation | depends on WP7 guidance + WP9-B triage | — | deferred:scope-amendment | +| REQ-BRIEFING-05 | Deferred (✅ amended §0 #14) | no triage-into-briefing path | depends on WP9-B + WP7 | — | deferred:scope-amendment | +| REQ-BRIEFING-06 | Deferred (✅ amended §0 #14) | no detail levels | per ADR-030 (briefing pipeline) | — | deferred:scope-amendment | +| REQ-CATALOG-01 | Satisfied | `analyze.rs:382,794`; e2e | — | — | — | +| REQ-CATALOG-02 | Satisfied | `host.rs:897`; `plugin.toml:38` | Positive "novel kind round-trip" test implicit | P3 | new | +| REQ-CATALOG-03 | Satisfied | `migrations/0001_initial_schema.sql:81`; `host.rs:1043` | `guides`/`emits_finding` reserved; no producer | — | deferred:scope-amendment | +| REQ-CATALOG-04 | Partial | `analyze.rs:1544` | git_churn/last_modified/authors/size/lines/mime never populated | P1 | new | +| REQ-CATALOG-05 | Satisfied | `analyze.rs:657`; `clustering.rs`; `tests/analyze.rs:730` | — | — | — | +| REQ-CATALOG-06 | Satisfied | `entity_id.rs`; `qualname.py` | Move-without-rename test missing | P3 | new | +| REQ-CATALOG-07 | **Missing** | `commands.rs:66`; call-sites write `None` | No HEAD-SHA capture | P1 | new | +| REQ-CONFIG-01 | Partial | `config.rs:16` | No `~/.config/clarion/defaults.yaml` merge; no `version:` field | P2 | new | +| REQ-CONFIG-02 | Deferred | no `profile`/`budget`/`default`/`deep` enum | WP6 deferral | — | deferred:scope-amendment | +| REQ-CONFIG-03 | Deferred | no dry-run estimator | WP6 + WP11 | — | deferred:scope-amendment | +| REQ-CONFIG-04 | Deferred | no `LlmPolicyConfig` | WP6 | — | deferred:scope-amendment | +| REQ-CONFIG-05 | Partial | `secret_scan.rs:574`; CLI flag | No `analysis.include`/`exclude` globs in config schema | P2 | new | +| REQ-FINDING-01 | Satisfied | `commands.rs:113-133`; `migrations/0001_initial_schema.sql:104-135` | — | — | — | +| REQ-FINDING-02 | Satisfied | `manifest.rs`; `limits.rs:37-54`; `pyright_session.py:34-41` | — | — | — | +| REQ-FINDING-03 | Deferred | scope-amendment §3-4 | WP9-B v1.1 | — | deferred:scope-amendment | +| REQ-FINDING-04 | Deferred | requirements.md:332 | WP10 SARIF translator | — | deferred:scope-amendment | +| REQ-FINDING-05 | Deferred | requirements.md:342 | scan_run_id mapping | — | deferred:scope-amendment | +| REQ-FINDING-06 | Deferred | requirements.md:352 | mark_unseen dedup | — | deferred:scope-amendment | +| REQ-GUIDANCE-01 | Deferred (✅ amended §0 #13) | view substrate exists; no behaviour | WP7 deferred whole | — | deferred:scope-amendment | +| REQ-GUIDANCE-02 | Deferred (✅ amended §0 #13) | `lib.rs:38 EMPTY_GUIDANCE_FINGERPRINT` placeholder | composition algo v0.2 | — | deferred:scope-amendment | +| REQ-GUIDANCE-03 | Deferred (✅ amended §0 #13) | no CLI / no `propose_guidance` MCP | WP7 authoring surface v0.2 | — | deferred:scope-amendment | +| REQ-GUIDANCE-04 | Deferred (✅ amended §0 #13) | no wardline_derived emitter | WP7 deferred | — | deferred:scope-amendment | +| REQ-GUIDANCE-05 | Deferred (✅ amended §0 #13) | no churn-vs-guidance code | WP7 deferred | — | deferred:scope-amendment | +| REQ-GUIDANCE-06 | Deferred (✅ amended §0 #13) | no `guidance export/import` CLI | WP7 deferred | — | deferred:scope-amendment | +| REQ-HTTP-01 | Deferred (✅ amended §0 #3) | `http_read.rs:364-372` ships ADR-014 subset | broader catalogue v0.2 per amendment §4 | — | deferred:scope-amendment | +| REQ-HTTP-02 | Deferred (✅ amended §0 #4) | `:resolve` handles file_path scheme only | multi-scheme oracle v0.2 (depends on WP9-B Wardline ingest) | — | deferred:scope-amendment | +| REQ-HTTP-03 | Satisfied | `http_read.rs:179-251,389-498`; `tests/serve.rs:1160..1342` | — | — | tracked:adr-034-refresh | +| REQ-HTTP-04 | Partial | `http_read.rs:735-816` | Uses `ETag` per-file vs spec'd `X-Clarion-State` run-level | P2 | new | +| REQ-MCP-01 | Deferred (✅ amended §0 #6) | `lib.rs:187-211` no cursor/breadcrumb state | cursor session model v0.2 (B.6 narrowed surface) | — | deferred:scope-amendment | +| REQ-MCP-02 | Deferred (✅ amended §0 #1) | `lib.rs:52-129` ships 8-tool MVP subset per amendment B.6 | broader catalogue v0.2 | — | deferred:scope-amendment | +| REQ-MCP-03 | Deferred (✅ amended §0 #2) | no `find_entry_points` etc.; depends on Phase-7 pre-compute | shortcuts v0.2 per ADR-030 + amendment §4 | — | deferred:scope-amendment | +| REQ-MCP-04 | Satisfied (✅ amended §0 #7) | `http_read.rs` execution_edge_cap; `lib.rs:74,93` bound 8-tool subset intrinsically | per-tool token budgets v0.2 (catalogue-deferred) | — | — | +| REQ-MCP-05 | Deferred (✅ amended §0 #8) | no write-effect tools in 8-tool MVP surface | consent gate v0.2 with catalogue | — | deferred:scope-amendment | +| REQ-MCP-06 | Deferred (✅ amended §0 #9) | `lib.rs:162-185` no session persistence | session model v0.2 with cursor (REQ-MCP-01) | — | deferred:scope-amendment | +| REQ-PLUGIN-01 | Satisfied | `transport.rs:113,287,42,57-63`; `server.py:73,119` | — | — | — | +| REQ-PLUGIN-02 | Satisfied | `manifest.rs:283,43`; `plugin.toml:38`; tests | Tags/prompt_templates absent (WP6) | — | — | +| REQ-PLUGIN-03 | Partial | `host.rs:815`; `server.py:179,238` | `build_prompt` RPC + `file_list` absent (ADR-030 narrowed) | — | deferred:scope-amendment | +| REQ-PLUGIN-04 | Partial | `plugin.toml`; extractor + resolvers | `protocol`/`global` kinds, `decorated_by`/`inherits_from`/`uses_type`/`alias_of` edges absent | P2 | new | +| REQ-PLUGIN-05 | **Missing** | `reference_resolver.py:4` only type-imports TYPE_CHECKING | No exclusion / src-prefix / placeholders / aliases | P1 | new | +| REQ-PLUGIN-06 | **Missing** | no `decorated_by` in `plugin.toml:40`; no emit-site | Decorator detection entirely absent | P1 | new | + +### 6.2 REQ-INTEG-* (cross-product) + +| ID | Verdict | Evidence | Gap | Sev | Tracked | +|----|---------|----------|-----|-----|---------| +| REQ-INTEG-FILIGREE-01 | Deferred | requirements.md:600 | WP9-B | — | deferred:scope-amendment | +| REQ-INTEG-FILIGREE-02 | Deferred | requirements.md:610 | WP9-B observation emission | — | deferred:scope-amendment | +| REQ-INTEG-FILIGREE-03 | Deferred | requirements.md:620 | Registry-backend consumption | — | deferred:scope-amendment | +| REQ-INTEG-FILIGREE-04 | Deferred | requirements.md:630 | `scan_source` ns + schema pin | — | deferred:scope-amendment | +| REQ-INTEG-FILIGREE-05 | Deferred | requirements.md:640 | Capability probe | — | deferred:scope-amendment | +| REQ-INTEG-WARDLINE-01 | Satisfied | `wardline_probe.py:35-56`; `plugin.toml:55-56`; tests | Fail-soft only; no mirror module, no `MIRRORED` finding | P2 | tracked:clarion-88d2ef40b6 | +| REQ-INTEG-WARDLINE-02 | Deferred | no `wardline.yaml` reader | WP9-B Wardline-config ingest | P2 | deferred:scope-amendment (row-level blockquote in `requirements.md:702`) | +| REQ-INTEG-WARDLINE-03 | Deferred | `entities.wardline_json` always `None` | WP9-B | P2 | deferred:scope-amendment (row-level blockquote in `requirements.md:712`) | +| REQ-INTEG-WARDLINE-04 | Deferred | no `wardline.exceptions.json` reader | WP9-B | P2 | deferred:scope-amendment (row-level blockquote in `requirements.md:722`) | +| REQ-INTEG-WARDLINE-05 | Deferred | no `clarion sarif import` | WP10 | P2 | deferred:scope-amendment (row-level blockquote in `requirements.md:732`) | +| REQ-INTEG-WARDLINE-06 | Deferred | no resolve oracle for wardline schemes | Depends on -03/-04 | P2 | deferred:scope-amendment (row-level blockquote in `requirements.md:742`) | + +### 6.3 NFR-* + +| ID | Verdict | Evidence | Gap | Sev | Tracked | +|----|---------|----------|-----|-----|---------| +| NFR-SEC-01 | Satisfied | `clarion-scanner/src/lib.rs`; `secret_scan.rs:212-263`; tests | — | — | — | +| NFR-SEC-02 | Deferred | no ``, no schema validation | ADR-009 Backlog + WP6/WP7 | — | deferred:scope-amendment | +| NFR-SEC-03 | Satisfied | `tests/serve.rs:1160..1317`; HMAC helper :2231 | Verification line citations stale | — | tracked:adr-034-refresh | +| NFR-SEC-04 | Partial | `secret_scan/findings.rs:17`; `secret_scan.rs:36`; `breaker.rs:16` | `CLA-INFRA-TOKEN-STORAGE-DEGRADED`, `CLA-INFRA-BRIEFING-INVALID`, `CLA-SEC-VOCABULARY-CANDIDATE-NOVEL` absent | P2 | deferred:scope-amendment | +| NFR-SEC-05 | Satisfied | `install.rs:97`; `tests/install.rs:40,45` | — | — | — | +| NFR-RELIABILITY-01 | Partial | `pragma.rs:17-30`; writer-actor; `analyze_lock.rs` | No `--resume`; no SIGKILL+reopen test | P2 | partly deferred; tracked:STO-04 (clarion-ee22d1d72c) | +| NFR-RELIABILITY-02 | Missing | no `--no-filigree`/`--no-wardline` flags; no `findings.jsonl` fallback | WP9-B | P2 | deferred:scope-amendment | +| NFR-RELIABILITY-03 | Partial | `host_findings.rs:17-76`; `breaker.rs:16`; `analyze.rs:1355,2279` | LLM rate/budget/schema-invalid emitters absent; no PRAGMA integrity_check in e2e | P2 | partly deferred; tracked:STO-04 | +| NFR-PERF-01 | Deferred (✅ amended §0 #15) | `analyze.rs:482-495` ends at Phase 3 | 60-min target needs Phases 4–6 per ADR-030 | — | deferred:scope-amendment | +| NFR-PERF-02 | Partial | `lib.rs:176,256,788,1339-1367` | No benchmark harness | P2 | tracked:briefing_blocked index (clarion-bdabfd6bca) | +| NFR-PERF-03 | Missing | no 20-turn token-budget assertion | Detail levels absent (BRIEFING-06) | P2 | new | +| NFR-SCALE-01 | Partial | `limits.rs:115-165` `EntityCountCap=500k` | No recorded elspeth run | P2 | new | +| NFR-SCALE-02 | Missing | no DB-size assertion or fixture | — | P3 | new | +| NFR-SCALE-03 | Partial | `reader.rs:46-49`; `serve.rs:46`; `tests/reader_pool.rs` | No combined-load saturation test | P3 | new | +| NFR-COST-01 | Deferred (✅ amended §0 #16) | per-call cost captured; no run-level budget gate | $15 elspeth target needs Phases 4–6 per ADR-030 | — | deferred:scope-amendment | +| NFR-COST-02 | Partial | `cache.rs:48-110`; `lib.rs:788-792` | No `cache_hit_rate` aggregation; no repeat-run measurement | P2 | new | +| NFR-COST-03 | Deferred (✅ amended §0 #17) | no preflight/dry-run code | Phase 0 deferred per ADR-030 | — | deferred:scope-amendment | +| NFR-OPS-01 | Satisfied | `.github/workflows/release.yml:163,180,306,391` | Matrix 3 of 5 targets | P2 | new | +| NFR-OPS-02 | Satisfied | no telemetry import; CHANGELOG | — | — | — | +| NFR-OPS-03 | Partial | `install.rs:78-97` | No `clarion db export --textual` or `db merge-helper` subcommand | P1 | new | +| NFR-OPS-04 | Partial | `pyproject.toml`; `release.yml:241` | Not on PyPI; SLSA covers rust only | P1 | tracked:slsa-python-sdist (clarion-f530101222) | +| NFR-OBSERV-01 | Partial | `main.rs:69-76` plain-text tracing | No JSON, no file sink, no rotation, no per-run log | P1 | new | +| NFR-OBSERV-02 | Partial | `analyze.rs:174,527,563,670`; `writer.rs:234,328` | `runs//stats.json` file never written | P1 | new | +| NFR-OBSERV-03 | **Missing** | no Prometheus surface | — | P1 | new | +| NFR-OBSERV-04 | **Missing** | no `CLA-INFRA-SUITE-COMPAT-REPORT` emitter | — | P1 | new | +| NFR-COMPAT-01 | **Missing** | no Filigree schema-pin CI job | — | P1 | new | +| NFR-COMPAT-02 | Partial | `wardline_probe.py:35`; `plugin.toml:50`; tests | Pins `__version__` not `REGISTRY_VERSION`; no `MIRRORED`/`ADDITIVE-SKEW` findings; no mirror fallback | P1 | new | +| NFR-COMPAT-03 | Missing | no Anthropic SDK dependency | LLM path deferred | P3 | deferred:scope-amendment | + +### 6.4 CON-* + +| ID | Verdict | Evidence | Gap | Sev | Tracked | +|----|---------|----------|-----|-----|---------| +| CON-LOOM-01 | Satisfied | no cross-product mediator; `filigree.rs` is read-only client | — | — | — | +| CON-FILIGREE-01 | Deferred | no `scan-results` POST | WP9-B | — | deferred:scope-amendment | +| CON-FILIGREE-02 | Satisfied | no `RegistryProtocol` impl; shadow-registry only | — | — | — | +| CON-WARDLINE-01 | Satisfied | `wardline_probe.py:38-43` direct import | Asterisk per loom.md §5 | — | — | +| CON-ANTHROPIC-01 | Partial | `mcp/src/config.rs:95-99`; `cli/src/serve.rs:11` | `CodexCliProvider` is non-Anthropic vendor surface | Med | new (recommend ticket) | +| CON-LOCAL-01 | Satisfied | CLI-only; LLM is only network egress | — | — | — | +| CON-RUST-01 | Satisfied | trivially | — | — | — | +| CON-SQLITE-01 | Satisfied | `rusqlite` + `deadpool-sqlite` | — | — | — | + +### 6.5 NG-* (inverted: all 25 honoured) + +All 25 non-goals returned the expected absence pattern. No drift detected. The codebase ships no rule engine, no taint, no SARIF export, no file watchers, no multi-branch analysis, no rename detection, no wiki UI, no second language plugin, no coverage ingestion, no advanced git analysis, no plugin hash-pinning, no triage-feedback loop, no Wardline HTTP state-pull, no BAR awareness, no Wardline annotation descriptor, no `EntityAlias`, no Phase-7 cross-cutting analyses, no incremental analysis, no Filigree server-side dedup. The two named asterisks (`docs/suite/loom.md` §5: Wardline REGISTRY import, Wardline pipeline coupling) remain exactly where the federation axiom documents them with the named retirement conditions still standing. + +--- + +## 7. Appendix: how to reproduce this audit + +Source artefacts: + +- Per-agent partial reports: `/tmp/clarion_gap_{A,B,C,D,E,F,G}.md` (transient; regenerated each run). +- In-flight Filigree snapshot: `/tmp/clarion_gap_filigree_ready.json` and `…_inprogress.json`. +- Trace bridge: `**Addresses**:` headers in `docs/clarion/1.0/system-design.md` lines 38, 122, 243, 389, 482, 575, 664, 749, 846, 1056, 1139. + +The audit was executed by seven concurrent subagents (general-purpose, capped at three in flight) over `docs/clarion/1.0/requirements.md` + `docs/clarion/1.0/system-design.md` + `docs/clarion/1.0/detailed-design.md` + the workspace source. Verdicts cite specific `file:line` evidence so each row is independently re-verifiable. diff --git a/docs/clarion/1.0/reviews/v1.0-publish-checklist-2026-05-24.md b/docs/clarion/1.0/reviews/v1.0-publish-checklist-2026-05-24.md new file mode 100644 index 00000000..d8af08a2 --- /dev/null +++ b/docs/clarion/1.0/reviews/v1.0-publish-checklist-2026-05-24.md @@ -0,0 +1,151 @@ +# Clarion v1.0 — Publish-Ready Master Checklist + +**Date:** 2026-05-24 +**Source of truth:** [`gap-analysis-2026-05-24.md`](./gap-analysis-2026-05-24.md) +**Definition of done:** every checkbox below ticked; requirements.md and shipped binary tell the same story; tracked Filigree issues for the remaining items closed or moved. +**Status legend:** `[ ]` pending · `[~]` in progress · `[x]` done · `[-]` decided out-of-scope. + +This is the working punch list to take the v1.0.0 tag from "cut" to "publishable." Each item names the requirement IDs it addresses, the specific action, file paths where known, an effort estimate (S = <1 hour, M = half-day, L = ≥1 day), and a back-reference to the gap analysis row. + +--- + +## Phase 1 — Doc cleanup (final sweep) + +The amendment work is 18 of 23 done. Five rows remain. + +- [x] **1.1 — Amend `REQ-INTEG-WARDLINE-02`** (S) — `> **Deferred to v0.2** … WP9-B Wardline-config ingest` blockquote added (`requirements.md:702`). → [gap §6.2](./gap-analysis-2026-05-24.md#62-req-integ--cross-product) +- [x] **1.2 — Amend `REQ-INTEG-WARDLINE-03`** (S) — fingerprint ingest, blockquote added (`requirements.md:712`). +- [x] **1.3 — Amend `REQ-INTEG-WARDLINE-04`** (S) — exceptions ingest, blockquote added (`requirements.md:722`). +- [x] **1.4 — Amend `REQ-INTEG-WARDLINE-05`** (S) — SARIF baseline ingest, WP10 cited (`requirements.md:732`). +- [x] **1.5 — Amend `REQ-INTEG-WARDLINE-06`** (S) — identity-reconciliation oracle, dependency chain on -03/-04 named (`requirements.md:742`). +- [x] **1.6 — Update `CHANGELOG.md` known-limitations** (S) — appended after REGISTRY-import asterisk; lists -02..-06 and names WP9-B + WP10 as the v0.2 landing surfaces (`CHANGELOG.md:123-128`). +- [x] **1.7 — Fix `NFR-SEC-03` verification line citations** (S) — replaced 1457/1495/1547/1579/1614 line numbers with test-name citations (`serve_rejects_non_loopback_http_bind_before_binding_without_opt_in`, `serve_http_refuses_startup_on_non_loopback_without_token`, `serve_http_refuses_startup_when_identity_env_is_missing`, `serve_http_files_endpoint_requires_hmac_identity_when_configured` + wrong-secret companion, `serve_http_files_endpoint_requires_bearer_token_when_configured` + wrong-token + batch companion, and the `serve_http_capabilities_does_not_require_token` carve-out). The three already-tracked ADR-034 refresh tickets (`clarion-7913f950d7`, `clarion-272b5bc1ec`, `clarion-461e78616f`) remain open for their own scope. + +**Exit criteria:** `grep -L "Deferred to v0.2" docs/clarion/1.0/requirements.md` matches expectations; no row claims v1.0 contract for deferred surface. + +--- + +## Phase 2 — Targeted code backfills (REQ rows outside any deferral) + +Small, well-scoped Rust + Python changes. Storage substrate already shaped for each. + +- [ ] **2.1 — `REQ-ANALYZE-03` — `--resume`** (M) — add `--resume ` flag to `crates/clarion-cli/src/cli.rs:30` Analyze variant; implement checkpoint reader; verify `kill -9` mid-run → resume continues. Also closes NFR-RELIABILITY-01's verification surface. → [gap §3.3](./gap-analysis-2026-05-24.md#33-p1--targeted-gaps-outside-any-deferral-envelope) +- [ ] **2.2 — `REQ-CATALOG-07` — HEAD-SHA capture** (S) — populate `first_seen_commit` / `last_seen_commit` columns in `crates/clarion-storage/src/commands.rs:66`. Call-sites currently write `None` at `crates/clarion-cli/src/analyze.rs` lines 813, 1619, 1692, 2156, 2348, 2417. Read `HEAD` once at BeginRun. +- [ ] **2.3 — `REQ-CATALOG-04` — file-entity git metadata** (M) — populate `git_churn_count` / `git_last_modified` / `git_last_modified_sha` / `git_authors` / `size_bytes` / `line_count` / `mime_type` in `crates/clarion-cli/src/analyze.rs:1544` `core_file_entity_record`. `git_churn_count` is already a generated column in `migrations/0001_initial_schema.sql:266` — wire the source-JSON path. + +--- + +## Phase 3 — Python plugin ontology coverage (REQ-PLUGIN-05/06) + +The single largest code-shaped P1 in the audit. Splits into two independent changes. + +- [ ] **3.1 — `REQ-PLUGIN-05` — import resolution policy** (L) — in `plugins/python/src/clarion_plugin_python/reference_resolver.py`: + - Detect and exclude `if TYPE_CHECKING:` blocks from runtime-import edges. + - Canonicalise `src.`-prefixed imports (strip when project layout uses src/). + - Emit `python:unresolved:` placeholder entities for unresolvable imports. + - Emit `alias_of` edges for `__init__.py` re-exports (definition site wins). +- [ ] **3.2 — `REQ-PLUGIN-06` — decorator detection policy** (L) — in `plugins/python/src/clarion_plugin_python/extractor.py`: + - Add `decorated_by` to `plugins/python/plugin.toml:40` edge kinds list. + - Detect decorators including factory invocations (`@app.route("/health")`), stacked (preserve order), class decorators, aliases (`validates = validates_shape`). + - Emit `decorated_by` edges with `properties` carrying decorator arguments. + - Also emit `inherits_from` (class hierarchy) and `uses_type` (type annotation references) — completes the v1.0 ontology declared in the plugin manifest. + +--- + +## Phase 4 — NFR completeness + +Wardline probe + Filigree schema pin + observability foundations. + +- [ ] **4.1 — `NFR-COMPAT-02` — Wardline probe pinning fix** (S) — `plugins/python/src/clarion_plugin_python/wardline_probe.py:35` currently pins `wardline.__version__`; should pin `wardline.core.registry.REGISTRY_VERSION` per `detailed-design.md:1169-1170`. Also emit `CLA-INFRA-WARDLINE-REGISTRY-MIRRORED` on out-of-range and `CLA-INFRA-WARDLINE-REGISTRY-ADDITIVE-SKEW` on additive-newer skew. Add test in `plugins/python/tests/test_wardline_probe.py`. +- [ ] **4.2 — `NFR-COMPAT-01` — Filigree schema-pin CI job** (S) — add a job to `.github/workflows/ci.yml` that fetches `GET /api/files/_schema` from a tagged Filigree release and asserts `valid_severities`, `valid_finding_statuses`, `valid_association_types` match the pinned fixture. Fixture location: `tests/fixtures/filigree-schema-pin.json`. +- [ ] **4.3 — `NFR-OBSERV-01` — JSON structured tracing** (M) — switch `crates/clarion-cli/src/main.rs:69-76` from default `tracing_subscriber::fmt` to `.json()` formatter; add file sink writing to `.clarion/clarion.log`; add per-run sink writing to `.clarion/runs//log.jsonl` (path already in install template's `.gitignore` per `install.rs:97`). Rotation: 100MB × 5. +- [ ] **4.4 — `NFR-OBSERV-02` — stats.json file emission** (S) — stats already computed at `crates/clarion-cli/src/analyze.rs:174,527,563,670` and persisted to `runs.stats` SQLite column. Add a tee that writes the same JSON to `.clarion/runs//stats.json` at CommitRun. +- [ ] **4.5 — `NFR-OBSERV-04` — `CLA-INFRA-SUITE-COMPAT-REPORT` emitter** (M) — collect: Filigree availability (already probed), Wardline REGISTRY status (from 4.1), HMAC/loopback posture, SARIF schema (when Wardline ingest lands; for now report "not configured"). Emit one consolidated finding per `clarion analyze` run. +- [-] **4.6 — `NFR-OBSERV-03` — Prometheus `/api/v1/metrics`** — **decision: defer**. Depends on the broader HTTP surface already deferred via REQ-HTTP-01 amendment. Bundle into a 1d amendment when convenient: add row-level "Deferred to v0.2 — depends on REQ-HTTP-01 broader HTTP surface" blockquote. + +--- + +## Phase 5 — Ops / packaging + +- [ ] **5.1 — `NFR-OPS-03` decision — `clarion db` subcommand** (M, or amend to L decision) — either: + - **Build it:** add `clarion db export --textual` (deterministic SQL dump) + `clarion db merge-helper` (subcommand stub) to `crates/clarion-cli/src/`. + - **Narrow it:** amend `NFR-OPS-03` text to scope to "commit-by-default" only; defer export/merge-helper to v0.2. + + Recommend the amendment path; the `db export --textual` surface is also referenced by REQ-ANALYZE-07 (determinism verification) and both could land together post-1.0. +- [ ] **5.2 — `NFR-OPS-04` — PyPI publish + SLSA over sdist** (M) — tracked in Filigree as `clarion-f530101222`. Two parts: + - Publish `clarion-plugin-python` to PyPI (operator-doc'd install path changes from GitHub Release URL to `pipx install clarion-plugin-python`). + - Extend `release-subjects` job at `.github/workflows/release.yml:201` to include the sdist; SLSA generator at `:391` picks it up. +- [ ] **5.3 — `NFR-OPS-01` matrix** (S, optional) — current 3-target matrix at `release.yml:163` covers macOS x86_64/aarch64 + Linux x86_64. `requirements.md:828` promises 5 targets (adds aarch64-linux + windows-x86_64). Either expand the matrix or narrow the requirement to the shipped 3. +- [ ] **5.4 — Existing v1.0 blocker tickets to verify on this pass:** + - `clarion-22c35e1c44` — Enable live GitHub governance (branch protection, ruleset, RELEASE_GOVERNANCE_TOKEN). + - `clarion-cf3f018cf4` — Add ruleset for `refs/tags/v*` and tag-protection. + - `clarion-6a622d5f7e` — Ancestor-of-main check in release.yml verify. + - `clarion-316e9feef9` — Verify-published-release job after create-release. + - `clarion-ee22d1d72c` — PRAGMA `integrity_check` in e2e + documented backup procedure. + - `clarion-04ec1044e9` — Storage deployment constraints doc (NFS, double-analyze, backup). + - `clarion-d59fc0b798` — Pre-WP5 `.clarion/` upgrade requirement doc. + +--- + +## Phase 6 — Tracker hygiene + governance + +- [ ] **6.1 — Close already-done Filigree issues** (S) — per standing authorisation in user memory; verify in HEAD before closing. + - `clarion-a4fb59a96a` — rollback runbook ships at `docs/operator/v1.0-release-rollback.md` (141 lines). + - `clarion-42f4fee904` — loopback trust banner emitted at `crates/clarion-cli/src/http_read.rs:244-248`; documented at `docs/operator/clarion-http-read-api.md:58-73`. +- [ ] **6.2 — `CON-ANTHROPIC-01` × `CodexCliProvider` decision** (S, governance) — `crates/clarion-mcp/src/config.rs:95-99` enumerates `CodexCli` (non-Anthropic). Two options: + - **Amend constraint:** widen `CON-ANTHROPIC-01` in `requirements.md` to recognise local-CLI providers as a separate category alongside API providers. + - **Gate code:** put `CodexCli` behind a default-off `cargo` feature. + - Recommend amend — the constraint's prompt-caching rationale doesn't apply to CLI shell-outs. + +--- + +## Phase 7 — Post-publish (immediately after tag, before any other work) + +Flips the five "implemented but unmeasured" P2 verdicts to Satisfied without writing product code. + +- [ ] **7.1 — Elspeth-slice measurement run** (M) — single recorded `clarion analyze /home/john/elspeth` run; capture entity count + DB size + cache hit rate + wall clock; commit `runs//stats.json` snapshot to `tests/fixtures/elspeth-scale-2026-05.json`. Satisfies NFR-SCALE-01, NFR-SCALE-02, NFR-COST-02 measurement gap. +- [ ] **7.2 — Criterion benchmark harness for MCP latency** (M) — add `crates/clarion-mcp/benches/mcp_latency.rs` measuring `initialize` (≤100ms) + hot-cache `summary` p95 (≤50ms). Satisfies NFR-PERF-02. +- [ ] **7.3 — Combined-load test for ReaderPool** (S) — extend `crates/clarion-storage/tests/reader_pool.rs` with a 16-reader scenario simulating the "1 consult agent + 1 Wardline-equivalent puller" load described in NFR-SCALE-03 design notes. + +--- + +## Definition of done + +v1.0 is publish-ready when: + +1. Phases 1–6 are all `[x]` or `[-]`. +2. `cargo nextest run --workspace --all-features` is green. +3. `bash tests/e2e/sprint_1_walking_skeleton.sh` and `tests/e2e/sprint_2_mcp_surface.sh` and `tests/e2e/phase3_subsystems.sh` all pass. +4. `gap-analysis-2026-05-24.md` §1 verdict tally has Missing = 0 in all categories. +5. `requirements.md` Δ-from-this-checklist is empty (every row that needed a blockquote got one). +6. `CHANGELOG.md` known-limitations section is complete and accurate. + +Phase 7 is post-publish and not gating. + +--- + +## Sizing summary + +| Phase | Items | Effort total | +|-------|------:|-------------:| +| 1 — Doc cleanup | 7 | ~1 day (mostly S) | +| 2 — REQ backfills | 3 | ~2 days | +| 3 — Python plugin | 2 | ~3 days | +| 4 — NFR completeness | 5 (+1 deferred) | ~3 days | +| 5 — Ops/packaging | 4 | ~2 days | +| 6 — Tracker hygiene | 2 | ~1 hour | +| 7 — Post-publish | 3 | ~1.5 days | +| **Total to publish** | **23** | **~11 dev-days** (Phases 1–6) | + +Phases 1, 4.1, 4.2, 5.3, 6.1, 6.2 can run in parallel and unlock most of the visible drift. Phase 3 (Python plugin) is the longest single workstream; if schedule-bound, it's the candidate to split into 3a (REQ-PLUGIN-05) + 3b (REQ-PLUGIN-06) with -05 prioritised (import resolution affects every analysis run's edge count; decorator coverage is additive value). + +--- + +## Cross-references + +- Full verdict + evidence per requirement: [`gap-analysis-2026-05-24.md` §6](./gap-analysis-2026-05-24.md#6-full-rtm). +- Amendment history: [`gap-analysis-2026-05-24.md` §0](./gap-analysis-2026-05-24.md#0-amendments-applied-this-session). +- Sprint-2 scope amendment (the load-bearing precedent doc): [`docs/implementation/sprint-2/scope-amendment-2026-05.md`](../../implementation/sprint-2/scope-amendment-2026-05.md). +- ADR-030 (on-demand summary scope, narrows WP6): [`docs/clarion/adr/ADR-030-on-demand-summary-scope.md`](../adr/ADR-030-on-demand-summary-scope.md). +- ADR-014 (Filigree registry-backend HTTP read API scope): [`docs/clarion/adr/ADR-014-filigree-registry-backend.md`](../adr/ADR-014-filigree-registry-backend.md). +- ADR-034 (HTTP read API hardening): [`docs/clarion/adr/ADR-034-federation-http-read-api-hardening.md`](../adr/ADR-034-federation-http-read-api-hardening.md).