promote: evidence-protocol-doc freshness + tripwire (#89)

CHANGELOG.md

@@ -8,6 +8,23 @@ follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ### Documentation
 
+- **Evidence-protocol-doc freshness.**
+  [`docs/EVIDENCE.md`](docs/EVIDENCE.md) now reflects v0.6.4 artifact semantics:
+  the verdict-preservation section (§4.5) carries all nine `Verdict` members
+  instead of the stale pre-0.6 five; the system-operations framing (§2) is
+  restated as the producer / read-only-consumer split (`run` / `minimize`
+  produce; `replay`, `inspect`, `diff`, `history`, `timeline`, `matrix`,
+  `verify`, `export` consume), replacing a mislabeled "five operations" list;
+  the materialized-spec section (§4.2) documents that `paraphrase` lineage
+  preserves `validity_score` / `validity_method`; identity (§4.1) cross-links
+  `cli_invocation` provenance to its §6 semantic boundary; and §4.5 clarifies
+  that semantic-oracle effects are preserved *through* the assigned verdict and
+  existing case/judgment fields, not as a separate stored oracle-results
+  payload. A coarse meta-test
+  ([`tests/meta/test_evidence_doc_freshness.py`](tests/meta/test_evidence_doc_freshness.py))
+  now guards the contract: every `Verdict` member and every read-only consumer
+  must be named in the doc. No runtime behavior changes.
+
 - **Architecture-doc freshness + preservation guardrails.**
   [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) now reflects the full
   11-command CLI surface (the read-only consumers `inspect`, `history`,

docs/EVIDENCE.md

@@ -73,16 +73,36 @@ supporting a reliability claim. Without evidence, claims are
 anecdotes; with evidence, claims become inspectable. The replay
 artifact is reliability evidence in physical form.
 
-The five operations of the system, stated in these terms:
+The operations of the system divide cleanly into **producers** (which
+generate, derive, and preserve evidence) and **read-only consumers**
+(which re-present preserved evidence without re-deriving the claim).
+
+Producers — the forward pipeline that mints an artifact:
 
 - **Perturbations** generate reliability evidence
 - **Invariants + resolver** derive a reliability claim from generated
   evidence
 - **The replay artifact** preserves the (claim + evidence) pair
-- **`replay`** re-presents preserved evidence without re-deriving the
-  claim
-- **`diff`** compares two preserved (claim + evidence) pairs
+
+`run` drives this pipeline; `minimize` is a producer variant that
+searches for a smaller perturbation set reproducing the same verdict.
+
+Read-only consumers — they read a preserved artifact and never
+re-derive the claim:
+
+- **`replay`** re-presents the original run output verbatim
 - **`inspect`** expands preserved evidence on demand for legibility
+- **`diff`** compares two preserved (claim + evidence) pairs
+- **`history`** / **`timeline`** / **`matrix`** present preserved
+  artifacts across runs
+- **`verify`** / **`export`** check and package a preserved artifact
+
+(`doctor` is an environment diagnostic, not an evidence-protocol
+operation.) This producer/consumer split is the load-bearing boundary:
+preservation is read-only after save, so no consumer can mutate the
+claim it reads. The exact command surface is enumerated in
+[`ARCHITECTURE.md`](ARCHITECTURE.md); this document names the
+operations only to the depth the protocol semantics require.
 
 Perturbation engines are **replaceable** evidence generators —
 different families (paraphrase, retrieval, ordering) all feed the
@@ -159,6 +179,12 @@ The canonical contents of an artifact, by category:
 - `falsifyai_version` — the framework version that produced the
   artifact. Required for forward compatibility decisions.
 
+Provenance — *which command produced this artifact* — is carried by
+`cli_invocation` (descriptive, not part of the deterministic identity).
+Its semantic boundary is defined once in §6: it records the normalized
+invocation, **not** a guarantee that re-running reproduces the outputs
+(replay determinism lives in `materialized_hash` and `bundle_id`).
+
 The three hashes are deliberately split. They answer three different
 audit questions: *"is this the same file?"* / *"did materialization
 produce the same inputs?"* / *"which specific invocation am I looking
@@ -176,6 +202,14 @@ The full set of realized perturbations and their lineage:
   resulting input string)
 - The materialization order (the order is part of the identity)
 
+For meaning-preserving families gated by validity (the `paraphrase`
+family under bidirectional NLI), the lineage also preserves the
+validity evidence for each accepted perturbation: `validity_score` and
+`validity_method` are stamped into the perturbation's parameters. This
+keeps the *reason a rewrite was admitted* inspectable — a reader can
+audit not just that a paraphrase was used, but that it cleared the
+validity gate and by what measure.
+
 The materialized spec is preserved *as data*, not as a reference to the
 source YAML. This is intentional: the source YAML may be deleted,
 edited, or renamed; the artifact must remain self-contained. A reader
@@ -214,8 +248,11 @@ judgment, not just accept it.
 
 For each case:
 
-- The assigned verdict (`STABLE`, `FRAGILE`, `CONSISTENTLY_WRONG`,
-  `INSUFFICIENT`, `INVALID_EVAL`)
+- The assigned verdict — one of the nine taxonomy members:
+  `INFORMATION_PRESENT`, `STABLE`, `CONSISTENTLY_WRONG`,
+  `ADVERSARIALLY_VULNERABLE`, `FRAGILE`, `INFORMATION_NULL`,
+  `AMBIGUOUS`, `INSUFFICIENT`, `INVALID_EVAL` (the claim shape of each
+  is tabled in §7)
 - The stratified per-perturbation-family stability distribution that
   drove the verdict
 - The bootstrap confidence interval per family
@@ -227,6 +264,17 @@ per-case verdicts via the documented priority chain.
 The verdict is *the claim*. The rest of the artifact is *what the
 claim rests on*.
 
+**Semantic-oracle effects are preserved through the verdict, not as a
+separate payload.** The grounding, information-null, and meta oracles
+contribute their signals *into* verdict resolution; the artifact does
+not store a distinct "oracle results" record alongside the verdict.
+What an oracle concluded is recoverable from the assigned verdict (an
+`INFORMATION_PRESENT` verdict *is* the grounding oracle's affirmative
+signal made durable) together with the existing per-case and
+per-judgment fields above. This is deliberate: a second stored copy of
+oracle output would be redundant with the verdict it already produced,
+and redundancy invites drift between the two.
+
 ### 4.6 Falsifiability scoring
 
 The suite falsifiability score and per-case contributions. This is a

tests/meta/test_evidence_doc_freshness.py

@@ -0,0 +1,69 @@
+"""Stale-doc tripwire for the evidence-protocol contract document.
+
+``docs/EVIDENCE.md`` is the artifact contract: it specifies what the replay
+artifact *is*, what its verdicts mean as claims, and what consumers may do with
+it. Stale claims there are higher-risk than stale claims anywhere else in the
+repo, because the document is the thing external readers trust.
+
+This is a **coarse** freshness guard, not a prose validator. It enforces two
+cheap, high-signal invariants and nothing semantic:
+
+1. Every ``Verdict`` enum member is named at least once in the document. This
+   catches the failure mode where a new verdict class is wired into the
+   taxonomy but the contract doc is never updated to mention it. It does *not*
+   catch a verdict named in one section but stale in another (e.g. an outdated
+   per-section list while §7 stays complete) -- a document-wide substring scan
+   cannot see intra-document inconsistency. That class of drift is caught by
+   review, not here.
+2. The document names the read-only preservation consumers at least once, so
+   the producer/consumer boundary the artifact depends on stays documented as
+   the command surface grows.
+
+If this test fails, update ``docs/EVIDENCE.md`` -- do not weaken the assertion.
+"""
+
+from pathlib import Path
+
+from falsifyai.verdict.models import Verdict
+
+_EVIDENCE_DOC = Path(__file__).resolve().parents[2] / "docs" / "EVIDENCE.md"
+
+# Read-only consumers of a preserved artifact. The protocol guarantee is that
+# these re-present preserved evidence and never re-derive the claim; the doc
+# must keep describing that surface as it grows.
+_READONLY_CONSUMERS = (
+    "replay",
+    "inspect",
+    "diff",
+    "history",
+    "timeline",
+    "matrix",
+    "verify",
+    "export",
+)
+
+
+def _doc_text() -> str:
+    assert _EVIDENCE_DOC.exists(), f"evidence contract doc missing: {_EVIDENCE_DOC}"
+    return _EVIDENCE_DOC.read_text(encoding="utf-8")
+
+
+def test_every_verdict_appears_in_evidence_doc() -> None:
+    text = _doc_text()
+    missing = [v.name for v in Verdict if v.name not in text]
+    assert not missing, (
+        f"docs/EVIDENCE.md does not mention these Verdict members: {missing}. "
+        f"A new verdict class was wired into the taxonomy but the artifact "
+        f"contract doc was not updated. Add the verdict (at minimum to the §7 "
+        f"claim table) before shipping."
+    )
+
+
+def test_evidence_doc_names_readonly_consumers() -> None:
+    text = _doc_text()
+    missing = [c for c in _READONLY_CONSUMERS if c not in text]
+    assert not missing, (
+        f"docs/EVIDENCE.md no longer names these read-only consumers: {missing}. "
+        f"The producer/consumer boundary is load-bearing protocol semantics; "
+        f"keep the consumer surface documented."
+    )