chore: standardize AGENTS.md + adopt scoped ruff (published core excluded)

.github/workflows/render-report.yml

@@ -24,6 +24,12 @@ jobs:
       - name: Install Jupyter runtime
         run: python -m pip install --upgrade pip jupyter papermill jsonschema
 
+      - name: Ruff lint and format check (published core excluded — see ruff.toml)
+        run: |
+          python -m pip install ruff
+          ruff check .
+          ruff format --check .
+
       - name: Run unit tests (includes frozen conformance vector)
         run: PYTHONPATH=src python -m unittest discover -s tests -v
 

AGENTS.md

@@ -1,9 +1,24 @@
+---
+title: "AGENTS.md — eval-lab-methodology (PUBLIC)"
+category: charter
+component: eval-lab-methodology
+status: active
+version: 0.2.0
+last_updated: 2026-07-17
+tags: [methodology, statistics, evidence-contract, identity-domain, public, quarto]
+priority: high
+---
+
 # AGENTS.md — eval-lab-methodology (PUBLIC)
 
 Read this before doing any work in this repo. The estimator/report build order —
 what to build here and in what order — lives in `PLAN.md`; this file is the
 boundary and the honesty/sanitization rules that gate every commit.
 
+**This is a PUBLIC repository.** Anything here is world-readable, and the site
+**auto-publishes from `main`** — so treat every change as a public release from the
+moment it lands. There is no separate publish step at which to catch a leak.
+
 ## What this repo is
 
 The **public face** of the Agentic-Coding Evaluation Lab: the methodology write-up,
@@ -14,6 +29,30 @@ by anyone with no access to private infrastructure.
 
 Anything here is world-readable. Treat every commit as a public release.
 
+## Boundaries
+
+**This repo owns:**
+
+- The **infra-agnostic statistical + reporting core** — the estimators (Wilson,
+  two-stage bootstrap, Wilcoxon, GLMM wrapper), the superiority-by-margin decision
+  rule, the power simulation, and the Quarto report templates — developed and tested
+  here, runnable on synthetic or sanitized data.
+- The **normative identity-domain spec** (`src/eval_lab_methodology/identity_domain.py`)
+  and its frozen conformance vector.
+- The versioned **public evidence contract**, independent recomputation, and the
+  published methodology write-up + figures + dashboard.
+- The public site, which **auto-deploys from `main`** to
+  <https://jvjohnson.dev/eval-lab-methodology/>.
+
+**This repo does NOT own (must not become):**
+
+- A router, transport, or provider registry — it holds no secrets and routes nothing.
+- The working evaluation lab, its real task suite, or any real-campaign plumbing —
+  those live in private infrastructure and are referred to here only functionally.
+- A fork or vendored copy of the editing harness under evaluation — cite it as a
+  dependency only.
+- A store of raw real-run artifacts, private fixtures, or unverified numbers.
+
 ## The ecosystem and the boundary (roles)
 
 | Component | Visibility | Role | May it appear here? |
@@ -69,6 +108,31 @@ published values that downstream parity tests pin — **never edit them in place
 A change to the spec's semantics is a **new `schema_version` (v2) with a new
 conformance vector**, published alongside the old one, never a mutation of v1.
 
+## Frozen / do-not-edit-in-place
+
+- `src/eval_lab_methodology/identity_domain.py` — the normative identity-domain spec.
+  Its `CONFORMANCE_IDENTITY_DOMAIN` and `CONFORMANCE_IDENTITY_DOMAIN_SHA256` are
+  frozen, published values pinned by downstream parity tests. **Never edit them in
+  place.** A semantic change is a new `schema_version` with a new conformance vector,
+  published alongside the old one — never a mutation of the existing vector.
+
+## Hard rules
+
+1. **Public firewall.** Never reference private repos or private infrastructure by
+   name, and never introduce private absolute paths. Describe private couplings
+   **functionally** only. Merge to `main` auto-publishes, so **treat every change as
+   public** and assume anything committed is world-readable immediately.
+2. **No private imports.** Do not pull code, config, task fixtures, secrets, or
+   identifiers from private repos into this one.
+3. **Sanitization contract binds at merge time** — see the redaction list below.
+   Confirm none of it appears before anything is committed.
+4. **Honesty contract** — every published number traces to a produced artifact; no
+   number appears that a run didn't produce. See Honesty rules below.
+5. **Identity-domain freeze** — never edit the frozen conformance values in place;
+   semantic changes are a new versioned conformance vector.
+6. **Gate parity** — the exact merge gate below must pass locally before commit; the
+   local command equals CI.
+
 ## Sanitization contract (hard requirement — the redaction list)
 
 Before anything is committed, confirm NONE of these appear (files, filenames, and
@@ -96,15 +160,40 @@ bootstrap, GLMM, Quarto). The public alias "Agentic-Coding Evaluation Lab" is fi
   plainly. The honesty is the point of the artifact.
 - Seeded runs make intervals reproducible; keep it that way and say so.
 
-## Gates before commit
+## Gate parity (local == CI)
+
+The canonical merge gate — run from the repo root; local invocation equals CI, and
+every stage must pass before merge:
+
+```
+PYTHONPATH=src python -m unittest discover -s tests -v && PYTHONPATH=src python -m analysis.run_method_tranche --check && PYTHONPATH=src python -m analysis.run_contract_v2 --check
+```
 
-Run all three from the repo root; every one must pass before a commit:
+### Additional gates before commit
+
+In addition to the merge gate above, run these from the repo root; every one must
+pass before a commit:
 
 - `PYTHONPATH=src python -m unittest discover -s tests -v`
 - `make validate-report EVIDENCE=evidence/sample-lab-report.json` — and repeat
   for each `evidence/campaigns/**/evidence.json` file
 - `python -m pip wheel . --no-deps -w dist/`
 
+## Safe vs held commands
+
+**Safe (run freely):** the gate and validation commands above, `unittest` discovery,
+`make validate-report`, `python -m analysis.run_method_tranche --check`,
+`python -m analysis.run_contract_v2 --check`, `pip wheel`, figure regeneration
+(`figures/generate.py`), and local Quarto renders of synthetic-data example reports.
+
+**Held (stop and confirm first):**
+
+- Committing any real-run report, evidence, or figure — run the sanitization pass and
+  get a human confirmation first.
+- Editing the frozen identity-domain conformance values.
+- Any change that could reference private repos/infra or add a private path.
+- Anything that reaches private infrastructure, secrets, or the private task suite.
+
 ## What agents may / may not do here
 
 May: develop and unit-test the stats + reporting core on synthetic data; render
@@ -113,3 +202,22 @@ example reports; sanitize and publish real results/figures; improve the write-up
 May not: import private-repo code/config/secrets; name private repos or providers;
 commit raw real-run artifacts or the private task suite; publish an unverified
 number.
+
+## Truth lanes
+
+- `PLAN.md` — the roadmap and estimator/report build order (authoritative for what to
+  build here and in what sequence).
+- `README.md` — the public methodology write-up and project status.
+
+## STOP and escalate
+
+Stop and escalate to a human maintainer before proceeding when:
+
+- A change would (or might) leak any item on the sanitization redaction list.
+- You are about to commit a real-run report/evidence/figure and cannot fully verify
+  the sanitization pass.
+- A change would require editing the frozen identity-domain conformance values in
+  place, or otherwise mutating a published v1 conformance vector.
+- The gate parity command cannot be made to pass, or CI and local disagree.
+- A change would introduce a private repo name, provider account, or private absolute
+  path — since merge to `main` auto-publishes, this is unrecoverable once merged.

analysis/_method_tranche/dependence.py

@@ -131,7 +131,9 @@ def session_mixture_weight(rho: float, pi_d: float) -> float:
         raise ValueError("pi_d must satisfy 0 < pi_d <= 1")
     if rho == 0.0:
         return 0.0
-    lam = (rho * (1.0 - pi_d) + math.sqrt(rho * rho * (1.0 - pi_d) ** 2 + 4.0 * rho * pi_d)) / 2.0
+    lam = (
+        rho * (1.0 - pi_d) + math.sqrt(rho * rho * (1.0 - pi_d) ** 2 + 4.0 * rho * pi_d)
+    ) / 2.0
     if lam > 1.0:
         raise ValueError("no feasible mixture weight lam <= 1 for these inputs")
     return lam

analysis/_method_tranche/paired_trinomial.py

@@ -55,7 +55,9 @@
 def _exact(value: Fraction | int, name: str) -> Fraction:
     """Coerce ``value`` to ``Fraction``, rejecting floats to preserve exactness."""
     if isinstance(value, float):
-        raise TypeError(f"{name} must be an exact rational (Fraction or int), not float")
+        raise TypeError(
+            f"{name} must be an exact rational (Fraction or int), not float"
+        )
     return Fraction(value)
 
 
@@ -75,7 +77,9 @@ def _cell_weights(p_plus: Fraction, p_minus: Fraction) -> tuple[int, int, int, i
     return w_plus, w_minus, q - w_plus - w_minus, q
 
 
-def sum_distribution(n: int, p_plus: Fraction, p_minus: Fraction) -> dict[int, Fraction]:
+def sum_distribution(
+    n: int, p_plus: Fraction, p_minus: Fraction
+) -> dict[int, Fraction]:
     """Deterministically enumerate the exact distribution of ``S = sum_i d_i``.
 
     Integer-DP convolution of ``n`` i.i.d. trinomial tasks: per-step
@@ -208,7 +212,9 @@ def critical_value(dist: Mapping[int, Fraction], alpha: Fraction) -> int:
     raise AssertionError("unreachable: the tail at max(support) + 1 is exactly zero")
 
 
-def boundary_configuration(delta0: Fraction, pi_d: Fraction) -> tuple[Fraction, Fraction]:
+def boundary_configuration(
+    delta0: Fraction, pi_d: Fraction
+) -> tuple[Fraction, Fraction]:
     """Composite-null boundary point with mean difference ``delta0`` and discordance ``pi_d``.
 
     Returns ``(p_plus, p_minus) = ((pi_d + delta0) / 2, (pi_d - delta0) / 2)``,
@@ -296,9 +302,7 @@ def lfc_calibrate(
     if pi_d_grid is None:
         start = max(1, math.ceil(100 * delta0))
         hundredths = tuple(
-            point
-            for k in range(start, 101)
-            if (point := Fraction(k, 100)) > delta0
+            point for k in range(start, 101) if (point := Fraction(k, 100)) > delta0
         )
         grid: tuple[Fraction, ...] = (
             hundredths if delta0 == 0 else (delta0, *hundredths)
@@ -318,9 +322,7 @@ def lfc_calibrate(
             suffix[i] = suffix[i + 1] + dist[support[i]]
         tails.append((support, suffix))
     for c in range(-n, n + 2):
-        sizes = tuple(
-            suffix[bisect_left(support, c)] for support, suffix in tails
-        )
+        sizes = tuple(suffix[bisect_left(support, c)] for support, suffix in tails)
         sup_size = max(sizes)
         if sup_size <= alpha:
             return LfcCalibration(
@@ -380,7 +382,9 @@ def _signflip_pvalue_scaled(
     return Fraction(favourable, 2**n)
 
 
-def signflip_pvalue(n_plus: int, n_zero: int, n_minus: int, delta0: Fraction) -> Fraction:
+def signflip_pvalue(
+    n_plus: int, n_zero: int, n_minus: int, delta0: Fraction
+) -> Fraction:
     """Exact one-sided (upper) p-value of the shifted sign-flip statistic.
 
     With ``n = n_plus + n_zero + n_minus`` observed differences, the observed

analysis/run_method_tranche.py

@@ -164,7 +164,10 @@
     (
         "het-two-block-zero",
         Fraction(0),
-        ((20, Fraction(7, 20), Fraction(1, 20)), (20, Fraction(1, 20), Fraction(7, 20))),
+        (
+            (20, Fraction(7, 20), Fraction(1, 20)),
+            (20, Fraction(1, 20), Fraction(7, 20)),
+        ),
     ),
 )
 
@@ -288,7 +291,11 @@ def _null_boundary(
                         }
                     )
                 near_boundary.append(
-                    {"offset": _frac(offset), "true_delta": _frac(delta), "cells": cells}
+                    {
+                        "offset": _frac(offset),
+                        "true_delta": _frac(delta),
+                        "cells": cells,
+                    }
                 )
             record: dict[str, Any] = {
                 "n": n,
@@ -448,7 +455,9 @@ def _margin_feasibility(
                             "pi_d": _frac(pi_d),
                             "skipped": False,
                             "power": _frac(
-                                exact_power(n, p_plus, p_minus, calibration.critical_value)
+                                exact_power(
+                                    n, p_plus, p_minus, calibration.critical_value
+                                )
                             ),
                             "signflip_shifted_power": _frac(
                                 signflip_rejection_probability(
@@ -472,17 +481,17 @@ def _margin_feasibility(
                 }
             )
     check_calibration = calibrations[(40, DELTA0_RETIRED)]
-    check_p_plus, check_p_minus = boundary_configuration(Fraction(3, 10), Fraction(3, 10))
+    check_p_plus, check_p_minus = boundary_configuration(
+        Fraction(3, 10), Fraction(3, 10)
+    )
     derived_power = exact_power(
         40, check_p_plus, check_p_minus, check_calibration.critical_value
     )
     # The commission describes the comparison construction as calibrated at
     # the maximum-discordance boundary point (p_zero = 0).  Recompute that
     # calibration from this package's own committed code — never from the
     # unconsulted decision inputs — so the route comparison is traceable.
-    max_discordance = lfc_calibrate(
-        40, DELTA0_RETIRED, ALPHA, pi_d_grid=(Fraction(1),)
-    )
+    max_discordance = lfc_calibrate(40, DELTA0_RETIRED, ALPHA, pi_d_grid=(Fraction(1),))
     cross_check = {
         "n": 40,
         "delta0": _frac(DELTA0_RETIRED),
@@ -566,7 +575,9 @@ def _superiority_mpib(
                     "mpib": _frac(mpib),
                     "point_estimate_threshold": point_threshold,
                     "joint_threshold": joint_threshold,
-                    "p_joint_pass": _frac(tail_probability(distribution, joint_threshold)),
+                    "p_joint_pass": _frac(
+                        tail_probability(distribution, joint_threshold)
+                    ),
                     "p_point_estimate_only": _frac(
                         tail_probability(distribution, point_threshold)
                     ),

build_backend.py

@@ -19,11 +19,15 @@
 PACKAGE_ROOT = ROOT / "src" / "eval_lab_methodology"
 
 
-def get_requires_for_build_wheel(config_settings: dict[str, Any] | None = None) -> list[str]:
+def get_requires_for_build_wheel(
+    config_settings: dict[str, Any] | None = None,
+) -> list[str]:
     return []
 
 
-def get_requires_for_build_sdist(config_settings: dict[str, Any] | None = None) -> list[str]:
+def get_requires_for_build_sdist(
+    config_settings: dict[str, Any] | None = None,
+) -> list[str]:
     return []
 
 
@@ -50,7 +54,9 @@ def build_wheel(
 
     for path in sorted(PACKAGE_ROOT.rglob("*")):
         if path.is_file() and not _is_bytecode(path):
-            arcname = f"eval_lab_methodology/{path.relative_to(PACKAGE_ROOT).as_posix()}"
+            arcname = (
+                f"eval_lab_methodology/{path.relative_to(PACKAGE_ROOT).as_posix()}"
+            )
             payloads.append((arcname, path.read_bytes()))
 
     payloads.extend(
@@ -96,7 +102,10 @@ def build_sdist(
             if path.is_dir():
                 for item in sorted(path.rglob("*")):
                     if item.is_file() and not _is_bytecode(item):
-                        tar.add(item, arcname=f"{prefix}/{item.relative_to(ROOT).as_posix()}")
+                        tar.add(
+                            item,
+                            arcname=f"{prefix}/{item.relative_to(ROOT).as_posix()}",
+                        )
             elif path.is_file():
                 tar.add(path, arcname=f"{prefix}/{relative}")
     return sdist_name