diff --git a/.github/workflows/render-report.yml b/.github/workflows/render-report.yml
index 72c18b7..51c9f59 100644
--- a/.github/workflows/render-report.yml
+++ b/.github/workflows/render-report.yml
@@ -24,6 +24,12 @@ jobs:
- name: Install Jupyter runtime
run: python -m pip install --upgrade pip jupyter papermill jsonschema
+ - name: Ruff lint and format check (published core excluded — see ruff.toml)
+ run: |
+ python -m pip install ruff
+ ruff check .
+ ruff format --check .
+
- name: Run unit tests (includes frozen conformance vector)
run: PYTHONPATH=src python -m unittest discover -s tests -v
diff --git a/AGENTS.md b/AGENTS.md
index 256f7b6..9b0f1ce 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -1,9 +1,24 @@
+---
+title: "AGENTS.md — eval-lab-methodology (PUBLIC)"
+category: charter
+component: eval-lab-methodology
+status: active
+version: 0.2.0
+last_updated: 2026-07-17
+tags: [methodology, statistics, evidence-contract, identity-domain, public, quarto]
+priority: high
+---
+
# AGENTS.md — eval-lab-methodology (PUBLIC)
Read this before doing any work in this repo. The estimator/report build order —
what to build here and in what order — lives in `PLAN.md`; this file is the
boundary and the honesty/sanitization rules that gate every commit.
+**This is a PUBLIC repository.** Anything here is world-readable, and the site
+**auto-publishes from `main`** — so treat every change as a public release from the
+moment it lands. There is no separate publish step at which to catch a leak.
+
## What this repo is
The **public face** of the Agentic-Coding Evaluation Lab: the methodology write-up,
@@ -14,6 +29,30 @@ by anyone with no access to private infrastructure.
Anything here is world-readable. Treat every commit as a public release.
+## Boundaries
+
+**This repo owns:**
+
+- The **infra-agnostic statistical + reporting core** — the estimators (Wilson,
+ two-stage bootstrap, Wilcoxon, GLMM wrapper), the superiority-by-margin decision
+ rule, the power simulation, and the Quarto report templates — developed and tested
+ here, runnable on synthetic or sanitized data.
+- The **normative identity-domain spec** (`src/eval_lab_methodology/identity_domain.py`)
+ and its frozen conformance vector.
+- The versioned **public evidence contract**, independent recomputation, and the
+ published methodology write-up + figures + dashboard.
+- The public site, which **auto-deploys from `main`** to
+ .
+
+**This repo does NOT own (must not become):**
+
+- A router, transport, or provider registry — it holds no secrets and routes nothing.
+- The working evaluation lab, its real task suite, or any real-campaign plumbing —
+ those live in private infrastructure and are referred to here only functionally.
+- A fork or vendored copy of the editing harness under evaluation — cite it as a
+ dependency only.
+- A store of raw real-run artifacts, private fixtures, or unverified numbers.
+
## The ecosystem and the boundary (roles)
| Component | Visibility | Role | May it appear here? |
@@ -69,6 +108,31 @@ published values that downstream parity tests pin — **never edit them in place
A change to the spec's semantics is a **new `schema_version` (v2) with a new
conformance vector**, published alongside the old one, never a mutation of v1.
+## Frozen / do-not-edit-in-place
+
+- `src/eval_lab_methodology/identity_domain.py` — the normative identity-domain spec.
+ Its `CONFORMANCE_IDENTITY_DOMAIN` and `CONFORMANCE_IDENTITY_DOMAIN_SHA256` are
+ frozen, published values pinned by downstream parity tests. **Never edit them in
+ place.** A semantic change is a new `schema_version` with a new conformance vector,
+ published alongside the old one — never a mutation of the existing vector.
+
+## Hard rules
+
+1. **Public firewall.** Never reference private repos or private infrastructure by
+ name, and never introduce private absolute paths. Describe private couplings
+ **functionally** only. Merge to `main` auto-publishes, so **treat every change as
+ public** and assume anything committed is world-readable immediately.
+2. **No private imports.** Do not pull code, config, task fixtures, secrets, or
+ identifiers from private repos into this one.
+3. **Sanitization contract binds at merge time** — see the redaction list below.
+ Confirm none of it appears before anything is committed.
+4. **Honesty contract** — every published number traces to a produced artifact; no
+ number appears that a run didn't produce. See Honesty rules below.
+5. **Identity-domain freeze** — never edit the frozen conformance values in place;
+ semantic changes are a new versioned conformance vector.
+6. **Gate parity** — the exact merge gate below must pass locally before commit; the
+ local command equals CI.
+
## Sanitization contract (hard requirement — the redaction list)
Before anything is committed, confirm NONE of these appear (files, filenames, and
@@ -96,15 +160,40 @@ bootstrap, GLMM, Quarto). The public alias "Agentic-Coding Evaluation Lab" is fi
plainly. The honesty is the point of the artifact.
- Seeded runs make intervals reproducible; keep it that way and say so.
-## Gates before commit
+## Gate parity (local == CI)
+
+The canonical merge gate — run from the repo root; local invocation equals CI, and
+every stage must pass before merge:
+
+```
+PYTHONPATH=src python -m unittest discover -s tests -v && PYTHONPATH=src python -m analysis.run_method_tranche --check && PYTHONPATH=src python -m analysis.run_contract_v2 --check
+```
-Run all three from the repo root; every one must pass before a commit:
+### Additional gates before commit
+
+In addition to the merge gate above, run these from the repo root; every one must
+pass before a commit:
- `PYTHONPATH=src python -m unittest discover -s tests -v`
- `make validate-report EVIDENCE=evidence/sample-lab-report.json` — and repeat
for each `evidence/campaigns/**/evidence.json` file
- `python -m pip wheel . --no-deps -w dist/`
+## Safe vs held commands
+
+**Safe (run freely):** the gate and validation commands above, `unittest` discovery,
+`make validate-report`, `python -m analysis.run_method_tranche --check`,
+`python -m analysis.run_contract_v2 --check`, `pip wheel`, figure regeneration
+(`figures/generate.py`), and local Quarto renders of synthetic-data example reports.
+
+**Held (stop and confirm first):**
+
+- Committing any real-run report, evidence, or figure — run the sanitization pass and
+ get a human confirmation first.
+- Editing the frozen identity-domain conformance values.
+- Any change that could reference private repos/infra or add a private path.
+- Anything that reaches private infrastructure, secrets, or the private task suite.
+
## What agents may / may not do here
May: develop and unit-test the stats + reporting core on synthetic data; render
@@ -113,3 +202,22 @@ example reports; sanitize and publish real results/figures; improve the write-up
May not: import private-repo code/config/secrets; name private repos or providers;
commit raw real-run artifacts or the private task suite; publish an unverified
number.
+
+## Truth lanes
+
+- `PLAN.md` — the roadmap and estimator/report build order (authoritative for what to
+ build here and in what sequence).
+- `README.md` — the public methodology write-up and project status.
+
+## STOP and escalate
+
+Stop and escalate to a human maintainer before proceeding when:
+
+- A change would (or might) leak any item on the sanitization redaction list.
+- You are about to commit a real-run report/evidence/figure and cannot fully verify
+ the sanitization pass.
+- A change would require editing the frozen identity-domain conformance values in
+ place, or otherwise mutating a published v1 conformance vector.
+- The gate parity command cannot be made to pass, or CI and local disagree.
+- A change would introduce a private repo name, provider account, or private absolute
+ path — since merge to `main` auto-publishes, this is unrecoverable once merged.
diff --git a/analysis/_method_tranche/dependence.py b/analysis/_method_tranche/dependence.py
index 382f1ee..7be1ca7 100644
--- a/analysis/_method_tranche/dependence.py
+++ b/analysis/_method_tranche/dependence.py
@@ -131,7 +131,9 @@ def session_mixture_weight(rho: float, pi_d: float) -> float:
raise ValueError("pi_d must satisfy 0 < pi_d <= 1")
if rho == 0.0:
return 0.0
- lam = (rho * (1.0 - pi_d) + math.sqrt(rho * rho * (1.0 - pi_d) ** 2 + 4.0 * rho * pi_d)) / 2.0
+ lam = (
+ rho * (1.0 - pi_d) + math.sqrt(rho * rho * (1.0 - pi_d) ** 2 + 4.0 * rho * pi_d)
+ ) / 2.0
if lam > 1.0:
raise ValueError("no feasible mixture weight lam <= 1 for these inputs")
return lam
diff --git a/analysis/_method_tranche/paired_trinomial.py b/analysis/_method_tranche/paired_trinomial.py
index 0cb1758..bce3525 100644
--- a/analysis/_method_tranche/paired_trinomial.py
+++ b/analysis/_method_tranche/paired_trinomial.py
@@ -55,7 +55,9 @@
def _exact(value: Fraction | int, name: str) -> Fraction:
"""Coerce ``value`` to ``Fraction``, rejecting floats to preserve exactness."""
if isinstance(value, float):
- raise TypeError(f"{name} must be an exact rational (Fraction or int), not float")
+ raise TypeError(
+ f"{name} must be an exact rational (Fraction or int), not float"
+ )
return Fraction(value)
@@ -75,7 +77,9 @@ def _cell_weights(p_plus: Fraction, p_minus: Fraction) -> tuple[int, int, int, i
return w_plus, w_minus, q - w_plus - w_minus, q
-def sum_distribution(n: int, p_plus: Fraction, p_minus: Fraction) -> dict[int, Fraction]:
+def sum_distribution(
+ n: int, p_plus: Fraction, p_minus: Fraction
+) -> dict[int, Fraction]:
"""Deterministically enumerate the exact distribution of ``S = sum_i d_i``.
Integer-DP convolution of ``n`` i.i.d. trinomial tasks: per-step
@@ -208,7 +212,9 @@ def critical_value(dist: Mapping[int, Fraction], alpha: Fraction) -> int:
raise AssertionError("unreachable: the tail at max(support) + 1 is exactly zero")
-def boundary_configuration(delta0: Fraction, pi_d: Fraction) -> tuple[Fraction, Fraction]:
+def boundary_configuration(
+ delta0: Fraction, pi_d: Fraction
+) -> tuple[Fraction, Fraction]:
"""Composite-null boundary point with mean difference ``delta0`` and discordance ``pi_d``.
Returns ``(p_plus, p_minus) = ((pi_d + delta0) / 2, (pi_d - delta0) / 2)``,
@@ -296,9 +302,7 @@ def lfc_calibrate(
if pi_d_grid is None:
start = max(1, math.ceil(100 * delta0))
hundredths = tuple(
- point
- for k in range(start, 101)
- if (point := Fraction(k, 100)) > delta0
+ point for k in range(start, 101) if (point := Fraction(k, 100)) > delta0
)
grid: tuple[Fraction, ...] = (
hundredths if delta0 == 0 else (delta0, *hundredths)
@@ -318,9 +322,7 @@ def lfc_calibrate(
suffix[i] = suffix[i + 1] + dist[support[i]]
tails.append((support, suffix))
for c in range(-n, n + 2):
- sizes = tuple(
- suffix[bisect_left(support, c)] for support, suffix in tails
- )
+ sizes = tuple(suffix[bisect_left(support, c)] for support, suffix in tails)
sup_size = max(sizes)
if sup_size <= alpha:
return LfcCalibration(
@@ -380,7 +382,9 @@ def _signflip_pvalue_scaled(
return Fraction(favourable, 2**n)
-def signflip_pvalue(n_plus: int, n_zero: int, n_minus: int, delta0: Fraction) -> Fraction:
+def signflip_pvalue(
+ n_plus: int, n_zero: int, n_minus: int, delta0: Fraction
+) -> Fraction:
"""Exact one-sided (upper) p-value of the shifted sign-flip statistic.
With ``n = n_plus + n_zero + n_minus`` observed differences, the observed
diff --git a/analysis/run_method_tranche.py b/analysis/run_method_tranche.py
index 575c1eb..bf7c0ca 100644
--- a/analysis/run_method_tranche.py
+++ b/analysis/run_method_tranche.py
@@ -164,7 +164,10 @@
(
"het-two-block-zero",
Fraction(0),
- ((20, Fraction(7, 20), Fraction(1, 20)), (20, Fraction(1, 20), Fraction(7, 20))),
+ (
+ (20, Fraction(7, 20), Fraction(1, 20)),
+ (20, Fraction(1, 20), Fraction(7, 20)),
+ ),
),
)
@@ -288,7 +291,11 @@ def _null_boundary(
}
)
near_boundary.append(
- {"offset": _frac(offset), "true_delta": _frac(delta), "cells": cells}
+ {
+ "offset": _frac(offset),
+ "true_delta": _frac(delta),
+ "cells": cells,
+ }
)
record: dict[str, Any] = {
"n": n,
@@ -448,7 +455,9 @@ def _margin_feasibility(
"pi_d": _frac(pi_d),
"skipped": False,
"power": _frac(
- exact_power(n, p_plus, p_minus, calibration.critical_value)
+ exact_power(
+ n, p_plus, p_minus, calibration.critical_value
+ )
),
"signflip_shifted_power": _frac(
signflip_rejection_probability(
@@ -472,7 +481,9 @@ def _margin_feasibility(
}
)
check_calibration = calibrations[(40, DELTA0_RETIRED)]
- check_p_plus, check_p_minus = boundary_configuration(Fraction(3, 10), Fraction(3, 10))
+ check_p_plus, check_p_minus = boundary_configuration(
+ Fraction(3, 10), Fraction(3, 10)
+ )
derived_power = exact_power(
40, check_p_plus, check_p_minus, check_calibration.critical_value
)
@@ -480,9 +491,7 @@ def _margin_feasibility(
# the maximum-discordance boundary point (p_zero = 0). Recompute that
# calibration from this package's own committed code — never from the
# unconsulted decision inputs — so the route comparison is traceable.
- max_discordance = lfc_calibrate(
- 40, DELTA0_RETIRED, ALPHA, pi_d_grid=(Fraction(1),)
- )
+ max_discordance = lfc_calibrate(40, DELTA0_RETIRED, ALPHA, pi_d_grid=(Fraction(1),))
cross_check = {
"n": 40,
"delta0": _frac(DELTA0_RETIRED),
@@ -566,7 +575,9 @@ def _superiority_mpib(
"mpib": _frac(mpib),
"point_estimate_threshold": point_threshold,
"joint_threshold": joint_threshold,
- "p_joint_pass": _frac(tail_probability(distribution, joint_threshold)),
+ "p_joint_pass": _frac(
+ tail_probability(distribution, joint_threshold)
+ ),
"p_point_estimate_only": _frac(
tail_probability(distribution, point_threshold)
),
diff --git a/build_backend.py b/build_backend.py
index 13c66e8..6bbcc57 100644
--- a/build_backend.py
+++ b/build_backend.py
@@ -19,11 +19,15 @@
PACKAGE_ROOT = ROOT / "src" / "eval_lab_methodology"
-def get_requires_for_build_wheel(config_settings: dict[str, Any] | None = None) -> list[str]:
+def get_requires_for_build_wheel(
+ config_settings: dict[str, Any] | None = None,
+) -> list[str]:
return []
-def get_requires_for_build_sdist(config_settings: dict[str, Any] | None = None) -> list[str]:
+def get_requires_for_build_sdist(
+ config_settings: dict[str, Any] | None = None,
+) -> list[str]:
return []
@@ -50,7 +54,9 @@ def build_wheel(
for path in sorted(PACKAGE_ROOT.rglob("*")):
if path.is_file() and not _is_bytecode(path):
- arcname = f"eval_lab_methodology/{path.relative_to(PACKAGE_ROOT).as_posix()}"
+ arcname = (
+ f"eval_lab_methodology/{path.relative_to(PACKAGE_ROOT).as_posix()}"
+ )
payloads.append((arcname, path.read_bytes()))
payloads.extend(
@@ -96,7 +102,10 @@ def build_sdist(
if path.is_dir():
for item in sorted(path.rglob("*")):
if item.is_file() and not _is_bytecode(item):
- tar.add(item, arcname=f"{prefix}/{item.relative_to(ROOT).as_posix()}")
+ tar.add(
+ item,
+ arcname=f"{prefix}/{item.relative_to(ROOT).as_posix()}",
+ )
elif path.is_file():
tar.add(path, arcname=f"{prefix}/{relative}")
return sdist_name
diff --git a/figures/generate.py b/figures/generate.py
index 2e92a77..46a263c 100644
--- a/figures/generate.py
+++ b/figures/generate.py
@@ -34,16 +34,32 @@ def esc(text: str) -> str:
return text.replace("&", "&").replace("<", "<").replace(">", ">")
-def text(x: float, y: float, s: str, *, size: int = 13, anchor: str = "start",
- fill: str = INK, weight: str = "normal") -> str:
+def text(
+ x: float,
+ y: float,
+ s: str,
+ *,
+ size: int = 13,
+ anchor: str = "start",
+ fill: str = INK,
+ weight: str = "normal",
+) -> str:
return (
f"{esc(s)}"
)
-def line(x1: float, y1: float, x2: float, y2: float, *, stroke: str = GRID,
- width: float = 1.0, dash: str | None = None) -> str:
+def line(
+ x1: float,
+ y1: float,
+ x2: float,
+ y2: float,
+ *,
+ stroke: str = GRID,
+ width: float = 1.0,
+ dash: str | None = None,
+) -> str:
d = f" stroke-dasharray='{dash}'" if dash else ""
return (
f" float:
return x0 + (delta + 1.0) / 2.0 * (x1 - x0)
body: list[str] = []
- body.append(text(left, 34, "Historical promotion-rule examples", size=18, weight="bold"))
- body.append(text(left, 56, "Paired success delta (candidate minus incumbent) with seeded bootstrap 95% CI",
- size=13, fill="#52606d"))
+ body.append(
+ text(left, 34, "Historical promotion-rule examples", size=18, weight="bold")
+ )
+ body.append(
+ text(
+ left,
+ 56,
+ "Paired success delta (candidate minus incumbent) with seeded bootstrap 95% CI",
+ size=13,
+ fill="#52606d",
+ )
+ )
# axis ticks
for tick in (-1.0, -0.5, 0.0, 0.5, 1.0):
xt = px(tick)
body.append(line(xt, top, xt, h - bot, stroke=GRID, width=1.0))
- body.append(text(xt, h - bot + 20, f"{tick:+.1f}", size=12, anchor="middle", fill="#52606d"))
+ body.append(
+ text(
+ xt,
+ h - bot + 20,
+ f"{tick:+.1f}",
+ size=12,
+ anchor="middle",
+ fill="#52606d",
+ )
+ )
# zero + band
body.append(line(px(0.0), top, px(0.0), h - bot, stroke=ZERO, width=2.0))
- body.append(text(px(0.0), top - 8, "0", size=12, anchor="middle", fill=ZERO, weight="bold"))
+ body.append(
+ text(px(0.0), top - 8, "0", size=12, anchor="middle", fill=ZERO, weight="bold")
+ )
for b in (band, -band):
- body.append(line(px(b), top, px(b), h - bot, stroke="#9aa5b1", width=1.0, dash="4 3"))
- body.append(text(px(band), h - bot + 38, "historical rule: bootstrap CI lower bound > +0.10 margin",
- size=11, anchor="middle", fill="#7b8794"))
+ body.append(
+ line(px(b), top, px(b), h - bot, stroke="#9aa5b1", width=1.0, dash="4 3")
+ )
+ body.append(
+ text(
+ px(band),
+ h - bot + 38,
+ "historical rule: bootstrap CI lower bound > +0.10 margin",
+ size=11,
+ anchor="middle",
+ fill="#7b8794",
+ )
+ )
cases = DATA["promotion_gate_cases"]
rows_y = [top + 40, top + 110]
@@ -101,11 +147,13 @@ def px(delta: float) -> float:
for cx in (lo, hi):
body.append(line(cx, cy - 7, cx, cy + 7, stroke=color, width=3.0))
body.append(f"")
- label = "Representative (stubbed harness)" if "representative" in case["provenance"] else "Live smoke (small-n)"
- body.append(text(left, cy - 14, label, size=13, weight="bold"))
- detail = (
- f"delta {case['paired_delta']:+.3f}, CI [{case['ci_low']:g}, {case['ci_high']:g}] -> NO-GO"
+ label = (
+ "Representative (stubbed harness)"
+ if "representative" in case["provenance"]
+ else "Live smoke (small-n)"
)
+ body.append(text(left, cy - 14, label, size=13, weight="bold"))
+ detail = f"delta {case['paired_delta']:+.3f}, CI [{case['ci_low']:g}, {case['ci_high']:g}] -> NO-GO"
body.append(text(x1, cy - 14, detail, size=12, anchor="end", fill=color))
return svg(w, h, body)
@@ -124,14 +172,38 @@ def py(val: float) -> float:
return y1 - val * (y1 - y0)
body: list[str] = []
- body.append(text(left, 34, "Capability by task-class (representative data)", size=18, weight="bold"))
- body.append(text(left, 56, "Stubbed harness, representative data. Design behavior, not a live A/B.",
- size=13, fill="#b45309"))
+ body.append(
+ text(
+ left,
+ 34,
+ "Capability by task-class (representative data)",
+ size=18,
+ weight="bold",
+ )
+ )
+ body.append(
+ text(
+ left,
+ 56,
+ "Stubbed harness, representative data. Design behavior, not a live A/B.",
+ size=13,
+ fill="#b45309",
+ )
+ )
for frac in (0.0, 0.25, 0.5, 0.75, 1.0):
yy = py(frac)
body.append(line(x0, yy, x1, yy, stroke=GRID, width=1.0))
- body.append(text(x0 - 8, yy + 4, f"{int(frac * 100)}%", size=12, anchor="end", fill="#52606d"))
+ body.append(
+ text(
+ x0 - 8,
+ yy + 4,
+ f"{int(frac * 100)}%",
+ size=12,
+ anchor="end",
+ fill="#52606d",
+ )
+ )
n = len(classes)
group_w = (x1 - x0) / n
@@ -142,8 +214,26 @@ def py(val: float) -> float:
bc = gx + 3
body.append(rect(bi, py(inc[i]), bar_w, y1 - py(inc[i]), INCUMBENT))
body.append(rect(bc, py(cand[i]), bar_w, y1 - py(cand[i]), CANDIDATE))
- body.append(text(bi + bar_w / 2, py(inc[i]) - 6, f"{int(inc[i] * 100)}", size=11, anchor="middle", fill=INCUMBENT))
- body.append(text(bc + bar_w / 2, py(cand[i]) - 6, f"{int(cand[i] * 100)}", size=11, anchor="middle", fill=CANDIDATE))
+ body.append(
+ text(
+ bi + bar_w / 2,
+ py(inc[i]) - 6,
+ f"{int(inc[i] * 100)}",
+ size=11,
+ anchor="middle",
+ fill=INCUMBENT,
+ )
+ )
+ body.append(
+ text(
+ bc + bar_w / 2,
+ py(cand[i]) - 6,
+ f"{int(cand[i] * 100)}",
+ size=11,
+ anchor="middle",
+ fill=CANDIDATE,
+ )
+ )
body.append(text(gx, y1 + 20, cls, size=11, anchor="middle", fill=INK))
ly = h - 34
@@ -152,8 +242,16 @@ def py(val: float) -> float:
body.append(rect(left + 250, ly - 10, 14, 14, CANDIDATE))
body.append(text(left + 270, ly + 2, cap["candidate"]["label"], size=12))
summary = cap["capability_summary"]
- body.append(text(x1, 80, f"overall delta +{summary['overall_delta']:g}; no recommendation",
- size=12, anchor="end", fill="#b45309"))
+ body.append(
+ text(
+ x1,
+ 80,
+ f"overall delta +{summary['overall_delta']:g}; no recommendation",
+ size=12,
+ anchor="end",
+ fill="#b45309",
+ )
+ )
return svg(w, h, body)
diff --git a/ruff.toml b/ruff.toml
new file mode 100644
index 0000000..a697bac
--- /dev/null
+++ b/ruff.toml
@@ -0,0 +1,12 @@
+# Ruff configuration for eval-lab-methodology.
+# Adopted 2026-07-17 as part of the agent-framework standardization.
+#
+# The published core in src/eval_lab_methodology/ is vendored BYTE-FOR-BYTE by downstream
+# consumers (content-hash pinned) and includes the frozen identity_domain.py conformance
+# source of record. It must stay byte-stable — exclude it entirely from ruff so an
+# incidental reformat can never break downstream vendored-core parity.
+extend-exclude = ["src/eval_lab_methodology"]
+
+[lint.per-file-ignores]
+# Test modules set PYTHONPATH/imports before importing the unit under test.
+"tests/**" = ["E402"]
diff --git a/scripts/render_methodology_report.py b/scripts/render_methodology_report.py
index 618a0b6..6facafb 100644
--- a/scripts/render_methodology_report.py
+++ b/scripts/render_methodology_report.py
@@ -83,11 +83,17 @@ def load_and_validate(evidence_path: Path) -> dict[str, Any]:
validate_evidence_report(document)
sanitization = document["manifest"].get("sanitization", {})
if sanitization.get("status") != "public-safe":
- raise SystemExit("Refusing to render: manifest.sanitization.status must be public-safe")
+ raise SystemExit(
+ "Refusing to render: manifest.sanitization.status must be public-safe"
+ )
if sanitization.get("checked") is not True:
- raise SystemExit("Refusing to render: manifest.sanitization.checked must be true")
+ raise SystemExit(
+ "Refusing to render: manifest.sanitization.checked must be true"
+ )
if sanitization.get("redactions_required") is True:
- raise SystemExit("Refusing to render: manifest.sanitization.redactions_required must be false")
+ raise SystemExit(
+ "Refusing to render: manifest.sanitization.redactions_required must be false"
+ )
decision = document["report"]["decision"]
enhanced = document["report"]["statistics"]["enhanced_estimators"]
@@ -103,7 +109,10 @@ def load_and_validate(evidence_path: Path) -> dict[str, Any]:
enhanced["wilcoxon_signed_rank"].get("label"),
"enhanced:wilcoxon-signed-rank",
),
- "enhanced.glmm.label": (enhanced["glmm"].get("label"), "enhanced:glmm-logistic"),
+ "enhanced.glmm.label": (
+ enhanced["glmm"].get("label"),
+ "enhanced:glmm-logistic",
+ ),
"sign_test.alternative": (sign.get("alternative"), "two-sided"),
"sign_test.reported_only": (sign.get("reported_only"), True),
}
@@ -133,11 +142,17 @@ def main() -> int:
quarto = os.environ.get("QUARTO", "quarto")
if shutil.which(quarto) is None:
- raise SystemExit("quarto executable not found; install Quarto or set QUARTO=/path/to/quarto")
+ raise SystemExit(
+ "quarto executable not found; install Quarto or set QUARTO=/path/to/quarto"
+ )
output_dir.mkdir(parents=True, exist_ok=True)
env = os.environ.copy()
- env["PYTHONPATH"] = f"{SRC_PATH}{os.pathsep}{env['PYTHONPATH']}" if env.get("PYTHONPATH") else str(SRC_PATH)
+ env["PYTHONPATH"] = (
+ f"{SRC_PATH}{os.pathsep}{env['PYTHONPATH']}"
+ if env.get("PYTHONPATH")
+ else str(SRC_PATH)
+ )
# Render as a single document (no --output-dir: with the website project's
# _quarto.yml at the repo root, Quarto rejects --output-dir for documents
# that are not project render targets). The output lands next to the .qmd
@@ -155,7 +170,9 @@ def main() -> int:
rendered_html = report_path.with_suffix(".html")
if not rendered_html.exists():
- raise SystemExit(f"Quarto succeeded but no output found at {display_path(rendered_html)}")
+ raise SystemExit(
+ f"Quarto succeeded but no output found at {display_path(rendered_html)}"
+ )
resources = report_path.parent / f"{report_path.stem}_files"
for source in (rendered_html, resources):
if not source.exists():
diff --git a/tests/test_contract_v2_delivery_manifest.py b/tests/test_contract_v2_delivery_manifest.py
index 4b6df1e..64b1bde 100644
--- a/tests/test_contract_v2_delivery_manifest.py
+++ b/tests/test_contract_v2_delivery_manifest.py
@@ -109,9 +109,7 @@ def assert_runtime_tree_matches(
for path in root.rglob("*")
if path.is_file() or path.is_symlink()
}
- expected_paths = {
- entry["path"] for entry in self.manifest["runtime_files"]
- }
+ expected_paths = {entry["path"] for entry in self.manifest["runtime_files"]}
self.assertEqual(actual_paths, expected_paths)
def test_manifest_is_canonical_exact_keyed_and_public_safe(self) -> None:
@@ -171,9 +169,7 @@ def test_runtime_layout_entrypoints_and_dependency_are_explicit(self) -> None:
"import_root": ".",
"layout": "preserve-relative-paths",
"python": ">=3.11",
- "render_entrypoint": (
- "analysis.contract_v2.report:render_markdown"
- ),
+ "render_entrypoint": ("analysis.contract_v2.report:render_markdown"),
"verify_entrypoint": "analysis.contract_v2:verify_exact_bytes",
},
)
diff --git a/tests/test_dependence.py b/tests/test_dependence.py
index b29f76b..c577d34 100644
--- a/tests/test_dependence.py
+++ b/tests/test_dependence.py
@@ -142,8 +142,12 @@ def test_values_and_length(self) -> None:
class SizeDistortionTests(unittest.TestCase):
def test_same_seed_gives_identical_result(self) -> None:
layout = SessionLayout(sessions=2, pairs_per_session=20)
- kwargs = dict(rho=0.05, pi_d=0.3, rule=lambda d: sum(d) >= 6, n_sims=500, seed=11)
- self.assertEqual(size_distortion(layout, **kwargs), size_distortion(layout, **kwargs))
+ kwargs = dict(
+ rho=0.05, pi_d=0.3, rule=lambda d: sum(d) >= 6, n_sims=500, seed=11
+ )
+ self.assertEqual(
+ size_distortion(layout, **kwargs), size_distortion(layout, **kwargs)
+ )
def test_result_fields_are_consistent(self) -> None:
layout = SessionLayout(sessions=2, pairs_per_session=20)
@@ -242,7 +246,9 @@ def test_transformed_cells_stay_an_exact_simplex(self) -> None:
self.assertEqual(sum(cells), Fraction(1))
def test_zero_floor_is_the_identity_transform(self) -> None:
- self.assertEqual(floor_transformed_cells(self.BASE_CELLS, Fraction(0)), self.BASE_CELLS)
+ self.assertEqual(
+ floor_transformed_cells(self.BASE_CELLS, Fraction(0)), self.BASE_CELLS
+ )
def test_hand_computed_cell_at_one_twentieth(self) -> None:
# p11' = (19/20)^2 * 2/5 + (19/20)(1/20) * 3/10 + (1/20)^2 * 3/10 = 47/125.
diff --git a/tests/test_enhanced_estimators.py b/tests/test_enhanced_estimators.py
index c905656..430ab49 100644
--- a/tests/test_enhanced_estimators.py
+++ b/tests/test_enhanced_estimators.py
@@ -68,7 +68,7 @@ def test_power_simulation_returns_planned_n(self) -> None:
def test_glmm_wrapper_has_clean_optional_dependency_boundary(self) -> None:
records = [
{"success": 1, "model": "candidate", "task": "task-1"},
- {"success": 0, "model": "incumbent", "task": "task-1"}
+ {"success": 0, "model": "incumbent", "task": "task-1"},
]
if importlib.util.find_spec("statsmodels") is None:
with self.assertRaises(OptionalDependencyError):
diff --git a/tests/test_experimental_note_publication.py b/tests/test_experimental_note_publication.py
index 98cfc9d..fe357ec 100644
--- a/tests/test_experimental_note_publication.py
+++ b/tests/test_experimental_note_publication.py
@@ -48,7 +48,9 @@ def test_historical_gate_is_labeled_historical(self) -> None:
plan = (ROOT / "PLAN.md").read_text(encoding="utf-8")
index = (ROOT / "index.qmd").read_text(encoding="utf-8")
self.assertIn("historical superiority helper", readme)
- self.assertIn("lower bound is strictly greater than the supplied margin", readme)
+ self.assertIn(
+ "lower bound is strictly greater than the supplied margin", readme
+ )
self.assertIn("historical implementation truth", plan)
self.assertIn("not tagged or published to a package registry", index)
@@ -67,7 +69,9 @@ def test_public_status_refuses_unearned_readiness_claims(self) -> None:
self.assertIn("no defensible powered real-model result", readme)
# Both conditions must survive rewording: adequate power AND scientific
# soundness. A well-powered but confounded comparison is still not evidence.
- self.assertIn("No scientifically sound, adequately powered real-model comparison", index)
+ self.assertIn(
+ "No scientifically sound, adequately powered real-model comparison", index
+ )
self.assertIn("necessary, not sufficient", identity)
self.assertIn("does not prove numerical equivalence", errata)
@@ -76,7 +80,9 @@ def test_public_status_refuses_unearned_readiness_claims(self) -> None:
self.assertNotIn("Calibrated small-n statistics", public_overviews)
self.assertNotIn("everything that can change the numerics", public_overviews)
- def test_historical_examples_use_the_implemented_rule_without_model_claims(self) -> None:
+ def test_historical_examples_use_the_implemented_rule_without_model_claims(
+ self,
+ ) -> None:
example = (ROOT / "evidence" / "false-positive-representative.md").read_text(
encoding="utf-8"
)
diff --git a/tests/test_identity_domain.py b/tests/test_identity_domain.py
index 73fee8d..a95d249 100644
--- a/tests/test_identity_domain.py
+++ b/tests/test_identity_domain.py
@@ -53,7 +53,9 @@ def test_non_ascii_changes_bytes_deterministically(self) -> None:
identity_domain_sha256(accented),
CONFORMANCE_IDENTITY_DOMAIN_SHA256,
)
- self.assertEqual(identity_domain_sha256(accented), identity_domain_sha256(accented))
+ self.assertEqual(
+ identity_domain_sha256(accented), identity_domain_sha256(accented)
+ )
def test_pattern_requires_prefixed_lowercase_64_hex(self) -> None:
digest = "0" * 64
@@ -78,7 +80,9 @@ def test_rejects_missing_section(self) -> None:
def test_rejects_missing_launch_key(self) -> None:
invalid = copy.deepcopy(CONFORMANCE_IDENTITY_DOMAIN)
invalid["launch"].pop("batch_invariant")
- self.assertIn("$.launch.batch_invariant is required", validate_identity_domain(invalid))
+ self.assertIn(
+ "$.launch.batch_invariant is required", validate_identity_domain(invalid)
+ )
def test_rejects_run_id_at_top_level(self) -> None:
invalid = copy.deepcopy(CONFORMANCE_IDENTITY_DOMAIN)
@@ -113,7 +117,9 @@ def test_conformance_vector_is_json_serializable_and_synthetic(self) -> None:
payload = json.loads(canonical_json(CONFORMANCE_IDENTITY_DOMAIN))
self.assertEqual(payload["schema_version"], IDENTITY_DOMAIN_SCHEMA_VERSION)
self.assertEqual(payload["model"]["repo"], "example-org/synthetic-model")
- self.assertEqual(payload["launch"]["activation_dtype"], payload["launch"]["dtype"])
+ self.assertEqual(
+ payload["launch"]["activation_dtype"], payload["launch"]["dtype"]
+ )
if __name__ == "__main__":
diff --git a/tests/test_method_tranche_results.py b/tests/test_method_tranche_results.py
index de1a094..b37bbbd 100644
--- a/tests/test_method_tranche_results.py
+++ b/tests/test_method_tranche_results.py
@@ -93,9 +93,7 @@ def test_citations_are_name_and_sha256_only(self) -> None:
self.assertEqual(sorted(record), ["name", "sha256"])
def test_all_scenario_families_present(self) -> None:
- self.assertEqual(
- set(self.committed["scenarios"]), EXPECTED_SCENARIO_FAMILIES
- )
+ self.assertEqual(set(self.committed["scenarios"]), EXPECTED_SCENARIO_FAMILIES)
def test_check_mode_accepts_matching_bytes_without_writing(self) -> None:
with tempfile.TemporaryDirectory() as directory:
diff --git a/tests/test_paired_trinomial.py b/tests/test_paired_trinomial.py
index 516d95f..8c97099 100644
--- a/tests/test_paired_trinomial.py
+++ b/tests/test_paired_trinomial.py
@@ -197,7 +197,12 @@ def test_tail_probability_monotone_nonincreasing_in_critical(self) -> None:
def test_critical_value_minimality(self) -> None:
dist = sum_distribution(12, Fraction(2, 5), Fraction(1, 5))
- for alpha in (Fraction(1, 100), Fraction(1, 20), Fraction(1, 10), Fraction(1, 4)):
+ for alpha in (
+ Fraction(1, 100),
+ Fraction(1, 20),
+ Fraction(1, 10),
+ Fraction(1, 4),
+ ):
c = critical_value(dist, alpha)
self.assertLessEqual(tail_probability(dist, c), alpha)
self.assertGreater(tail_probability(dist, c - 1), alpha)
@@ -292,7 +297,9 @@ def test_derived_operating_points_at_n40(self) -> None:
calibration = lfc_calibrate(40, Fraction(1, 10), ALPHA)
self.assertEqual(calibration.critical_value, 15)
self.assertLessEqual(calibration.sup_size, ALPHA)
- self.assertAlmostEqual(float(calibration.sup_size), 0.04325358925091875, places=14)
+ self.assertAlmostEqual(
+ float(calibration.sup_size), 0.04325358925091875, places=14
+ )
# The grid supremum lands adjacent to — not exactly at — maximum
# discordance: at pi_d = 1 the support of S is even-parity only, so an
# odd-parity boundary point just inside pi_d = 1 carries the supremum.
@@ -300,9 +307,13 @@ def test_derived_operating_points_at_n40(self) -> None:
self.assertEqual(calibration.lfc_pi_d, Fraction(97, 100))
superiority = lfc_calibrate(40, Fraction(0), ALPHA)
self.assertEqual(superiority.critical_value, 11)
- self.assertAlmostEqual(float(superiority.sup_size), 0.045117717694842636, places=14)
+ self.assertAlmostEqual(
+ float(superiority.sup_size), 0.045117717694842636, places=14
+ )
- def test_delta0_zero_matches_hand_computed_binomial_critical_at_pi_d_one(self) -> None:
+ def test_delta0_zero_matches_hand_computed_binomial_critical_at_pi_d_one(
+ self,
+ ) -> None:
# At delta0 = 0 and pi_d = 1 every task is discordant, so
# S = 2X - n with X ~ Binomial(n, 1/2): the calibrated critical value
# must equal the binomial sign-test critical computed by hand.
@@ -310,9 +321,7 @@ def test_delta0_zero_matches_hand_computed_binomial_critical_at_pi_d_one(self) -
expected = None
for c in range(-n, n + 2):
k_min = max(0, -((c + n) // -2)) # ceil((c + n) / 2), integer-exact
- tail = Fraction(
- sum(math.comb(n, x) for x in range(k_min, n + 1)), 2**n
- )
+ tail = Fraction(sum(math.comb(n, x) for x in range(k_min, n + 1)), 2**n)
if tail <= ALPHA:
expected = c
break
@@ -437,7 +446,9 @@ def test_pvalue_at_delta0_zero_reduces_to_binomial_tail(self) -> None:
n_plus, n_zero, n_minus = 7, 3, 2
m = n_plus + n_minus
expected = Fraction(sum(math.comb(m, k) for k in range(n_plus, m + 1)), 2**m)
- self.assertEqual(signflip_pvalue(n_plus, n_zero, n_minus, Fraction(0)), expected)
+ self.assertEqual(
+ signflip_pvalue(n_plus, n_zero, n_minus, Fraction(0)), expected
+ )
def test_rejection_probability_matches_full_outcome_enumeration(self) -> None:
# Every task has positive discordance probability, so the task count
@@ -481,9 +492,7 @@ def test_input_validation(self) -> None:
with self.assertRaises(TypeError):
signflip_pvalue(1, 1, 1, 0.1)
with self.assertRaises(ValueError):
- signflip_rejection_probability(
- {}, n=1, delta0=Fraction(0), alpha=ALPHA
- )
+ signflip_rejection_probability({}, n=1, delta0=Fraction(0), alpha=ALPHA)
with self.assertRaises(ValueError):
signflip_rejection_probability(
{(0, 0): Fraction(1)}, n=0, delta0=Fraction(0), alpha=ALPHA
diff --git a/tests/test_primitives.py b/tests/test_primitives.py
index 9413e59..f36febd 100644
--- a/tests/test_primitives.py
+++ b/tests/test_primitives.py
@@ -2,13 +2,20 @@
import unittest
-from eval_lab_methodology import bootstrap_ci, sign_test, superiority_by_margin, wilson_interval
+from eval_lab_methodology import (
+ bootstrap_ci,
+ sign_test,
+ superiority_by_margin,
+ wilson_interval,
+)
class PrimitiveParityTests(unittest.TestCase):
def test_wilson_interval_matches_reference_outputs(self) -> None:
self.assertEqual(wilson_interval(0, 0), (0.0, 0.0))
- self.assertEqual(wilson_interval(5, 10), (0.23658959361548731, 0.7634104063845126))
+ self.assertEqual(
+ wilson_interval(5, 10), (0.23658959361548731, 0.7634104063845126)
+ )
self.assertEqual(wilson_interval(10, 10), (0.7224598312333834, 1.0))
self.assertEqual(wilson_interval(0, 10), (0.0, 0.2775401687666166))
diff --git a/tests/test_schema_and_version.py b/tests/test_schema_and_version.py
index f313992..7b57282 100644
--- a/tests/test_schema_and_version.py
+++ b/tests/test_schema_and_version.py
@@ -36,11 +36,19 @@ def test_core_markers_are_queryable_and_stable(self) -> None:
self.assertRegex(__core_content_hash__, r"^sha256:[0-9a-f]{64}$")
def test_schema_is_versioned_and_sample_validates(self) -> None:
- schema = json.loads((ROOT / "evidence" / "schema.json").read_text(encoding="utf-8"))
- sample = json.loads((ROOT / "evidence" / "sample-lab-report.json").read_text(encoding="utf-8"))
+ schema = json.loads(
+ (ROOT / "evidence" / "schema.json").read_text(encoding="utf-8")
+ )
+ sample = json.loads(
+ (ROOT / "evidence" / "sample-lab-report.json").read_text(encoding="utf-8")
+ )
- self.assertEqual(schema["$id"], "urn:agentic-coding-evaluation-lab:evidence:1.1.0")
- self.assertEqual(schema["properties"]["schema_version"]["const"], EVIDENCE_SCHEMA_VERSION)
+ self.assertEqual(
+ schema["$id"], "urn:agentic-coding-evaluation-lab:evidence:1.1.0"
+ )
+ self.assertEqual(
+ schema["properties"]["schema_version"]["const"], EVIDENCE_SCHEMA_VERSION
+ )
self.assertEqual(sample["schema_version"], EVIDENCE_SCHEMA_VERSION)
self.assertEqual(sample["core"]["core_version"], __core_version__)
self.assertEqual(sample["core"]["core_content_hash"], __core_content_hash__)
@@ -59,7 +67,9 @@ def test_schema_is_versioned_and_sample_validates(self) -> None:
validate_evidence_report(sample)
def test_validator_fails_closed_on_missing_raw_outcomes(self) -> None:
- sample = json.loads((ROOT / "evidence" / "sample-lab-report.json").read_text(encoding="utf-8"))
+ sample = json.loads(
+ (ROOT / "evidence" / "sample-lab-report.json").read_text(encoding="utf-8")
+ )
invalid = copy.deepcopy(sample)
invalid.pop("raw_outcomes")
with self.assertRaises(EvidenceValidationError):
@@ -68,19 +78,26 @@ def test_validator_fails_closed_on_missing_raw_outcomes(self) -> None:
def test_validator_fails_closed_on_empty_identity_domain(self) -> None:
invalid = self._sample()
invalid["manifest"]["identity_domain"] = {}
- with self.assertRaisesRegex(EvidenceValidationError, "identity_domain.status is required"):
+ with self.assertRaisesRegex(
+ EvidenceValidationError, "identity_domain.status is required"
+ ):
validate_evidence_report(invalid)
def test_validator_fails_closed_on_empty_core_and_report_objects(self) -> None:
for key in ("core", "report", "manifest"):
invalid = self._sample()
invalid[key] = {}
- with self.assertRaises(EvidenceValidationError, msg=f"empty {key} must not validate"):
+ with self.assertRaises(
+ EvidenceValidationError, msg=f"empty {key} must not validate"
+ ):
validate_evidence_report(invalid)
def test_validator_rejects_matched_with_mismatched_domain_hashes(self) -> None:
other = "sha256:" + "9" * 64
- for key in ("candidate_identity_domain_sha256", "incumbent_identity_domain_sha256"):
+ for key in (
+ "candidate_identity_domain_sha256",
+ "incumbent_identity_domain_sha256",
+ ):
invalid = self._sample()
invalid["manifest"]["identity_domain"][key] = other
with self.assertRaisesRegex(
@@ -92,14 +109,18 @@ def test_validator_rejects_matched_with_mismatched_domain_hashes(self) -> None:
def test_validator_rejects_matched_hash_missing_from_observed(self) -> None:
other = "sha256:" + "9" * 64
invalid = self._sample()
- invalid["manifest"]["identity_domain"]["observed_identity_domain_sha256s"] = [other]
+ invalid["manifest"]["identity_domain"]["observed_identity_domain_sha256s"] = [
+ other
+ ]
with self.assertRaisesRegex(
EvidenceValidationError,
"must appear in observed_identity_domain_sha256s when status is matched",
):
validate_evidence_report(invalid)
- def test_validator_accepts_bridge_authorized_and_rejects_empty_bridge_fields(self) -> None:
+ def test_validator_accepts_bridge_authorized_and_rejects_empty_bridge_fields(
+ self,
+ ) -> None:
document = self._bridge_authorized_sample()
validate_evidence_report(document)
@@ -125,17 +146,25 @@ def test_validator_rejects_bridge_authorized_with_null_bridge(self) -> None:
jsonschema is None and os.environ.get("CI", "").lower() != "true",
"jsonschema is not installed outside CI",
)
- def test_schema_json_enforces_status_conditional_identity_domain_rules(self) -> None:
+ def test_schema_json_enforces_status_conditional_identity_domain_rules(
+ self,
+ ) -> None:
if jsonschema is None:
- self.fail("jsonschema must be installed in CI for schema conditional-rule coverage")
- schema = json.loads((ROOT / "evidence" / "schema.json").read_text(encoding="utf-8"))
+ self.fail(
+ "jsonschema must be installed in CI for schema conditional-rule coverage"
+ )
+ schema = json.loads(
+ (ROOT / "evidence" / "schema.json").read_text(encoding="utf-8")
+ )
validator = jsonschema.Draft202012Validator(schema)
validator.validate(self._sample())
validator.validate(self._bridge_authorized_sample())
matched_without_hash = self._sample()
- matched_without_hash["manifest"]["identity_domain"]["identity_domain_sha256"] = None
+ matched_without_hash["manifest"]["identity_domain"][
+ "identity_domain_sha256"
+ ] = None
matched_with_bridge = self._sample()
matched_with_bridge["manifest"]["identity_domain"]["bridge_authorization"] = {
@@ -144,18 +173,25 @@ def test_schema_json_enforces_status_conditional_identity_domain_rules(self) ->
}
bridge_without_authorization = self._bridge_authorized_sample()
- bridge_without_authorization["manifest"]["identity_domain"]["bridge_authorization"] = None
+ bridge_without_authorization["manifest"]["identity_domain"][
+ "bridge_authorization"
+ ] = None
for name, invalid in (
("matched with null identity_domain_sha256", matched_without_hash),
("matched with non-null bridge_authorization", matched_with_bridge),
- ("bridge-authorized with null bridge_authorization", bridge_without_authorization),
+ (
+ "bridge-authorized with null bridge_authorization",
+ bridge_without_authorization,
+ ),
):
with self.assertRaises(jsonschema.ValidationError, msg=name):
validator.validate(invalid)
def _sample(self) -> dict:
- return json.loads((ROOT / "evidence" / "sample-lab-report.json").read_text(encoding="utf-8"))
+ return json.loads(
+ (ROOT / "evidence" / "sample-lab-report.json").read_text(encoding="utf-8")
+ )
def _bridge_authorized_sample(self) -> dict:
document = self._sample()
@@ -175,7 +211,9 @@ def _bridge_authorized_sample(self) -> dict:
return document
def test_sample_report_numbers_recompute_from_raw_outcomes(self) -> None:
- sample = json.loads((ROOT / "evidence" / "sample-lab-report.json").read_text(encoding="utf-8"))
+ sample = json.loads(
+ (ROOT / "evidence" / "sample-lab-report.json").read_text(encoding="utf-8")
+ )
raw = sample["raw_outcomes"]
incumbent_successes = [
@@ -195,26 +233,43 @@ def test_sample_report_numbers_recompute_from_raw_outcomes(self) -> None:
]
for task in raw
}
- task_mean_deltas = [sum(values) / len(values) for values in task_deltas.values()]
+ task_mean_deltas = [
+ sum(values) / len(values) for values in task_deltas.values()
+ ]
stats = sample["report"]["statistics"]
incumbent = stats["capability"]["incumbent"]
candidate = stats["capability"]["candidate"]
self.assertEqual(incumbent["successes"], sum(incumbent_successes))
self.assertEqual(candidate["successes"], sum(candidate_successes))
- self.assertEqual(incumbent["wilson"]["low"], wilson_interval(sum(incumbent_successes), 6)[0])
- self.assertEqual(candidate["wilson"]["high"], wilson_interval(sum(candidate_successes), 6)[1])
+ self.assertEqual(
+ incumbent["wilson"]["low"], wilson_interval(sum(incumbent_successes), 6)[0]
+ )
+ self.assertEqual(
+ candidate["wilson"]["high"], wilson_interval(sum(candidate_successes), 6)[1]
+ )
single_stage_ci = bootstrap_ci(
[delta for values in task_deltas.values() for delta in values],
seed=12345,
)
- self.assertLessEqual(single_stage_ci[0], stats["paired_delta"]["point_estimate"])
+ self.assertLessEqual(
+ single_stage_ci[0], stats["paired_delta"]["point_estimate"]
+ )
two_stage = two_stage_bootstrap(task_deltas, iterations=2000, seed=12345)
- self.assertEqual(two_stage.estimate, stats["enhanced_estimators"]["two_stage_bootstrap"]["estimate"])
- self.assertEqual(two_stage.ci_low, stats["enhanced_estimators"]["two_stage_bootstrap"]["ci"]["low"])
- self.assertEqual(two_stage.ci_high, stats["enhanced_estimators"]["two_stage_bootstrap"]["ci"]["high"])
+ self.assertEqual(
+ two_stage.estimate,
+ stats["enhanced_estimators"]["two_stage_bootstrap"]["estimate"],
+ )
+ self.assertEqual(
+ two_stage.ci_low,
+ stats["enhanced_estimators"]["two_stage_bootstrap"]["ci"]["low"],
+ )
+ self.assertEqual(
+ two_stage.ci_high,
+ stats["enhanced_estimators"]["two_stage_bootstrap"]["ci"]["high"],
+ )
wins, losses, ties, p_value = sign_test(task_mean_deltas)
decision = sample["report"]["decision"]
@@ -222,8 +277,14 @@ def test_sample_report_numbers_recompute_from_raw_outcomes(self) -> None:
self.assertTrue(decision["sign_test"]["reported_only"])
wilcoxon = wilcoxon_signed_rank(task_mean_deltas)
- self.assertEqual(wilcoxon.statistic, stats["enhanced_estimators"]["wilcoxon_signed_rank"]["statistic"])
- self.assertEqual(wilcoxon.p_value, stats["enhanced_estimators"]["wilcoxon_signed_rank"]["p_value"])
+ self.assertEqual(
+ wilcoxon.statistic,
+ stats["enhanced_estimators"]["wilcoxon_signed_rank"]["statistic"],
+ )
+ self.assertEqual(
+ wilcoxon.p_value,
+ stats["enhanced_estimators"]["wilcoxon_signed_rank"]["p_value"],
+ )
gate = superiority_by_margin(
stats["paired_delta"]["point_estimate"],