diff --git a/docs/SESSION_HANDOFF_iter17_phase2_complete.md b/docs/SESSION_HANDOFF_iter17_phase2_complete.md new file mode 100644 index 0000000..445bf56 --- /dev/null +++ b/docs/SESSION_HANDOFF_iter17_phase2_complete.md @@ -0,0 +1,258 @@ +# Session Handoff: iter17 — Phase 2 Complete, Paused Before Phase 3 + +**Branch:** `iter17-ab-regression-audit` (off `main`, 17 commits ahead) +**Last session ended:** 2026-04-11 +**Next session starts at:** Phase 3 (annotation runs, ~$45 LLM budget) + +--- + +## TL;DR + +iter17 is an A/B regression audit gating iter18's ~$400-600 90k annotation run on proof that iter16's longer `SYSTEM_PROMPT` did not silently degrade any of the 40 pre-existing detection labels. The plan has 18 tasks across 6 phases. Phases 0-2 (11 code tasks) are complete, tested, and committed. Phases 3-5 (annotation runs + report + gate decision) remain and need real LLM spend + human judgment. + +**To resume:** read this doc, verify the branch is checked out, then start Task 3.1 (git worktree setup) per the implementation plan at `docs/superpowers/plans/2026-04-10-iter17-ab-regression-audit.md`. + +--- + +## What's Done (Phases 0-2, 11 code tasks) + +### Phase 0 — Operational Fixes ✅ + +| Task | Commit | What | +|---|---|---| +| 0.1 | `7eafced` | Added `if __name__ == "__main__": main()` guard to `training/trainr/core/annotate_detections.py`. `python -m` invocation now works as a backup diagnostic path. | +| 0.2 | `497588c` | Tightened `INJECTION_PATTERNS["stack_trace"]` in `build_audit_sample.py`. Python requires Traceback+File adjacency; Java requires Exception-header+frame adjacency; Rust uses `(?m)^` line anchor to exclude `// error-pattern:` test directives. Polars-compatible (uses `[\s\S]{0,N}` instead of `(?s:.)`, `(?m)^` instead of lookbehind). Does NOT rebuild `iter16_5k_input.parquet` — reusing existing file is load-bearing for apples-to-apples A/B. | + +### Phase 1 — Audit Helpers ✅ + +| Task | Commit(s) | What | +|---|---|---| +| 1.1 | `8f1ef7d`, `81adffc` | `compute_prevalence_per_label(dfs: dict[str, pl.DataFrame]) -> dict[str, float]` — majority-of-N fire rate per label on stratified rows. Filters internally (callers pass raw frames). Follow-up commit added schema-divergence coverage. | +| 1.2 | `99159b4`, `0d67183` | `load_annotator_parquets(paths: list[Path]) -> dict[str, pl.DataFrame]` with `EXPECTED_MODEL_SLUGS = frozenset({"gemini3flash", "sonnet", "gpt54mini"})` constant. Parses model slug from filename, asserts slug set equality, rejects wrong count / duplicates / unknown slugs. Follow-up commit added true-duplicate slug path coverage. | +| 1.3 | `e71e16c` | Byte-for-byte reproducibility smoke test for the iter16 audit report. Regenerates the committed report from archived parquets and fails loud on any drift. **Confirms Phase 1 made no regressions to existing audit output.** | + +### Phase 2 — `compare_prompt_versions.py` Module ✅ + +| Task | Commit | What | +|---|---|---| +| 2.1 | `2b78826` | New module `training/trainr/core/compare_prompt_versions.py` with `LabelCategory`, `LabelVerdict`, `LabelRow`, `DeltaReport` dataclasses. `compare_prompt_versions()` function handles the happy-path case. `_compute_prev_ratio()` implements zero-handling: `0/0 → 1.0`, `0/>0 → inf`, `>0/0 → 0.0`. | +| 2.2 | `37383a8` | Dynamic `det_*` column introspection → `shared / iter15-only / iter16-only` categorization. Hard schema assertion between `after_frames` and `noise_floor_frames` (noise floor correctness depends on identical column sets). Partial-metric `LabelRow` entries for asymmetric labels. | +| 2.3 | `445751c` | `_compute_verdict()` pure helper combining hard agreement gate (`|Δagr| > 0.005`) with soft prevalence gate (`prev_ratio ∉ [0.5, 2.0]`). Verdicts can co-occur (`FAIL_AND_WARN`). Override eligibility is reported but NOT applied in the verdict — human review applies it per the Gate Decision Protocol. | +| 2.4 | `0846735` | `_assert_fingerprint_matches_input()` helper — hashes all non-`det_*` columns row-by-row against the input parquet. Fingerprint validation runs FIRST in `compare_prompt_versions()` body, before the schema assertion and before metric computation. Error messages name the diverging row index, column name, and source slug (`iter15/`, `iter16a/`, or `iter16b/`). | +| 2.5 | `739757f` | `format_delta_report(report) -> str` — pure markdown formatter rendering all 8 sections from the spec's §Report Schema (header, gate verdict summary, shared labels table, iter15-only section, iter16-only section, FAIL-agreement detail with override-eligibility hint, WARN-prevalence detail, noise floor table). Three formatting helpers: `_fmt_float` (4 dp), `_fmt_signed` (signed 4 dp), `_fmt_ratio` (3 dp). `None` renders as `—`. | +| 2.6 | `62cbd6d` | `trainr data compare-prompts` CLI wire-up. Module `main()` parses argv (argparse), expands globs via Python's `glob` module (NOT shell), calls `load_annotator_parquets` with slug validation, runs the comparison, writes the report, exits `2` on any FAIL-agreement (report is written BEFORE exit). Click wire-up in `training/trainr/commands/data.py`. | + +### Test Coverage + +**24 tests** in `training/tests/test_compare_prompt_versions.py`: + +| Class | Count | Coverage | +|---|---:|---| +| `TestHappyPath` | 1 | All-shared, all-PASS baseline | +| `TestColumnCategorization` | 4 | iter15-only, iter16-only, mixed asymmetry, schema mismatch raise | +| `TestVerdictLogic` | 7 | PASS, FAIL_AGREEMENT, WARN_PREVALENCE (low + high), FAIL_AND_WARN co-occur, 0/0 no-warn, 0/>0 warn | +| `TestRowAlignmentFingerprint` | 4 | Passing, text-mutation-with-row-index, row-count-mismatch, missing-non-det-column | +| `TestFormatReport` | 3 | All sections present, FAIL count in summary, inf-ratio clean formatting | +| `TestMainCLI` | 3 | End-to-end from parquets, exits 2 on FAIL, raises on empty glob | +| Constants | 2 | `AGREEMENT_DELTA_THRESHOLD == 0.005`, `PREVALENCE_RATIO_LOW == 0.5`, `PREVALENCE_RATIO_HIGH == 2.0` | + +Plus **14 tests** in `training/tests/test_audit_semantic_labels.py` (up from 4 pre-iter17): +- 5 `TestComputePrevalencePerLabel` (with schema-divergence coverage) +- 6 `TestLoadAnnotatorParquets` (with true-duplicate slug coverage) +- 1 `TestIter16ReportReproducibility` (byte-for-byte iter16 audit reproduction) +- 2 original/pre-existing tests + +Plus **6 new tests** in `training/tests/test_build_audit_sample.py` under `TestStackTraceInjectionPatternTightening` (3 anti-cases, 3 positive cases). + +**All tests green.** An end-to-end smoke test of the compare CLI against real iter16 parquets (running iter16 vs iter16 as both sides) produced a valid PASS report with all 42 shared labels and zero deltas — confirming the full code path works against real data. + +### Architecture Summary + +- **Workspace boundary respected**: Python-only changes under `training/`. Zero Rust code touched. +- **Polars-end-to-end**: no pandas introduced. +- **Dictionary-keyed-by-slug convention**: `dict[str, pl.DataFrame]` throughout, matching existing `compute_agreement_across_models`. +- **Pure functions where possible**: `_compute_verdict`, `_compute_prev_ratio`, `format_delta_report`, all formatting helpers. +- **Defensive guards at every boundary**: fingerprint before schema assert before metric compute; raise ValueError with specific row/column/slug context. + +--- + +## What's Next (Phases 3-5, 7 tasks, ~$45 LLM spend) + +### Phase 3 — Annotation Runs (~$45, real money) + +Produces 9 parquet files in `training/data/audit/`. Spec concurrency cap: **≤ 2 annotation jobs in flight at any time, 40 in-flight LLM requests total**. + +**Model ID lookup (do this first in the next session):** +```bash +grep -E 'google/gemini|anthropic/claude|openai/gpt' docs/accuracy_runs/2026-04-10-iteration-16.md +``` + +Expected mapping (verify against iter16 iteration doc): +- `gemini3flash` → `google/gemini-3-flash-preview` +- `sonnet` → `anthropic/claude-sonnet-4.6` +- `gpt54mini` → `openai/gpt-5.4-mini` + +**Task 3.1 — Worktree setup.** Off commit `22bc292` (iter15 prompt state, Phase 0 test repairs applied): +```bash +git worktree add .worktrees/iter17-iter15-prompt 22bc292 +cd .worktrees/iter17-iter15-prompt/training && uv sync && cd - +# Sanity: verify iter15 prompt state +grep -c 'log_content\|stack_trace\|diff_patch' .worktrees/iter17-iter15-prompt/training/trainr/core/annotate_detections.py # expect 0 +grep -c 'log_lines' .worktrees/iter17-iter15-prompt/training/trainr/core/annotate_detections.py # expect >0 +export REPO_ROOT="$(git rev-parse --show-toplevel)" +``` + +**Task 3.2 — iter15 side (3 runs, ~$15).** Run from worktree, one model at a time (concurrency cap): +```bash +cd .worktrees/iter17-iter15-prompt/training +for model_slug_pair in "gemini3flash google/gemini-3-flash-preview" "sonnet anthropic/claude-sonnet-4.6" "gpt54mini openai/gpt-5.4-mini"; do + slug=${model_slug_pair% *} + model=${model_slug_pair#* } + uv run trainr data annotate-detections \ + --input "$REPO_ROOT/training/data/audit/iter16_5k_input.parquet" \ + --model "$model" \ + --backend openrouter \ + --output "$REPO_ROOT/training/data/audit/iter17_ab_iter15_${slug}.parquet" +done +``` + +Verify: 3 parquets, each 5065 rows, each has `det_log_lines` column, none has `det_log_content`. + +**Task 3.3 — iter16a side (3 runs, ~$15).** From main repo. May run concurrently with Task 3.2 in a separate terminal (2 jobs in flight max across both sides). Same loop but change `iter15` → `iter16a` in output paths, run from `"$REPO_ROOT"/training`: +```bash +cd "$REPO_ROOT"/training +for model_slug_pair in "gemini3flash google/gemini-3-flash-preview" "sonnet anthropic/claude-sonnet-4.6" "gpt54mini openai/gpt-5.4-mini"; do + slug=${model_slug_pair% *} + model=${model_slug_pair#* } + uv run trainr data annotate-detections \ + --input data/audit/iter16_5k_input.parquet \ + --model "$model" \ + --backend openrouter \ + --output "data/audit/iter17_ab_iter16a_${slug}.parquet" +done +``` + +Verify: 3 parquets, each 5065 rows, each has `det_log_content`, none has `det_log_lines`. + +**Task 3.4 — iter16b side (3 runs, ~$15).** After iter15 and iter16a complete. Same as 3.3 but output `iter16b_{slug}.parquet`. This is the same-prompt noise floor companion. + +**Sanity after Task 3.4:** iter16a vs iter16b should have slightly different fire patterns (not identical — that would signal a duplicate run or deterministic backend). Spot-check: +```bash +uv run --directory training python -c " +import polars as pl +a = pl.read_parquet('training/data/audit/iter17_ab_iter16a_gemini3flash.parquet') +b = pl.read_parquet('training/data/audit/iter17_ab_iter16b_gemini3flash.parquet') +same = (a['det_python'] == b['det_python']).sum() +print(f'det_python agreement: {same}/{len(a)}') +" +``` +Expected: high but not 100% (typically >95% but <100% due to LLM nondeterminism). + +### Phase 4 — Comparison + Report + +**Task 4.1** — Run `trainr data compare-prompts`: +```bash +cd /home/bfirestone/devspace/personal/sentiolabs/text-classifier-rs +uv run --directory training trainr data compare-prompts \ + --before "$(pwd)/training/data/audit/iter17_ab_iter15_*.parquet" \ + --after "$(pwd)/training/data/audit/iter17_ab_iter16a_*.parquet" \ + --noise-floor "$(pwd)/training/data/audit/iter17_ab_iter16b_*.parquet" \ + --input "$(pwd)/training/data/audit/iter16_5k_input.parquet" \ + --output "$(pwd)/docs/accuracy_runs/2026-04-10-iter17-regression-report.md" +``` +Exit 0 = PASS. Exit 2 = FAIL (at least one FAIL-agreement row; report still written). + +Commit the report: +```bash +git add docs/accuracy_runs/2026-04-10-iter17-regression-report.md +git commit -m "docs(iter17): A/B regression audit report" +``` + +**Task 4.2** — Write iter17 iteration doc at `docs/accuracy_runs/2026-04-10-iteration-17.md` following the template at the bottom of the implementation plan (Task 4.2 section). Key sections to fill in: +- Gate decision: PASS / FAIL / PASS-with-override +- Per-row human review for each FAIL-agreement row (override eligibility: `|Δagr| ≤ 2 × noise_floor`) +- Per-row review for each WARN-prevalence row (semantic judgment on whether drift is legitimate tightening or silent bug) +- Key learnings + +### Phase 5 — Gate Decision + +**Task 5.1** — Final explicit gate decision commit + push: +```bash +# PASS case: +git commit --allow-empty -m "docs(iter17): gate decision — PASS" +# FAIL case (back to Task 3.3 with a revised prompt; each iteration is ~$30: iter16a + iter16b rerun): +git commit --allow-empty -m "docs(iter17): gate decision — FAIL" + +git push +``` + +**FAIL path gotcha** — if the gate fails and a prompt iteration is needed, rerun **both** iter16a AND iter16b against the new prompt (~$30 per iteration). Reusing the cached iter16b from the old prompt would turn the noise floor into a prompt-drift measurement. iter15 parquets stay cached (the iter15 prompt state doesn't change). + +--- + +## Gate Decision Protocol Reference + +From the spec (`docs/superpowers/specs/2026-04-10-iter17-ab-regression-audit-design.md`): + +**Hard gate (blocks iter18):** any shared label with `|Δagr| > 0.005` → FAIL-agreement. Overridable if `|Δagr| ≤ 2 × noise_floor` with documented reviewer sign-off. + +**Soft gate (requires sign-off):** any shared label with `prev_ratio ∉ [0.5, 2.0]` → WARN-prevalence. Needs per-row human review in the iteration doc. + +**Noise floor:** `|agr(iter16a) - agr(iter16b)|` per label. Informational floor for override eligibility; does NOT adjust the 0.005 threshold. + +**iter15-only labels** (`det_log_lines` removed in iter16): reported for context but not gated. + +**iter16-only labels** (`det_log_content`, `det_stack_trace`, `det_diff_patch` added in iter16): reported for context; cross-ref with iter16's own audit report. + +--- + +## Gotchas and Notes for the Next Session + +1. **Pyright diagnostic noise is expected.** The workspace-root Pyright can't resolve `polars` / `pytest` / `trainr.core.*` imports because they live in `training/.venv`. This cascades into false-positive "symbol not accessed" / "not defined" hints on otherwise-working code. Every Task in this session hit these warnings; all were verified false positives by running the actual tests under `uv run --directory training pytest ...`. Ignore unless you see something truly new. + +2. **Test runner convention:** all `pytest` invocations go through `uv run --directory training pytest ...`. Running `pytest` directly from the repo root won't work — polars et al live in `training/.venv`. + +3. **CLI runtime:** `trainr data ` only works from `training/` or with `uv run --directory training trainr ...`. + +4. **10 pre-existing test failures in unrelated modules.** `test_eval_onnx`, `test_pull_real_data`, `test_vote_labels`, `test_voting_pilot` have pre-existing failures that are NOT caused by iter17 changes. They were present before Task 0.1 and shouldn't gate iter17 landing. + +5. **The archived iter16 parquets (`training/data/audit/iter16_5k_*.parquet`) must NOT be used as a noise floor reference.** They were produced before commit `c1ec175` (the CSV-log refinement), so comparing them against fresh iter16 runs measures prompt drift, not same-prompt variance. This was caught by SPEC_REVIEW during brainstorming and is why we use the fresh iter16a + iter16b approach. + +6. **`iter16_5k_input.parquet` is immutable across iter17.** Do NOT rebuild it under any circumstances — doing so would invalidate the apples-to-apples A/B comparison. The injection-regex tightening in Task 0.2 was explicitly designed NOT to rebuild it (benefits are deferred to iter18+ audits). + +7. **Concurrency discipline:** max 2 annotation jobs in flight, 40 in-flight LLM requests total. Running all 9 parquets in parallel would create rate-limit asymmetry and weaken the controlled-experiment claim. + +8. **iter17 doesn't touch retraining or the Rust artifact.** Those are iter19 and iter20 respectively, each on their own branch per the spec's seam decomposition. + +--- + +## Branch State at Session End + +- **Branch:** `iter17-ab-regression-audit` (off `main`, 17 commits ahead) +- **Last commit:** `62cbd6d feat(cli): trainr data compare-prompts` +- **Uncommitted files:** this handoff doc (being committed in the same pause-and-push step) +- **Total tests in compare module:** 24 (all green) +- **Full audit suite:** 75 passed, 1 skipped (no regressions) +- **Annotation parquets for Phase 3:** 0 of 9 produced +- **Regression report:** not yet generated +- **Iteration doc:** not yet written +- **Gate decision:** not yet made + +**Cost spent so far this branch:** $0 (code work only). +**Cost remaining to complete iter17:** ~$45 base + ~$30 per prompt iteration on FAIL. + +--- + +## Plan and Spec References + +- **Spec:** [`docs/superpowers/specs/2026-04-10-iter17-ab-regression-audit-design.md`](superpowers/specs/2026-04-10-iter17-ab-regression-audit-design.md) +- **Plan:** [`docs/superpowers/plans/2026-04-10-iter17-ab-regression-audit.md`](superpowers/plans/2026-04-10-iter17-ab-regression-audit.md) +- **Prior iteration:** [`docs/accuracy_runs/2026-04-10-iteration-16.md`](accuracy_runs/2026-04-10-iteration-16.md) + +To resume implementation in a future session: + +1. `git checkout iter17-ab-regression-audit` +2. Read this handoff doc +3. Confirm OpenRouter account is funded for ~$45 +4. Verify exact model IDs via `grep -E 'google/|anthropic/|openai/' docs/accuracy_runs/2026-04-10-iteration-16.md` +5. Start with Task 3.1 per the plan diff --git a/training/tests/test_audit_semantic_labels.py b/training/tests/test_audit_semantic_labels.py index b0f8a5d..59ecfd6 100644 --- a/training/tests/test_audit_semantic_labels.py +++ b/training/tests/test_audit_semantic_labels.py @@ -1,6 +1,12 @@ """Smoke tests for trainr.core.audit_semantic_labels.""" +import re +from pathlib import Path + import polars as pl +import pytest + +from trainr.core.audit_semantic_labels import compute_prevalence_per_label def _make_fake_annotated( @@ -72,3 +78,196 @@ def test_agreement_excludes_injected_rows(): filtered = filter_for_agreement(df) assert len(filtered) == 2 assert all(s == "stratified" for s in filtered["audit_source"].to_list()) + + +class TestComputePrevalencePerLabel: + def test_majority_fire_rate_single_model(self): + # Single model: prevalence is simply mean fire rate. + df = pl.DataFrame({ + "audit_source": ["stratified"] * 10, + "det_python": [1, 0, 1, 1, 0, 0, 0, 0, 0, 0], + "det_markdown": [0] * 10, + }) + result = compute_prevalence_per_label({"only": df}) + assert result["python"] == 0.3 + assert result["markdown"] == 0.0 + + def test_majority_of_three_fire(self): + # 3 models: prevalence = fraction of rows where majority (>=2 of 3) fires. + base = pl.DataFrame({ + "audit_source": ["stratified"] * 4, + }) + df1 = base.with_columns(pl.Series("det_python", [1, 1, 0, 0])) + df2 = base.with_columns(pl.Series("det_python", [1, 0, 1, 0])) + df3 = base.with_columns(pl.Series("det_python", [0, 1, 1, 0])) + # Row-by-row: 2+, 2+, 2+, 0 → majority fires on rows 0, 1, 2 → prev = 3/4 + result = compute_prevalence_per_label({"a": df1, "b": df2, "c": df3}) + assert result["python"] == 0.75 + + def test_zero_rows_returns_zero(self): + df = pl.DataFrame({ + "audit_source": pl.Series([], dtype=pl.Utf8), + "det_python": pl.Series([], dtype=pl.Int64), + }) + result = compute_prevalence_per_label({"only": df}) + assert result["python"] == 0.0 + + def test_filters_out_injected_rows(self): + # Only stratified rows count. Injected rows that fire must be excluded. + df = pl.DataFrame({ + "audit_source": ["stratified", "stratified", "inject_det_python"], + "det_python": [0, 0, 1], # only the injected row fires + }) + result = compute_prevalence_per_label({"only": df}) + assert result["python"] == 0.0 + + def test_skips_label_missing_from_some_models(self): + # det_markdown is in df_a only. Function must skip it rather than crash. + df_a = pl.DataFrame({ + "audit_source": ["stratified"] * 2, + "det_python": [1, 0], + "det_markdown": [0, 0], + }) + df_b = pl.DataFrame({ + "audit_source": ["stratified"] * 2, + "det_python": [1, 1], + # intentionally missing det_markdown + }) + result = compute_prevalence_per_label({"a": df_a, "b": df_b}) + assert "python" in result + assert "markdown" not in result, ( + "Labels missing from one frame must be skipped, not included" + ) + + +class TestLoadAnnotatorParquets: + def test_expected_slug_set_constant(self): + from trainr.core.audit_semantic_labels import EXPECTED_MODEL_SLUGS + assert EXPECTED_MODEL_SLUGS == frozenset({"gemini3flash", "sonnet", "gpt54mini"}) + + def test_loads_three_parquets_by_slug(self, tmp_path): + from trainr.core.audit_semantic_labels import load_annotator_parquets + + def _make(path: Path, val: int): + pl.DataFrame({ + "audit_source": ["stratified"], + "det_python": [val], + }).write_parquet(path) + + _make(tmp_path / "iter17_ab_iter15_gemini3flash.parquet", 1) + _make(tmp_path / "iter17_ab_iter15_sonnet.parquet", 0) + _make(tmp_path / "iter17_ab_iter15_gpt54mini.parquet", 1) + + paths = sorted(tmp_path.glob("iter17_ab_iter15_*.parquet")) + result = load_annotator_parquets(paths) + + assert set(result.keys()) == {"gemini3flash", "sonnet", "gpt54mini"} + assert result["gemini3flash"]["det_python"][0] == 1 + assert result["sonnet"]["det_python"][0] == 0 + + def test_rejects_wrong_count(self, tmp_path): + from trainr.core.audit_semantic_labels import load_annotator_parquets + + pl.DataFrame({"audit_source": ["stratified"]}).write_parquet( + tmp_path / "iter17_ab_iter15_gemini3flash.parquet" + ) + paths = [tmp_path / "iter17_ab_iter15_gemini3flash.parquet"] + with pytest.raises(ValueError, match="expected 3 parquets"): + load_annotator_parquets(paths) + + def test_rejects_duplicate_slug(self, tmp_path): + from trainr.core.audit_semantic_labels import load_annotator_parquets + + def _make(path: Path): + pl.DataFrame({"audit_source": ["stratified"]}).write_parquet(path) + + _make(tmp_path / "iter17_ab_iter15_gemini3flash.parquet") + _make(tmp_path / "iter17_ab_iter15_sonnet.parquet") + _make(tmp_path / "iter17_ab_iter15_sonnet_copy.parquet") + # The "copy" one won't match the regex expected set and will be rejected. + paths = sorted(tmp_path.glob("iter17_ab_iter15_*.parquet")) + with pytest.raises(ValueError): + load_annotator_parquets(paths) + + def test_rejects_true_duplicate_slug(self, tmp_path): + """Two paths that both parse to the same canonical slug must raise.""" + from trainr.core.audit_semantic_labels import load_annotator_parquets + + def _make(path: Path): + pl.DataFrame({"audit_source": ["stratified"]}).write_parquet(path) + + # Two files in different dirs that both parse to slug "gemini3flash". + dir_a = tmp_path / "a" + dir_b = tmp_path / "b" + dir_a.mkdir() + dir_b.mkdir() + _make(dir_a / "iter17_ab_gemini3flash.parquet") + _make(dir_b / "iter17_ab_gemini3flash.parquet") + _make(dir_a / "iter17_ab_sonnet.parquet") + + paths = [ + dir_a / "iter17_ab_gemini3flash.parquet", + dir_b / "iter17_ab_gemini3flash.parquet", + dir_a / "iter17_ab_sonnet.parquet", + ] + with pytest.raises(ValueError, match="duplicate slug"): + load_annotator_parquets(paths) + + def test_rejects_unknown_slug(self, tmp_path): + from trainr.core.audit_semantic_labels import load_annotator_parquets + + def _make(path: Path): + pl.DataFrame({"audit_source": ["stratified"]}).write_parquet(path) + + _make(tmp_path / "iter17_ab_iter15_gemini3flash.parquet") + _make(tmp_path / "iter17_ab_iter15_sonnet.parquet") + _make(tmp_path / "iter17_ab_iter15_claude35.parquet") + + paths = sorted(tmp_path.glob("iter17_ab_iter15_*.parquet")) + with pytest.raises(ValueError, match="unexpected slugs"): + load_annotator_parquets(paths) + + +class TestIter16ReportReproducibility: + """Guard against silent drift in audit_semantic_labels output. + + The iter16 audit report is committed at docs/accuracy_runs/... . If any + refactor changes agreement math, rounding, column ordering, or + format_report output, this test fails loud. + """ + + def test_iter16_audit_report_reproduces_byte_for_byte(self): + from trainr.core.audit_semantic_labels import ( + compute_agreement_across_models, + compute_recall_majority, + filter_for_agreement, + format_report, + ) + + repo_root = Path(__file__).resolve().parents[2] + parquet_dir = repo_root / "training" / "data" / "audit" + report_path = ( + repo_root / "docs" / "accuracy_runs" + / "2026-04-10-iteration-16-audit-report.md" + ) + + if not parquet_dir.exists() or not report_path.exists(): + pytest.skip("iter16 fixture data not available in this checkout") + + dfs = { + "gemini3flash": pl.read_parquet(parquet_dir / "iter16_5k_gemini3flash.parquet"), + "sonnet": pl.read_parquet(parquet_dir / "iter16_5k_sonnet.parquet"), + "gpt54mini": pl.read_parquet(parquet_dir / "iter16_5k_gpt54mini.parquet"), + } + stratified_dfs = {name: filter_for_agreement(df) for name, df in dfs.items()} + agreement = compute_agreement_across_models(stratified_dfs) + recall = compute_recall_majority(dfs) + regenerated = format_report(dfs, agreement, recall) + + committed = report_path.read_text() + assert regenerated == committed, ( + "iter16 audit report drift detected — a refactor changed the " + "audit output. If this change is intentional, regenerate the " + "committed report and update this test's fixture reference." + ) + diff --git a/training/tests/test_build_audit_sample.py b/training/tests/test_build_audit_sample.py index f9e03bb..37be839 100644 --- a/training/tests/test_build_audit_sample.py +++ b/training/tests/test_build_audit_sample.py @@ -156,3 +156,78 @@ def test_build_audit_sample_injection_regexes_match_positives(): assert inject_sources.count("inject_stack_trace") == 2 assert inject_sources.count("inject_diff_patch") == 2 assert inject_sources.count("inject_log_content") == 2 + + +# --------------------------------------------------------------------------- +# Regression tests for tightened stack_trace INJECTION_PATTERNS (iter17) +# --------------------------------------------------------------------------- + +import re + +from trainr.core.build_audit_sample import INJECTION_PATTERNS + + +def _matches(label: str, text: str) -> bool: + """Return True if any pattern for `label` matches `text`. + + Each pattern is compiled individually so that Rust-regex inline flags + like `(?m)` (which Python's `re` only allows at the start of a full + pattern) are accepted. This matches the semantics of polars' + `str.contains` OR composition used in `find_injection_candidates`. + """ + for pattern in INJECTION_PATTERNS[label]: + if re.search(pattern, text) is not None: + return True + return False + + +class TestStackTraceInjectionPatternTightening: + """Regression tests: each case SHOULD NOT match after the tightening.""" + + def test_rust_error_pattern_directive_does_not_match(self): + text = "// error-pattern:thread 'main' panicked at" + assert not _matches("stack_trace", text), ( + "Rust test directive must not be injected as a stack_trace candidate" + ) + + def test_python_prose_at_line_without_traceback_does_not_match(self): + text = "the parser errored at line 42 of the config" + assert not _matches("stack_trace", text), ( + "Prose mentioning 'at line N' without a Traceback header must not match" + ) + + def test_java_frame_without_exception_header_does_not_match(self): + text = " See also: at com.foo.Bar.method(Bar.java:15) for details" + assert not _matches("stack_trace", text), ( + "Java frame with no 'Exception in thread' context must not match" + ) + + def test_real_python_traceback_still_matches(self): + text = ( + 'Traceback (most recent call last):\n' + ' File "foo.py", line 5, in \n' + ' raise ValueError("bad")\n' + 'ValueError: bad' + ) + assert _matches("stack_trace", text), ( + "Real Python traceback must still be detected" + ) + + def test_real_java_trace_still_matches(self): + text = ( + 'Exception in thread "main" java.lang.NullPointerException\n' + ' at com.foo.Bar.method(Bar.java:15)\n' + ' at com.foo.Baz.run(Baz.java:22)' + ) + assert _matches("stack_trace", text), ( + "Real Java trace with Exception header must still be detected" + ) + + def test_real_rust_panic_still_matches(self): + text = ( + "thread 'main' panicked at 'assertion failed', src/lib.rs:42\n" + "note: run with `RUST_BACKTRACE=1`" + ) + assert _matches("stack_trace", text), ( + "Real Rust panic at runtime must still match" + ) diff --git a/training/tests/test_compare_prompt_versions.py b/training/tests/test_compare_prompt_versions.py new file mode 100644 index 0000000..89a1619 --- /dev/null +++ b/training/tests/test_compare_prompt_versions.py @@ -0,0 +1,623 @@ +"""Unit tests for compare_prompt_versions.py. + +Fixture pattern: construct small polars DataFrames in-memory and call the +public API directly. No filesystem I/O except where explicitly testing +filesystem-interacting functions (glob handling, parquet reading). +""" + +from __future__ import annotations + +import math + +import polars as pl +import pytest + +from trainr.core.compare_prompt_versions import ( + AGREEMENT_DELTA_THRESHOLD, + DeltaReport, + LabelCategory, + LabelVerdict, + compare_prompt_versions, +) + + +def _make_input_frame(n_strat: int = 10, n_inject: int = 2) -> pl.DataFrame: + """Minimal input parquet fixture matching the real iter16_5k_input schema.""" + rows = [ + {"text": f"row-{i}", "sub_type": "python", "audit_source": "stratified"} + for i in range(n_strat) + ] + [ + {"text": f"inj-{i}", "sub_type": "python", "audit_source": "inject_det_python"} + for i in range(n_inject) + ] + return pl.DataFrame(rows) + + +def _make_annotator_frame( + input_frame: pl.DataFrame, + det_columns: dict[str, list[int]], +) -> pl.DataFrame: + """Clone the input frame and append det_* columns with given values.""" + result = input_frame.clone() + for col, values in det_columns.items(): + result = result.with_columns(pl.Series(col, values)) + return result + + +class TestHappyPath: + def test_all_shared_all_pass(self): + """Baseline: 3 shared labels, zero delta, identical noise floor, all PASS.""" + input_frame = _make_input_frame(n_strat=10, n_inject=0) + # Every model fires det_python on rows 0-2 (30% prevalence). + votes = [1, 1, 1, 0, 0, 0, 0, 0, 0, 0] + before_frames = { + "gemini3flash": _make_annotator_frame(input_frame, {"det_python": votes}), + "sonnet": _make_annotator_frame(input_frame, {"det_python": votes}), + "gpt54mini": _make_annotator_frame(input_frame, {"det_python": votes}), + } + after_frames = {k: v.clone() for k, v in before_frames.items()} + noise_frames = {k: v.clone() for k, v in before_frames.items()} + + report = compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + + assert isinstance(report, DeltaReport) + assert "python" in report.shared_labels + py = report.labels["python"] + assert py.verdict == LabelVerdict.PASS + assert py.delta_agreement == pytest.approx(0.0) + assert py.iter15_prevalence == pytest.approx(0.3) + assert py.iter16_prevalence == pytest.approx(0.3) + assert py.prevalence_ratio == pytest.approx(1.0) + assert py.noise_floor == pytest.approx(0.0) + + +class TestColumnCategorization: + def _make_frames(self, det_columns: list[str]) -> tuple[dict[str, pl.DataFrame], pl.DataFrame]: + input_frame = _make_input_frame(n_strat=5, n_inject=0) + frames = {} + for slug in ("gemini3flash", "sonnet", "gpt54mini"): + frames[slug] = _make_annotator_frame( + input_frame, + {col: [0] * 5 for col in det_columns}, + ) + return frames, input_frame + + def test_iter15_only_label_categorized(self): + before_frames, input_frame = self._make_frames(["det_python", "det_log_lines"]) + after_frames, _ = self._make_frames(["det_python"]) + noise_frames = {k: v.clone() for k, v in after_frames.items()} + + report = compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + + assert "log_lines" in report.iter15_only_labels + assert "log_lines" in report.labels + assert report.labels["log_lines"].category == LabelCategory.ITER15_ONLY + assert report.labels["log_lines"].verdict == LabelVerdict.NO_VERDICT + assert report.labels["log_lines"].iter16_agreement is None + assert report.labels["log_lines"].delta_agreement is None + assert "python" in report.shared_labels + + def test_iter16_only_label_categorized(self): + before_frames, input_frame = self._make_frames(["det_python"]) + after_frames, _ = self._make_frames(["det_python", "det_log_content"]) + noise_frames = {k: v.clone() for k, v in after_frames.items()} + + report = compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + + assert "log_content" in report.iter16_only_labels + assert report.labels["log_content"].category == LabelCategory.ITER16_ONLY + assert report.labels["log_content"].verdict == LabelVerdict.NO_VERDICT + assert report.labels["log_content"].iter15_agreement is None + + def test_mixed_asymmetry(self): + before_frames, input_frame = self._make_frames(["det_python", "det_log_lines"]) + after_frames, _ = self._make_frames(["det_python", "det_log_content"]) + noise_frames = {k: v.clone() for k, v in after_frames.items()} + + report = compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + + assert report.shared_labels == ["python"] + assert report.iter15_only_labels == ["log_lines"] + assert report.iter16_only_labels == ["log_content"] + + def test_after_and_noise_floor_column_mismatch_raises(self): + before_frames, input_frame = self._make_frames(["det_python"]) + after_frames, _ = self._make_frames(["det_python", "det_log_content"]) + # noise_floor missing det_log_content — this must fail loud. + noise_frames, _ = self._make_frames(["det_python"]) + + with pytest.raises(ValueError, match="det_log_content"): + compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + + +class TestVerdictLogic: + """Verdict logic per the spec's hard/soft gate rules.""" + + def _frames_with_votes( + self, + before_votes: list[list[int]], + after_votes: list[list[int]], + noise_votes: list[list[int]] | None = None, + ) -> tuple[dict, dict, dict, pl.DataFrame]: + """Build 3-model frame sets from per-model vote lists. + + Each *_votes arg is a list of 3 lists (one per model), each of + length n_rows. Returns (before_frames, after_frames, noise_frames, + input_frame). + """ + n_rows = len(before_votes[0]) + input_frame = _make_input_frame(n_strat=n_rows, n_inject=0) + noise_votes_actual = noise_votes if noise_votes is not None else after_votes + + def _build(vote_lists): + return { + slug: _make_annotator_frame(input_frame, {"det_python": votes}) + for slug, votes in zip( + ("gemini3flash", "sonnet", "gpt54mini"), + vote_lists, + ) + } + + return _build(before_votes), _build(after_votes), _build(noise_votes_actual), input_frame + + def test_pass_when_delta_zero(self): + before_frames, after_frames, noise_frames, input_frame = self._frames_with_votes( + before_votes=[[1, 1, 0, 0]] * 3, # unanimous 0.5 prev, agr=1.0 + after_votes=[[1, 1, 0, 0]] * 3, + ) + report = compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + assert report.labels["python"].verdict == LabelVerdict.PASS + + def test_fail_agreement_hard_gate(self): + # iter15 unanimous on all 10 rows (agr=1.0); iter16 has 1 split (2-1) + # on row 9, giving iter16_agr = (9 + 2/3) / 10 ≈ 0.9666. Δ ≈ -0.0333 + # which exceeds the 0.005 threshold → FAIL-agreement. + before_frames, after_frames, noise_frames, input_frame = self._frames_with_votes( + before_votes=[ + [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], + [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], + [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], + ], + after_votes=[ + [1, 1, 1, 1, 1, 1, 1, 1, 1, 0], # row 9 dissent + [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], + [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], + ], + # noise == after → noise_floor = 0 + ) + report = compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + row = report.labels["python"] + assert abs(row.delta_agreement) > AGREEMENT_DELTA_THRESHOLD + assert row.verdict == LabelVerdict.FAIL_AGREEMENT + + def test_warn_prevalence_low_ratio(self): + # iter15 fires 3/4 rows, iter16 fires 1/4 → ratio = 0.33 → WARN. + # Agreement is full 1.0 on both sides so no FAIL — pure WARN test. + before_frames, after_frames, noise_frames, input_frame = self._frames_with_votes( + before_votes=[[1, 1, 1, 0]] * 3, + after_votes=[[1, 0, 0, 0]] * 3, + ) + report = compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + row = report.labels["python"] + assert row.iter15_prevalence == 0.75 + assert row.iter16_prevalence == 0.25 + assert row.prevalence_ratio == pytest.approx(0.333, rel=0.01) + # Note: Δagr=0 so this should be WARN_PREVALENCE, NOT FAIL_AND_WARN. + assert row.verdict == LabelVerdict.WARN_PREVALENCE + + def test_warn_prevalence_high_ratio(self): + before_frames, after_frames, noise_frames, input_frame = self._frames_with_votes( + before_votes=[[1, 0, 0, 0]] * 3, + after_votes=[[1, 1, 1, 0]] * 3, + ) + report = compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + row = report.labels["python"] + assert row.prevalence_ratio == pytest.approx(3.0) + assert row.verdict == LabelVerdict.WARN_PREVALENCE + + def test_fail_and_warn_co_occur(self): + # Construct a case where Δagr > 0.005 AND prev_ratio > 2.0 both hold. + # iter15: 1/10 rows fire unanimously (agr=1.0, prev=0.1) + # iter16: 3/10 rows with 2-1 split on one of them + # → prev = 3/10 = 0.3, ratio = 3.0 (out of [0.5, 2.0], WARN) + # → agr = (9 + 2/3)/10 ≈ 0.9666, Δ ≈ -0.0333 (FAIL) + before_frames, after_frames, noise_frames, input_frame = self._frames_with_votes( + before_votes=[ + [1, 0, 0, 0, 0, 0, 0, 0, 0, 0], + [1, 0, 0, 0, 0, 0, 0, 0, 0, 0], + [1, 0, 0, 0, 0, 0, 0, 0, 0, 0], + ], + after_votes=[ + [1, 1, 1, 0, 0, 0, 0, 0, 0, 1], # 4 fires with 1 dissent on row 9 + [1, 1, 1, 0, 0, 0, 0, 0, 0, 0], + [1, 1, 1, 0, 0, 0, 0, 0, 0, 0], + ], + ) + report = compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + row = report.labels["python"] + # Agreement: iter15=1.0 (all unanimous); iter16 has 1 row with 2-1 split. + assert abs(row.delta_agreement) > AGREEMENT_DELTA_THRESHOLD + # Prevalence ratio: iter15=1/10=0.1; iter16 majority fires on rows + # 0,1,2 (the 2-1 row on row 9 does NOT meet majority). ratio=0.3/0.1=3.0 + assert row.prevalence_ratio == pytest.approx(3.0) + assert row.verdict == LabelVerdict.FAIL_AND_WARN + + def test_zero_prevalence_both_sides_no_warn(self): + before_frames, after_frames, noise_frames, input_frame = self._frames_with_votes( + before_votes=[[0, 0, 0, 0]] * 3, + after_votes=[[0, 0, 0, 0]] * 3, + ) + report = compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + row = report.labels["python"] + assert row.prevalence_ratio == 1.0 + assert row.verdict == LabelVerdict.PASS + + def test_zero_to_nonzero_warn(self): + before_frames, after_frames, noise_frames, input_frame = self._frames_with_votes( + before_votes=[[0, 0, 0, 0]] * 3, + after_votes=[[1, 1, 1, 1]] * 3, + ) + report = compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + row = report.labels["python"] + assert math.isinf(row.prevalence_ratio) + assert row.verdict == LabelVerdict.WARN_PREVALENCE + + +class TestRowAlignmentFingerprint: + def test_passing_fingerprint_does_not_raise(self): + input_frame = _make_input_frame(n_strat=5, n_inject=0) + before_frames = { + slug: _make_annotator_frame(input_frame, {"det_python": [0] * 5}) + for slug in ("gemini3flash", "sonnet", "gpt54mini") + } + after_frames = {k: v.clone() for k, v in before_frames.items()} + noise_frames = {k: v.clone() for k, v in before_frames.items()} + + # Should succeed without raising. + compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + + def test_text_mutation_raises_with_row_index(self): + input_frame = _make_input_frame(n_strat=5, n_inject=0) + before_frames = { + slug: _make_annotator_frame(input_frame, {"det_python": [0] * 5}) + for slug in ("gemini3flash", "sonnet", "gpt54mini") + } + # Corrupt one model's parquet by mutating a text row. + bad = before_frames["gemini3flash"].with_columns( + pl.Series("text", ["row-0", "row-1", "CORRUPTED", "row-3", "row-4"]) + ) + before_frames["gemini3flash"] = bad + after_frames = {k: v.clone() for k, v in before_frames.items()} + noise_frames = {k: v.clone() for k, v in before_frames.items()} + + with pytest.raises(ValueError) as excinfo: + compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + msg = str(excinfo.value) + # Error must name the row index and the diverging column. + assert "row 2" in msg or "index 2" in msg, f"row index missing from error: {msg!r}" + assert "text" in msg, f"column name missing from error: {msg!r}" + assert "gemini3flash" in msg, f"source slug missing from error: {msg!r}" + + def test_row_count_mismatch_raises(self): + input_frame = _make_input_frame(n_strat=5, n_inject=0) + smaller = input_frame.head(3) + before_frames = { + "gemini3flash": _make_annotator_frame(smaller, {"det_python": [0, 0, 0]}), + "sonnet": _make_annotator_frame(input_frame, {"det_python": [0] * 5}), + "gpt54mini": _make_annotator_frame(input_frame, {"det_python": [0] * 5}), + } + after_frames = {k: v.clone() for k, v in before_frames.items()} + noise_frames = {k: v.clone() for k, v in before_frames.items()} + + with pytest.raises(ValueError, match="row count"): + compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + + def test_missing_non_det_column_raises(self): + input_frame = _make_input_frame(n_strat=5, n_inject=0) + # Drop sub_type from one annotation frame to simulate a column-schema + # mismatch on a non-det column. The fingerprint check should catch it. + before_frames = { + slug: _make_annotator_frame(input_frame, {"det_python": [0] * 5}) + for slug in ("gemini3flash", "sonnet", "gpt54mini") + } + before_frames["sonnet"] = before_frames["sonnet"].drop("sub_type") + after_frames = {k: v.clone() for k, v in before_frames.items()} + # Restore sub_type on after/noise so the mismatch is only on before. + after_frames["sonnet"] = _make_annotator_frame(input_frame, {"det_python": [0] * 5}) + noise_frames = {k: v.clone() for k, v in before_frames.items()} + noise_frames["sonnet"] = _make_annotator_frame(input_frame, {"det_python": [0] * 5}) + + with pytest.raises(ValueError, match="sub_type|non-det"): + compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + + +class TestFormatReport: + def test_report_contains_all_required_sections(self): + from trainr.core.compare_prompt_versions import format_delta_report + + input_frame = _make_input_frame(n_strat=5, n_inject=0) + before_frames = { + slug: _make_annotator_frame( + input_frame, {"det_python": [1, 0, 0, 0, 0], "det_log_lines": [0] * 5} + ) + for slug in ("gemini3flash", "sonnet", "gpt54mini") + } + after_frames = { + slug: _make_annotator_frame( + input_frame, + {"det_python": [1, 0, 0, 0, 0], "det_log_content": [0] * 5}, + ) + for slug in ("gemini3flash", "sonnet", "gpt54mini") + } + noise_frames = {k: v.clone() for k, v in after_frames.items()} + + report = compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + text = format_delta_report(report) + + assert "# iter17 A/B Regression Audit Report" in text + assert "## Gate verdict" in text + assert "## Shared labels" in text + assert "## iter15-only labels" in text + assert "## iter16-only labels" in text + assert "## Noise floor table" in text + # Shared label row present + assert "python" in text + # iter15-only label present in its section + assert "log_lines" in text + # iter16-only label present in its section + assert "log_content" in text + # No FAIL rows in this fixture → PASS gate verdict + assert "**PASS**" in text + + def test_report_summary_shows_fail_count(self): + from trainr.core.compare_prompt_versions import format_delta_report + + input_frame = _make_input_frame(n_strat=10, n_inject=0) + # Construct a FAIL-agreement row: iter15 unanimous, iter16 has 2-1 split. + before_votes = [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1]] * 3 + after_votes = [ + [1, 1, 1, 1, 1, 1, 1, 1, 1, 0], + [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], + [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], + ] + before_frames = { + slug: _make_annotator_frame(input_frame, {"det_python": v}) + for slug, v in zip(("gemini3flash", "sonnet", "gpt54mini"), before_votes) + } + after_frames = { + slug: _make_annotator_frame(input_frame, {"det_python": v}) + for slug, v in zip(("gemini3flash", "sonnet", "gpt54mini"), after_votes) + } + noise_frames = {k: v.clone() for k, v in after_frames.items()} + + report = compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + text = format_delta_report(report) + assert "**FAIL**" in text + # Summary line should mention the FAIL count + assert "1 FAIL-agreement" in text or "FAIL-agreement: 1" in text or "1 FAIL" in text + + def test_report_handles_inf_prevalence_ratio(self): + """A label with iter15_prev=0 and iter16_prev>0 should format cleanly.""" + from trainr.core.compare_prompt_versions import format_delta_report + + input_frame = _make_input_frame(n_strat=4, n_inject=0) + # iter15: 0 fires; iter16: all 4 fire → ratio = inf + before_votes = [[0, 0, 0, 0]] * 3 + after_votes = [[1, 1, 1, 1]] * 3 + before_frames = { + slug: _make_annotator_frame(input_frame, {"det_python": v}) + for slug, v in zip(("gemini3flash", "sonnet", "gpt54mini"), before_votes) + } + after_frames = { + slug: _make_annotator_frame(input_frame, {"det_python": v}) + for slug, v in zip(("gemini3flash", "sonnet", "gpt54mini"), after_votes) + } + noise_frames = {k: v.clone() for k, v in after_frames.items()} + + report = compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + text = format_delta_report(report) + # Should not crash, should not say "nan". "inf" is OK as a formatted value. + assert "nan" not in text.lower() or "inf" in text + assert "python" in text + + +def test_agreement_delta_threshold_constant(): + """Pin the threshold so future edits don't silently move it.""" + from trainr.core.compare_prompt_versions import AGREEMENT_DELTA_THRESHOLD + assert AGREEMENT_DELTA_THRESHOLD == 0.005 + + +def test_prevalence_ratio_bounds_constants(): + from trainr.core.compare_prompt_versions import ( + PREVALENCE_RATIO_HIGH, + PREVALENCE_RATIO_LOW, + ) + assert PREVALENCE_RATIO_LOW == 0.5 + assert PREVALENCE_RATIO_HIGH == 2.0 + + +class TestMainCLI: + def test_main_end_to_end_from_parquets(self, tmp_path): + """Run main() against a small set of real parquet files on disk.""" + from trainr.core.compare_prompt_versions import main as compare_main + + # Build a tiny input parquet and 9 annotation parquets. + input_frame = _make_input_frame(n_strat=5, n_inject=0) + input_path = tmp_path / "input.parquet" + input_frame.write_parquet(input_path) + + def _write_side(side: str): + for slug in ("gemini3flash", "sonnet", "gpt54mini"): + df = _make_annotator_frame(input_frame, {"det_python": [0] * 5}) + df.write_parquet(tmp_path / f"iter17_ab_{side}_{slug}.parquet") + + _write_side("iter15") + _write_side("iter16a") + _write_side("iter16b") + + output_report = tmp_path / "report.md" + compare_main([ + "--before", str(tmp_path / "iter17_ab_iter15_*.parquet"), + "--after", str(tmp_path / "iter17_ab_iter16a_*.parquet"), + "--noise-floor", str(tmp_path / "iter17_ab_iter16b_*.parquet"), + "--input", str(input_path), + "--output", str(output_report), + ]) + + assert output_report.exists() + content = output_report.read_text() + assert "# iter17 A/B Regression Audit Report" in content + assert "**PASS**" in content + + def test_main_exits_2_on_fail(self, tmp_path): + """main() should sys.exit(2) if any FAIL-agreement row exists.""" + import sys + + from trainr.core.compare_prompt_versions import main as compare_main + + input_frame = _make_input_frame(n_strat=10, n_inject=0) + input_path = tmp_path / "input.parquet" + input_frame.write_parquet(input_path) + + # iter15 unanimous all-1, iter16 has a 2-1 split on row 9 → FAIL. + before_votes = [[1] * 10] * 3 + after_votes = [ + [1, 1, 1, 1, 1, 1, 1, 1, 1, 0], + [1] * 10, + [1] * 10, + ] + + def _write_side(side: str, votes_list): + for slug, votes in zip(("gemini3flash", "sonnet", "gpt54mini"), votes_list): + df = _make_annotator_frame(input_frame, {"det_python": votes}) + df.write_parquet(tmp_path / f"iter17_ab_{side}_{slug}.parquet") + + _write_side("iter15", before_votes) + _write_side("iter16a", after_votes) + _write_side("iter16b", after_votes) + + output_report = tmp_path / "report.md" + with pytest.raises(SystemExit) as excinfo: + compare_main([ + "--before", str(tmp_path / "iter17_ab_iter15_*.parquet"), + "--after", str(tmp_path / "iter17_ab_iter16a_*.parquet"), + "--noise-floor", str(tmp_path / "iter17_ab_iter16b_*.parquet"), + "--input", str(input_path), + "--output", str(output_report), + ]) + assert excinfo.value.code == 2 + # Report should still be written before the exit. + assert output_report.exists() + assert "**FAIL**" in output_report.read_text() + + def test_main_raises_on_empty_glob(self, tmp_path): + from trainr.core.compare_prompt_versions import main as compare_main + + # No parquets exist at these paths. + input_path = tmp_path / "input.parquet" + _make_input_frame(n_strat=5, n_inject=0).write_parquet(input_path) + output_report = tmp_path / "report.md" + + with pytest.raises(ValueError, match="matched zero files"): + compare_main([ + "--before", str(tmp_path / "nonexistent_*.parquet"), + "--after", str(tmp_path / "also_missing_*.parquet"), + "--noise-floor", str(tmp_path / "also_gone_*.parquet"), + "--input", str(input_path), + "--output", str(output_report), + ]) diff --git a/training/trainr/commands/data.py b/training/trainr/commands/data.py index 4e0a649..3b978bb 100644 --- a/training/trainr/commands/data.py +++ b/training/trainr/commands/data.py @@ -94,6 +94,24 @@ def annotate_detections_cmd(**kwargs): _main(argv) +@data.command("compare-prompts") +@click.option("--before", required=True, help="Glob for iter15-side parquets (3 files).") +@click.option("--after", required=True, help="Glob for iter16a-side parquets (3 files, A/B after side).") +@click.option( + "--noise-floor", "noise_floor", required=True, + help="Glob for iter16b-side parquets (3 files, same-prompt noise companion).", +) +@click.option("--input", required=True, help="Path to input parquet (iter16_5k_input.parquet).") +@click.option("--output", required=True, help="Markdown report output path.") +def compare_prompts_cmd(**kwargs): + """A/B regression audit between two SYSTEM_PROMPT versions.""" + from trainr.core.compare_prompt_versions import main as _main + + # Click uses underscores in kwargs; _build_argv converts to dashes. + argv = _build_argv(kwargs) + _main(argv) + + @data.command("relabel-unknowns") @click.option("--input", required=True, help="Path to input Parquet file with unknown sub_types.") @click.option("--output", required=True, help="Path to output Parquet file.") diff --git a/training/trainr/core/annotate_detections.py b/training/trainr/core/annotate_detections.py index 964e9cf..84b1630 100644 --- a/training/trainr/core/annotate_detections.py +++ b/training/trainr/core/annotate_detections.py @@ -783,3 +783,7 @@ def main(argv: list[str] | None = None) -> None: print(" Checkpoint removed (run complete).", file=sys.stderr) print(f" Done. {len(result_df)} samples with {len(DETECTION_LABELS)} detection columns.", file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/training/trainr/core/audit_semantic_labels.py b/training/trainr/core/audit_semantic_labels.py index 1952c28..2569a4d 100644 --- a/training/trainr/core/audit_semantic_labels.py +++ b/training/trainr/core/audit_semantic_labels.py @@ -21,6 +21,7 @@ import argparse import math +import re import sys from pathlib import Path @@ -30,6 +31,17 @@ STRATIFIED = "stratified" +# Canonical model slug set for iter16/iter17 annotation parquets. This is +# the authoritative set — every loader that accepts a glob of 3 parquets +# asserts its parsed slugs equal this set. Duplicates, misnames, and +# unknown slugs all fail loud here rather than silently corrupting metrics. +EXPECTED_MODEL_SLUGS: frozenset[str] = frozenset({"gemini3flash", "sonnet", "gpt54mini"}) + +# Filename pattern: _.parquet where slug is alphanumeric. +# The capture group extracts the slug. Validation against EXPECTED_MODEL_SLUGS +# happens in load_annotator_parquets. +_SLUG_RE = re.compile(r".*_(?P[a-z0-9]+)\.parquet$") + # Pass criteria thresholds (from spec) AGREEMENT_THRESHOLD = 0.995 RECALL_THRESHOLD = 0.90 @@ -45,6 +57,63 @@ def detection_columns(df: pl.DataFrame) -> list[str]: return [c for c in df.columns if c.startswith("det_")] +def load_annotator_parquets( + paths: list[Path], +) -> dict[str, pl.DataFrame]: + """Load annotator parquets keyed by model slug parsed from filename. + + Asserts that exactly 3 parquets are passed AND that their parsed slug + set equals EXPECTED_MODEL_SLUGS. This catches the "three files but one + is a duplicate or misnamed" failure mode that a count check alone + cannot — critical because gate metrics depend on knowing which model + produced which parquet. + + Args: + paths: List of 3 parquet file paths. + + Returns: + {slug: DataFrame} with keys exactly equal to EXPECTED_MODEL_SLUGS. + + Raises: + ValueError: If count != 3, a filename doesn't match the slug regex, + or the parsed slug set != EXPECTED_MODEL_SLUGS. + """ + if len(paths) != 3: + raise ValueError( + f"load_annotator_parquets: expected 3 parquets, got {len(paths)}: " + f"{[str(p) for p in paths]}" + ) + + parsed: dict[str, Path] = {} + for path in paths: + match = _SLUG_RE.match(path.name) + if match is None: + raise ValueError( + f"load_annotator_parquets: could not parse model slug from " + f"{path.name!r}; expected format '_.parquet' " + f"with slug in {sorted(EXPECTED_MODEL_SLUGS)}" + ) + slug = match.group("slug") + if slug in parsed: + raise ValueError( + f"load_annotator_parquets: duplicate slug {slug!r} " + f"(already mapped to {parsed[slug].name}, now {path.name})" + ) + parsed[slug] = path + + parsed_slugs = frozenset(parsed.keys()) + if parsed_slugs != EXPECTED_MODEL_SLUGS: + missing = EXPECTED_MODEL_SLUGS - parsed_slugs + extra = parsed_slugs - EXPECTED_MODEL_SLUGS + raise ValueError( + f"load_annotator_parquets: parsed slugs do not match expected set. " + f"missing={sorted(missing)}, unexpected slugs={sorted(extra)}. " + f"expected={sorted(EXPECTED_MODEL_SLUGS)}" + ) + + return {slug: pl.read_parquet(path) for slug, path in parsed.items()} + + def compute_agreement_across_models( dfs: dict[str, pl.DataFrame], ) -> dict[str, float]: @@ -106,6 +175,60 @@ def compute_agreement_across_models( return result +def compute_prevalence_per_label( + dfs: dict[str, pl.DataFrame], +) -> dict[str, float]: + """Majority-of-N fire rate per label on stratified rows. + + For each `det_*` column present in the first DataFrame, compute the + fraction of stratified rows where at least ceil(N/2) of the N models + fired (det == 1). Zero-row inputs return 0.0. + + Args: + dfs: {model_slug: DataFrame}. All DataFrames must already be + filtered to stratified rows (use filter_for_agreement upstream) + and share row ordering. + + Returns: + {label_without_det_prefix: prevalence_float in [0.0, 1.0]}. + """ + model_names = list(dfs.keys()) + if not model_names: + return {} + + first = dfs[model_names[0]] + # Filter all frames to stratified rows. Doing this inside the function + # makes the function safe to call with raw annotator output (the alt is + # requiring every caller to remember to filter first). + stratified_dfs = { + name: df.filter(pl.col("audit_source") == STRATIFIED) + for name, df in dfs.items() + } + stratified_first = stratified_dfs[model_names[0]] + n_rows = len(stratified_first) + if n_rows == 0: + return {label[len("det_"):]: 0.0 for label in detection_columns(first)} + + n_models = len(model_names) + majority_threshold = (n_models + 1) // 2 # ceil(N/2); for N=3 this is 2 + + result: dict[str, float] = {} + for col in detection_columns(first): + label = col[len("det_"):] + # Skip columns that don't exist on every frame (asymmetric schemas + # are the compare module's problem, not this function's). + if not all(col in stratified_dfs[m].columns for m in model_names): + continue + votes_per_model = [stratified_dfs[m][col].to_list() for m in model_names] + fire_count = 0 + for row_idx in range(n_rows): + row_votes = [votes_per_model[m][row_idx] for m in range(n_models)] + if sum(row_votes) >= majority_threshold: + fire_count += 1 + result[label] = fire_count / n_rows + return result + + def compute_recall_on_injected(df: pl.DataFrame) -> dict[str, float]: """Per-semantic-label recall on injected positives. diff --git a/training/trainr/core/build_audit_sample.py b/training/trainr/core/build_audit_sample.py index 90a8544..cfc41d0 100644 --- a/training/trainr/core/build_audit_sample.py +++ b/training/trainr/core/build_audit_sample.py @@ -32,12 +32,21 @@ # true positives for manual review. INJECTION_PATTERNS: dict[str, list[str]] = { "stack_trace": [ - r"Traceback \(most recent call last\)", - r"Exception in thread", + # Python: Traceback header + "File " adjacency within ~200 chars. + # [\s\S]{0,200} is the polars-compatible equivalent of (?s:.{0,200}) — + # "any char including newlines", up to a bounded count. + r'Traceback \(most recent call last\):[\s\S]{0,200}File "', + # Java: "Exception in thread" header + "at .(File.java:N)" + # frame adjacency within ~400 chars. + r"Exception in thread[\s\S]{0,400}\s+at [\w.$]+\(.*\.java:\d+\)", + # Go is unchanged — "goroutine N [" header is already specific. r"goroutine \d+ \[", - r"panicked at", - r"\s+at [\w.$]+\(.*\.java:\d+\)", - r"^\s+at \w+\.\w+\.\w+\(\) in .*\.cs:line \d+", # .NET + # Rust: (?m)^ anchors to line start, which excludes the test directive + # "// error-pattern:thread 'main' panicked at" (not at line start) while + # still matching real runtime panics (printed at line start). + r"(?m)^thread '[^']+' panicked at", + # .NET is unchanged — the "in .cs:line N" suffix is specific. + r"^\s+at \w+\.\w+\.\w+\(\) in .*\.cs:line \d+", ], "diff_patch": [ r"(?m)^@@ -\d+(,\d+)? \+\d+(,\d+)? @@", diff --git a/training/trainr/core/compare_prompt_versions.py b/training/trainr/core/compare_prompt_versions.py new file mode 100644 index 0000000..fc4123a --- /dev/null +++ b/training/trainr/core/compare_prompt_versions.py @@ -0,0 +1,578 @@ +"""A/B regression audit comparison — iter15 prompt vs iter16 prompt. + +Given three sets of annotator parquets (iter15 before, iter16a after, and +iter16b noise-floor companion), computes per-label agreement delta, +prevalence ratio, and same-prompt noise floor, then emits a gate verdict. + +See docs/superpowers/specs/2026-04-10-iter17-ab-regression-audit-design.md. +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass, field +from enum import Enum +from pathlib import Path + +import polars as pl + +from trainr.core.audit_semantic_labels import ( + compute_agreement_across_models, + compute_prevalence_per_label, + detection_columns, + filter_for_agreement, +) + +# Gate thresholds, from the spec. +AGREEMENT_DELTA_THRESHOLD = 0.005 +PREVALENCE_RATIO_LOW = 0.5 +PREVALENCE_RATIO_HIGH = 2.0 + + +class LabelCategory(str, Enum): + SHARED = "shared" + ITER15_ONLY = "iter15-only" + ITER16_ONLY = "iter16-only" + + +class LabelVerdict(str, Enum): + PASS = "PASS" + FAIL_AGREEMENT = "FAIL-agreement" + WARN_PREVALENCE = "WARN-prevalence" + FAIL_AND_WARN = "FAIL-agreement+WARN-prevalence" + NO_VERDICT = "N/A" # iter15-only / iter16-only rows + + +@dataclass +class LabelRow: + label: str + category: LabelCategory + iter15_agreement: float | None + iter16_agreement: float | None + delta_agreement: float | None + iter15_prevalence: float | None + iter16_prevalence: float | None + prevalence_ratio: float | None + noise_floor: float | None + verdict: LabelVerdict + + +@dataclass +class DeltaReport: + labels: dict[str, LabelRow] = field(default_factory=dict) + shared_labels: list[str] = field(default_factory=list) + iter15_only_labels: list[str] = field(default_factory=list) + iter16_only_labels: list[str] = field(default_factory=list) + + @property + def fail_agreement_rows(self) -> list[LabelRow]: + return [ + r for r in self.labels.values() + if r.verdict in (LabelVerdict.FAIL_AGREEMENT, LabelVerdict.FAIL_AND_WARN) + ] + + @property + def warn_prevalence_rows(self) -> list[LabelRow]: + return [ + r for r in self.labels.values() + if r.verdict in (LabelVerdict.WARN_PREVALENCE, LabelVerdict.FAIL_AND_WARN) + ] + + +def compare_prompt_versions( + before_frames: dict[str, pl.DataFrame], + after_frames: dict[str, pl.DataFrame], + noise_floor_frames: dict[str, pl.DataFrame], + input_frame: pl.DataFrame, +) -> DeltaReport: + """Compute the A/B regression report. + + All three frame dicts must be keyed by the canonical model slugs + ({"gemini3flash", "sonnet", "gpt54mini"}) and share row ordering with + `input_frame`. The caller is responsible for loading and filtering — + this function works in-memory only. + + `input_frame` is used for fingerprint validation: every annotation + frame must row-match the input on all non-det_* columns. This catches + row-order drift, row count mismatches, and silent corruption. + """ + report = DeltaReport() + + # --- Fingerprint validation. Every annotation frame must row-match the + # input parquet on all non-det_* columns. This catches row-order drift, + # row count mismatches, and silent corruption of the input passthrough + # columns. Must run BEFORE any metric computation so failures are + # caught early with a clear row index. + for slug, frame in before_frames.items(): + _assert_fingerprint_matches_input(frame, input_frame, f"iter15/{slug}") + for slug, frame in after_frames.items(): + _assert_fingerprint_matches_input(frame, input_frame, f"iter16a/{slug}") + for slug, frame in noise_floor_frames.items(): + _assert_fingerprint_matches_input(frame, input_frame, f"iter16b/{slug}") + + # --- Hard schema assertion: after and noise_floor must agree on det_* cols. + # Noise floor correctness depends on both sides having the same label set. + after_first = next(iter(after_frames.values())) + noise_first = next(iter(noise_floor_frames.values())) + after_det = set(detection_columns(after_first)) + noise_det = set(detection_columns(noise_first)) + if after_det != noise_det: + diff = sorted(after_det.symmetric_difference(noise_det)) + raise ValueError( + f"compare_prompt_versions: after and noise_floor have differing " + f"det_* column sets. symmetric_difference={diff}. " + f"after_only={sorted(after_det - noise_det)}, " + f"noise_only={sorted(noise_det - after_det)}" + ) + + # --- Column categorization via dynamic introspection. + before_first = next(iter(before_frames.values())) + before_det = set(detection_columns(before_first)) + + def _strip(col: str) -> str: + return col[len("det_"):] + + before_labels = {_strip(c) for c in before_det} + after_labels = {_strip(c) for c in after_det} + + shared = sorted(before_labels & after_labels) + iter15_only = sorted(before_labels - after_labels) + iter16_only = sorted(after_labels - before_labels) + + report.shared_labels = shared + report.iter15_only_labels = iter15_only + report.iter16_only_labels = iter16_only + + # --- Filter to stratified rows for agreement (prevalence filters itself). + before_strat = {k: filter_for_agreement(v) for k, v in before_frames.items()} + after_strat = {k: filter_for_agreement(v) for k, v in after_frames.items()} + noise_strat = {k: filter_for_agreement(v) for k, v in noise_floor_frames.items()} + + iter15_agr = compute_agreement_across_models(before_strat) + iter16_agr = compute_agreement_across_models(after_strat) + noise_agr = compute_agreement_across_models(noise_strat) + + iter15_prev = compute_prevalence_per_label(before_frames) + iter16_prev = compute_prevalence_per_label(after_frames) + + # --- Shared labels: full metrics, placeholder PASS verdict (Task 2.3 fixes). + for label in shared: + # Noise floor: same-prompt variance between iter16a and iter16b + # measured as |agr(iter16a) - agr(iter16b)|. Available iff the label + # is in noise_agr (guaranteed for shared labels given the schema + # assert above, but defensive check retained). + nf = abs(noise_agr[label] - iter16_agr[label]) if label in noise_agr else None + report.labels[label] = LabelRow( + label=label, + category=LabelCategory.SHARED, + iter15_agreement=iter15_agr[label], + iter16_agreement=iter16_agr[label], + delta_agreement=iter16_agr[label] - iter15_agr[label], + iter15_prevalence=iter15_prev.get(label, 0.0), + iter16_prevalence=iter16_prev.get(label, 0.0), + prevalence_ratio=_compute_prev_ratio( + iter15_prev.get(label, 0.0), + iter16_prev.get(label, 0.0), + ), + noise_floor=nf, + verdict=_compute_verdict( + delta_agreement=iter16_agr[label] - iter15_agr[label], + prevalence_ratio=_compute_prev_ratio( + iter15_prev.get(label, 0.0), + iter16_prev.get(label, 0.0), + ), + ), + ) + + # --- iter15-only labels: partial metrics, no verdict. + for label in iter15_only: + report.labels[label] = LabelRow( + label=label, + category=LabelCategory.ITER15_ONLY, + iter15_agreement=iter15_agr[label], + iter16_agreement=None, + delta_agreement=None, + iter15_prevalence=iter15_prev.get(label, 0.0), + iter16_prevalence=None, + prevalence_ratio=None, + noise_floor=None, + verdict=LabelVerdict.NO_VERDICT, + ) + + # --- iter16-only labels: partial metrics, no verdict. + for label in iter16_only: + nf = abs(noise_agr[label] - iter16_agr[label]) if label in noise_agr else None + report.labels[label] = LabelRow( + label=label, + category=LabelCategory.ITER16_ONLY, + iter15_agreement=None, + iter16_agreement=iter16_agr[label], + delta_agreement=None, + iter15_prevalence=None, + iter16_prevalence=iter16_prev.get(label, 0.0), + prevalence_ratio=None, + noise_floor=nf, + verdict=LabelVerdict.NO_VERDICT, + ) + + return report + + +def _non_det_columns(frame: pl.DataFrame) -> list[str]: + """Non-`det_*` columns in the frame's natural order.""" + return [c for c in frame.columns if not c.startswith("det_")] + + +def _assert_fingerprint_matches_input( + frame: pl.DataFrame, + input_frame: pl.DataFrame, + source: str, +) -> None: + """Assert that `frame`'s non-det_* columns row-match `input_frame`. + + Fingerprint = concatenation of all non-`det_*` column values per row. + The input_frame defines the column set; frame must have every column + the input has (and may have more — the extras are det_* columns + added by the annotator). + + Raises ValueError with the first diverging row index and column name + on mismatch. `source` is a tag included in the error message so the + caller can identify which parquet failed (e.g., "iter15/gemini3flash"). + """ + if len(frame) != len(input_frame): + raise ValueError( + f"{source}: row count mismatch — frame has {len(frame)} rows, " + f"input has {len(input_frame)}" + ) + + cols = _non_det_columns(input_frame) + missing = [c for c in cols if c not in frame.columns] + if missing: + raise ValueError( + f"{source}: annotation parquet missing non-det columns from " + f"input: {missing}" + ) + + # Column-by-column equality check. Stop on first diverging row. + for col in cols: + left = input_frame[col].to_list() + right = frame[col].to_list() + for row_idx, (l, r) in enumerate(zip(left, right)): + if l != r: + raise ValueError( + f"{source}: fingerprint mismatch at row {row_idx}, " + f"column {col!r}: input={l!r} vs annotation={r!r}" + ) + + +def _compute_prev_ratio(iter15_prev: float, iter16_prev: float) -> float: + """Zero-handling rules from the spec: 0/0 → 1.0, 0/>0 → inf, >0/0 → 0.0.""" + if iter15_prev == 0 and iter16_prev == 0: + return 1.0 + if iter15_prev == 0: + return math.inf + return iter16_prev / iter15_prev + + +def format_delta_report(report: DeltaReport) -> str: + """Render the DeltaReport as markdown per the spec's §Report Schema. + + Sections in order: + 1. Gate verdict summary (PASS/FAIL + row counts) + 2. Shared labels table (per-label metrics + verdicts) + 3. iter15-only labels section (removed in iter16) + 4. iter16-only labels section (new in iter16) + 5. FAIL-agreement detail with override-eligibility hint + 6. WARN-prevalence detail + 7. Noise floor table (per-label) + """ + lines: list[str] = [] + + lines.append("# iter17 A/B Regression Audit Report") + lines.append("") + lines.append("**Date:** 2026-04-10") + lines.append("") + + # --- Gate verdict summary + n_shared = len(report.shared_labels) + n_fail = len(report.fail_agreement_rows) + n_warn = len(report.warn_prevalence_rows) + overall = "**FAIL**" if n_fail > 0 else "**PASS**" + lines.append("## Gate verdict") + lines.append("") + lines.append(overall) + lines.append("") + lines.append( + f"Summary: {n_shared} shared labels, {n_fail} FAIL-agreement, " + f"{n_warn} WARN-prevalence." + ) + lines.append("") + + # --- Shared labels table + lines.append("## Shared labels (gate applies)") + lines.append("") + lines.append( + "| label | iter15_agr | iter16_agr | Δagr | iter15_prev | iter16_prev | " + "prev_ratio | noise_floor | verdict |" + ) + lines.append("|---|---:|---:|---:|---:|---:|---:|---:|---|") + for label in report.shared_labels: + row = report.labels[label] + lines.append( + f"| det_{row.label} " + f"| {_fmt_float(row.iter15_agreement)} " + f"| {_fmt_float(row.iter16_agreement)} " + f"| {_fmt_signed(row.delta_agreement)} " + f"| {_fmt_float(row.iter15_prevalence)} " + f"| {_fmt_float(row.iter16_prevalence)} " + f"| {_fmt_ratio(row.prevalence_ratio)} " + f"| {_fmt_float(row.noise_floor)} " + f"| {row.verdict.value} |" + ) + lines.append("") + + # --- iter15-only labels (removed in iter16) + lines.append("## iter15-only labels (removed in iter16, context only)") + lines.append("") + if report.iter15_only_labels: + lines.append("| label | iter15_agr | iter15_prev | note |") + lines.append("|---|---:|---:|---|") + for label in report.iter15_only_labels: + row = report.labels[label] + lines.append( + f"| det_{row.label} " + f"| {_fmt_float(row.iter15_agreement)} " + f"| {_fmt_float(row.iter15_prevalence)} " + f"| Not in iter16 prompt |" + ) + else: + lines.append("_None._") + lines.append("") + + # --- iter16-only labels (new in iter16) + lines.append("## iter16-only labels (new in iter16, cross-ref iter16 audit)") + lines.append("") + if report.iter16_only_labels: + lines.append("| label | iter16_agr | iter16_prev | noise_floor |") + lines.append("|---|---:|---:|---:|") + for label in report.iter16_only_labels: + row = report.labels[label] + lines.append( + f"| det_{row.label} " + f"| {_fmt_float(row.iter16_agreement)} " + f"| {_fmt_float(row.iter16_prevalence)} " + f"| {_fmt_float(row.noise_floor)} |" + ) + else: + lines.append("_None._") + lines.append("") + + # --- FAIL-agreement detail with override-eligibility hint + lines.append("## FAIL-agreement rows (if any)") + lines.append("") + if report.fail_agreement_rows: + for row in report.fail_agreement_rows: + if row.noise_floor and row.noise_floor > 0 and row.delta_agreement is not None: + ratio = abs(row.delta_agreement) / row.noise_floor + overridable = ( + "ELIGIBLE for override" if ratio <= 2.0 else "NOT overridable" + ) + lines.append( + f"- **det_{row.label}**: " + f"Δagr={_fmt_signed(row.delta_agreement)}, " + f"noise_floor={_fmt_float(row.noise_floor)}, " + f"|Δagr|/noise_floor={ratio:.2f} → {overridable}" + ) + else: + lines.append( + f"- **det_{row.label}**: " + f"Δagr={_fmt_signed(row.delta_agreement)}, " + f"noise_floor={_fmt_float(row.noise_floor)} → " + f"override eligibility cannot be computed" + ) + else: + lines.append("_None._") + lines.append("") + + # --- WARN-prevalence detail + lines.append("## WARN-prevalence rows (if any)") + lines.append("") + if report.warn_prevalence_rows: + for row in report.warn_prevalence_rows: + lines.append( + f"- **det_{row.label}**: " + f"iter15_prev={_fmt_float(row.iter15_prevalence)}, " + f"iter16_prev={_fmt_float(row.iter16_prevalence)}, " + f"ratio={_fmt_ratio(row.prevalence_ratio)}" + ) + else: + lines.append("_None._") + lines.append("") + + # --- Noise floor table (informational, all categorized labels) + lines.append("## Noise floor table") + lines.append("") + lines.append( + "Per-label same-prompt variance from iter16a vs iter16b runs. " + "Used to bound override eligibility for FAIL-agreement rows per " + "the Gate Decision Protocol." + ) + lines.append("") + lines.append("| label | category | noise_floor |") + lines.append("|---|---|---:|") + for label in sorted(report.labels.keys()): + row = report.labels[label] + lines.append( + f"| det_{row.label} | {row.category.value} | {_fmt_float(row.noise_floor)} |" + ) + lines.append("") + + return "\n".join(lines) + "\n" + + +def _fmt_float(value: float | None) -> str: + if value is None: + return "—" + if math.isinf(value): + return "inf" + if math.isnan(value): + return "nan" + return f"{value:.4f}" + + +def _fmt_signed(value: float | None) -> str: + if value is None: + return "—" + if math.isinf(value): + return "inf" + if math.isnan(value): + return "nan" + return f"{value:+.4f}" + + +def _fmt_ratio(value: float | None) -> str: + if value is None: + return "—" + if math.isinf(value): + return "inf" + if math.isnan(value): + return "nan" + return f"{value:.3f}" + + +def _compute_verdict( + delta_agreement: float, + prevalence_ratio: float, +) -> LabelVerdict: + """Combine hard agreement gate + soft prevalence gate into a verdict. + + - |Δagr| > AGREEMENT_DELTA_THRESHOLD → FAIL-agreement (hard gate) + - prev_ratio outside [PREVALENCE_RATIO_LOW, PREVALENCE_RATIO_HIGH] → WARN-prevalence (soft gate) + - Both → FAIL_AND_WARN + - Neither → PASS + + Note: override eligibility (|Δagr| ≤ 2 × noise_floor) is NOT applied + here. The verdict reflects the raw gate outcome; human review applies + override logic per the Gate Decision Protocol. + """ + fail = abs(delta_agreement) > AGREEMENT_DELTA_THRESHOLD + # math.inf, math.nan, 0.0, and any number outside [0.5, 2.0] warn. + # Only 0/0 → 1.0 is explicitly in-range via _compute_prev_ratio; all + # other zero cases produce inf or 0.0 (both outside the band). + warn = ( + math.isinf(prevalence_ratio) + or math.isnan(prevalence_ratio) + or prevalence_ratio < PREVALENCE_RATIO_LOW + or prevalence_ratio > PREVALENCE_RATIO_HIGH + ) + if fail and warn: + return LabelVerdict.FAIL_AND_WARN + if fail: + return LabelVerdict.FAIL_AGREEMENT + if warn: + return LabelVerdict.WARN_PREVALENCE + return LabelVerdict.PASS + + +def main(argv: list[str] | None = None) -> None: + """CLI entry point: `trainr data compare-prompts`. + + Parses argv, expands globs via Python's `glob` module, loads parquets + via `load_annotator_parquets` (which validates the slug set), runs + the comparison, writes the markdown report, and exits 2 if any + FAIL-agreement row exists so shell callers can gate on it. + """ + import argparse + import glob as _glob + import sys + + from trainr.core.audit_semantic_labels import load_annotator_parquets + + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--before", required=True, + help="Glob for the iter15-side parquets (3 files, one per model slug).", + ) + parser.add_argument( + "--after", required=True, + help="Glob for the iter16a-side parquets (the A/B 'after' side, 3 files).", + ) + parser.add_argument( + "--noise-floor", required=True, dest="noise_floor", + help="Glob for the iter16b-side parquets (noise floor companion, 3 files).", + ) + parser.add_argument( + "--input", required=True, + help="Path to the input parquet used for all annotation runs " + "(iter16_5k_input.parquet).", + ) + parser.add_argument( + "--output", required=True, + help="Path to write the markdown regression report.", + ) + args = parser.parse_args(argv) + + def _resolve(glob_arg: str, name: str) -> list[Path]: + paths = sorted(Path(p) for p in _glob.glob(glob_arg)) + if not paths: + raise ValueError( + f"--{name}: glob {glob_arg!r} matched zero files" + ) + return paths + + before_paths = _resolve(args.before, "before") + after_paths = _resolve(args.after, "after") + noise_paths = _resolve(args.noise_floor, "noise-floor") + + before_frames = load_annotator_parquets(before_paths) + after_frames = load_annotator_parquets(after_paths) + noise_frames = load_annotator_parquets(noise_paths) + + input_frame = pl.read_parquet(args.input) + + report = compare_prompt_versions( + before_frames=before_frames, + after_frames=after_frames, + noise_floor_frames=noise_frames, + input_frame=input_frame, + ) + text = format_delta_report(report) + + output_path = Path(args.output) + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_text(text) + + n_fail = len(report.fail_agreement_rows) + print( + f"compare-prompts: wrote {args.output}. " + f"{len(report.shared_labels)} shared labels, " + f"{n_fail} FAIL-agreement, " + f"{len(report.warn_prevalence_rows)} WARN-prevalence.", + file=sys.stderr, + ) + if n_fail > 0: + sys.exit(2) + + +if __name__ == "__main__": + main()