diff --git a/README.md b/README.md index 615885b..c091a48 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,13 @@ Read the [phase-one findings](docs/blog/sota-v2-findings.md), inspect the [generated analysis](results/analysis/publication-panel-analysis.json), or follow the [clean-clone reproduction guide](docs/REPRODUCING_SOTA_V2_RELEASE.md). +The tagged study remains frozen under its literal `sota-v2` contract. Current +development is `sota-v3` because post-release fixes to numeric-input validation +and negotiation-window semantics change simulator behavior; the v2 evidence is +not retroactively rewritten or rerun. See the +[living publication-readiness log](docs/PUBLISH_READINESS.md) for the decision +boundary and remaining work. + GM-Bench includes: - A seeded league simulator with aging, development, injuries, contracts, @@ -64,7 +71,9 @@ full baseline panel; `GM_BENCH_PRIVATE_SEEDS` swaps in a private panel). Every run records usage telemetry — tokens, dollar cost (from `gm_bench/pricing.json` or adapter-reported cost), and per-decision latency — alongside scores. Results intended for serious frontier-model comparison should pass the stricter -`sota-v2` validator in [`docs/production_benchmark.md`](docs/production_benchmark.md). +current-contract validator (`sota-v3` on development HEAD) described in +[`docs/production_benchmark.md`](docs/production_benchmark.md). Frozen release +artifacts remain verifiable with `--policy sota-v2`. Runs also stamp a seed-panel hash so private held-out panels can be verified locally (integrity for a known panel, not secrecy). Use `redact-result` before publishing private-panel artifacts so the seed list is not committed. @@ -73,9 +82,9 @@ publishing private-panel artifacts so the seed list is not committed. OPENAI_API_KEY=... python -m gm_bench model --provider openai --model gpt-5.4 \ --preset leaderboard --repeats 3 --json > /tmp/openai-gpt-5.4.raw.json python -m gm_bench validate-result /tmp/openai-gpt-5.4.raw.json \ - --policy sota-v2 + --policy sota-v3 python -m gm_bench compact-result /tmp/openai-gpt-5.4.raw.json \ - --output results/leaderboard/openai-gpt-5.4.json --policy sota-v2 + --output /tmp/openai-gpt-5.4.compact.json --policy sota-v3 python -m gm_bench validate-contract python -m gm_bench calibrate-score --json python web/scripts/build_leaderboard.py # refresh web/src/data/leaderboard.json diff --git a/docs/PUBLISH_READINESS.md b/docs/PUBLISH_READINESS.md index f0bb121..cd41a5b 100644 --- a/docs/PUBLISH_READINESS.md +++ b/docs/PUBLISH_READINESS.md @@ -7,20 +7,25 @@ > to preserve this first draft; the goal is to make it more accurate as the > project develops. -**Last reviewed:** 2026-07-19 -**Current target:** Publish a validated `sota-v2` leaderboard, accompanying blog -post, GitHub release, and public site. -**Current state:** The frozen phase-one public panel is complete. Eight of ten +**Last reviewed:** 2026-07-24 +**Current target:** Preserve the published `sota-v2` study as frozen historical +evidence while hardening the next contract and finishing the public +presentation. +**Current state:** The frozen phase-one public panel, blog, GitHub release, and +results-first site are published. Eight of ten registered models produced route-matched, cost-complete `sota-v2` rows at the shared 4,096-token native-minimum-reasoning cap, clearing the predeclared minimum. Grok 4.5 and Mistral Medium 3.5 completed but remain diagnostic because their artifacts lacked complete usage or cost coverage. The generated site now exposes the eight eligible rows. All models overlap in one uncertainty tier, -and every eligible model trails `pick-trader`; the remaining release work is -claim review, raw public-trace packaging, presentation polish, and independent -reproduction rather than more phase-one model runs. -**Current weekly focus:** [#63 — Publication sprint: freeze and ship GM-Bench -`sota-v2`](https://github.com/nedcut/gm-bench/issues/63) +and every eligible model trails `pick-trader`. The immediate work is +[Issue #84](https://github.com/nedcut/gm-bench/issues/84): land the three P0 +correctness and artifact-integrity fixes as an explicit `sota-v3` contract, +keep all `sota-v2` evidence immutable, complete the remaining claim/framing +audit, and seek an independent reproduction. No additional paid model runs are +authorized or needed for this hardening step. +**Current weekly focus:** [#84 — Close the gap between `sota-v2` evidence and +framing](https://github.com/nedcut/gm-bench/issues/84) **Broader roadmap:** [#60 — Roadmap to a publishable leaderboard + blog post](https://github.com/nedcut/gm-bench/issues/60) @@ -81,8 +86,8 @@ The project is publish-ready only when all four gates pass. | Current model evidence | Public panel complete | Eight registered, route-matched, cost-complete `sota-v2` rows clear the publication floor; Grok and Mistral are retained as diagnostics. | | Statistical evidence | Ready but low-resolution | Seed-paired intervals, exact sign-flip tests, full-family Holm adjustment, and overlap tiers are generated. All eight rows share one tier and trail `pick-trader`. | | External validation | Missing | No independent reproduction or third-party result has been recorded. | -| GitHub presentation | Needs polish | Repository metadata, release packaging, README framing, and final site still need work. | -| Blog | Scaffolded | The validated results now exist; the narrative still needs its final evidence-backed update. | +| GitHub presentation | Strong | The results-first site, README, tagged evidence release, and clean-clone guide are public; the remaining work is consistency and disclosure polish. | +| Blog | Published, living | The evidence-backed phase-one findings are public and must retain post-release integrity notes as the implementation evolves. | ## Critical path @@ -522,6 +527,30 @@ These should be refined, not quietly removed: remains frozen long enough to build a meaningful panel. - Performance inside the benchmark may not transfer to other strategic domains. +## Post-publication contract hardening + +- [x] Publish and tag the frozen `sota-v2` evidence before changing simulator + semantics. +- [ ] Merge [#85](https://github.com/nedcut/gm-bench/pull/85) only after the + complete test, lint, contract, historical-artifact, analyzer, and site gates + pass on its exact head. +- [x] Reject non-finite action inputs and forbid non-finite publication JSON. +- [x] Make negotiation walk-aways persist for the full decision window. +- [x] Recompute compact-artifact statistics and verify raw-artifact hashes when + raw evidence is supplied. +- [x] Preserve a literal historical `sota-v2` validation contract rather than + reinterpreting released rows against current source. +- [x] Start corrected development at `sota-v3`; do not re-record old smokes, + silently unlock the old publication runner, or spend on a new panel without a + new pre-registered registry and lane. +- [ ] Add the post-release integrity disclosure to every public surface where a + reader might confuse the frozen v2 study with current development behavior. +- [ ] Complete Issue #84's claim/framing items in order of scientific impact: + scoring decomposition, same-view/scaffold comparisons, power/no-ranking + framing, memo ablation, then realism polish. +- [ ] Obtain and link an independent clean-clone reproduction. A closed issue + without a report is not evidence of reproduction. + ## What should wait until after publication - [ ] Keep [#62 — strategic contract mechanics](https://github.com/nedcut/gm-bench/pull/62) @@ -573,6 +602,7 @@ decision and why. | 2026-07-18 | Amend GLM 5.2 from the unhealthy first-party Z.AI FP8 endpoint to Novita FP8. | The frozen `z-ai/fp8` endpoint remained at OpenRouter status `-2` across repeated launch preflights, while the exact dated Novita FP8 endpoint was healthy and advertised the common lane parameters. No full-panel GLM result existed. | Pin `novita/fp8`, replace rather than reuse the Z.AI smoke entry, refresh route pricing/runtime evidence, and require a clean exact-route smoke before restoring panel unlock. The replacement smoke completed 4/4 decisions with zero failures or truncations for $0.009225. | | 2026-07-19 | Publish eight eligible phase-one rows without an ordinal winner claim. | Eight of ten registered cells passed exact-route and complete-cost gates. Grok recorded usage for 476/480 decisions and cost for 474/480; Mistral recorded cost for 479/480 after one fallback. All eight eligible seed-paired intervals form one connected tier, and every Holm-adjusted primary contrast is 0.078125. | Publish the eight eligible rows as one uncertainty tier, retain Grok and Mistral as diagnostics, do not rerun completed cells, and attach exact raw evidence plus checksums to the tagged release. | | 2026-07-18 | Reconcile the frozen publication protocol and reserve repair-call contingency before launch. | Independent Fable 5 review found that the runner and lane correctly enforced 4,096/3,072/8,192 native-minimum reasoning, but `publication_protocol.json` still described the retired 1,024/768/2,048 policy. It also noted that the prior reservation covered only primary calls even though one bounded repair is configured. No full-panel result existed. | Record the current lane as an explicit pre-data protocol amendment. Reserve one full-price call for every configured repair attempt and apply the committed 1.2x cost contingency before admitting each serial cell. Use a sub-$100 operator ceiling and monitor measured spend after every cell. | +| 2026-07-24 | Preserve the released `sota-v2` study and open `sota-v3` for the Issue #84 P0 fixes. | Non-finite input rejection and decision-window walk-away persistence change simulator/action semantics; compact-artifact recomputation strengthens the validator. Quietly changing the v2 fingerprint or re-recording old smokes would make the released contract mutable after results were known. | Pin the historical v2 contract and validator, identify current corrected runs as v3, block paid publication execution until a v3 registry and lane are pre-registered, retain the tagged v2 evidence and its narrow claim, and require no paid reruns for this repair. | ## Experiment and release log @@ -594,6 +624,8 @@ than pasting large outputs. | 2026-07-16 | First fixed-1,024 smoke series | Superseded by deliberate panel revision | `docs/run_logs/sota-v2-smokes-2026-07-16.md` | Six routes were accepted, two completed with protocol failures, Nemotron Nano exhausted infrastructure retries, and Claude direct remained unhealthy. The evidence remains auditable but cannot unlock the revised 4,096-token native-reasoning panel. | | 2026-07-15 | Statistical analysis plan | Frozen | `config/publication_protocol.json` | Pre-registered pre-data: unit of inference, primary paired contrast, Holm-Bonferroni multiplicity, descriptive inference labels, tiered ranking, power disclosure, temperature policy, and registry exclusion criteria. | | 2026-07-18 | Final Fable 5 launch audit | Conditions resolved pre-data | `docs/run_logs/sota-v2-final-launch-audit-2026-07-18.md` | No P0 blocker. Reconciled the stale output-policy text, strengthened reservations for repairs plus contingency, selected a $95 operator ceiling, and retained Tencent timing and per-cell spend monitoring as launch conditions. | +| 2026-07-24 | P0 integrity hardening and v3 boundary | In review | [#85](https://github.com/nedcut/gm-bench/pull/85) | Fixes non-finite actions, negotiation-window resets, and compact-artifact integrity without mutating the frozen v2 release contract. | +| 2026-07-24 | Results-first public site | Merged | [#87](https://github.com/nedcut/gm-bench/pull/87) | Reframes the public result around one unresolved model tier, the scripted-reference gap, compute, and auditability. | ## Living-document maintenance checklist diff --git a/docs/blog/sota-v2-findings.md b/docs/blog/sota-v2-findings.md index 2e2b4cc..f14c177 100644 --- a/docs/blog/sota-v2-findings.md +++ b/docs/blog/sota-v2-findings.md @@ -101,6 +101,39 @@ front office. actual model token use still ranged from 7,611 to 10,764 tokens per decision. - The result evaluates model-plus-standardized-scaffold systems on this task; it is not a claim about general intelligence or universal strategic ability. +- The published panel and scripted references are committed and public. That + makes replay easy, but it also permits benchmark-specific adaptation; future + generalization claims need a private or prospective panel. +- Opponents are scripted front-office policies, not autonomous learning GMs. + Morale is shown as decorative state but does not currently affect strength, + development, or score; readers should not treat it as a decision-relevant mechanic. + +## Decision history + +The protocol was frozen through an iterative process, not discovered whole. +The public [publish-readiness decision log](../PUBLISH_READINESS.md) records +panel reshuffles, cap-policy changes, route substitutions, and evidence resets. +“Frozen” describes the released protocol and artifact set; it does not erase +those pre-freeze researcher decisions. + +## Post-release integrity note (2026-07-24) + +A post-release code audit found three defects: non-finite numeric action inputs +could bypass comparison-based validation; negotiation walk-away counters reset +between interaction rounds instead of lasting for the full decision window; and +the routine compact-artifact validator trusted several derived statistics +instead of recomputing them from retained episode rows. The tagged release +manifest and checksums still bind the published files, and the committed compact +rows contain finite values. The compact rows do not preserve enough action-level +detail to prove retrospectively that no model benefited from the negotiation +reset. + +The repair therefore does not rewrite or silently “re-certify” `sota-v2`. +Released v2 artifacts remain frozen historical evidence under their literal +contract and the narrow claim below. Corrected simulator/action behavior and +the stronger artifact validator begin at `sota-v3`; a future v3 model panel +would require a new pre-registered lane and fresh evidence. No paid reruns are +part of this repair. ## Audit and reproduce diff --git a/docs/production_benchmark.md b/docs/production_benchmark.md index 6fd937f..2befd5f 100644 --- a/docs/production_benchmark.md +++ b/docs/production_benchmark.md @@ -1,26 +1,29 @@ # Production Benchmark Standard -GM-Bench has two result tiers: +GM-Bench has one general result tier plus versioned strict policies: - `public-leaderboard`: a structurally valid public leaderboard result. -- `sota-v2`: the stricter standard for claims about frontier model GM ability. +- `sota-v3`: the current strict validator for new development results. +- `sota-v2`: the frozen historical validator for the published phase-one study. - `sota-v1`: the frozen historical validator for archived v1 evidence. It does not make a v1 row comparable to v2; it only keeps the archived contract independently auditable. The public leaderboard can show development and diagnostic rows, including local -models that are below the scripted baselines. A `sota-v2` result is the minimum -bar for a result that should be compared as a serious model benchmark. +models that are below the scripted baselines. A current `sota-v3` result is the +minimum technical bar for a new serious comparison, but no v3 publication lane +or model registry is pre-registered yet. Do not spend on or publish a v3 panel +until those conditions are frozen. Committed official artifacts belong in `results/leaderboard/` and must pass the `public-leaderboard` validator in CI. Ineligible runs that are retained for transparency belong in `results/diagnostics/`. The site builder separates -current v2 rows from explicitly archived pre-v2 evidence while preserving the -official-artifact gate. +the frozen phase-one v2 rows from current development and explicitly archived +pre-v2 evidence while preserving the official-artifact gate. -## SOTA-v2 Requirements +## Strict result requirements -A `sota-v2` result must be produced by: +A new strict result on development HEAD is produced under `sota-v3`: ```bash python -m gm_bench model \ @@ -33,7 +36,7 @@ python -m gm_bench model \ Model / external-process adapters run **serially by default** (one episode at a time). Parallel fan-out across seeds×repeats will burn provider rate limits and -fill rows with fallback `noop`s, which then fails the sota-v2 failure-rate gate. +fill rows with fallback `noop`s, which then fails the strict failure-rate gate. Opt into concurrency only when the provider can handle it: `GM_BENCH_WORKERS=N` or `--workers N`. Scripted in-process baselines still parallelize. @@ -43,7 +46,7 @@ default — leave it unset or force `GM_BENCH_WORKERS=1` for Claude. On 2026-07- parallel Sonnet leaderboard panel emptied a Claude Pro 5h usage limit in ~5 minutes wall clock and produced a 0.873 decision failure rate. The multi-megabyte failed artifact is intentionally not retained. Prefer -`--preset smoke` first; a clean serial sota-v2 panel is +`--preset smoke` first; a clean serial strict panel is multi-hour quota spend, not a quick retry. Fresh-spawn serial model panels write an atomic checkpoint after every completed @@ -69,19 +72,19 @@ python -m gm_bench model \ --json > /tmp/gm-bench---private.raw.json python -m gm_bench redact-result \ /tmp/gm-bench---private.raw.json \ - --output results/leaderboard/--private.redacted.json \ - --policy sota-v2 + --output /tmp/gm-bench---private.redacted.json \ + --policy sota-v3 ``` It must also satisfy the machine validator: ```bash python -m gm_bench validate-result \ - results/leaderboard/-.json \ - --policy sota-v2 + /tmp/gm-bench--.json \ + --policy sota-v3 ``` -Before publishing SOTA-v2 claims from a new source contract, run the benchmark +Before publishing claims from a new source contract, run the benchmark validity canaries: ```bash @@ -124,7 +127,7 @@ The validator enforces: - Fresh-spawn condition: `run_info.session` must be absent or false. Session rows (`--session`, model keeps its full trajectory in context) are a separate labeled condition — publishable, but never comparable with memo-only rows and - never `sota-v2`. + never eligible for the active strict policy. - Scaffold provenance: new rows record `run_info.scaffold_fingerprint`, a per-provider hash of the prompt scaffold (shared prompt builder plus the provider's adapter script and spec). A recorded fingerprint that does not @@ -225,10 +228,11 @@ still. Under the freeze: -- New model rows run against the frozen contract; `validate-result` already - rejects rows whose fingerprint does not match the current source. +- The released v2 rows validate against a literal historical contract rather + than the current source fingerprint. Reproduce them from the tagged release; + current HEAD emits v3. - Simulator, scoring, preset, or schema changes that alter the fingerprint do - not amend `sota-v2` — they start a new claim lane (`sota-v3`) with its own + not amend `sota-v2` — the Issue #84 fixes started `sota-v3`, which needs its own re-cached baseline panel and reference means, exactly as `sota-v2` superseded `sota-v1`. Existing `sota-v2` rows stay published and comparable with each other under their own contract. diff --git a/docs/submitting_results.md b/docs/submitting_results.md index 06e0d3f..168a41f 100644 --- a/docs/submitting_results.md +++ b/docs/submitting_results.md @@ -5,7 +5,13 @@ true for it to pass the machine validator. Every requirement below is enforced b `gm_bench/official.py` (`validate_leaderboard_payload`); nothing here is aspirational unless it is called out as a convention. Read [production_benchmark.md](production_benchmark.md) first for the two result tiers -(`public-leaderboard`, `sota-v2`) and the contract freeze. +(`public-leaderboard`, strict versioned policies) and the contract freeze. + +> **Current status:** the public site is the frozen `sota-v2` phase-one study. +> Development HEAD emits `sota-v3`, but a v3 publication registry and lane have +> not been pre-registered. New strict submissions should therefore remain local +> or diagnostic until that protocol is frozen; do not add a v3 row to +> `results/leaderboard/` and expect the v2 site to publish it. ## Produce the row @@ -30,11 +36,11 @@ For a contamination-resistant private-panel row, set a held-out panel first, kee the raw JSON local, and publish only the redacted artifact: ```bash -export GM_BENCH_PRIVATE_SEEDS="101,102,110-115" # >= 8 seeds for sota-v2 +export GM_BENCH_PRIVATE_SEEDS="101,102,110-115" # >= 8 seeds for the strict policy python -m gm_bench model --provider

--model \ --preset leaderboard --repeats 3 --json > /tmp/

--private.raw.json python -m gm_bench redact-result /tmp/

--private.raw.json \ - --output results/leaderboard/

--private.redacted.json --policy sota-v2 + --output /tmp/

--private.redacted.json --policy sota-v3 ``` `redact-result` writes the output file **only if the selected policy passes**; an @@ -45,9 +51,9 @@ per-episode detail, and `paired.per_seed` rows. ## Validate before you submit ```bash -python3 -m gm_bench validate-result /tmp/.raw.json --policy sota-v2 +python3 -m gm_bench validate-result /tmp/.raw.json --policy sota-v3 python3 -m gm_bench compact-result /tmp/.raw.json \ - --output results/leaderboard/.json --policy sota-v2 + --output /tmp/.compact.json --policy sota-v3 ``` Use `--policy public-leaderboard` for a development/diagnostic row. Exit code is @@ -73,9 +79,9 @@ Both policies require these; the values are read straight from the payload: - Every candidate episode present exactly once per seed/repeat, each with `seasons == 5`; each baseline episode present once per seed. -`public-leaderboard` is lenient where `sota-v2` is strict: +`public-leaderboard` is lenient where the current `sota-v3` policy is strict: -| Check | `public-leaderboard` | `sota-v2` | +| Check | `public-leaderboard` | `sota-v3` | |---|---|---| | Candidate repeats | ≥ 1 | ≥ 3 | | Seed count | ≥ 1 | ≥ 8 (full leaderboard panel) | @@ -84,10 +90,10 @@ Both policies require these; the values are read straight from the payload: | Seed-panel provenance | warning if missing | **required** | | Baseline panel | any known subset, no dupes | **exact** full panel: `random`, `conservative`, `win-now`, `rebuild`, `value`, `shrewd`, `strategic`, `pick-trader` | -For `sota-v2`, `run_info.benchmark_contract` must match `expected_contract()` field -for field — including `contract_fingerprint`, frozen at `558e8f35ea1d66b9`. A row -built against a different simulator/scoring/schema source is rejected, not merely -flagged. +For `sota-v3`, `run_info.benchmark_contract` must match `expected_contract()` +field for field. The historical `sota-v2` policy instead matches the literal +released contract, including fingerprint `558e8f35ea1d66b9`. A row built against +a different simulator/scoring/schema source is rejected, not merely flagged. Seed-panel provenance (`run_info.seed_panel`) must name one of two identities; `custom` panels are rejected outright: @@ -101,7 +107,7 @@ Seed-panel provenance (`run_info.seed_panel`) must name one of two identities; These are recorded as warnings, keep the row eligible, and surface on the site (`sota_v2_issues`): illegal actions present, failed-query rate above 0.25 -(misfired scout/inspect lookups; rates above 1.0 are a hard `sota-v2` error, +(misfired scout/inspect lookups; rates above 1.0 are a hard strict-policy error, not a warning), any adapter fallback/error decisions, lift not significant at 95%, candidate not beating the strongest baseline, or the strongest baseline not being `pick-trader`. @@ -131,7 +137,7 @@ traces would leak the held-out seeds. ## Eligible vs diagnostic -The site builder (`web/scripts/build_leaderboard.py`) re-runs the `sota-v2` +The phase-one site builder (`web/scripts/build_leaderboard.py`) re-runs the frozen `sota-v2` validator on every artifact under `results/leaderboard/` and sets `sota_v2_eligible` from `report.ok`, carrying all errors and warnings into `sota_v2_issues`. A row is: diff --git a/examples/gm_agent_common.py b/examples/gm_agent_common.py index 4b69edd..b6a9dfa 100644 --- a/examples/gm_agent_common.py +++ b/examples/gm_agent_common.py @@ -233,7 +233,7 @@ def parse_actions(text: Any) -> list[dict[str, Any]]: raise ValueError(f"model response content must be a string, got {type(text).__name__}") stripped = strip_terminal_codes(text).strip() try: - parsed = json.loads(stripped) + parsed = json.loads(stripped, parse_constant=_reject_non_finite_json_constant) return _actions_from_json(parsed) except json.JSONDecodeError: pass @@ -241,7 +241,7 @@ def parse_actions(text: Any) -> list[dict[str, Any]]: fenced = re.search(r"```(?:json)?\s*(.*?)\s*```", stripped, re.DOTALL) if fenced: try: - parsed = json.loads(fenced.group(1)) + parsed = json.loads(fenced.group(1), parse_constant=_reject_non_finite_json_constant) return _actions_from_json(parsed) except json.JSONDecodeError: pass @@ -260,6 +260,10 @@ def parse_actions(text: Any) -> list[dict[str, Any]]: raise ValueError("model did not return a JSON action array") +def _reject_non_finite_json_constant(value: str) -> None: + raise ValueError(f"non-finite JSON number {value!r} is not allowed") + + def _actions_from_json(parsed: Any) -> list[dict[str, Any]]: if isinstance(parsed, dict) and isinstance(parsed.get("actions"), list): parsed = parsed["actions"] @@ -327,7 +331,7 @@ def _has_usable_value(value: Any) -> bool: def _scan_json_values(text: str, opener: str) -> list[Any]: - decoder = json.JSONDecoder() + decoder = json.JSONDecoder(parse_constant=_reject_non_finite_json_constant) values: list[Any] = [] for match in re.finditer(re.escape(opener), text): try: diff --git a/gm_bench/cli.py b/gm_bench/cli.py index 4aa979d..baccc31 100644 --- a/gm_bench/cli.py +++ b/gm_bench/cli.py @@ -31,7 +31,7 @@ from gm_bench.official import ( POLICIES, PUBLIC_LEADERBOARD_POLICY, - SOTA_V2_POLICY, + SOTA_V3_POLICY, redact_leaderboard_payload, validate_leaderboard_payload, ) @@ -235,7 +235,7 @@ def main(argv: list[str] | None = None) -> None: redact_parser.add_argument( "--policy", choices=sorted(POLICIES), - default=SOTA_V2_POLICY.name, + default=SOTA_V3_POLICY.name, help="validation policy to record before redaction", ) @@ -246,7 +246,7 @@ def main(argv: list[str] | None = None) -> None: compact_parser.add_argument("path", help="raw model result JSON") compact_parser.add_argument("--output", required=True, help="path for the compact publication artifact") compact_parser.add_argument( - "--policy", choices=sorted(POLICIES), default=SOTA_V2_POLICY.name, help="policy that must pass before writing" + "--policy", choices=sorted(POLICIES), default=SOTA_V3_POLICY.name, help="policy that must pass before writing" ) validate_contract_parser = subparsers.add_parser( diff --git a/gm_bench/contract.py b/gm_bench/contract.py index 60a7cc9..2425e55 100644 --- a/gm_bench/contract.py +++ b/gm_bench/contract.py @@ -15,11 +15,21 @@ from gm_bench.scoring import SCORING_VERSION, scoring_scale_fingerprint -BENCHMARK_VERSION = "sota-v2" -ACTION_PROTOCOL_VERSION = "actions-v2" -SIMULATOR_VERSION = "sim-v2" +BENCHMARK_VERSION = "sota-v3" +ACTION_PROTOCOL_VERSION = "actions-v3" +SIMULATOR_VERSION = "sim-v3" OBSERVATION_VERSION = "observation-v1" +SOTA_V2_CONTRACT = { + "benchmark_version": "sota-v2", + "action_protocol_version": "actions-v2", + "scoring_version": "score-v1", + "scoring_scale_fingerprint": "05a60ff4f691e734", + "simulator_version": "sim-v2", + "observation_version": "observation-v1", + "contract_fingerprint": "558e8f35ea1d66b9", +} + _ROOT = Path(__file__).resolve().parents[1] # Fingerprint covers score-affecting simulator/protocol sources only. # Pricing/telemetry (gm_bench/pricing.json, gm_bench/telemetry.py) and diff --git a/gm_bench/official.py b/gm_bench/official.py index e4ffbb6..8137c8c 100644 --- a/gm_bench/official.py +++ b/gm_bench/official.py @@ -3,6 +3,7 @@ from __future__ import annotations import copy +import math import os import re from dataclasses import dataclass @@ -17,10 +18,11 @@ _parse_seeds, seed_panel_hash, ) -from gm_bench.contract import expected_contract, scaffold_fingerprint +from gm_bench.contract import SOTA_V2_CONTRACT, expected_contract, scaffold_fingerprint PUBLIC_LEADERBOARD_POLICY_NAME = "public-leaderboard" SOTA_V2_POLICY_NAME = "sota-v2" +SOTA_V3_POLICY_NAME = "sota-v3" OUTPUT_BUDGET_SWEEP_POLICY_NAME = "output-budget-sweep" SOTA_V1_POLICY_NAME = "sota-v1" ARCHIVE_V1_POLICY_NAME = "archive-v1" @@ -71,6 +73,18 @@ class ResultPolicy: require_contract_provenance=True, require_seed_panel_provenance=True, require_scaffold_provenance=True, + expected_contract=SOTA_V2_CONTRACT, + validate_current_scaffold=False, + max_failed_query_rate=1.0, +) +SOTA_V3_POLICY = ResultPolicy( + name=SOTA_V3_POLICY_NAME, + min_repeats=3, + min_seed_count=len(PRESETS["leaderboard"]["seeds"]), + max_decision_failure_rate=0.02, + require_contract_provenance=True, + require_seed_panel_provenance=True, + require_scaffold_provenance=True, expected_contract=expected_contract(), max_failed_query_rate=1.0, ) @@ -85,7 +99,8 @@ class ResultPolicy: require_contract_provenance=True, require_seed_panel_provenance=True, require_scaffold_provenance=True, - expected_contract=expected_contract(), + expected_contract=SOTA_V2_CONTRACT, + validate_current_scaffold=False, max_failed_query_rate=1.0, ) SOTA_V1_POLICY = ResultPolicy( @@ -122,6 +137,7 @@ class ResultPolicy: OUTPUT_BUDGET_SWEEP_POLICY.name: OUTPUT_BUDGET_SWEEP_POLICY, SOTA_V1_POLICY.name: SOTA_V1_POLICY, SOTA_V2_POLICY.name: SOTA_V2_POLICY, + SOTA_V3_POLICY.name: SOTA_V3_POLICY, ARCHIVE_V1_POLICY.name: ARCHIVE_V1_POLICY, } REDACTED_SEEDS_SENTINEL = "" @@ -205,11 +221,15 @@ def validate_leaderboard_payload( ) elif run_info.get("scaffold_fingerprint"): warnings.append("historical scaffold fingerprint retained but cannot be re-derived from current source") - strict_v2_lane = policy.name in {SOTA_V2_POLICY_NAME, OUTPUT_BUDGET_SWEEP_POLICY_NAME} + strict_sota_lane = policy.name in { + SOTA_V2_POLICY_NAME, + SOTA_V3_POLICY_NAME, + OUTPUT_BUDGET_SWEEP_POLICY_NAME, + } if run_info.get("session"): - if strict_v2_lane: + if strict_sota_lane: errors.append( - "sota-v2 rows must be fresh-spawn (memo-only memory); " + f"{policy.name} rows must be fresh-spawn (memo-only memory); " "session-condition rows are a separate lane and not comparable" ) else: @@ -217,13 +237,13 @@ def validate_leaderboard_payload( "session-condition row: model retains full trajectory in context; " "not comparable with fresh-spawn rows" ) - if strict_v2_lane: + if strict_sota_lane: repair_attempts = run_info.get("protocol_repair_attempts") option_repair = (run_info.get("provider_options") or {}).get("GM_BENCH_PROTOCOL_REPAIR_ATTEMPTS") parsed_repairs: dict[str, int] = {} for label, raw in (("protocol_repair_attempts", repair_attempts), ("provider_options", option_repair)): if raw in (None, ""): - errors.append(f"run_info.{label} repair attempts are required for sota-v2") + errors.append(f"run_info.{label} repair attempts are required for {policy.name}") continue try: parsed = int(raw) @@ -231,7 +251,9 @@ def validate_leaderboard_payload( errors.append(f"run_info.{label} repair attempts must be an integer") continue if not 0 <= parsed <= 1: - errors.append(f"sota-v2 repair attempts must be between zero and one; got {parsed} via {label}") + errors.append( + f"{policy.name} repair attempts must be between zero and one; got {parsed} via {label}" + ) continue parsed_repairs[label] = parsed if len(parsed_repairs) == 2 and len(set(parsed_repairs.values())) != 1: @@ -259,7 +281,7 @@ def validate_leaderboard_payload( baselines = [_dict(result) for result in _list(payload.get("baselines"))] baseline_names = [result.get("agent") for result in baselines] - if policy.name in {SOTA_V2_POLICY_NAME, OUTPUT_BUDGET_SWEEP_POLICY_NAME}: + if policy.name in {SOTA_V2_POLICY_NAME, SOTA_V3_POLICY_NAME, OUTPUT_BUDGET_SWEEP_POLICY_NAME}: _expect_equal(errors, "baselines", baseline_names, expected_baselines) else: if not baseline_names: @@ -322,7 +344,7 @@ def validate_leaderboard_payload( warnings, run_info, usage, - strict=policy.name in {SOTA_V2_POLICY_NAME, OUTPUT_BUDGET_SWEEP_POLICY_NAME}, + strict=policy.name in {SOTA_V2_POLICY_NAME, SOTA_V3_POLICY_NAME, OUTPUT_BUDGET_SWEEP_POLICY_NAME}, ) for baseline in baselines: @@ -332,6 +354,9 @@ def validate_leaderboard_payload( elif expected_seeds is not None: _validate_episode_panel(errors, f"baseline[{name}]", baseline, expected_seeds, expected_seasons, repeats=1) + if isinstance(payload.get("publication"), dict) and not redacted_private: + _validate_compact_integrity(errors, payload, candidate, baselines, expected_seeds or []) + normalized = _dict(payload.get("normalized")) paired = _dict(payload.get("paired")) if not normalized: @@ -400,7 +425,7 @@ def _validate_openrouter_route( def redact_leaderboard_payload( payload: dict[str, Any], *, - policy: ResultPolicy = SOTA_V2_POLICY, + policy: ResultPolicy = SOTA_V3_POLICY, ) -> tuple[dict[str, Any], ValidationReport]: """Return a public-safe copy of a leaderboard payload. @@ -656,6 +681,69 @@ def _validate_episode_panel( errors.append(f"{label}.episodes missing seed/repeat pairs: {missing}") +def _validate_compact_integrity( + errors: list[str], + payload: dict[str, Any], + candidate: dict[str, Any], + baselines: list[dict[str, Any]], + seeds: list[int], +) -> None: + """Treat compact summaries as derived views, never independent evidence.""" + _validate_finite_numbers(errors, payload) + if not candidate or not baselines or not seeds: + return + from gm_bench.runner import _paired_analysis, _precise_mean_score, summarize_episodes + + rebuilt_candidate = {**candidate, "summary": summarize_episodes(candidate.get("episodes") or [])} + rebuilt_baselines = [ + {**baseline, "summary": summarize_episodes(baseline.get("episodes") or [])} for baseline in baselines + ] + _compare_present_values(errors, "candidate.summary", candidate.get("summary"), rebuilt_candidate["summary"]) + for original, rebuilt in zip(baselines, rebuilt_baselines, strict=True): + _compare_present_values( + errors, f"baseline[{original.get('agent', 'unknown')}].summary", original.get("summary"), rebuilt["summary"] + ) + baseline_mean = sum(_precise_mean_score(block) for block in rebuilt_baselines) / len(rebuilt_baselines) + candidate_mean = _precise_mean_score(rebuilt_candidate) + expected_normalized = { + "candidate_mean_score": round(candidate_mean, 3), + "baseline_panel_mean_score": round(baseline_mean, 3), + "score_lift": round(candidate_mean - baseline_mean, 3), + "score_lift_pct": round(((candidate_mean / baseline_mean) - 1.0) * 100.0, 2) if baseline_mean else 0.0, + "candidate_illegal_actions": rebuilt_candidate["summary"]["illegal_actions"], + "baseline_illegal_actions": sum(block["summary"]["illegal_actions"] for block in rebuilt_baselines), + } + _compare_present_values(errors, "normalized", payload.get("normalized"), expected_normalized) + _compare_present_values( + errors, "paired", payload.get("paired"), _paired_analysis(seeds, rebuilt_candidate, rebuilt_baselines) + ) + + +def _validate_finite_numbers(errors: list[str], value: Any, path: str = "payload") -> None: + if isinstance(value, float) and not math.isfinite(value): + errors.append(f"{path} must not contain a non-finite number") + elif isinstance(value, dict): + for key, child in value.items(): + _validate_finite_numbers(errors, child, f"{path}.{key}") + elif isinstance(value, list): + for index, child in enumerate(value): + _validate_finite_numbers(errors, child, f"{path}[{index}]") + + +def _compare_present_values(errors: list[str], path: str, actual: Any, expected: Any) -> None: + if not isinstance(actual, dict) or not isinstance(expected, dict): + return + for key, value in actual.items(): + if key not in expected: + continue + expected_value = expected[key] + child_path = f"{path}.{key}" + if isinstance(value, dict) and isinstance(expected_value, dict): + _compare_present_values(errors, child_path, value, expected_value) + elif value != expected_value: + errors.append(f"{child_path} does not match episode-derived value") + + def _expect_equal(errors: list[str], name: str, actual: Any, expected: Any) -> None: if actual != expected: errors.append(f"{name} must be {expected!r}, got {actual!r}") diff --git a/gm_bench/publication.py b/gm_bench/publication.py index ca5c224..bbbef21 100644 --- a/gm_bench/publication.py +++ b/gm_bench/publication.py @@ -17,7 +17,7 @@ def canonical_sha256(payload: dict[str, Any]) -> str: - encoded = json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode() + encoded = json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=False, allow_nan=False).encode() return hashlib.sha256(encoded).hexdigest() @@ -83,14 +83,19 @@ def smoke_manifest_issues( manifest: dict[str, Any] | None, registry: dict[str, Any], lane: dict[str, Any], + *, + expected_contract_fingerprint: str | None = None, + validate_current_scaffold: bool = True, ) -> list[str]: """Machine-check the pre-panel smoke evidence against the frozen lane. The full panel (and any published ranking) must not be unlockable by editing a status string: every registered model needs an accepted smoke manifest entry recorded from a real artifact, at the frozen cap, under the - current scaffold and contract, with complete finish-reason telemetry and - no cap-pressure or truncation trigger. + selected scaffold and contract, with complete finish-reason telemetry and + no cap-pressure or truncation trigger. Historical release builders may pass + their frozen contract fingerprint and disable current-scaffold comparison; + new publication runners retain the current-source defaults. """ from gm_bench.benchmark_config import PRESETS from gm_bench.contract import contract_fingerprint, scaffold_fingerprint @@ -172,11 +177,13 @@ def smoke_manifest_issues( not isinstance(reasoning_tokens, int) or isinstance(reasoning_tokens, bool) or reasoning_tokens < 0 ): issues.append(f"{prefix} is missing reasoning-token telemetry for a mandatory-reasoning model") - if entry.get("contract_fingerprint") != contract_fingerprint(): + expected_contract = expected_contract_fingerprint or contract_fingerprint() + if entry.get("contract_fingerprint") != expected_contract: issues.append(f"{prefix} was recorded under a different benchmark contract") - expected_scaffold = scaffold_fingerprint(str(model.get("provider") or "")) - if expected_scaffold is not None and entry.get("scaffold_fingerprint") != expected_scaffold: - issues.append(f"{prefix} was recorded under a different prompt scaffold") + if validate_current_scaffold: + expected_scaffold = scaffold_fingerprint(str(model.get("provider") or "")) + if expected_scaffold is not None and entry.get("scaffold_fingerprint") != expected_scaffold: + issues.append(f"{prefix} was recorded under a different prompt scaffold") artifact_sha = entry.get("artifact_sha256") if not isinstance(artifact_sha, str) or re.fullmatch(r"[0-9a-f]{64}", artifact_sha) is None: issues.append(f"{prefix} must record the raw artifact sha256") diff --git a/gm_bench/runner.py b/gm_bench/runner.py index 6dc30fd..6262d2a 100644 --- a/gm_bench/runner.py +++ b/gm_bench/runner.py @@ -166,6 +166,9 @@ def run_decision_point( timed into harness latency, usages are collected, and the decision is marked failed if ANY round's actions fail. """ + # Interaction rounds are one decision window. Negotiation walk-aways must + # survive query follow-ups and reopen only at the next decision point. + league.begin_decision_window() tier = _observation_tier_for_agent(agent, config) action_results: list[dict[str, Any]] | None = None last_results: list[dict[str, Any]] = [] diff --git a/gm_bench/simulator.py b/gm_bench/simulator.py index a15659a..928e86d 100644 --- a/gm_bench/simulator.py +++ b/gm_bench/simulator.py @@ -199,9 +199,6 @@ def _available_actions(self, phase: str) -> list[str]: return actions def apply_actions(self, actions: list[dict[str, Any]], phase: str) -> list[ActionResult]: - # Walk-aways last one decision window: each call to apply_actions is one - # window, so counterparties who broke off talks come back next window. - self.window_walkaways = {} self._action_results = [] if not isinstance(actions, list): self._record({}, phase, False, "agent response must be a list of actions") @@ -217,6 +214,10 @@ def apply_actions(self, actions: list[dict[str, Any]], phase: str) -> list[Actio self._dispatch_action(action_type, action, phase) return self._action_results + def begin_decision_window(self) -> None: + """Reset negotiation walk-aways once before an agent decision.""" + self.window_walkaways = {} + def _dispatch_action(self, action_type: str, action: dict[str, Any], phase: str) -> ActionResult: if action_type == "noop": return self._record(action, phase, True, "no-op", penalize=False) @@ -488,7 +489,12 @@ def _inspect_player(self, action: dict[str, Any], phase: str) -> ActionResult: def _list_free_agents(self, action: dict[str, Any], phase: str) -> ActionResult: position = action.get("position") - min_overall = float(action.get("min_overall", 0.0)) + try: + min_overall = float(action.get("min_overall", 0.0)) + except (TypeError, ValueError): + return self._record(action, phase, False, "min_overall must be a finite number", penalize=False) + if not math.isfinite(min_overall): + return self._record(action, phase, False, "min_overall must be a finite number", penalize=False) limit = min(24, max(1, int(action.get("limit", 12)))) players = [self.players[pid] for pid in self.free_agents] if position in {"F", "D", "G"}: @@ -510,9 +516,13 @@ def _memo(self, action: dict[str, Any], phase: str) -> None: self._record({**action, "text": memo}, phase, True, "memo saved") def _sign_free_agent(self, action: dict[str, Any], phase: str) -> None: - player_id = int(action.get("player_id", -1)) - years = int(action.get("years", 1)) - salary = float(action.get("salary", 0.0)) + try: + player_id = int(action.get("player_id", -1)) + years = int(action.get("years", 1)) + salary = float(action.get("salary", 0.0)) + except (TypeError, ValueError): + self._record(action, phase, False, "signing fields must be valid numbers") + return if player_id not in self.free_agents or player_id not in self.players: self._record(action, phase, False, "player is not an available free agent") return @@ -521,7 +531,7 @@ def _sign_free_agent(self, action: dict[str, Any], phase: str) -> None: return # A non-positive salary is not a negotiable bid — it is a malformed # action, so it must not ride the penalty-free rejected-offer path. - if salary <= 0: + if not math.isfinite(salary) or salary <= 0: self._record(action, phase, False, "salary must be a positive amount") return player = self.players[player_id] diff --git a/scripts/analyze_publication_panel.py b/scripts/analyze_publication_panel.py index 5748d22..d48a4c0 100644 --- a/scripts/analyze_publication_panel.py +++ b/scripts/analyze_publication_panel.py @@ -309,12 +309,18 @@ def _registered_lane_issues( return issues -def analyze(registry: Mapping[str, Any], payloads: Sequence[Mapping[str, Any]]) -> dict[str, Any]: +def analyze( + registry: Mapping[str, Any], + payloads: Sequence[Mapping[str, Any]], + *, + raw_payloads: Sequence[Mapping[str, Any]] = (), +) -> dict[str, Any]: """Analyze valid artifacts matching the registry and report missing/rejected rows.""" specs, config_errors = _registry_specs(registry) specs_by_identity = {(spec["provider"], spec["model"]): spec for spec in specs} candidates: dict[str, list[dict[str, Any]]] = {} rejected: list[dict[str, Any]] = [] + raw_by_hash = {canonical_sha256(dict(raw)): raw for raw in raw_payloads} for index, payload in enumerate(payloads): artifact_sha256 = canonical_sha256(dict(payload)) @@ -334,6 +340,8 @@ def analyze(registry: Mapping[str, Any], payloads: Sequence[Mapping[str, Any]]) not isinstance(raw_artifact_sha256, str) or re.fullmatch(r"[0-9a-f]{64}", raw_artifact_sha256) is None ): reasons.append("publication.raw_artifact_sha256 must be a 64-character lowercase hex digest") + elif raw_artifact_sha256 is not None and raw_by_hash and raw_artifact_sha256 not in raw_by_hash: + reasons.append("publication.raw_artifact_sha256 does not match any supplied raw artifact") try: per_seed = per_seed_pick_trader_lifts( payload, @@ -426,6 +434,7 @@ def main() -> int: parser.add_argument("artifacts", nargs="*", type=Path) parser.add_argument("--registry", type=Path, default=REGISTRY_PATH) parser.add_argument("--artifacts-dir", type=Path, default=DEFAULT_ARTIFACT_DIR) + parser.add_argument("--raw-artifacts-dir", type=Path, help="raw JSON evidence used to verify compact hash links") parser.add_argument("--output", type=Path, default=DEFAULT_OUTPUT_PATH) args = parser.parse_args() @@ -433,10 +442,15 @@ def main() -> int: try: registry = _read_json(args.registry) payloads = [_read_json(path) for path in artifact_paths] + raw_payloads = ( + [_read_json(path) for path in sorted(args.raw_artifacts_dir.glob("*.json"))] + if args.raw_artifacts_dir + else [] + ) except (OSError, ValueError, json.JSONDecodeError) as exc: sys.exit(f"analyze_publication_panel: {exc}") - result = analyze(registry, payloads) + result = analyze(registry, payloads, raw_payloads=raw_payloads) text = json.dumps(result, indent=2, sort_keys=True) + "\n" if result["eligible_model_count"] == 0: print(text, end="") diff --git a/scripts/run_publication_matrix.py b/scripts/run_publication_matrix.py index db38f3e..557ffb9 100644 --- a/scripts/run_publication_matrix.py +++ b/scripts/run_publication_matrix.py @@ -37,9 +37,9 @@ sys.path.insert(0, str(ROOT)) from gm_bench.benchmark_config import PRESETS # noqa: E402 -from gm_bench.contract import contract_fingerprint, scaffold_fingerprint # noqa: E402 +from gm_bench.contract import BENCHMARK_VERSION, contract_fingerprint, scaffold_fingerprint # noqa: E402 from gm_bench.environment import load_environment_files # noqa: E402 -from gm_bench.official import SOTA_V2_POLICY, validate_leaderboard_payload # noqa: E402 +from gm_bench.official import POLICIES, validate_leaderboard_payload # noqa: E402 from gm_bench.protocol import PHASES # noqa: E402 from gm_bench.publication import SMOKE_MANIFEST_FORMAT, smoke_manifest_issues # noqa: E402 @@ -71,6 +71,17 @@ def _read_json(path: Path) -> dict[str, Any]: return payload +def _require_current_publication_contract() -> None: + registry_contract = _read_json(PANEL_CONFIG).get("contract") + lane_contract = _read_json(LANE_CONFIG).get("contract") + if registry_contract != BENCHMARK_VERSION or lane_contract != BENCHMARK_VERSION: + raise ValueError( + "the committed publication registry/lane are frozen historical evidence " + f"({registry_contract!r}/{lane_contract!r}); current code is {BENCHMARK_VERSION!r}. " + "Create and pre-register a new contract lane before recording smokes or running a panel." + ) + + def _validate_models(models: list[dict[str, Any]], *, exact_routes: bool = True) -> None: if not models: raise ValueError("publication matrix contains no models") @@ -935,7 +946,7 @@ def _reusable_smoke_artifact(cell: Cell, run_dir: Path) -> Path | None: def _panel_artifact_issues(cell: Cell, artifact: dict[str, Any]) -> list[str]: """Return blocking publication and registered-route issues for one panel artifact.""" - issues = list(validate_leaderboard_payload(artifact, policy=SOTA_V2_POLICY).errors) + issues = list(validate_leaderboard_payload(artifact, policy=POLICIES[BENCHMARK_VERSION]).errors) run_info = artifact.get("run_info") run_info = run_info if isinstance(run_info, dict) else {} for key, expected in ( @@ -1249,9 +1260,18 @@ def main(argv: list[str] | None = None) -> int: parser.error("record-smoke requires --model-id") if args.artifact is None: parser.error("record-smoke requires --artifact") + try: + _require_current_publication_contract() + except (OSError, ValueError, json.JSONDecodeError) as exc: + parser.error(str(exc)) return _record_smoke(args.model_id, args.artifact, args.manifest) if args.max_spend_usd is not None and args.max_spend_usd <= 0: parser.error("--max-spend-usd must be positive") + if args.phase == "panel" or (args.phase == "smoke" and not args.dry_run and not args.preflight_only): + try: + _require_current_publication_contract() + except (OSError, ValueError, json.JSONDecodeError) as exc: + parser.error(str(exc)) try: cells = build_cells(args.phase, args.model_id, args.cap) except (OSError, ValueError, json.JSONDecodeError) as exc: diff --git a/tests/test_action_validation.py b/tests/test_action_validation.py index c11b861..a73dc75 100644 --- a/tests/test_action_validation.py +++ b/tests/test_action_validation.py @@ -2,8 +2,12 @@ from __future__ import annotations +import json import random +import pytest + +from examples.gm_agent_common import parse_actions from gm_bench.simulator import League @@ -51,3 +55,26 @@ def test_trade_rejected_when_give_value_too_low() -> None: assert league.illegal_actions == illegal_before assert league.rejected_offers > rejected_before assert league.transactions[-1].accepted is False + + +@pytest.mark.parametrize("value", ["NaN", "Infinity", "-Infinity"]) +def test_non_finite_action_numbers_are_rejected_at_parser_and_simulator_boundaries(value: str) -> None: + with pytest.raises(ValueError, match="non-finite"): + parse_actions(f'{{"actions":[{{"type":"sign_free_agent","salary":{value}}}]}}') + + league = League.new(seed=7) + player_id = league.free_agents[0] + league.apply_actions( + [{"type": "sign_free_agent", "player_id": player_id, "years": 1, "salary": json.loads(value)}], "preseason" + ) + assert not league.transactions[-1].accepted + assert player_id in league.free_agents + assert league.user_team.roster.count(player_id) == 0 + + +@pytest.mark.parametrize("value", [float("nan"), float("inf"), float("-inf")]) +def test_non_finite_query_threshold_is_rejected(value: float) -> None: + league = League.new(seed=7) + league.apply_actions([{"type": "list_free_agents", "min_overall": value}], "preseason") + assert not league.transactions[-1].accepted + assert "finite number" in league.transactions[-1].message diff --git a/tests/test_contract_versioning.py b/tests/test_contract_versioning.py index f2d851b..ac8098b 100644 --- a/tests/test_contract_versioning.py +++ b/tests/test_contract_versioning.py @@ -9,35 +9,42 @@ ACTION_PROTOCOL_VERSION, BENCHMARK_VERSION, SIMULATOR_VERSION, + SOTA_V2_CONTRACT, benchmark_contract, ) from gm_bench.official import ( POLICIES, SOTA_V1_POLICY, SOTA_V2_POLICY, + SOTA_V3_POLICY, validate_leaderboard_payload, ) -def test_contract_reports_v2_version_strings() -> None: - assert BENCHMARK_VERSION == "sota-v2" - assert ACTION_PROTOCOL_VERSION == "actions-v2" - assert SIMULATOR_VERSION == "sim-v2" +def test_contract_reports_v3_version_strings() -> None: + assert BENCHMARK_VERSION == "sota-v3" + assert ACTION_PROTOCOL_VERSION == "actions-v3" + assert SIMULATOR_VERSION == "sim-v3" contract = benchmark_contract() - assert contract["benchmark_version"] == "sota-v2" - assert contract["action_protocol_version"] == "actions-v2" - assert contract["simulator_version"] == "sim-v2" - # Scoring scale is deliberately unchanged in v2. + assert contract["benchmark_version"] == "sota-v3" + assert contract["action_protocol_version"] == "actions-v3" + assert contract["simulator_version"] == "sim-v3" + # The P0 fixes change action/simulator semantics, not the scoring scale. assert contract["scoring_version"] == "score-v1" def test_current_and_historical_sota_policies_are_distinct() -> None: assert SOTA_V2_POLICY.name == "sota-v2" assert POLICIES["sota-v2"] is SOTA_V2_POLICY + assert SOTA_V3_POLICY.name == "sota-v3" + assert POLICIES["sota-v3"] is SOTA_V3_POLICY assert SOTA_V1_POLICY.name == "sota-v1" assert POLICIES["sota-v1"] is SOTA_V1_POLICY assert SOTA_V1_POLICY is not SOTA_V2_POLICY assert SOTA_V1_POLICY.expected_contract["contract_fingerprint"] == "cf2607e59dba0c7f" + assert SOTA_V2_POLICY.expected_contract == SOTA_V2_CONTRACT + assert SOTA_V2_POLICY.expected_contract["contract_fingerprint"] == "558e8f35ea1d66b9" + assert SOTA_V3_POLICY.expected_contract == benchmark_contract() def test_archived_v1_result_remains_auditable_but_not_v2_eligible() -> None: @@ -45,7 +52,7 @@ def test_archived_v1_result_remains_auditable_but_not_v2_eligible() -> None: payload = json.loads(path.read_text()) historical = validate_leaderboard_payload(payload, policy=SOTA_V1_POLICY) - current = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + current = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert historical.ok assert not current.ok diff --git a/tests/test_negotiation.py b/tests/test_negotiation.py index 54ee9fc..4f4e4d5 100644 --- a/tests/test_negotiation.py +++ b/tests/test_negotiation.py @@ -84,6 +84,7 @@ def test_free_agent_walks_away_after_repeated_lowballs() -> None: assert league.rejected_offers == REJECTED_OFFER_LIMIT_PER_WINDOW + 1 assert league.illegal_actions == 0 # Talks reopen at the next decision window. + league.begin_decision_window() league.apply_actions([{"type": "sign_free_agent", "player_id": player_id, "years": 1, "salary": ask}], "preseason") assert league.transactions[-1].accepted is True diff --git a/tests/test_official_results.py b/tests/test_official_results.py index 0a5a0f2..2152a1a 100644 --- a/tests/test_official_results.py +++ b/tests/test_official_results.py @@ -7,7 +7,7 @@ from gm_bench import cli as cli_module from gm_bench.benchmark_config import PRESETS, PRIVATE_SEEDS_ENV, seed_panel_metadata -from gm_bench.contract import benchmark_contract, scaffold_fingerprint +from gm_bench.contract import SOTA_V2_CONTRACT, benchmark_contract, scaffold_fingerprint from gm_bench.official import ( ARCHIVE_V1_POLICY, OUTPUT_BUDGET_SWEEP_POLICY, @@ -15,10 +15,11 @@ REDACTED_SEEDS_SENTINEL, SOTA_V1_CONTRACT, SOTA_V1_POLICY, - SOTA_V2_POLICY, + SOTA_V3_POLICY, redact_leaderboard_payload, validate_leaderboard_payload, ) +from gm_bench.publication import compact_result from scripts.analyze_output_budget import analyze from web.scripts.build_leaderboard import model_row @@ -136,8 +137,19 @@ def test_public_leaderboard_policy_accepts_single_repeat_payload() -> None: assert report.ok +def test_compact_artifact_rejects_tampered_episode_and_aggregate() -> None: + payload = compact_result(_official_payload(repeats=3)) + payload["candidate"]["episodes"][0]["final_score"] = 9999.0 + payload["candidate"]["summary"]["mean_score"] = 9999.0 + payload["normalized"]["candidate_mean_score"] = 9999.0 + report = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) + assert not report.ok + assert any("episode-derived" in error for error in report.errors) + + def test_output_budget_policy_preserves_high_failure_cells() -> None: payload = _official_payload(repeats=3, failure_rate=1.0) + payload["run_info"]["benchmark_contract"] = SOTA_V2_CONTRACT report = validate_leaderboard_payload(payload, policy=OUTPUT_BUDGET_SWEEP_POLICY) assert report.ok assert any("adapter fallback" in warning for warning in report.warnings) @@ -154,31 +166,31 @@ def test_historical_baseline_panel_is_diagnostic_but_not_sota() -> None: assert public_report.ok assert "historical baseline panel differs from the current official panel" in public_report.warnings - sota_report = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + sota_report = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert not sota_report.ok assert any(error.startswith("baselines must be") for error in sota_report.errors) -def test_sota_v2_policy_requires_repeats() -> None: - report = validate_leaderboard_payload(_official_payload(repeats=1), policy=SOTA_V2_POLICY) +def test_sota_v3_policy_requires_repeats() -> None: + report = validate_leaderboard_payload(_official_payload(repeats=1), policy=SOTA_V3_POLICY) assert not report.ok - assert "candidate.repeats must be >= 3 for sota-v2" in report.errors + assert "candidate.repeats must be >= 3 for sota-v3" in report.errors -def test_sota_v2_policy_rejects_high_failure_rate() -> None: - report = validate_leaderboard_payload(_official_payload(repeats=3, failure_rate=0.05), policy=SOTA_V2_POLICY) +def test_sota_v3_policy_rejects_high_failure_rate() -> None: + report = validate_leaderboard_payload(_official_payload(repeats=3, failure_rate=0.05), policy=SOTA_V3_POLICY) assert not report.ok assert any("decision_failure_rate" in error for error in report.errors) -def test_sota_v2_policy_rejects_runaway_failed_queries() -> None: +def test_sota_v3_policy_rejects_runaway_failed_queries() -> None: # The v1 scout-contract break produced 1,124 silently-rejected lookups across # 480 decisions (2.34/decision) while reporting a clean summary. That row must # not be publishable again. payload = _official_payload(repeats=3) decisions = int(payload["candidate"]["summary"]["decisions"]) payload["candidate"]["summary"]["failed_queries"] = decisions * 2 + 1 - report = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + report = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert not report.ok assert any("failed queries" in error for error in report.errors) @@ -187,13 +199,13 @@ def test_failed_queries_warn_below_the_hard_gate() -> None: payload = _official_payload(repeats=3) decisions = int(payload["candidate"]["summary"]["decisions"]) payload["candidate"]["summary"]["failed_queries"] = int(decisions * 0.5) - report = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + report = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert report.ok assert any("failed queries" in warning for warning in report.warnings) # A handful of misfired lookups is normal exploration, not a signal. payload["candidate"]["summary"]["failed_queries"] = int(decisions * 0.1) - quiet = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + quiet = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert quiet.ok assert not any("failed queries" in warning for warning in quiet.warnings) @@ -222,16 +234,16 @@ def test_archive_v1_policy_rejects_a_non_v1_artifact() -> None: assert any("contract" in error.lower() for error in report.errors) -def test_sota_v2_policy_requires_full_usage() -> None: +def test_sota_v3_policy_requires_full_usage() -> None: payload = _official_payload(repeats=3) payload["candidate"]["summary"]["usage"]["decisions_with_usage"] = 0 - report = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + report = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert not report.ok assert "candidate usage must cover every decision point" in report.errors payload = _official_payload(repeats=3) payload["candidate"]["summary"]["usage"]["cost_usd"] = "missing" - report = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + report = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert not report.ok assert "candidate usage.cost_usd is required, use null only when pricing is unknown" in report.errors @@ -240,17 +252,18 @@ def test_sota_v2_policy_requires_full_usage() -> None: ("run_value", "option_value"), [(None, "1"), (1, None), (-1, "-1"), (2, "2"), (0, "1"), ("bad", "1")], ) -def test_sota_v2_requires_matching_bounded_repair_provenance(run_value: object, option_value: object) -> None: +def test_sota_v3_requires_matching_bounded_repair_provenance(run_value: object, option_value: object) -> None: payload = _official_payload(repeats=3) payload["run_info"]["protocol_repair_attempts"] = run_value payload["run_info"]["provider_options"]["GM_BENCH_PROTOCOL_REPAIR_ATTEMPTS"] = option_value - report = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + report = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert not report.ok assert any("repair" in error for error in report.errors) def test_output_budget_analysis_rejects_duplicate_cells() -> None: payload = _official_payload(repeats=3) + payload["run_info"]["benchmark_contract"] = SOTA_V2_CONTRACT payload["run_info"]["profile"] = "compact" payload["run_info"]["transport"] = "direct-api" payload["run_info"]["provider_options"] = { @@ -308,12 +321,12 @@ def test_openrouter_price_route_is_public_diagnostic_but_not_sota() -> None: assert public.ok assert any("price-routed OpenRouter diagnostic" in warning for warning in public.warnings) - sota = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + sota = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert not sota.ok assert any("OPENROUTER_PROVIDER_ONLY" in error for error in sota.errors) -def test_sota_v2_accepts_pinned_single_upstream_openrouter_route() -> None: +def test_sota_v3_accepts_pinned_single_upstream_openrouter_route() -> None: payload = _official_payload(repeats=3) payload["agent"] = "openrouter:openai/gpt-test" payload["candidate"]["agent"] = payload["agent"] @@ -333,11 +346,11 @@ def test_sota_v2_accepts_pinned_single_upstream_openrouter_route() -> None: ) payload["candidate"]["summary"]["usage"]["upstream_providers"] = ["OpenAI"] - report = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + report = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert report.ok -def test_sota_v2_rejects_openrouter_upstream_that_differs_from_pin() -> None: +def test_sota_v3_rejects_openrouter_upstream_that_differs_from_pin() -> None: payload = _official_payload(repeats=3) payload["agent"] = "openrouter:openai/gpt-test" payload["candidate"]["agent"] = payload["agent"] @@ -357,83 +370,83 @@ def test_sota_v2_rejects_openrouter_upstream_that_differs_from_pin() -> None: ) payload["candidate"]["summary"]["usage"]["upstream_providers"] = ["Azure"] - report = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + report = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert not report.ok assert any("does not match" in error for error in report.errors) -def test_sota_v2_policy_requires_contract_provenance() -> None: +def test_sota_v3_policy_requires_contract_provenance() -> None: payload = _official_payload(repeats=3) del payload["run_info"]["benchmark_contract"] - report = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + report = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert not report.ok assert "run_info.benchmark_contract is required for current-contract validation" in report.errors -def test_sota_v2_policy_rejects_missing_scaffold_fingerprint() -> None: +def test_sota_v3_policy_rejects_missing_scaffold_fingerprint() -> None: payload = _official_payload(repeats=3) del payload["run_info"]["scaffold_fingerprint"] - sota_report = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + sota_report = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert not sota_report.ok - assert any("scaffold_fingerprint is required for sota-v2 rows" in error for error in sota_report.errors) + assert any("scaffold_fingerprint is required for sota-v3 rows" in error for error in sota_report.errors) public_report = validate_leaderboard_payload(payload, policy=PUBLIC_LEADERBOARD_POLICY) assert public_report.ok assert any("scaffold_fingerprint missing" in warning for warning in public_report.warnings) -def test_sota_v2_policy_requires_seed_panel_provenance() -> None: +def test_sota_v3_policy_requires_seed_panel_provenance() -> None: payload = _official_payload(repeats=3) del payload["run_info"]["seed_panel"] - report = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + report = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert not report.ok assert "run_info.seed_panel is required for official seed-panel validation" in report.errors -def test_sota_v2_policy_rejects_contract_fingerprint_mismatch() -> None: +def test_sota_v3_policy_rejects_contract_fingerprint_mismatch() -> None: payload = _official_payload(repeats=3) payload["run_info"]["benchmark_contract"]["contract_fingerprint"] = "stale" - report = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + report = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert not report.ok assert any("contract_fingerprint" in error for error in report.errors) -def test_sota_v2_policy_accepts_private_panel_when_env_matches(monkeypatch: pytest.MonkeyPatch) -> None: +def test_sota_v3_policy_accepts_private_panel_when_env_matches(monkeypatch: pytest.MonkeyPatch) -> None: private_seeds = [101, 102, 110, 111, 112, 113, 114, 115] monkeypatch.setenv(PRIVATE_SEEDS_ENV, "101,102,110-115") report = validate_leaderboard_payload( _official_payload(repeats=3, seeds=private_seeds), - policy=SOTA_V2_POLICY, + policy=SOTA_V3_POLICY, ) assert report.ok -def test_sota_v2_policy_rejects_too_small_private_panel(monkeypatch: pytest.MonkeyPatch) -> None: +def test_sota_v3_policy_rejects_too_small_private_panel(monkeypatch: pytest.MonkeyPatch) -> None: private_seeds = [101, 102, 110, 111] monkeypatch.setenv(PRIVATE_SEEDS_ENV, "101,102,110-111") report = validate_leaderboard_payload( _official_payload(repeats=3, seeds=private_seeds), - policy=SOTA_V2_POLICY, + policy=SOTA_V3_POLICY, ) assert not report.ok - assert "seeds must contain at least 8 seed(s) for sota-v2" in report.errors + assert "seeds must contain at least 8 seed(s) for sota-v3" in report.errors -def test_sota_v2_policy_rejects_private_panel_without_env(monkeypatch: pytest.MonkeyPatch) -> None: +def test_sota_v3_policy_rejects_private_panel_without_env(monkeypatch: pytest.MonkeyPatch) -> None: private_seeds = [101, 102, 110, 111, 112, 113, 114, 115] monkeypatch.setenv(PRIVATE_SEEDS_ENV, "101,102,110-115") payload = _official_payload(repeats=3, seeds=private_seeds) monkeypatch.delenv(PRIVATE_SEEDS_ENV) - report = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + report = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert not report.ok assert f"{PRIVATE_SEEDS_ENV} is required to validate a private leaderboard seed panel" in report.errors -def test_sota_v2_policy_rejects_seed_panel_hash_mismatch() -> None: +def test_sota_v3_policy_rejects_seed_panel_hash_mismatch() -> None: payload = _official_payload(repeats=3) payload["run_info"]["seed_panel"]["sha256"] = "stale" - report = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + report = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert not report.ok assert any("seed_panel.sha256" in error for error in report.errors) @@ -444,7 +457,7 @@ def test_redact_leaderboard_payload_removes_private_seed_details(monkeypatch: py redacted, report = redact_leaderboard_payload(_official_payload(repeats=3, seeds=private_seeds)) assert report.ok - assert redacted["validation_reports"]["sota-v2"]["ok"] is True + assert redacted["validation_reports"]["sota-v3"]["ok"] is True assert redacted["redaction"]["applied"] is True assert redacted["seeds"] == REDACTED_SEEDS_SENTINEL assert redacted["candidate"]["seeds"] == REDACTED_SEEDS_SENTINEL @@ -466,25 +479,25 @@ def test_cli_redact_result_writes_public_safe_artifact(tmp_path: Path, monkeypat payload = json.loads(redacted_path.read_text()) assert payload["seeds"] == REDACTED_SEEDS_SENTINEL assert payload["candidate"]["episodes"] == [] - assert payload["validation_reports"]["sota-v2"]["ok"] is True + assert payload["validation_reports"]["sota-v3"]["ok"] is True -def test_leaderboard_builder_accepts_redacted_private_artifact(monkeypatch: pytest.MonkeyPatch) -> None: +def test_v2_leaderboard_builder_excludes_redacted_v3_artifact(monkeypatch: pytest.MonkeyPatch) -> None: private_seeds = [101, 102, 110, 111, 112, 113, 114, 115] monkeypatch.setenv(PRIVATE_SEEDS_ENV, "101,102,110-115") redacted, _report = redact_leaderboard_payload(_official_payload(repeats=3, seeds=private_seeds)) monkeypatch.delenv(PRIVATE_SEEDS_ENV) # Revalidation must succeed without the private seed env: only the commitment remains. - report = validate_leaderboard_payload(redacted, policy=SOTA_V2_POLICY) + report = validate_leaderboard_payload(redacted, policy=SOTA_V3_POLICY) assert report.ok row = model_row(redacted) assert row["seeds"] is None assert row["seed_panel"] == "private-env" - assert row["sota_v2_eligible"] is True - assert row["sota_v2_issues"] == [] + assert row["sota_v2_eligible"] is False + assert any("benchmark_version" in issue for issue in row["sota_v2_issues"]) def test_leaderboard_builder_revalidates_forged_sota_report() -> None: @@ -515,7 +528,7 @@ def test_leaderboard_builder_rejects_forged_redacted_sota_report() -> None: payload["redaction"] = {"applied": True, "seed_panel": "private-env", "removed": ["seeds"]} payload["validation_reports"] = {"sota-v2": {"policy": "sota-v2", "ok": True, "errors": [], "warnings": []}} - report = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + report = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert not report.ok assert any("candidate.repeats" in error for error in report.errors) assert any("decision_failure_rate" in error for error in report.errors) @@ -524,14 +537,14 @@ def test_leaderboard_builder_rejects_forged_redacted_sota_report() -> None: assert row["sota_v2_eligible"] is False -def test_sota_v2_policy_accepts_valid_redacted_private_artifact(monkeypatch: pytest.MonkeyPatch) -> None: +def test_sota_v3_policy_accepts_valid_redacted_private_artifact(monkeypatch: pytest.MonkeyPatch) -> None: private_seeds = [101, 102, 110, 111, 112, 113, 114, 115] monkeypatch.setenv(PRIVATE_SEEDS_ENV, "101,102,110-115") redacted, report = redact_leaderboard_payload(_official_payload(repeats=3, seeds=private_seeds)) assert report.ok monkeypatch.delenv(PRIVATE_SEEDS_ENV) - revalidated = validate_leaderboard_payload(redacted, policy=SOTA_V2_POLICY) + revalidated = validate_leaderboard_payload(redacted, policy=SOTA_V3_POLICY) assert revalidated.ok diff --git a/tests/test_oracle.py b/tests/test_oracle.py index 430d3c5..69da7e9 100644 --- a/tests/test_oracle.py +++ b/tests/test_oracle.py @@ -7,10 +7,9 @@ def test_oracle_preserves_frozen_contract_fingerprint() -> None: - # Re-frozen 2026-07-15 (failed_queries split + ambiguous-scout rejection); - # see the PUBLISH_READINESS decision log for why the pin moved off - # a65a4359ca3c6e64. - assert contract_fingerprint() == "558e8f35ea1d66b9" + # sota-v3 begins with the post-release numeric-validation and negotiation + # window corrections; sota-v2 remains pinned in SOTA_V2_CONTRACT. + assert contract_fingerprint() == "89c89e53c26740c7" assert "oracle" not in AGENTS diff --git a/tests/test_pick_trading.py b/tests/test_pick_trading.py index 50cbac2..25a0966 100644 --- a/tests/test_pick_trading.py +++ b/tests/test_pick_trading.py @@ -30,6 +30,7 @@ def find_accepted_pick_sale(league: League) -> dict | None: "receive_pick_seasons": [league.season + 1], } before = league.illegal_actions + league.begin_decision_window() # each probe is its own window league.apply_actions([action], "preseason") if league.transactions[-1].accepted: return action diff --git a/tests/test_publication.py b/tests/test_publication.py index 80f66b1..9dc56ad 100644 --- a/tests/test_publication.py +++ b/tests/test_publication.py @@ -1,6 +1,7 @@ from __future__ import annotations import json +import math import subprocess from pathlib import Path @@ -64,6 +65,11 @@ def test_compact_result_rejects_already_compacted_artifact() -> None: compact_result(compact) +def test_canonical_hash_refuses_non_finite_publication_json() -> None: + with pytest.raises(ValueError, match="Out of range float values"): + canonical_sha256({"score": math.nan}) + + def test_budget_analysis_refuses_empty_sweep(tmp_path: Path) -> None: output = tmp_path / "analysis.json" subprocess.run(["python3", "scripts/analyze_output_budget.py", "--output", str(output)], check=True) diff --git a/tests/test_publication_claims.py b/tests/test_publication_claims.py index ee82ad2..76cb9c6 100644 --- a/tests/test_publication_claims.py +++ b/tests/test_publication_claims.py @@ -19,6 +19,8 @@ def test_blog_result_table_matches_generated_leaderboard() -> None: site = json.loads(Path("web/src/data/leaderboard.json").read_text()) blog = Path("docs/blog/sota-v2-findings.md").read_text() + assert site["contract"]["benchmark_version"] == "sota-v2" + assert site["contract"]["contract_fingerprint"] == "558e8f35ea1d66b9" rows = {} pattern = re.compile( r"^\| (?P[^|]+?) \| (?P-?[0-9.]+) \| (?P-?[0-9.]+) " @@ -44,6 +46,8 @@ def test_public_claim_surfaces_use_the_frozen_lane_and_current_gate() -> None: readme = Path("README.md").read_text() assert "4,096-token" in blog assert "1,024-token safety ceiling" not in blog + assert "Post-release integrity note (2026-07-24)" in blog + assert "sota-v3" in readme assert "`publishable_ranking: true`" in readme assert "no ordinal \u201cbest model\u201d claim" in " ".join(readme.split()) diff --git a/tests/test_publication_runner.py b/tests/test_publication_runner.py index a40a2fe..3acf176 100644 --- a/tests/test_publication_runner.py +++ b/tests/test_publication_runner.py @@ -11,7 +11,7 @@ import pytest import scripts.run_publication_matrix as publication_runner -from gm_bench.contract import contract_fingerprint, scaffold_fingerprint +from gm_bench.contract import BENCHMARK_VERSION, contract_fingerprint, scaffold_fingerprint from scripts.run_publication_matrix import ( _artifact_spend_usd, _cell_reservation_usd, @@ -36,8 +36,10 @@ def _frozen_panel_files( monkeypatch: pytest.MonkeyPatch, ) -> tuple[dict, dict, Path]: registry = json.loads(Path("config/sota_v2_models.json").read_text()) + registry["contract"] = BENCHMARK_VERSION registry["selection_status"] = "frozen" lane = json.loads(Path("config/sota_v2_lane.json").read_text()) + lane["contract"] = BENCHMARK_VERSION lane["output_budget_status"] = "frozen-native-reasoning-cap" lane.pop("smoke_manifest", None) registry_path = tmp_path / "models.json" @@ -293,8 +295,9 @@ def test_validate_models_rejects_mandatory_minimum_policy_with_no_effort_declare publication_runner._validate_models([model]) -def test_committed_panel_is_unlocked_after_registry_and_smoke_freeze() -> None: - assert len(build_cells("panel")) == 10 +def test_committed_v2_panel_is_locked_after_current_contract_advances() -> None: + with pytest.raises(ValueError, match="different benchmark contract"): + build_cells("panel") def test_panel_stays_locked_when_frozen_registry_has_no_manifest( @@ -628,8 +631,11 @@ def test_artifact_spend_uses_completed_result_telemetry(tmp_path: Path) -> None: def test_paid_openrouter_run_requires_explicit_spend_ceiling( - tmp_path: Path, capsys: pytest.CaptureFixture[str] + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], ) -> None: + _frozen_panel_files(tmp_path, monkeypatch) with pytest.raises(SystemExit) as exc: main( [ @@ -660,7 +666,12 @@ def test_cell_reservation_blocks_launch_before_ceiling_overrun(tmp_path: Path) - assert not (tmp_path / "openrouter-reservations.json").exists() -def test_cell_reservation_covers_repairs_and_cost_contingency() -> None: +def test_cell_reservation_covers_repairs_and_cost_contingency( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + registry, lane, manifest_path = _frozen_panel_files(tmp_path, monkeypatch) + manifest_path.write_text(json.dumps(_valid_manifest(registry, lane))) cell = build_cells("panel", model_id="openrouter-gpt-5.6-luna-openai")[0] pricing = json.loads(Path("config/openrouter_pricing_snapshot.json").read_text()) assumptions = pricing["planning_assumptions"] @@ -803,8 +814,11 @@ def test_unsettled_failed_attempt_remains_part_of_next_reservation_guard(tmp_pat def test_panel_artifact_gate_requires_complete_cost_and_registered_route_telemetry( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: + registry, lane, manifest_path = _frozen_panel_files(tmp_path, monkeypatch) + manifest_path.write_text(json.dumps(_valid_manifest(registry, lane))) cell = build_cells("panel", model_id="openrouter-grok-4.5-xai")[0] artifact = { "run_info": { @@ -846,6 +860,8 @@ def test_existing_ineligible_panel_artifact_is_not_reused_or_overwritten( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: + registry, lane, manifest_path = _frozen_panel_files(tmp_path, monkeypatch) + manifest_path.write_text(json.dumps(_valid_manifest(registry, lane))) cell = build_cells("panel", model_id="openrouter-grok-4.5-xai")[0] raw_dir = tmp_path / "raw" raw_dir.mkdir() diff --git a/tests/test_session_lane.py b/tests/test_session_lane.py index 6f89fa7..823b72a 100644 --- a/tests/test_session_lane.py +++ b/tests/test_session_lane.py @@ -1,6 +1,6 @@ """Session-lane wiring: provider construction, provenance, and validation.""" -from gm_bench.official import SOTA_V2_POLICY, validate_leaderboard_payload +from gm_bench.official import SOTA_V3_POLICY, validate_leaderboard_payload from gm_bench.providers import build_provider_agent from gm_bench.session import PersistentProcessAgent @@ -17,12 +17,12 @@ def test_build_provider_agent_default_is_fresh_spawn() -> None: assert agent.metadata["session"] is False -def test_sota_v2_rejects_session_rows() -> None: +def test_sota_v3_rejects_session_rows() -> None: from test_official_results import _official_payload payload = _official_payload(repeats=3) payload["run_info"]["session"] = True - report = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + report = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert any("session-condition" in error for error in report.errors) @@ -31,5 +31,5 @@ def test_fresh_spawn_rows_still_pass_sota_v1() -> None: payload = _official_payload(repeats=3) payload["run_info"]["session"] = False - report = validate_leaderboard_payload(payload, policy=SOTA_V2_POLICY) + report = validate_leaderboard_payload(payload, policy=SOTA_V3_POLICY) assert report.ok diff --git a/tests/test_smoke_manifest.py b/tests/test_smoke_manifest.py index 4d8cd08..518702a 100644 --- a/tests/test_smoke_manifest.py +++ b/tests/test_smoke_manifest.py @@ -63,6 +63,25 @@ def test_complete_valid_smoke_manifest_has_no_issues() -> None: assert smoke_manifest_issues(_valid_manifest(registry, lane), registry, lane) == [] +def test_historical_smoke_manifest_uses_frozen_contract_without_current_scaffold() -> None: + registry, lane = _registry_and_lane() + manifest = _valid_manifest(registry, lane) + for entry in manifest["entries"].values(): + entry["contract_fingerprint"] = "frozen-v2" + entry["scaffold_fingerprint"] = "historical-scaffold" + + assert ( + smoke_manifest_issues( + manifest, + registry, + lane, + expected_contract_fingerprint="frozen-v2", + validate_current_scaffold=False, + ) + == [] + ) + + @pytest.mark.parametrize( ("mutation", "message"), [ diff --git a/web/scripts/build_leaderboard.py b/web/scripts/build_leaderboard.py index d7bc443..0d60cc7 100644 --- a/web/scripts/build_leaderboard.py +++ b/web/scripts/build_leaderboard.py @@ -8,7 +8,7 @@ python -m gm_bench model --provider

--model --preset leaderboard --repeats 3 --json > results/leaderboard/.json or a redacted private-panel artifact from ``python -m gm_bench redact-result``. -Only rows on the current ``sota-v2`` contract are published; anything else is +Only rows on the frozen ``sota-v2`` release contract are published; anything else is skipped with a note on stderr. It writes ``web/src/data/leaderboard.json`` with one row per model plus the @@ -34,7 +34,7 @@ from gm_bench.agents import AGENTS # noqa: E402 from gm_bench.baseline_cache import cache_key, load_cache # noqa: E402 from gm_bench.benchmark_config import PRESETS, PRIVATE_LEADERBOARD_PANEL_NAME # noqa: E402 -from gm_bench.contract import benchmark_contract # noqa: E402 +from gm_bench.contract import SOTA_V2_CONTRACT # noqa: E402 from gm_bench.official import REDACTED_SEEDS_SENTINEL, SOTA_V2_POLICY, validate_leaderboard_payload # noqa: E402 from gm_bench.oracle import OracleAgent # noqa: E402 from gm_bench.protocol import PHASES # noqa: E402 @@ -507,7 +507,13 @@ def main() -> None: loaded = None if isinstance(loaded, dict): manifest = loaded - smoke_issues = smoke_manifest_issues(manifest, model_config, lane_config) + smoke_issues = smoke_manifest_issues( + manifest, + model_config, + lane_config, + expected_contract_fingerprint=str(SOTA_V2_CONTRACT["contract_fingerprint"]), + validate_current_scaffold=False, + ) panel_analysis: dict[str, Any] | None = None if PANEL_ANALYSIS_PATH.is_file(): try: @@ -559,7 +565,7 @@ def main() -> None: updated = max((stamp for stamp in timestamps if stamp), default="")[:10] dataset = { "updated": updated, - "contract": benchmark_contract(), + "contract": SOTA_V2_CONTRACT, "preset": { "name": "leaderboard", "seeds": LEADERBOARD["seeds"], diff --git a/web/src/data/leaderboard.json b/web/src/data/leaderboard.json index 0a4a46e..c04a1b9 100644 --- a/web/src/data/leaderboard.json +++ b/web/src/data/leaderboard.json @@ -137,6 +137,7 @@ "significant": true, "sota_v2_eligible": true, "sota_v2_issues": [ + "historical scaffold fingerprint retained but cannot be re-derived from current source", "candidate has illegal actions; score includes protocol penalties", "candidate has 453 failed queries across 480 decisions (0.94 per decision); misfired scout/inspect lookups may indicate the model is not reading query errors", "candidate does not beat the strongest scripted baseline" @@ -223,6 +224,7 @@ "significant": true, "sota_v2_eligible": true, "sota_v2_issues": [ + "historical scaffold fingerprint retained but cannot be re-derived from current source", "candidate has illegal actions; score includes protocol penalties", "candidate used adapter fallback/error output on at least one decision", "candidate has 209 failed queries across 480 decisions (0.44 per decision); misfired scout/inspect lookups may indicate the model is not reading query errors", @@ -310,6 +312,7 @@ "significant": true, "sota_v2_eligible": true, "sota_v2_issues": [ + "historical scaffold fingerprint retained but cannot be re-derived from current source", "candidate has illegal actions; score includes protocol penalties", "candidate used adapter fallback/error output on at least one decision", "candidate does not beat the strongest scripted baseline" @@ -396,6 +399,7 @@ "significant": true, "sota_v2_eligible": true, "sota_v2_issues": [ + "historical scaffold fingerprint retained but cannot be re-derived from current source", "candidate has illegal actions; score includes protocol penalties", "candidate does not beat the strongest scripted baseline" ], @@ -481,6 +485,7 @@ "significant": true, "sota_v2_eligible": true, "sota_v2_issues": [ + "historical scaffold fingerprint retained but cannot be re-derived from current source", "candidate has illegal actions; score includes protocol penalties", "candidate does not beat the strongest scripted baseline" ], @@ -566,6 +571,7 @@ "significant": true, "sota_v2_eligible": true, "sota_v2_issues": [ + "historical scaffold fingerprint retained but cannot be re-derived from current source", "candidate has illegal actions; score includes protocol penalties", "candidate does not beat the strongest scripted baseline" ], @@ -651,6 +657,7 @@ "significant": true, "sota_v2_eligible": true, "sota_v2_issues": [ + "historical scaffold fingerprint retained but cannot be re-derived from current source", "candidate has illegal actions; score includes protocol penalties", "candidate has 137 failed queries across 480 decisions (0.29 per decision); misfired scout/inspect lookups may indicate the model is not reading query errors", "candidate does not beat the strongest scripted baseline" @@ -737,6 +744,7 @@ "significant": true, "sota_v2_eligible": true, "sota_v2_issues": [ + "historical scaffold fingerprint retained but cannot be re-derived from current source", "candidate has illegal actions; score includes protocol penalties", "candidate used adapter fallback/error output on at least one decision", "candidate does not beat the strongest scripted baseline"