Publication sprint: freeze and ship GM-Bench sota-v2

[nedcut]

Goal

Make publication readiness the project's primary focus for the next seven days
(2026-07-14 through 2026-07-21). The target is an evidence-complete, frozen
sota-v2 release candidate—or a precise external blocker with every safe
in-repo step finished—not more simulator scope.

The authoritative source of truth is the living checklist in
docs/PUBLISH_READINESS.md.
Update it whenever evidence, a frozen decision, cost, provider state, or a
blocker changes. This issue is the compact execution tracker; #60 remains the
broader roadmap.

Focus rule for the week

• Publication work is the default priority.
• Keep Make contract length a strategic decision #62 and all behavior-changing sota-v3 work parked.
• Do not alter the frozen v2 simulator, scoring, presets, or schemas after
  official model runs begin.
• Do not replace, retune, or selectively rerun a valid result because of its
  score.
• Keep all paid calls serial, checkpointed, route-pinned, and behind an explicit
  spend ceiling.

Current state

• GM-Bench v2: scout contract fix, failed-query telemetry, sota-v2 contract #58 and Harden model panel execution and recovery #59 are merged into main.
• Build the publishable sota-v2 leaderboard pipeline #61 merged to main after independent review and full finding disposition.
• 361 local tests, Ruff, contract canaries, calibration, archived-v1
  verification, web lint/build, and empty-v2-leaderboard generation pass on the
  current publication branch.
• The sweep is frozen at 3 models × 4 common caps (256 / 1,024 / 4,096 /
  16,384) × 3 repeats.
• The primary endpoint, deterministic cap rule, retry/exclusion/stopping
  policy, and minimum eight-model headline gate are committed.
• Provider-dependent “uncapped” was removed before official results.
• JSON mode is standardized on, reasoning is standardized off, and exact
  endpoints are pinned.
• All three sweep models completed a clean standardized 1,024-token smoke.
• All 11 intended headline routes accept the common privacy, parameter, JSON,
  reasoning-off, and bounded-output policy.
• The sweep estimate is recorded: $27 planning, $32.40 with cost
  contingency, $94.51 token-ceiling contingency; 8.96 projected serial API
  hours, 13.44 with runtime contingency.
• No official sweep or headline result exists yet; the public v2 ranking is
  correctly empty.

1. Land the publication pipeline

• Obtain a fresh independent review of Build the publishable sota-v2 leaderboard pipeline #61 at its latest head.
• Resolve or explicitly disposition every substantive finding.
• Re-run the full local release gate and verify all GitHub checks at the
  exact final SHA.
• Merge Build the publishable sota-v2 leaderboard pipeline #61 to main.
• Verify a clean clone reproduces the empty gated leaderboard, cost
  estimate, contract checks, and web build.

Exit: main contains the frozen protocol and safe paid-run path.

2. Run and interpret the output-budget sweep

• Recheck live OpenRouter prices, account balance, endpoint names,
  capabilities, and route health immediately before the run.
• Record the approved operator spend ceiling. The committed $100 ceiling is
  a conservative guard, not automatic authorization.
• Run all 12 official cells serially with one new run directory:
  scripts/run_publication_matrix.py sweep --max-spend-usd <approved>.
• Preserve raw artifacts, checkpoints, provider failures, stderr/log
  summaries, and account-cost deltas outside git.
• Do not rerun metered invalid content, protocol failures, illegal actions,
  poor scores, or surprising valid cells.
• Regenerate results/analysis/output-budget-sweep.json.
• Audit configured cap versus actual output tokens, score, repairs, failures,
  cost, and latency for every model/cap cell.
• Apply the predeclared analyzer recommendation: lowest saturated cap, or
  16,384 plus compute-elastic curves if no lower cap saturates.
• Freeze config/sota_v2_lane.json with the decision, date, artifact hash,
  rationale, and limitations before any full-panel result.

Exit: one compute policy is frozen from complete sweep evidence.

3. Run and validate the headline model panel

• Re-estimate exact full-panel cost and runtime at the frozen cap and obtain
  explicit spend approval.
• Recheck all 11 exact routes; document withdrawals rather than substituting
  models after results are visible.
• Run a benchmark-level smoke for every model at the frozen cap.
• Run the full API panel serially: 8 seeds × 5 seasons × 3 repeats.
• Require complete route, input/output token, cost, latency, repair, failure,
  and provenance telemetry.
• Keep valid poor model behavior in the result; resume only the
  pre-registered infrastructure cases.
• Strictly validate every row under sota-v2.
• Put interesting ineligible rows in results/diagnostics/; never weaken
  the headline gate.
• Require at least eight eligible pre-registered rows before a ranking can
  appear.
• Compact only after validation, retain canonical raw hashes, and prepare raw
  public traces as release assets.
• Regenerate tables and site data from artifacts only.

Exit: the generated leaderboard contains a meaningful, compute-comparable,
strictly eligible panel.

4. Analyze, write, and package

• Report score, uncertainty, paired effect sizes, sign-flip p-values,
  compute, cost, failures, repairs, and per-mechanic outcomes.
• Treat per-model p-values as descriptive unless the final family-wise
  analysis justifies stronger claims.
• Run power, score-weight sensitivity, leave-one-seed-out, and
  extreme-episode checks.
• Show output-budget curves before the ranking and show Oracle →
  pick-trader → best eligible model → random headroom.
• Keep API and coding-harness results in separate tables.
• Finish the blog from generated evidence, including the withdrawn-v1 story,
  compute confound, limitations, and exact scope.
• Update the README/site with the primary result, what the benchmark does and
  does not measure, architecture/evaluation flow, quickstarts, cost guidance,
  and links to evidence.
• Add CITATION.cff, changelog/release notes, checksums, reproducibility
  manifest, raw assets, and a tagged v2 release.
• Check mobile, accessibility, commands, internal links, and clean-clone
  behavior.
• Obtain an independent final read and, if possible, one outside
  clean-clone reproduction.
• Decide before writing claims whether a salted private-panel precommit/run
  is feasible; otherwise scope v2 explicitly to reproducible public-panel
  evidence.

Release gate

Do not publish a headline ranking unless all are true:

• Frozen benchmark and compute policies are fingerprinted and reproducible.
• Every headline row is strictly eligible and compute-comparable.
• At least eight pre-registered models pass.
• Raw evidence is available and hash-linked from compact artifacts.
• Failures, exclusions, uncertainty, compute, cost, and limitations are
  visible.
• Claims stay inside this synthetic model-plus-scaffold condition.
• CI, artifact validation, site build, commands, links, and clean-clone
  checks pass.
• No v3 behavior has leaked into the frozen v2 lane.

Daily operating rhythm

• Start each work session with this issue and the living document.
• End each session by checking completed items and linking durable evidence.
• Record provider/quota drift immediately instead of burning repeat calls.
• Keep the next unblocked critical-path item explicit.
• Update cost/runtime estimates whenever observed data materially changes
  them.

Out of scope this week

• New v3 mechanics from Contract mechanics: make contract length a real decision #24/Make contract length a strategic decision #62.
• Broad simulator feature expansion.
• Cosmetic site work without validated evidence.
• Retuning the benchmark after seeing results.
• Publishing a paper; the deliverables are the benchmark, GitHub release/site,
  reproducible artifacts, and a rigorous blog post.

Closes when v2 is published, or when the only remaining work is an explicitly
documented external run/review blocker and every safe in-repo task is complete.

Related: #60, #58, #59, #61, #62, #46.

[nedcut]

2026-07-14 sprint handoff:

• GM-Bench v2: scout contract fix, failed-query telemetry, sota-v2 contract #58 and Harden model panel execution and recovery #59 are merged.
• Build the publishable sota-v2 leaderboard pipeline #61 is retargeted to main, ready for review, mergeable, and green across Python 3.11/3.12/3.13, Ruff, benchmark-contract, and web checks.
• The living checklist is now committed in Build the publishable sota-v2 leaderboard pipeline #61 at docs/PUBLISH_READINESS.md.
• Three sweep models and an 11-model headline panel are pre-registered with exact routes/options.
• The serial publication driver, failure-tolerant sweep policy, spend measurement, minimum-eight-row gate, and route/provenance/cost checks are implemented.
• Bounded smoke evidence: Qwen at 1,024/4,096 exposed genuine output-budget protocol failures; GPT-5.4 mini at 4,096 completed cleanly. Completed artifacts reported under $0.04 total.
• A detached clean-worktree install, contract validation, 12-cell dry-run, lint, and production web build passed.

Next unblocked critical path: independent review and merge #61, freeze final retry/exclusion/stopping wording plus the approved sweep budget, then smoke every remaining model/cap path before starting the paid 12-cell sweep. Do not start the full panel until the sweep is interpreted and config/sota_v2_lane.json is frozen.

[nedcut]

2026-07-14 repository hygiene and main handoff:

• Primary checkout is now clean on main at 3f2c086, exactly synchronized with origin/main.
• Preserved 86 untracked benchmark/run artifacts (61 MB) outside the repo at /Users/nedcutler/Documents/Code/gm_bench-local-artifacts-2026-07-14; nothing was discarded.
• Reduced registered worktrees from 20 to 3: the primary checkout plus the two intentional draft-PR worktrees for Rewrite landing site around the official eval story #46 and Make contract length a strategic decision #62.
• Preserved six uncommitted files from an old merged Claude worktree as named stash@{0} before removing that worktree.
• Pruned local branches already merged into main; retained unmerged/history branches and every required remote branch. origin/v2-bench-fixes remains because Make contract length a strategic decision #62 intentionally uses it as its base.
• Revalidated current main: 361 tests, Ruff format/lint, validate-contract, calibration, web lint, and production build all pass. GitHub CI and Lint are green at the exact main SHA.
• Filled in the public repository description, GitHub Pages homepage, and topics.

Next unblocked critical path:

1. Recheck live OpenRouter route names, capabilities, health, prices, and account balance without starting benchmark calls.
2. Record explicit operator approval for the sweep spend ceiling; the committed $100 guard is not authorization by itself.
3. Run exactly one new serial/checkpointed 12-cell output-budget sweep and preserve raw evidence outside git.
4. Analyze all cells and freeze config/sota_v2_lane.json from the predeclared deterministic rule.
5. Only then re-estimate/approve/smoke/run the full headline panel.

Keep #62 parked as post-publication v3 work. Keep #46 parked until validated v2 rows exist, then rebase it deliberately onto current main and preserve the v2 empty/archive/lane behavior.

[nedcut]

2026-07-15 model-panel finalization and first paid cell:

Final 11-model headline panel (exact identities/routes are in config/sota_v2_models.json):

1. GPT-5.6 Luna — OpenAI
2. GPT-5.4 Mini — OpenAI
3. GLM 5.2 — StreamLake
4. Claude Sonnet 5 — Anthropic
5. Nemotron 3 Ultra 550B — Together
6. Qwen 3.7 Plus — Alibaba
7. Claude Haiku 4.5 — Anthropic
8. Mistral Medium 3.5 — Mistral
9. Mistral Small 4 (2603) — Mistral
10. MiniMax M3 — MiniMax
11. Qwen 3.5 9B — DeepInfra

The live preflight found the previously pinned SiliconFlow Qwen and DeepInfra Nemotron endpoints unavailable. Both were moved to healthy policy-compatible endpoints before any official sweep result existed. Gemini, Grok, Kimi, and DeepSeek remain excluded from this headline lane because the pre-result probes could not place them under the same reasoning-off, JSON, privacy, and pinned-provider conditions.

GPT-5.6 Luna replaced GPT-5.4 Mini as the frontier sweep model. Its 1,024-token full cell is now accepted:

• strict output-budget-sweep validation passes
• mean score 200.799; strategy score 293.508
• 2,225 protocol penalty from 890 illegal actions; 84 failed queries
• zero failed decisions and zero repairs across 480 decisions
• exact OpenAI upstream; 154 output tokens/decision
• cost $5.637672
• canonical raw SHA-256 b681bdc56f3d176d194c9c1e20cc688be4ef4b58f7669862fb2268af99a0e37a
• byte SHA-256 74d342c5d4c799524dadd6f668350eede67b8b74e5522833723e25ab4f50480b

PR #65 contains the timestamped selection, route amendments, provenance fix, updated cost plan, and living-document evidence. This remains one accepted sweep cell—not a cap decision or headline row.

Current OpenRouter balance after smoke + full cell: approximately $39.61.

Next critical action: review/merge #65, refresh the changed DeepInfra Qwen smoke, then choose the next sweep cells without changing the frozen model/route list in response to Luna's score.

[nedcut]

Draft PR #66 resets the publication scaffold and provisional model registry before more paid calls: #66

Key effects:

• phase-aware action examples and explicit current-pick draft limits
• prior Luna smoke and 1,024 cell preserved as superseded diagnostics under the old scaffold fingerprint
• provisional 10-model, nine-lab registry; DeepSeek V4 Pro, Kimi K2.6, and Nemotron 3 Nano require route smokes before freeze
• Gemini 3.1 Pro and Grok 4.5 explicitly excluded from the reasoning-disabled lane because OpenRouter currently marks reasoning mandatory
• paid four-cap sweep locked while we decide whether to keep the cap experiment or use one generous safety ceiling and report actual token efficiency

No paid model calls were made. Local validation: 366 tests, Ruff, contract canaries, generated publication artifacts, and web lint/build all green.

[nedcut]

PR #66 now contains the frozen output policy and executable next gate.

• Retired the paid four-cap sweep before replacement official cells.
• Froze a common 1,024-token safety ceiling with reasoning disabled.
• Actual input/output tokens, cost, and latency are secondary efficiency metrics, not score inputs.
• smoke now expands to all 10 provisional models, serially, at 1,024.
• Predeclared trigger: if any smoke call emits >=768 output tokens or shows cap-induced truncation, raise the whole lane to 2,048 before any full-panel score.
• Full panel remains locked until the model registry is frozen after all route smokes.
• Public ranking also remains locked while the registry is provisional and until at least 8 strictly eligible rows exist.

Local validation: 369 tests passed, Ruff/contract validation green, 10-cell dry-run verified, and web lint/build green. No paid calls were made.

Next execution order after #66 review/merge: dry-run -> endpoint preflight -> one paid smoke -> inspect artifact and cap pressure -> continue one model at a time -> freeze registry -> refresh panel cost/runtime plan -> begin full fixed-cap runs.

[nedcut]

2026-07-16 status (day 3).

Section 2 of this issue is superseded: PR #66 retired the 3-model × 4-cap sweep in favor of the frozen 1,024-token fixed safety ceiling (escalation to 2,048 predeclared at the 75%/768 cap-pressure trigger). The live path is: ten serial route smokes at 1,024 → accepted entries in config/sota_v2_smoke_manifest.json (machine-gated by #70/#71) → registry freeze → 10-model × 8-seed × 5-season × 3-repeat fixed-cap panel. docs/PUBLISH_READINESS.md remains authoritative and was synced today.

Landed / in flight today:

• Harden publication gates before the paid sota-v2 panel #70 + Close publication smoke-gate review gaps #71 merged: machine-enforced smoke gate hardening.
• Redesign the site and reconcile it with the frozen stats plan #72 (open): site redesign + tiers-not-ranks reconciliation (ordinal ranks and the uncorrected significance badge removed per the frozen plan).
• Close out pre-smoke readiness: contrast amendment, cost rework, runbook sync #73 (open): pre-data headline-contrast amendment — primary contrast is now paired lift vs pick-trader, recorded additively with dated rationale while zero paid evidence exists; new scripts/analyze_publication_panel.py implements the frozen analysis; fixed-panel cost estimate replaces the retired 12-cell artifact ($48.95 panel / $0.41 smokes / $59.23 at 1.2× contingency, 4,840 calls); data/publication-runs/ is now gitignored (raw paid artifacts could previously land in git).
• Pin the Semgrep CI scan to the reviewed ruleset #74 (open): Semgrep CI pinned to the reviewed ruleset; the 13 open code-scanning alerts trace to a local-vs-CI config mismatch, scan is now blocking.

Pre-smoke state after #73/#74 merge: smoke dry-run generates all ten pinned cells cleanly; OPENROUTER_API_KEY present; remaining paid-run prerequisites are operator actions only (balance/route recheck, spend-ceiling approval, then scripts/run_publication_matrix.py smoke one model at a time).

[nedcut]

The sota-v2 publication sprint is shipped.

• Release: https://github.com/nedcut/gm-bench/releases/tag/sota-v2-phase-one-2026-07-19
• Evidence/package PRs: Harden publication run completion gates #78, Publish the phase-one frontier panel #79, Package the phase-one benchmark release #80
• 8/10 registered cells are strict headline rows; Grok 4.5 and Mistral Medium 3.5 remain diagnostic for incomplete usage/cost coverage
• all eight eligible rows occupy one overlapping uncertainty tier and every row trails pick-trader; no ordinal winner claim is made
• exact raw traces, frozen configs, analysis, final run metadata, manifest, and checksums are attached to the release
• maintainer clean-clone reproduction passes from the public tag and assets

The remaining genuinely external gate is tracked in #81: an independent clean-clone reproduction by someone outside the release-generation process. Closing this sprint rather than leaving the stale pre-run checklist open.