Package the phase-one benchmark release

[nedcut]

Outcome

Packages the merged phase-one panel into a deterministic, independently verifiable release and updates public claims to match the actual evidence.

• adds a deterministic raw-evidence archive builder/verifier
• commits the 10-cell release manifest and archive checksum
• includes 8 headline and 2 diagnostic classifications with compact-to-raw hash links
• adds a clean-clone, no-provider-cost reproduction guide
• replaces the stale 1,024-token draft blog with the frozen 4,096-token result
• puts the narrow primary finding near the top of the README
• adds tests that bind the blog table to generated leaderboard data

Claim boundary

All eight eligible models trail pick-trader. Muse Spark 1.1 has the highest observed model mean, but all eight rows occupy one overlapping uncertainty tier and every full-family Holm-adjusted p-value is 0.078125. The release therefore makes no ordinal winner claim.

Grok 4.5 and Mistral Medium 3.5 remain diagnostic for incomplete usage/cost coverage and are included in the raw archive without being promoted to the headline table.

Verification

• python3 -m pytest -q (473 passed)
• Ruff format/lint
• contract canaries and scoring calibration
• web lint and production build
• deterministic archive byte-for-byte rebuild
• archive manifest and compact-to-raw hash verification
• SHA256SUMS.txt verification
• secret scan across all raw artifacts and final run metadata

The raw archive is intentionally kept out of git and will be uploaded to the tagged GitHub release after this exact branch is reviewed and merged.

Summary by CodeRabbit

• New Features

  • Published the frozen SOTA v2 Phase One benchmark release with model results, evidence artifacts, checksums, and reproducibility guidance.
  • Added a step-by-step process for downloading, validating, reproducing, and rebuilding the published results.
  • Added a deterministic release archive with manifest metadata and integrity verification.

• Documentation

  • Updated benchmark findings, publication readiness, leaderboard status, and release interpretation guidance.
  • Clarified headline versus diagnostic results and documented study limitations.

• Tests

  • Added checks to validate published claims, leaderboard consistency, archive repeatability, and artifact integrity.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 288353f6-53e7-4d7c-993a-d50b4ee837d7

📥 Commits

Reviewing files that changed from the base of the PR and between 604e983 and 7523dd7.

📒 Files selected for processing (10)

• README.md
• docs/PUBLISH_READINESS.md
• docs/REPRODUCING_SOTA_V2_RELEASE.md
• docs/blog/sota-v2-findings.md
• releases/sota-v2-phase-one-2026-07-19/README.md
• releases/sota-v2-phase-one-2026-07-19/SHA256SUMS.txt
• releases/sota-v2-phase-one-2026-07-19/manifest.json
• scripts/package_publication_release.py
• tests/test_publication_claims.py
• tests/test_publication_release.py

---

📝 Walkthrough

Walkthrough

The PR publishes the frozen sota-v2 phase-one results, adds release evidence and reproduction documentation, introduces deterministic archive packaging and verification, and adds tests for publication claims and artifact integrity.

Changes

Phase-one publication release

Layer / File(s)	Summary
Freeze status and publication claims README.md, docs/PUBLISH_READINESS.md, docs/blog/sota-v2-findings.md, tests/test_publication_claims.py	Updates phase-one result messaging, readiness records, findings, model metrics, interpretation limits, and claim consistency tests.
Release evidence and reproduction contract docs/REPRODUCING_SOTA_V2_RELEASE.md, releases/sota-v2-phase-one-2026-07-19/...	Defines the release manifest, checksums, included evidence, release notes, and verification procedure.
Deterministic archive packaging and verification scripts/package_publication_release.py	Builds deterministic ZIP releases, records file and artifact hashes, validates compact-artifact links, and verifies archive structure and checksums.
Archive fixture and integrity validation tests/test_publication_release.py	Tests byte-identical archive generation, manifest verification, and rejection of mismatched compact-artifact hashes.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related issues

• Publication sprint: freeze and ship GM-Bench sota-v2 #63 — Covers the broader sota-v2 publication, evidence packaging, reproduction, and leaderboard release work implemented here.

Possibly related PRs

• nedcut/gm-bench#61 — Provides the canonical/compact artifact hashing and publication eligibility machinery used by this release packaging work.
• nedcut/gm-bench#76 — Updates model routing and publication identifiers referenced by the frozen phase-one validation and leaderboard claims.

Sequence Diagram(s)

sequenceDiagram
  participant PackageCLI
  participant PublicationInputs
  participant ReleaseZIP
  participant VerificationTests
  PackageCLI->>PublicationInputs: read frozen configs, analysis, and model traces
  PackageCLI->>ReleaseZIP: write manifest, artifacts, and checksums
  VerificationTests->>ReleaseZIP: verify archive membership and hashes
  ReleaseZIP-->>VerificationTests: return validated release metadata

Loading

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/release-frontier-panel

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.