Close the gap between sota-v2 evidence and framing: scoring, observation asymmetry, scaffolds, power, and site claims

[nedcut]

Summary

An independent review of the sota-v2 phase-one release found that the methodology, disclosure, and reproducibility engineering are strong, while the headline comparison is heavily shaped by a hand-designed composite score, a white-box reference policy, a fresh-spawn/memo-only scaffold, and an eight-seed public panel. That does not make the result invalid; it narrows what the result measures.

The durable, defensible finding today is: under the frozen native-minimum-reasoning, fresh-spawn/memo-only, hand-scored public protocol, all eight eligible model-plus-scaffold systems trailed the white-box pick-trader reference on all eight seeds. The items below either fix concrete correctness/integrity defects or help the benchmark support broader claims without outrunning its evidence.

---

Priority 0 — Correctness and evidence-integrity blockers

These are new findings reproduced against local HEAD 1591b1d without modifying the checkout.

0A. Non-finite salaries bypass contract and cap validation

_sign_free_agent converts salary with float(...) and then uses ordinary comparisons. Python accepts NaN through json.loads; every comparison against it is false, so a NaN offer bypasses the positive-salary, hard-cap, and reservation-price checks and is accepted. A direct probe signed 24 free agents in one round at salary: NaN, grew the user roster from 23 to 47, and left payroll/cap room as NaN.

Evidence: gm_bench/simulator.py:512-551; permissive parsing begins in examples/gm_agent_common.py:231-237.

• Reject non-finite numeric action inputs with math.isfinite, especially salary and query thresholds.
• Add adversarial tests for NaN, Infinity, and -Infinity at the adapter/parser and simulator boundaries.
• Serialize publication JSON with non-finite values forbidden so poisoned state cannot become an artifact.

0B. Negotiation walk-aways reset per interaction round, not per decision window

The prompt promises that a counterparty stops negotiating after two declines “until your next decision window.” However, League.apply_actions() resets window_walkaways on every call (simulator.py:201-205), while run_decision_point() calls it once per interaction round (runner.py:176-202). A probe produced two declines, then a third decline after the next call instead of the expected “broken off negotiations” result; the counter reset from 2 to 1.

An agent can keep the window alive with an accepted query and resume hidden-value probing on the next round, defeating the stated binary-search guard.

• Reset walk-away state once at decision-window entry, not inside each round's apply_actions.
• Add an end-to-end multi-round regression covering both free-agent and team counterparties.

0C. A tampered compact artifact passes both strict validation and panel analysis

_validate_episode_panel() checks seed/repeat coverage and season counts, but it does not validate finite score fields or recompute summaries, normalized values, or paired statistics from episode rows (official.py:623-656). The publication analyzer checks that publication.raw_artifact_sha256 merely looks like a 64-character hash; it does not bind the compact artifact to the raw artifact (scripts/analyze_publication_panel.py:319-363).

In an in-memory probe, setting every Muse candidate episode score and its displayed aggregates to 9999 still yielded:

• validate_leaderboard_payload(..., SOTA_V2_POLICY).ok == True
• one eligible panel-analysis row
• computed mean lift 9587.381125
• the old declared raw-artifact hash left untouched

The tagged release manifest/checksums are good external evidence, but the routine validator/analyzer/site path does not enforce that link.

• Validate every compact numeric field as finite.
• Recompute candidate summaries, normalized blocks, and paired statistics from compact episode rows and reject mismatches.
• When raw evidence or a release manifest is available, verify publication.raw_artifact_sha256 and compact-to-raw linkage rather than checking hash syntax only.
• Add a tampered-artifact regression that must fail the strict validator, analyzer, and site builder.

Priority 1 — Scoring validity

Problem. The composite is hand-designed and encodes a defensible but non-neutral preference for sustainable asset accumulation. Published compact rows do not retain score components, so weight sensitivity cannot be recomputed on model rows and readers cannot see how much of a result comes from wins/titles versus terminal assets, cap room, strength, and depth.

A prior draft of this issue used an invalid championship-rate comparison. The correct denominators strengthen the need for careful reporting but do not show GLM outperforming pick-trader on titles:

Agent	Score	Titles	Titles / 5-season episode	Title rate / season
pick-trader	411.619	14 / 8 episodes	1.750	35.0%
GLM 5.2	217.539	11 / 24 episodes	0.458	9.2%
Muse Spark 1.1	231.851	8 / 24 episodes	0.333	6.7%

So the current evidence does not establish a “won more, scored less” contradiction. The real limitation is that the composite's value judgments cannot yet be stress-tested on published model end states.

Proposed work:

• Persist per-episode score components in result artifacts (a new contract-lane change) so weight sensitivity can run on model rows.
• Report an outcome-only secondary view (championships, playoff rounds, wins) alongside the composite; label it descriptive rather than silently redefining the primary endpoint.
• Document explicitly that the primary composite rewards sustainable asset accumulation over win-now mortgaging.
• Run a predeclared sensitivity sweep over model-row components and report whether the “all models trail pick-trader” conclusion survives.
• Normalize championships consistently by episodes or total seasons everywhere; never compare candidate repeat totals directly with a deterministic baseline's single-repeat totals.

Priority 2 — Quantify the white-box/scaffold asymmetry

Problem. Every baseline is authored with direct knowledge of the environment and score. The published oracle adds only 19.5 points over pick-trader, but OracleAgent explicitly calls itself a partial hidden-information reference and does not optimize globally (gm_bench/oracle.py:49-62). Therefore 19.5 points is the gain achieved by this particular partial oracle, not proof that pick-trader is near-optimal or that the environment has only 19.5 points of headroom.

The observation paths also differ materially. Scripted agents receive the full simulator observation. The compact model scaffold keeps the full 23-player user roster but preselects at most 16 of 36 initial free agents, 16 of 60 draft prospects, and 12 trade-market entries (examples/gm_agent_common.py:112-181), while also injecting a host-computed legal fallback lineup (:184-227). This is a legitimate model-plus-scaffold condition, but not a controlled “same information, different decision-maker” comparison.

Proposed work:

• Replace “near-optimal” / “near-ceiling” language with “strongest tested white-box reference” unless a real optimization bound is built.
• Build a stronger search/planning oracle or a provable upper bound before interpreting the oracle gap as total strategic headroom.
• Add a same-view reference that operates only on the exact compact payload shown to models, including the same candidate truncation.
• Conversely, run a model companion lane over the untruncated full observation to estimate the cost of scaffold compaction.
• Add a weight-blind baseline track, tuned against a held-out scoring variant, to estimate how much of pick-trader's edge comes from exact score knowledge.
• Report the partial-oracle gap (19.5) next to the MDD (62), with the explicit caveat that the oracle is not an upper bound.

Priority 3 — Separate scaffold effects from model capability

Problem. The frozen lane pins a 4,096-token output ceiling, uses native-minimum reasoning (disabled where optional and lowest supported effort where mandatory), and fresh-spawns the agent between decision points so only a 2,000-character memo carries state. That is a valid, reproducible condition, but it measures model-plus-scaffold behavior under constrained continuity; it should not be generalized to unconstrained model planning capability without companion conditions.

Additional scaffold confounds:

• Soft failure substitution: non-strict fallback_actions still drafts the best public asset and sets a legal lineup while tagging model_error. Failures are counted (decision_failure_rate / sota-v2 ≤2% gate) but still move the roster, biasing scores upward for flaky adapters unless GM_AGENT_STRICT=1.
• "Same lane" ≠ equal effective compute: usage still spans ~7.6k–10.8k tokens/decision across eligible rows; score tracks tokens/decision nearly monotonically (README already notes this).
• Memo / fresh-spawn cost falls almost entirely on models: scripted strategic memo ablation is 0 points because it reconstructs policy from the observation (docs/scoring_calibration.md).

Proposed work:

• Add a reasoning-enabled companion lane (clearly separate from the frozen headline lane) so the reasoning-off result can be read as a condition, not a verdict on capability.
• Promote the --session (full-context) condition to a first-class reported lane; the delta between memo-only and full-context is itself the measurement the benchmark is designed to produce.
• Report an output-token-budget sweep result for at least one model so readers can see how sensitive the score is to the 4,096 ceiling.
• Make GM_AGENT_STRICT=1 (noop-on-failure) the default for any publication lane, or publish both soft-fallback and strict scores so fallback contribution is measurable.
• Show tokens/decision (and preferably input vs output) beside every model row on every public surface that sorts by score — already required by policy; enforce visually so bare-score reading is hard.

---

Priority 4 — Statistical power / the "leaderboard" framing

Problem. With 8 seeds the study cannot separate any two models (all lifts in one tier; pick-trader→oracle band is below the MDD). The 12/16/24-seed MDD rows are extrapolations from resampled 8-seed residuals, not measured. For GPT-5.6 Luna, within-seed run-to-run stddev (35.8) exceeds across-seed stddev (24.8) — model sampling noise is larger than seed variation, absorbed by only 3 repeats. This sits in tension with the new sports-wire "box score" site and publishable_ranking: true.

Holm-adjusted exact sign-flip results for every eligible model vs pick-trader are 0.078125 with holm_reject_at_0_05: false (results/analysis/publication-panel-analysis.json). Unadjusted (p = 0.0078125) (lose on every seed) is real; family-wise significance at α=0.05 is not. "None beat" is an observed-mean claim, not a corrected inference claim.

publishable_ranking: true means "the evidence table may be shown," but the name invites readers to treat the sorted table as a resolved ranking — conflicting with the overlapping-tier / no-ordinal-winner messaging.

Proposed work:

• Reconcile the site presentation with the no-ranking result: the box-score/leaderboard layout implies an ordinal ranking the analysis explicitly withdraws. Add a persistent "why there is no Add repo hygiene: CI, .gitignore, LICENSE, and dev deps #1" callout, or render models as an unranked tier rather than a sorted ladder.
• Rename or gloss publishable_ranking in user-facing copy (e.g. "table publishable / ranking unresolved") so the flag cannot be read as "ranking is valid."
• Increase repeats where within-seed > across-seed stddev (Luna, GLM), or report an explicit note per-row when model noise dominates seed noise.
• Expand the seed panel (even to 12–16) for any future lane that wants to make any model-vs-model claim; label the current 8-seed MDD rows clearly as extrapolation on the site, not just in the doc.
• Put Holm-adjusted vs unadjusted status next to the hero/mean-gap claim wherever "none beat pick-trader" appears.

---

Priority 5 — Validate memo, the mechanism the benchmark exists to test

Problem. memo is sold as what makes "multi-season coherence observable," but it has exactly zero effect for the deterministic reference (per calibration), and there is no ablation isolating whether models benefit from it. The one capability distinguishing this benchmark from a sequence of independent greedy decisions is asserted, not demonstrated.

Proposed work:

• Add a memo-ablation for model rows: run at least one model with memo disabled vs enabled and report the score delta. If models gain nothing, the "long-horizon coherence" claim needs to be dropped or reframed.

---

Priority 6 — Environment realism vs marketing claims

Problem. Several simulator surfaces look like GM realism but do not feed decisions or score, while marketing language overstates opponent sophistication.

• morale is updated from standings (_update_morale_from_standings) and shown in public_dict, but is never consumed by strength, development, or scoring — observation noise / decorative state.
• On-ice sample is tiny (REGULAR_SEASON_GAMES_PER_PAIR = 3 → ~33 games/team). Wins enter score at 0.42 each; seasons are high-variance relative to the weight they carry.
• Opponent FA/draft/trade policies are greedy / formulaic / limited one-for-ones — not learning multi-agent GMs. Site copy that implies "eleven AI rivals" (or similar) overstates the competitive pressure models actually face.
• Protocol competence remains a large measurement surface even after the strategy/protocol split: best published mean_strategy_score ~237 vs pick-trader ~412. Illegal-action and failed-query rates show dense-schema literacy is still part of what is being measured (prompt scaffolding already summarizes score terms).

Proposed work:

• Either wire morale into strength/development/score, or stop emitting it in observations and marketing copy.
• Soften or precise opponent language on the site ("scripted opponent offices," not implied autonomous GMs).
• Optionally offer a higher-games / lower-noise season profile as a companion lane if win terms are meant to discriminate skill rather than luck.
• Keep strategy vs protocol split prominent; add a one-line "what this measures" note that JSON/API literacy is part of the task under the current scaffold.

---

Priority 7 — Site / framing consistency

Scope note. The two Hero findings below are from the current local cursor/sports-wire-redesign-0350 worktree and are not present in committed HEAD 1591b1d. Track them here as pre-merge review findings, not as defects in the tagged phase-one release.

Problem. The findings blog is careful; the redesign's first viewport is not.

• Redesign Hero: "Eight frontier models. None beat a scripted GM" — observed means only; omits Holm non-rejection and observation/scaffold conditions.
• Redesign Hero subcopy uses {modelCount} for "against N scripted heuristics" — that is the model count, not the baseline count (web/src/components/Hero.tsx). Happens to be 8/8 today; still the wrong variable.
• Public seeds 11–18 are committed in-repo; observations include "seed"; league gen and even true_potential are regenerable (gm_bench/oracle.py). Docs correctly call public rows reproducibility artifacts, not contamination-resistant SOTA — the site still leads with them without that caveat in the first viewport.
• Researcher degrees of freedom are logged (docs/PUBLISH_READINESS.md decision log: panel reshuffles, cap policy flips, route substitutions, evidence resets). Good that it exists; public surfaces should point at it so "frozen" is not read as "never forked."

Proposed work:

• Rewrite the hero claim to match the findings blog's narrow durable conclusion (conditions + observed means + no ordinal ranking).
• Fix the baseline-count copy bug (use actual baseline panel length).
• Surface public-panel contamination / adaptation risk on the site, not only in the spec (also listed under Smaller items below).
• Link the publish-readiness decision log from the results / about surface.

---

Smaller items

• Keep the predeclared telemetry-completeness headline gate: excluding Grok and Mistral from the strict lane is conservative and avoids post-hoc imputation. Show them prominently in a separate diagnostic table, and only introduce an imputed-telemetry category prospectively under a frozen rule.
• On any leaderboard-styled surface, guard against the free Tencent HY3 ($0.00) > paid Claude/GPT rows being read as a quality ranking — the no-ranking stance needs to be visually enforced, not just stated in prose.
• Note the public-seed-panel + committed-baselines adaptation risk on the site, not only in the spec.
• Historical proof this class of bug matters: archive the v1 scout prospect_id vs player_id failure mode next to any "models can't GM" claim (up to 1,124 silent scout failures; baselines unscathed) so readers see how protocol bugs can wreck rankings unevenly.
• Website protocol snippets should match production envelopes ({"actions":[...],"usage":...}) where that is what adapters actually emit.

---

What NOT to change

The self-correcting process is the project's biggest asset and should be preserved: pre-registration, Holm correction across the full model family, the strategy/protocol score split, contract/scoring fingerprints, seed-panel hashing, the source-fingerprinted baseline cache, and the exploit canary + regression test. The v1→v2 scout-contract retraction is exemplary. These are why the benchmark's own decomposition was able to show the model deficit is real strategy (pre-penalty strategy_score ~235 vs 411), not JSON-discipline noise — keep that instrument.

Also keep: failed_queries visibility, API vs coding-harness lane separation, usage/cost/latency as first-class comparison fields, and the refusal to silently rerun poor eligible rows for a better number.

Filed from an independent review of the sota-v2-phase-one-2026-07-19 release; augmented with a second independent code/methodology pass (observation truncation, soft fallbacks, public-potential scoring, site framing, environment realism).