Harden benchmark integrity: mispriced tokens, false-certainty stat, COI + parity disclosure#23
Draft
macanderson wants to merge 1 commit into
Draft
Harden benchmark integrity: mispriced tokens, false-certainty stat, COI + parity disclosure#23macanderson wants to merge 1 commit into
macanderson wants to merge 1 commit into
Conversation
…t, disclose COI + parity gaps Pre-launch fairness/methodology audit fixes for the benchmark. Statistics - wilsonInterval(0, n=0) returned [0,0] — false certainty of 0% from zero data. A run with zero scored trials (e.g. every trial excluded as agent-error) would make `arena gate --require-significant` certify a "100% resolve-rate drop". Now returns [0,1] (maximal uncertainty); test updated. - Paired-bootstrap CI used floor(0.025N)/ceil(0.975N) — asymmetric tails (trims 2.5% below, 2.45% above). Switched to the type-1 empirical quantile (ceil(q·N)-1) on both bounds. Pricing - pricing.json priced Anthropic Opus 4.8 at $15/$75 — 3x the real $5/$25 — and Haiku 4.5 at $0.8/$4 vs $1/$5. Every computed cost involving those models (and any unmatched run against a cheaper competitor) was fiction. Corrected to published rates; added source URL + asOf date + a cache-write-TTL note to each Anthropic entry, matching the "verify before publishing" discipline the third-party entries already carried. Claims (README + web) — disclose what the code actually does - Added a prominent conflict-of-interest note: Arena is built by the team behind Oxagen and Stella, two of the agents it benchmarks. METHODOLOGY already required this of publishers; the project now meets its own bar. - "held-out tests the agent can never see or author" was false for local runs (the verify suites sit on disk and are public in this repo). Reworded to "copied in only after the agent exits / not in the workspace", with an explicit not-network-sandboxed caveat pointing at Harbor. - "Autonomy levels are matched as closely as each CLI allows" was contradicted by the adapters: claude-code/gemini are capped at edits-only while oxagen/stella run one-shot ungated (shell included) — an advantage for the author's agents on test-graded tasks. Disclosed the asymmetry and the matched-autonomy flags, and noted the full-env-passthrough (every CLI sees every vendor's keys). Not changed here (needs an owner decision, see the audit report): pinning the adapters to a matched autonomy level, nulling tokens/cost on envelope-parse failure, decoupling the agent-error classifier from parsed token counts, and env-allowlisting spawned agents. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Sorry @macanderson, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-launch adversarial audit of the benchmark's fairness and methodology. A benchmark that ranks the author's own agents against competitors gets the harshest scrutiny in software — these are the things a hostile expert would lead with. All fixes verified:
tscclean, 53/53 vitest, biome clean.📉 Statistics
wilsonInterval(0, n=0)returned[0,0]— false certainty of 0% from zero data. A run where every trial is excluded asagent-error(scoredTrials = 0) would makearena gate --require-significantcertify a "100% resolve-rate drop" — the exact case the agent-error category exists to prevent. Now returns[0,1].floor(0.025N)/ceil(0.975N)— trims 2.5% below but 2.45% above). Switched to the type-1 empirical quantileceil(q·N)-1on both bounds.💵 Pricing (
pricing.json)computedUsdinvolving those models was fiction, and any unmatched run against a cheaper competitor inflated the competitor's cost gap ~3×. Corrected to published rates, and addedsource+asOf+ a cache-write-TTL note to each Anthropic entry (the third-party entries already carried "verify before publishing" notes; the wrong ones didn't).🏷️ Claims vs. reality (README + web)
claude-code/geminiare capped at edits-only whileoxagen/stellarun one-shot ungated (shell included) — a real edge for the author's agents on test-graded tasks. Disclosed, with the matched-autonomy flags and the full-env-passthrough caveat.🧭 Deliberately left for an owner decision (documented in the audit report, not silently changed here)
These change measured behavior or need a judgment call:
--dangerously-skip-permissions/--approval-mode yolofor all).0 tokens / $0.00(today one CLI JSON-shape drift silently hands that vendor the cost column).agent-errorclassifier from parsed token counts (currently vendor-asymmetric).PWD, don't hand every CLI every vendor's keys).Part of a two-repo launch audit (Arena + Stella); the companion report ranks the top 25 pitfalls across both.
🤖 Generated with Claude Code