Skip to content

bench: add reproducible cross-model matrix#107

Merged
jmagly merged 1 commit into
elder-plinius:mainfrom
jmagly:fix/7-14-model-benchmark-matrix
Jul 20, 2026
Merged

bench: add reproducible cross-model matrix#107
jmagly merged 1 commit into
elder-plinius:mainfrom
jmagly:fix/7-14-model-benchmark-matrix

Conversation

@jmagly

@jmagly jmagly commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a versioned model/harness matrix manifest, deterministic aggregator, JSON receipt, and Markdown comparison
  • derive a real Opus 4.7 vs 4.8 pass@1 comparison from 15 identical committed Cybench task artifacts
  • keep success, benchmark failure, abstention, infrastructure error, and skipped outcomes separate
  • integrate matrix verification into verify-claims and normal CI tests
  • document arbitrary multi-model refusal-frontier runs, including no-quota dry runs and API-backed cost boundaries
  • add provider/harness/evidence metadata and multi-model regression coverage to refusal-frontier

Results

The committed common-corpus matrix re-derives:

  • Claude Opus 4.7: 13/15 (86.7%)
  • Claude Opus 4.8: 12/15 (80.0%)

These are historical full-system results, not an isolated model ranking. Every task cell links to its committed oracle-graded source artifact.

Verification

  • npm test — 60 files / 644 tests; 11/11 ops-preflight checks; model-matrix fixtures; 19/19 refusal-frontier checks
  • npm run lint — 0 errors (pre-existing warnings only)
  • npm run typecheck
  • npm run verify-claims — 27/27
  • npm run verify:model-matrix
  • arbitrary two-model --dry-run completed without API access
  • git diff --check

Contribution receipt

  • Scope class: committed benchmark artifacts and offline aggregation
  • Target authority: existing Cybench CTF-range receipts only; no new target execution
  • Network use: GitHub delivery only; no model or target calls
  • Run mode: static aggregation, offline tests, historical tool-backed receipts
  • Redaction: raw historical transcripts remain stripped; no credentials or private target data added
  • Residual risk: run dates/model versions differ, so the matrix explicitly labels itself as a historical system comparison

Closes: #7
Closes: #14

@jmagly
jmagly marked this pull request as ready for review July 20, 2026 18:52
@jmagly
jmagly merged commit 937c6e9 into elder-plinius:main Jul 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Using different models for variant tests Benchmarks with different models

1 participant