Skip to content

SPEC.md

Latest commit

 

History

History
31 lines (21 loc) · 1.06 KB

SPEC.md

File metadata and controls

31 lines (21 loc) · 1.06 KB
Raw

OpenMathBench Spec

OpenMathBench reports two layers and never averages them into one opaque score:

  1. Public LLM bench context: externally measured model-level math ability supplied as metadata rows.
  2. OpenMath reliability benchmark: task-specific generation reliability measured through the same OpenMath-compatible POST /api/generate SSE contract for every compared system.

Primary metric:

verified_yield = rows where overall == "verified" / requested rows

Default matrix:

systems × 12 demo-safe units × 2 modes × 5 repeats

Required report outputs:

  • reports/openmathbench/openmathbench-<timestamp>.json
  • reports/openmathbench/openmathbench-<timestamp>.jsonl
  • reports/openmathbench/openmathbench-<timestamp>.md

Non-goals:

  • Do not average public LLM benchmark scores with OpenMathBench reliability metrics.
  • Do not count only accepted OpenMath outputs; rejected/missing rows remain part of the denominator.
  • Do not use LLM judges as correctness proof.
  • Do not claim whole-curriculum coverage from the v0 12-unit suite.