OpenMathBench

Reliability benchmark for verified math-problem generation.

OpenMathBench is the benchmark product for OpenMath's core claim: compared with a plain LLM-generation path, OpenMath should produce more mathematically verified, intent-preserving structural/conceptual transformations.

It deliberately reports two layers instead of merging them into one score:

1. Public LLM bench layer — optional contextual scores such as GSM8K/MATH/MathBench for the base model's general math ability.
2. OpenMath reliability layer — task-specific generation reliability: verified yield, gate pass rates, objective/isomorphism failures, retries, latency, and fallback usage.

Run

From this repository:

pnpm bench -- \
  --system full=http://localhost:31415 \
  --system pure_llm=http://localhost:31416 \
  --public-bench "GSM8K|gpt-5.5(xhigh)|92.0|external report" \
  --repeats 5 \
  --concurrency 4

From the OpenMath monorepo workspace:

pnpm -F @openmath/openmathbench bench -- \
  --system full=http://localhost:31415 \
  --system pure_llm=http://localhost:31416 \
  --public-bench "GSM8K|gpt-5.5(xhigh)|92.0|external report" \
  --repeats 5 \
  --concurrency 4

Each --system name=url must point to an OpenMath agent-compatible POST /api/generate SSE endpoint. To compare pipeline variants fairly, start separate agent instances with different env settings, then pass each endpoint as a system.

Example baseline instance:

cd packages/agent
PORT=31416 DETERMINISTIC_FALLBACK=off ./node_modules/.bin/tsx src/index.ts

Smoke-test without live services:

pnpm smoke

Outputs

The CLI writes:

• reports/openmathbench/openmathbench-<timestamp>.json
• reports/openmathbench/openmathbench-<timestamp>.jsonl
• reports/openmathbench/openmathbench-<timestamp>.md

The Markdown report is presentation-ready: it separates public LLM bench context from OpenMathBench reliability metrics and includes per-system, per-mode, per-gate, per-unit, and non-verified run tables.

Interpreting results

Primary success is verified_yield = verified / requested. Warnings are not counted as verified because OpenMath's claim is about problems that pass deterministic verification, not merely problems that look usable.

deterministic-topic-generator in refined_by is reported as template fallback usage. It is useful engineering telemetry, but should not be confused with LLM generation quality.

What this can and cannot prove

OpenMathBench can support claims about verifier-backed generation reliability: verified yield, transparent rejection, gate-level bottlenecks, retries, fallback usage, and latency. It does not prove that one base LLM is smarter than another, and it does not replace human review for Korean naturalness or classroom usefulness.

License

CC BY-NC 4.0, matching OpenMath.