Scoring Module Dev: atomic stage-pipeline protocols with official-result validation

[ZTWHHH]

Summary

Introduce the scoring module for simple-mmeval: score result.json outputs against ground truth using dataset-defined protocols from the mm-eval new format (flat sample fields + metadata.json grading config). Every protocol is an ordered list of atomic stages (no composite scorers), so grading is composable, inspectable, and user-definable from the CLI, and reproduces official published results with each dataset's declared pipeline unchanged.

What's included

• mmeval/score.py CLI and scoring pipeline: discover result.json files, run per-sample grading, write score.json with summary and full config provenance.
• Atomic stages: six composable stages replace all composite scorers — exact-match, rule-match (rule-based, free), llm-match, llm-judge (LLM), and the fractional graders vqa-accuracy, anls. Stages run in order (pass → next, score → short-circuit); a grader must be last. LLM/API failures never become verdicts — they surface as llm_error and are re-scored on resume.
• Metadata-driven protocols: read dataset_meta (task_type, score_pipeline, score_params, protocol notes) injected at load time from each dataset's metadata.json. Running the declared list stamps official_protocol: true; --score_pipeline overrides it (precedence: CLI flag > dataset metadata > default).
• New-format contract: grading fields (answer, question_type, etc.) live as flat top-level sample fields; messages carries render inputs only.
• Loader integration: HF/TSV loaders emit the new format and attach dataset_meta for downstream scoring.
• Judge providers: LLM stages run through --judge_provider — local (open-weight model via the framework's own loader, no external service), openai, or azure_openai.

Coverage & validation

• Audited every mm-eval repo: 169 subsets / 128 repos — 158 supported, 11 refused; all declare score_pipeline natively.
• Ten end-to-end checks run each dataset's official pipeline unchanged and agree with official references within ±1.0 pp; per-sample verdicts match the official comparators with zero mismatches across 33,300 samples.

Type of Change

• Bug fix
• New Feature
• New model support
• New dataset support
• Documentation update
• Other (specify):

Testing

• Tested locally
• Added example script (if new model/dataset)

Test command used:

# Command you used to test

Checklist

• Updated mmeval/registery.py (if new model)
• Updated README (if new model/dataset)
• Code follows existing style

[Irisicy4]

Scoring design looks solid overall (fingerprinted resume, llm_error vs wrong separation, refuse-don't-guess). A few things to address — inline comments below, plus one repo-level ask:

Can we trim the ~220k lines of generated result.json/score.json under tests/examples/ (two 95k-line score.json files)? They'll churn on every scoring change; small fixtures (e.g. 20-row slices) or CI-generated outputs would cover the same paths.

[Irisicy4]

judge_include_reason changes the judge prompt ("Include a concise reason" vs "Set reason to an empty string") and judge_max_tokens changes truncation/parse-failure behavior — both affect verdicts but are missing from the resume fingerprint, so resuming across a toggle silently mixes verdicts from two judge configs. Please add both to this LLM branch.

[ZTWHHH]

Done, both are now in the LLM branch of the fingerprint.

[Irisicy4]

Asymmetry: compact "AC" is accepted for predictions here, but parse_gt_letters rejects compact gts — so a dataset storing gt as "AC" scores 0 forever, silently. Suggest making it symmetric by NOT accepting compact form anywhere (it's uncommon and ambiguity-prone) and leaving compact-format datasets to a user patch.

[ZTWHHH]

Rather than dropping compact parsing (it's a real output format), we made it symmetric where it's unambiguous: matchers now accept compact gts once the row is typed mcq, restricted to the sample's actual candidate letters. Type inference alone still refuses compact form since "BED"-style words would misclassify.

[Irisicy4]

accuracy divides by total including invalid samples, which deflates scores on answer-withheld/malformed splits. Please emit both accuracy and accuracy_valid (correct/valid) with a 1-line comment explaining the difference.

[ZTWHHH]

Done, summary now carries both: accuracy (over all rows, invalid counts as wrong) and accuracy_valid (over gradable rows only).

[Irisicy4]

When gt == 0 this turns rel_tol into an absolute tolerance (pred 0.04 passes at rel_tol 0.05). ChartQA's official relaxed accuracy falls back to exact equality at target 0 — please exact-match when gt==0 (with abs_tol still honored), otherwise the official_protocol stamp is inaccurate for these rows.

[ZTWHHH]

Fixed, numbers_match no longer applies numeric_rel_tol at gt == 0 (abs_tol still honored), matching the official implementation (lmms-eval chartqa relaxed_correctness: the target_float truthiness check
routes target==0 to exact match).

[Irisicy4]

If tmp passes the fingerprint we return only tmp's rows, dropping a fuller same-fingerprint final score.json. Union (final first, tmp overwrites) recovers more resumable rows — though that may require synced changes elsewhere (flush/finalize logic), so fine as a follow-up.

[ZTWHHH]

Fixed, final score.json loads first, tmp overlays it, both fingerprint-gated. A completed run also always clears its tmp now, so a stale tmp can never overlay a fresher final.

[Irisicy4]

qwenvl2d5_my (plus *-merged now living in two series) looks like a personal experiment series leaking into the shared registry — and makes get_series order-dependent. Drop or rename before merge.

[ZTWHHH]

Dropped.

[Irisicy4]

Docstring says the loader "ensures grading fields sit at the sample top level", but _process_sample never lifts any fields — the scorer rejects old-layout files instead. Please fix the docstring to match.

[ZTWHHH]

Fixed. Docstring now states the actual contract: rows are expected flat; non-conforming rows pass through unchanged and surface at scoring time via the scorer's explicit layout rejection.

[ZTWHHH]

Scoring design looks solid overall (fingerprinted resume, llm_error vs wrong separation, refuse-don't-guess). A few things to address — inline comments below, plus one repo-level ask:

Can we trim the ~220k lines of generated result.json/score.json under tests/examples/ (two 95k-line score.json files)? They'll churn on every scoring change; small fixtures (e.g. 20-row slices) or CI-generated outputs would cover the same paths.

Done. The four files are now 20-sample runs produced end-to-end by using the framework's native sampling flags.

[Irisicy4]

Thanks for the quick turnaround — fingerprint, tmp/final union, gt==0, accuracy_valid, registry and docstring all look good. Two issues on the new revision:

[Irisicy4]

All four regenerated fixtures are 100% invalid: MMBench 20/20, hf_template 20/20, LLaVABench 60/60, modality_test 18/18 — MMBench's answer is '' (answer-withheld split) and the LLaVABench rows lack reference_response. So the fixtures exercise only the invalid-sample path: zero matcher/grader/judge code is covered, and they'd green-light a fully broken scorer. Please regenerate from a split with answers (e.g. MMBench dev) and add an invalid == 0 assertion to the fixture-generation step.

[ZTWHHH]

Fixed as you suggested.

[Irisicy4]

Note this goes the opposite direction from my suggestion (reject compact everywhere) — accepting compact gts at match time is a defensible alternative, but it has a residual collision: with the A–F fallback candidates (options-in-image datasets), word-like gts parse as letter sets — "BED" → (B,D,E), "FACE" → (A,C,E,F). Please guard it: only accept compact gts when the candidates come from real parsed options, not the A–F fallback (same for template.py / llm.py) — or revert to rejecting compact form.

[ZTWHHH]

Fixed as you suggested. Compact gt parsing now requires the candidates to come from real parsed options; under the A–F fallback a word-like gt ("BED", "FACE") is treated as a whole-string answer instead of a letter set.

[Irisicy4]

This fixture is still 60/60 invalid (and modality_test is 18/18) — the round-2 regeneration covered only the two MMBench fixtures, and the invalid == 0 gate lives only in those two scripts. LLaVABench is the judge-path example, so the llm-judge stage still has zero fixture coverage; everything here exercises only the missing-gt path. Please regenerate it with reference_response present (dev rows) and extend the fixture gate to this script — or drop the degenerate score.json fixtures (LLaVABench, modality_test) rather than committing all-invalid examples.

[ZTWHHH]

Regenerated.

Root cause: LLaVABench keeps its reference answer in gpt4_ans, not the default answer field, so gt resolved to None and every row fell to missing_gt_or_pred; with no dataset_meta the default exact-match,rule-match also ran in place of the declared llm-judge protocol.

Re-scored with --score_gt_field gpt4_ans --score_pipeline llm-judge under an open-weight local judge (Qwen2.5-7B-Instruct, greedy): 60/60 valid, 0 invalid, 0 llm_errors, all 60 rows decided by llm-judge.

[Irisicy4]

The resume fingerprint covers every config knob but carries no dataset identity — and the cache's only binding to its result.json is "same folder". Same-dataset variants collide silently: rerun the same benchmark into the same out_dir with --split test after a dev run (or --subset cn after en, or after the dataset is re-pushed upstream with corrected rows) and the fingerprint passes, eval-ids 0..N collide, and the old split's verdicts are stamped onto the new rows with no trace.

Please have the loader inject dataset_name / subset / split into dataset_meta (it already knows all three) and include them here — that also makes score.json self-describing about what it scored, which VALIDATION.md's table currently asserts from outside the artifact.

(Doesn't cover same-dataset re-inference with a new checkpoint — a per-row sha256(pred) check would; fine as a follow-up.)

[ZTWHHH]

Done. The loader now injects dataset_name/subset/split into dataset_meta (mmeval_hf.py). A rerun of a different split/subset/dataset into the same out_dir is now a cache miss and is re-scored instead of inheriting stale verdicts.

[Irisicy4]

Stale remediation: "the current loader translates published metadata to score_pipeline" is no longer true — the translation boundary was deleted this commit, and the loader now hard-fails on score_type with its own retirement error (mmeval_hf.py). A user following this message re-runs inference and dead-ends on the loader's error, each message pointing at the other component. Please change the remediation here (and keep the loader message consistent) to: "re-push the dataset with the score_pipeline schema, then re-run inference."

[ZTWHHH]

Fixed. Neither message references the old format's history anymore; they state that the score_type key is unsupported and point to the score_pipeline schema (docs/en/SCORING.md, 'Dataset metadata contract'), then re-run inference.

[Irisicy4]

This PR's history includes b0b89a8 ("eval: render MCQ options for HRBench/MMVP"), which added ~15 lines right here in build_prompt: rebuild options from flat per-letter keys (HR-Bench A/B/C/D) or a single options_text string (MMVP) so the template's {% if options %} actually fires. Per that commit's own message, without it "the model never sees the options and scores collapse to ~random" — it was validated as part of this branch's parity work.

The sync-merge cd1fb16 resolved the base.py conflict by taking main's side, and the hunk is gone from the head tree (no options_text anywhere in mmeval/data/) — so the PR's commit log advertises a fix its diff no longer contains. Anyone reading merged history will believe it landed.

Was dropping it intentional? If deferred as out-of-scope for the scoring PR, please say so in the PR description and file a follow-up to restore it; if not, cherry-pick the hunk back from b0b89a8. Either is fine — the current silent state is the only bad option.

[ZTWHHH]

It is intentional. Option normalization was moved out of build_prompt and into the dataset loader part: datasets can declare their own prompt template in metadata.json, and the TSV loader rebuilds the options dict itself, which makes a generic rebuild inside build_prompt redundant.

Both datasets are still covered, and I verified the affected paths end-to-end: mm-eval/MMVP renders through its own "{{ options.options_text }}" template; evalkit's MMVP.tsv is index/image/question/A/B/answer, so tsv.py:158 rebuilds the options dict for the default template; HRBench4K is covered the same way on both paths.

Restoring the hunk would only add dead code: its guards fire only when options is empty and the choices sit in top-level keys, and neither holds in the current sample shape.

[Irisicy4]

Looks good and have reproduction validation now