Scoring Module Dev: atomic stage-pipeline protocols with official-result validation

.gitignore

@@ -173,4 +173,7 @@ cython_debug/
 # PyPI configuration file
 .pypirc
 mydata
-.tmp/
+datasets
+.claude
+work_dirs
+.tmp/

docs/en/OPEN_JUDGE.md

@@ -0,0 +1,84 @@
+# Open-weight judge
+
+Two scoring stages use an LLM to grade a sample: `llm-judge` (the LLM judges
+correctness directly) and `llm-match` (the LLM extracts the chosen option for an
+MCQ, then rules compare it). By default these call an OpenAI-compatible API. The
+**local** provider runs an open-weight model in-process instead, so you can grade
+datasets whose official protocol uses an LLM stage without any external service
+or API key.
+
+## Enabling it
+
+```bash
+PYTHONPATH=. python mmeval/score.py --score_out_dir work_dirs/my_run \
+  --score_pipeline llm-judge \
+  --judge_provider local --judge_model Qwen/Qwen2.5-7B-Instruct
+```
+
+The model is loaded with `transformers` (`device_map="auto"`); the first call
+loads the weights and the rest of the run reuses them.
+
+## Configuration
+
+| Flag | Purpose | Default |
+|---|---|---|
+| `--judge_provider` | `local`, `openai`, or `azure_openai` | `openai` |
+| `--judge_model` | for `local`, a Hugging Face causal-LM id | — |
+| `--judge_temperature` | generation temperature | `0.0` |
+| `--judge_max_tokens` | max completion tokens | env `JUDGE_MAX_TOKENS`, else 2048 |
+| `--judge_max_retry` | attempts per call | env `JUDGE_MAX_RETRY`, else 3 |
+| `--judge_concurrency` | max concurrent calls | env `JUDGE_MAX_CONCURRENCY`, else 4 |
+| `--judge_include_reason` | keep the judge's reason in the output | off |
+
+These flags affect only the two LLM stages; a rule-only pipeline (for example
+`exact-match,rule-match`) ignores them.
+
+## Supported models
+
+`--judge_model` accepts any open-weight text instruction model (a Hugging Face
+causal-LM id) — for example `Qwen/Qwen2.5-7B-Instruct`,
+`Qwen/Qwen2.5-72B-Instruct`, `mistralai/Mistral-7B-Instruct-v0.3`, or
+`microsoft/Phi-3.5-mini-instruct`. Which one to use is your choice.
+
+## Measured against a GPT judge
+
+Reliability as a substitute for a GPT judge was measured directly. For each
+baseline below, an open judge re-graded the same model responses that the
+benchmark's official GPT grader had already scored, and the two sets of verdicts
+were compared sample by sample. Baselines are separated by what the recorded GPT
+label actually is: a correctness verdict, which tests `llm-judge`, or an answer
+extraction, which tests `llm-match`.
+
+Per-sample agreement is the primary figure. A dataset score can match by
+coincidence when errors in opposite directions cancel out, whereas agreement
+counts every disagreement.
+
+| Baseline (GPT reference) | Metric | Qwen2.5-7B | Qwen2.5-72B | Mistral-7B | Phi-3.5-mini |
+|---|---|---|---|---|---|
+| **MathVerse** — GPT correctness verdict (`llm-judge`, n=500, GPT score 44.2) | Per-sample agreement | 93.6% | 96.2% | 90.2% | 90.4% |
+| | Δ vs GPT score | +0.8 | +2.2 | +9.8 | +8.4 |
+| **MM-Vet** — GPT graded verdict (`llm-judge`, n=191) | Per-sample agreement † | 88.2% | 86.1% | 85.4% | 81.8% |
+| | Δ vs GPT score | — † | — † | — † | — † |
+| **MathVista** — GPT answer extraction (`llm-match`, n=300, GPT score 45.3) | Per-sample agreement | 97.3% | 97.0% | 98.7% | 98.7% |
+| | Δ vs GPT score | +1.3 | +0.3 | 0.0 | 0.0 |
+
+† MM-Vet's official metric is partial credit (0.0–1.0) while a judge verdict is
+binary, so the dataset-level scores are not directly comparable. Agreement is
+measured on the 144 rows where GPT gave an unambiguous correct/incorrect verdict.
+
+Two limits are worth reading off the table. Agreement depends on the model
+chosen, so these are not one result but a range across the models tested. And all
+three baselines are math and reasoning benchmarks, so agreement in other domains
+is not established.
+
+## Behaviour that affects your runs
+
+- **Deterministic.** Judging is greedy (temperature 0), so re-scoring the same
+  run with the same judge produces byte-identical output.
+- **Resume cache.** On resume a scored row is reused only when the judge
+  fingerprint matches: provider, model, temperature, max tokens, the reason flag,
+  and — for `local` — the model's resolved snapshot revision. Change any of these
+  and the affected rows are re-judged.
+- **Failures are never guessed.** If a judge call fails or its reply cannot be
+  parsed, that row is not turned into a verdict. It is recorded with an
+  `llm_error` reason, counted in `summary.llm_errors`, and re-judged on resume.

docs/en/SCORING.md

@@ -0,0 +1,148 @@
+# Scoring
+
+`mmeval/score.py` grades the `result.json` files produced by inference into
+`score.json` files:
+
+```bash
+PYTHONPATH=. python3 mmeval/score.py --score_out_dir work_dirs/my_run
+```
+
+For how well the framework reproduces official published results, see
+[VALIDATION.md](VALIDATION.md).
+
+## The pipeline model
+
+All scoring is expressed as a **pipeline: an ordered list of atomic stages**.
+There are no composite scorer types — a protocol like "rules first, LLM judge
+for the rest" is written as the list `exact-match,rule-match,llm-judge`.
+The naming scheme: `X-match` = candidate extraction via X (exact form /
+rule heuristics / an LLM) followed by rule comparison; `llm-judge` = the LLM
+decides correctness directly; `vqa-accuracy` / `anls` = named official
+metrics (grader stages).
+
+| Stage | Mechanism | What it does |
+|---|---|---|
+| `exact-match` | rules, free | Strict-form comparison: bare option letter / clean letter set, pure yes/no, whole-string number or text. Near-zero false positives; passes anything ambiguous. |
+| `rule-match` | rules, free | Robust answer extraction from verbose / chain-of-thought output (option-letter inference, answer cues, `\boxed{}`, option-text restatement), then comparison. |
+| `llm-match` | LLM | The LLM extracts the chosen option letter(s); rule-based set compare (VLMEvalKit protocol). MCQ only; passes non-MCQ samples. |
+| `llm-judge` | LLM | The LLM judges correctness directly from question + response + ground truth. Both verdicts decide the sample (yes → 1.0, no → 0.0). |
+| `vqa-accuracy` | grader, fractional | Official VQA consensus accuracy (leave-one-out `min(1, matches/3)` over the multi-annotator answer list). |
+| `anls` | grader, fractional | ANLS: `1 − min` normalized Levenshtein distance over the references, zeroed below the threshold. |
+
+The two LLM stages run their judge through a configurable provider
+(`--judge_provider`): `local` (an in-process open-weight model — no external
+service), `openai`, or `azure_openai`. The stage logic, prompt, retry, and
+`llm_error` accounting are identical across providers. For running the local
+open-weight judge and how its scores relate to GPT-judged numbers, see
+[OPEN_JUDGE.md](OPEN_JUDGE.md).
+
+### Execution semantics
+
+Stages run in order; each returns one of three outcomes:
+
+- **pass** — the stage cannot decide the sample; the next stage runs. If every
+  stage passes, the sample scores `0.0` with `decided_by: null` (undecided,
+  counted wrong). Rule stages never decide a miss — they pass instead, so a
+  later stage can still grade the sample.
+- **score** — the stage decided; the pipeline short-circuits. Each sample row
+  records `score` (0..1), `is_correct` (`1` only at full credit) and
+  `decided_by` (the stage name). Fractional protocols read
+  `summary.mean_score` as the headline; `summary.accuracy` is the strict
+  full-credit rate.
+- **invalid** — the sample cannot be graded under the stage's protocol (e.g.
+  `vqa-accuracy` without the multi-annotator reference list); recorded as
+  `status: invalid` with the stage's reason.
+
+The grader stages (`vqa-accuracy`, `anls`) always decide, so they are only
+valid as the **last** stage, at most one per pipeline (validated with a clear
+error). Compositions like
+`exact-match,anls` (exact tier first, fractional similarity for the rest) are
+legal pipelines.
+
+LLM/API failures never become verdicts: the failing stage passes with an
+`llm_error: ...` reason that survives into the sample row and
+`summary.llm_errors`, and such rows are re-scored on resume — never guessed.
+
+### CLI
+
+```bash
+--score_pipeline exact-match,rule-match     # default
+--score_pipeline rule-match,llm-match       # MCQ letter-extraction protocol
+--score_pipeline llm-judge                  # pure LLM judging
+--score_pipeline vqa-accuracy               # official VQA accuracy
+```
+
+Stage parameters are pipeline-level flags, consumed by the stages that use
+them (per-knob precedence: explicit CLI flag > dataset metadata > default;
+`score.json config.knob_sources` records who decided each knob):
+
+| Flag | Consumed by |
+|---|---|
+| `--score_numeric_rel_tol`, `--score_numeric_abs_tol` | `exact-match`, `rule-match` (numeric answers) |
+| `--score_string_match` (`exact`/`contains`/`anls`) | `exact-match`, `rule-match` (open answers) |
+| `--score_anls_threshold` | `anls` and the `anls` string-match mode of the rule-based stages |
+| `--judge_provider/model/temperature/max_tokens/...` | the LLM stages (`llm-match`, `llm-judge`) |
+
+## Dataset metadata contract
+
+mm-eval datasets declare their official protocol per subset in the repo-root
+`metadata.json`. The scorer reads it from the `dataset_meta` block the loader
+injects into every sample (so it travels inside `result.json`):
+
+```jsonc
+"subsets": {
+  "en": {
+    "task_type": "multiple_choice_qa",
+    "score_pipeline": ["rule-match", "llm-match"],   // the official protocol
+    "score_params": {"numeric_rel_tol": 0.05},            // optional stage params
+    "score_protocol": {"note": "..."}                     // verbatim protocol note
+  }
+}
+```
+
+`score_pipeline` takes one of three forms:
+
+1. **A stage list** — the official protocol; running it stamps
+   `config.official_protocol: true`. An explicit `--score_pipeline` that differs
+   overrides it, stamps the output `official_protocol: false`, and appends a
+   protocol note saying so.
+2. **`[]`** — the dataset explicitly declares that **no official protocol
+   exists**; the scorer runs the default pipeline and stamps
+   `official_protocol: false` with a note.
+3. **`{"unsupported": "<protocol>", "reason": "..."}`** — the official
+   protocol cannot be executed by this scorer (e.g. corpus-level caption
+   metrics, code execution). Scoring is **refused** with the reason; an
+   explicit `--score_pipeline` forces approximate scoring, stamped non-official.
+
+Datasets with no `score_pipeline` declaration score under the default
+pipeline with `official_protocol: null` (unknown).
+
+### Metadata schema
+
+Every dataset on the mm-eval org declares its scoring protocol with
+`score_pipeline` (described above). A `score_type` key is not supported and is
+rejected with an actionable error: the loader refuses a metadata.json that
+carries it, and the scorer refuses a `result.json` whose `dataset_meta`
+carries it. In both cases the fix is the same — declare the protocol with the
+`score_pipeline` schema, then re-run inference.
+
+The audited per-subset pipeline for all published datasets is listed in
+[SCORING_COVERAGE.md](SCORING_COVERAGE.md).
+
+## Resume
+
+Scoring resumes from `score.json` / `score.json.tmp` caches keyed by a config
+fingerprint that captures everything verdict-determining: the **dataset
+identity** (`dataset_name` / `subset` / `split`, carried in each sample's
+`dataset_meta`), the resolved pipeline, stage params, and — only when an LLM
+stage is present — the judge identity
+(provider/model/temperature/include_reason/resolved max_tokens), plus the
+model's resolved snapshot revision for the `local` provider. Any change
+invalidates the cache; rule-only pipelines survive judge-config edits. The
+dataset-identity keys mean a cache produced for a different split/subset (or a
+re-pushed dataset) scored into the same `out_dir` is discarded rather than
+reused with colliding eval-ids — an old `result.json` without identity keys
+matches only another identity-less run. The fingerprint is embedded verbatim
+in `score.json`'s `config` block, so a score file is self-describing about
+what it scored. Rows that failed with `llm_error` are always re-scored.
+Reruns of an unchanged config are byte-identical.