diff --git a/.gitignore b/.gitignore index 94a66840..63adecaf 100644 --- a/.gitignore +++ b/.gitignore @@ -173,4 +173,7 @@ cython_debug/ # PyPI configuration file .pypirc mydata -.tmp/ \ No newline at end of file +datasets +.claude +work_dirs +.tmp/ diff --git a/docs/en/OPEN_JUDGE.md b/docs/en/OPEN_JUDGE.md new file mode 100644 index 00000000..5fd151cd --- /dev/null +++ b/docs/en/OPEN_JUDGE.md @@ -0,0 +1,84 @@ +# Open-weight judge + +Two scoring stages use an LLM to grade a sample: `llm-judge` (the LLM judges +correctness directly) and `llm-match` (the LLM extracts the chosen option for an +MCQ, then rules compare it). By default these call an OpenAI-compatible API. The +**local** provider runs an open-weight model in-process instead, so you can grade +datasets whose official protocol uses an LLM stage without any external service +or API key. + +## Enabling it + +```bash +PYTHONPATH=. python mmeval/score.py --score_out_dir work_dirs/my_run \ + --score_pipeline llm-judge \ + --judge_provider local --judge_model Qwen/Qwen2.5-7B-Instruct +``` + +The model is loaded with `transformers` (`device_map="auto"`); the first call +loads the weights and the rest of the run reuses them. + +## Configuration + +| Flag | Purpose | Default | +|---|---|---| +| `--judge_provider` | `local`, `openai`, or `azure_openai` | `openai` | +| `--judge_model` | for `local`, a Hugging Face causal-LM id | — | +| `--judge_temperature` | generation temperature | `0.0` | +| `--judge_max_tokens` | max completion tokens | env `JUDGE_MAX_TOKENS`, else 2048 | +| `--judge_max_retry` | attempts per call | env `JUDGE_MAX_RETRY`, else 3 | +| `--judge_concurrency` | max concurrent calls | env `JUDGE_MAX_CONCURRENCY`, else 4 | +| `--judge_include_reason` | keep the judge's reason in the output | off | + +These flags affect only the two LLM stages; a rule-only pipeline (for example +`exact-match,rule-match`) ignores them. + +## Supported models + +`--judge_model` accepts any open-weight text instruction model (a Hugging Face +causal-LM id) — for example `Qwen/Qwen2.5-7B-Instruct`, +`Qwen/Qwen2.5-72B-Instruct`, `mistralai/Mistral-7B-Instruct-v0.3`, or +`microsoft/Phi-3.5-mini-instruct`. Which one to use is your choice. + +## Measured against a GPT judge + +Reliability as a substitute for a GPT judge was measured directly. For each +baseline below, an open judge re-graded the same model responses that the +benchmark's official GPT grader had already scored, and the two sets of verdicts +were compared sample by sample. Baselines are separated by what the recorded GPT +label actually is: a correctness verdict, which tests `llm-judge`, or an answer +extraction, which tests `llm-match`. + +Per-sample agreement is the primary figure. A dataset score can match by +coincidence when errors in opposite directions cancel out, whereas agreement +counts every disagreement. + +| Baseline (GPT reference) | Metric | Qwen2.5-7B | Qwen2.5-72B | Mistral-7B | Phi-3.5-mini | +|---|---|---|---|---|---| +| **MathVerse** — GPT correctness verdict (`llm-judge`, n=500, GPT score 44.2) | Per-sample agreement | 93.6% | 96.2% | 90.2% | 90.4% | +| | Δ vs GPT score | +0.8 | +2.2 | +9.8 | +8.4 | +| **MM-Vet** — GPT graded verdict (`llm-judge`, n=191) | Per-sample agreement † | 88.2% | 86.1% | 85.4% | 81.8% | +| | Δ vs GPT score | — † | — † | — † | — † | +| **MathVista** — GPT answer extraction (`llm-match`, n=300, GPT score 45.3) | Per-sample agreement | 97.3% | 97.0% | 98.7% | 98.7% | +| | Δ vs GPT score | +1.3 | +0.3 | 0.0 | 0.0 | + +† MM-Vet's official metric is partial credit (0.0–1.0) while a judge verdict is +binary, so the dataset-level scores are not directly comparable. Agreement is +measured on the 144 rows where GPT gave an unambiguous correct/incorrect verdict. + +Two limits are worth reading off the table. Agreement depends on the model +chosen, so these are not one result but a range across the models tested. And all +three baselines are math and reasoning benchmarks, so agreement in other domains +is not established. + +## Behaviour that affects your runs + +- **Deterministic.** Judging is greedy (temperature 0), so re-scoring the same + run with the same judge produces byte-identical output. +- **Resume cache.** On resume a scored row is reused only when the judge + fingerprint matches: provider, model, temperature, max tokens, the reason flag, + and — for `local` — the model's resolved snapshot revision. Change any of these + and the affected rows are re-judged. +- **Failures are never guessed.** If a judge call fails or its reply cannot be + parsed, that row is not turned into a verdict. It is recorded with an + `llm_error` reason, counted in `summary.llm_errors`, and re-judged on resume. diff --git a/docs/en/SCORING.md b/docs/en/SCORING.md new file mode 100644 index 00000000..5a466831 --- /dev/null +++ b/docs/en/SCORING.md @@ -0,0 +1,148 @@ +# Scoring + +`mmeval/score.py` grades the `result.json` files produced by inference into +`score.json` files: + +```bash +PYTHONPATH=. python3 mmeval/score.py --score_out_dir work_dirs/my_run +``` + +For how well the framework reproduces official published results, see +[VALIDATION.md](VALIDATION.md). + +## The pipeline model + +All scoring is expressed as a **pipeline: an ordered list of atomic stages**. +There are no composite scorer types — a protocol like "rules first, LLM judge +for the rest" is written as the list `exact-match,rule-match,llm-judge`. +The naming scheme: `X-match` = candidate extraction via X (exact form / +rule heuristics / an LLM) followed by rule comparison; `llm-judge` = the LLM +decides correctness directly; `vqa-accuracy` / `anls` = named official +metrics (grader stages). + +| Stage | Mechanism | What it does | +|---|---|---| +| `exact-match` | rules, free | Strict-form comparison: bare option letter / clean letter set, pure yes/no, whole-string number or text. Near-zero false positives; passes anything ambiguous. | +| `rule-match` | rules, free | Robust answer extraction from verbose / chain-of-thought output (option-letter inference, answer cues, `\boxed{}`, option-text restatement), then comparison. | +| `llm-match` | LLM | The LLM extracts the chosen option letter(s); rule-based set compare (VLMEvalKit protocol). MCQ only; passes non-MCQ samples. | +| `llm-judge` | LLM | The LLM judges correctness directly from question + response + ground truth. Both verdicts decide the sample (yes → 1.0, no → 0.0). | +| `vqa-accuracy` | grader, fractional | Official VQA consensus accuracy (leave-one-out `min(1, matches/3)` over the multi-annotator answer list). | +| `anls` | grader, fractional | ANLS: `1 − min` normalized Levenshtein distance over the references, zeroed below the threshold. | + +The two LLM stages run their judge through a configurable provider +(`--judge_provider`): `local` (an in-process open-weight model — no external +service), `openai`, or `azure_openai`. The stage logic, prompt, retry, and +`llm_error` accounting are identical across providers. For running the local +open-weight judge and how its scores relate to GPT-judged numbers, see +[OPEN_JUDGE.md](OPEN_JUDGE.md). + +### Execution semantics + +Stages run in order; each returns one of three outcomes: + +- **pass** — the stage cannot decide the sample; the next stage runs. If every + stage passes, the sample scores `0.0` with `decided_by: null` (undecided, + counted wrong). Rule stages never decide a miss — they pass instead, so a + later stage can still grade the sample. +- **score** — the stage decided; the pipeline short-circuits. Each sample row + records `score` (0..1), `is_correct` (`1` only at full credit) and + `decided_by` (the stage name). Fractional protocols read + `summary.mean_score` as the headline; `summary.accuracy` is the strict + full-credit rate. +- **invalid** — the sample cannot be graded under the stage's protocol (e.g. + `vqa-accuracy` without the multi-annotator reference list); recorded as + `status: invalid` with the stage's reason. + +The grader stages (`vqa-accuracy`, `anls`) always decide, so they are only +valid as the **last** stage, at most one per pipeline (validated with a clear +error). Compositions like +`exact-match,anls` (exact tier first, fractional similarity for the rest) are +legal pipelines. + +LLM/API failures never become verdicts: the failing stage passes with an +`llm_error: ...` reason that survives into the sample row and +`summary.llm_errors`, and such rows are re-scored on resume — never guessed. + +### CLI + +```bash +--score_pipeline exact-match,rule-match # default +--score_pipeline rule-match,llm-match # MCQ letter-extraction protocol +--score_pipeline llm-judge # pure LLM judging +--score_pipeline vqa-accuracy # official VQA accuracy +``` + +Stage parameters are pipeline-level flags, consumed by the stages that use +them (per-knob precedence: explicit CLI flag > dataset metadata > default; +`score.json config.knob_sources` records who decided each knob): + +| Flag | Consumed by | +|---|---| +| `--score_numeric_rel_tol`, `--score_numeric_abs_tol` | `exact-match`, `rule-match` (numeric answers) | +| `--score_string_match` (`exact`/`contains`/`anls`) | `exact-match`, `rule-match` (open answers) | +| `--score_anls_threshold` | `anls` and the `anls` string-match mode of the rule-based stages | +| `--judge_provider/model/temperature/max_tokens/...` | the LLM stages (`llm-match`, `llm-judge`) | + +## Dataset metadata contract + +mm-eval datasets declare their official protocol per subset in the repo-root +`metadata.json`. The scorer reads it from the `dataset_meta` block the loader +injects into every sample (so it travels inside `result.json`): + +```jsonc +"subsets": { + "en": { + "task_type": "multiple_choice_qa", + "score_pipeline": ["rule-match", "llm-match"], // the official protocol + "score_params": {"numeric_rel_tol": 0.05}, // optional stage params + "score_protocol": {"note": "..."} // verbatim protocol note + } +} +``` + +`score_pipeline` takes one of three forms: + +1. **A stage list** — the official protocol; running it stamps + `config.official_protocol: true`. An explicit `--score_pipeline` that differs + overrides it, stamps the output `official_protocol: false`, and appends a + protocol note saying so. +2. **`[]`** — the dataset explicitly declares that **no official protocol + exists**; the scorer runs the default pipeline and stamps + `official_protocol: false` with a note. +3. **`{"unsupported": "", "reason": "..."}`** — the official + protocol cannot be executed by this scorer (e.g. corpus-level caption + metrics, code execution). Scoring is **refused** with the reason; an + explicit `--score_pipeline` forces approximate scoring, stamped non-official. + +Datasets with no `score_pipeline` declaration score under the default +pipeline with `official_protocol: null` (unknown). + +### Metadata schema + +Every dataset on the mm-eval org declares its scoring protocol with +`score_pipeline` (described above). A `score_type` key is not supported and is +rejected with an actionable error: the loader refuses a metadata.json that +carries it, and the scorer refuses a `result.json` whose `dataset_meta` +carries it. In both cases the fix is the same — declare the protocol with the +`score_pipeline` schema, then re-run inference. + +The audited per-subset pipeline for all published datasets is listed in +[SCORING_COVERAGE.md](SCORING_COVERAGE.md). + +## Resume + +Scoring resumes from `score.json` / `score.json.tmp` caches keyed by a config +fingerprint that captures everything verdict-determining: the **dataset +identity** (`dataset_name` / `subset` / `split`, carried in each sample's +`dataset_meta`), the resolved pipeline, stage params, and — only when an LLM +stage is present — the judge identity +(provider/model/temperature/include_reason/resolved max_tokens), plus the +model's resolved snapshot revision for the `local` provider. Any change +invalidates the cache; rule-only pipelines survive judge-config edits. The +dataset-identity keys mean a cache produced for a different split/subset (or a +re-pushed dataset) scored into the same `out_dir` is discarded rather than +reused with colliding eval-ids — an old `result.json` without identity keys +matches only another identity-less run. The fingerprint is embedded verbatim +in `score.json`'s `config` block, so a score file is self-describing about +what it scored. Rows that failed with `llm_error` are always re-scored. +Reruns of an unchanged config are byte-identical. diff --git a/docs/en/SCORING_COVERAGE.md b/docs/en/SCORING_COVERAGE.md new file mode 100644 index 00000000..43e05c35 --- /dev/null +++ b/docs/en/SCORING_COVERAGE.md @@ -0,0 +1,208 @@ +# Scoring pipeline coverage — mm-eval org audit + +Survey of every dataset repo on https://huggingface.co/mm-eval: the atomic +`score_pipeline` each repo declares per subset in its `metadata.json` (see +`docs/en/SCORING.md`, "Metadata schema"). Every supported pipeline below was +validated against the stage registry; REFUSED rows raise an explicit, +actionable error instead of silently mis-grading. + +**169 subsets across 128 dataset repos: 158 supported, 11 refused.** (`mm-eval/VLMEvalKit` carries no metadata.json — it is a raw TSV mirror, not an mm-eval-format dataset — and is excluded.) + +## Stage usage census + +Official-protocol demand per atomic stage (datasets / subsets): + +- `exact-match`: 61 datasets / 80 subsets +- `rule-match`: 91 datasets / 127 subsets +- `llm-match`: 30 datasets / 47 subsets +- `llm-judge`: 30 datasets / 39 subsets +- `vqa-accuracy`: 4 datasets / 4 subsets +- `anls`: 4 datasets / 4 subsets + +Narrowest stages: `vqa-accuracy` (VQAv2, OK-VQA, TextVQA, VizWiz-VQA) and +`anls` (DocVQA, InfographicVQA, ST-VQA, ChartNet; the ANLS comparator also +backs `string_match=anls`, e.g. ChartQAPro) — 4 datasets each, above the +1–2-dataset prune threshold, and both are the ecosystem's canonical +fractional metrics. + +| Dataset | Subset | score_pipeline | Status | Notes | +|---|---|---|---|---| +| 3DSRBench | main | rule-match,llm-match | supported | | +| A-OKVQA | main | exact-match,rule-match | supported | | +| AI2D | main | rule-match,llm-match | supported | | +| AlgoPuzzleVQA | default | exact-match,rule-match | supported | | +| ArxivQA | main | rule-match,llm-match | supported | | +| BLINK | Art_Style | rule-match,llm-match | supported | | +| BLINK | Counting | rule-match,llm-match | supported | | +| BLINK | Forensic_Detection | rule-match,llm-match | supported | | +| BLINK | Functional_Correspondence | rule-match,llm-match | supported | | +| BLINK | IQ_Test | rule-match,llm-match | supported | | +| BLINK | Jigsaw | rule-match,llm-match | supported | | +| BLINK | Multi_view_Reasoning | rule-match,llm-match | supported | | +| BLINK | Object_Localization | rule-match,llm-match | supported | | +| BLINK | Relative_Depth | rule-match,llm-match | supported | | +| BLINK | Relative_Reflectance | rule-match,llm-match | supported | | +| BLINK | Semantic_Correspondence | rule-match,llm-match | supported | | +| BLINK | Spatial_Relation | rule-match,llm-match | supported | | +| BLINK | Visual_Correspondence | rule-match,llm-match | supported | | +| BLINK | Visual_Similarity | rule-match,llm-match | supported | | +| BenchLMM | main | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [0, 1], 'official_judge_model': 'gpt-4-0613'} | +| CMMMU | art_and_design | exact-match,rule-match | supported | | +| CMMMU | business | exact-match,rule-match | supported | | +| CMMMU | health_and_medicine | exact-match,rule-match | supported | | +| CMMMU | humanities_and_social_sciences | exact-match,rule-match | supported | | +| CMMMU | science | exact-match,rule-match | supported | | +| CMMMU | technology_and_engineering | exact-match,rule-match | supported | | +| CRPE | exist | exact-match,rule-match | supported | | +| CRPE | relation | exact-match,rule-match | supported | | +| CV-Bench | main | exact-match,rule-match | supported | | +| CVQA | main | rule-match,llm-match | supported | | +| CharXiv | reasoning | exact-match,rule-match,llm-judge | supported | | +| ChartNet | main | anls | supported | score_params: {'threshold': 0.0} | +| ChartQA | main | exact-match,rule-match | supported | score_params: {'numeric_rel_tol': 0.05} | +| ChartQAPro | main | exact-match,rule-match | supported | score_params: {'numeric_rel_tol': 0.05, 'string_match': 'anls', 'anls_threshold': 0.5} | +| DocVQA | main | anls | supported | score_params: {'threshold': 0.5} | +| DynaMath | main | rule-match,llm-match | supported | score_params: {'numeric_abs_tol': 0.001} | +| EMMA | main | exact-match,rule-match,llm-judge | supported | | +| ENEM | main | exact-match,rule-match | supported | | +| EXAMS-V | default | exact-match,rule-match | supported | | +| EndoBench | main | rule-match,llm-match | supported | | +| Ferret-Bench | default | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [1, 10], 'official_judge_model': 'gpt-4-0314'} | +| Flickr8k | main | unsupported: caption-metrics | REFUSED | requires corpus-level caption metrics (CIDEr/BLEU/ROUGE/BERTScore); use the benchmark's own evaluator (score_params: {'metrics': ['cider', 'bleu4', 'meteor', 'rouge_l']}) | +| GMAI-MMBench | default | rule-match,llm-match | supported | | +| GQA | main | exact-match,rule-match | supported | score_params: {'string_match': 'exact'} | +| GQA-Spatial | main | exact-match,rule-match | supported | | +| Geometry3K | main | [] (no official protocol) | supported | default pipeline, stamped non-official | +| HRBench4K | main | rule-match,llm-match | supported | | +| HRBench8K | main | rule-match,llm-match | supported | | +| HallusionBench | main | exact-match,rule-match,llm-judge | supported | score_params: {'judge_inputs': ['gt_answer_details']} | +| HallusionBench | non_image | exact-match,rule-match,llm-judge | supported | score_params: {'judge_inputs': ['gt_answer_details']} | +| HumanEval-V | main | unsupported: code-execution | REFUSED | requires official test-case execution (pass@1); use the benchmark's own evaluator (score_params: {'metric': 'pass@1'}) | +| IconQA | choose_img | exact-match,rule-match | supported | | +| IconQA | choose_txt | exact-match,rule-match | supported | | +| IconQA | fill_in_blank | exact-match,rule-match | supported | score_params: {'string_match': 'exact'} | +| InfographicVQA | main | anls | supported | score_params: {'threshold': 0.5} | +| LEGO-Puzzles | default | rule-match,llm-match | supported | | +| LLaVA-Bench-COCO | default | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [1, 10], 'official_judge_model': 'gpt-4-0314'} | +| LLaVA-Bench-Wilder | main | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [1, 10], 'official_judge_model': 'gpt-4-vision-preview'} | +| LLaVA-Bench-in-the-Wild | main | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [1, 10], 'official_judge_model': 'gpt-4-0314', 'judge_inputs': ['caption']} | +| LiveBench | 2024-05 | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [0, 10], 'official_judge_model': 'gpt-4o'} | +| LiveBench | 2024-06 | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [0, 10], 'official_judge_model': 'gpt-4o', 'judge_inputs': ['criteria']} | +| LiveBench | 2024-07 | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [0, 10], 'official_judge_model': 'gpt-4o', 'judge_inputs': ['criteria']} | +| LiveBench | 2024-08 | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [0, 10], 'official_judge_model': 'gpt-4o', 'judge_inputs': ['criteria']} | +| LiveBench | 2024-09 | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [0, 10], 'official_judge_model': 'gpt-4o', 'judge_inputs': ['criteria']} | +| LogicVista | main | rule-match,llm-match | supported | | +| M3CoT | main | exact-match,rule-match | supported | | +| MEGA-Bench | main | unsupported: per-task-metric | REFUSED | requires the benchmark's own evaluator (per-row metric dispatch) (score_params: {'metric_field': 'task_name'}) | +| MM-IFEval | main | llm-judge | supported | score_params: {'rubric': 'partial_credit', 'judge_inputs': ['constraints']} | +| MM-Vet-v2 | default | llm-judge | supported | score_params: {'rubric': 'partial_credit', 'scale': [0.0, 1.0], 'official_judge_model': 'gpt-4-0613', 'reference_syntax': 'and_or_composition'} | +| MMBench | cc | rule-match,llm-match | supported | | +| MMBench | cn | rule-match,llm-match | supported | | +| MMBench | en | rule-match,llm-match | supported | | +| MMBench-V11 | cn | rule-match,llm-match | supported | | +| MMBench-V11 | en | rule-match,llm-match | supported | | +| MME | main | exact-match,rule-match | supported | | +| MME-RealWorld-Lite | main | exact-match,rule-match | supported | | +| MMEvalPro | main | exact-match,rule-match | supported | | +| MMHal-Bench | main | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [0, 6], 'official_judge_model': 'gpt-4', 'judge_inputs': ['image_content']} | +| MMIU | default | rule-match,llm-match | supported | | +| MMMU | main | exact-match,rule-match | supported | | +| MMMU-Pro | standard_10_options | exact-match,rule-match | supported | | +| MMMU-Pro | standard_4_options | exact-match,rule-match | supported | | +| MMMU-Pro | vision | exact-match,rule-match | supported | | +| MMStar | main | rule-match,llm-match | supported | | +| MMT-Bench | main | rule-match,llm-match | supported | | +| MMVP | main | rule-match,llm-match | supported | | +| MMVU | main | exact-match,rule-match,llm-judge | supported | | +| MMVet | main | llm-judge | supported | score_params: {'rubric': 'partial_credit', 'scale': [0.0, 1.0], 'official_judge_model': 'gpt-4-0613', 'reference_syntax': 'and_or_composition'} | +| MMVet-v2 | main | llm-judge | supported | score_params: {'rubric': 'partial_credit', 'scale': [0.0, 1.0], 'official_judge_model': 'gpt-4-0613', 'reference_syntax': 'and_or_composition'} | +| MS-COCO-Captions | main | unsupported: caption-metrics | REFUSED | requires corpus-level caption metrics (CIDEr/BLEU/ROUGE/BERTScore); use the benchmark's own evaluator (score_params: {'metrics': ['cider', 'bleu4', 'meteor', 'rouge_l']}) | +| MTVQA | main | exact-match,rule-match | supported | score_params: {'string_match': 'contains'} | +| MaRVL | main | exact-match,rule-match | supported | | +| Mantis-Eval | main | exact-match,rule-match | supported | | +| MathVerse | testmini | exact-match,rule-match,llm-judge | supported | | +| MathVerse | testmini_text_only | exact-match,rule-match,llm-judge | supported | | +| MathVision | main | exact-match,rule-match | supported | | +| MathVista | main | rule-match,llm-match | supported | | +| MedXpertQA | MM | exact-match,rule-match | supported | | +| Mementos | default | unsupported: keyword-set-f1 | REFUSED | requires keyword extraction + set-F1 grading (Mementos); use the benchmark's own evaluator (score_params: {'compare': 'keyword_set_f1'}) | +| MuirBench | main | exact-match,rule-match | supported | | +| NExT-QA | mc | exact-match,rule-match | supported | | +| NLVR2 | main | exact-match,rule-match | supported | | +| NaturalBench | default | exact-match,rule-match | supported | | +| NoCaps | main | unsupported: caption-metrics | REFUSED | requires corpus-level caption metrics (CIDEr/BLEU/ROUGE/BERTScore); use the benchmark's own evaluator (score_params: {'metrics': ['cider', 'bleu4', 'meteor', 'rouge_l']}) | +| OCR-VQA | main | exact-match,rule-match | supported | score_params: {'string_match': 'exact'} | +| OCRBench | main | exact-match,rule-match | supported | score_params: {'string_match': 'contains'} | +| OCRBench-v2 | default | unsupported: per-task-metric | REFUSED | requires the benchmark's own evaluator (per-row metric dispatch) (score_params: {'metric_field': 'type'}) | +| OK-VQA | main | vqa-accuracy | supported | | +| OlympiadBench | main | exact-match,rule-match | supported | | +| PMC-VQA | main | exact-match,rule-match | supported | | +| POPE | main | exact-match,rule-match | supported | | +| PathVQA | main | exact-match,rule-match,llm-judge | supported | | +| PhyX | mc | exact-match,rule-match,llm-judge | supported | | +| PuzzleVQA | default | exact-match,rule-match | supported | | +| Q-Bench-Plus | default | rule-match,llm-match | supported | | +| R-Bench-V | main | llm-judge | supported | score_params: {'rubric': 'binary', 'official_judge_model': 'gpt-4o'} | +| RealWorldQA | main | rule-match,llm-match | supported | | +| SAT | real | exact-match,rule-match | supported | | +| SAT | synthetic | exact-match,rule-match | supported | | +| SEED-Bench-2-Plus | main | rule-match,llm-match | supported | | +| SEED-Bench-H | main | rule-match,llm-match | supported | | +| SEEDBench | main | rule-match,llm-match | supported | | +| SLAKE | main | exact-match,rule-match,llm-judge | supported | | +| ST-VQA | main | anls | supported | score_params: {'threshold': 0.5} | +| SciFIBench | caption2figure | rule-match,llm-match | supported | | +| SciFIBench | figure2caption | rule-match,llm-match | supported | | +| SciVQA | default | unsupported: caption-metrics | REFUSED | requires corpus-level caption metrics (CIDEr/BLEU/ROUGE/BERTScore); use the benchmark's own evaluator (score_params: {'metrics': ['rouge1', 'rouge_l', 'bertscore']}) | +| ScienceQA | main | exact-match,rule-match | supported | | +| ScienceQA-IMG | default | exact-match,rule-match | supported | | +| ScreenQA | main | exact-match,rule-match | supported | score_params: {'string_match': 'exact'} | +| SeePhys | default | exact-match,rule-match,llm-judge | supported | | +| SpatialEval | vqa | exact-match,rule-match | supported | | +| SpatialEval | vtqa | exact-match,rule-match | supported | | +| TableVQA-Bench | fintabnetqa | exact-match,rule-match | supported | | +| TableVQA-Bench | vtabfact | exact-match,rule-match | supported | | +| TableVQA-Bench | vwtq | exact-match,rule-match | supported | | +| TableVQA-Bench | vwtq_syn | exact-match,rule-match | supported | | +| TallyQA | main | exact-match,rule-match | supported | score_params: {'string_match': 'exact'} | +| TempCompass | caption_matching | exact-match,rule-match,llm-judge | supported | | +| TempCompass | captioning | llm-judge | supported | score_params: {'rubric': 'binary', 'official_judge_model': 'gpt-3.5-turbo-1106'} | +| TempCompass | multi_choice | exact-match,rule-match,llm-judge | supported | | +| TempCompass | yes_no | exact-match,rule-match,llm-judge | supported | | +| TextCaps | main | unsupported: caption-metrics | REFUSED | requires corpus-level caption metrics (CIDEr/BLEU/ROUGE/BERTScore); use the benchmark's own evaluator (score_params: {'metrics': ['cider', 'bleu4', 'meteor', 'rouge_l']}) | +| TextVQA | main | vqa-accuracy | supported | | +| TheoremQA | image | exact-match,rule-match | supported | score_params: {'numeric_rel_tol': 0.04} | +| VLMsAreBlind | default | exact-match,rule-match | supported | | +| VQA-RAD | main | exact-match,rule-match,llm-judge | supported | | +| VQAv2 | main | vqa-accuracy | supported | | +| VSI-Bench | mc | exact-match,rule-match | supported | | +| VSR | zeroshot | rule-match,llm-match | supported | | +| VStarBench | main | rule-match,llm-match | supported | | +| VibeEval | main | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [1, 5], 'official_judge_model': 'reka-core-20240501'} | +| VisIT-Bench | default | llm-judge | supported | score_params: {'rubric': 'binary', 'official_judge_model': 'gpt-4', 'judge_inputs': ['instruction_conditioned_caption']} | +| VisOnlyQA | main | rule-match,llm-match | supported | | +| Visual7W | main | rule-match,llm-match | supported | | +| VisualMRC | main | unsupported: caption-metrics | REFUSED | requires corpus-level caption metrics (CIDEr/BLEU/ROUGE/BERTScore); use the benchmark's own evaluator (score_params: {'metrics': ['rouge_l', 'bleu4', 'meteor', 'cider']}) | +| VisualSimpleQA | main | llm-judge | supported | score_params: {'rubric': 'three_way', 'official_judge_model': 'SimpleQA-style ChatGPT grader (GPT-4o class)'} | +| VizWiz-Captions | main | unsupported: caption-metrics | REFUSED | requires corpus-level caption metrics (CIDEr/BLEU/ROUGE/BERTScore); use the benchmark's own evaluator (score_params: {'metrics': ['cider', 'bleu4', 'meteor', 'rouge_l', 'spice']}) | +| VizWiz-VQA | main | vqa-accuracy | supported | | +| WHOOPS | main | exact-match,rule-match,llm-judge | supported | | +| We-Math | main | exact-match,rule-match | supported | | +| WebSRC | main | exact-match,rule-match | supported | score_params: {'string_match': 'exact'} | +| WildVision-Bench | main | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [-2, 2], 'official_judge_model': 'gpt-4o'} | +| Winoground | main | exact-match,rule-match | supported | | +| WorldMedQA-V | default | rule-match,llm-match | supported | | +| WorldVQA | main | llm-judge | supported | score_params: {'rubric': 'three_way', 'official_judge_model': 'gpt-4o-1120'} | +| ZeroBench | default | exact-match,rule-match | supported | | +| ZoomBench | main | exact-match,rule-match,llm-judge | supported | | +| xGQA | main | exact-match,rule-match | supported | score_params: {'string_match': 'exact'} | + +## Refused protocols + +- `caption_metrics` (7 subsets) → corpus-level caption metrics (CIDEr/BLEU/ROUGE/BERTScore); use the benchmark's own evaluator. +- `per_task_metric` (2) → per-row metric dispatch; use the benchmark's own evaluator. +- `execution` (1) → official test-case execution (pass@1). +- `llm_extract` + `compare=keyword_set_f1` (1, Mementos) → keyword extraction + set-F1 grading. + +An explicit `--score_pipeline` can force approximate scoring for these; the +output is then stamped `official_protocol: false` with a protocol note. diff --git a/docs/en/SUPPORTED.md b/docs/en/SUPPORTED.md index ddbdedd6..bdf57db0 100644 --- a/docs/en/SUPPORTED.md +++ b/docs/en/SUPPORTED.md @@ -163,7 +163,7 @@ Currently available mm-eval HuggingFace datasets: | Dataset | Splits | Usage | |---------|--------|-------| | [MMBench-en](https://huggingface.co/datasets/mm-eval/MMBench-en) | dev, test | `mmeval_hf@mm-eval/MMBench-en` | -| [MMBench-en-V11](https://huggingface.co/datasets/mm-eval/MMBench-en-V11) | dev, test | `mmeval_hf@mm-eval/MMBench-en-V11` | +| [MMBench-V11](https://huggingface.co/datasets/mm-eval/MMBench-V11) | dev, test | `mmeval_hf@mm-eval/MMBench-V11 --subset en` (subsets: en, cn) | ### 3. VLMEvalKit Datasets (`evalkit@`) diff --git a/docs/en/USAGE.md b/docs/en/USAGE.md index eb1874d0..be0fc429 100644 --- a/docs/en/USAGE.md +++ b/docs/en/USAGE.md @@ -42,7 +42,8 @@ Run evaluation on a HuggingFace-hosted dataset with built-in prompt templates: ```bash python mmeval/run.py \ --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \ - --dataset mmeval_hf@mm-eval/MMBench-en-V11 \ + --dataset mmeval_hf@mm-eval/MMBench-V11 \ + --subset en \ --split test \ --out_dir work_dirs/mmbench_test \ --gpu_per_parallel 1 \ @@ -98,7 +99,8 @@ Distribute inference across multiple GPUs with automatic data sharding: ```bash python mmeval/run.py \ --model_name_or_path Qwen/Qwen2.5-VL-72B-Instruct \ - --dataset mmeval_hf@mm-eval/MMBench-en-V11 \ + --dataset mmeval_hf@mm-eval/MMBench-V11 \ + --subset en \ --split test \ --out_dir work_dirs/mmbench_72b \ --gpu_per_parallel 4 \ @@ -114,7 +116,8 @@ Run a quick deterministic smoke test on 20 randomly selected samples: ```bash python mmeval/run.py \ --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \ - --dataset mmeval_hf@mm-eval/MMBench-en-V11 \ + --dataset mmeval_hf@mm-eval/MMBench-V11 \ + --subset en \ --split test \ --out_dir work_dirs/mmbench_smoke \ --gpu_per_parallel 1 \ diff --git a/docs/en/VALIDATION.md b/docs/en/VALIDATION.md new file mode 100644 index 00000000..dbcf8725 --- /dev/null +++ b/docs/en/VALIDATION.md @@ -0,0 +1,54 @@ +# Validation: reproduced official results + +Has this framework been validated against official evaluation results? Yes. +Each check below runs a real model end to end — the framework's own inference +and scoring — using the dataset's **official scoring pipeline unchanged**, and +compares the result against an official reference: a model card, a benchmark's +technical report, the [OpenVLM leaderboard](https://huggingface.co/spaces/opencompass/open_vlm_leaderboard), +or the upstream [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) +evaluator run on the same machine. + +## How to read the table + +- **`eval_score`** is this framework's result; **`reference_score`** is the + official number; **`delta` = `eval_score` − `reference_score`** (percentage + points). +- Scores are each benchmark's official headline metric, as a percentage: + accuracy for the rule-based pipelines (ChartQA, OCRBench) and mean + ANLS / VQA-accuracy for the fractional metrics (DocVQA, InfographicVQA, + TextVQA). +- **`score_pipeline`** is the ordered list of scoring stages that ran (see + [SCORING.md](SCORING.md)); it is the dataset's declared official protocol. +- **`custom_template`** = `yes` where the model was prompted with the official + benchmark's own prompt template rather than the dataset's shipped default + (see notes below). + +**Agreement.** All ten checks agree with their official reference to within +**±1.0 percentage point** on the inference side, and the scoring side is +**exact**: on identical model predictions the framework's per-sample verdicts +match the official comparators with **zero mismatches across 33,300 samples** +(ANLS and VQA-accuracy agree to 1e-9). The small delta that remains is +inference-side variance — prompt construction, image handling, generation +settings — not scoring. + +## Reproductions + +| model | model_revision | dataset | split | score_pipeline | custom_template | samples | eval_score | reference_score | delta | reference_source | notes | +|---|---|---|---|---|---|---|---|---|---|---|---| +| Qwen2-VL-2B-Instruct | 895c3a49bc3f | ChartQA | test | exact-match, rule-match (numeric_rel_tol 0.05) | no | 2500 | 74.24 | 73.50 | +0.74 | Qwen2-VL-2B-Instruct model card (ChartQA_test) | | +| Qwen2-VL-2B-Instruct | 895c3a49bc3f | OCRBench | test | exact-match, rule-match (string_match contains) | no | 1000 | 79.10 | 79.70 | -0.60 | OpenVLM leaderboard (797/1000); model card 79.4 | | +| Qwen2-VL-2B-Instruct | 895c3a49bc3f | DocVQA | validation | anls (threshold 0.5) | no | 5349 | 88.48 | 89.24 | -0.76 | Upstream VLMEvalKit run, same machine | | +| Qwen2-VL-2B-Instruct | 895c3a49bc3f | InfographicVQA | validation | anls (threshold 0.5) | yes | 2801 | 64.31 | 64.20 | +0.11 | Upstream VLMEvalKit run, same machine | matched official inference config | +| Qwen2-VL-2B-Instruct | 895c3a49bc3f | TextVQA | validation | vqa-accuracy | no | 5000 | 79.16 | 79.70 | -0.54 | Qwen2-VL-2B-Instruct model card (TextVQA_val) | | +| Qwen2.5-VL-3B-Instruct | 66285546d2b8 | ChartQA | test | exact-match, rule-match (numeric_rel_tol 0.05) | no | 2500 | 83.64 | 84.00 | -0.36 | Qwen2.5-VL Technical Report Table 5 (arXiv:2502.13923) | | +| Qwen2.5-VL-3B-Instruct | 66285546d2b8 | OCRBench | test | exact-match, rule-match (string_match contains) | no | 1000 | 82.70 | 82.80 | -0.10 | OpenVLM leaderboard (828/1000); same-machine upstream 82.6 | matched official inference config | +| Qwen2.5-VL-3B-Instruct | 66285546d2b8 | DocVQA | validation | anls (threshold 0.5) | no | 5349 | 92.46 | 93.01 | -0.55 | Upstream VLMEvalKit run, same machine | | +| Qwen2.5-VL-3B-Instruct | 66285546d2b8 | InfographicVQA | validation | anls (threshold 0.5) | no | 2801 | 75.15 | 76.03 | -0.88 | Upstream VLMEvalKit run, same machine | matched official inference config | +| Qwen2.5-VL-3B-Instruct | 66285546d2b8 | TextVQA | validation | vqa-accuracy | no | 5000 | 79.82 | 79.30 | +0.52 | Qwen2.5-VL Technical Report Table 5 + model card | | + +**Matched official inference config.** Three checks needed the official +inference configuration — image pixel window, prompt construction, and +generation length — reproduced before the model's outputs agreed with the +reference to within tolerance. Where the official prompt template itself was +reproduced, `custom_template` is `yes` (here, 2B InfographicVQA). This is +inference-side alignment only; no scoring behaviour changes. diff --git a/mmeval/data/base.py b/mmeval/data/base.py index faf627aa..2a4bbd85 100644 --- a/mmeval/data/base.py +++ b/mmeval/data/base.py @@ -9,7 +9,6 @@ from jinja2 import Environment - class BaseDataset(ABC): """Dataset base class for loading and processing datasets. @@ -209,7 +208,7 @@ def build_prompt(self, prompt_template: str, sample: Dict[str, Any]) -> str: Rendered template string """ env = Environment() - env.globals.update({'zip': zip, 'enumerate': enumerate, 'len': len, 'range': range, 'list': list, + env.globals.update({'zip': zip, 'enumerate': enumerate, 'len': len, 'range': range, 'list': list, 'dict': dict, 'str': str, 'int': int, 'float': float, 'bool': bool, 'sum': sum, 'max': max, 'min': min}) template = env.from_string(prompt_template) return template.render(**sample) diff --git a/mmeval/data/mmeval_hf.py b/mmeval/data/mmeval_hf.py index cfdd74b9..74e76ae2 100644 --- a/mmeval/data/mmeval_hf.py +++ b/mmeval/data/mmeval_hf.py @@ -7,7 +7,15 @@ class MMEvalHFDataset(BaseDataset): - """Dataset loader for HuggingFace datasets in mm-eval format.""" + """Dataset loader for HuggingFace datasets in mm-eval format. + + Reads the subset manifest from the dataset repo's metadata.json and injects + the scoring-relevant metadata as the flat top-level `dataset_meta` sample + field. Rows are expected in the flat layout (grading fields as top-level + columns; messages[0] carries render inputs only) — non-conforming rows are + passed through unchanged and surface at scoring time via the scorer's + explicit layout rejection. + """ def __init__(self, args): self.dataset_name = args.dataset.split("@", 1)[1] @@ -40,9 +48,50 @@ def _load_raw_data(self, args) -> tuple: raise ValueError( f"{self.dataset_name}: subset {self.subset!r} not in {list(subsets)}" ) - dataset_template = subsets[self.subset].get("prompt_template") + subset_block = subsets[self.subset] + dataset_template = subset_block.get("prompt_template") + + # Scoring data-flow contract: inject the subset's scoring-relevant + # metadata into every sample as the flat top-level `dataset_meta` + # field, so it lands in result.json and the scorer can resolve the + # official grading protocol without access to the HF manifest. CLI + # flags on the scorer always override these values. + if "score_type" in subset_block: + raise ValueError( + f"{self.dataset_name} subset {self.subset!r}: metadata.json " + f"contains an unsupported `score_type` key. Declare the protocol " + f"with the `score_pipeline` schema instead (docs/en/SCORING.md, " + f"'Dataset metadata contract'), then re-run inference." + ) + # Dataset identity always travels in result.json, even when no scoring + # protocol is declared, so the scorer's resume fingerprint can tell + # apart caches produced for a different split/subset/dataset scored + # into the same out_dir (eval-ids 0..N would otherwise collide). + meta_block = { + "dataset_name": self.dataset_name, + "subset": self.subset, + "split": self.split, + } + for key in ("task_type", "score_params"): + value = subset_block.get(key) + if value: + meta_block[key] = value + if "score_pipeline" in subset_block: + # Declared protocol passes through verbatim; [] (explicitly no + # official protocol) is a meaningful value, so test presence. + meta_block["score_pipeline"] = subset_block["score_pipeline"] + note = ((subset_block.get("score_protocol") or {}).get("note") or "").strip() + if note: + meta_block["score_note"] = note + self._dataset_meta = meta_block - ds = load_dataset(self.dataset_name, name="default", split=self.split) + # Prefer multi-config layout: HF config name == mm-eval subset name. + # Fall back to the single-`default` config layout, where one config + # holds all subsets' splits (named `_`). + try: + ds = load_dataset(self.dataset_name, name=self.subset, split=self.split) + except (ValueError, FileNotFoundError): + ds = load_dataset(self.dataset_name, name="default", split=self.split) return ds, dataset_template def convert_circular(self, **kwargs) -> any: @@ -52,14 +101,12 @@ def convert_circular(self, **kwargs) -> any: def _process_sample(self, idx: int): sample = dict(self._raw_dataset[idx]) - # Add eval-id if not present if "eval-id" not in sample: sample["eval-id"] = idx messages = sample["messages"] message_list = json.loads(messages) if isinstance(messages, str) else messages - # Get media from sample level (HF datasets may store as single image or list) media = sample.get("media") if media is None: media_list = [] @@ -68,10 +115,10 @@ def _process_sample(self, idx: int): else: media_list = [media] - # Ensure sample["media"] is always a list sample["media"] = media_list - - # Process all messages with sample-level media indexed by placeholder order sample["messages"] = self._process_messages(message_list, media_list) + if self._dataset_meta: + sample.setdefault("dataset_meta", dict(self._dataset_meta)) + return sample diff --git a/mmeval/data/tsv.py b/mmeval/data/tsv.py index ec88c41c..bada9868 100644 --- a/mmeval/data/tsv.py +++ b/mmeval/data/tsv.py @@ -58,7 +58,8 @@ def __init__(self, args): self.dataset_dir = os.getenv('DATASET_DIR') or "./datasets" self.dataset_url = None self.resize = args.resize - + self.split = getattr(args, "split", None) + if args.dataset.startswith("http"): self.file_name = args.dataset.split('/')[-1].replace('.tsv', '') self.dataset_url = args.dataset @@ -67,6 +68,16 @@ def __init__(self, args): else: self.file_name = args.dataset + # Dataset identity injected into every sample's dataset_meta (see + # mmeval_hf.py) so the scorer's resume fingerprint discards a cache + # produced for a different TSV scored into the same out_dir. A TSV is a + # single flat file, so it has no subset. + self._dataset_meta = { + "dataset_name": self.file_name, + "subset": None, + "split": self.split, + } + super().__init__(args) def _load_raw_data(self, args) -> tuple: @@ -104,7 +115,6 @@ def _extract_media_paths(self, sample: Dict[str, Any]) -> list: """ media_paths = [] - # Priority 1: Check for image_url if 'image_url' in sample and pd.notna(sample['image_url']): image_url = sample['image_url'] if image_url.startswith('[') and image_url.endswith(']'): @@ -112,7 +122,6 @@ def _extract_media_paths(self, sample: Dict[str, Any]) -> list: else: media_paths = [image_url] - # Priority 2: Check for base64 image data elif 'image' in sample and pd.notna(sample['image']): image = sample['image'] if image.startswith('[') and image.endswith(']'): @@ -145,7 +154,6 @@ def _process_sample(self, index: int) -> Dict[str, Any]: # If placeholder/media counts mismatch, all placeholders are moved to prefix. question = normalize_question_with_media(question, len(media_list)) - # Build options dict and choices list options = { choice_index: sample[choice_index] for choice_index in string.ascii_uppercase if choice_index in sample and not pd.isna(sample[choice_index]) @@ -156,7 +164,7 @@ def _process_sample(self, index: int) -> Dict[str, Any]: if options: message["options"] = options message["choices"] = list(options.keys()) - + hint = sample.get("hint", None) if hint and pd.notna(hint): message["hint"] = hint @@ -168,7 +176,8 @@ def _process_sample(self, index: int) -> Dict[str, Any]: sample.pop("image", None) sample.pop("image_url", None) sample["media"] = media_list - + sample.setdefault("dataset_meta", dict(self._dataset_meta)) + return sample def __repr__(self): diff --git a/mmeval/infer/qwen3_vl.py b/mmeval/infer/qwen3_vl.py index 0d3ddecb..d73e40c9 100644 --- a/mmeval/infer/qwen3_vl.py +++ b/mmeval/infer/qwen3_vl.py @@ -1,3 +1,4 @@ +import os import re import copy @@ -14,8 +15,16 @@ class TaskRunner(Task): def __init__(self, args): self.args = args self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") - self.dtype = getattr(args, "dtype") or "auto" - self.default_model_kwargs = {"device_map": "auto"} + # Qwen3-VL is a bf16-native model; "auto" mis-resolves to fp32 on this merged ckpt, + # which both doubles attention memory AND breaks flash_attention_2 (needs fp16/bf16). + # Default to bf16 unless the user explicitly passes --dtype. + _dt = getattr(args, "dtype", None) + self.dtype = _dt if _dt and _dt != "auto" else torch.bfloat16 + # flash_attention_2 is EXACT attention (same math as sdpa, just memory-efficient): + # avoids materializing the O(N^2) attention matrix that OOMs on no-cap full-res images. + # Overridable via --attn_implementation; default on since flash-attn is installed. + _attn = getattr(args, "attn_implementation", None) or os.environ.get("MMEVAL_ATTN", "flash_attention_2") + self.default_model_kwargs = {"device_map": "auto", "attn_implementation": _attn} self.default_gen_kwargs = {"max_new_tokens": 128} self.model_kwargs = parse_model_kwargs(args, self.default_model_kwargs) self.gen_kwargs = parse_gen_kwargs(args, self.default_gen_kwargs) @@ -53,10 +62,13 @@ def parse_input(self, message): { "type": "image", "image": media, - "min_pixels": 4 * 32 * 32, - "max_pixels": 256 * 32 * 32, + # Aligned with VLMEvalKit: no hardcoded min/max_pixels, so the + # processor's defaults apply (shortest_edge=65536, + # longest_edge=16777216). The previous cap of max_pixels=256*32*32 + # (262144 ≈ 512x512) downscaled ~17% of MMStar images and hurt + # fine-perception accuracy. } - ) + ) elif chunk == constants.video: media = media_list[media_idx] media_idx += 1 @@ -94,9 +106,9 @@ def _generate_response(self, inputs): def run_sample(self, sample: dict): ori_sample = copy.deepcopy(sample) message = sample["messages"][0] - + user_message = self.parse_input(message) - + text = self.processor.apply_chat_template( user_message, tokenize=False, add_generation_prompt=True ) @@ -112,12 +124,12 @@ def run_sample(self, sample: dict): video_metadatas = None inputs = self.processor( - text=text, - images=images, - videos=videos, + text=text, + images=images, + videos=videos, video_metadata=video_metadatas, - return_tensors="pt", - do_resize=False, + return_tensors="pt", + do_resize=False, **video_kwargs ) inputs = inputs.to(self.model.device) diff --git a/mmeval/infer/qwenvl2.py b/mmeval/infer/qwenvl2.py index 824b9fe0..e8a0ed84 100644 --- a/mmeval/infer/qwenvl2.py +++ b/mmeval/infer/qwenvl2.py @@ -27,9 +27,12 @@ def __init__(self, args): def load_model(self, args): self.model = Qwen2VLForConditionalGeneration.from_pretrained(args.model_name_or_path, torch_dtype=self.dtype, **self.model_kwargs) self.tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path) - min_pixels = 256 * 28 * 28 - max_pixels = 1280 * 28 * 28 - self.processor = AutoProcessor.from_pretrained(args.model_name_or_path, min_pixels=min_pixels, max_pixels=max_pixels) + processor_kwargs = {"min_pixels": 256 * 28 * 28, "max_pixels": 1280 * 28 * 28} + if getattr(args, "min_pixels", None) is not None: + processor_kwargs["min_pixels"] = args.min_pixels + if getattr(args, "max_pixels", None) is not None: + processor_kwargs["max_pixels"] = args.max_pixels + self.processor = AutoProcessor.from_pretrained(args.model_name_or_path, **processor_kwargs) def parse_input(self, message:dict): question = message["prompt"] diff --git a/mmeval/infer/qwenvl2d5.py b/mmeval/infer/qwenvl2d5.py index 0c6ee5ff..b337ea35 100644 --- a/mmeval/infer/qwenvl2d5.py +++ b/mmeval/infer/qwenvl2d5.py @@ -32,9 +32,16 @@ def __init__(self, args): def load_model(self, args): self.model = Qwen2_5_VLForConditionalGeneration.from_pretrained(args.model_name_or_path, torch_dtype=self.dtype, **self.model_kwargs) self.tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path) - min_pixels = 256 * 28 * 28 - max_pixels = 1280 * 28 * 28 - self.processor = AutoProcessor.from_pretrained(args.model_name_or_path, min_pixels=min_pixels, max_pixels=max_pixels) + # Aligned with VLMEvalKit: do NOT pass min/max_pixels by default, so the + # processor's defaults apply (min_pixels=3136, max_pixels=12845056 ≈ 12.8M). + # The previous cap of max_pixels=1280*28*28 (~1MP) was ~12.8x lower and + # downscaled images. --min_pixels/--max_pixels override per run. + processor_kwargs = {} + if getattr(args, "min_pixels", None) is not None: + processor_kwargs["min_pixels"] = args.min_pixels + if getattr(args, "max_pixels", None) is not None: + processor_kwargs["max_pixels"] = args.max_pixels + self.processor = AutoProcessor.from_pretrained(args.model_name_or_path, **processor_kwargs) def parse_input(self, message): question = message["prompt"] @@ -145,4 +152,4 @@ def run_sample(self, sample: dict): if __name__ == "__main__": args = parse_args() model_evaluator = TaskRunner(args) - model_evaluator.inference_dataset() + model_evaluator.inference_dataset() \ No newline at end of file diff --git a/mmeval/run.py b/mmeval/run.py index 5eb809b8..0c211219 100644 --- a/mmeval/run.py +++ b/mmeval/run.py @@ -19,7 +19,7 @@ def get_series(model_name: str): args = parse_args() model_name_or_path = args.model_name_or_path - series = get_series(model_name_or_path.split("/")[-1]) + series = args.model_series or get_series(model_name_or_path.split("/")[-1]) infer_file = series_infer_env_mapping[series]["infer_file"] infer_env = series_infer_env_mapping[series]["env"] diff --git a/mmeval/score.py b/mmeval/score.py new file mode 100644 index 00000000..f2e8700d --- /dev/null +++ b/mmeval/score.py @@ -0,0 +1,31 @@ +from mmeval.scoring.pipeline import discover_result_files, score_result_file +from mmeval.utils.argparser import parse_args + + +def run_score(args): + files = discover_result_files(out_dir=args.score_out_dir, pattern=args.score_result_glob) + if not files: + raise RuntimeError(f"No result files found under {args.score_out_dir} with pattern {args.score_result_glob}") + + done = [] + failed = [] + for result_file in files: + try: + done.append(score_result_file(result_file, args)) + except Exception as exc: + failed.append({"result_file": result_file, "error": str(exc)}) + + for item in done: + print( + f"✅ [score] {item['result_file']} -> {item['score_file']} " + f"(acc={item['summary']['accuracy']:.4f}, resumed={item.get('resumed_count', 0)})" + ) + if failed: + for item in failed: + print(f"❌ [score] {item['result_file']}: {item['error']}") + raise SystemExit(1) + + +if __name__ == "__main__": + args = parse_args() + run_score(args) diff --git a/mmeval/scoring/__init__.py b/mmeval/scoring/__init__.py new file mode 100644 index 00000000..0a22a81b --- /dev/null +++ b/mmeval/scoring/__init__.py @@ -0,0 +1,6 @@ +from mmeval.scoring.pipeline import score_result_file, discover_result_files + +__all__ = [ + "score_result_file", + "discover_result_files", +] diff --git a/mmeval/scoring/extraction.py b/mmeval/scoring/extraction.py new file mode 100644 index 00000000..8e36ead1 --- /dev/null +++ b/mmeval/scoring/extraction.py @@ -0,0 +1,607 @@ +import re +import unicodedata +from typing import Any, Dict, List, Optional, Tuple + + +OPTION_LETTERS = list("ABCDEFGHIJKLMNOPQRSTUVWXYZ") + +# Question-type taxonomy. Every type is justified by at least one mm-eval org +# dataset (per-subset audit: docs/en/SCORING_COVERAGE.md): mcq (ScienceQA-IMG, MMStar, ...; +# multi-letter answers like LogicVista's "A, C" are mcq with |letters|>1), +# yes_no (VSR, GQA-Spatial verify rows), numeric (CharXiv, VLMsAreBlind, SeePhys), +# open (MM-Vet-v2, CharXiv descriptive). +QUESTION_TYPES = ("mcq", "yes_no", "numeric", "open") + + + +def _to_text(value: Any) -> str: + if isinstance(value, list): + if not value: + return "" + value = value[0] + if value is None: + return "" + return str(value) + + +def _extract_boxed_content(text: str) -> str: + marker = "\\boxed{" + start = text.find(marker) + if start < 0: + return text + i = start + len(marker) + depth = 1 + content = [] + while i < len(text): + ch = text[i] + if ch == "{": + depth += 1 + elif ch == "}": + depth -= 1 + if depth == 0: + return "".join(content).strip() + content.append(ch) + i += 1 + return text + + +# normalize_text machinery, compiled once (hot path: called several times per sample). +_COT_TAG_RE = re.compile(r"", re.IGNORECASE) +_LATEX_CMD_RE = re.compile(r"\\(?:left|right|displaystyle|mathrm|textbf|textit|text)\b") +_LATEX_SPACING_RE = re.compile(r"\\(?:,|;|!|\:)") +_WHITESPACE_RE = re.compile(r"\s+") +# Inference artifacts and unicode operator variants folded via one translate/replace pass. +_ARTIFACT_REPLACEMENTS = ( + ("```", " "), ("<|end_of_sentence|>", ""), ("<|end▁of▁sentence|>", ""), + ("", ""), ("", ""), ("", ""), ("Falcon: ", ""), +) +_OPERATOR_TRANSLATION = str.maketrans({"−": "-", "–": "-", "—": "-", "×": "*", "·": "*", "÷": "/"}) + + +def normalize_text( + value: Any, + lower: bool = False, + extract_boxed: bool = True, + strip_latex_commands: bool = True, +) -> str: + """Syntax-level, semantics-preserving normalization of a model answer: + NFKC folding, \boxed{} content extraction, / tag removal, + inference-artifact stripping, light LaTeX cleanup, unicode operator folding, + whitespace collapse (and optional lowercasing).""" + text = _to_text(value) + + text = unicodedata.normalize("NFKC", text) + if extract_boxed: + text = _extract_boxed_content(text) + + text = _COT_TAG_RE.sub(" ", text) + for old_str, new_str in _ARTIFACT_REPLACEMENTS: + text = text.replace(old_str, new_str) + + if strip_latex_commands: + text = _LATEX_CMD_RE.sub(" ", text) + text = _LATEX_SPACING_RE.sub(" ", text) + text = text.replace("\\%", "%") + + text = text.translate(_OPERATOR_TRANSLATION) + + text = _WHITESPACE_RE.sub(" ", text.strip()) + if lower: + text = text.lower() + return text + + +def normalize_for_exact(value: Any) -> str: + return normalize_text(value, lower=True).strip() + + +_WRAPPING_QUOTES_RE = re.compile(r"^[\"'`]+|[\"'`]+$") +_TRAILING_SENTENCE_PUNCT_RE = re.compile(r"[\.。!！?？]+$") + + +def normalize_open_answer(value: Any) -> str: + """normalize_for_exact plus open-answer specifics: strip wrapping quotes and + trailing sentence punctuation ("... Bridge." vs "bridge").""" + text = normalize_for_exact(value) + text = _WRAPPING_QUOTES_RE.sub("", text) + text = _TRAILING_SENTENCE_PUNCT_RE.sub("", text) + return _WHITESPACE_RE.sub(" ", text).strip() + + +# Render inputs — the fields a dataset's prompt template can reference; they are +# the ONLY fields that live inside messages[0] (dataset-standard boundary rule). +_RENDER_INPUT_FIELDS = frozenset({"question", "options", "choices", "hint", "prompt"}) + + +def get_field(sample: Dict[str, Any], field: str, default: Any = None) -> Any: + """Read a field from the ONE canonical result.json layout: the prediction + at messages[-1].response; render inputs (question/options/choices/hint/ + prompt) inside messages[0]; every grading field (answer, question_type, + reference_response, and any --score_gt_field) at the sample TOP level. + All backends emit this layout; files with grading fields embedded in + messages[0] are rejected by the scorer.""" + if not field: + return default + if field == "messages[-1].response": + messages = sample.get("messages") or [] + last = messages[-1] if messages else None + return last.get("response", default) if isinstance(last, dict) else default + if field in _RENDER_INPUT_FIELDS: + messages = sample.get("messages") or [] + first = messages[0] if messages else None + return first.get(field, default) if isinstance(first, dict) else default + return sample.get(field, default) + + +def get_question(sample: Dict[str, Any]) -> str: + q = get_field(sample, "question") + return "" if q is None else str(q) + + +_PAREN_OPTION_RE = re.compile(r"\(([A-H])\)") +_LABELLED_OPTION_RE = re.compile(r"(?:^|[\n\s])([A-H])[\.\):]") + + +def _option_letters_from_question(question: str) -> List[str]: + """Parse option labels embedded in the question text. Handles both the + "A: ... B: ..." (MMStar) and "(A) ... (B) ..." (BLINK) inline formats. Returns + only the leading contiguous run from A (A,B,C,D,...) to avoid stray letters.""" + if not isinstance(question, str): + return [] + found = set() + # "(A)" / "(A) " style and "A." / "A:" / "A)" at a token boundary. + for m in _PAREN_OPTION_RE.findall(question): + found.add(m.upper()) + for m in _LABELLED_OPTION_RE.findall(question): + found.add(m.upper()) + letters = [] + for ch in OPTION_LETTERS[:8]: + if ch in found: + letters.append(ch) + else: + break # keep only the contiguous A,B,C,... prefix + return letters + + +def _choices_to_letters(choices: List[Any]) -> List[str]: + """A `choices` list is either option letters (['A','B',...]) or option values + (['0','3',...]). Letters are used directly; values map positionally to A,B,C,...""" + items = [str(c).strip() for c in choices] + if items and all(len(c) == 1 and c.upper() in OPTION_LETTERS for c in items): + return [c.upper() for c in items] + return OPTION_LETTERS[: len(items)] + + +_LETTER_PREFIX_RE = re.compile(r"\s*([A-Z])\s*[\.\):]\s*(.+)$") + + +def _strip_letter_prefix(text: str, letter: str) -> str: + """VSI-Bench stores choices as letter-prefixed strings ('A. back-right'); + strip the positional letter prefix so option-text matching sees the bare text.""" + m = _LETTER_PREFIX_RE.match(text) + return m.group(2).strip() if m and m.group(1) == letter else text + + +def _flat_letter_options(src: Dict[str, Any]) -> Dict[str, str]: + """Options stored as per-letter keys with non-empty values (GMAI-MMBench, + WorldMedQA-V, HRBench store msg['A']='...' with empty options/choices; an + empty value, e.g. GMAI's E='', means the option does not exist).""" + out = {} + for letter in OPTION_LETTERS[:8]: + value = src.get(letter) + if value is None or str(value).strip() == "": + break # contiguous prefix only, like the inline-question parser + out[letter] = str(value) + return out + + +def parsed_options(sample: Dict[str, Any]) -> Optional[List[str]]: + """Candidate option letters from an explicit option source, or None when the + sample carries none (options-in-image datasets). All option sources — the + `options` dict, the `choices` list, inline-question text and flat per-letter + keys — are render inputs and therefore live inside messages[0] + (dataset-standard boundary rule); only grading fields moved to the sample + top level.""" + messages = sample.get("messages") or [] + msg = messages[0] if messages and isinstance(messages[0], dict) else None + if not isinstance(msg, dict): + return None + + options = msg.get("options") + if isinstance(options, dict) and options: + return [str(k).strip().upper() for k in options.keys()] + choices = msg.get("choices") + if isinstance(choices, list) and choices: + return _choices_to_letters(choices) + + # Flat per-letter keys inside the user message (GMAI-MMBench msg["A"].."E", + # WorldMedQA-V, HRBench) — empty values excluded. + flat = _flat_letter_options(msg) + if flat: + return list(flat.keys()) + + # Inline-options datasets (MMStar/BLINK keep options in the question text and + # ship empty `options`/`choices`): derive the real letters from the question so + # we don't fall back to A-F and mis-count stray letters (e.g. "F = ma"). + letters = _option_letters_from_question(msg.get("question") or msg.get("prompt") or "") + if letters: + return letters + return None + + +def extract_options(sample: Dict[str, Any]) -> List[str]: + """Candidate option letters: the sample's parsed option source, else A-F + (some MCQ datasets, e.g. LogicVista, keep the choices in the image and don't + list them in the question text). Whether the item is actually MCQ vs open is + decided by infer_question_type from the ground-truth shape, not by this + default.""" + return parsed_options(sample) or OPTION_LETTERS[:6] + + +def extract_option_texts(sample: Dict[str, Any], letters: List[str]) -> Dict[str, str]: + """Map option letter -> option text, from whichever source the sample uses + (all render inputs inside messages[0] — see extract_options).""" + messages = sample.get("messages") or [] + msg = messages[0] if messages and isinstance(messages[0], dict) else None + if isinstance(msg, dict): + msg_options = msg.get("options") + if isinstance(msg_options, dict) and msg_options: + return {str(k).strip().upper(): str(v) for k, v in msg_options.items()} + # BLINK-style: `choices` holds the option *values* positionally (A,B,C,...). + msg_choices = msg.get("choices") + if isinstance(msg_choices, list) and msg_choices: + vals = [str(c) for c in msg_choices] + if not all(len(v.strip()) == 1 and v.strip().upper() in OPTION_LETTERS for v in vals): + # VSI-Bench stores letter-prefixed values ("A. back-right"); strip + # the positional letter so text matching sees the bare option. + return { + OPTION_LETTERS[i]: _strip_letter_prefix(vals[i], OPTION_LETTERS[i]) + for i in range(min(len(vals), len(OPTION_LETTERS))) + } + + # Flat per-letter option keys inside the user message (HRBench msg["A"]="27B", + # GMAI-MMBench, WorldMedQA-V) with empty options/choices. + option_texts = {} + for letter in letters: + if isinstance(msg, dict) and letter in msg: + option_texts[letter] = str(msg[letter]) + return option_texts + + +_GT_LABEL_STRIP_RE = re.compile(r"[()\[\].:;\s\*]") +_LETTER_SEPARATOR_RE = re.compile(r"[,;/&\s]+|\band\b", re.IGNORECASE) + + +def parse_gt_letters(gt: Any) -> Optional[Tuple[str, ...]]: + """MCQ ground truth -> sorted letter tuple. A single bare letter ("B", "(D)") + gives a 1-tuple; multi-letter answers must be SEPARATED letters like "A, C" / + "A and C" (LogicVista format), >=2 unique letters within A-H. Compact "AC" is + deliberately NOT accepted — as a *type signal* it would misread open answers + that happen to be A-H-only words ("BED").""" + g = _GT_LABEL_STRIP_RE.sub("", str(gt)).upper() + if len(g) == 1 and g in OPTION_LETTERS: + return (g,) + tokens = [t for t in _LETTER_SEPARATOR_RE.split(str(gt).strip()) if t] + if len(tokens) < 2: + return None + letters = [] + for t in tokens: + if len(t) == 1 and t.upper() in OPTION_LETTERS[:8]: + letters.append(t.upper()) + else: + return None + if len(set(letters)) != len(letters): + return None + return tuple(sorted(letters)) + + +def parse_number(value: Any) -> Optional[float]: + """The whole (normalized) answer is a number, else None. A trailing + percent sign reads as /100 — the official numeric semantics (ChartQA + relaxed_correctness `_to_float`, lmms-eval chartqa/utils.py, taken from + Qwen-VL evaluate_vqa.py:L113): "24%" is 0.24, NOT 24, so a model + answering "24%" against gt "24" does not match.""" + text = normalize_open_answer(value).replace(",", "") + percent = text.endswith("%") + if percent: + text = text.rstrip("%").strip() + try: + number = float(text) + except ValueError: + return None + return number / 100.0 if percent else number + + +_WORD_RE = re.compile(r"[a-z]+") + + +def extract_yes_no(value: Any) -> Optional[str]: + """VLMEvalKit YOrN_Extraction parity (vlmeval/dataset/utils/yorn.py:254-261): + word-level; "yes" without "no" -> yes, "no" without "yes" -> no, both or + neither -> None (upstream 'Unknown').""" + words = set(_WORD_RE.findall(normalize_text(value, lower=True))) + if "yes" in words and "no" not in words: + return "yes" + if "no" in words and "yes" not in words: + return "no" + return None + + +def infer_question_type(sample: Dict[str, Any], question_type: str = "auto", gt: Any = None) -> str: + """Resolve the sample's question type. Order: forced CLI value -> the sample's + standardized `question_type` field (recognized vocabulary only — mm-eval org + datasets like VSI-Bench reuse that field for subtask names, which fall + through) -> gt shape -> options presence.""" + forced = (question_type or "auto").strip().lower() + if forced in QUESTION_TYPES: + return forced + + # Per-sample field contract: exactly the four grading values. A full-org + # sweep of every mm-eval dataset/split (577k+ rows, datasets-server + # statistics API) found ONLY these; dataset-level task_type stays in + # metadata, source-benchmark labels stay in `source_question_type`. + field_value = str(get_field(sample, "question_type", "") or "").strip().lower() + if field_value in QUESTION_TYPES: + return field_value + + # The ground-truth shape is the most reliable signal and correctly splits + # mixed benchmarks (RealWorldQA interleaves letter, yes/no and numeric gts; + # GQA-Spatial mixes verify and query rows; LogicVista mixes single- and + # multi-letter answers — both are mcq). + if gt is not None: + norm = normalize_open_answer(gt) + if norm in {"yes", "no"}: + return "yes_no" + if parse_gt_letters(gt) is not None: + return "mcq" + if parse_number(gt) is not None: + return "numeric" + if _GT_LABEL_STRIP_RE.sub("", str(gt)) != "": + return "open" + + # No usable gt signal (missing/blank gt — the sample is marked invalid before + # any stage runs): default to mcq. NOTE extract_options never returns empty + # (A-F fallback), so an "options present?" check here would be dead logic. + return "mcq" + + +_STRICT_CUE_RES = ( + re.compile(r"^(?:answer|option|choice)\s*[:\-]?\s*(?-i:([A-Z]))(?:[\s\.\),:;]|$)", re.IGNORECASE), + re.compile(r"(?:final answer|the answer is|answer is|choice is|option is)\s*[:\-]?\s*(?-i:([A-Z]))(?:[\s\.\),:;]|$)", re.IGNORECASE), +) + + +def extract_option_strict(text: Any, candidates: List[str]) -> Optional[str]: + # Prefer the content of the LAST ... tag if the model emitted + # the reasoning-style "...X" format. Otherwise + # the buried letter gets diluted by surrounding reasoning text and the regexes + # below — which anchor on the start of the string — miss it. + raw_text = _to_text(text) + ans = _ANSWER_TAG_RE.findall(raw_text) + clean = normalize_text(ans[-1]) if ans else normalize_text(text) + if not clean: + return None + + stripped = clean.strip() + uppercase = stripped.upper() + # The whole answer is a single option letter (any case) — unambiguous. + if uppercase in candidates: + return uppercase + # Labelled answer-first form "B. ..." / "B) ..." / "B: ...". Punctuation (not a + # space) and an uppercase letter are required, so "A teddy bear" is NOT option A. + if len(stripped) >= 2 and uppercase[0] in candidates and stripped[0].isupper() and stripped[1] in {".", ")", ":"}: + return uppercase[0] + + # Cue phrases are case-insensitive but the letter is matched case-SENSITIVELY + # ((?-i:...)), so "the answer is a teddy bear" never reads as option A. + for pattern in _STRICT_CUE_RES: + match = pattern.search(stripped) + if match: + candidate = match.group(1) + if candidate in candidates: + return candidate + return None + + +# Punctuation/markdown replaced by whitespace so option letters become standalone +# tokens — VLMEvalKit's character set plus full-width CJK punctuation (without it +# "最终答案：A" never splits and the trailing letter is lost). +_CHOICE_PUNCT_TRANSLATION = str.maketrans({ + ch: " " for ch in ".()[],:;!*#{}" + "：；，。、！？「」『』（）【】《》" +}) + +# Explicit "final answer" style cues (English + CJK markers 答案/最终答案/选). +# Cues resolve answers even when several option letters appear in the reasoning +# (where pure token-position logic gives up and upstream defers to its GPT +# extractor). Cue phrases are case-insensitive but the option letter is matched +# case-SENSITIVELY via (?-i:...) so a lowercase article ("the answer is a teddy +# bear") is never mistaken for option "A". +_ANSWER_CUE = re.compile( + r"(?:final answer|the answer is|answer is|answer\s*:|option is|" + r"(?:the\s+)?correct\s+(?:option|answer|choice)(?:\s+is)?|" # "correct option: (C)" + r"最终答案|答案|选)\s*[:\-：]?\s*" + # The letter may be wrapped by any mix of '(' / '*' / full-width parens in any + # order: "A", "(A)", "**A**", "**(A)**", "(A) text", "**(A) 0.0"... + r"[\(（\*]{0,4}(?-i:([A-Z]))[\)）\*]{0,4}(?:[\s\.\),:;、，。]|$)", + re.IGNORECASE, +) + + +def _strip_choice_punct(text: str) -> str: + return text.translate(_CHOICE_PUNCT_TRANSLATION) + + +def can_infer_option(answer: str, choices: List[str]) -> Optional[str]: + """VLMEvalKit-style: strip punctuation -> tokens -> the unique option letter + appearing near the end of the answer. Case-sensitive like upstream: pass the + ORIGINAL-case answer, or lowercase articles ("a cute cat") count as option A.""" + answer_mod = _strip_choice_punct(answer) + splits = [x.strip() for x in answer_mod.split() if x.strip()] + present = [c for c in choices if c in splits] + if len(present) == 1: + ch = present[0] + # Near the end (the model's final pick), mirroring VLMEvalKit's last-5 + # window; additionally a leading letter in a short (<=5 token) answer + # such as "B is the correct answer" is accepted — upstream defers those + # to its GPT extractor, which resolves them the same way. + if splits.index(ch) > len(splits) - 5 or (splits.count(ch) == 1 and len(splits) <= 5): + return ch + return None + + +def can_infer_text(answer: str, option_texts: Dict[str, str]) -> Optional[str]: + """VLMEvalKit-style: the answer restates exactly one option's text. Upstream's + length guard applies: a long CoT that merely mentions one option somewhere is + not an answer restatement.""" + if not option_texts: + return None + low = answer.lower() + if len(low) > 2 * sum(len(str(v)) for v in option_texts.values()): + return None + cands = [k for k, v in option_texts.items() if v and str(v).strip().lower() in low] + if len(cands) == 1: + return cands[0] + return None + + +# extract_option_robust machinery, compiled once. Rule order matters and is +# documented on each step below. +_ANSWER_TAG_RE = re.compile(r"\s*(.*?)\s*", re.DOTALL | re.IGNORECASE) +_WRAPPING_BRACKETS_RE = re.compile(r"^[\(\[]|[\)\]\.]$") +_BOXED_LETTER_RE = re.compile(r"\\boxed\{\s*(?:\\[a-zA-Z]+\s*\{\s*)?[\(\*]*\s*([A-Za-z])") +_CUE_TAIL_RE = re.compile( + r"(?:final answer|the answer is|answer is|answer\s*[:\-：]|option is|choice is)" + r"\s*[\(（\*]{0,4}([A-Za-z])[\)）\*]{0,4}\s*[\.。!！]?\s*$", + re.IGNORECASE, +) +_HEAD_LABEL_RE = re.compile(r"\s*\*{0,2}\(?([A-Z])\)?\s*[:\.\)]\s") +_BOLD_LABEL_RE = re.compile(r"\*\*\s*\(?([A-Z])\)?\s*[:\.]") + + +def extract_option_robust( + text: Any, candidates: List[str], option_texts: Optional[Dict[str, str]] = None +) -> Optional[str]: + """Robust (tier-2) MCQ letter extraction for chain-of-thought output: + VLMEvalKit's can_infer core (token position + option-text restatement) + extended with -tag, \boxed{}, answer-cue (incl. CJK), and + answer-label rules. Returns the extracted candidate letter or None. + + Args: the raw prediction text (str or 1-element list), the candidate + letters, and the letter->option-text map for restatement matching. + """ + raw = _to_text(text) + if not raw: + return None + cand_set = [str(c).strip().upper() for c in (candidates or OPTION_LETTERS)] + + # 1. The whole (normalized) answer is one bare option letter, any case + # ("b", "(C)", "\boxed{D}") — unambiguous, accept directly. + whole = _WRAPPING_BRACKETS_RE.sub("", normalize_text(raw).strip()).strip() + if whole.upper() in cand_set: + return whole.upper() + + # 2. The LAST ... tag, when present, is the model's final + # pick — resolve inside it (bare letter, token match, text restatement). + tags = _ANSWER_TAG_RE.findall(raw) + if tags: + inner_raw = tags[-1].strip() + if inner_raw.upper() in cand_set: + return inner_raw.upper() + inner = can_infer_option(inner_raw, cand_set) or can_infer_text(inner_raw, option_texts or {}) + if inner: + return inner + + # 3. The LAST \boxed{...} letter, tolerating an inner \text{}/\mathrm{} + # wrap and a leading '(' / '*': "$\boxed{C}$", "$\boxed{\text{C}}$". + boxes = [b.upper() for b in _BOXED_LETTER_RE.findall(raw) if b.upper() in cand_set] + if boxes: + return boxes[-1] + + # 4. The LAST explicit answer cue (markdown/CJK tolerant, uppercase letter). + cues = [c.upper() for c in _ANSWER_CUE.findall(raw) if c.upper() in cand_set] + if cues: + return cues[-1] + + # 5. A lowercase letter after a cue counts only when it TERMINATES the + # response ("the answer is b") — there it cannot be an article. Mid-text + # lowercase letters stay ignored ("the answer is a cute cat" is not A). + tail = _CUE_TAIL_RE.search(raw) + if tail and tail.group(1).upper() in cand_set: + return tail.group(1).upper() + + # 6. VLMEvalKit core: the unique candidate letter near the end of the + # answer, ORIGINAL case (uppercasing first would make every article "a" + # a hit for option A). + opt = can_infer_option(raw, cand_set) + if opt: + return opt + + # 7. Answer-label formats the position/count test drops: + # 7a — answer-first label ("C: A man surfing") in SHORT responses only; + # long answers often reconsider the leading letter later, so they + # defer to the end-position/cue rules above. + if len(raw.strip()) <= 80: + head = _HEAD_LABEL_RE.match(raw) + if head and head.group(1).upper() in cand_set: + return head.group(1).upper() + # 7b — a uniquely bolded option label "**D:**" anywhere (the model's + # final restated pick). + bolds = {b.upper() for b in _BOLD_LABEL_RE.findall(raw) if b.upper() in cand_set} + if len(bolds) == 1: + return next(iter(bolds)) + + # 8. Fallback: the answer restates exactly one option's text. + return can_infer_text(raw, option_texts or {}) + + +def _letters_from_segment(segment: str, candidates: List[str]) -> Optional[Tuple[str, ...]]: + """A text segment that consists ONLY of candidate letters and separators + ("A, C" / "A and C" / "AC") -> sorted unique letters; anything else None.""" + text = normalize_text(segment).strip().rstrip(".。") + if not text: + return None + tokens = [t for t in _LETTER_SEPARATOR_RE.split(text) if t] + letters: List[str] = [] + for t in tokens: + up = t.upper() + if len(up) == 1 and up in candidates: + letters.append(up) + elif 2 <= len(up) <= len(candidates) and all(c in candidates for c in up) and len(set(up)) == len(up): + letters.extend(up) # compact form "AC" + else: + return None + if not letters or len(set(letters)) != len(letters): + return None + return tuple(sorted(letters)) + + +_MULTI_ANSWER_CUE_RE = re.compile( + r"(?:final answers?|the answers? (?:are|is)|answers? (?:are|is)|answers?\s*[:\-：]|最终答案|答案)", + re.IGNORECASE, +) + + +def extract_letter_set(text: Any, candidates: List[str]) -> Optional[Tuple[str, ...]]: + """Multi-select prediction -> sorted letter tuple. Tries, in order: the LAST + tag, the segment after the LAST answer cue, then the whole answer. + Only letters-and-separators segments are accepted — free text defers to the + LLM extractor (official LogicVista protocol: GPT extracts the letters, then + sorted set equality; vlmeval/dataset/utils/logicvista.py:50-68).""" + raw = _to_text(text) + if not raw: + return None + cand = [str(c).strip().upper() for c in (candidates or OPTION_LETTERS[:8])] + + tags = _ANSWER_TAG_RE.findall(raw) + if tags: + got = _letters_from_segment(tags[-1], cand) + if got: + return got + + cues = list(_MULTI_ANSWER_CUE_RE.finditer(raw)) + if cues: + got = _letters_from_segment(raw[cues[-1].end():], cand) + if got: + return got + + return _letters_from_segment(raw, cand) diff --git a/mmeval/scoring/pipeline.py b/mmeval/scoring/pipeline.py new file mode 100644 index 00000000..c8aa57e8 --- /dev/null +++ b/mmeval/scoring/pipeline.py @@ -0,0 +1,430 @@ +import glob +import json +import os +import platform +import subprocess +from collections import Counter +from concurrent.futures import ThreadPoolExecutor, as_completed +from typing import Any, Dict, List, Optional + +from mmeval.scoring.stages import INVALID, LLM_STAGE_NAMES, SCORE, STAGE_REGISTRY +from mmeval.scoring.stages.llm import _resolve_setting, local_judge_revision +from mmeval.scoring.protocol import ResolvedProtocol, extract_dataset_meta, resolve_protocol +from mmeval.scoring.report import build_summary +from mmeval.scoring.extraction import ( + OPTION_LETTERS, + extract_option_texts, + extract_options, + get_field, + get_question, + infer_question_type, + parse_gt_letters, + parsed_options, +) + +try: + from tqdm import tqdm +except Exception: # pragma: no cover - fallback when tqdm is unavailable + tqdm = None + + +def discover_result_files(out_dir: str, pattern: str = "**/result.json") -> List[str]: + if not out_dir: + return [] + search_pattern = os.path.join(out_dir, pattern) + return sorted(x for x in glob.glob(search_pattern, recursive=True) if os.path.isfile(x)) + + +def build_stages(args, pipeline): + """Instantiate the validated pipeline (list of atomic stage names). + Construction is uniform; LLM-backed stages read their client/config + from args, the rest ignore it.""" + return [STAGE_REGISTRY[name](args) for name in pipeline] + + +def _resume_fingerprint(args, proto: ResolvedProtocol, + dataset_meta: Optional[Dict[str, Any]] = None) -> Dict[str, Any]: + """The RESOLVED config subset that determines a sample's verdict (post + CLI/dataset_meta/default precedence). Cached rows produced under a different + fingerprint must not be reused — resuming a rule-only run into a + `rule-match,llm-match` rerun would silently keep the old verdicts.""" + meta = dataset_meta or {} + fingerprint = { + # Dataset identity binds the cache to WHAT was scored, not just the + # out_dir folder: a different split/subset/dataset scored into the same + # directory (eval-ids 0..N collide) is a cache miss and re-scored. + # Absent-vs-absent (old result.json / local@json) matches on None; + # absent-vs-present discards the stale cache. + "dataset_name": meta.get("dataset_name"), + "subset": meta.get("subset"), + "split": meta.get("split"), + "score_pipeline": list(proto.pipeline), + "score_gt_field": args.score_gt_field, + "score_pred_field": args.score_pred_field, + "score_question_type": args.score_question_type, + "score_numeric_rel_tol": proto.numeric_rel_tol, + "score_numeric_abs_tol": proto.numeric_abs_tol, + "score_string_match": proto.string_match, + "score_anls_threshold": proto.anls_threshold, + } + # With an LLM stage in the pipeline, the judge's identity determines + # verdicts too — a gpt-4o-mini cache must not resume into a gpt-5 run. + # Judge-free pipelines deliberately exclude these so a judge-config edit + # doesn't invalidate them. + if any(name in LLM_STAGE_NAMES for name in proto.pipeline): + fingerprint["judge_provider"] = args.judge_provider + fingerprint["judge_model"] = args.judge_model + fingerprint["judge_temperature"] = args.judge_temperature + # Both change verdicts: include_reason alters the judge prompt, and + # max_tokens changes truncation/parse-failure behavior. Fingerprint the + # RESOLVED max_tokens (CLI > env > default), like the stage uses. + fingerprint["judge_include_reason"] = bool(args.judge_include_reason) + fingerprint["judge_max_tokens"] = _resolve_setting( + getattr(args, "judge_max_tokens", None), "JUDGE_MAX_TOKENS", 2048) + if (args.judge_provider or "").strip().lower() == "local": + # A local model name doesn't pin weights the way an API model + # string does — the resolved snapshot commit must invalidate too. + fingerprint["judge_model_revision"] = local_judge_revision(args.judge_model) + return fingerprint + + +def _provenance() -> Dict[str, Any]: + """Environment identity embedded in score.json so any run can be reproduced. + Deterministic on a given checkout (no timestamps — reruns stay byte-identical).""" + try: + commit = subprocess.run( + ["git", "rev-parse", "HEAD"], + capture_output=True, text=True, timeout=5, + cwd=os.path.dirname(os.path.abspath(__file__)), + ).stdout.strip() or None + except Exception: + commit = None + try: + from importlib.metadata import version + openai_version = version("openai") + except Exception: + openai_version = None + return { + "git_commit": commit, + "python": platform.python_version(), + "openai": openai_version, + } + + + +def _load_cached_results(out_path: str, resume_enabled: bool, fingerprint: Dict[str, Any]) -> Dict[str, Dict[str, Any]]: + if not resume_enabled: + return {} + + cache_by_id: Dict[str, Dict[str, Any]] = {} + tmp_path = f"{out_path}.tmp" + # Union of both caches: the final score.json first, then the tmp overlays + # it (fresher rows win). Both must pass the fingerprint check, so every + # retained row was produced under the current verdict-determining config. + for path in (out_path, tmp_path): + if not os.path.exists(path): + continue + try: + with open(path, "r") as f: + payload = json.load(f) + if not isinstance(payload, dict): + continue # a bare-list tmp carries no config fingerprint; don't trust it + config = payload.get("config") or {} + if any(config.get(k) != v for k, v in fingerprint.items()): + print(f"[score] ignoring resume cache {path}: scoring config changed", flush=True) + continue + rows = payload.get("samples") + if not isinstance(rows, list): + continue + for row in rows: + if not isinstance(row, dict): + continue + if "eval-id" not in row: + continue + # Rows that failed because the judge API failed are not verdicts; + # re-score them instead of freezing a transient outage into 0s. + if str(row.get("reason", "")).startswith("llm_error"): + continue + cache_by_id[str(row["eval-id"])] = row + except Exception: + continue + return cache_by_id + + +def _flush_resume_tmp(tmp_path: str, samples: List[Dict[str, Any]], fingerprint: Dict[str, Any]) -> None: + # Atomic write: a crash mid-flush must never leave a truncated tmp that a + # resume would then have to discard (losing all progress since the last flush). + part_path = f"{tmp_path}.part" + with open(part_path, "w") as f: + json.dump({"config": fingerprint, "samples": samples}, f, indent=2) + os.replace(part_path, tmp_path) + + +def _score_single_sample(sample: Dict[str, Any], args, stages, proto: ResolvedProtocol) -> Dict[str, Any]: + gt = get_field(sample, args.score_gt_field, None) + if (gt is None or (isinstance(gt, str) and not gt.strip())) and args.score_gt_field == "answer": + # Judge-graded datasets carry the reference in the reserved + # `reference_response` field instead of `answer` — generic fallback, + # never per-dataset. + gt = get_field(sample, "reference_response", None) + pred = get_field(sample, args.score_pred_field, None) + question = get_question(sample) + question_type = infer_question_type(sample, args.score_question_type, gt=gt) + options = extract_options(sample) if question_type == "mcq" else [] + if question_type == "mcq": + # The gt letters prove candidate membership: when option discovery + # provably missed some (malformed inline lists, e.g. a "B Only ..." + # line without label punctuation), widen to the contiguous A..max + # span instead of failing on an option the dataset says exists. + gt_letters = parse_gt_letters(gt) or () + missing = [l for l in gt_letters if l not in options] + if missing: + top = max(OPTION_LETTERS.index(l) for l in list(options) + list(gt_letters) + if l in OPTION_LETTERS) + options = OPTION_LETTERS[:top + 1] + # Whether the candidates are a real parsed option source (vs the A-F + # fallback) — gates compact multi-letter gt parsing in the rule stages. + options_parsed = question_type == "mcq" and parsed_options(sample) is not None + option_texts = extract_option_texts(sample, options) + + eval_id = sample.get("eval-id", sample.get("id")) + output = { + "eval-id": eval_id, + "question_type": question_type, + "gt": gt, + "pred": pred, + "decided_by": None, + "matched": None, + "score": 0.0, + "is_correct": 0, + "status": "ok", + } + + # Blank gt (answer-withheld test splits) must be invalid, not "confidently 0". + if gt is None or pred is None or (isinstance(gt, str) and not gt.strip()): + output["status"] = "invalid" + output["reason"] = "missing_gt_or_pred" + return output + + context = { + "question_type": question_type, + "question": question, + "pred": pred, + "gt": gt, + "options": options, + "options_parsed": options_parsed, + "option_texts": option_texts, + "numeric_rel_tol": proto.numeric_rel_tol, + "numeric_abs_tol": proto.numeric_abs_tol, + "string_match": proto.string_match, + "anls_threshold": proto.anls_threshold, + } + + trace = [] + for stage in stages: + result = stage.run(sample, context) + if args.score_debug: + trace.append( + { + "stage": stage.name, + "outcome": result.outcome, + "score": result.score, + "matched": result.matched, + "reason": result.reason, + "meta": result.meta, + } + ) + if result.meta and "judge_prob" in result.meta: + # Observability only (P of the judge's verdict token); never a verdict input. + output["judge_prob"] = result.meta["judge_prob"] + if result.outcome == INVALID: + # The sample cannot be graded under this stage's protocol. + output["status"] = "invalid" + output["reason"] = result.reason + break + if result.outcome == SCORE: + # Decided: short-circuit. is_correct = full credit only; the + # fractional per-sample `score` feeds summary.mean_score. + output["decided_by"] = stage.name + output["matched"] = result.matched + output["score"] = float(result.score) + output["is_correct"] = int(result.score >= 1.0) + if result.reason is not None and args.judge_include_reason: + output["reason"] = result.reason + break + if stage.uses_llm and result.reason is not None: + # LLM stage pass-reasons are error markers ("llm_error: ...") or — + # only when --judge_include_reason — the judge's explanation; both + # must survive into the output unless a later stage decides + # (errors especially: see summary.llm_errors). + output["reason"] = result.reason + + if args.score_debug: + output["trace"] = trace + return output + + +def _score_sample_at(idx: int, sample: Dict[str, Any], args, stages, proto: ResolvedProtocol): + """Worker wrapper: score one sample, keep its position, default a missing eval-id.""" + scored = _score_single_sample(sample, args, stages, proto) + if scored["eval-id"] is None: + scored["eval-id"] = sample.get("eval-id", sample.get("id", idx)) + return idx, scored + + +def _reject_nonconforming_layout(result_file: str, samples: List[Dict[str, Any]], gt_field: str) -> None: + """Grading fields live ONLY at the sample top level; fail fast instead of + scoring every sample as missing-gt.""" + for sample in samples: + if not isinstance(sample, dict): + continue + messages = sample.get("messages") or [] + msg = messages[0] if messages and isinstance(messages[0], dict) else {} + if gt_field in msg and gt_field not in sample: + raise ValueError( + f"{result_file}: non-conforming result format — `{gt_field}` found inside " + f"messages[0]. Grading fields (answer, question_type, reference_response, " + f"--score_gt_field) must be TOP-LEVEL sample fields; messages carry only " + f"render inputs and the model response." + ) + + +def score_result_file(result_file: str, args) -> Dict[str, Any]: + with open(result_file, "r") as f: + samples = json.load(f) + if not isinstance(samples, list): + raise ValueError(f"{result_file} must be a JSON list.") + _reject_nonconforming_layout(result_file, samples, args.score_gt_field) + + # Metadata-driven protocol resolution (docs/en/SCORING.md): per knob, + # explicit CLI flag > dataset_meta from result.json > built-in default. + # Conflicting dataset_meta across samples or unsupported protocols raise here. + dataset_meta = extract_dataset_meta(samples, result_file) + proto = resolve_protocol(args, dataset_meta, result_file) + + out_path = os.path.join(os.path.dirname(result_file), args.score_output_name) + tmp_path = f"{out_path}.tmp" + fingerprint = _resume_fingerprint(args, proto, dataset_meta) + cached_by_id = _load_cached_results(out_path=out_path, resume_enabled=args.score_resume, fingerprint=fingerprint) + stages = build_stages(args, proto.pipeline) + + scored_samples: List[Dict[str, Any]] = [None] * len(samples) + resumed_count = 0 + pbar = None + if args.score_progress_bar and tqdm is not None: + pbar = tqdm( + total=len(samples), + desc=f"Scoring {os.path.basename(os.path.dirname(result_file))}", + leave=False, + ) + + # Samples whose normalized eval-id is not unique (true duplicates, or an int/str + # collision like 1 vs "1") cannot be safely resumed from cache: one cached row + # would stand in for several distinct samples. Always re-score those. + id_counts = Counter( + str(s.get("eval-id", s.get("id", i))) if isinstance(s, dict) else str(i) + for i, s in enumerate(samples) + ) + dup_ids = {k for k, v in id_counts.items() if v > 1} + if dup_ids and cached_by_id: + print(f"[score] {len(dup_ids)} non-unique eval-id(s) in {result_file}; those samples will be re-scored, not resumed", flush=True) + + uncached_samples = [] + for idx, sample in enumerate(samples): + eval_id = str(sample.get("eval-id", sample.get("id", idx))) + cached = cached_by_id.get(eval_id) if eval_id not in dup_ids else None + if cached is not None: + scored_samples[idx] = cached + resumed_count += 1 + if pbar is not None: + pbar.update(1) + continue + uncached_samples.append((idx, sample)) + + # Threads only help when samples block on API calls (LLM stages); for + # judge-free pipelines the GIL makes them a net slowdown (measured + # ~3x on 10k samples). Default to serial there; an explicit + # --parallel_per_task always wins. + uses_api = any(n in LLM_STAGE_NAMES for n in proto.pipeline) + if "score_parallel_per_task" in getattr(args, "_explicit_flags", frozenset()) or uses_api: + sample_workers = max(1, int(getattr(args, "score_parallel_per_task", 1))) + else: + sample_workers = 1 + processed_new = 0 + + if sample_workers <= 1 or len(uncached_samples) <= 1: + for idx, sample in uncached_samples: + i, scored = _score_sample_at(idx, sample, args, stages, proto) + scored_samples[i] = scored + processed_new += 1 + if pbar is not None: + pbar.update(1) + if args.score_resume and processed_new % max(1, int(args.score_save_freq)) == 0: + _flush_resume_tmp(tmp_path, [x for x in scored_samples if x is not None], fingerprint) + else: + with ThreadPoolExecutor(max_workers=sample_workers) as executor: + futures = [ + executor.submit(_score_sample_at, idx, sample, args, stages, proto) + for idx, sample in uncached_samples + ] + for future in as_completed(futures): + i, scored = future.result() + scored_samples[i] = scored + processed_new += 1 + if pbar is not None: + pbar.update(1) + if args.score_resume and processed_new % max(1, int(args.score_save_freq)) == 0: + _flush_resume_tmp(tmp_path, [x for x in scored_samples if x is not None], fingerprint) + + if pbar is not None: + pbar.close() + + if any(x is None for x in scored_samples): + raise RuntimeError(f"Incomplete scoring results found for {result_file}") + + summary = build_summary(scored_samples) + + payload = { + "summary": summary, + "config": { + # RESOLVED per-knob values (CLI > dataset_meta > default); which + # source decided each knob is recorded in knob_sources. The resume + # fingerprint is embedded verbatim: _load_cached_results validates + # cached rows against this block, so it must stay a superset of + # every fingerprint key. + **fingerprint, + "task_type": proto.task_type, + "official_protocol": proto.official_protocol, + "knob_sources": proto.knob_sources, + "judge_provider": args.judge_provider, + "judge_model": args.judge_model, + "judge_temperature": args.judge_temperature, + "judge_include_reason": args.judge_include_reason, + "score_progress_bar": args.score_progress_bar, + "score_resume": args.score_resume, + "score_save_freq": args.score_save_freq, + "score_parallel_per_task": sample_workers, + "resumed_count": resumed_count, + "provenance": _provenance(), + # List of notes: the scorer-wide caveat, rubric degradations, the + # none-stamp, and score_protocol.note verbatim (score_note). + "protocol_notes": proto.protocol_notes, + }, + "samples": scored_samples, + } + + with open(out_path, "w") as f: + json.dump(payload, f, indent=2) + # A finished score.json supersedes any resume tmp regardless of how this + # run was configured — a stale tmp left behind would overlay OLDER rows + # onto a fresher final in a later resume's cache union. + if os.path.exists(tmp_path): + os.remove(tmp_path) + + return { + "result_file": result_file, + "score_file": out_path, + "summary": summary, + "resumed_count": resumed_count, + } + diff --git a/mmeval/scoring/protocol.py b/mmeval/scoring/protocol.py new file mode 100644 index 00000000..7273ae56 --- /dev/null +++ b/mmeval/scoring/protocol.py @@ -0,0 +1,239 @@ +"""score_pipeline -> execution resolution (the metadata-driven scoring contract). + +Every sample may carry a flat top-level `dataset_meta` block (injected at +inference time by the mmeval_hf loader, persisted in result.json). Its +`score_pipeline` declares the dataset's official scoring protocol as an +ORDERED LIST OF ATOMIC STAGE NAMES (see mmeval/scoring/stages/): + + "score_pipeline": ["exact-match", "rule-match", "llm-judge"] + +Two non-list forms are part of the contract: +- [] — the dataset explicitly declares that no + official protocol exists; the scorer runs the default pipeline and stamps + the output non-official. +- {"unsupported": "", — the official protocol cannot be executed + "reason": ""} by this scorer; scoring is REFUSED with + an actionable error (an explicit --score_pipeline forces approximate scoring, + stamped non-official). + +The scorer resolves each knob with the precedence + explicit CLI flag > dataset_meta > built-in default, +records which source decided every knob (knob_sources), and stamps +official-protocol status. +""" +import json +from dataclasses import dataclass, field +from typing import Any, Dict, List, Optional + +from mmeval.scoring.stages import STAGE_REGISTRY, GRADER_STAGE_NAMES + +# The scorer-wide caveat recorded into every score.json (first protocol note). +GLOBAL_PROTOCOL_NOTE = ( + "Per-sample scoring; official aggregations noted per dataset in these notes " + "are not reproduced. LLM failures are marked wrong and counted in " + "summary.llm_errors (never randomly guessed, unlike VLMEvalKit)." +) + +NO_PROTOCOL_NOTE = ( + "score_pipeline=[]: the dataset declares that no official eval protocol " + "exists; numbers are mm-eval-convention, not official benchmark results." +) + +_STRING_MATCH_VALUES = ("exact", "contains", "anls") + + +def parse_pipeline(spec, where: str) -> List[str]: + """Validate a pipeline spec (comma-separated string from the CLI, or a JSON + array from dataset metadata) into an ordered list of atomic stage names.""" + if isinstance(spec, str): + stages = [x.strip() for x in spec.split(",") if x.strip()] + elif isinstance(spec, (list, tuple)): + stages = [str(x).strip() for x in spec if str(x).strip()] + else: + raise ValueError(f"{where}: score pipeline must be a list of stage names, got {type(spec).__name__}") + if not stages: + raise ValueError(f"{where}: empty pipeline; known stages: {sorted(STAGE_REGISTRY)}") + unknown = [s for s in stages if s not in STAGE_REGISTRY] + if unknown: + raise ValueError(f"{where}: unknown stage(s) {unknown}; known stages: {sorted(STAGE_REGISTRY)}") + if len(set(stages)) != len(stages): + raise ValueError(f"{where}: duplicate stages in pipeline {stages}") + for s in stages[:-1]: + if s in GRADER_STAGE_NAMES: + raise ValueError( + f"{where}: `{s}` is a grader stage — it always decides every " + f"sample, so stages after it can never run; a grader must be " + f"the pipeline's last stage (and at most one per pipeline)" + ) + return stages + + +@dataclass +class ResolvedProtocol: + """Effective per-file scoring configuration after precedence resolution.""" + pipeline: List[str] + numeric_rel_tol: float + numeric_abs_tol: float + string_match: str # exact | contains | anls + anls_threshold: float + task_type: Optional[str] + official_protocol: Optional[bool] # None = unknown (no declaration) + protocol_notes: List[str] = field(default_factory=list) + knob_sources: Dict[str, str] = field(default_factory=dict) # knob -> cli|dataset_meta|default + + +def extract_dataset_meta(samples: List[Dict[str, Any]], result_file: str) -> Optional[Dict[str, Any]]: + """The single dataset_meta block of a result file, or None. Conflicting + blocks across samples mean a hand-merged file — refuse, never pick one.""" + distinct: Dict[str, Dict[str, Any]] = {} + for sample in samples: + if not isinstance(sample, dict): + continue + meta = sample.get("dataset_meta") + if isinstance(meta, dict) and meta: + distinct[json.dumps(meta, sort_keys=True)] = meta + if len(distinct) > 1: + keys = "\n ".join(sorted(distinct)) + raise ValueError( + f"{result_file} carries {len(distinct)} conflicting dataset_meta blocks " + f"(hand-merged file?). Score each source file separately, or strip the " + f"blocks and pass explicit CLI flags.\n {keys}" + ) + return next(iter(distinct.values()), None) + + +def _explicit(args) -> frozenset: + return getattr(args, "_explicit_flags", frozenset()) + + +def resolve_protocol(args, dataset_meta: Optional[Dict[str, Any]], result_file: str) -> ResolvedProtocol: + """Apply the per-knob precedence CLI > dataset_meta > default. Raises + ValueError for unsupported protocols (unless the user explicitly forced a + pipeline via --score_pipeline) and for an unsupported `score_type` key.""" + explicit = _explicit(args) + meta = dataset_meta or {} + if "score_type" in meta: + raise ValueError( + f"{result_file}: dataset_meta contains an unsupported `score_type` " + f"key. Declare the protocol with the `score_pipeline` schema in the " + f"dataset's metadata.json (docs/en/SCORING.md, 'Dataset metadata " + f"contract'), then re-run inference to regenerate this result.json." + ) + declared = meta.get("score_pipeline") + params = meta.get("score_params") or {} + + notes: List[str] = [GLOBAL_PROTOCOL_NOTE] + sources: Dict[str, str] = {} + + # --- pipeline ------------------------------------------------------------ + declared_stages: Optional[List[str]] = None # a declared, runnable protocol + unsupported: Optional[Dict[str, Any]] = None + if isinstance(declared, dict): + if "unsupported" not in declared: + raise ValueError( + f"{result_file}: malformed score_pipeline object {declared!r}; " + f'expected {{"unsupported": "", "reason": "..."}} or a stage list' + ) + unsupported = declared + elif isinstance(declared, (list, tuple)): + declared_stages = parse_pipeline(declared, f"{result_file} dataset_meta.score_pipeline") if declared else [] + elif declared is not None: + raise ValueError( + f"{result_file}: score_pipeline must be a stage list or an " + f'{{"unsupported": ...}} object, got {type(declared).__name__}' + ) + + if "score_pipeline" in explicit: + pipeline = parse_pipeline(args.score_pipeline, "--score_pipeline") + sources["score_pipeline"] = "cli" + if unsupported is not None: + notes.append( + f"explicit --score_pipeline overrides the dataset's unsupported official " + f"protocol ({unsupported.get('unsupported')}): numbers are approximate, " + f"NOT comparable to official results." + ) + elif declared_stages and pipeline != declared_stages: + notes.append( + f"explicit --score_pipeline overrides the dataset's declared pipeline " + f"{declared_stages}: the official protocol is NOT being executed; " + f"numbers are not comparable to official results." + ) + elif unsupported is not None: + name = unsupported.get("unsupported", "unknown") + reason = unsupported.get("reason", "no reason recorded") + raise ValueError( + f"{result_file}: the dataset's official protocol ({name}) is unsupported: " + f"{reason}. Pass an explicit --score_pipeline to force approximate scoring " + f"(numbers will NOT be official)." + ) + elif declared_stages: + pipeline = declared_stages + sources["score_pipeline"] = "dataset_meta" + else: # no declaration ([] or absent): built-in default pipeline + pipeline = parse_pipeline(args.score_pipeline, "--score_pipeline default") + sources["score_pipeline"] = "default" + + # --- numeric / string knobs (CLI > score_params > default) -------------- + def _knob(cli_name: str, param_name: str, default): + if cli_name in explicit: + sources[cli_name] = "cli" + return getattr(args, cli_name) + if param_name in params: + sources[cli_name] = "dataset_meta" + return params[param_name] + sources[cli_name] = "default" + return default + + numeric_rel_tol = float(_knob("score_numeric_rel_tol", "numeric_rel_tol", args.score_numeric_rel_tol)) + numeric_abs_tol = float(_knob("score_numeric_abs_tol", "numeric_abs_tol", args.score_numeric_abs_tol)) + string_match = str(_knob("score_string_match", "string_match", args.score_string_match)) + if string_match not in _STRING_MATCH_VALUES: + raise ValueError(f"{result_file}: invalid string_match {string_match!r}; allowed: {_STRING_MATCH_VALUES}") + # anls params spell it `threshold`; the rule-stage comparator knob is + # `anls_threshold` — both feed the same resolved value. + param_thresh = params.get("threshold", params.get("anls_threshold")) + if "score_anls_threshold" in explicit: + anls_threshold = float(args.score_anls_threshold) + sources["score_anls_threshold"] = "cli" + elif param_thresh is not None: + anls_threshold = float(param_thresh) + sources["score_anls_threshold"] = "dataset_meta" + else: + anls_threshold = float(args.score_anls_threshold) + sources["score_anls_threshold"] = "default" + + # --- judge rubric degradation note -------------------------------------- + rubric = params.get("rubric") + if rubric and rubric != "binary": + notes.append( + f"official rubric `{rubric}`" + + (f" scale={params['scale']}" if params.get("scale") else "") + + " degraded to a binary llm-judge; official numbers aggregate " + "the fractional/scaled judge scores and are not directly comparable." + ) + + # --- official-protocol stamp + score_note ------------------------------- + if declared is None: + official = None # nothing declared at all + elif declared_stages == []: + official = False # explicit "no official protocol" + notes.append(NO_PROTOCOL_NOTE) + elif sources["score_pipeline"] == "cli" and (unsupported is not None or pipeline != declared_stages): + official = False # declared protocol overridden from the CLI + else: + official = True + score_note = (meta.get("score_note") or "").strip() + if score_note: + notes.append(score_note) # verbatim, per the metadata contract + + return ResolvedProtocol( + pipeline=pipeline, + numeric_rel_tol=numeric_rel_tol, + numeric_abs_tol=numeric_abs_tol, + string_match=string_match, + anls_threshold=anls_threshold, + task_type=meta.get("task_type"), + official_protocol=official, + protocol_notes=notes, + knob_sources=sources, + ) diff --git a/mmeval/scoring/report.py b/mmeval/scoring/report.py new file mode 100644 index 00000000..b1ed1cb1 --- /dev/null +++ b/mmeval/scoring/report.py @@ -0,0 +1,60 @@ +from collections import defaultdict +from typing import Any, Dict, List + + +def build_summary(sample_results: List[Dict[str, Any]]) -> Dict[str, Any]: + total = len(sample_results) + correct = sum(1 for x in sample_results if x.get("is_correct") == 1) + # accuracy divides by ALL samples (invalid rows count as wrong); + # accuracy_valid divides by the gradable samples only — the honest number + # on splits with withheld answers or malformed rows. + accuracy = (correct / total) if total else 0.0 + + by_stage = defaultdict(lambda: {"total": 0, "correct": 0}) + by_question_type = defaultdict(lambda: {"total": 0, "correct": 0}) + invalid = 0 + llm_errors = 0 + judge_probs = [] + for item in sample_results: + stage = item.get("decided_by") or "undecided" + qtype = item.get("question_type") or "unknown" + by_stage[stage]["total"] += 1 + by_stage[stage]["correct"] += int(item.get("is_correct") == 1) + by_question_type[qtype]["total"] += 1 + by_question_type[qtype]["correct"] += int(item.get("is_correct") == 1) + if item.get("status") == "invalid": + invalid += 1 + # Samples whose judge-stage API call failed (API/parse error) — their 0s are + # not verdicts; a non-zero count here means the accuracy is a lower bound. + if str(item.get("reason", "")).startswith("llm_error"): + llm_errors += 1 + if item.get("judge_prob") is not None: + judge_probs.append(item["judge_prob"]) + + # accuracy = strict full-credit rate (mean is_correct). mean_score averages + # the fractional per-sample `score` (metric stages emit true fractions; + # rule/judge stages emit 1.0/0.0) — the OFFICIAL headline for fractional + # protocols. Equal to accuracy whenever no sample carries a fractional score. + mean_score = ( + sum((x["score"] if x.get("score") is not None else float(x.get("is_correct") == 1)) + for x in sample_results) / total + if total else 0.0 + ) + valid = total - invalid + summary = { + "total": total, + "valid": valid, + "correct": correct, + "accuracy": accuracy, + "accuracy_valid": (correct / valid) if valid else 0.0, + "mean_score": mean_score, + "invalid": invalid, + "llm_errors": llm_errors, + "by_stage": dict(by_stage), + "by_question_type": dict(by_question_type), + } + # Mean P(verdict) over judge-graded samples that returned logprobs — + # observability for how confident the judge was, never a verdict input. + if judge_probs: + summary["judge_prob_mean"] = sum(judge_probs) / len(judge_probs) + return summary diff --git a/mmeval/scoring/stages/__init__.py b/mmeval/scoring/stages/__init__.py new file mode 100644 index 00000000..2e1c933e --- /dev/null +++ b/mmeval/scoring/stages/__init__.py @@ -0,0 +1,43 @@ +from mmeval.scoring.stages.base import BaseStage, StageResult, INVALID, PASS, SCORE +from mmeval.scoring.stages.exact_match import ExactMatchStage +from mmeval.scoring.stages.rule_match import RuleMatchStage +from mmeval.scoring.stages.llm import LLMMatchStage, LLMJudgeStage +from mmeval.scoring.stages.metrics import AnlsStage, VqaAccuracyStage + +# Every atomic scoring stage, by pipeline name (mainstream eval vocabulary: +# exact match, answer extraction, LLM-as-judge, VQA accuracy, ANLS). +STAGE_REGISTRY = { + "exact-match": ExactMatchStage, + "rule-match": RuleMatchStage, + # LLM extracts the chosen option, then rule-based set compare (VLMEvalKit). + "llm-match": LLMMatchStage, + # LLM judges whether the response is correct (our approach). + "llm-judge": LLMJudgeStage, + "vqa-accuracy": VqaAccuracyStage, + "anls": AnlsStage, +} + +# Derived from the stage classes' own declarations — the class attribute is +# the single source of truth, so a new stage cannot silently miss these sets. +# uses_llm: the stage calls an LLM API; its presence pulls the judge identity +# into the resume fingerprint. grader: the stage always decides (or +# invalidates) and never abstains, so it is only valid as the last stage. +LLM_STAGE_NAMES = frozenset(n for n, c in STAGE_REGISTRY.items() if c.uses_llm) +GRADER_STAGE_NAMES = frozenset(n for n, c in STAGE_REGISTRY.items() if c.grader) + +__all__ = [ + "BaseStage", + "StageResult", + "PASS", + "SCORE", + "INVALID", + "ExactMatchStage", + "RuleMatchStage", + "LLMMatchStage", + "LLMJudgeStage", + "VqaAccuracyStage", + "AnlsStage", + "STAGE_REGISTRY", + "LLM_STAGE_NAMES", + "GRADER_STAGE_NAMES", +] diff --git a/mmeval/scoring/stages/base.py b/mmeval/scoring/stages/base.py new file mode 100644 index 00000000..2cd59ad4 --- /dev/null +++ b/mmeval/scoring/stages/base.py @@ -0,0 +1,67 @@ +"""The atomic scoring stage: the single primitive behind every pipeline. + +A scoring pipeline is an ordered list of stages. For each sample, stages run +in order; each returns a StageResult with one of three outcomes: + +- PASS — the stage cannot decide this sample; the next stage runs. If every + stage passes, the sample scores 0.0 with decided_by null + (undecided, counted wrong). A pass may carry a reason — notably + "llm_error: ..." markers, which survive into the output row and + summary.llm_errors unless a later stage decides the sample. +- SCORE — the stage decided the sample and the pipeline short-circuits. + `score` is in [0, 1]: rule stages emit 1.0 on a match and never + decide a miss (they pass instead); llm-judge emits 1.0/0.0; + metric stages emit fractional values. is_correct = score >= 1.0. +- INVALID — the sample cannot be graded under this stage's protocol (e.g. + vqa-accuracy without the multi-annotator reference list). Terminal: + the sample is recorded as status=invalid with the stage's reason. +""" +from dataclasses import dataclass, field +from typing import Any, Dict, Optional + +PASS = "pass" +SCORE = "score" +INVALID = "invalid" + + +@dataclass +class StageResult: + outcome: str + score: float = 0.0 + matched: Optional[str] = None + reason: Optional[str] = None + meta: Dict[str, Any] = field(default_factory=dict) + + @classmethod + def passed(cls, reason: Optional[str] = None, meta: Optional[Dict[str, Any]] = None) -> "StageResult": + return cls(outcome=PASS, reason=reason, meta=meta or {}) + + @classmethod + def scored(cls, score: float, matched: Optional[str] = None, + reason: Optional[str] = None, meta: Optional[Dict[str, Any]] = None) -> "StageResult": + return cls(outcome=SCORE, score=float(score), matched=matched, reason=reason, meta=meta or {}) + + @classmethod + def not_gradable(cls, reason: str) -> "StageResult": + return cls(outcome=INVALID, reason=reason) + + +class BaseStage: + name = "base" + # Declared by each stage class; the registry derives LLM_STAGE_NAMES / + # GRADER_STAGE_NAMES from these, so no behavior is keyed off name sets. + uses_llm = False # calls an LLM API: judge identity enters the fingerprint + # grader: the stage always decides (fractional score 0..1) or invalidates, + # never abstains -> only valid as the last pipeline stage, at most one. + # DELIBERATE COUPLING: `grader` bundles "fractional output" with "always + # decides / last-only". If a future stage ever needs one without the + # other, split this flag — known, accepted trade-off, not an oversight. + grader = False + + def __init__(self, args=None): + # Uniform construction: every stage accepts the parsed args; stages + # without external clients ignore them. + self.args = args + + def run(self, sample: Dict[str, Any], context: Dict[str, Any]) -> StageResult: + raise NotImplementedError diff --git a/mmeval/scoring/stages/exact_match.py b/mmeval/scoring/stages/exact_match.py new file mode 100644 index 00000000..7246e0df --- /dev/null +++ b/mmeval/scoring/stages/exact_match.py @@ -0,0 +1,147 @@ +from mmeval.scoring.stages.base import BaseStage, StageResult +from mmeval.scoring.extraction import ( + extract_letter_set, + extract_option_strict, + normalize_for_exact, + normalize_open_answer, + parse_gt_letters, + parse_number, +) + + +def extract_letters_strict(pred, candidates): + """Strict-tier letter-set extraction: a clean letters-and-separators answer + ("B", "A, C", "BD"), else a single unambiguous strict-form letter.""" + letters = extract_letter_set(pred, candidates) + if letters: + return letters + single = extract_option_strict(pred, candidates) + return (single,) if single else None + + +def numbers_match(pred: float, gt: float, context) -> bool: + """Exact float equality by default; the protocol's tolerances widen it: + relative (`numeric_rel_tol` — ChartQA 0.05) and/or absolute + (`numeric_abs_tol` — DynaMath 0.001, OlympiadBench-style epsilon). Either + satisfied tolerance matches. Relative tolerance does NOT apply when the + ground truth is 0 — the official relaxed-accuracy protocol falls back to + exact matching there (lmms-eval chartqa/utils.py relaxed_correctness: + `target_float` truthiness sends target==0 to the exact-match branch).""" + if pred == gt: + return True + abs_tol = float(context.get("numeric_abs_tol") or 0.0) + if abs_tol > 0.0 and abs(pred - gt) <= abs_tol: + return True + rel_tol = float(context.get("numeric_rel_tol") or 0.0) + if rel_tol > 0.0 and gt != 0.0: + return abs(pred - gt) / abs(gt) <= rel_tol + return False + + +def numeric_branch_active(context) -> bool: + """The numeric comparator applies when the protocol declares numeric + tolerances or uses the default exact string matching. A protocol that + declares a NON-exact string_match without numeric tolerances (OCRBench: + `contains`) makes that string comparator official for every free-form + row, digit-string answers included (VLMEvalKit + vlmeval/dataset/utils/ocrbench.py:36-42 applies the substring rule to all + categories). A protocol declaring BOTH (ChartQAPro: `anls` + + numeric_rel_tol) dispatches by answer shape, so tolerances keep the + numeric branch alive.""" + return (str(context.get("string_match") or "exact") == "exact" + or float(context.get("numeric_rel_tol") or 0.0) > 0.0 + or float(context.get("numeric_abs_tol") or 0.0) > 0.0) + + +def gt_references(gt) -> list: + """A gt may be a single value or a reference LIST (OCRBench answer lists, + VQA-style multi-annotator answers).""" + if isinstance(gt, (list, tuple)): + return [x for x in gt if x is not None] + return [gt] + + +def open_text_match(pred_text: str, pred_raw, gt_raw, context) -> bool: + """The rule stages' text comparator, per the protocol's `string_match`: + exact (default) — normalized equality against any reference; + contains — OCRBench protocol: a normalized reference appears as a substring + of the normalized prediction; + anls — threshold ANLS against the references (ChartQAPro).""" + mode = context.get("string_match") or "exact" + refs = gt_references(gt_raw) + if mode == "anls": + from mmeval.scoring.stages.metrics import anls + score, error = anls(pred_raw, [str(r) for r in refs], + threshold=float(context.get("anls_threshold") or 0.5)) + return error is None and score > 0.0 + pred_norm = normalize_open_answer(pred_text) + if not pred_norm: + return False + for ref in refs: + ref_norm = normalize_open_answer(ref) + if not ref_norm: + continue + if mode == "contains": + if ref_norm in pred_norm: + return True + elif pred_norm == ref_norm: + return True + return False + + +class ExactMatchStage(BaseStage): + """Strict-form deterministic comparison: near-zero false positives, zero + cost. Matches only unambiguous answers (a bare option letter, a clean + letter set, a pure yes/no, a whole-string number or text); anything it + cannot decide is passed to the next stage.""" + + name = "exact-match" + + def run(self, sample, context): + question_type = context["question_type"] + pred_raw = context["pred"] + gt_raw = context["gt"] + + if gt_raw is None or pred_raw is None: + return StageResult.passed() + + if question_type == "mcq": + # Letter-SET grading: single-select is the |set|=1 special case, + # multi-letter gts like LogicVista's "A, C" the general one. + candidates = context["options"] + # Compact multi-letter gts ("AC") are accepted HERE — the row is + # already typed mcq — while parse_gt_letters stays conservative for + # type inference. Only with REAL parsed options, though: under the + # A-F fallback (options-in-image datasets) word-like gts would + # parse as letter sets ("BED" -> B,D,E). + gt_letters = parse_gt_letters(gt_raw) + if gt_letters is None and context.get("options_parsed"): + gt_letters = extract_letter_set(gt_raw, candidates) + if gt_letters is None: + g = extract_option_strict(gt_raw, candidates) or normalize_for_exact(gt_raw).upper() + gt_letters = (g,) if g else None + pred_letters = extract_letters_strict(pred_raw, candidates) + if gt_letters and pred_letters and pred_letters == gt_letters: + return StageResult.scored(1.0, matched=", ".join(gt_letters)) + return StageResult.passed() + + if question_type == "yes_no": + pred_norm = normalize_open_answer(pred_raw) + if pred_norm in {"yes", "no"} and pred_norm == normalize_open_answer(gt_raw): + return StageResult.scored(1.0, matched=pred_norm) + return StageResult.passed() + + if question_type == "numeric" and numeric_branch_active(context): + # Strict tier: the whole prediction is a number. + gt_num = parse_number(gt_raw) + pred_num = parse_number(pred_raw) + if gt_num is not None and pred_num is not None and numbers_match(pred_num, gt_num, context): + return StageResult.scored(1.0, matched=str(gt_raw)) + return StageResult.passed() + # numeric-shaped rows under a declared non-exact string_match fall + # through to the protocol's string comparator below. + + if open_text_match(str(pred_raw if not isinstance(pred_raw, list) else (pred_raw[0] if pred_raw else "")), + pred_raw, gt_raw, context): + return StageResult.scored(1.0, matched=str(gt_raw)) + return StageResult.passed() diff --git a/mmeval/scoring/stages/llm.py b/mmeval/scoring/stages/llm.py new file mode 100644 index 00000000..d16fed5c --- /dev/null +++ b/mmeval/scoring/stages/llm.py @@ -0,0 +1,469 @@ +import json +import math +import os +import re +import threading +import time +from typing import Any, Dict + +from mmeval.scoring.stages.base import BaseStage, StageResult +from mmeval.scoring.extraction import ( + can_infer_option, + can_infer_text, + extract_letter_set, + extract_option_robust, + normalize_for_exact, + normalize_text, + parse_gt_letters, +) + + +class _LocalJudgeClient: + """In-process open-weight judge (judge_provider=local). The judge is a + text-only task — both LLM stages build pure-text prompts and this client + only ever tokenizes text — so the model is loaded as a causal LM + (AutoModelForCausalLM + AutoTokenizer, dtype/device_map auto), which admits + the strong instruction-following LLMs best suited to judging. It exposes the + minimal chat-completions surface the judge stages consume, so prompts, + parsing, retry/llm_error accounting and the semaphore are identical across + providers. Generation is serialized by a lock: the bound is the GPU, not an + API rate limit.""" + + def __init__(self, model_name: str): + import types as _types + from transformers import AutoModelForCausalLM, AutoTokenizer + self._tok = AutoTokenizer.from_pretrained(model_name) + self._model = AutoModelForCausalLM.from_pretrained( + model_name, dtype="auto", device_map="auto") + self._model.eval() + self._lock = threading.Lock() + self.chat = _types.SimpleNamespace( + completions=_types.SimpleNamespace(create=self._create)) + + def _create(self, model, messages, temperature=0.0, max_tokens=512, + logprobs=False, **_ignored): + import torch + from types import SimpleNamespace as NS + try: + # Thinking-mode families (Qwen3/Qwen3.5) reason at length before + # the terse judge/extraction answer; disable it — the verdict + # contract is unchanged and per-call latency drops ~10x. + prompt = self._tok.apply_chat_template( + messages, add_generation_prompt=True, tokenize=False, + enable_thinking=False) + except TypeError: + prompt = self._tok.apply_chat_template( + messages, add_generation_prompt=True, tokenize=False) + inputs = self._tok(prompt, return_tensors="pt").to(self._model.device) + gen_kwargs = dict(max_new_tokens=int(max_tokens), + output_scores=bool(logprobs), + return_dict_in_generate=True, + pad_token_id=self._tok.pad_token_id or self._tok.eos_token_id) + if temperature and float(temperature) > 0: + gen_kwargs.update(do_sample=True, temperature=float(temperature)) + else: + gen_kwargs.update(do_sample=False) # greedy: deterministic verdicts + with self._lock, torch.inference_mode(): + out = self._model.generate(**inputs, **gen_kwargs) + new_tokens = out.sequences[0][inputs["input_ids"].shape[1]:] + text = self._tok.decode(new_tokens, skip_special_tokens=True) + content = None + if logprobs and getattr(out, "scores", None): + content = [NS(token=self._tok.decode([tid]), + logprob=torch.log_softmax(step[0].float(), dim=-1)[tid].item()) + for tid, step in zip(new_tokens.tolist(), out.scores)] + choice = NS(message=NS(content=text), + logprobs=NS(content=content) if content is not None else None) + return NS(choices=[choice]) + + +def local_judge_revision(model_name: str): + """Resolved snapshot commit of a locally cached judge model. Enters the + resume fingerprint for judge_provider=local: the model NAME alone does not + pin the weights the way an API model string does.""" + base = os.path.join( + os.path.expanduser(os.getenv("HF_HOME", "~/.cache/huggingface")), "hub", + "models--" + str(model_name).replace("/", "--"), "refs", "main") + try: + with open(base) as f: + return f.read().strip() + except OSError: + return None + + +_LOCAL_CLIENT_CACHE: Dict[str, "_LocalJudgeClient"] = {} + + +def _build_judge_client(args): + """Build the judge client: local (in-process open-weight model via the + framework's native transformers loading), openai, or azure_openai. + + Returns (client, model_or_deployment_name). + """ + provider = (args.judge_provider or "openai").strip().lower() + if provider == "local": + if not args.judge_model: + raise ValueError("`--judge_model` is required for the local judge.") + # Process-wide cache: stages are rebuilt per result file, and a + # multi-file scoring run must not reload the judge weights each time. + if args.judge_model not in _LOCAL_CLIENT_CACHE: + _LOCAL_CLIENT_CACHE[args.judge_model] = _LocalJudgeClient(args.judge_model) + return _LOCAL_CLIENT_CACHE[args.judge_model], args.judge_model + + try: + from openai import AzureOpenAI, OpenAI + except Exception as exc: # pragma: no cover + raise RuntimeError("openai package is required for API judge providers.") from exc + + if provider == "azure_openai": + key = os.getenv("AZURE_OPENAI_KEY") + endpoint = os.getenv("AZURE_OPENAI_ENDPOINT") + if not key or not endpoint: + raise ValueError("AZURE_OPENAI_KEY and AZURE_OPENAI_ENDPOINT are required for azure_openai judge.") + model = os.getenv("AZURE_OPENAI_DEPLOYNAME", args.judge_model) + client = AzureOpenAI( + api_key=key, + azure_endpoint=endpoint, + api_version=os.getenv("AZURE_OPENAI_API_VERSION", "2024-02-15-preview"), + ) + return client, model + + key = os.getenv("OPENAI_API_KEY") + base_url = os.getenv("OPENAI_BASE_URL") + if not key: + raise ValueError("OPENAI_API_KEY is required for openai judge.") + kwargs = {"api_key": key} + if base_url: + kwargs["base_url"] = base_url + return OpenAI(**kwargs), args.judge_model + + +def _resolve_setting(cli_value, env_name, default): + """Single source of truth for judge knobs: CLI flag > env var > default.""" + if cli_value is not None: + return int(cli_value) + return int(os.getenv(env_name, str(default))) + + +# HTTP statuses worth retrying: timeout/conflict/rate-limit and server errors. +_RETRYABLE_STATUS = {408, 409, 429} +_RETRYABLE_EXC_NAMES = {"APIConnectionError", "APITimeoutError", "RateLimitError", "InternalServerError"} + + +def _is_retryable(exc) -> bool: + status = getattr(exc, "status_code", None) + if isinstance(status, int): + return status in _RETRYABLE_STATUS or status >= 500 + return type(exc).__name__ in _RETRYABLE_EXC_NAMES + + +def _verdict_probability(logprobs_content, verdict) -> Any: + """P of the generated verdict digit, from API token logprobs. The judge reply + is tiny JSON ({"is_correct": 1, ...}) whose FIRST digit token is the verdict; + returns exp(logprob of that token), or None when logprobs are unavailable.""" + if not logprobs_content: + return None + target = str(int(verdict)) + for entry in logprobs_content: + token = str(getattr(entry, "token", "")).strip().strip('"') + if token == target: + lp = getattr(entry, "logprob", None) + return math.exp(lp) if lp is not None else None + if token and token.strip("01") == "" and token != target: + return None # first digit token disagrees with the parsed verdict — don't guess + return None + + +class _LLMStageBase(BaseStage): + uses_llm = True + """Shared client/config + a `_chat` helper with retry and reasoning-model + handling (gpt-5/o* use max_completion_tokens and reject temperature overrides). + + Concurrent LLM calls are capped by a process-wide semaphore (--judge_concurrency / + JUDGE_MAX_CONCURRENCY, default 4), DECOUPLED from the scorer's sample-worker count. + Firing 16+ judge calls at once over a full dataset saturates the API's tokens/min + quota; the gateway then returns degraded-but-valid responses (not errors), silently + tanking accuracy. Capping concurrency keeps sustained throughput under quota. + The semaphore is sized once per process by the first judge stage constructed.""" + + _sem = None + _sem_lock = threading.Lock() + + @classmethod + def _get_sem(cls, n): + if _LLMStageBase._sem is None: + with _LLMStageBase._sem_lock: + if _LLMStageBase._sem is None: + _LLMStageBase._sem = threading.Semaphore(max(1, n)) + return _LLMStageBase._sem + + def __init__(self, args): + self.max_retry = max(1, _resolve_setting(getattr(args, "judge_max_retry", None), "JUDGE_MAX_RETRY", 3)) + self.concurrency = _resolve_setting(getattr(args, "judge_concurrency", None), "JUDGE_MAX_CONCURRENCY", 4) + self.max_tokens = _resolve_setting(getattr(args, "judge_max_tokens", None), "JUDGE_MAX_TOKENS", 2048) + self.temperature = float(args.judge_temperature) + self.include_reason = bool(args.judge_include_reason) + self.client, self.model = _build_judge_client(args) + if not self.model: + raise ValueError("`--judge_model` is required when a judge stage is enabled.") + + # Process-wide: set to False after a provider rejects the logprobs parameter, + # so we don't pay a failed round-trip per sample on such deployments. + _logprobs_supported = True + + def _chat(self, messages, want_logprobs=False): + """Returns (text, error, logprobs): text is None on failure, error describes + the last problem so callers can surface it instead of silently scoring + 'incorrect'. logprobs is the response token-logprob list (or None) — purely + observational, requested only when want_logprobs and the provider allows it.""" + is_reasoning = bool(re.match(r"\s*(gpt-5|o1|o3|o4)", str(self.model), re.I)) + kwargs = {"model": self.model, "messages": messages} + if is_reasoning: + kwargs["max_completion_tokens"] = self.max_tokens + else: + kwargs["max_tokens"] = self.max_tokens + kwargs["temperature"] = self.temperature + use_logprobs = want_logprobs and _LLMStageBase._logprobs_supported + if use_logprobs: + kwargs["logprobs"] = True + sem = self._get_sem(self.concurrency) + last_error = "no_attempt" + for attempt in range(self.max_retry): + try: + with sem: # cap concurrent API calls; don't hold it during backoff + resp = self.client.chat.completions.create(**kwargs) + text = resp.choices[0].message.content or "" + if text.strip(): + lp = getattr(resp.choices[0], "logprobs", None) + content = getattr(lp, "content", None) if lp is not None else None + return text, None, content + last_error = "empty_response" + except Exception as exc: + last_error = f"{type(exc).__name__}: {exc}" + if not _is_retryable(exc): + if use_logprobs and getattr(exc, "status_code", None) == 400: + # 400 with logprobs requested: some gateways reject the + # parameter outright. Drop it (process-wide) and redo the + # call — the verdict matters more than the observability + # field. Auth/permission errors (401/403) fail fast below. + print(f"[{self.name}] provider rejected logprobs; disabling: {last_error}", flush=True) + _LLMStageBase._logprobs_supported = False + return self._chat(messages, want_logprobs=False) + # auth / bad request / not found — retrying cannot help + print(f"[{self.name}] non-retryable API error: {last_error}", flush=True) + return None, last_error, None + print(f"[{self.name}] attempt {attempt + 1}/{self.max_retry} failed: {last_error}", flush=True) + if attempt + 1 < self.max_retry: + time.sleep(1.5 * (2 ** attempt)) # 1.5s, 3s, 6s exponential backoff + return None, last_error, None + + +class LLMJudgeStage(_LLMStageBase): + """The LLM directly judges whether the response is correct given + QUESTION + MODEL_RESPONSE + GROUND_TRUTH, returning is_correct 0/1. + Both verdicts DECIDE the sample (score 1.0/0.0); only transport/parse + failures pass, carrying an llm_error marker.""" + + name = "llm-judge" + + def run(self, sample: Dict[str, Any], context: Dict[str, Any]) -> StageResult: + pred = normalize_text(context["pred"], extract_boxed=False, strip_latex_commands=False) + gt = normalize_text(context["gt"], extract_boxed=False, strip_latex_commands=False) + question = normalize_text(context["question"], extract_boxed=False, strip_latex_commands=False) + question_type = context["question_type"] + + prompt = self._build_prompt(question_type, question, pred, gt, context) + messages = [ + {"role": "system", "content": "You are a strict evaluator for VLM outputs."}, + {"role": "user", "content": prompt}, + ] + + text, error, logprobs = self._chat(messages, want_logprobs=True) + parsed = self._parse_response(text) if text else None + if parsed is None: + # Transport/parse failure, NOT a "judged incorrect" verdict — the reason + # marker keeps it distinguishable in score.json and in the summary. + detail = error or "unparseable_judge_response" + return StageResult.passed(reason=f"llm_error: {detail}") + + is_correct = int(parsed.get("is_correct", 0)) == 1 + # Observability only: P(verdict token) from API logprobs; never a + # model-self-reported number, never an input to the verdict. + prob = _verdict_probability(logprobs, parsed["is_correct"]) + meta = {"judge_prob": prob} + reason = parsed.get("reason") if self.include_reason else None + if is_correct: + return StageResult.scored(1.0, matched=str(context["gt"]), reason=reason, meta=meta) + return StageResult.scored(0.0, matched=None, reason=reason, meta=meta) + + def _build_prompt(self, question_type, question, pred, gt, context): + options = context.get("option_texts", {}) + options_text = "" + if question_type == "mcq" and options: + rendered = [f"{k}. {v}" for k, v in options.items()] + options_text = "\nOptions:\n" + "\n".join(rendered) + + reason_instruction = "Include a concise reason." if self.include_reason else "Set reason to an empty string." + return ( + "Judge whether MODEL_RESPONSE is correct given QUESTION and GROUND_TRUTH.\n" + f"QuestionType: {question_type}\n" + f"QUESTION: {question}\n" + f"{options_text}\n" + f"MODEL_RESPONSE: {pred}\n" + f"GROUND_TRUTH: {gt}\n\n" + "Return JSON only with schema:\n" + '{"is_correct": 0_or_1, "reason": "string"}\n' + f"{reason_instruction}\n" + "No markdown, no extra fields." + ) + + def _parse_response(self, text): + text = text.strip() + if not text: + return None + try: + return self._normalize_judge_obj(json.loads(text)) + except Exception: + pass + match = re.search(r"\{.*\}", text, flags=re.DOTALL) + if match: + try: + return self._normalize_judge_obj(json.loads(match.group(0))) + except Exception: + pass + # Transport-level recovery from slightly malformed JSON that still carries + # an UNAMBIGUOUS verdict (e.g. some instruct models drop the "reason" key: + # {"is_correct": 1, ""}). This recovers the value the judge already + # produced; it does NOT change how correctness is decided and never + # guesses — only a single, consistent 0/1 (or true/false) is accepted, so + # conflicting or absent verdicts remain llm_errors. + verdicts = {v.lower() for v in re.findall( + r'"is_correct"\s*:\s*(true|false|0|1)\b', text, flags=re.IGNORECASE)} + if len(verdicts) == 1: + v = verdicts.pop() + reason = re.search(r'"reason"\s*:\s*"([^"]*)"', text) + return self._normalize_judge_obj( + {"is_correct": 1 if v in ("1", "true") else 0, + "reason": reason.group(1) if reason else ""}) + return None + + @staticmethod + def _normalize_judge_obj(obj): + if not isinstance(obj, dict): + return None + raw = obj.get("is_correct", 0) + if isinstance(raw, bool): + value = 1 if raw else 0 + else: + try: + value = int(raw) + except Exception: + value = 0 + reason = obj.get("reason", "") + return {"is_correct": 1 if value == 1 else 0, "reason": str(reason)} + + +class LLMMatchStage(_LLMStageBase): + """VLMEvalKit's approach: the LLM only EXTRACTS which option(s) the response + picks, then we rule-based exact-compare against the ground truth. Mirrors + extract_answer_from_item for mcq (rule-based can_infer prefetch, GPT extraction + only as a fallback) and the LogicVista letters extractor for multi-letter gts + (vlmeval/dataset/utils/logicvista.py:15-21, sorted-letters compare :50-68).""" + + name = "llm-match" + + # Modeled on VLMEvalKit's LogicVista extraction instruction (logicvista.py:15-21). + EXTRACT_MULTI_TMPL = ( + 'You are an AI assistant who will help me to match an answer with several ' + 'options of a multiple-selection question. You are provided with a question, ' + 'several options, and an answer, and you need to output which option letters ' + 'the answer selects. Output ONLY the chosen uppercase letters separated by ' + 'commas (e.g. "A, C"). If the answer selects none of the options, output Z.\n' + 'Question: {}?\nOptions: {}\nAnswer: {}\nYour output: ' + ) + + # Verbatim from VLMEvalKit (vlmeval/dataset/utils/multiple_choice.py build_prompt). + EXTRACT_TMPL = ( + 'You are an AI assistant who will help me to match ' + 'an answer with several options of a single-choice question. ' + 'You are provided with a question, several options, and an answer, ' + 'and you need to find which option is most similar to the answer. ' + 'If the meaning of all options are significantly different from the answer, output Z. ' + 'Your should output a single uppercase character in A, B, C, D (if they are valid options), and Z. \n' + 'Example 1: \n' + 'Question: What is the main object in image?\nOptions: A. teddy bear B. rabbit C. cat D. dog\n' + 'Answer: a cute teddy bear\nYour output: A\n' + 'Example 2: \n' + 'Question: What is the main object in image?\nOptions: A. teddy bear B. rabbit C. cat D. dog\n' + 'Answer: Spider\nYour output: Z\n' + 'Example 3: \n' + 'Question: {}?\nOptions: {}\nAnswer: {}\nYour output: ' + ) + + def run(self, sample: Dict[str, Any], context: Dict[str, Any]) -> StageResult: + if context["question_type"] != "mcq": + return StageResult.passed(reason="llm-match supports mcq only") + + pred = normalize_text(context["pred"], extract_boxed=False, strip_latex_commands=False) + question = normalize_text(context["question"], extract_boxed=False, strip_latex_commands=False) + # Drop media placeholders from the text prompt. + question = re.sub(r"<(?:image|video)>", " ", question).strip() + choices = [str(c).strip().upper() for c in (context["options"] or [])] + option_texts = context.get("option_texts", {}) + + # Compact gts only with real parsed options (see exact.py: the A-F + # fallback would read word-like gts as letter sets). + gt_letters = parse_gt_letters(context["gt"]) + if gt_letters is None and context.get("options_parsed"): + gt_letters = extract_letter_set(context["gt"], choices) + if gt_letters is not None and len(gt_letters) > 1: + # Multi-letter gt (LogicVista): rule prefetch, then the letters-extraction + # prompt; official sorted-set compare (utils/logicvista.py:50-68). + got = extract_letter_set(pred, choices) + error = None + if got is None: + rendered = " ".join(f"{k}. {v}" for k, v in option_texts.items()) + prompt = self.EXTRACT_MULTI_TMPL.format(question, rendered, pred) + text, error, _ = self._chat([{"role": "user", "content": prompt}]) + if text: + got = extract_letter_set(text, choices) + if got and got == gt_letters: + return StageResult.scored(1.0, matched=", ".join(gt_letters)) + reason = f"llm_error: {error}" if error else None + return StageResult.passed(reason=reason) + + # Single-letter gt: VLMEvalKit extract_answer_from_item protocol. + gt_letter = (gt_letters[0] if gt_letters else None) or \ + extract_option_robust(context["gt"], choices, option_texts) or \ + normalize_for_exact(context["gt"]).upper() + + opt, error = self._extract_option(question, pred, choices, option_texts) + if opt and gt_letter and opt == gt_letter: + return StageResult.scored(1.0, matched=gt_letter) + reason = f"llm_error: {error}" if error else None + return StageResult.passed(reason=reason) + + def _extract_option(self, question, pred, choices, option_texts): + """Returns (option_or_None, api_error_or_None).""" + # Prefetch with rules (VLMEvalKit can_infer = can_infer_option | can_infer_text). + # Original case, matching VLMEvalKit — uppercasing first would turn every + # article "a" into a hit for option A. + opt = can_infer_option(str(pred), choices) or can_infer_text(str(pred), option_texts) + if opt: + return opt, None + # GPT extraction fallback. + rendered = [f"{k}. {v}" for k, v in option_texts.items()] if option_texts else [] + options_str = " ".join(rendered) + prompt = self.EXTRACT_TMPL.format(question, options_str, pred) + text, error, _ = self._chat([{"role": "user", "content": prompt}]) + if not text: + return None, error + valid = choices + ["Z"] + got = can_infer_option(text.upper(), valid) + if got: + return got, None + # The extractor replies with a bare letter; accept any valid candidate (not + # just A-D — benchmarks with options E/F+ must not fall off this cliff). + m = re.search(rf"\b([{re.escape(''.join(valid))}])\b", text.upper()) + return (m.group(1) if m else None), None + diff --git a/mmeval/scoring/stages/metrics.py b/mmeval/scoring/stages/metrics.py new file mode 100644 index 00000000..22024ebb --- /dev/null +++ b/mmeval/scoring/stages/metrics.py @@ -0,0 +1,126 @@ +"""Fractional grader stages. Both always DECIDE (score 0..1) or mark the +sample invalid — they never pass — so they are only valid as a pipeline's +last stage. is_correct = 1 only at full credit; the official headline +aggregate for fractional protocols is summary.mean_score. + +Protocol sources (implemented 1:1): +- vqa-accuracy: lmms-eval lmms_eval/tasks/vqav2/utils.py:15-42 + (vqav2_process_results) — whitespace-normalize, EvalAIAnswerProcessor + punctuation+digit/article normalization, then the official leave-one-out + consensus accuracy: for each reference answer, acc = min(1, #matching OTHER + references / 3); sample score = mean over references. +- anls: lmms-eval lmms_eval/api/metrics.py:293-321 (anls) — per + reference, normalized Levenshtein distance over lowercased + whitespace-collapsed strings; score = 1 - min distance, zeroed below the + threshold (default 0.5). +""" +from typing import Any, List, Optional, Tuple + +from mmeval.scoring.stages.base import BaseStage, StageResult +from mmeval.scoring.vqa_eval_metric import EvalAIAnswerProcessor + +_VQA_PROCESSOR = EvalAIAnswerProcessor() + + +def _reference_list(gt: Any) -> Optional[List[str]]: + if isinstance(gt, (list, tuple)) and gt: + return [str(x) for x in gt] + if isinstance(gt, str) and gt.strip(): + return [gt] + return None + + +def _vqa_normalize(text: str) -> str: + # lmms-eval vqav2_process_results: \n/\t -> space, strip, then the official + # EvalAI punctuation and digit/article passes. + text = text.replace("\n", " ").replace("\t", " ").strip() + text = _VQA_PROCESSOR.process_punctuation(text) + text = _VQA_PROCESSOR.process_digit_article(text) + return text + + +def vqa_consensus(pred: Any, gt: Any) -> Tuple[Optional[float], Optional[str]]: + """Official VQA consensus accuracy. Returns (score, error): score is None + on error. Requires the multi-annotator reference list (VQAv2/OK-VQA/ + TextVQA/VizWiz ship 10 answers) — the leave-one-out formula is undefined + for a single reference.""" + refs = _reference_list(gt) + if refs is None or len(refs) < 2: + return None, "vqa-accuracy requires the multi-annotator answer list (>=2 references) as gt" + pred_text = pred[0] if isinstance(pred, list) and pred else pred + if not isinstance(pred_text, str): + return None, "vqa-accuracy: prediction is not text" + res = _vqa_normalize(pred_text) + norm_refs = [_vqa_normalize(r) for r in refs] + accs = [] + for i in range(len(norm_refs)): + others = norm_refs[:i] + norm_refs[i + 1:] + matching = sum(1 for o in others if o == res) + accs.append(min(1.0, matching / 3.0)) + return sum(accs) / len(accs), None + + +def _levenshtein(s1: str, s2: str) -> int: + # lmms-eval api/metrics.py levenshtein_distance, verbatim algorithm. + if len(s1) > len(s2): + s1, s2 = s2, s1 + distances = range(len(s1) + 1) + for i2, c2 in enumerate(s2): + distances_ = [i2 + 1] + for i1, c1 in enumerate(s1): + if c1 == c2: + distances_.append(distances[i1]) + else: + distances_.append(1 + min((distances[i1], distances[i1 + 1], distances_[-1]))) + distances = distances_ + return distances[-1] + + +def anls(pred: Any, gt: Any, threshold: float = 0.5) -> Tuple[Optional[float], Optional[str]]: + """ANLS (lmms-eval api/metrics.py:293-321): 1 - min normalized Levenshtein + over the references, zeroed below `threshold`. Accepts a single reference + string or a reference list.""" + refs = _reference_list(gt) + if refs is None: + return None, "anls: empty ground truth" + pred_text = pred[0] if isinstance(pred, list) and pred else pred + if not isinstance(pred_text, str): + return None, "anls: prediction is not text" + det = " ".join(pred_text.strip().lower().split()) + values = [] + for answer in refs: + gt_answer = " ".join(answer.strip().lower().split()) + dist = _levenshtein(gt_answer, det) + length = max(len(answer.upper()), len(pred_text.upper())) + values.append(0.0 if length == 0 else float(dist) / float(length)) + score = 1 - min(values) + if score < threshold: + score = 0.0 + return score, None + + +class VqaAccuracyStage(BaseStage): + """Official VQA consensus accuracy (fractional grader).""" + + name = "vqa-accuracy" + grader = True + + def run(self, sample, context): + score, error = vqa_consensus(context["pred"], context["gt"]) + if score is None: + return StageResult.not_gradable(error) + return StageResult.scored(score) + + +class AnlsStage(BaseStage): + """ANLS similarity against the references (fractional grader).""" + + name = "anls" + grader = True + + def run(self, sample, context): + score, error = anls(context["pred"], context["gt"], + threshold=float(context.get("anls_threshold") or 0.5)) + if score is None: + return StageResult.not_gradable(error) + return StageResult.scored(score) diff --git a/mmeval/scoring/stages/rule_match.py b/mmeval/scoring/stages/rule_match.py new file mode 100644 index 00000000..a3ff7135 --- /dev/null +++ b/mmeval/scoring/stages/rule_match.py @@ -0,0 +1,117 @@ +import re + +from mmeval.scoring.stages.base import BaseStage, StageResult +from mmeval.scoring.stages.exact_match import (gt_references, numbers_match, + numeric_branch_active, open_text_match) +from mmeval.scoring.extraction import ( + extract_letter_set, + extract_option_robust, + extract_yes_no, + normalize_for_exact, + normalize_open_answer, + normalize_text, + parse_gt_letters, + parse_number, +) + +# Answer cues that introduce the model's final value in a chain of thought. +_NUMERIC_CUE = re.compile(r"(?:final answer|the answer is|answer is|answer\s*[:\-]|=)") +# Optional trailing % so extracted percentages keep the official /100 +# semantics (see extraction.parse_number). +_NUMBER_RE = re.compile(r"-?\d+(?:\.\d+)?%?") +_OPEN_ANSWER_RES = ( + re.compile(r"(?:final answer|answer is|the answer is)\s*[:\-]?\s*(.+)$", re.IGNORECASE), + re.compile(r"^\s*(.+?)\s*$", re.IGNORECASE), +) + + +class RuleMatchStage(BaseStage): + """Robust rule-based answer extraction: per-type extraction tuned for + verbose / chain-of-thought output, then deterministic comparison. Still + fully rule-based — no API calls.""" + + name = "rule-match" + + def run(self, sample, context): + if context["gt"] is None or context["pred"] is None: + return StageResult.passed() + + handler = { + "mcq": self._match_mcq, + "yes_no": self._match_yes_no, + "numeric": self._match_numeric, + }.get(context["question_type"], self._match_open) + return handler(context) + + def _match_mcq(self, context): + # Letter-SET grading (single-select = |set|=1; LogicVista multi-letter + # answers compare sorted sets, vlmeval/dataset/utils/logicvista.py:50-68). + pred_raw = context["pred"] + gt_raw = context["gt"] + candidates = context["options"] + option_texts = context.get("option_texts", {}) + + # Compact multi-letter gts ("AC") are accepted at match time (the row + # is already typed mcq) — but only with REAL parsed options: under the + # A-F fallback, word-like gts would parse as letter sets ("BED" -> B,D,E). + gt_letters = parse_gt_letters(gt_raw) + if gt_letters is None and context.get("options_parsed"): + gt_letters = extract_letter_set(gt_raw, candidates) + if gt_letters is None: + g = extract_option_robust(gt_raw, candidates, option_texts) or normalize_for_exact(gt_raw).upper() + gt_letters = (g,) if g else None + + # Clean letters-and-separators segments first ("A, C", "BD", + # "the answers are A and C"); then the robust single-letter extractor + # (VLMEvalKit can_infer + our patches: tags, cues, CJK, \boxed, + # option-text restatement). + pred_letters = extract_letter_set(pred_raw, candidates) + if pred_letters is None: + option = extract_option_robust(pred_raw, candidates, option_texts) + pred_letters = (option,) if option else None + + if gt_letters and pred_letters and pred_letters == gt_letters: + return StageResult.scored(1.0, matched=", ".join(gt_letters)) + return StageResult.passed() + + def _match_yes_no(self, context): + # VLMEvalKit YOrN_Extraction parity (vlmeval/dataset/utils/yorn.py:254-261): + # word-level yes XOR no; a response containing both defers to the next + # stage (upstream marks it 'Unknown'). + gt_norm = normalize_open_answer(context["gt"]) + pred_yn = extract_yes_no(context["pred"]) + if pred_yn is not None and pred_yn == gt_norm: + return StageResult.scored(1.0, matched=pred_yn) + return StageResult.passed() + + def _match_numeric(self, context): + if not numeric_branch_active(context): + # e.g. OCRBench: the declared non-exact string comparator is the + # official rule for digit-string answers too (see exact_match.py). + return self._match_open(context) + gt_nums = [n for n in (parse_number(r) for r in gt_references(context["gt"])) if n is not None] + if not gt_nums: + return StageResult.passed() + pred_norm = normalize_open_answer(context["pred"]) + # Prefer the segment after the LAST answer cue (a CoT usually walks + # through intermediate numbers first); fall back to the whole text. + cues = list(_NUMERIC_CUE.finditer(pred_norm)) + segment = pred_norm[cues[-1].end():] if cues else pred_norm + nums = _NUMBER_RE.findall(segment) or _NUMBER_RE.findall(pred_norm) + pred_num = parse_number(nums[0]) if nums else None + if pred_num is not None and any(numbers_match(pred_num, g, context) for g in gt_nums): + return StageResult.scored(1.0, matched=str(context["gt"])) + return StageResult.passed() + + def _match_open(self, context): + pred_norm_text = normalize_text(context["pred"]) + + # generic "final answer is ..." extraction, then whole-string restatement; + # each candidate compared per the protocol's string_match (exact/contains/anls). + for pattern in _OPEN_ANSWER_RES: + m = pattern.search(pred_norm_text) + if not m: + continue + if open_text_match(m.group(1), m.group(1), context["gt"], context): + return StageResult.scored(1.0, matched=str(context["gt"])) + return StageResult.passed() diff --git a/mmeval/scoring/vqa_eval_metric.py b/mmeval/scoring/vqa_eval_metric.py new file mode 100644 index 00000000..a885657a --- /dev/null +++ b/mmeval/scoring/vqa_eval_metric.py @@ -0,0 +1,218 @@ +# Verbatim copy of lmms-eval's EvalAIAnswerProcessor +# (lmms_eval/tasks/_task_utils/vqa_eval_metric.py @ EvolvingLMMs-Lab/lmms-eval, +# itself copied from facebookresearch/mmf pythia/tasks/processors.py#L897) — +# the OFFICIAL VQA-accuracy answer normalization. Do not edit: protocol fidelity +# requires the exact contraction/digit/punctuation tables. +import re + + +class EvalAIAnswerProcessor: + """ + Processes an answer similar to Eval AI + copied from + https://github.com/facebookresearch/mmf/blob/c46b3b3391275b4181567db80943473a89ab98ab/pythia/tasks/processors.py#L897 + """ + + CONTRACTIONS = { + "aint": "ain't", + "arent": "aren't", + "cant": "can't", + "couldve": "could've", + "couldnt": "couldn't", + "couldn'tve": "couldn't've", + "couldnt've": "couldn't've", + "didnt": "didn't", + "doesnt": "doesn't", + "dont": "don't", + "hadnt": "hadn't", + "hadnt've": "hadn't've", + "hadn'tve": "hadn't've", + "hasnt": "hasn't", + "havent": "haven't", + "hed": "he'd", + "hed've": "he'd've", + "he'dve": "he'd've", + "hes": "he's", + "howd": "how'd", + "howll": "how'll", + "hows": "how's", + "Id've": "I'd've", + "I'dve": "I'd've", + "Im": "I'm", + "Ive": "I've", + "isnt": "isn't", + "itd": "it'd", + "itd've": "it'd've", + "it'dve": "it'd've", + "itll": "it'll", + "let's": "let's", + "maam": "ma'am", + "mightnt": "mightn't", + "mightnt've": "mightn't've", + "mightn'tve": "mightn't've", + "mightve": "might've", + "mustnt": "mustn't", + "mustve": "must've", + "neednt": "needn't", + "notve": "not've", + "oclock": "o'clock", + "oughtnt": "oughtn't", + "ow's'at": "'ow's'at", + "'ows'at": "'ow's'at", + "'ow'sat": "'ow's'at", + "shant": "shan't", + "shed've": "she'd've", + "she'dve": "she'd've", + "she's": "she's", + "shouldve": "should've", + "shouldnt": "shouldn't", + "shouldnt've": "shouldn't've", + "shouldn'tve": "shouldn't've", + "somebody'd": "somebodyd", + "somebodyd've": "somebody'd've", + "somebody'dve": "somebody'd've", + "somebodyll": "somebody'll", + "somebodys": "somebody's", + "someoned": "someone'd", + "someoned've": "someone'd've", + "someone'dve": "someone'd've", + "someonell": "someone'll", + "someones": "someone's", + "somethingd": "something'd", + "somethingd've": "something'd've", + "something'dve": "something'd've", + "somethingll": "something'll", + "thats": "that's", + "thered": "there'd", + "thered've": "there'd've", + "there'dve": "there'd've", + "therere": "there're", + "theres": "there's", + "theyd": "they'd", + "theyd've": "they'd've", + "they'dve": "they'd've", + "theyll": "they'll", + "theyre": "they're", + "theyve": "they've", + "twas": "'twas", + "wasnt": "wasn't", + "wed've": "we'd've", + "we'dve": "we'd've", + "weve": "we've", + "werent": "weren't", + "whatll": "what'll", + "whatre": "what're", + "whats": "what's", + "whatve": "what've", + "whens": "when's", + "whered": "where'd", + "wheres": "where's", + "whereve": "where've", + "whod": "who'd", + "whod've": "who'd've", + "who'dve": "who'd've", + "wholl": "who'll", + "whos": "who's", + "whove": "who've", + "whyll": "why'll", + "whyre": "why're", + "whys": "why's", + "wont": "won't", + "wouldve": "would've", + "wouldnt": "wouldn't", + "wouldnt've": "wouldn't've", + "wouldn'tve": "wouldn't've", + "yall": "y'all", + "yall'll": "y'all'll", + "y'allll": "y'all'll", + "yall'd've": "y'all'd've", + "y'alld've": "y'all'd've", + "y'all'dve": "y'all'd've", + "youd": "you'd", + "youd've": "you'd've", + "you'dve": "you'd've", + "youll": "you'll", + "youre": "you're", + "youve": "you've", + } + + NUMBER_MAP = { + "none": "0", + "zero": "0", + "one": "1", + "two": "2", + "three": "3", + "four": "4", + "five": "5", + "six": "6", + "seven": "7", + "eight": "8", + "nine": "9", + "ten": "10", + } + ARTICLES = ["a", "an", "the"] + PERIOD_STRIP = re.compile(r"(?!<=\d)(\.)(?!\d)") + COMMA_STRIP = re.compile(r"(?<=\d)(\,)+(?=\d)") + PUNCTUATIONS = [ + ";", + r"/", + "[", + "]", + '"', + "{", + "}", + "(", + ")", + "=", + "+", + "\\", + "_", + "-", + ">", + "<", + "@", + "`", + ",", + "?", + "!", + ] + + def __init__(self, *args, **kwargs): + pass + + def word_tokenize(self, word): + word = word.lower() + word = word.replace(",", "").replace("?", "").replace("'s", " 's") + return word.strip() + + def process_punctuation(self, in_text): + out_text = in_text + for p in self.PUNCTUATIONS: + if (p + " " in in_text or " " + p in in_text) or (re.search(self.COMMA_STRIP, in_text) is not None): + out_text = out_text.replace(p, "") + else: + out_text = out_text.replace(p, " ") + out_text = self.PERIOD_STRIP.sub("", out_text, re.UNICODE) + return out_text + + def process_digit_article(self, in_text): + out_text = [] + temp_text = in_text.lower().split() + for word in temp_text: + word = self.NUMBER_MAP.setdefault(word, word) + if word not in self.ARTICLES: + out_text.append(word) + else: + pass + for word_id, word in enumerate(out_text): + if word in self.CONTRACTIONS: + out_text[word_id] = self.CONTRACTIONS[word] + out_text = " ".join(out_text) + return out_text + + def __call__(self, item): + item = self.word_tokenize(item) + item = item.replace("\n", " ").replace("\t", " ").strip() + item = self.process_punctuation(item) + item = self.process_digit_article(item) + return item diff --git a/mmeval/utils/argparser.py b/mmeval/utils/argparser.py index d0f83a54..e4c621c1 100644 --- a/mmeval/utils/argparser.py +++ b/mmeval/utils/argparser.py @@ -1,5 +1,7 @@ import argparse import inspect +import os +import sys import warnings from dataclasses import dataclass, field, fields from typing import Dict, Optional, Sequence, get_args @@ -7,6 +9,7 @@ @dataclass class ModelArguments: model_name_or_path: Optional[str] = field(default=None) + model_series: Optional[str] = field(default=None) # parameters for model low_cpu_mem_usage: Optional[bool] = field(default=None, metadata={"help": "Tries to not use more than 1x model size in CPU memory (including peak memory) while loading the model."}) @@ -14,6 +17,8 @@ class ModelArguments: # parameters for model inference dtype: Optional[str] = field(default=None, metadata={"help": "Override the default torch.dtype and load the model under a specific dtype."}) + min_pixels: Optional[int] = field(default=None, metadata={"help": "Image processor min_pixels override (Qwen-VL family); None keeps the runner's default."}) + max_pixels: Optional[int] = field(default=None, metadata={"help": "Image processor max_pixels override (Qwen-VL family); None keeps the runner's default."}) device_map: Optional[str] = field(default=None, metadata={"help": "A map that specifies where each submodule should go."}) # parameters that control the length of the output @@ -84,7 +89,48 @@ class ExperimentArguments: no_conda: bool = field(default=False, metadata={"help": "use current python env instead of conda"}) -ARGUMENT_DATACLASSES = (ModelArguments, DataArguments, InferenceArguments, ExperimentArguments) +@dataclass +class ScoreRuntimeArguments: + # Score-mode CLI convention: every dest is prefixed `score_` (scoring + # pipeline/IO/knobs) or `judge_` (the LLM-judge client group — already + # score-mode-only and self-descriptive; `score_judge_*` would stutter). + # No unprefixed or infer-colliding dests. + score_out_dir: Optional[str] = field(default=None, metadata={"help": "output directory containing result.json files"}) + score_parallel_per_task: int = field(default=4, metadata={"help": "number of sample workers inside each result.json"}) + + +@dataclass +class ScoreArguments: + score_result_glob: str = field(default="**/result.json", metadata={"help": "glob pattern for result files in score mode"}) + score_output_name: str = field(default="score.json", metadata={"help": "output score json file name"}) + score_progress_bar: bool = field(default=True, metadata={"help": "show progress bar while scoring samples"}) + score_resume: bool = field(default=True, metadata={"help": "resume scoring from score tmp/final files when available"}) + score_save_freq: int = field(default=20, metadata={"help": "flush frequency for score resume tmp file"}) + score_debug: bool = field(default=False, metadata={"help": "write per-stage trace into score outputs"}) + score_pipeline: str = field(default="exact-match,rule-match", metadata={"help": "ordered scoring pipeline: comma-separated atomic stage names (exact-match, rule-match, llm-match, llm-judge, vqa-accuracy, anls)"}) + + score_gt_field: str = field(default="answer", metadata={"help": "field name for ground-truth in result sample"}) + score_pred_field: str = field(default="messages[-1].response", metadata={"help": "field path for prediction in result sample"}) + score_question_type: str = field(default="auto", metadata={"help": "question type: auto|mcq|yes_no|numeric|open. auto = read the sample's question_type field (recognized values only), then infer from gt shape"}) + score_numeric_rel_tol: float = field(default=0.0, metadata={"help": "relative tolerance for numeric answers (0 = exact float equality)"}) + score_numeric_abs_tol: float = field(default=0.0, metadata={"help": "absolute tolerance for numeric answers (0 = off; DynaMath uses 0.001)"}) + score_string_match: str = field(default="exact", metadata={"help": "rule-chain text comparison: exact|contains|anls (contains = OCRBench substring protocol; anls = threshold ANLS)"}) + score_anls_threshold: float = field(default=0.5, metadata={"help": "ANLS threshold (string_match=anls and the anls grader)"}) + + judge_provider: Optional[str] = field(default=None, metadata={"help": "llm judge provider: local (in-process open-weight text LLM, loaded as a causal LM) | openai | azure_openai"}) + judge_model: Optional[str] = field(default=None, metadata={"help": "llm judge model/deployment name; for local, an open-weight text LLM HF id (e.g. Qwen/Qwen2.5-7B-Instruct)"}) + # For the three knobs below, resolution order is: CLI flag > env var > default + # (env names JUDGE_MAX_RETRY / JUDGE_MAX_CONCURRENCY / JUDGE_MAX_TOKENS). + judge_max_retry: Optional[int] = field(default=None, metadata={"help": "llm judge max attempts per call (default: env JUDGE_MAX_RETRY or 3)"}) + judge_temperature: float = field(default=0.0, metadata={"help": "llm judge generation temperature"}) + judge_concurrency: Optional[int] = field(default=None, metadata={"help": "max concurrent llm judge calls, process-wide (default: env JUDGE_MAX_CONCURRENCY or 4)"}) + judge_max_tokens: Optional[int] = field(default=None, metadata={"help": "llm judge max completion tokens (default: env JUDGE_MAX_TOKENS or 2048)"}) + judge_include_reason: bool = field(default=False, metadata={"help": "include llm judge reason in output"}) + + +ARGUMENT_DATACLASSES = (ModelArguments, DataArguments, InferenceArguments, ExperimentArguments, ScoreArguments) +INFER_ARGUMENT_DATACLASSES = (ModelArguments, DataArguments, InferenceArguments, ExperimentArguments) +SCORE_ARGUMENT_DATACLASSES = (ScoreRuntimeArguments, ScoreArguments) BOOL_DEFAULTS = { @@ -100,9 +146,8 @@ class ExperimentArguments: ) } -def parse_args(): - parser = argparse.ArgumentParser() - for dc in ARGUMENT_DATACLASSES: +def _add_dataclass_arguments(parser, dataclasses, suppress_defaults=False): + for dc in dataclasses: for f in fields(dc): tp = f.type type_args = getattr(tp, '__args__', None) @@ -114,23 +159,57 @@ def parse_args(): if f.default is True: parser.add_argument(f"--no-{cli_name}", f"--no_{f.name}", dest=f.name, action="store_false", - default=True, help=help_text) + default=argparse.SUPPRESS if suppress_defaults else True, + help=help_text) elif f.default is False: parser.add_argument(f"--{cli_name}", f"--{f.name}", dest=f.name, action="store_true", - default=False, help=help_text) + default=argparse.SUPPRESS if suppress_defaults else False, + help=help_text) else: parser.add_argument(f"--{cli_name}", f"--{f.name}", dest=f.name, action="store_true", - default=None, help=help_text) + default=argparse.SUPPRESS if suppress_defaults else None, + help=help_text) parser.add_argument(f"--no-{cli_name}", f"--no_{f.name}", - dest=f.name, action="store_false") + dest=f.name, action="store_false", + **({"default": argparse.SUPPRESS} if suppress_defaults else {})) else: parser.add_argument(f"--{cli_name}", f"--{f.name}", dest=f.name, type=tp, - default=f.default, help=help_text) - return parser.parse_args() + default=argparse.SUPPRESS if suppress_defaults else f.default, + help=help_text) + + +def _track_explicit_flags(args, argv): + """Record which score-mode dests the user explicitly provided (shadow parse + with suppressed defaults). The scorer's per-knob resolution needs this: + explicit CLI flag > dataset_meta from result.json > built-in default.""" + shadow = argparse.ArgumentParser() + _add_dataclass_arguments(shadow, SCORE_ARGUMENT_DATACLASSES, suppress_defaults=True) + args._explicit_flags = frozenset(vars(shadow.parse_known_args(argv)[0]).keys()) + return args + +def parse_args(mode: str = "auto"): + mode = (mode or "auto").strip().lower() + if mode == "auto": + script_name = os.path.basename(sys.argv[0]) + mode = "score" if script_name == "score.py" else "infer" + + if mode == "infer": + dataclasses = INFER_ARGUMENT_DATACLASSES + elif mode == "score": + dataclasses = SCORE_ARGUMENT_DATACLASSES + else: + raise ValueError(f"Unsupported parse mode: {mode}. Expected infer|score|auto.") + + parser = argparse.ArgumentParser() + _add_dataclass_arguments(parser, dataclasses) + args = parser.parse_args() + if mode == "score": + args = _track_explicit_flags(args, sys.argv[1:]) + return args def parse_model_kwargs(args, default_kwargs=None): default_kwargs = default_kwargs or {} model_kwargs = {} diff --git a/scripts/examples/evalkit_dataset.sh b/scripts/examples/evalkit_dataset.sh index 7e637d1f..69319a2b 100644 --- a/scripts/examples/evalkit_dataset.sh +++ b/scripts/examples/evalkit_dataset.sh @@ -1,7 +1,22 @@ export PYTHONPATH=./:$PYTHONPATH export DATASET_DIR=./datasets -# LLaVABench +# Inference examples (produce result.json) for several VLMEvalKit datasets. +# LLaVABench is additionally SCORED here with an open-weight local judge: its +# official protocol is the `llm-judge` path, and the `local` provider runs it +# with no API key (see run_score_evalkit_llm.sh / llm_judge_api_reference.md for +# the API judge). The reference answer is in the `gpt4_ans` field, so scoring +# passes `--score_gt_field gpt4_ans`. +# +# IMPORTANT — this is a DEGRADED approximation, not LLaVABench's official number: +# the official metric is a GPT-graded RELATIVE score (assistant vs reference on a +# 1-10 scale, per category, with the image caption given to the judge), whereas +# our llm-judge emits a BINARY correctness verdict and the caption is not fed to +# the judge. Treat the committed accuracy as a per-sample binary rate under an +# open judge, not as the official LLaVABench score. See docs/en/OPEN_JUDGE.md. +# The other datasets below are inference-only. + +# LLaVABench — inference python mmeval/run.py \ --dataset evalkit@LLaVABench \ --out_dir work_dirs/examples/evalkit_dataset/LLaVABench \ @@ -9,6 +24,23 @@ python mmeval/run.py \ --gpu_per_parallel 1 \ --parallel_per_task 1 +# LLaVABench — score with a local open-weight judge (deterministic, greedy) +python mmeval/score.py \ + --score_out_dir work_dirs/examples/evalkit_dataset/LLaVABench \ + --score_pipeline llm-judge \ + --score_gt_field gpt4_ans \ + --judge_provider local --judge_model Qwen/Qwen2.5-7B-Instruct + +# Fixture gate: a committed judge fixture must be a genuine graded run. +python3 - <<'PY' +import json +s = json.load(open("work_dirs/examples/evalkit_dataset/LLaVABench/score.json"))["summary"] +assert s["invalid"] == 0, f"fixture has {s['invalid']} invalid samples" +assert s["llm_errors"] == 0, f"fixture has {s['llm_errors']} llm_errors" +print(f"LLaVABench fixture OK: {s['total']} samples, invalid=0, " + f"llm_errors=0, accuracy={s['accuracy']:.2f}") +PY + # MicroBench python mmeval/run.py \ --model_name_or_path Qwen/Qwen3-VL-2B-Instruct \ diff --git a/scripts/examples/hf_dataset.sh b/scripts/examples/hf_dataset.sh index bdcbc341..649cd238 100644 --- a/scripts/examples/hf_dataset.sh +++ b/scripts/examples/hf_dataset.sh @@ -1,9 +1,37 @@ export PYTHONPATH=./:$PYTHONPATH +# +# The committed outputs under work_dirs/examples/hf_dataset/ are exactly what +# this script produces: a deterministic 20-sample subset of the MMBench dev +# split (dev carries answers; the test split withholds them), selected with +# the framework's native sampling flags — see docs/en/USAGE.md "Run a subset +# of samples". python mmeval/run.py \ --model_name_or_path Qwen/Qwen3-VL-2B-Instruct \ - --dataset mmeval_hf@mm-eval/MMBench-en-V11 \ - --split test \ + --dataset mmeval_hf@mm-eval/MMBench-V11 \ + --subset en \ + --split dev \ + --sample_num 20 \ + --sample_order random \ + --sample_seed 42 \ --out_dir work_dirs/examples/hf_dataset/MMBench_en_V11 \ --gpu_per_parallel 1 \ - --parallel_per_task 1 \ No newline at end of file + --parallel_per_task 1 + +# Score the run. The explicit rule pipeline keeps this credential-free +# (MMBench's declared pipeline includes an LLM judge stage, which needs API +# keys); the override is recorded in score.json's knob_sources. +python mmeval/score.py \ + --score_out_dir work_dirs/examples/hf_dataset/MMBench_en_V11 \ + --score_result_glob 'result.json' \ + --score_pipeline exact-match,rule-match \ + --no_score_resume + +# Fixture gate: every sample must be gradable — a fixture whose rows are all +# invalid would exercise nothing but the missing-gt path. +python3 - <<'PY' +import json +s = json.load(open("work_dirs/examples/hf_dataset/MMBench_en_V11/score.json"))["summary"] +assert s["invalid"] == 0, f"fixture has {s['invalid']} invalid samples" +print(f"fixture OK: {s['total']} samples, invalid=0, accuracy={s['accuracy']:.2f}") +PY diff --git a/scripts/examples/local_dataset.sh b/scripts/examples/local_dataset.sh index 24cb8434..963762fe 100644 --- a/scripts/examples/local_dataset.sh +++ b/scripts/examples/local_dataset.sh @@ -1,5 +1,11 @@ export PYTHONPATH=./:$PYTHONPATH +# Inference-only example. These inputs are modality / text smoke samples +# (tests/samples/*.json) that carry NO ground truth, so scoring them would be +# 100% invalid and verify nothing — no score.json fixture is committed and the +# `invalid == 0` fixture gate (see hf_dataset.sh) does not apply here. The +# committed result.json shows the inference output format for local@json. + python mmeval/run.py \ --model_name_or_path Qwen/Qwen3-VL-2B-Instruct \ --dataset local@json \ diff --git a/scripts/examples/prompt_template.sh b/scripts/examples/prompt_template.sh index 8da791da..11afcce8 100644 --- a/scripts/examples/prompt_template.sh +++ b/scripts/examples/prompt_template.sh @@ -1,4 +1,13 @@ export PYTHONPATH=./:$PYTHONPATH +# +# NOTE on committed example outputs: the result.json/score.json checked into +# work_dirs/examples/ are the outputs of these commands as written — the HF +# example (Test 4) selects a deterministic 20-sample subset of the answered +# dev split with the framework's native sampling flags +# (--sample_num/--sample_order/--sample_seed, see docs/en/USAGE.md "Run a +# subset of samples"), scores it, and asserts every sample is gradable. +# Score the local-JSON outputs (Tests 1-2) with: +# PYTHONPATH=. python3 mmeval/score.py --score_out_dir --no_score_resume # Test 1: Local JSON with template file path python mmeval/run.py \ @@ -34,13 +43,32 @@ python mmeval/run.py \ --gpu_per_parallel 1 \ --parallel_per_task 1 -# Test 4: HuggingFace MMBench-en-V11 +# Test 4: HuggingFace MMBench-V11 (en subset), 20-sample subset via native +# sampling. The dev split carries answers (test withholds them), so the +# committed fixture exercises real grading. python mmeval/run.py \ --model_name_or_path Qwen/Qwen3-VL-2B-Instruct \ - --dataset mmeval_hf@mm-eval/MMBench-en-V11 \ - --split test \ + --dataset mmeval_hf@mm-eval/MMBench-V11 \ + --subset en \ + --split dev \ + --sample_num 20 \ + --sample_order random \ + --sample_seed 42 \ --out_dir work_dirs/examples/prompt_template/hf_template \ --gpu_per_parallel 1 \ --parallel_per_task 1 +# Score Test 4 and gate the fixture: every sample must be gradable. +python mmeval/score.py \ + --score_out_dir work_dirs/examples/prompt_template/hf_template \ + --score_result_glob 'result.json' \ + --score_pipeline exact-match,rule-match \ + --no_score_resume +python3 - <<'PY' +import json +s = json.load(open("work_dirs/examples/prompt_template/hf_template/score.json"))["summary"] +assert s["invalid"] == 0, f"fixture has {s['invalid']} invalid samples" +print(f"fixture OK: {s['total']} samples, invalid=0, accuracy={s['accuracy']:.2f}") +PY + diff --git a/scripts/templates/vlmevalkit_mcq.txt b/scripts/templates/vlmevalkit_mcq.txt new file mode 100644 index 00000000..0e2f142a --- /dev/null +++ b/scripts/templates/vlmevalkit_mcq.txt @@ -0,0 +1,5 @@ +{{ question }}{% if options %} +Options: +{% for k, v in options.items() %}{{ k }}. {{ v }} +{% endfor %}{% endif %} +Please select the correct answer from the options above. \ No newline at end of file diff --git a/work_dirs/examples/evalkit_dataset/LLaVABench/score.json b/work_dirs/examples/evalkit_dataset/LLaVABench/score.json new file mode 100644 index 00000000..df6e73d1 --- /dev/null +++ b/work_dirs/examples/evalkit_dataset/LLaVABench/score.json @@ -0,0 +1,910 @@ +{ + "summary": { + "total": 60, + "valid": 60, + "correct": 20, + "accuracy": 0.3333333333333333, + "accuracy_valid": 0.3333333333333333, + "mean_score": 0.3333333333333333, + "invalid": 0, + "llm_errors": 0, + "by_stage": { + "llm-judge": { + "total": 60, + "correct": 20 + } + }, + "by_question_type": { + "open": { + "total": 60, + "correct": 20 + } + }, + "judge_prob_mean": 0.9942218726982717 + }, + "config": { + "dataset_name": null, + "subset": null, + "split": null, + "score_pipeline": [ + "llm-judge" + ], + "score_gt_field": "gpt4_ans", + "score_pred_field": "messages[-1].response", + "score_question_type": "auto", + "score_numeric_rel_tol": 0.0, + "score_numeric_abs_tol": 0.0, + "score_string_match": "exact", + "score_anls_threshold": 0.5, + "judge_provider": "local", + "judge_model": "Qwen/Qwen2.5-7B-Instruct", + "judge_temperature": 0.0, + "judge_include_reason": false, + "judge_max_tokens": 2048, + "judge_model_revision": "a09a35458c702b33eeacc393d103063234e8bc28", + "task_type": null, + "official_protocol": null, + "knob_sources": { + "score_pipeline": "cli", + "score_numeric_rel_tol": "default", + "score_numeric_abs_tol": "default", + "score_string_match": "default", + "score_anls_threshold": "default" + }, + "score_progress_bar": true, + "score_resume": true, + "score_save_freq": 20, + "score_parallel_per_task": 4, + "resumed_count": 0, + "provenance": { + "git_commit": null, + "python": "3.10.20", + "openai": null + }, + "protocol_notes": [ + "Per-sample scoring; official aggregations noted per dataset in these notes are not reproduced. LLM failures are marked wrong and counted in summary.llm_errors (never randomly guessed, unlike VLMEvalKit)." + ] + }, + "samples": [ + { + "eval-id": 0, + "question_type": "open", + "gt": "The famous sight in the photo is Diamond Head.", + "pred": [ + "This is the famous landmark known as the Diamond Head, a volcanic crater located in Honolulu, Hawaii. It is a popular tourist attraction and a key feature of the city's skyline." + ], + "decided_by": "llm-judge", + "matched": "The famous sight in the photo is Diamond Head.", + "score": 1.0, + "is_correct": 1, + "status": "ok", + "judge_prob": 0.9999998807907247 + }, + { + "eval-id": 1, + "question_type": "open", + "gt": "This photo features an aerial view of Diamond Head, located in the Hawaiian Islands. The volcanic tuff cone appears prominent and impressive, with its ridges and slopes covered in lush green vegetation. The crater of Diamond Head is clearly visible, surrounded by the steep walls of the structure. In the background, you can see the vast expanse of the ocean with varying shades of blue, showcasing the pristine waters surrounding the island.", + "pred": [ + "This is an aerial photograph capturing a dramatic and scenic coastal landscape, likely taken from a high vantage point on a mountain or cliff overlooking the ocean. The image is rich in detail and color, showcasing the contrast between the rugged landmass and the vibrant sea.\n\nThe central focus is a large, steep, and rugged mountain that dominates the mid-ground. Its surface is a mix of dark, earthy tones\u2014browns, tans, and dark reds\u2014indicating a heavily eroded volcanic rock. The mountain has a distinct, steep-sided crater or bowl shape, with a large, prominent crater that appears to be the result" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 0.9999998807907247 + }, + { + "eval-id": 2, + "question_type": "open", + "gt": "Diamond Head is a volcanic cone formed by the eruption of a volcano. The formation of this sight can be attributed to volcanic activity, where magma from the Earth's mantle rises and breaks through the Earth's crust, creating eruptions. Over time, the eruptions build up layers of volcanic ash and rock, which solidify and form this distinct cone-shaped structure. The erosion caused by wind, rain, and waves can also shape the landscape, contributing to its current appearance.", + "pred": [ + "The image you've provided shows the iconic **Dolphin's Back**, a natural landform located in the **Hawaiian Islands**. This is a classic example of a **volcanic crater** formed by the collapse of a volcano.\n\nHere are the possible reasons for the formation of this sight:\n\n---\n\n### 1. **Volcanic Origin**\n- The most likely cause is that this is a **volcanic crater** formed during the eruption of a volcano.\n- The area is part of the **Hawaiian Islands**, which are formed by the slow movement of the Pacific Plate over a hotspot (a region of" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 0.9993774000965375 + }, + { + "eval-id": 3, + "question_type": "open", + "gt": "Title: Aloha from Hawaii: Our Unforgettable Adventure at Diamond Head and Beyond!\n\nDay 1: Exploring the Cultural Riches of the Island\n\nUpon our arrival in the breathtaking Hawaiian Islands, we couldn't wait to start exploring the myriad cultural experiences that awaited us. We started by visiting the renowned 'Iolani Palace, witnessing the rich history that exudes from the architecture and artifacts. Next, we strolled through the quirky and art-filled streets of Kaka'ako, a colorful neighborhood with a welcoming atmosphere.\n\nDay 2: A Journey to Diamond Head\n\nWaking up early to the sound of crashing waves, we eagerly prepared for today's main event: hiking Diamond Head. This iconic volcanic cone is a must-see attraction in the Hawaiian Islands. As we made our ascent, we marveled at the majestic aerial view of Diamond Head, showcasing its vast expanse and lush, green beauty. The jaw-dropping views from the summit, featuring a stunning panorama of Waikiki and endless ocean vistas, were undoubtedly worth the effort.\n\nDay 3: Cultural Immersion and Culinary Delights\n\nOur next day was devoted to experiencing Hawaiian culture and flavors. We joined a luau and were enthralled by the captivating stories told through traditional music and dance. The feast which followed satisfied our taste buds with mouth-watering dishes like Kalua pig, poi, and luscious haupia, an authentic coconut dessert.\n\nDay 4: Discovering the Island's Natural Wonders\n\nOur travel would not be complete without exploring Oahu's other natural gems. The ethereal Hanauma Bay wowed us with its vibrant marine life and crystal-clear waters, offering an unparalleled snorkeling experience. We rounded off our day with a picturesque hike at Manoa Falls, where towering trees and a magnificent waterfall welcomed us.\n\nDay 5: Time to Say Aloha!\n\nOur Hawaiian adventure was unforgettable - from the breathtaking summit at Diamond Head and the aquatic paradise of Hanauma Bay to the cultural heartbeat pulsating within Kaka'ako and the delectable luau feast. If you haven't visited the Hawaiian Islands, what are you waiting for? Paradise is calling!\n\nAloha!", + "pred": [ + "### **A Journey to the Heart of Hawaii: Exploring the Majestic Haleakal\u0101 and the Cultural Tapestry of Maui**\n\n**By [Your Name] | May 2024**\n\nIf you're a fan of breathtaking landscapes, vibrant culture, and a truly unforgettable experience, then your dream destination might just be the island of Maui, Hawaii. This is not just a vacation; it's a journey into a world where nature and tradition harmoniously coexist. I had the privilege of visiting Maui this past spring, and the experience was nothing short of magical. From the awe-inspiring Haleakal\u0101 volcano to the" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 1.0 + }, + { + "eval-id": 4, + "question_type": "open", + "gt": "This is a mangosteen fruit.", + "pred": [ + "Based on the visual characteristics in the image, this is a **mangosteen**.\n\nHere are the key features that identify it:\n\n- **Appearance:** The fruit has a deep, dark purple or maroon skin.\n- **Stem:** It has a distinctive green, leaf-like cap at the top, which is the fruit's calyx.\n- **Internal Structure:** The fruit is cut in half, revealing a very white, fleshy, and star-shaped interior. This is the edible flesh, which is known for its sweet and slightly tangy taste.\n- **Texture:** The flesh is described as juicy and soft," + ], + "decided_by": "llm-judge", + "matched": "This is a mangosteen fruit.", + "score": 1.0, + "is_correct": 1, + "status": "ok", + "judge_prob": 1.0 + }, + { + "eval-id": 5, + "question_type": "open", + "gt": "There are three uncut fruits in the image.", + "pred": [ + "Based on the image provided, we can count the uncut fruits by identifying the whole, intact fruits.\n\n- There are three whole mangosteens with their green tops still attached. They are located in the upper left, middle, and lower right portions of the image.\n- There is one cut mangosteen in the foreground, showing its white, star-shaped interior. This is not an uncut fruit.\n- There is one cut mangosteen on the right side, also showing its white interior. This is also not an uncut fruit.\n- The other fruit is the one in the lower right, which is cut in half and" + ], + "decided_by": "llm-judge", + "matched": "There are three uncut fruits in the image.", + "score": 1.0, + "is_correct": 1, + "status": "ok", + "judge_prob": 0.9981844070129096 + }, + { + "eval-id": 6, + "question_type": "open", + "gt": "This photo features four mangosteens on a wooden table. Three of them are uncut, showing their round, dark purple exterior. The fourth mangosteen is cut open, revealing its white, fleshy interior, which is divided into segments. The table surface has a natural wood grain pattern and adds a rustic touch to the image.", + "pred": [ + "This is a detailed, close-up photograph of several mangosteens (also known as mangosteens or mangosteens) arranged on a rustic wooden surface. The image is composed to highlight the fruit's unique and appealing characteristics.\n\nThe central focus is on the mangosteens, which are shown in various states of presentation:\n- **Whole fruits:** Several whole mangosteens are visible, each with a deep, dark purple to almost black skin. The skin is smooth and has a slight sheen, and the top of each fruit features a distinctive, leaf-like crown. This crown is composed of a cluster of green, fleshy" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 0.9999597088831099 + }, + { + "eval-id": 7, + "question_type": "open", + "gt": "The fragrance of the mangosteens in the image can be described as sweet and slightly floral, with a hint of citrus aroma. It is a delicate and pleasant smell that entices you to try the fruit.", + "pred": [ + "The scent of the mangosteen is a delightful and unique blend of tropical freshness and a hint of subtle sweetness, which is truly unforgettable. If you were to describe it to someone who has never had this fruit before, you might say:\n\n\"Imagine a crisp, clean aroma of fresh citrus and sweet, tropical fruit, like a blend of lemon and lime, but with a hint of a slightly sweet, almost floral note. It\u2019s not overpowering, but rather delicate and refreshing, like the scent of a warm, sunlit orchard. This fragrance is so distinct and pleasant that it\u2019s almost like a burst of sunshine on a warm" + ], + "decided_by": "llm-judge", + "matched": "The fragrance of the mangosteens in the image can be described as sweet and slightly floral, with a hint of citrus aroma. It is a delicate and pleasant smell that entices you to try the fruit.", + "score": 1.0, + "is_correct": 1, + "status": "ok", + "judge_prob": 0.945686706476785 + }, + { + "eval-id": 8, + "question_type": "open", + "gt": "This painting is a creative fusion of the famous Mona Lisa and a dog. The dog is dressed as the Mona Lisa, with a similar facial expression and hairstyle as the iconic figure. The canine's fur seamlessly blends with the clothing, imitating the original painting's style. The background features a landscape with gentle rolling hills and a hazy sky, just like in the original Mona Lisa artwork. The artist has skillfully combined the two subjects into one, giving the piece a whimsical and amusing quality.", + "pred": [ + "This is a digitally altered image that is a parody of the famous Mona Lisa, but with a dog instead of a woman. The image is a portrait of a long-haired, brown dog, likely a dachshund or a similar breed, dressed in a manner that mimics the Renaissance-era fashion of the Mona Lisa. The dog is depicted with a human-like expression, with large, dark, and expressive eyes that look directly at the viewer. Its fur is a rich brown color, and it has a long, flowing coat with a slightly shaggy texture.\n\nThe dog is wearing a dark, draped garment that resembles the famous Mona" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 0.9994471746722718 + }, + { + "eval-id": 9, + "question_type": "open", + "gt": "The intended effect of this painting might be to create a humorous and whimsical twist on the famous Mona Lisa artwork by featuring a dog dressed in her attire, which can evoke feelings of amusement and curiosity.", + "pred": [ + "This painting is a humorous and creative reinterpretation of the iconic \"Mona Lisa,\" but with a canine subject. The intended effect is to create a playful, absurd, and entertaining piece of art that combines the classic elements of the original painting with a modern, unexpected twist.\n\nHere are the key aspects of the intended effect:\n\n- **Absurdity and Satire**: By replacing the human figure with a dog, the artist subverts the traditional narrative of the Mona Lisa. The dog's expression, though it appears to be a \"smile,\" is more like a neutral, slightly sad or confused look, which is a perfect contrast to" + ], + "decided_by": "llm-judge", + "matched": "The intended effect of this painting might be to create a humorous and whimsical twist on the famous Mona Lisa artwork by featuring a dog dressed in her attire, which can evoke feelings of amusement and curiosity.", + "score": 1.0, + "is_correct": 1, + "status": "ok", + "judge_prob": 0.9997293489777506 + }, + { + "eval-id": 10, + "question_type": "open", + "gt": "This creative twist on the classic Mona Lisa painting, featuring a dog dressed as the iconic figure, can be interpreted differently by various audiences. Art enthusiasts may view this as an interesting fusion of popular culture and classical art, appreciating the creativity and humor behind the concept. Animal lovers, on the other hand, might be delighted to see a dog take center stage in such a famous piece of art, showcasing the artist's love and appreciation for animals.\n\nFor those with a fondness for satire, the image can be seen as poking fun at the seriousness often associated with classical works while simultaneously paying homage to Leonardo da Vinci's masterpiece. Others might interpret this image as a commentary on how iconic images and art can be reimagined and reinterpreted over time, reflecting the ever-evolving nature of culture and society. Regardless of the audience, this playful adaptation of the Mona Lisa serves as a conversation-starter and offers a fresh perspective on a timeless classic.", + "pred": [ + "This image presents a striking and humorous reinterpretation of Leonardo da Vinci's *Mona Lisa*, but its meaning is likely to be interpreted differently by various audiences based on their perspectives, cultural backgrounds, and personal experiences.\n\nFor many, especially those familiar with the original artwork, the image is a playful and affectionate parody. The central figure, a dog with a human-like expression and a classic pose, is a clear homage to the Mona Lisa's enigmatic smile and the artistic style of the Renaissance. The dog's gaze, with its soft, slightly melancholic eyes, mirrors the Mona Lisa's subtle, enigmatic smile. This twist is" + ], + "decided_by": "llm-judge", + "matched": "This creative twist on the classic Mona Lisa painting, featuring a dog dressed as the iconic figure, can be interpreted differently by various audiences. Art enthusiasts may view this as an interesting fusion of popular culture and classical art, appreciating the creativity and humor behind the concept. Animal lovers, on the other hand, might be delighted to see a dog take center stage in such a famous piece of art, showcasing the artist's love and appreciation for animals.\n\nFor those with a fondness for satire, the image can be seen as poking fun at the seriousness often associated with classical works while simultaneously paying homage to Leonardo da Vinci's masterpiece. Others might interpret this image as a commentary on how iconic images and art can be reimagined and reinterpreted over time, reflecting the ever-evolving nature of culture and society. Regardless of the audience, this playful adaptation of the Mona Lisa serves as a conversation-starter and offers a fresh perspective on a timeless classic.", + "score": 1.0, + "is_correct": 1, + "status": "ok", + "judge_prob": 0.9974071433150283 + }, + { + "eval-id": 11, + "question_type": "open", + "gt": "The man in the photo is Elon Musk.", + "pred": [ + "The man in the photo is Elon Musk, the CEO of Tesla, SpaceX, and other companies." + ], + "decided_by": "llm-judge", + "matched": "The man in the photo is Elon Musk.", + "score": 1.0, + "is_correct": 1, + "status": "ok", + "judge_prob": 0.9999991655356624 + }, + { + "eval-id": 12, + "question_type": "open", + "gt": "The meme is parodying an iconic scene from The Lion King, where Rafiki holds up Simba to all the other animals while standing on a mountaintop.", + "pred": [ + "This meme is a parody of the iconic scene from the 1999 Disney animated film *The Lion King*.\n\nThe scene depicted is where the character Simba, a young lion, is on the Pride Rock, looking up at the sky and holding a small, golden lion cub (often referred to as \"the cub\") in his arms. The cub is the same one that is being held by the character, which is a reference to the \"Doge\" meme, a popular internet meme featuring a cartoon dog with a human face.\n\nThe meme is a clever blend of two different cultural references: the *The Lion King*" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 0.9999749666753994 + }, + { + "eval-id": 13, + "question_type": "open", + "gt": "This meme reflects Elon Musk's public image and actions by humorously comparing him to the iconic scene from The Lion King. It suggests that he is presenting a new and valuable \"creation\" to the world, much like Rafiki holding up Simba. In the context of Elon Musk, this could allude to his innovative ventures in technology and entrepreneurship, along with his eccentric and playful personality that often captures attention and generates discussions. The dog in the image might also be a nod to his frequent engagement with internet culture and his tendency to influence trends, including those involving cryptocurrencies like Dogecoin.", + "pred": [ + "This meme reflects a blend of satire, internet culture, and commentary on Elon Musk's public image, personality, and actions through the juxtaposition of his public persona with a fictional, comedic character.\n\nHere's a breakdown of how it comments on him:\n\n- **Satire of the \"Doge\" Culture**: The central figure is a dog with the face of a Doge, a popular internet meme that combines the image of a dog with the face of a meme character. The meme uses this to satirize Musk's often exaggerated public persona. The Doge face is a symbol of internet humor, and its use here suggests Musk's" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 1.0 + }, + { + "eval-id": 14, + "question_type": "open", + "gt": "The meme features a dog lying on a cyan wooden floor, looking exhausted or unenthusiastic. The text at the top of the meme reads \"MONDAY,\" and at the bottom, it says \"JUST...MONDAY.\" The meme humorously captures the feeling of a lack of motivation and energy that is often associated with Mondays, represented by the dog's pose and expression.", + "pred": [ + "Of course. Here is a detailed explanation of the meme.\n\nThis image is a classic internet meme that uses a simple, relatable scene to convey a common feeling of exhaustion or dread. It is a well-known \"just...monday\" meme, popularized on platforms like Reddit and Imgflip.\n\n### The Visual Elements\n\nThe image is composed of a photograph and text overlaid on it.\n\n- **The Subject:** The central figure is a small, light-brown or cream-colored French Bulldog (also known as a Frenchie). The dog is lying on its back on a wooden deck. Its body is relaxed and it appears" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 0.9999988079084972 + }, + { + "eval-id": 15, + "question_type": "open", + "gt": "Someone could express the same sentiment as this meme by saying things like \"Mondays, am I right?\" or \"Struggling to get through Monday\" or even using a hashtag like #MondayBlues. Visual alternatives might include a person dragging themselves out of bed or showing a gloomy face with the text \"Monday Mood.\"", + "pred": [ + "The meme \"Monday. Just... Monday.\" expresses the common sentiment of feeling overwhelmed, stressed, or drained by the start of the week. Here are several other ways someone might express the same sentiment:\n\n- **\"I'm not even sure I'm awake anymore. It's like I've been in a time loop.\"** This conveys a sense of mental fatigue and disorientation, similar to the dog's state of exhaustion.\n\n- **\"This week is just a blur of tasks and deadlines. I'm not even sure what I'm supposed to do.\"** This expresses the feeling of being overwhelmed by the volume of responsibilities.\n\n- **" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 0.9999997615814777 + }, + { + "eval-id": 16, + "question_type": "open", + "gt": "Yes, the Mona Lisa was painted by the famous Italian artist, Leonardo da Vinci.", + "pred": [ + "Yes, the painting you're referring to is *The Mona Lisa* by Leonardo da Vinci. It is one of the most famous artworks in the world, created in the early 16th century." + ], + "decided_by": "llm-judge", + "matched": "Yes, the Mona Lisa was painted by the famous Italian artist, Leonardo da Vinci.", + "score": 1.0, + "is_correct": 1, + "status": "ok", + "judge_prob": 0.9999998807907247 + }, + { + "eval-id": 17, + "question_type": "open", + "gt": "This painting, the Mona Lisa, features a woman with a serene and enigmatic smile. She has long, dark hair that flows down onto her shoulders, and her skin tone is fair. Her attire consists of a blue and green garment. In the background, there is a landscape with a winding river, trees, mountains, and a distant bridge, imbuing the scene with depth and misty atmosphere. The gentle softness of the forms, the delicate colors, and the subtle transitions between light and shadow are specific to the sfumato technique used by the artist Leonardo da Vinci.", + "pred": [ + "This is a detailed description of the painting, which is a famous work by the Italian Renaissance artist Leonardo da Vinci.\n\nThe painting is a portrait of a woman, commonly known as the Mona Lisa, painted in the early 1500s. The artwork is renowned for its enigmatic expression, the subtle use of light and shadow, and its innovative use of perspective.\n\n**Subject and Composition:**\nThe central figure is a woman with long, dark brown hair that falls over her shoulders and frames her face. Her gaze is directed slightly to the right of the viewer, creating a sense of engagement and mystery. Her facial expression is one" + ], + "decided_by": "llm-judge", + "matched": "This painting, the Mona Lisa, features a woman with a serene and enigmatic smile. She has long, dark hair that flows down onto her shoulders, and her skin tone is fair. Her attire consists of a blue and green garment. In the background, there is a landscape with a winding river, trees, mountains, and a distant bridge, imbuing the scene with depth and misty atmosphere. The gentle softness of the forms, the delicate colors, and the subtle transitions between light and shadow are specific to the sfumato technique used by the artist Leonardo da Vinci.", + "score": 1.0, + "is_correct": 1, + "status": "ok", + "judge_prob": 0.9693021123577238 + }, + { + "eval-id": 18, + "question_type": "open", + "gt": "The Mona Lisa, painted by Leonardo da Vinci in the early 16th century, has had a significant historical impact on the art world. As a masterpiece of the Italian Renaissance, it showcases da Vinci's exceptional talents in both painting techniques and psychological insight.\n\nThe painting is notable for its use of sfumato, a technique that blends colors together without stark lines, creating a more realistic and softened appearance. This technique was groundbreaking at the time and has since inspired countless artists.\n\nAdditionally, the Mona Lisa's enigmatic expression and the subject's gaze, which seems to follow the viewer, have captured the imaginations of viewers for centuries. This has led to extensive analysis and discussion of the portrait, making it a widely recognized symbol of not only the Italian Renaissance but also of the broader history of Western art.\n\nThe Mona Lisa has also been a source of fascination for its mysterious history, including theft, vandalism, and conspiracy theories. These events have added to the painting's allure and contributed to its iconic status in popular culture.\n\nIn summary, the historical impact and significance of the Mona Lisa can be attributed to its revolutionary techniques, enigmatic subject, and storied history, all of which have cemented its place as a crucial work in the art world.", + "pred": [ + "The painting you've shown is *The Mona Lisa*, created by Leonardo da Vinci in the early 1500s. It is one of the most famous and influential works in the history of art. Here's a discussion of its historical impact and significance.\n\n### Historical Impact\n\n1. **Revolution in Artistic Technique**:\n - *The Mona Lisa* is a masterclass in the use of sfumato, a technique that involves the delicate blending of tones to create a soft, almost invisible transition between colors and shadows. This technique allowed da Vinci to create a sense of depth and realism that was groundbreaking at the time" + ], + "decided_by": "llm-judge", + "matched": "The Mona Lisa, painted by Leonardo da Vinci in the early 16th century, has had a significant historical impact on the art world. As a masterpiece of the Italian Renaissance, it showcases da Vinci's exceptional talents in both painting techniques and psychological insight.\n\nThe painting is notable for its use of sfumato, a technique that blends colors together without stark lines, creating a more realistic and softened appearance. This technique was groundbreaking at the time and has since inspired countless artists.\n\nAdditionally, the Mona Lisa's enigmatic expression and the subject's gaze, which seems to follow the viewer, have captured the imaginations of viewers for centuries. This has led to extensive analysis and discussion of the portrait, making it a widely recognized symbol of not only the Italian Renaissance but also of the broader history of Western art.\n\nThe Mona Lisa has also been a source of fascination for its mysterious history, including theft, vandalism, and conspiracy theories. These events have added to the painting's allure and contributed to its iconic status in popular culture.\n\nIn summary, the historical impact and significance of the Mona Lisa can be attributed to its revolutionary techniques, enigmatic subject, and storied history, all of which have cemented its place as a crucial work in the art world.", + "score": 1.0, + "is_correct": 1, + "status": "ok", + "judge_prob": 0.9693021123577238 + }, + { + "eval-id": 19, + "question_type": "open", + "gt": "In this photo, the Space Needle, a famous observation tower, is prominently displayed against a clear blue sky in the background. The structure's futuristic design features an elongated, slender column supporting a flying saucer-like top with observation decks and antennas. The overall atmosphere is bright and sunny, showcasing the iconic Seattle landmark in all its glory.", + "pred": [ + "This is a detailed photograph of the iconic Seattle Space Needle, a famous landmark in Seattle, Washington. The image is taken from a low angle, looking up towards the top of the structure, which emphasizes its height and architectural design.\n\nThe Space Needle is the central focus of the image. It is a white, modern skyscraper with a distinctive saucer-shaped top. The top section is a wide, circular platform with a flat, white surface. A red, white, and blue American flag is flying from the top of the structure. The main body of the needle is a tall, slender, white tower with three vertical support legs, which" + ], + "decided_by": "llm-judge", + "matched": "In this photo, the Space Needle, a famous observation tower, is prominently displayed against a clear blue sky in the background. The structure's futuristic design features an elongated, slender column supporting a flying saucer-like top with observation decks and antennas. The overall atmosphere is bright and sunny, showcasing the iconic Seattle landmark in all its glory.", + "score": 1.0, + "is_correct": 1, + "status": "ok", + "judge_prob": 0.998387794420291 + }, + { + "eval-id": 20, + "question_type": "open", + "gt": "The best weather for visiting the Space Needle would be during clear or sunny days with minimal clouds, which allows for better visibility and enjoyment of the views. The best seasons would be late spring or summer when the weather is warmer and more predictable. The best time of day would be during late mornings or early afternoons when natural light is at its peak, making it ideal for sightseeing and photography.\n\nThe image shows the Space Needle with a clear blue sky in the background, which indicates that the photo was taken during a beautiful day with great weather. This would be a good time to visit the Space Needle.", + "pred": [ + "Based on the image provided, here is an analysis of the best weather, season, time of day, and whether the photo was taken at a good time to visit the Space Needle.\n\n### Weather and Season\nThe weather appears to be **sunny and pleasant**. The sky is a clear, bright blue with no clouds, and the lighting is bright and even, suggesting a sunny day. This type of weather is ideal for outdoor activities and sightseeing.\n\nThe season is likely **late spring or summer**. This is inferred from the lush green foliage on the trees and the overall vibrant appearance of the city. While the photo was taken" + ], + "decided_by": "llm-judge", + "matched": "The best weather for visiting the Space Needle would be during clear or sunny days with minimal clouds, which allows for better visibility and enjoyment of the views. The best seasons would be late spring or summer when the weather is warmer and more predictable. The best time of day would be during late mornings or early afternoons when natural light is at its peak, making it ideal for sightseeing and photography.\n\nThe image shows the Space Needle with a clear blue sky in the background, which indicates that the photo was taken during a beautiful day with great weather. This would be a good time to visit the Space Needle.", + "score": 1.0, + "is_correct": 1, + "status": "ok", + "judge_prob": 0.9999891520726072 + }, + { + "eval-id": 21, + "question_type": "open", + "gt": "The name of the character in the image is Conan Edogawa.", + "pred": [ + "The character in the image is Conan Edogawa, a fictional detective from the anime and manga series *Detective Conan* (also known as *Case Closed*). He is known for his sharp intellect and keen observational skills, often solving complex mysteries." + ], + "decided_by": "llm-judge", + "matched": "The name of the character in the image is Conan Edogawa.", + "score": 1.0, + "is_correct": 1, + "status": "ok", + "judge_prob": 1.0 + }, + { + "eval-id": 22, + "question_type": "open", + "gt": "Conan Edogawa appears to be a confident and intelligent character, as indicated by his focused gaze and slightly raised eyebrows. His big eyes and the charming, youthful visage make him more approachable and appealing to audiences of various ages. The iconic glasses and bow tie add a touch of quirkiness and sophistication to his design, making him stand out as a unique character. These aspects of his design may have contributed to his popularity among fans of Japanese cartoons.", + "pred": [ + "Based on the image provided, the character is **Kaito Kid**, a central figure in the anime and manga series *Case Closed* (also known as *Detective Conan*). The image shows him in his signature attire, which is a key element of his character design.\n\nHere is an analysis of his personality and the design elements that contributed to his popularity:\n\n### Personality\n\nKaito Kid is a complex and multifaceted character whose personality is defined by a combination of traits:\n\n- **Intense Focus and Determination**: Kaito is driven by a singular purpose: to solve crimes and bring justice to the world." + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 1.0 + }, + { + "eval-id": 23, + "question_type": "open", + "gt": "When visiting this serene lakeside location, be cautious about the following:\n\n1. Wooden pier safety: Ensure the T-shaped wooden pier is sturdy and safe to walk on, and be mindful of any loose planks or slippery surfaces.\n\n2. Swimming conditions: If you plan to swim in the lake, check for potential hazards such as submerged objects or strong underwater currents, and be aware of the water temperature.\n\n3. Wildlife: Some lakes may have local wildlife, such as snakes or alligators, which could pose a threat. Remain cautious and observant, especially near the water's edge.\n\n4. Weather conditions: The sky is cloudy, so be prepared for potential rainfall or changes in temperature, and monitor weather forecasts before your visit.\n\n5. Hiking in the surrounding area: If you plan to explore the green trees and mountains in the background, be cautious of potential hazards like steep and slippery terrain or unstable trail conditions. Carry appropriate hiking gear and wear sturdy footwear.\n\n6. Protect the environment: Be respectful of the natural beauty of the area, and avoid littering, damaging any trees or plants, or disturbing wildlife.", + "pred": [ + "Based on the image provided, this appears to be a serene and scenic location, likely a lake or a calm body of water in a mountainous or forested region. While the image itself is beautiful and peaceful, it's important to be cautious about the potential risks and considerations when visiting such a place. Here are some things to be mindful of:\n\n---\n\n### 1. **Weather and Environmental Conditions**\n- **Extreme Weather:** The landscape is surrounded by mountains and forests, which can be exposed to sudden weather changes. Even on a seemingly calm day, there could be a sudden storm, heavy rain, or snowfall. The sky in" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 1.0 + }, + { + "eval-id": 24, + "question_type": "open", + "gt": "As a photographer looking to capture the essence of this location, I would choose early morning or late afternoon during the golden hour, and with slightly cloudy or clear weather conditions. The reasons behind my choice are:\n\n1. Golden hour light: Early morning or late afternoon light is soft, warm, and directional, which tends to be flattering for landscapes. It can help to bring out the vibrant colors of the green trees and the mountains, add depth to the scene, and create interesting shadows and highlights.\n\n2. Slightly cloudy or clear weather: These conditions can allow for a more dynamic and visually interesting sky, providing a beautiful backdrop that complements the serene scene. Slightly cloudy weather can also help to diffuse the sunlight, creating a softer light, which can enhance the overall mood of the image.\n\n3. Calm water: Capturing the scene when the lake's surface is calm will create reflections, adding visual interest and making the scene even more picturesque.\n\n4. Emphasize the T-shaped wooden pier: By shooting at an angle that makes the pier the main focal point, it will draw the viewer's attention and evoke a feeling of peacefulness and tranquility.\n\nBy choosing the right time of day and weather conditions, a photographer can effectively capture the serene essence of this location, highlighting its natural beauty and creating a visually captivating image.", + "pred": [ + "If I were a photographer aiming to capture the essence of this location, I would choose **early morning on a calm, overcast day**. Here\u2019s why:\n\n- **Time of Day: Early Morning (around 6:30 AM)**\n - The soft, diffused light of early morning is ideal for this scene. It avoids the harsh shadows and intense sunlight of midday, creating a gentle, even illumination that enhances the natural colors and textures of the landscape. The light is also less likely to create glare on the water, allowing for a clearer view of the wooden dock and the surrounding environment.\n - The air" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 0.975647791945821 + }, + { + "eval-id": 25, + "question_type": "open", + "gt": "The image features a man standing on the rear bumper of a yellow taxi on a busy city street. He is ironing a blue T-shirt on a foldable ironing table attached to the back of the taxi. The man is dressed in a yellow sweater, matching the taxi's color. Another yellow taxi is visible on the left side of the scene. The street is surrounded by tall buildings, and visible street lights have red flags attached to them.", + "pred": [ + "This is a candid, humorous photograph taken on a city street, likely in New York City, capturing a man in an unconventional and amusing situation.\n\nThe central focus is a man with a bald head and glasses, wearing a bright yellow long-sleeved shirt. He is standing on the rear bumper of a yellow SUV, which appears to be a Ford Explorer or similar model, with a silver lower bumper. The vehicle is parked on a street, and the man is actively ironing a blue piece of clothing, which looks like a shirt or a towel, onto a small, makeshift ironing board. This board is constructed from a metal folding" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 1.0 + }, + { + "eval-id": 26, + "question_type": "open", + "gt": "The unusual aspect of this image is that a man is standing on the rear bumper of a yellow taxi, using a foldable ironing table attached to the back of the taxi to iron a blue T-shirt. This is not a typical sight on a city street.", + "pred": [ + "The image is unusual because it depicts a man ironing clothes on the back of a yellow taxi, which is parked on the street. This is not a typical scene of someone ironing at home or in a laundry room, but rather an unexpected and unconventional situation. The man is using a portable ironing board, which is placed on the back of the taxi, and is ironing a blue shirt. The yellow taxi is parked on the street, and there are other vehicles and buildings in the background. The image is likely a humorous or ironic depiction of a man who has taken his ironing to the streets, possibly as a form of" + ], + "decided_by": "llm-judge", + "matched": "The unusual aspect of this image is that a man is standing on the rear bumper of a yellow taxi, using a foldable ironing table attached to the back of the taxi to iron a blue T-shirt. This is not a typical sight on a city street.", + "score": 1.0, + "is_correct": 1, + "status": "ok", + "judge_prob": 0.9828358314220359 + }, + { + "eval-id": 27, + "question_type": "open", + "gt": "In the left part of the fridge, there is a plastic box of strawberries.", + "pred": [ + "Based on the image provided, the fruit in the left part of the refrigerator is **strawberries**.\n\nThey are located in a clear plastic container on the lower left side of the fridge, next to a bag of carrots. The strawberries are red and appear to be fresh." + ], + "decided_by": "llm-judge", + "matched": "In the left part of the fridge, there is a plastic box of strawberries.", + "score": 1.0, + "is_correct": 1, + "status": "ok", + "judge_prob": 0.999993205116773 + }, + { + "eval-id": 28, + "question_type": "open", + "gt": "The brand of the blueberry-flavored yogurt is Fage.", + "pred": [ + "Based on the image provided, the brand of the blueberry-flavored yogurt is **Fay-eh!**.\n\nThis is visible on the label of the white tub of yogurt in the refrigerator, which is labeled \"GRADE A\" and \"Fay-eh!\"." + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 0.9995092178991526 + }, + { + "eval-id": 29, + "question_type": "open", + "gt": "There is no strawberry-flavored yogurt in the fridge. There is a large bottle of Fage non-fat yogurt, a smaller cup of Fage blueberry yogurt, and another smaller cup with an unknown brand and flavor.", + "pred": [ + "Based on a careful examination of the image provided, there is no strawberry-flavored yogurt in the fridge.\n\nHere is a breakdown of the contents visible in the refrigerator:\n\n- **In the foreground:** A clear plastic container holds several strawberries.\n- **On the right side:** There is a container of yogurt. The label is clearly visible and reads:\n - **\"All Natural\"**\n - **\"Tropical Blueberry\"**\n - **\"Green Stewed Yogurt\"**\n - The flavor is explicitly **blueberry**, not strawberry.\n- **In the background:** There is a large" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 0.9958322430461789 + }, + { + "eval-id": 30, + "question_type": "open", + "gt": "The image shows an open refrigerator filled with various food items. In the left part of the compartment, you can see a plastic box of strawberries with a small bag of baby carrots on top, and towards the back, there is a stack of sauce containers. In the middle of the compartment, there is a green plastic box towards the front, with an unidentified plastic bag resting on it. A carton of milk is located towards the back of this section. \n\nOn the right side of the compartment, there is a box of blueberries with three yogurts stacked on top. The large bottle of yogurt is Fage non-fat yogurt, and one of the smaller cups is Fage blueberry yogurt. The brand and flavor of the other smaller cup are unknown. Towards the back of this section, there is a container with an unknown content.", + "pred": [ + "This is a detailed description of the image:\n\nThe photograph captures the interior of a refrigerator, focusing on the shelves filled with various food items. The scene is well-lit, likely by the refrigerator's internal lighting, which highlights the contents.\n\n- **Left Side:** The leftmost shelf holds a clear plastic container filled with fresh strawberries. The strawberries are a mix of red and green, and they are arranged in a neat, circular pattern. Above the container of strawberries, there is a colorful bag of baby carrots, which are orange and wrapped in a purple and yellow plastic bag. The bag is slightly crumpled.\n\n- **Center:**" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 0.9999996423722521 + }, + { + "eval-id": 31, + "question_type": "open", + "gt": "With the available ingredients, you could potentially make:\n\n1. A fruit salad using the strawberries and blueberries.\n2. A yogurt parfait by layering the Fage non-fat yogurt, blueberry yogurt, and fresh fruit (strawberries and blueberries).\n3. A smoothie using the milk, strawberries, blueberries, and yogurt. You could add other ingredients if you have them, like a banana or some spinach.\n4. A healthy snack by enjoying the baby carrots with a dip, such as hummus or ranch, if you have them in the sauce containers.\n\nIt's difficult to suggest more specific meals without knowing the contents of the unidentified plastic bag, green plastic box, and the unknown container.", + "pred": [ + "Based on the items visible in the refrigerator, you have a variety of ingredients that can be used to cook a range of meals. Here's a breakdown of what you can make with these ingredients:\n\n### **Protein & Carbohydrates**\n- **Protein:** This is a strong foundation for many meals.\n - **Chicken or Turkey:** You can use the chicken or turkey from the frozen package (likely a frozen chicken breast or turkey cutlet) to make a simple stir-fry, a chicken and vegetable stew, or a chicken and vegetable bake.\n - **Tuna or Salmon:** The tuna is a great source of protein" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 1.0 + }, + { + "eval-id": 32, + "question_type": "open", + "gt": "There are three coffee mugs in the set.", + "pred": [ + "Based on the image provided, we can count the coffee mugs.\n\n- There are three mugs visible in the image, arranged in a row.\n- The mug on the far left is partially cut off by the frame.\n- The mug in the middle is the most clearly focused.\n- The mug on the right is also clearly visible.\n\nTherefore, there are **3** coffee mugs in the set." + ], + "decided_by": "llm-judge", + "matched": "There are three coffee mugs in the set.", + "score": 1.0, + "is_correct": 1, + "status": "ok", + "judge_prob": 0.9999997615814777 + }, + { + "eval-id": 33, + "question_type": "open", + "gt": "Introducing our delightful set of three identical coffee mugs, featuring the iconic and beloved character, Mario! These vibrant, ceramic mugs boast a high-quality design that playfully pays homage to the world-renowned plumber and hero of the Mushroom Kingdom. Wake up to a nostalgic cup of coffee or hot cocoa and add a touch of Super Mario magic to your kitchen with these charming mugs. Perfect as a gift for both collectors and fans alike, these mugs are sure to level up your beverage game! So, power up your mornings with everyone's favorite gaming legend and get your set of three Mario coffee mugs today!", + "pred": [ + "### **The Ultimate Mario Mug: A Celebration of Fun and Flavor**\n\nStep into the world of endless adventures with the **Mario Mug**\u2014a vibrant, high-quality ceramic mug that brings the magic of the Mushroom Kingdom to your daily coffee ritual! Designed for the ultimate gaming and coffee lovers, this mug is more than just a drinkware item\u2014it's a piece of fun, a nostalgic tribute, and a stylish addition to any kitchen or office.\n\n---\n\n### **Why the Mario Mug Stands Out**\n\n- **Iconic Design, Realistic Detail**: Every mug features a stunning, high-resolution print of Mario in his classic red cap and blue" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 0.9999952316511552 + }, + { + "eval-id": 34, + "question_type": "open", + "gt": "As an AI visual assistant, I cannot display or provide a full recipe. However, from the image I can tell you the main ingredients and components of this dish. It consists of cooked lobsters covered with melted butter, minced garlic, rosemary, and parsley. There are also lemon slices placed on the tray. To make this dish, you would need to prepare these ingredients and cook the lobsters according to a recipe you find.", + "pred": [ + "Of course! Based on the image provided, this is a delicious and visually appealing dish of baked lobster tails. Here is a detailed recipe for this dish.\n\n---\n\n### **Baked Lemon Herb Lobster Tails**\n\nThis recipe is a simple, elegant way to prepare lobster tails, highlighting their rich flavor with a bright, zesty lemon and fresh herbs. The dish is best enjoyed as a main course, but it's also delicious as a side dish or a hearty appetizer.\n\n#### **Ingredients**\n\n- **For the lobster tails:**\n - 1 (12-14 oz) lobster tail, preferably from a fresh," + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 0.9999997615814777 + }, + { + "eval-id": 35, + "question_type": "open", + "gt": "This meme features an image of an oven tray with fried chicken nuggets arranged in the shape of the continents on a world map, resembling a bird's-eye view of Earth. Above the image, there is a caption that reads, \"Sometimes I just look at pictures of the Earth from space and I marvel at how beautiful it all is.\" The humor in this meme comes from the juxtaposition of the earnest caption with the amusing representation of Earth using chicken nuggets.", + "pred": [ + "Of course! This meme is a clever and humorous take on a common internet joke, using a visual metaphor to create a funny and relatable scenario.\n\nHere is a detailed breakdown of the meme:\n\n### The Visual: The \"World\" of Fried Chicken\n- The image shows a baking sheet with several pieces of breaded and fried chicken, likely chicken nuggets or nugget-style chicken.\n- The pieces are arranged on the sheet to form a recognizable silhouette of the world map.\n- The \"countries\" are formed by the shapes of the chicken pieces:\n - The **United States** is represented by the two large, elongated pieces" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 1.0 + }, + { + "eval-id": 36, + "question_type": "open", + "gt": "The two machine learning concepts mentioned in the meme are \"statistical learning\" and \"neural networks.\"", + "pred": [ + "Based on the image provided, the two machine learning concepts mentioned are:\n\n- **Statistical Learning**: This is the top section of the meme, where a stick figure is explaining that people are overgeneralizing because the VC-Dimension of the kernel is too high. This is a concept from the theory of statistical learning, which deals with the generalization ability of models.\n\n- **Neural Networks**: This is the bottom section of the meme, where a stick figure is advocating for \"stack more layers\" to solve the problem. Neural networks are a type of machine learning model that are based on artificial neurons, and stacking more layers is a" + ], + "decided_by": "llm-judge", + "matched": "The two machine learning concepts mentioned in the meme are \"statistical learning\" and \"neural networks.\"", + "score": 1.0, + "is_correct": 1, + "status": "ok", + "judge_prob": 0.9999946356109105 + }, + { + "eval-id": 37, + "question_type": "open", + "gt": "This meme consists of two distinct parts placed vertically. In the top half, the caption reads \"statistical learning,\" and there's an image of a person standing in front of a whiteboard with a concerned expression. On the whiteboard, there's a plot showing a red curve gradually decreasing. The person's chat bubble reads: \"People, our learner overgeneralizes because the VC-Dimension of our Kernel is too high. Get some experts and minimize the structural risk in a new one. Rework our loss function, make the next kernel stable, unbiased, and consider using a soft margin.\"\n\nIn the bottom half of the image, the caption reads \"neural networks,\" and there's another person standing in front of a whiteboard with a happy expression. This time, the whiteboard displays a plot with a green curve, with both x and y axes labeled \"LAYERS.\" The person's chat bubble reads: \"STACK MORE LAYERS.\" \n\nAt the bottom of the meme, a text caption says: \"But unironically.\" This implies that stacking more layers is genuinely considered a useful approach in neural networks, in contrast to the complex instructions given in the statistical learning section.", + "pred": [ + "This meme is a satirical and humorous take on the challenges and misconceptions in machine learning, specifically comparing two different approaches to model complexity and generalization.\n\nThe meme is composed of two panels, each with a stick figure representing a different approach to machine learning, and a chart or diagram illustrating the underlying concept.\n\n---\n\n### Panel 1: The \"Statistical Learning\" Approach\n\n- **Title:** `STATISTICAL LEARNING`\n- **Figure:** A stick figure with an angry expression, representing a traditional machine learning approach.\n- **Text in speech bubble:** \n - \"People, our learner overgeneralizes because the VC-D" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 1.0 + }, + { + "eval-id": 38, + "question_type": "open", + "gt": "This meme is funny because it highlights the difference in complexity and jargon between two machine learning approaches: statistical learning and neural networks.\n\nIn the top half, the meme features a person discussing statistical learning. They use technical terms, such as \"VC-Dimension,\" \"kernel,\" \"structural risk,\" and \"soft margin.\" The text and the person's concerned expression suggest that this approach requires a deep understanding of various concepts, making it seem complicated and challenging.\n\nIn contrast, the bottom half shows a person with a happy expression, talking about neural networks. The advice given here is simply to \"stack more layers,\" referring to adding layers to a neural network model to improve its performance. This part of the meme pokes fun at the perceived simplicity of neural networks compared to statistical learning.\n\nThe humor comes from the exaggerated simplicity of the neural networks approach compared to the complex jargon of statistical learning, implying that neural networks practitioners might not need the same level of expertise as those working with statistical learning methods. The \"but unironically\" text suggests that this is a genuine observation made in the field, adding to the humor.", + "pred": [ + "Of course. This comic is funny because it uses a clever, layered joke about the history and evolution of machine learning, playing on the contrast between two different approaches to solving problems. Let's break it down step by step.\n\n### Step 1: The Setup - The Two Approaches\n\nThe comic presents two distinct methods for solving a problem, which are presented as \"statistical learning\" and \"neural networks.\"\n\n- **Top Panel: Statistical Learning**\n - The speaker is a stick figure with a serious, frustrated expression.\n - The topic is \"statistical learning,\" which is a well-known field in machine learning.\n " + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 1.0 + }, + { + "eval-id": 39, + "question_type": "open", + "gt": "Panel 1: This panel features a smartphone that resembles an iPhone, with a white VGA cable plugged into its charging port. The VGA connector is blue. The phone has a glass screen protector and is encased in a protective case.\n\nPanel 2: This panel shows a product package containing a VGA connector similar to the one in the first panel. The brand is ELECOM, and the specific series is the P-APLTDCN. There is a green sticker at the bottom of the package that reads, \"For Lightning Cable.\" \n\nPanel 3: The final panel is a top view of the connector, which appears similar to the ones in the previous panels. In this panel, the actual connector is visible, revealing that it is, in fact, a Lightning connector.", + "pred": [ + "This image is a collage of three photographs that document the use of a specific electronic adapter. The overall theme is the adaptation of a modern device (a smartphone) to an older, legacy system (a VGA monitor or projector). The collage is presented in a way that highlights the unusual and somewhat humorous nature of this connection.\n\nHere is a detailed description of each panel:\n\n---\n\n- **Panel 1 (Left):** This is the largest panel, showing a close-up of a smartphone with a black, protective case. The phone is positioned at an angle, with its charging port facing the viewer. A white, flexible cable is plugged into the" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 0.9999996423722521 + }, + { + "eval-id": 40, + "question_type": "open", + "gt": "The humor in this image comes from the misleading setup across the three panels. \n\nPanel 1: It shows a smartphone with a VGA cable plugged into its charging port, which is unexpected, as smartphones typically don't connect with VGA cables. \n\nPanel 2: The product package is shown, containing a VGA connector from the ELECOM P-APLTDCN series, making it appear related to the previous panel. The green sticker at the bottom says, \"For Lightning Cable,\" adding to the confusion.\n\nPanel 3: The top view of the connector reveals that it is actually a Lightning connector instead of a VGA connector, which explains how it can be plugged into the smartphone. This surprise twist creates the humor in the meme.", + "pred": [ + "Of course, here is a detailed description of the funny aspects of the image, broken down panel by panel.\n\nThe image is a composite of three photos that together tell a humorous story about a specific, and very awkward, electronic accessory.\n\n---\n\n### Panel 1 (Left, Main Image)\nThis panel shows a smartphone, which appears to be a modern model with a black screen and a protective case, connected to a **blue VGA (or DVI) cable**. The cable is plugged into the phone's charging port. The cable's plug is a standard, bulky blue VGA connector, which is designed for connecting a computer to a monitor" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 1.0 + }, + { + "eval-id": 41, + "question_type": "open", + "gt": "The creature appears to be made up of hot, red lava with flames on its body and cooled-down black volcanic rock for its four feet.", + "pred": [ + "Based on the visual characteristics of the creature, it appears to be made of a combination of materials, primarily designed to look like a 3D printed or sculpted object.\n\nHere's a breakdown of the materials and textures visible:\n\n- **The Body and Fur:** The main body of the creature is a vibrant, fiery red-orange color. The surface has a smooth, glossy finish, which suggests a material like **plastic resin** or **acrylic**. The way the light reflects off it gives it a polished, almost metallic sheen, which is characteristic of these materials. The texture is also smooth, but it's not perfectly" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 1.0 + }, + { + "eval-id": 42, + "question_type": "open", + "gt": "The logo of LLaVA features a creature that resembles a llama, creatively formed with lava. The body of the llama appears like hot, red lava with flames around it, giving it a sense of energy and movement. Its four feet look like black volcanic rocks that have cooled down after a lava flow, providing a striking contrast to the fiery body. The llama is also wearing a pair of red glasses, which adds an element of intelligence and quirkiness to the logo. \n\nThe design of the logo is innovative and unique, as it incorporates the concept of both language and vision in the form of a lava-formed llama. The creative combination of these elements makes the logo memorable and effectively represents the LLaMA architecture.", + "pred": [ + "The image you've provided is not the logo of LLaVA (Large Language and Vision Assistant), but rather a stylized, cartoonish figurine that appears to be a digital creation, possibly an AI-generated character. The character is a fiery, red creature with a flamboyant, cartoonish design. It has large, expressive eyes, a pair of red-rimmed glasses, and is surrounded by flames, giving it a dynamic, energetic appearance. The creature stands on two legs, with a textured, rocky base that suggests it might be a creature from a fantasy or sci-fi world. The overall design is whimsical and imaginative" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 1.0 + }, + { + "eval-id": 43, + "question_type": "open", + "gt": "The painting features three animals: a brown bear on the left who is eating a chocolate cookie, a blue-grey cat in the middle who is holding a blue mug, and a light-brown rabbit on the right who is sipping from a pink coffee mug. They are all gathered around a table made of a tree stump, enjoying a picnic.", + "pred": [ + "Based on the painting, the animals are a bear, a cat, and a rabbit. They are gathered around a tree stump that serves as a small table, enjoying a meal together.\n\nHere is a breakdown of what each animal is doing:\n\n- **The Bear**: On the left, the bear is sitting and holding a small plate of cookies in its paws. It is also holding a small cup, possibly a teacup, to its mouth. It appears to be eating or drinking from the cup.\n- **The Cat**: In the center, the cat is sitting upright, looking at the bear and the rabbit. It" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 0.9997293489777506 + }, + { + "eval-id": 44, + "question_type": "open", + "gt": "Once upon a time, in a magical forest nestled between rolling hills and a sparkling river, there lived a brown bear, a blue-grey cat, and a light-brown rabbit. They were the best of friends, and they loved nothing more than spending their days together in the peaceful embrace of their beloved forest.\n\nOne lovely day, the sun was shining brightly, and the gentle breeze tickled the leaves of the tall trees, creating a beautiful symphony. The three friends decided it was the perfect time for a picnic. They chose a picturesque spot underneath a canopy of leaves, where a tree stump served as their table. \n\nThe bear, who had a sweet tooth, brought a plate filled with various kinds of delicious cookies. Beaming with excitement, he took a large bite of a chocolate cookie, savoring the rich and indulgent taste. The cat, who loved her tea, brought her favorite blue mug filled with the finest blend. She held the mug in her paws, enjoying the warm aroma that filled the air. The rabbit, known for her love of coffee, delicately sipped her beverage from a pink mug, feeling the warmth and comfort spread throughout her tiny body.\n\nThere they were, surrounded by the lush green grass, each friend engrossed in their own little world of delight. The bear's chocolate cookie crumbs dusted the ground beneath him, the cat's purrs harmonizing with the rustling leaves, and the rabbit's whiskers twitching with joy and satisfaction.\n\nTheir laughter and sweet conversation filled the clearing, and the sounds of their merrymaking brought curious forest creatures out of their hiding places. Slowly, they approached the merry trio, wanting to be part of the joy they witnessed. One by one, they joined the picnic, and soon they were all sharing a feast of cookies and happiness.\n\nIn that magical forest, a bond beyond friendship united those woodland creatures, as they spent countless days celebrating their love for each other and the enchanting world they called home. And so, they lived together in harmony, basking in the beauty of their friendship, for many, many years to come. And the tale of the Picnic of the Forest Friends was passed down through generations, forever immortalizing their wonder and joy.", + "pred": [ + "### The Tale of the Forest's Last Tea\n\nIn the heart of a lush, sun-dappled forest, where the trees whispered secrets to the wind and the leaves rustled like secrets, lived a trio of unlikely friends: a wise bear, a curious cat, and a gentle rabbit. They were not the usual forest dwellers, but they were the guardians of the forest's quiet magic. The bear, named Thistle, was a stout guardian of the forest's ancient roots, while the cat, named Marmot, was a silent observer of the world, and the rabbit, named Pippin, was the quietest" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 0.9999998807907247 + }, + { + "eval-id": 45, + "question_type": "open", + "gt": "The sketch is a website layout on a page of a notebook. It has a heading that reads \"My Joke Website\". Below the heading, there are several rows, presumably representing different sections of the website. Each row contains a joke followed by a button or interactive element. The first row includes \"[Really funny joke 1]\" and \"[Push to reveal punchline]\". The second row has \"[Same, but joke 2]\" and \"[Push to reveal punchline]\". At the bottom of the sketch, there's a copyright notice: \"Copyright OpenAI 2023\". The overall layout is simple and linear, focusing on the presentation of jokes and their punchlines through interactive user engagement.", + "pred": [ + "This is a hand-drawn, minimalist sketch of a website layout, likely for a personal or humorous website. The entire drawing is contained within a rectangular frame, mimicking the appearance of a page in a notebook.\n\nThe sketch is organized with a clear structure, using a hand-drawn line to separate the title from the content.\n\n**Top Section: The Title**\n- The title, \"My Joke website,\" is written in a casual, handwritten script.\n- The text is centered at the top of the page.\n- The title is enclosed within a horizontal line, which is drawn with a single, thick line across the top of the" + ], + "decided_by": "llm-judge", + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "ok", + "judge_prob": 1.0 + }, + { + "eval-id": 46, + "question_type": "open", + "gt": "Here is a brief and simple HTML/JS code to create a colorful website using the notebook sketch with two real jokes:\n\n```html\n\n\n\n \n \n My Joke Website\n \n\n\n

My Joke Website

\n

Why do we never tell secrets on a farm?

\n \n \n\n

Why don't scientists trust atoms?

\n \n \n \n

Copyright OpenAI 2023

\n\n \n\n\n```\n\nThis code creates a simple and colorful webpage with a light blue background and a purple heading. The jokes and buttons are displayed on the page, and when the user clicks the \"Push to reveal punchline\" button, the respective punchlines will be revealed using JavaScript.", + "pred": [ + "```html\n\n\n\n \n \n My Joke Website\n \n\n\n

My Joke Website

\n

Funny Joke

\n \n\n \n\n\n```\n\nThis code snippet creates a simple colorful webpage with a heading \"My Joke Website\", a joke placeholder, and a button to reveal the punchline. It fetches a joke from the Chuck Norris Jokes API (https://api.chucknorris.io/jokes/random) when the user clicks the button.", + "pred": [ + "```html\n\n\n\n \n \n My Joke Website\n