feat(aa-lcr): add dataset + 0-shot LLM-equality-checker-graded task

[jack-scitix-ai]

Type

• feature — new benchmark, task, or capability

Summary

• Add AA-LCR (Artificial Analysis Long Context Reasoning): 100 hard multi-document reasoning questions (~71k–114k input tokens each). Free-form answers are graded by an LLM equality checker (binary CORRECT/INCORRECT); headline metric = accuracy.
• Upstream ships no eval code — the input prompt and the equality-checker prompt are reproduced verbatim from the dataset card (Apache-2.0).
• Dataset loader reads AA-LCR_Dataset.csv and pulls each document straight from the shipped extracted_text/AA-LCR_extracted-text.zip in data_source_filenames order; recovers the archive's non-UTF8 (cp437-mojibake) / NFD member names so lookups match the CSV.
• Empty/truncated answers are graded INCORRECT without calling the grader — verified that the checker returns CORRECT for an empty candidate, which would otherwise inflate the score.
• Refs: dataset card https://huggingface.co/datasets/ArtificialAnalysis/AA-LCR · scoring/sampling per AA methodology https://artificialanalysis.ai/methodology/intelligence-benchmarking (reasoning models: temperature 0.6; AA-LCR = 3 repeats, pass@1 aggregated).

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check or mypy --strict)
• Unit tests pass (pdm run pytest)

Manual

Validated against the Artificial Analysis public leaderboard (https://artificialanalysis.ai/evaluations/artificial-analysis-long-context-reasoning). All runs: temperature=0.6, n=3 repeats (AA-aligned), grader = Qwen3-235B-A22B-Instruct-2507. gpt-oss-20b served locally (sglang, bf16); gpt-oss-120b via a remote OpenAI-compatible endpoint.

Model	Expected	Actual	Diff
gpt-oss-120b (reasoning=high)	50.7	53.00	+2.30(+2.30%)
gpt-oss-120b (reasoning=low)	43.7	45.00	+1.30(+1.30%)
gpt-oss-20b (reasoning=high)	30.7	33.67	+2.97(+2.97%)
gpt-oss-20b (reasoning=low)	31.0	21.33	−9.67(-9.67%)

• 3 of 4 configurations reproduce AA within ~3 points; sieval dataset download aa_lcr verified end-to-end (100 questions / 776 documents load, 0 failures).

Known discrepancy — gpt-oss-20b low (−9.7): AA's own numbers show the 20b is effort-insensitive (low 31 ≈ high 30.7) while the 120b is effort-sensitive (43.7 → 50.7), i.e. the small model saturates at high effort. Our high-effort runs match AA for both sizes, and 120b matches at both effort levels — so the eval pipeline and grader are sound. The 20b-low gap is attributed to a serving-level difference in how much reasoning the "low" level elicits on the small model (our local run produced ~230 median reasoning tokens at low), not the benchmark implementation. Not resolvable from our side without AA's per-question outputs.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/
• Deleted code verified — no remaining call sites depend on it

If: New or Modified Benchmark

• Reference paper/repo linked in Summary
• Score comparison table included (model, expected, actual, diff)
• Dataset loading tested (sieval dataset download <name> succeeds)
• Task registered in package-level __init__.py

If: community/ Changes

• Upstream diff documented (what differs and why)
• License attribution preserved

[ethan-scitix]

Approve. Reproduction confirmed empirically against the AA-LCR leaderboard: gpt-oss-120b/20b (high) scored 53.0 / 33.7 vs official 50.7 / 30.7 — within ~2–3 pts, and the exact 1/300 granularity confirms the n=3 protocol is faithfully replicated. Loader verified against the real archive at the pinned SHA (100/100 questions, 776/776 document refs resolve; the cp437→UTF-8 + NFC recovery is load-bearing for 8 non-ASCII filenames and correct). Preflight green, 36 tests pass, accuracy recomputes exactly from raw grades. Prompt/grader templates are verbatim from the card.

Nit (do first, 1 line): Pin the source URL to a revision permalink in sieval/community/aa_lcr.py:2 and reference_impl.url (sieval/tasks/aa_lcr_0shot_gen.py:77) — the templates are reproduced verbatim from a specific revision, so link .../blob/bdae010bbce259820c0e34c1d7cce210d966fb75/README.md rather than the repo root.

Nit (nice-to-have): Add the validation to reference_impl.notes the way simpleqa_verified does, e.g. "reproduced gpt-oss-120b/20b (high) within ~2–3 pts of AA leaderboard 50.7/30.7; grader Qwen3-235B-Instruct-2507." It's the evidence that justifies status="experimental".

Question (low): The empty-answer guard in sieval/tasks/aa_lcr_0shot_gen.py:167 is a good defensive choice but diverges from the browsecomp/simpleqa_verified siblings (it didn't fire in any of these runs). Worth a one-line comment noting it's aa_lcr-specific by design, so the next person diffing the family doesn't read it as an oversight.

status="experimental" is the correct call and should stay — no published upstream harness to diff against, and the score depends on a grader endpoint that isn't version-pinnable. Nothing here gates merge.