feat(hle): add Humanity's Last Exam dataset + 0-shot LLM-judge task

[jack-scitix-ai]

Type

feature — new benchmark, task, or capability

Summary

• Adds Humanity's Last Exam (HLE) (Center for AI Safety; Phan et al., 2025): dataset loader, a 0-shot generative task, and a vendored metric kernel.
• Reference repo: https://github.com/centerforaisafety/hle/tree/26dca2e253b405105b4c3d8c2f5af06f86f90c66/hle_eval — SYSTEM_PROMPT / JUDGE_PROMPT / calib_err vendored byte-for-byte.
• Task: model answers under the HLE Explanation/Answer/Confidence prompt; a separate LLM judge (mandatory grader arg) decides correctness; metrics are accuracy, 95% Wald CI, and calibration error. Text-only subset by default.
• Dataset load() stays minimal; auxiliary Image columns are set decode=False to stay Pillow-free while preserving the raw image column.
• Deviations from upstream documented in module docstrings (judge reached via ChatModel text parsing instead of beta.chat.completions.parse; o1 system→user role swap dropped).
• Review cleanup: dropped a dead _text_only attribute; added byte-level pinning tests on the vendored prompts so upstream drift fails loudly.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check or mypy --strict)
• Unit tests pass (pdm run pytest)

Manual

Dataset loading validated end-to-end: full runs load cais/hle at the pinned revision and materialize 2158 text-only samples (same loader path as sieval dataset download hle).

0-shot, single attempt (n=1), non-tool — matching the official HLE protocol. Expected column: gpt-oss-20b HLE (text-only, no tools) from the gpt-oss model card (arXiv:2508.10925).

Model	Expected	Actual	Diff
gpt-oss-20b (reasoning=high)	10.9	12.14	+1.24(+1.24%)
gpt-oss-20b (reasoning=low)	4.2	3.61	−0.59(-0.59%)

Both within the <3 pp target; runs are clean (low: fails=1, no truncation; high: fails=0). The high-effort run flags 517/2158 (24%) truncated_output anomalies — long reasoning chains hitting max_tokens with an empty answer channel — yet still lands within target; expected reasoning-model behavior, not an eval defect.

Note on methodology: our judge is GPT-5.2 (text-only subset); the model card's HLE judge may differ, and HLE accuracy is judge-sensitive — treat this as a close, not identical-protocol, comparison.

• Ran both effort settings; scores + fail/truncation counts captured.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/
• Deleted code verified — no remaining call sites depend on it

If: New or Modified Benchmark

• Reference paper/repo linked in Summary
• Score comparison table included (model, expected, actual, diff)
• Dataset loading tested (sieval dataset download <name> succeeds)
• Task registered in package-level __init__.py

If: community/ Changes

• Upstream diff documented (what differs and why)
• License attribution preserved

If: New Dependency

• Added to correct PDM dependency group
• Justified in Summary (why this package, no lighter alternative?)

[ethan-scitix]

Approve — faithful port, prompts/metrics byte-exact vs upstream@26dca2e, and the gpt-oss-20b validation checks out (12.14/3.61 vs card 10.9/4.2, <3pp, like-for-like incl. truncation). stable is justified. A few non-blocking nits:

• Nit: Mirror the validation numbers into reference_impl.notes — PR bodies don't ship with the artifact, and simpleqa_verified already sets the precedent of putting its validation line in the notes (they get embedded in index.json). One line is enough, e.g. "VALIDATION: gpt-oss-20b 12.14/3.61 vs card 10.9/4.2 (<3pp), judge GPT-5.2, text-only."
• Nit: sieval/tasks/hle_0shot_gen.py:241 — rename n_judged → n_graded to match the judge siblings (browsecomp, simpleqa_verified).
• Nit: report() could echo the truncated/anomaly count alongside fails. The 12.14 headline is a valid like-for-like number, but a reader of report.json alone can't see that 24% of outputs truncated — surfacing it makes the headline self-documenting for a collapse-prone benchmark.
• Nit: sieval/community/hle.py:70 — _CORRECT_RE matches inside "incorrect:"; safe today (last-match + field order), but \bcorrect removes the alias for free.