feat(ruler): add RULER long-context benchmark (datasets + tasks)

[Mea1Ma]

Note: This PR is stacked on #9 (add local: source handler and fix url: gzip truncation check
). Those commits appear in this diff too because GitHub can't target a fork branch as the base. Please review the downloader changes in #9; once it merges I'll rebase this onto main to drop them. The RULER work here depends on the local: handler (the Paul Graham haystack is bundled in-tree and staged via local:).

Type

• feature — new benchmark, task, or capability

Summary

Adds the RULER long-context benchmark (13 configs across 4 categories) as
synthetic datasets + thin generative tasks, adapted from NVIDIA RULER

• datasets/ruler/ — 5 synthetic loaders (niah/qa/vt/cwe/fwe) that build the
  full prompt at load time; haystack corpus (Paul Graham Essays) is generated
  once and bundled in-tree, staged via the local: scheme.
• tasks/ruler/ — thin chat tasks that send the prompt and score by RULER's
  string_match_all/string_match_part. VT/CWE are kshot (n_shot=1) because
  upstream bakes one in-context demonstration into the prompt; the rest are 0-shot.
• community/ruler/ — vendored RULER constants/tokenizer/scoring, kept close
  to upstream with attribution.
• Decoding params stay in user config (tasks never inject them); HotpotQA source
  is revision-pinned for reproducibility.

Reference: https://github.com/NVIDIA/RULER

Related Issues

Test Plan

CI note: ty is pinned to 0.0.20 in the lock for parity with main
(an upstream merge had re-resolved the loose ty>=0.0.5 spec up to 0.0.59,
whose stricter checks flag ~150 pre-existing diagnostics repo-wide). RULER's
deps (tiktoken/wonderwords/nltk/scipy/transformers) are absent from
the light CI env, so the ruler tests are gated behind @_needs_ruler_deps
and skip there.

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check; ty pinned to 0.0.20)
• Unit tests pass — CI light env (no ruler extra):
  pdm run pytest tests/unit -m "not stress" --cov → 2433 passed, 39 skipped,
  98.44% coverage (coverage scope is sieval/core). RULER tests requiring the
  ruler extra skip here; pdm install -G ruler && pdm run pytest runs the full
  suite (ruler tests green locally).

Manual

Reproduces RULER aligned with NVIDIA RULER. Compared against the
RULER score in the Qwen3 technical report (arXiv:2505.09388) — note Qwen does
not publish the exact prompt/decoding/answer-extraction config, so the target
is "land close to the report" rather than byte-exact reproduction.

• Model tested: Qwen3-8B, both non-thinking and thinking modes (matches the report's settings)
• Coverage: full 4k–128k length sweep (4k/8k/16k/32k/64k/128k), 13-task avg, for both modes
• Reference (Qwen3 tech report, RULER avg over 4k–128k): 89.1 (non-thinking) / 84.4 (thinking)
• Actual (this implementation): 88.12 (non-thinking) / 86.86 (thinking)
• Diff: -0.98 (-1.10%) non-thinking, +2.46 (+2.91%) thinking — both within target < 3%

Model	Mode	Expected	Actual	Diff
Qwen3-8B	Non-thinking	89.1	88.12	-0.98 (-1.10%)
Qwen3-8B	Thinking	84.4	86.86	+2.46 (+2.91%)

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) marker
• No new upper-layer dependencies added to core/
• Deleted code verified — no remaining call sites depend on it

If: New or Modified Benchmark

• Reference paper/repo linked in Summary
• Score comparison table included (model, expected, actual, diff)
• Dataset loading tested (sieval dataset download path; local/url/hf sources resolve)
• Task registered in package-level __init__.py

If: community/ Changes

• Upstream diff documented (vendored from NVIDIA RULER, attribution in headers)
• License attribution preserved (Apache-2.0 headers restored)

If: New Dependency

• Added to correct PDM dependency group (ruler: tiktoken, wonderwords, numpy, scipy, nltk, transformers)
• Justified in Summary (tiktoken/wonderwords for prompt synthesis, scipy/numpy for sampling, nltk for sentence tokenization, transformers for the HF/Qwen3 tokenizer path)

[ethan-scitix]

Review: RULER benchmark onboarding

The port is faithful — I verified the generation logic against NVIDIA RULER at the pinned SHA, and the 6-length Qwen3-8B run reproduces published recall numbers closely (cwe@16k 93.5 vs 93.2, vt/niah ~100). Requesting changes on packaging and structure before merge.

Blocking

• Bundled corpus doesn't belong in the repo. sieval/datasets/_data/paul_graham_essays/PaulGrahamEssays.json.gz (1.1MB) should not ship in-tree. RULER's corpus isn't an official artifact — NVIDIA ships only a URL list + scraper, so this .gz is just one local scrape frozen. Host one copy (HF @sha, or any immutable/public OSS object via url:) and keep scripts/gen_paul_graham_essays.py as the BYO/regeneration path. The loader is host-agnostic once the file lands in the data dir.
• Overlaps with the still-open #9. This branch re-includes #9's downloaders/{base,local,url}.py + tests verbatim. Land the url: gzip fix via #9; rebase this PR on top and drop those files. local: itself should become true BYO (no in-wheel bundling; download = no-op + instructions) rather than copy-from-package.
• nltk missing from the ruler extra. pyproject.toml:39 lists tiktoken/wonderwords/numpy/scipy but not nltk, which the default niah/vt essay path imports (_common.py:71, ruler_niah.py:285, ruler_vt.py:315). check_dep_coverage passes only because nltk is in ifeval; a clean pip install sieval[ruler] breaks at load. Add nltk to the ruler group.

Structure (the bigger one)

• Collapse to 1 dataset + 1 task, mirroring MMLU. The 5 datasets / 5 empty task classes + the new leaderboard ruler-avg command are all symptoms of the same gap. MMLU already models 57 subtasks as one dataset with a Subject column and aggregates per-group in report() (mmlu_0shot_gen.py:92,102). Do the same: one RulerDataset emitting a subtask column, one RulerTask whose report() groups by subtask (right matcher per group — string_match_all vs string_match_part) and emits per-subtask scores + the headline mean. The context-length dimension stays as separate runs (64k/128k need YaRN-scaled serving). This removes _ruler_avg.py, the ruler-avg command, and the output.py renderer entirely — no benchmark-specific logic in the CLI, no parsing length out of task-name strings.

Other

• Bug: CWE english_words.json pinned to main (ruler_cwe.py:44), unlike the deliberately-pinned HotpotQA revision. It's the long-context vocab fallback used exactly at 64k/128k. Pin to a SHA.
• Nit: Category. Logic/TextualReasoning is a poor fit for retrieval/aggregation tasks. Move to Language/SemanticUnderstanding; the long-context tag already carries the capability axis.
• Reproducibility: ship at least one example eval config (now small — one ruler task per length) and document the non-obvious prerequisites: continue_final_message: true (without it the answer-prefix continuation silently breaks), YaRN for >32k, and tokenizer_path must match the target model (RULER sizes prompts to the model's tokenizer). The config that produced the run was dropped in 4c1d1749 and examples/ has none.

(Dockerfile / deps-in-image handled separately.)

[ethan-scitix]

Verdict: Comment — port is faithful and reproduction checks out; a few consistency/onboarding items before merge.

Verified empirically: string_match_all/string_match_part, templates, and token budgets are byte-exact to NVIDIA/RULER at ab17b78; the vendored matcher reproduces the stored subtask scores exactly (cwe=98.54, qa_squad=76.8, fwe=97.53), and the per-length means average to ≈88, consistent with the reported 88.2.

Should fix before merge

• Bug (consistency): the per-sample retry loops differ across siblings. Upstream RULER's niah/qa/vt/cwe all lack the floor else (all can spin at used <= incremental). This port added else: break to _niah.py:159, _vt.py:155, _cwe.py:84 but not _qa.py:81 — so three deviate from upstream and one matches it. Practically unreachable at 4k+, but pick one behavior for all four: either match upstream exactly (drop the else), or keep the guard everywhere and annotate the deviation in code. Author's call, but it must be consistent.
• Nit (architecture): _ChatGenBase (ruler_0shot_gen.py:51) has a single subclass. Inline preprocess/infer/postprocess into RulerZeroShotGenTask and delete the base.
• Nit (encapsulation): ruler_0shot_gen.py:73,85 reads self.model._kwargs/_model; use the public self.model.meta() instead.
• Nit (robustness): sieval dataset download ruler swallows LocalSourceUnavailable and exits 0 (cli/dataset/commands.py:190). Track missing required local: sources and print a clear final summary + exit non-zero, so a missing corpus isn't mistaken for success.
• Nit (onboarding): the BYO corpus step is only in the script docstring / error message, not in the config setup or docs, and the generator's deps (html2text, beautifulsoup4, certifi) are undeclared — so following the example config fails at load time, and running the script fails on import. Please: (1) document the full flow (install extra → generate corpus → download the rest → run), (2) declare the generator deps (e.g. a ruler-gen optional group), and (3) have download print the exact generate command rather than auto-scraping.

Nits

• Revert the .pre-commit-config.yaml --maxkb=2000 bump — the corpus is BYO/not committed, so it's vestigial and its comment (references bundling under sieval/datasets/_data/) is misleading.
• ty overrides (pyproject.toml): narrow sieval/datasets/ruler/** to the specific loader files (or install the ruler extra in the type-check env); drop the scripts/ exemption and declare its deps instead — exempting it hides that the script isn't runnable out of the box.
• wonderwords.random_word._get_words_from_text_file (_niah.py:224, _cwe.py:144): private API, faithful to upstream — keep, but add a comment noting the risk and cap the version (<3).
• Trim examples to nonthinking + thinking (flat, no subdir); fold the 64k/128k yarn variants into a comment.

PR body

• "Actual: 87.7" contradicts the table's 88.2 (89.1 − 0.9 = 88.2, −1.01%) — 88.2 is the real number.
• Summary says answer_prefix is appended to the user message "without a prefilled assistant turn," but the shipped nonthinking config and the reproduction runs use the assistant-prefill path (continue_final_message: true). Align the description with what was validated.

[ethan-scitix]

Bug: qa_hotpotqa never reads the staged snapshot; silently falls back to a live online fetch. ruler.py:56 routes it to the base data dir, so _qa.py:171 calls load_dataset("<data_dir>", "distractor", …) — which raises ValueError: BuilderConfig 'distractor' not found (reproduced against the real staged layout), caught at _qa.py:172 → always drops to online load_dataset("hotpotqa/hotpot_qa", …). So the pinned copy download stages is dead, and eval fails under HF_HUB_OFFLINE=1 despite a completed download. Point the loader at <data_dir>/hotpotqa/hotpot_qa and keep online as an explicit last resort.

Nit: unrelated churn in leaderboard/commands.py:64-73. Loop→comprehension rewrite of report(), no RULER connection — leftover from the removed ruler-avg command. Please restore origin/main's version so the PR touches no unrelated CLI file.

Nit: two of the four external sources are downloaded but never consumed. Besides HotpotQA above, english_words.json is loaded into randle_words (_cwe.py:42-45) but never sampled — the CWE cap keeps num_words ≤ vocab. This is the documented ≤32k stable scope, so it's fine; just add one line to the source docstring so nobody assumes it's load-bearing.

Nit: _read_hotpotqa builds a dict it only uses as a set (_qa.py:180-189 — values discarded by sorted(set(...))). Use a set comprehension.

Housekeeping: rebase onto main to drop the closed-#9 downloader commits (branch is also behind #30/#34/#35/#37). ruler-gen generator deps stay undeclared — a conscious call, but running the script still ImportErrors with no install hint; a one-line pip install … in the usage/error path would close it.

[ethan-scitix]

To address

1. Question: Thinking long-context reproduction is unverified. In the real runs, think-32k-YaRN collapsed (overall 6.89, everything 0 except fwe) and think 64k/128k have no report. This is not a port bug — the same code gives non-think-32k=92.3 and think-4k/8k are fine, so it's the 32k+YaRN serving config. But the "reproduces published recall" claim in reference_impl.notes (sieval/tasks/ruler_0shot_gen.py:62) is currently only backed by non-think + think-≤8k. Please re-run think ≥32k to confirm, or scope the status="stable" / repro claim to the validated configs and mark thinking long-context experimental.

2. Nit: The divergence notes read as exhaustive but omit 4 (all score-neutral, confirmed by the runs) — add to the notes or align to upstream:

   • estimated_max bounding formula differs from upstream (sieval/datasets/ruler/_niah.py:210 / _vt.py:246 / _cwe.py:123 / _qa.py:133) → different binary-search probe trajectory → different global-RNG consumption → samples not byte-reproducible vs upstream. Prefer aligning to upstream (max_seq_length -= model_template_token up front, use the raw token count).
   • HotpotQA is sourced from the HF mirror, not upstream's derived json; questions are taken in native order and sliced to the first N, so per-index selection may differ (sieval/datasets/ruler/_qa.py:182). The notes only cover doc ordering.
   • essay escape_snob=True (scripts/gen_paul_graham_essays.py:131) escapes markdown, but upstream's escape_all is a no-op (unescaped — confirmed in the staged corpus) → diverges from upstream's actual output. Either revert to unescaped, or record it as a divergence and note the repro corpus must be regenerated.
   • VT ICL fragment is sized against 500, not 500−T (sieval/datasets/ruler/_vt.py:128, currently only in a helper docstring).

3. Nit: CWE drops sharply in thinking mode under the cap (non-think 8k 98.3 → think 8k 69.1); predictions are verbose lists truncated mid-answer. Upstream's cap=120 assumes a terse answer — worth noting its effect on thinking-mode CWE in the notes.

4. Nit: Inconsistent qwen3 detection: thinking_prefill uses substring in (sieval/datasets/ruler/_shared.py:164) while get_template / tokens_to_generate use .startswith (:224 / :112). For non-standard model ids, packing and inference can disagree on the <think> block — align them.