feat(ruler): add RULER long-context benchmark (datasets + tasks)#11

Open

Mea1Ma wants to merge 106 commits into

scitix:mainfrom

Mea1Ma:feat/ruler

Commits on Jul 6, 2026

fix(dataset): compare on-the-wire bytes so gzip responses aren't flagged as truncated downloads

Mea1Ma
and
claude
committed
feat(datasets): add local: downloader for package-bundled dataset files
Mea1Ma
committed
feat(datasets): subpackage auto-discovery + vendored RULER assets

Mea1Ma
and
claude
committed
feat(datasets): RULER synthetic datasets (niah/qa/vt/cwe/fwe)

Mea1Ma
and
claude
committed
feat(tasks): RULER tasks (chat + base-gen) and effective-length report

Mea1Ma
and
claude
committed
docs(examples): RULER eval configs and multi-length sweep generator

Mea1Ma
and
claude
committed
feat(examples): inject tokenizer_model into generated RULER sweep

Mea1Ma
and
claude
committed
fix(datasets): load hotpotqa from HF instead of the defunct CMU URL

Mea1Ma
and
claude
committed
fix(examples): correct hotpotqa dataset path in RULER sweep generator

claude
authored and
Mea1Ma
committed
feat(ruler): remove base_gen endpoint support

claude
authored and
Mea1Ma
committed
refactor(ruler): simplify to chat-only endpoint

claude
authored and
Mea1Ma
committed
fix(ruler-qa): handle hotpotqa config fallback and path correction

claude
authored and
Mea1Ma
committed
fix(ruler-qa): correct hotpotqa loading strategy

claude
authored and
Mea1Ma
committed
refactor(ruler): align scoring with community.ruler metrics

claude
authored and
Mea1Ma
committed
feat(ruler): support task ordering from config file

claude
authored and
Mea1Ma
committed
feat(ruler): include detailed per-task results in output

claude
authored and
Mea1Ma
committed
feat(output): display per-task scores in ruler-effective text renderer

claude
authored and
Mea1Ma
committed
fix(ruler-gen): add SGLang deterministic inference to server overrides

claude
authored and
Mea1Ma
committed
fix(output): match task_order names with task_base names in per-task display

claude
authored and
Mea1Ma
committed
feat(ruler): auto-detect effective_config.yaml for task ordering

claude
authored and
Mea1Ma
committed
fix(ruler): strip suffix from task_order before matching with task base names

claude
authored and
Mea1Ma
committed
feat(ruler): add Qwen3 8B SGLang configurations

claude
authored and
Mea1Ma
committed
refactor(ruler): rework RULER datasets and tasks to reproduce model continuation

Mea1Ma
and
claude
committed
feat(ruler): adapt dataset generation for Qwen3 with model-aware tokenization

authored and
Mea1Ma
committed
chore(examples): restore infer-recipe-override.yaml

Mea1Ma
and
claude
committed
build(deps): restore project metadata and add ruler dependency group

Mea1Ma
and
claude
committed
refactor(ruler): centralize thinking-tag prefill into a config-driven helper

Mea1Ma
and
claude
committed
fix(ruler): correct VT/CWE shot count to n_shot=1 (kshot, not 0-shot)

Mea1Ma
and
claude
committed
docs(ruler): restore NVIDIA license headers and upstream attribution

Mea1Ma
and
claude
committed
style(ruler): satisfy ruff — drop dead code and fix lint

Mea1Ma
and
claude
committed
fix(ruler): satisfy ty — type the vendored TASKS table at the call sites

Mea1Ma
and
claude
committed
refactor(leaderboard): remove the ruler-effective command

Mea1Ma
and
claude
committed
feat(ruler): pin the HotpotQA HF revision for reproducibility

Mea1Ma
and
claude
committed
chore(examples): drop the stray run.sh dev script

Mea1Ma
and
claude
committed
chore(ruler): drop the qwen3-8b sweep generator from the PR

Mea1Ma
and
claude
committed
docs(ruler): correct attribution to NVIDIA RULER and pin reference SHAs

Mea1Ma
and
claude
committed
feat(ruler): add leaderboard ruler-avg for the 13-subtask headline mean

Mea1Ma
and
claude
committed
feat(ruler): round ruler-avg means to one decimal

Mea1Ma
and
claude
committed
style(ruler): drop fixed-width padding on ruler-avg per-length value

Mea1Ma
and
claude
committed
test(downloaders): add AI-Generated marker to test_local

Mea1Ma
and
claude
committed
style(ruler): ruff-format _ruler_avg

Mea1Ma
and
claude
committed
docs(ruler): fix last opencompass reference in niah task docstring

Mea1Ma
and
claude
committed
fix(datasets): make local: copy atomic and extend basename guard to local:

Mea1Ma
and
claude
committed
refactor(ruler): unify 5+5 dataset/task classes into one RulerDataset + RulerZeroShotGenTask

Mea1Ma
and
claude
committed
refactor(ruler): support subpackage dataset discovery and fix thinking prefill

claude
authored and
Mea1Ma
committed
fix: pre-commit formatting and code quality checks

claude
authored and
Mea1Ma
committed
fix(ruler): pre-commit formatting and docs for long-context eval config

claude
authored and
Mea1Ma
committed
refactor(ruler): drop assistant prefill, move answer_prefix to user turn, add think_budget

claude
authored and
Mea1Ma
committed
refactor(ruler): add Qwen3 thinking tag overhead and multi-subtask loading

authored and
Mea1Ma
committed
refactor(ruler): add dual message pattern support for assistant vs user prefill

Mea1Ma
and
claude
committed
docs: clarify extra_body config detection logic for assistant prefill pattern
Mea1Ma
committed
docs(examples): add comprehensive comments to all ruler YAML configs

Claude Code
authored and
Mea1Ma
committed
fix(dataset): reword url size-mismatch error; pin gzip test to RULER's dev-v2.0.json source

authored and
Mea1Ma
committed
refactor(tasks): shorten comment lines to satisfy ruff line-length check

Claude Code
authored and
Mea1Ma
committed
feat(tasks): add HMMT Feb 2025 and IMO-AnswerBench benchmarks (#22)

authored and
Mea1Ma
committed
fix(ruler): make feat/ruler pass CI (lint, types, preflight, tests)

Mea1Ma
and
claude
committed
docs(downloaders): correct local scheme docstring to match generate-not-commit flow

Mea1Ma
and
claude
committed
refactor(ruler): true bring-your-own local: corpus (+ nltk dep, example fix)

Mea1Ma
and
claude
committed
fix(ruler): consistency and robustness improvements per maintainer review

Mea1Ma
and
claude
committed
refactor(ruler): onboarding + encapsulation nits

Mea1Ma
and
claude
committed
fix(ruler): resolve script and type-checker issues in final nits

Mea1Ma
and
claude
committed
feat(mbpp): add dataset and few-shot base-model task (#12)

jack-scitix-ai
authored and
Mea1Ma
committed
feat(openbookqa): add dataset and k-shot generative task (#19)

jack-scitix-ai
authored and
Mea1Ma
committed
feat(models): add SglangGenModel for echoed-input logprobs via sglang /generate (#21)

authored and
Mea1Ma
committed
feat(gsm8k): add DeepSeek-Math-aligned 0-shot chat-model task (#29)

jack-scitix-ai
authored and
Mea1Ma
committed
feat(ifbench): add dataset and few-shot base-model task (#13)

authored and
Mea1Ma
committed
feat(tasks): add clp eval mode + naming category (#23)

authored and
Mea1Ma
committed
fix(imo-answerbench): normalize in extraction + verbatim grader, promote to stable (#28)

authored and
Mea1Ma
committed
chore: release 0.6.0

authored and
Mea1Ma
committed
style(pyproject.toml): format with taplo

Mea1Ma
and
claude
committed
chore: regenerate stubs and meta index with ruler

Mea1Ma
and
claude
committed
chore(pdm): update lock file

Mea1Ma
and
claude
committed
Merge branch 'main' into feat/ruler
Mea1Ma
authored

Commits on Jul 7, 2026

Commits on Jul 16, 2026

Commits on Jul 17, 2026

Commits on Jul 22, 2026

fix(ruler): simplify think_budget context_length checks to 128k only

Dev
and
claude
committed
style(ruler): apply ruff formatting

Mea1Ma
and
claude
committed

Commits on Jul 23, 2026

Commits on Jul 24, 2026

Merge branch 'main' into feat/ruler
Mea1Ma
authored