Skip to content

Add Dr. Bench deep-research benchmark #75

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
reacher-z wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Add Dr. Bench deep-research benchmark #75

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,6 +386,7 @@ Most "awesome" lists are link dumps. This one is **annotated and verified**: eve
- **[ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery](https://github.com/OSU-NLP-Group/ScienceAgentBench)** — OSU-NLP Group (Ohio State) — <https://github.com/OSU-NLP-Group/ScienceAgentBench> · *benchmark* — 102 expert-validated tasks from 44 peer-reviewed papers; grades self-contained Python programs by execution + success rate; best agent solves only ~34% (ICLR 2025). 🆕
- **[CORE-Bench: Computational Reproducibility Agent Benchmark](https://arxiv.org/abs/2409.11363)** — Siegel, Kapoor, Narayanan et al. (Princeton) — <https://arxiv.org/abs/2409.11363> · *benchmark* — 270 tasks over 90 papers (CS/social science/medicine) that grade whether an agent can reproduce published results from code+data; from the Princeton AI-Snake-Oil group.
- **[DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://arxiv.org/abs/2506.11763)** — Mingxuan Du et al. — <https://arxiv.org/abs/2506.11763> · *benchmark* — 100 PhD-level tasks across 22 fields; reference-based adaptive-rubric grader for analyst-grade citation-rich reports, validated for human-judgment alignment — the standard deep-research-report eval. 🆕
- **[Dr. Bench: A Multidimensional Evaluation for Deep Research Agents, from Answers to Reports](https://arxiv.org/abs/2510.02190)** — Yao et al. — <https://github.com/EVIGBYEN/DrBench> · *benchmark* — 214 expert-curated deep-research tasks across 10 domains, evaluating long reports for semantic quality, topical focus, and retrieval trustworthiness. 🆕
- **[BixBench: A Comprehensive Benchmark for LLM-based Agents in Computational Biology](https://arxiv.org/abs/2503.00096)** — FutureHouse + ScienceMachine — <https://arxiv.org/abs/2503.00096> · *benchmark* — 50+ real bioinformatics analysis scenarios with ~300 open-answer questions over multi-step Jupyter trajectories; frontier models hit only ~17% — serious wet-lab-adjacent science agent eval. 🆕
- **[Introducing LifeSciBench](https://openai.com/index/introducing-life-sci-bench/)** — OpenAI — <https://openai.com/index/introducing-life-sci-bench/> · *benchmark* — 🆕 750 expert-authored life-science research tasks (7 workflows × 7 biological domains) graded by 19,020 rubric criteria from 173 PhD-level scientist contributors, independently validated by 453 expert reviewers; requires interpreting genomic sequence files, chemical structures, and experimental figures; best model (GPT-Rosalind) scores 36.1% overall — the largest expert-rubric-graded wet-lab-workflow agent benchmark.
- **[Gaia2 and ARE: Scaling Up Agent Environments and Evaluations](https://arxiv.org/abs/2509.17158)** — Meta (Meta Agents Research Environments) — <https://arxiv.org/abs/2509.17158> · *benchmark* — Successor to GAIA: dynamic, time-driven, multi-agent simulated environments with async world events and a verifiable scenario grader; frontier success ~42% — the serious general-assistant env from Meta. 🆕
Expand Down Expand Up @@ -573,4 +574,3 @@ To the extent possible under law, [BenchFlow](https://benchflow.ai) and contribu




Loading