Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -311,6 +311,7 @@ Most "awesome" lists are link dumps. This one is **annotated and verified**: eve
- **[A Survey on LLM-as-a-Judge](https://arxiv.org/abs/2411.15594)** — Jiawei Gu et al. — <https://arxiv.org/abs/2411.15594> · *paper* — The most-cited survey organizing the LLM-judge space (bias taxonomy, reliability methods, agreement metrics). Serves as the one-stop map/bibliography the section currently lacks.
- **[One Token to Fool LLM-as-a-Judge](https://arxiv.org/abs/2507.08794)** — Yulai Zhao, Haolin Liu, Dian Yu et al. (Tencent AI Lab / Princeton) — <https://arxiv.org/abs/2507.08794> · *paper* — Shows 'master-key' tokens (a colon, 'Solution:') trigger false-positive rewards up to 80% even on GPT-o1/Claude-4 judges, plus a robust Master-RM fix. Core evidence on judge/verifier reward-hacking fragility. 🆕
- **[Weaver: Closing the Generation-Verification Gap with Weak Verifiers](https://hazyresearch.stanford.edu/blog/2025-06-18-weaver)** — Jon Saad-Falcon et al. — Stanford Hazy Research / Scaling Intelligence — <https://hazyresearch.stanford.edu/blog/2025-06-18-weaver> · *blog* — Directly operationalizes 'verifiable vs judgeable': aggregates many weak judges/reward models (unlabeled) to shrink the generator-verifier gap, reaching o3-mini accuracy from Llama-3.3-70B. Paper: arxiv.org/abs/2506.18203. 🆕
- **[greenproof](https://github.com/zxyasfas/greenproof)** — <https://github.com/zxyasfas/greenproof> · *tool* — Deterministic (no LLM) verifier for whether a coding agent's green test run is real: snapshots the repo's tests before the agent runs, then reruns the agent's current code against the untouched originals — if the code fails them, the green depended on the agent's test edits, not a real fix. Self-reported: caught every constructed cheat and flagged no legitimate edit across 15 hand-built scenarios on 5 real Python repos; not a statistical benchmark. 🆕
- **[Agent-as-a-Judge: Evaluate Agents with Agents](https://arxiv.org/abs/2410.10934)** — Mingchen Zhuge et al. (Meta AI / KAUST) — <https://arxiv.org/abs/2410.10934> · *paper* — Extends LLM-as-judge to agentic trajectories—grading intermediate steps, not just final outputs—with the DevAI benchmark. The agent-specific evaluation case this agent-evals library specifically needs.
- **[VerifyBench: A Systematic Benchmark for Evaluating Reasoning Verifiers Across Domains](https://arxiv.org/abs/2507.09884)** — Various (AAAI 2026) — <https://arxiv.org/abs/2507.09884> · *benchmark* — Cross-domain benchmark exposing verifier precision/recall trade-offs (specialized verifiers high-accuracy but low-recall; general models inclusive but unstable). Quantifies how trustworthy a verifier actually is for RLVR. 🆕
- **[Enhancing LLM-as-a-Judge with Grading Notes / From Pilot to Production with Custom Judges](https://www.databricks.com/blog/pilot-production-custom-judges)** — Databricks (Mosaic Research) — <https://www.databricks.com/blog/pilot-production-custom-judges> · *blog* — Enterprise-grade judge-building playbook: 20-30 calibration examples, batched SME annotation, Krippendorff's alpha agreement gating—a production-side complement to the Hamel/Shankar academic alignment loop. 🆕
Expand Down
Loading