Add general:posttrain:dev suite#193
Conversation
…ch + GPQA Diamond) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2ac3ed1d93
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "general:posttrain:dev", | ||
| ( | ||
| get_suite("math:posttrain:dev"), | ||
| "scicode", |
There was a problem hiding this comment.
Remove external eval from task suite definition
Replace "scicode" here with a real task spec (or split this into a separate external-eval workflow), because task suites are validated through validate_tasks() which only accepts entries where task_exists() or suite_exists() is true (src/olmo_eval/common/configs.py:72-101). scicode is registered as an external eval name (SciCodeExternalEval.name returns "scicode" in src/olmo_eval/evals/external/benchmarks/scicode/eval.py:79-82), not a task, so general:posttrain:dev expands to an invalid spec and cannot be executed via normal task/suite runs.
Useful? React with 👍 / 👎.
Summary
general:posttrain:devsuite combiningmath:posttrain:dev, SciCode, IFBench, and GPQA DiamondAVERAGE_OF_AVERAGESaggregation, matching sibling dev-style suitesTest plan
get_suite("general:posttrain:dev")resolves and expands correctly🤖 Generated with Claude Code