Skip to content

Add general:posttrain:dev suite#193

Open
finbarrtimbers wants to merge 1 commit into
mainfrom
finbarr/general-posttrain-dev
Open

Add general:posttrain:dev suite#193
finbarrtimbers wants to merge 1 commit into
mainfrom
finbarr/general-posttrain-dev

Conversation

@finbarrtimbers

Copy link
Copy Markdown
Contributor

Summary

  • New general:posttrain:dev suite combining math:posttrain:dev, SciCode, IFBench, and GPQA Diamond
  • Uses AVERAGE_OF_AVERAGES aggregation, matching sibling dev-style suites
  • Expands to 12 underlying tasks

Test plan

  • get_suite("general:posttrain:dev") resolves and expands correctly
  • Pre-commit ruff format / check pass

🤖 Generated with Claude Code

…ch + GPQA Diamond) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2ac3ed1d93

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

"general:posttrain:dev",
(
get_suite("math:posttrain:dev"),
"scicode",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Remove external eval from task suite definition

Replace "scicode" here with a real task spec (or split this into a separate external-eval workflow), because task suites are validated through validate_tasks() which only accepts entries where task_exists() or suite_exists() is true (src/olmo_eval/common/configs.py:72-101). scicode is registered as an external eval name (SciCodeExternalEval.name returns "scicode" in src/olmo_eval/evals/external/benchmarks/scicode/eval.py:79-82), not a task, so general:posttrain:dev expands to an invalid spec and cannot be executed via normal task/suite runs.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant