You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The new public API needs real builtin environments, not just abstract examples. This PR ports the builtin benchmark domains into environment classes that own row loading and sample construction.
What Actually Changes
Adds builtin environment classes under ergon_builtins/environments:
MiniF2FEnvironment
SweBenchVerifiedEnvironment
ResearchRubricsEnvironment
GDPEvalEnvironment
Each environment owns loading plus _sample_from_row(...) and produces Sample.from_tasks(...) with existing task classes.
Adds shared row-dependent resolution in _resolve.py so worker/evaluator/sandbox values can be objects or row callables.
Adds SWE streaming support.
Keeps MiniF2F, ResearchRubrics, and GDPEval materialized-only.
Existing benchmark modules remain as compatibility wrappers that delegate build_instances() through environment all_samples().
Gotchas / Review Risks
Old Benchmark classes are still present as shims. This PR ports behavior but does not delete compatibility wrappers.
Default loaders can hit external/Hugging Face data; tests use fake loaders to keep unit coverage deterministic.
SWE streaming rejects all_samples(); the other environments are materialized-only.
loader: Any | None is intentionally flexible but weakly typed.
ResearchRubrics has special handling for empty rubric criteria from row payloads.
Cleaned Later
Builtin benchmark wrappers and old benchmark-facing docs are removed/rewired later.
Examples are updated in PR 07.
CLI/read/dashboard consumption lands after the builtins exist.
Validation Notes
Review the environment unit tests under ergon_builtins/tests/unit/environments, especially row-dependent worker/evaluator/sandbox resolution, SWE streaming behavior, and compatibility wrapper coverage.
Closed after landing the cumulative episode/sample stack snapshot through PR132 directly onto dev in merge commit 1c22130. This PR's changes are included in dev.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review Guide
Stack position: PR 06, base
codex/episode-sample-pr05-submit-materialize, headcodex/episode-sample-pr06-builtin-environments.Implements RFC Docs
04-ecosystem/02-builtins.md: builtin environment classes, loading, row-to-sample conversion, row-dependent components.02-core-authoring-api.md: concrete Environment -> Sample -> Task examples.09-implementation-plan/07-pr-06-builtin-environments.md.Why This PR Exists
The new public API needs real builtin environments, not just abstract examples. This PR ports the builtin benchmark domains into environment classes that own row loading and sample construction.
What Actually Changes
ergon_builtins/environments:MiniF2FEnvironmentSweBenchVerifiedEnvironmentResearchRubricsEnvironmentGDPEvalEnvironment_sample_from_row(...)and producesSample.from_tasks(...)with existing task classes._resolve.pyso worker/evaluator/sandbox values can be objects or row callables.build_instances()through environmentall_samples().Gotchas / Review Risks
all_samples(); the other environments are materialized-only.loader: Any | Noneis intentionally flexible but weakly typed.Cleaned Later
Validation Notes
Review the environment unit tests under
ergon_builtins/tests/unit/environments, especially row-dependent worker/evaluator/sandbox resolution, SWE streaming behavior, and compatibility wrapper coverage.