You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Retires the legacy benchmark authoring layer in favor of the RFC's Experiment -> Environment -> Sample path. Builtin examples now compose data loaders, explicit workers/evaluators/sandboxes, and make_*_sample(...) helpers through Environment.from_dataset(...) / Environment.from_records(...).
What changed
Added row-backed Environment.from_dataset(...) and Environment.from_records(...) constructors.
Replaced bespoke builtin environment classes with benchmark-local dataset.py, sample.py, and concrete task.py helpers.
Deleted thin worker_factory.py, _default_*_sandbox, benchmark class, and variable-alias compatibility layers.
Ported examples, smoke fixtures, real-LLM harnesses, and builtin unit tests to explicit Pythonic composition.
Renamed CLI benchmark setup surfaces to environment setup to match the public vocabulary.
Tightened the remaining cleanup polish: Pydantic preflight result, clearer row-backed test fixture names, no name-to-name aliases.
Why
The old builtin path hid the important authoring choices behind factories and bespoke environment classes. That made examples harder to inspect and conflicted with the RFC goal of having one public composition shape for users and builtins. This PR makes the examples read directly: load rows, construct worker/evaluator/sandbox objects, build samples, submit an experiment.
Review notes / gotchas
make_*_sample(...) helpers remain intentionally: they are row-to-sample builders, not thin object aliases.
load_*_rows(...) / iter_*_rows(...) remain intentionally as source helpers.
The removed environment classes are guarded by tests so they do not silently re-enter the public API.
This PR is stacked on PR 11 (codex/episode-sample-pr11-delete-definition-runtime), not directly on dev.
Closed after landing the cumulative episode/sample stack snapshot through PR132 directly onto dev in merge commit 1c22130. This PR's changes are included in dev.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Retires the legacy benchmark authoring layer in favor of the RFC's
Experiment -> Environment -> Samplepath. Builtin examples now compose data loaders, explicit workers/evaluators/sandboxes, andmake_*_sample(...)helpers throughEnvironment.from_dataset(...)/Environment.from_records(...).What changed
Environment.from_dataset(...)andEnvironment.from_records(...)constructors.dataset.py,sample.py, and concretetask.pyhelpers.worker_factory.py,_default_*_sandbox, benchmark class, and variable-alias compatibility layers.Why
The old builtin path hid the important authoring choices behind factories and bespoke environment classes. That made examples harder to inspect and conflicted with the RFC goal of having one public composition shape for users and builtins. This PR makes the examples read directly: load rows, construct worker/evaluator/sandbox objects, build samples, submit an experiment.
Review notes / gotchas
make_*_sample(...)helpers remain intentionally: they are row-to-sample builders, not thin object aliases.load_*_rows(...)/iter_*_rows(...)remain intentionally as source helpers.codex/episode-sample-pr11-delete-definition-runtime), not directly ondev.Validation
uv run pytest ergon_builtins/tests/unit/environments tests/examples ergon_core/tests/unit/api/test_environment_constructors.py ergon_builtins/tests/unit/benchmarks/test_source_sample_helpers.py ergon_core/tests/unit/read_models/test_sample_read_service.py ergon_core/tests/unit/runtime/test_execute_task_evaluator_fanout.py ergon_core/tests/unit/core/application/jobs/test_worker_execute_live_sandbox_attach.py -quv run ruff check examples/getting_started/_shared/env.py ergon_builtins/tests/unit/environments/conftest.py tests/fixtures/smoke_components/__init__.py ergon_core/ergon_core/core/application/runtime/sample_records.py ergon_core/ergon_core/core/application/samples/event_views.py ergon_core/ergon_core/core/jobs/task/worker_execute/contract.py tests/e2e/_asserts.py ergon_core/tests/unit/read_models/test_sample_read_service.py ergon_core/ergon_core/core/jobs/task/execute/job.py ergon_core/ergon_core/core/jobs/task/worker_execute/job.py ergon_core/tests/unit/core/application/jobs/test_worker_execute_live_sandbox_attach.py ergon_core/tests/unit/runtime/test_execute_task_evaluator_fanout.py