You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This PR shows the user-side API in real scripts. It moves examples away from definition persistence and launch calls toward Python composition with environments, experiments, and experiment.submit(...).
What Actually Changes
Adds example scripts under examples/experiment_api for:
MiniF2F single environment.
Mixed environments.
Streaming SWE with candidate_pool_size > k.
Curriculum sampler.
Row-dependent runtime configuration.
Moves the MiniF2F llama.cpp getting-started flow to examples/getting_started/01_minif2f_local_llamacpp/submit.py.
Replaces persist_benchmark / launch_run in that getting-started example with MiniF2FEnvironment, Experiment, and experiment.submit(...).
Keeps run.py as a compatibility delegator to submit.py.
Updates CLI catalogue so ergon examples launch minif2f-local-llamacpp points at submit.py.
Adds stable print_submission_summary(...) output with experiment_id, sampler_invocation_id, batch_id, and sample_ids.
Gotchas / Review Risks
Some simple examples print result.model_dump_json(...), while getting-started normalizes experiment_id; review vocabulary consistency here.
The curriculum sampler sorts candidates but does not itself slice to k. That matches the public sampler design, but only if the service enforces selected count correctly.
Some examples import underscored builtin helper functions for default sandbox/evaluator setup. That may be okay as example scaffolding, but it is a review point.
ERGON_REPO_ROOT is strict in the getting-started path.
Cleaned Later
run.py remains as a shim for now.
Read APIs, CLI inspection, dashboard pages, and definition deletion happen in later PRs.
Architecture allowances are expanded as this stack adds example/import paths.
Validation Notes
Review the example tests and smoke-facing getting-started script. At the time of agent review, this PR had Python lint/type passing, while frontend and smoke checks were still failing from stack-level issues.
Closed after landing the cumulative episode/sample stack snapshot through PR132 directly onto dev in merge commit 1c22130. This PR's changes are included in dev.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review Guide
Stack position: PR 07, base
codex/episode-sample-pr06-builtin-environments, headcodex/episode-sample-pr07-examples-vertical.Implements RFC Docs
04-ecosystem/03-examples.md: public examples and vertical calling flows.02-core-authoring-api.md: practicalEnvironment,Experiment, and sampler usage.03b-implementation-churn.md: CLI/examples move toward sample/experiment vocabulary.09-implementation-plan/08-pr-07-examples-vertical.md.Why This PR Exists
This PR shows the user-side API in real scripts. It moves examples away from definition persistence and launch calls toward Python composition with environments, experiments, and
experiment.submit(...).What Actually Changes
examples/experiment_apifor:candidate_pool_size > k.examples/getting_started/01_minif2f_local_llamacpp/submit.py.persist_benchmark/launch_runin that getting-started example withMiniF2FEnvironment,Experiment, andexperiment.submit(...).run.pyas a compatibility delegator tosubmit.py.ergon examples launch minif2f-local-llamacpppoints atsubmit.py.print_submission_summary(...)output withexperiment_id,sampler_invocation_id,batch_id, andsample_ids.Gotchas / Review Risks
result.model_dump_json(...), while getting-started normalizesexperiment_id; review vocabulary consistency here.k. That matches the public sampler design, but only if the service enforces selected count correctly.ERGON_REPO_ROOTis strict in the getting-started path.Cleaned Later
run.pyremains as a shim for now.Validation Notes
Review the example tests and smoke-facing getting-started script. At the time of agent review, this PR had Python lint/type passing, while frontend and smoke checks were still failing from stack-level issues.