Skip to content

PR 12: Retire benchmark authoring indirection - #131

Closed
cm2435 wants to merge 15 commits into
codex/episode-sample-pr11-delete-definition-runtimefrom
codex/episode-sample-pr12-trainer-adapter
Closed

PR 12: Retire benchmark authoring indirection#131
cm2435 wants to merge 15 commits into
codex/episode-sample-pr11-delete-definition-runtimefrom
codex/episode-sample-pr12-trainer-adapter

Conversation

@cm2435

@cm2435 cm2435 commented May 26, 2026

Copy link
Copy Markdown
Contributor

Summary

Retires the legacy benchmark authoring layer in favor of the RFC's Experiment -> Environment -> Sample path. Builtin examples now compose data loaders, explicit workers/evaluators/sandboxes, and make_*_sample(...) helpers through Environment.from_dataset(...) / Environment.from_records(...).

What changed

  • Added row-backed Environment.from_dataset(...) and Environment.from_records(...) constructors.
  • Replaced bespoke builtin environment classes with benchmark-local dataset.py, sample.py, and concrete task.py helpers.
  • Deleted thin worker_factory.py, _default_*_sandbox, benchmark class, and variable-alias compatibility layers.
  • Ported examples, smoke fixtures, real-LLM harnesses, and builtin unit tests to explicit Pythonic composition.
  • Renamed CLI benchmark setup surfaces to environment setup to match the public vocabulary.
  • Tightened the remaining cleanup polish: Pydantic preflight result, clearer row-backed test fixture names, no name-to-name aliases.

Why

The old builtin path hid the important authoring choices behind factories and bespoke environment classes. That made examples harder to inspect and conflicted with the RFC goal of having one public composition shape for users and builtins. This PR makes the examples read directly: load rows, construct worker/evaluator/sandbox objects, build samples, submit an experiment.

Review notes / gotchas

  • make_*_sample(...) helpers remain intentionally: they are row-to-sample builders, not thin object aliases.
  • load_*_rows(...) / iter_*_rows(...) remain intentionally as source helpers.
  • The removed environment classes are guarded by tests so they do not silently re-enter the public API.
  • This PR is stacked on PR 11 (codex/episode-sample-pr11-delete-definition-runtime), not directly on dev.

Validation

  • uv run pytest ergon_builtins/tests/unit/environments tests/examples ergon_core/tests/unit/api/test_environment_constructors.py ergon_builtins/tests/unit/benchmarks/test_source_sample_helpers.py ergon_core/tests/unit/read_models/test_sample_read_service.py ergon_core/tests/unit/runtime/test_execute_task_evaluator_fanout.py ergon_core/tests/unit/core/application/jobs/test_worker_execute_live_sandbox_attach.py -q
  • uv run ruff check examples/getting_started/_shared/env.py ergon_builtins/tests/unit/environments/conftest.py tests/fixtures/smoke_components/__init__.py ergon_core/ergon_core/core/application/runtime/sample_records.py ergon_core/ergon_core/core/application/samples/event_views.py ergon_core/ergon_core/core/jobs/task/worker_execute/contract.py tests/e2e/_asserts.py ergon_core/tests/unit/read_models/test_sample_read_service.py ergon_core/ergon_core/core/jobs/task/execute/job.py ergon_core/ergon_core/core/jobs/task/worker_execute/job.py ergon_core/tests/unit/core/application/jobs/test_worker_execute_live_sandbox_attach.py ergon_core/tests/unit/runtime/test_execute_task_evaluator_fanout.py

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

No PNG screenshots were uploaded for this leg. See screenshots/pr-131 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

No PNG screenshots were uploaded for this leg. See screenshots/pr-131 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

No PNG screenshots were uploaded for this leg. See screenshots/pr-131 for the uploaded placeholder.

@cm2435
cm2435 force-pushed the codex/episode-sample-pr11-delete-definition-runtime branch from 8ff02fd to 0866dd6 Compare May 26, 2026 23:22
@cm2435
cm2435 force-pushed the codex/episode-sample-pr12-trainer-adapter branch from 6918036 to ed35a12 Compare May 26, 2026 23:25
@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

No PNG screenshots were uploaded for this leg. See screenshots/pr-131 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

No PNG screenshots were uploaded for this leg. See screenshots/pr-131 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

No PNG screenshots were uploaded for this leg. See screenshots/pr-131 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

No PNG screenshots were uploaded for this leg. See screenshots/pr-131 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

No PNG screenshots were uploaded for this leg. See screenshots/pr-131 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

No PNG screenshots were uploaded for this leg. See screenshots/pr-131 for the uploaded placeholder.

@cm2435
cm2435 force-pushed the codex/episode-sample-pr11-delete-definition-runtime branch from 0866dd6 to 16696c3 Compare May 26, 2026 23:34
@cm2435
cm2435 force-pushed the codex/episode-sample-pr12-trainer-adapter branch from 13658b1 to 3fd39b9 Compare May 26, 2026 23:34
@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-131.

swebench-verified/920cecd9-de3d-4f65-ada3-74564dc68a3e-activity-stack.png
swebench-verified/920cecd9-de3d-4f65-ada3-74564dc68a3e-sad.png
swebench-verified/920cecd9-de3d-4f65-ada3-74564dc68a3e-visual-debugger-full.png
swebench-verified/b7607076-dd40-495a-a87c-934f2fc521bd-activity-stack.png
swebench-verified/b7607076-dd40-495a-a87c-934f2fc521bd-happy.png
swebench-verified/b7607076-dd40-495a-a87c-934f2fc521bd-visual-debugger-full.png
swebench-verified/e15915a2-821b-43a2-aa26-5095a4027ce2-activity-stack.png
swebench-verified/e15915a2-821b-43a2-aa26-5095a4027ce2-happy.png
swebench-verified/e15915a2-821b-43a2-aa26-5095a4027ce2-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-131.

minif2f/add3443f-efd7-4e0f-908c-0b825abf658c-activity-stack.png
minif2f/add3443f-efd7-4e0f-908c-0b825abf658c-sad.png
minif2f/add3443f-efd7-4e0f-908c-0b825abf658c-visual-debugger-full.png
minif2f/c14ac509-1f72-4108-b04c-44ec9985ae97-activity-stack.png
minif2f/c14ac509-1f72-4108-b04c-44ec9985ae97-happy.png
minif2f/c14ac509-1f72-4108-b04c-44ec9985ae97-visual-debugger-full.png
minif2f/c9bdea0f-5957-4d61-b969-db137963636c-activity-stack.png
minif2f/c9bdea0f-5957-4d61-b969-db137963636c-happy.png
minif2f/c9bdea0f-5957-4d61-b969-db137963636c-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-131.

researchrubrics/5f5df76f-1b63-4b1a-ac56-a6d5bedf0950-activity-stack.png
researchrubrics/5f5df76f-1b63-4b1a-ac56-a6d5bedf0950-happy.png
researchrubrics/5f5df76f-1b63-4b1a-ac56-a6d5bedf0950-visual-debugger-full.png
researchrubrics/7c7e2b9f-ca8d-4902-bfda-36b48ac283d7-activity-stack.png
researchrubrics/7c7e2b9f-ca8d-4902-bfda-36b48ac283d7-happy.png
researchrubrics/7c7e2b9f-ca8d-4902-bfda-36b48ac283d7-visual-debugger-full.png
researchrubrics/b6d398b3-609f-4865-8b99-b95fe497a32c-activity-stack.png
researchrubrics/b6d398b3-609f-4865-8b99-b95fe497a32c-sad.png
researchrubrics/b6d398b3-609f-4865-8b99-b95fe497a32c-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-131.

researchrubrics/ad3a4e14-8bf2-4b8f-87d2-fb28f936287b-activity-stack.png
researchrubrics/ad3a4e14-8bf2-4b8f-87d2-fb28f936287b-happy.png
researchrubrics/ad3a4e14-8bf2-4b8f-87d2-fb28f936287b-visual-debugger-full.png
researchrubrics/e619deca-c0e3-4cfd-8917-2f5031a981f6-activity-stack.png
researchrubrics/e619deca-c0e3-4cfd-8917-2f5031a981f6-happy.png
researchrubrics/e619deca-c0e3-4cfd-8917-2f5031a981f6-visual-debugger-full.png
researchrubrics/f42592cc-7ae2-4d81-8ff2-6f04b15e2854-activity-stack.png
researchrubrics/f42592cc-7ae2-4d81-8ff2-6f04b15e2854-sad.png
researchrubrics/f42592cc-7ae2-4d81-8ff2-6f04b15e2854-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-131.

minif2f/385a072f-0cf7-4008-89a1-53ae43c1b1eb-activity-stack.png
minif2f/385a072f-0cf7-4008-89a1-53ae43c1b1eb-sad.png
minif2f/385a072f-0cf7-4008-89a1-53ae43c1b1eb-visual-debugger-full.png
minif2f/6dbec722-8b7b-4751-bb29-c1c42da2352f-activity-stack.png
minif2f/6dbec722-8b7b-4751-bb29-c1c42da2352f-happy.png
minif2f/6dbec722-8b7b-4751-bb29-c1c42da2352f-visual-debugger-full.png
minif2f/e6eef70d-2657-43d2-b3b0-e0d31eff5142-activity-stack.png
minif2f/e6eef70d-2657-43d2-b3b0-e0d31eff5142-happy.png
minif2f/e6eef70d-2657-43d2-b3b0-e0d31eff5142-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-131.

swebench-verified/416bf030-f1c2-437e-9663-a6aeb9acdec8-activity-stack.png
swebench-verified/416bf030-f1c2-437e-9663-a6aeb9acdec8-happy.png
swebench-verified/416bf030-f1c2-437e-9663-a6aeb9acdec8-visual-debugger-full.png
swebench-verified/d018932e-4b9e-42ad-a91e-d272a71d0737-activity-stack.png
swebench-verified/d018932e-4b9e-42ad-a91e-d272a71d0737-sad.png
swebench-verified/d018932e-4b9e-42ad-a91e-d272a71d0737-visual-debugger-full.png
swebench-verified/e1f5f066-3cb9-481c-900f-e3cb8d67f1a1-activity-stack.png
swebench-verified/e1f5f066-3cb9-481c-900f-e3cb8d67f1a1-happy.png
swebench-verified/e1f5f066-3cb9-481c-900f-e3cb8d67f1a1-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-131.

researchrubrics/67c042c4-7e50-466a-9423-d1fe0f50bb2c-activity-stack.png
researchrubrics/67c042c4-7e50-466a-9423-d1fe0f50bb2c-happy.png
researchrubrics/67c042c4-7e50-466a-9423-d1fe0f50bb2c-visual-debugger-full.png
researchrubrics/bdf9bf2f-2e4b-49a5-9c3f-76c525b2b665-activity-stack.png
researchrubrics/bdf9bf2f-2e4b-49a5-9c3f-76c525b2b665-happy.png
researchrubrics/bdf9bf2f-2e4b-49a5-9c3f-76c525b2b665-visual-debugger-full.png
researchrubrics/eaa10ba2-9f62-4c02-9f2a-b66706ac5a66-activity-stack.png
researchrubrics/eaa10ba2-9f62-4c02-9f2a-b66706ac5a66-sad.png
researchrubrics/eaa10ba2-9f62-4c02-9f2a-b66706ac5a66-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-131.

swebench-verified/2178525a-9be9-4ada-a677-85f7b5d74a87-activity-stack.png
swebench-verified/2178525a-9be9-4ada-a677-85f7b5d74a87-happy.png
swebench-verified/2178525a-9be9-4ada-a677-85f7b5d74a87-visual-debugger-full.png
swebench-verified/76790d91-2875-4ce3-8455-74083f9f2c31-activity-stack.png
swebench-verified/76790d91-2875-4ce3-8455-74083f9f2c31-sad.png
swebench-verified/76790d91-2875-4ce3-8455-74083f9f2c31-visual-debugger-full.png
swebench-verified/7f3d8744-c865-46b3-ae5d-15d185116522-activity-stack.png
swebench-verified/7f3d8744-c865-46b3-ae5d-15d185116522-happy.png
swebench-verified/7f3d8744-c865-46b3-ae5d-15d185116522-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-131.

minif2f/abdb5ef2-420b-4d9d-9966-e950d656cd79-activity-stack.png
minif2f/abdb5ef2-420b-4d9d-9966-e950d656cd79-happy.png
minif2f/abdb5ef2-420b-4d9d-9966-e950d656cd79-visual-debugger-full.png
minif2f/b02b57b7-2ba1-4545-a058-11346fae0e51-activity-stack.png
minif2f/b02b57b7-2ba1-4545-a058-11346fae0e51-sad.png
minif2f/b02b57b7-2ba1-4545-a058-11346fae0e51-visual-debugger-full.png
minif2f/d0121053-3615-46d9-9cac-32b38a0ab00a-activity-stack.png
minif2f/d0121053-3615-46d9-9cac-32b38a0ab00a-happy.png
minif2f/d0121053-3615-46d9-9cac-32b38a0ab00a-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-131.

minif2f/3fbb2e2f-75b2-4c70-bfed-4f2941ad3422-activity-stack.png
minif2f/3fbb2e2f-75b2-4c70-bfed-4f2941ad3422-sad.png
minif2f/3fbb2e2f-75b2-4c70-bfed-4f2941ad3422-visual-debugger-full.png
minif2f/5a77710c-bbe1-4409-b055-a2e0e330832c-activity-stack.png
minif2f/5a77710c-bbe1-4409-b055-a2e0e330832c-happy.png
minif2f/5a77710c-bbe1-4409-b055-a2e0e330832c-visual-debugger-full.png
minif2f/f92dcedb-9047-4c84-a140-2917989466b5-activity-stack.png
minif2f/f92dcedb-9047-4c84-a140-2917989466b5-happy.png
minif2f/f92dcedb-9047-4c84-a140-2917989466b5-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-131.

researchrubrics/49388cf5-ed5d-4bc3-a022-592718d4a106-activity-stack.png
researchrubrics/49388cf5-ed5d-4bc3-a022-592718d4a106-sad.png
researchrubrics/49388cf5-ed5d-4bc3-a022-592718d4a106-visual-debugger-full.png
researchrubrics/98a3e3a3-4700-44f3-9ab8-a148dc33b333-activity-stack.png
researchrubrics/98a3e3a3-4700-44f3-9ab8-a148dc33b333-happy.png
researchrubrics/98a3e3a3-4700-44f3-9ab8-a148dc33b333-visual-debugger-full.png
researchrubrics/ca4cffdb-16f1-44a1-b808-3ea1946890cf-activity-stack.png
researchrubrics/ca4cffdb-16f1-44a1-b808-3ea1946890cf-happy.png
researchrubrics/ca4cffdb-16f1-44a1-b808-3ea1946890cf-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-131.

swebench-verified/0dfac6f1-4493-4ff3-a97a-d0bd991d8526-activity-stack.png
swebench-verified/0dfac6f1-4493-4ff3-a97a-d0bd991d8526-sad.png
swebench-verified/0dfac6f1-4493-4ff3-a97a-d0bd991d8526-visual-debugger-full.png
swebench-verified/ab67d7da-8de2-4ffc-8842-0d897ff12772-activity-stack.png
swebench-verified/ab67d7da-8de2-4ffc-8842-0d897ff12772-happy.png
swebench-verified/ab67d7da-8de2-4ffc-8842-0d897ff12772-visual-debugger-full.png
swebench-verified/cf26f6bf-01be-46b4-8ae0-be13adfb0b1d-activity-stack.png
swebench-verified/cf26f6bf-01be-46b4-8ae0-be13adfb0b1d-happy.png
swebench-verified/cf26f6bf-01be-46b4-8ae0-be13adfb0b1d-visual-debugger-full.png

@cm2435
cm2435 force-pushed the codex/episode-sample-pr12-trainer-adapter branch from b0dc189 to 7315f89 Compare May 27, 2026 13:53
@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

No PNG screenshots were uploaded for this leg. See screenshots/pr-131 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

No PNG screenshots were uploaded for this leg. See screenshots/pr-131 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

No PNG screenshots were uploaded for this leg. See screenshots/pr-131 for the uploaded placeholder.

@cm2435
cm2435 force-pushed the codex/episode-sample-pr11-delete-definition-runtime branch from 17563ab to f9c1062 Compare May 27, 2026 14:00
@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-131.

swebench-verified/046729a0-272d-4d09-9853-244b8e8193bd-activity-stack.png
swebench-verified/046729a0-272d-4d09-9853-244b8e8193bd-happy.png
swebench-verified/046729a0-272d-4d09-9853-244b8e8193bd-visual-debugger-full.png
swebench-verified/4d8a0655-6b0d-44da-8a4d-e34cd5c196d5-activity-stack.png
swebench-verified/4d8a0655-6b0d-44da-8a4d-e34cd5c196d5-sad.png
swebench-verified/4d8a0655-6b0d-44da-8a4d-e34cd5c196d5-visual-debugger-full.png
swebench-verified/55e4a8d3-8627-4fc0-a0ae-122649e8bd58-activity-stack.png
swebench-verified/55e4a8d3-8627-4fc0-a0ae-122649e8bd58-happy.png
swebench-verified/55e4a8d3-8627-4fc0-a0ae-122649e8bd58-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-131.

minif2f/0bd9a788-ccb5-4d42-85ba-016808d17647-activity-stack.png
minif2f/0bd9a788-ccb5-4d42-85ba-016808d17647-happy.png
minif2f/0bd9a788-ccb5-4d42-85ba-016808d17647-visual-debugger-full.png
minif2f/94638ce4-481b-4e38-9d51-e614e770c66e-activity-stack.png
minif2f/94638ce4-481b-4e38-9d51-e614e770c66e-sad.png
minif2f/94638ce4-481b-4e38-9d51-e614e770c66e-visual-debugger-full.png
minif2f/cb729a3a-daa7-4fbf-815f-0a9dea9b1fac-activity-stack.png
minif2f/cb729a3a-daa7-4fbf-815f-0a9dea9b1fac-happy.png
minif2f/cb729a3a-daa7-4fbf-815f-0a9dea9b1fac-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-131.

researchrubrics/3d26d17f-5e68-40e1-9266-ed72ae6d233b-activity-stack.png
researchrubrics/3d26d17f-5e68-40e1-9266-ed72ae6d233b-sad.png
researchrubrics/3d26d17f-5e68-40e1-9266-ed72ae6d233b-visual-debugger-full.png
researchrubrics/601df8a7-6a70-4867-9e7c-b1e466477436-activity-stack.png
researchrubrics/601df8a7-6a70-4867-9e7c-b1e466477436-happy.png
researchrubrics/601df8a7-6a70-4867-9e7c-b1e466477436-visual-debugger-full.png
researchrubrics/a6f62390-8cd1-487f-9f67-3ae90e2e012e-activity-stack.png
researchrubrics/a6f62390-8cd1-487f-9f67-3ae90e2e012e-happy.png
researchrubrics/a6f62390-8cd1-487f-9f67-3ae90e2e012e-visual-debugger-full.png

@cm2435 cm2435 changed the title PR 12: Add experiment rollout trainer adapter path PR 12: Retire benchmark authoring indirection May 28, 2026
@cm2435
cm2435 force-pushed the codex/episode-sample-pr12-trainer-adapter branch 2 times, most recently from 5d708ee to 62d36f0 Compare May 28, 2026 13:53
@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-131.

swebench-verified/048c14d3-2414-4326-b4d7-b31e2d4da9b6-activity-stack.png
swebench-verified/048c14d3-2414-4326-b4d7-b31e2d4da9b6-happy.png
swebench-verified/048c14d3-2414-4326-b4d7-b31e2d4da9b6-visual-debugger-full.png
swebench-verified/2424c8eb-2c96-4af3-b302-708071409c74-activity-stack.png
swebench-verified/2424c8eb-2c96-4af3-b302-708071409c74-happy.png
swebench-verified/2424c8eb-2c96-4af3-b302-708071409c74-visual-debugger-full.png
swebench-verified/45bc4b41-092d-49f9-86bb-12dfe3abba2f-activity-stack.png
swebench-verified/45bc4b41-092d-49f9-86bb-12dfe3abba2f-sad.png
swebench-verified/45bc4b41-092d-49f9-86bb-12dfe3abba2f-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-131.

researchrubrics/1859842d-a856-40e9-bde8-1fd6ea8523c7-activity-stack.png
researchrubrics/1859842d-a856-40e9-bde8-1fd6ea8523c7-happy.png
researchrubrics/1859842d-a856-40e9-bde8-1fd6ea8523c7-visual-debugger-full.png
researchrubrics/6eeef03f-c8db-488d-8204-7af6085b8fd3-activity-stack.png
researchrubrics/6eeef03f-c8db-488d-8204-7af6085b8fd3-happy.png
researchrubrics/6eeef03f-c8db-488d-8204-7af6085b8fd3-visual-debugger-full.png
researchrubrics/942290fd-8dd3-4162-a572-4c02e374c6ee-activity-stack.png
researchrubrics/942290fd-8dd3-4162-a572-4c02e374c6ee-sad.png
researchrubrics/942290fd-8dd3-4162-a572-4c02e374c6ee-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-131.

minif2f/55590c8a-968c-4c48-a034-e75e06a4b115-activity-stack.png
minif2f/55590c8a-968c-4c48-a034-e75e06a4b115-happy.png
minif2f/55590c8a-968c-4c48-a034-e75e06a4b115-visual-debugger-full.png
minif2f/8682f309-8a15-4c9a-8874-f266732b986d-activity-stack.png
minif2f/8682f309-8a15-4c9a-8874-f266732b986d-happy.png
minif2f/8682f309-8a15-4c9a-8874-f266732b986d-visual-debugger-full.png
minif2f/f00c5fb3-4206-4d1b-a75a-fbea4688838e-activity-stack.png
minif2f/f00c5fb3-4206-4d1b-a75a-fbea4688838e-sad.png
minif2f/f00c5fb3-4206-4d1b-a75a-fbea4688838e-visual-debugger-full.png

@cm2435
cm2435 force-pushed the codex/episode-sample-pr12-trainer-adapter branch from 62d36f0 to 02990b9 Compare May 28, 2026 13:55
@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-131.

minif2f/71c0bab4-5811-4d03-92cc-f5fbeaf482a4-activity-stack.png
minif2f/71c0bab4-5811-4d03-92cc-f5fbeaf482a4-happy.png
minif2f/71c0bab4-5811-4d03-92cc-f5fbeaf482a4-visual-debugger-full.png
minif2f/d3f66300-8bd2-4f3d-85c3-47e58af94cc3-activity-stack.png
minif2f/d3f66300-8bd2-4f3d-85c3-47e58af94cc3-happy.png
minif2f/d3f66300-8bd2-4f3d-85c3-47e58af94cc3-visual-debugger-full.png
minif2f/eef9fd25-b1b1-4094-b364-6169caaa1ae0-activity-stack.png
minif2f/eef9fd25-b1b1-4094-b364-6169caaa1ae0-sad.png
minif2f/eef9fd25-b1b1-4094-b364-6169caaa1ae0-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-131.

swebench-verified/01a261b5-0db0-4de8-8489-b0aa545b8ba5-activity-stack.png
swebench-verified/01a261b5-0db0-4de8-8489-b0aa545b8ba5-happy.png
swebench-verified/01a261b5-0db0-4de8-8489-b0aa545b8ba5-visual-debugger-full.png
swebench-verified/3a83021e-918b-49de-bec4-4caef283d3ef-activity-stack.png
swebench-verified/3a83021e-918b-49de-bec4-4caef283d3ef-happy.png
swebench-verified/3a83021e-918b-49de-bec4-4caef283d3ef-visual-debugger-full.png
swebench-verified/61e7f115-1db6-40d3-b259-643b73d90844-activity-stack.png
swebench-verified/61e7f115-1db6-40d3-b259-643b73d90844-sad.png
swebench-verified/61e7f115-1db6-40d3-b259-643b73d90844-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-131.

researchrubrics/97161938-8287-4f08-812a-976b75fe6d9b-activity-stack.png
researchrubrics/97161938-8287-4f08-812a-976b75fe6d9b-happy.png
researchrubrics/97161938-8287-4f08-812a-976b75fe6d9b-visual-debugger-full.png
researchrubrics/c8f81d7f-48d8-40d6-a5f3-5ffc1f06cba9-activity-stack.png
researchrubrics/c8f81d7f-48d8-40d6-a5f3-5ffc1f06cba9-sad.png
researchrubrics/c8f81d7f-48d8-40d6-a5f3-5ffc1f06cba9-visual-debugger-full.png
researchrubrics/d3246431-f730-42c2-8432-03f859b44e36-activity-stack.png
researchrubrics/d3246431-f730-42c2-8432-03f859b44e36-happy.png
researchrubrics/d3246431-f730-42c2-8432-03f859b44e36-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-131.

minif2f/2968f231-4a2d-4d40-a404-d6224ccf1d14-activity-stack.png
minif2f/2968f231-4a2d-4d40-a404-d6224ccf1d14-happy.png
minif2f/2968f231-4a2d-4d40-a404-d6224ccf1d14-visual-debugger-full.png
minif2f/34de0d63-a526-40ec-99f1-bba0b5baf308-activity-stack.png
minif2f/34de0d63-a526-40ec-99f1-bba0b5baf308-sad.png
minif2f/34de0d63-a526-40ec-99f1-bba0b5baf308-visual-debugger-full.png
minif2f/e40c0049-71c5-47c8-a8ce-a59e9a5171fc-activity-stack.png
minif2f/e40c0049-71c5-47c8-a8ce-a59e9a5171fc-happy.png
minif2f/e40c0049-71c5-47c8-a8ce-a59e9a5171fc-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-131.

swebench-verified/2baa420b-2fb1-456d-99e5-4190028b22e0-activity-stack.png
swebench-verified/2baa420b-2fb1-456d-99e5-4190028b22e0-sad.png
swebench-verified/2baa420b-2fb1-456d-99e5-4190028b22e0-visual-debugger-full.png
swebench-verified/5947e947-2ffd-4d53-a6f3-a375b645bd16-activity-stack.png
swebench-verified/5947e947-2ffd-4d53-a6f3-a375b645bd16-happy.png
swebench-verified/5947e947-2ffd-4d53-a6f3-a375b645bd16-visual-debugger-full.png
swebench-verified/6c1bb5d4-a16b-4db3-9431-b9cd2fb3c81e-activity-stack.png
swebench-verified/6c1bb5d4-a16b-4db3-9431-b9cd2fb3c81e-happy.png
swebench-verified/6c1bb5d4-a16b-4db3-9431-b9cd2fb3c81e-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-131.

researchrubrics/12812ee0-5cc5-4a41-8bdb-80259dbef41a-activity-stack.png
researchrubrics/12812ee0-5cc5-4a41-8bdb-80259dbef41a-sad.png
researchrubrics/12812ee0-5cc5-4a41-8bdb-80259dbef41a-visual-debugger-full.png
researchrubrics/4284b6c5-6136-4cc3-98c4-831145c0c74f-activity-stack.png
researchrubrics/4284b6c5-6136-4cc3-98c4-831145c0c74f-happy.png
researchrubrics/4284b6c5-6136-4cc3-98c4-831145c0c74f-visual-debugger-full.png
researchrubrics/f412b4a8-3ba1-44de-93ff-a3977f739c68-activity-stack.png
researchrubrics/f412b4a8-3ba1-44de-93ff-a3977f739c68-happy.png
researchrubrics/f412b4a8-3ba1-44de-93ff-a3977f739c68-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-131.

minif2f/015a16b7-d370-483c-bcc3-26e5888412b3-activity-stack.png
minif2f/015a16b7-d370-483c-bcc3-26e5888412b3-happy.png
minif2f/015a16b7-d370-483c-bcc3-26e5888412b3-visual-debugger-full.png
minif2f/3908652b-cb51-4f4b-8b05-2d79554df560-activity-stack.png
minif2f/3908652b-cb51-4f4b-8b05-2d79554df560-sad.png
minif2f/3908652b-cb51-4f4b-8b05-2d79554df560-visual-debugger-full.png
minif2f/5216fc6e-dfff-4d50-b0a0-21470f58c1b1-activity-stack.png
minif2f/5216fc6e-dfff-4d50-b0a0-21470f58c1b1-happy.png
minif2f/5216fc6e-dfff-4d50-b0a0-21470f58c1b1-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-131.

researchrubrics/6ad73b8a-3f8e-4182-9b54-8a63b00f2f83-activity-stack.png
researchrubrics/6ad73b8a-3f8e-4182-9b54-8a63b00f2f83-sad.png
researchrubrics/6ad73b8a-3f8e-4182-9b54-8a63b00f2f83-visual-debugger-full.png
researchrubrics/93ce51f1-5b6c-4b18-9ec4-fb167c63d1a1-activity-stack.png
researchrubrics/93ce51f1-5b6c-4b18-9ec4-fb167c63d1a1-happy.png
researchrubrics/93ce51f1-5b6c-4b18-9ec4-fb167c63d1a1-visual-debugger-full.png
researchrubrics/e2c1d05a-b6a9-4e83-b03a-b62c83e66cc1-activity-stack.png
researchrubrics/e2c1d05a-b6a9-4e83-b03a-b62c83e66cc1-happy.png
researchrubrics/e2c1d05a-b6a9-4e83-b03a-b62c83e66cc1-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-131.

swebench-verified/0197e5da-a845-4f4d-8e73-780e531eb3d8-activity-stack.png
swebench-verified/0197e5da-a845-4f4d-8e73-780e531eb3d8-sad.png
swebench-verified/0197e5da-a845-4f4d-8e73-780e531eb3d8-visual-debugger-full.png
swebench-verified/273487d6-e983-4514-9a34-436f625da7f1-activity-stack.png
swebench-verified/273487d6-e983-4514-9a34-436f625da7f1-happy.png
swebench-verified/273487d6-e983-4514-9a34-436f625da7f1-visual-debugger-full.png
swebench-verified/8fca95e4-5f24-4824-bed0-3aca7a19cd6f-activity-stack.png
swebench-verified/8fca95e4-5f24-4824-bed0-3aca7a19cd6f-happy.png
swebench-verified/8fca95e4-5f24-4824-bed0-3aca7a19cd6f-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-131.

swebench-verified/433bf726-d97f-4a80-916c-13868a809200-activity-stack.png
swebench-verified/433bf726-d97f-4a80-916c-13868a809200-happy.png
swebench-verified/433bf726-d97f-4a80-916c-13868a809200-visual-debugger-full.png
swebench-verified/929fc9ca-3eb5-40a8-abf3-5548706ac8de-activity-stack.png
swebench-verified/929fc9ca-3eb5-40a8-abf3-5548706ac8de-sad.png
swebench-verified/929fc9ca-3eb5-40a8-abf3-5548706ac8de-visual-debugger-full.png
swebench-verified/aaf87879-b009-46b3-af01-9250cbae117a-activity-stack.png
swebench-verified/aaf87879-b009-46b3-af01-9250cbae117a-happy.png
swebench-verified/aaf87879-b009-46b3-af01-9250cbae117a-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-131.

researchrubrics/40f79de4-508a-40f5-b7a7-5650b01a9b54-activity-stack.png
researchrubrics/40f79de4-508a-40f5-b7a7-5650b01a9b54-sad.png
researchrubrics/40f79de4-508a-40f5-b7a7-5650b01a9b54-visual-debugger-full.png
researchrubrics/65cd53da-8cba-4338-a916-6348d45fe2c1-activity-stack.png
researchrubrics/65cd53da-8cba-4338-a916-6348d45fe2c1-happy.png
researchrubrics/65cd53da-8cba-4338-a916-6348d45fe2c1-visual-debugger-full.png
researchrubrics/fd7ac757-86e1-4d4c-ad81-b682328c4a33-activity-stack.png
researchrubrics/fd7ac757-86e1-4d4c-ad81-b682328c4a33-happy.png
researchrubrics/fd7ac757-86e1-4d4c-ad81-b682328c4a33-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-131.

minif2f/87805961-ce1c-4a04-ae64-e9c3bf21e7f9-activity-stack.png
minif2f/87805961-ce1c-4a04-ae64-e9c3bf21e7f9-happy.png
minif2f/87805961-ce1c-4a04-ae64-e9c3bf21e7f9-visual-debugger-full.png
minif2f/8eb37145-e603-4d50-9272-89f87e2551a5-activity-stack.png
minif2f/8eb37145-e603-4d50-9272-89f87e2551a5-sad.png
minif2f/8eb37145-e603-4d50-9272-89f87e2551a5-visual-debugger-full.png
minif2f/befee523-7438-417a-a6b3-6958496d6823-activity-stack.png
minif2f/befee523-7438-417a-a6b3-6958496d6823-happy.png
minif2f/befee523-7438-417a-a6b3-6958496d6823-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-131.

researchrubrics/1686c1cd-2e6b-4cb6-90e7-ed52d60a7faf-activity-stack.png
researchrubrics/1686c1cd-2e6b-4cb6-90e7-ed52d60a7faf-happy.png
researchrubrics/1686c1cd-2e6b-4cb6-90e7-ed52d60a7faf-visual-debugger-full.png
researchrubrics/e7d89300-71fd-44bd-b0fc-6891f9fe46aa-activity-stack.png
researchrubrics/e7d89300-71fd-44bd-b0fc-6891f9fe46aa-sad.png
researchrubrics/e7d89300-71fd-44bd-b0fc-6891f9fe46aa-visual-debugger-full.png
researchrubrics/f467e170-9251-42a6-950d-8f4e3c61c987-activity-stack.png
researchrubrics/f467e170-9251-42a6-950d-8f4e3c61c987-happy.png
researchrubrics/f467e170-9251-42a6-950d-8f4e3c61c987-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-131.

minif2f/56c0556a-9b4d-46aa-85eb-6d1435d4b042-activity-stack.png
minif2f/56c0556a-9b4d-46aa-85eb-6d1435d4b042-happy.png
minif2f/56c0556a-9b4d-46aa-85eb-6d1435d4b042-visual-debugger-full.png
minif2f/5a667ed7-9738-4b51-a2dc-cc8ed04b98bb-activity-stack.png
minif2f/5a667ed7-9738-4b51-a2dc-cc8ed04b98bb-sad.png
minif2f/5a667ed7-9738-4b51-a2dc-cc8ed04b98bb-visual-debugger-full.png
minif2f/cbb45495-b480-4fa4-8c5c-9af37405cc50-activity-stack.png
minif2f/cbb45495-b480-4fa4-8c5c-9af37405cc50-happy.png
minif2f/cbb45495-b480-4fa4-8c5c-9af37405cc50-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-131.

swebench-verified/85961a2b-5451-4074-a07b-9920cc10ab7f-activity-stack.png
swebench-verified/85961a2b-5451-4074-a07b-9920cc10ab7f-sad.png
swebench-verified/85961a2b-5451-4074-a07b-9920cc10ab7f-visual-debugger-full.png
swebench-verified/aa2d2d5e-7a96-4d64-bc24-a2b440e2c3ab-activity-stack.png
swebench-verified/aa2d2d5e-7a96-4d64-bc24-a2b440e2c3ab-happy.png
swebench-verified/aa2d2d5e-7a96-4d64-bc24-a2b440e2c3ab-visual-debugger-full.png
swebench-verified/d9ac5be3-f441-4d22-ade6-6a90f2863316-activity-stack.png
swebench-verified/d9ac5be3-f441-4d22-ade6-6a90f2863316-happy.png
swebench-verified/d9ac5be3-f441-4d22-ade6-6a90f2863316-visual-debugger-full.png

@cm2435

cm2435 commented Jun 7, 2026

Copy link
Copy Markdown
Contributor Author

Closed after landing the cumulative episode/sample stack snapshot through PR132 directly onto dev in merge commit 1c22130. This PR's changes are included in dev.

@cm2435 cm2435 closed this Jun 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant