Skip to content

PR 06: Add builtin sample environments#125

Closed
cm2435 wants to merge 1 commit into
codex/episode-sample-pr05-submit-materializefrom
codex/episode-sample-pr06-builtin-environments
Closed

PR 06: Add builtin sample environments#125
cm2435 wants to merge 1 commit into
codex/episode-sample-pr05-submit-materializefrom
codex/episode-sample-pr06-builtin-environments

Conversation

@cm2435

@cm2435 cm2435 commented May 26, 2026

Copy link
Copy Markdown
Contributor

Review Guide

Stack position: PR 06, base codex/episode-sample-pr05-submit-materialize, head codex/episode-sample-pr06-builtin-environments.

Implements RFC Docs

  • 04-ecosystem/02-builtins.md: builtin environment classes, loading, row-to-sample conversion, row-dependent components.
  • 02-core-authoring-api.md: concrete Environment -> Sample -> Task examples.
  • 09-implementation-plan/07-pr-06-builtin-environments.md.

Why This PR Exists

The new public API needs real builtin environments, not just abstract examples. This PR ports the builtin benchmark domains into environment classes that own row loading and sample construction.

What Actually Changes

  • Adds builtin environment classes under ergon_builtins/environments:
    • MiniF2FEnvironment
    • SweBenchVerifiedEnvironment
    • ResearchRubricsEnvironment
    • GDPEvalEnvironment
  • Each environment owns loading plus _sample_from_row(...) and produces Sample.from_tasks(...) with existing task classes.
  • Adds shared row-dependent resolution in _resolve.py so worker/evaluator/sandbox values can be objects or row callables.
  • Adds SWE streaming support.
  • Keeps MiniF2F, ResearchRubrics, and GDPEval materialized-only.
  • Existing benchmark modules remain as compatibility wrappers that delegate build_instances() through environment all_samples().

Gotchas / Review Risks

  • Old Benchmark classes are still present as shims. This PR ports behavior but does not delete compatibility wrappers.
  • Default loaders can hit external/Hugging Face data; tests use fake loaders to keep unit coverage deterministic.
  • SWE streaming rejects all_samples(); the other environments are materialized-only.
  • loader: Any | None is intentionally flexible but weakly typed.
  • ResearchRubrics has special handling for empty rubric criteria from row payloads.

Cleaned Later

  • Builtin benchmark wrappers and old benchmark-facing docs are removed/rewired later.
  • Examples are updated in PR 07.
  • CLI/read/dashboard consumption lands after the builtins exist.

Validation Notes

Review the environment unit tests under ergon_builtins/tests/unit/environments, especially row-dependent worker/evaluator/sandbox resolution, SWE streaming behavior, and compatibility wrapper coverage.

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

No PNG screenshots were uploaded for this leg. See screenshots/pr-125 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

No PNG screenshots were uploaded for this leg. See screenshots/pr-125 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

No PNG screenshots were uploaded for this leg. See screenshots/pr-125 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-125.

swebench-verified/0ffc7926-b85b-484a-8cbd-5deec7e0f6ad-activity-stack.png
swebench-verified/0ffc7926-b85b-484a-8cbd-5deec7e0f6ad-sad.png
swebench-verified/0ffc7926-b85b-484a-8cbd-5deec7e0f6ad-visual-debugger-full.png
swebench-verified/4ab1c1ef-2d2f-40ab-bd92-5fa5160423f6-activity-stack.png
swebench-verified/4ab1c1ef-2d2f-40ab-bd92-5fa5160423f6-happy.png
swebench-verified/4ab1c1ef-2d2f-40ab-bd92-5fa5160423f6-visual-debugger-full.png
swebench-verified/59ba5852-133e-4400-ab26-477500e8e7c9-activity-stack.png
swebench-verified/59ba5852-133e-4400-ab26-477500e8e7c9-happy.png
swebench-verified/59ba5852-133e-4400-ab26-477500e8e7c9-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-125.

minif2f/2da942ac-e395-461b-8017-2b728c059fc2-activity-stack.png
minif2f/2da942ac-e395-461b-8017-2b728c059fc2-sad.png
minif2f/2da942ac-e395-461b-8017-2b728c059fc2-visual-debugger-full.png
minif2f/bbf3609d-4e58-467e-9683-203c153add27-activity-stack.png
minif2f/bbf3609d-4e58-467e-9683-203c153add27-happy.png
minif2f/bbf3609d-4e58-467e-9683-203c153add27-visual-debugger-full.png
minif2f/da88d66b-3646-4bbd-a04b-409f7e6ab414-activity-stack.png
minif2f/da88d66b-3646-4bbd-a04b-409f7e6ab414-happy.png
minif2f/da88d66b-3646-4bbd-a04b-409f7e6ab414-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-125.

researchrubrics/1f3af6ea-4a38-4695-9b0a-5912eb367c82-activity-stack.png
researchrubrics/1f3af6ea-4a38-4695-9b0a-5912eb367c82-happy.png
researchrubrics/1f3af6ea-4a38-4695-9b0a-5912eb367c82-visual-debugger-full.png
researchrubrics/4699be71-8332-4380-b3a6-1ae5418b3003-activity-stack.png
researchrubrics/4699be71-8332-4380-b3a6-1ae5418b3003-happy.png
researchrubrics/4699be71-8332-4380-b3a6-1ae5418b3003-visual-debugger-full.png
researchrubrics/b80ab169-fb6b-44d5-b36f-c9deff44eaac-activity-stack.png
researchrubrics/b80ab169-fb6b-44d5-b36f-c9deff44eaac-sad.png
researchrubrics/b80ab169-fb6b-44d5-b36f-c9deff44eaac-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-125.

researchrubrics/95444951-3ae9-42b5-a3a1-99a6a4ea7637-activity-stack.png
researchrubrics/95444951-3ae9-42b5-a3a1-99a6a4ea7637-happy.png
researchrubrics/95444951-3ae9-42b5-a3a1-99a6a4ea7637-visual-debugger-full.png
researchrubrics/cb9c063d-ff60-4557-a588-9cb15eff008e-activity-stack.png
researchrubrics/cb9c063d-ff60-4557-a588-9cb15eff008e-happy.png
researchrubrics/cb9c063d-ff60-4557-a588-9cb15eff008e-visual-debugger-full.png
researchrubrics/ce8fe075-a2f1-40cb-b5fc-373728e0bc00-activity-stack.png
researchrubrics/ce8fe075-a2f1-40cb-b5fc-373728e0bc00-sad.png
researchrubrics/ce8fe075-a2f1-40cb-b5fc-373728e0bc00-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-125.

minif2f/1611f9a1-83d5-498b-a9b1-a4a7befcfe64-activity-stack.png
minif2f/1611f9a1-83d5-498b-a9b1-a4a7befcfe64-happy.png
minif2f/1611f9a1-83d5-498b-a9b1-a4a7befcfe64-visual-debugger-full.png
minif2f/290205de-7ec8-4489-83fd-d9f1d2c5493e-activity-stack.png
minif2f/290205de-7ec8-4489-83fd-d9f1d2c5493e-sad.png
minif2f/290205de-7ec8-4489-83fd-d9f1d2c5493e-visual-debugger-full.png
minif2f/93552d5b-7de8-4a89-b2b5-fdda54e2e2b0-activity-stack.png
minif2f/93552d5b-7de8-4a89-b2b5-fdda54e2e2b0-happy.png
minif2f/93552d5b-7de8-4a89-b2b5-fdda54e2e2b0-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-125.

swebench-verified/440c35b3-d970-4cb8-bfd3-41657c33d1f1-activity-stack.png
swebench-verified/440c35b3-d970-4cb8-bfd3-41657c33d1f1-happy.png
swebench-verified/440c35b3-d970-4cb8-bfd3-41657c33d1f1-visual-debugger-full.png
swebench-verified/e47129e1-5d0c-42f6-9696-8ee514afe715-activity-stack.png
swebench-verified/e47129e1-5d0c-42f6-9696-8ee514afe715-happy.png
swebench-verified/e47129e1-5d0c-42f6-9696-8ee514afe715-visual-debugger-full.png
swebench-verified/ec84121f-3193-46ca-ab6e-56593270c3ed-activity-stack.png
swebench-verified/ec84121f-3193-46ca-ab6e-56593270c3ed-sad.png
swebench-verified/ec84121f-3193-46ca-ab6e-56593270c3ed-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-125.

minif2f/11262902-b6c6-414e-844d-2e234fdca6e0-activity-stack.png
minif2f/11262902-b6c6-414e-844d-2e234fdca6e0-happy.png
minif2f/11262902-b6c6-414e-844d-2e234fdca6e0-visual-debugger-full.png
minif2f/b9a52cbe-9809-447b-8b8c-c7c23bd3cf10-activity-stack.png
minif2f/b9a52cbe-9809-447b-8b8c-c7c23bd3cf10-happy.png
minif2f/b9a52cbe-9809-447b-8b8c-c7c23bd3cf10-visual-debugger-full.png
minif2f/be7fba46-b611-42da-b3a3-0f6beaca1d67-activity-stack.png
minif2f/be7fba46-b611-42da-b3a3-0f6beaca1d67-sad.png
minif2f/be7fba46-b611-42da-b3a3-0f6beaca1d67-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-125.

swebench-verified/6e12e1d4-5c63-47ea-ac42-95722f2ed71a-activity-stack.png
swebench-verified/6e12e1d4-5c63-47ea-ac42-95722f2ed71a-sad.png
swebench-verified/6e12e1d4-5c63-47ea-ac42-95722f2ed71a-visual-debugger-full.png
swebench-verified/eac358d4-1c1b-42bd-b656-0fe3d95d73fc-activity-stack.png
swebench-verified/eac358d4-1c1b-42bd-b656-0fe3d95d73fc-happy.png
swebench-verified/eac358d4-1c1b-42bd-b656-0fe3d95d73fc-visual-debugger-full.png
swebench-verified/f4af2a59-cb9d-4e4d-9e8b-3e3e37d4f828-activity-stack.png
swebench-verified/f4af2a59-cb9d-4e4d-9e8b-3e3e37d4f828-happy.png
swebench-verified/f4af2a59-cb9d-4e4d-9e8b-3e3e37d4f828-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-125.

researchrubrics/b75fd425-3cd5-46a2-aa48-3b085643e5a5-activity-stack.png
researchrubrics/b75fd425-3cd5-46a2-aa48-3b085643e5a5-sad.png
researchrubrics/b75fd425-3cd5-46a2-aa48-3b085643e5a5-visual-debugger-full.png
researchrubrics/d43ccebf-e9a5-465a-a2bd-cf5b3557fe4e-activity-stack.png
researchrubrics/d43ccebf-e9a5-465a-a2bd-cf5b3557fe4e-happy.png
researchrubrics/d43ccebf-e9a5-465a-a2bd-cf5b3557fe4e-visual-debugger-full.png
researchrubrics/e5d318e7-4c34-4162-abae-91c2f97ccc1c-activity-stack.png
researchrubrics/e5d318e7-4c34-4162-abae-91c2f97ccc1c-happy.png
researchrubrics/e5d318e7-4c34-4162-abae-91c2f97ccc1c-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-125.

researchrubrics/3c09480a-8f6b-4945-ae3f-db7a1fb8e395-activity-stack.png
researchrubrics/3c09480a-8f6b-4945-ae3f-db7a1fb8e395-happy.png
researchrubrics/3c09480a-8f6b-4945-ae3f-db7a1fb8e395-visual-debugger-full.png
researchrubrics/cd59e311-b158-4192-80e1-74235946be74-activity-stack.png
researchrubrics/cd59e311-b158-4192-80e1-74235946be74-sad.png
researchrubrics/cd59e311-b158-4192-80e1-74235946be74-visual-debugger-full.png
researchrubrics/e46ba2ba-e2d5-4582-9c1f-e54483439e25-activity-stack.png
researchrubrics/e46ba2ba-e2d5-4582-9c1f-e54483439e25-happy.png
researchrubrics/e46ba2ba-e2d5-4582-9c1f-e54483439e25-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-125.

minif2f/28238446-2165-4b1c-8145-0a011f252958-activity-stack.png
minif2f/28238446-2165-4b1c-8145-0a011f252958-happy.png
minif2f/28238446-2165-4b1c-8145-0a011f252958-visual-debugger-full.png
minif2f/cc4e39b3-0604-4f48-9d6f-fc6639f972bc-activity-stack.png
minif2f/cc4e39b3-0604-4f48-9d6f-fc6639f972bc-happy.png
minif2f/cc4e39b3-0604-4f48-9d6f-fc6639f972bc-visual-debugger-full.png
minif2f/ea266b38-0033-4917-aeeb-e0ea2d49e7a8-activity-stack.png
minif2f/ea266b38-0033-4917-aeeb-e0ea2d49e7a8-sad.png
minif2f/ea266b38-0033-4917-aeeb-e0ea2d49e7a8-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-125.

swebench-verified/40e2e317-0b82-4440-8257-be0bc7381056-activity-stack.png
swebench-verified/40e2e317-0b82-4440-8257-be0bc7381056-happy.png
swebench-verified/40e2e317-0b82-4440-8257-be0bc7381056-visual-debugger-full.png
swebench-verified/ec02df56-34fa-4a8f-8a9a-b61abc5ac833-activity-stack.png
swebench-verified/ec02df56-34fa-4a8f-8a9a-b61abc5ac833-happy.png
swebench-verified/ec02df56-34fa-4a8f-8a9a-b61abc5ac833-visual-debugger-full.png
swebench-verified/f49d1bd3-d599-4d9b-9899-d202059afa26-activity-stack.png
swebench-verified/f49d1bd3-d599-4d9b-9899-d202059afa26-sad.png
swebench-verified/f49d1bd3-d599-4d9b-9899-d202059afa26-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-125.

swebench-verified/1af8a192-52cf-4a9e-990a-f8a2eb7dd2a4-activity-stack.png
swebench-verified/1af8a192-52cf-4a9e-990a-f8a2eb7dd2a4-sad.png
swebench-verified/1af8a192-52cf-4a9e-990a-f8a2eb7dd2a4-visual-debugger-full.png
swebench-verified/5bf41230-457e-4a5d-ada3-01d29e550434-activity-stack.png
swebench-verified/5bf41230-457e-4a5d-ada3-01d29e550434-happy.png
swebench-verified/5bf41230-457e-4a5d-ada3-01d29e550434-visual-debugger-full.png
swebench-verified/608078c0-42c2-40fd-8ce1-5df313b1fd31-activity-stack.png
swebench-verified/608078c0-42c2-40fd-8ce1-5df313b1fd31-happy.png
swebench-verified/608078c0-42c2-40fd-8ce1-5df313b1fd31-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-125.

minif2f/27989e8d-f9c8-4563-8cf0-148b816b9668-activity-stack.png
minif2f/27989e8d-f9c8-4563-8cf0-148b816b9668-happy.png
minif2f/27989e8d-f9c8-4563-8cf0-148b816b9668-visual-debugger-full.png
minif2f/7809cbc1-5b49-4d8f-8d84-55c53d5e8c85-activity-stack.png
minif2f/7809cbc1-5b49-4d8f-8d84-55c53d5e8c85-sad.png
minif2f/7809cbc1-5b49-4d8f-8d84-55c53d5e8c85-visual-debugger-full.png
minif2f/de91d5ff-fc14-462d-8db6-f02a2b8aa90b-activity-stack.png
minif2f/de91d5ff-fc14-462d-8db6-f02a2b8aa90b-happy.png
minif2f/de91d5ff-fc14-462d-8db6-f02a2b8aa90b-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-125.

researchrubrics/2d70c961-db6d-485a-8dfd-4f3ab21d529e-activity-stack.png
researchrubrics/2d70c961-db6d-485a-8dfd-4f3ab21d529e-happy.png
researchrubrics/2d70c961-db6d-485a-8dfd-4f3ab21d529e-visual-debugger-full.png
researchrubrics/7bdb9f0e-c85a-46c9-82ba-7b2da8e2e39b-activity-stack.png
researchrubrics/7bdb9f0e-c85a-46c9-82ba-7b2da8e2e39b-happy.png
researchrubrics/7bdb9f0e-c85a-46c9-82ba-7b2da8e2e39b-visual-debugger-full.png
researchrubrics/eca353c6-7606-4da0-99f3-68f5e322c4e5-activity-stack.png
researchrubrics/eca353c6-7606-4da0-99f3-68f5e322c4e5-sad.png
researchrubrics/eca353c6-7606-4da0-99f3-68f5e322c4e5-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-125.

minif2f/50964865-bfcc-45e6-957a-2bd53966f956-activity-stack.png
minif2f/50964865-bfcc-45e6-957a-2bd53966f956-happy.png
minif2f/50964865-bfcc-45e6-957a-2bd53966f956-visual-debugger-full.png
minif2f/cd4b996e-8d81-4248-bba5-71f39c160d27-activity-stack.png
minif2f/cd4b996e-8d81-4248-bba5-71f39c160d27-sad.png
minif2f/cd4b996e-8d81-4248-bba5-71f39c160d27-visual-debugger-full.png
minif2f/d6abed6f-8cd4-4f5e-843c-774971455af7-activity-stack.png
minif2f/d6abed6f-8cd4-4f5e-843c-774971455af7-happy.png
minif2f/d6abed6f-8cd4-4f5e-843c-774971455af7-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-125.

researchrubrics/13156809-3eca-45a8-bc8c-bb5528a8939d-activity-stack.png
researchrubrics/13156809-3eca-45a8-bc8c-bb5528a8939d-happy.png
researchrubrics/13156809-3eca-45a8-bc8c-bb5528a8939d-visual-debugger-full.png
researchrubrics/34cd2ef3-e5f0-42b9-8d4a-6bc44456a2e8-activity-stack.png
researchrubrics/34cd2ef3-e5f0-42b9-8d4a-6bc44456a2e8-sad.png
researchrubrics/34cd2ef3-e5f0-42b9-8d4a-6bc44456a2e8-visual-debugger-full.png
researchrubrics/65825842-a959-4c28-9604-d64e685169a6-activity-stack.png
researchrubrics/65825842-a959-4c28-9604-d64e685169a6-happy.png
researchrubrics/65825842-a959-4c28-9604-d64e685169a6-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-125.

swebench-verified/0b58fbf9-a9e2-4594-bd9b-ac273e39d689-activity-stack.png
swebench-verified/0b58fbf9-a9e2-4594-bd9b-ac273e39d689-happy.png
swebench-verified/0b58fbf9-a9e2-4594-bd9b-ac273e39d689-visual-debugger-full.png
swebench-verified/653e4e0e-b8d1-4560-99a7-8b06c4cfbae4-activity-stack.png
swebench-verified/653e4e0e-b8d1-4560-99a7-8b06c4cfbae4-sad.png
swebench-verified/653e4e0e-b8d1-4560-99a7-8b06c4cfbae4-visual-debugger-full.png
swebench-verified/83055462-3bd6-4231-98eb-53ac1f7725b2-activity-stack.png
swebench-verified/83055462-3bd6-4231-98eb-53ac1f7725b2-happy.png
swebench-verified/83055462-3bd6-4231-98eb-53ac1f7725b2-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-125.

minif2f/14bfd350-ccf7-4460-9dae-dcba8d19cd9c-activity-stack.png
minif2f/14bfd350-ccf7-4460-9dae-dcba8d19cd9c-sad.png
minif2f/14bfd350-ccf7-4460-9dae-dcba8d19cd9c-visual-debugger-full.png
minif2f/21743271-c433-460d-8371-6e4db1f38398-activity-stack.png
minif2f/21743271-c433-460d-8371-6e4db1f38398-happy.png
minif2f/21743271-c433-460d-8371-6e4db1f38398-visual-debugger-full.png
minif2f/49e41031-2af1-4951-ae1f-48ea959078db-activity-stack.png
minif2f/49e41031-2af1-4951-ae1f-48ea959078db-happy.png
minif2f/49e41031-2af1-4951-ae1f-48ea959078db-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-125.

swebench-verified/4208cb4b-ecff-4f76-a351-303c7e90356f-activity-stack.png
swebench-verified/4208cb4b-ecff-4f76-a351-303c7e90356f-happy.png
swebench-verified/4208cb4b-ecff-4f76-a351-303c7e90356f-visual-debugger-full.png
swebench-verified/be3e38e4-dbb0-4a95-93b1-e926bbbe151e-activity-stack.png
swebench-verified/be3e38e4-dbb0-4a95-93b1-e926bbbe151e-happy.png
swebench-verified/be3e38e4-dbb0-4a95-93b1-e926bbbe151e-visual-debugger-full.png
swebench-verified/e7e3a270-5eab-4471-a9f8-18c5829b011e-activity-stack.png
swebench-verified/e7e3a270-5eab-4471-a9f8-18c5829b011e-sad.png
swebench-verified/e7e3a270-5eab-4471-a9f8-18c5829b011e-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-125.

researchrubrics/aa092bab-9a8d-4134-a16f-884ff643c54a-activity-stack.png
researchrubrics/aa092bab-9a8d-4134-a16f-884ff643c54a-happy.png
researchrubrics/aa092bab-9a8d-4134-a16f-884ff643c54a-visual-debugger-full.png
researchrubrics/bf39f62f-cd38-4ad0-b322-13aac20e5723-activity-stack.png
researchrubrics/bf39f62f-cd38-4ad0-b322-13aac20e5723-happy.png
researchrubrics/bf39f62f-cd38-4ad0-b322-13aac20e5723-visual-debugger-full.png
researchrubrics/cb989c6e-fea7-4cbf-841c-bae5deffe888-activity-stack.png
researchrubrics/cb989c6e-fea7-4cbf-841c-bae5deffe888-sad.png
researchrubrics/cb989c6e-fea7-4cbf-841c-bae5deffe888-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-125.

swebench-verified/01b91cf7-2be4-4f23-8aa8-f4068968dd3c-activity-stack.png
swebench-verified/01b91cf7-2be4-4f23-8aa8-f4068968dd3c-happy.png
swebench-verified/01b91cf7-2be4-4f23-8aa8-f4068968dd3c-visual-debugger-full.png
swebench-verified/3966994b-8ff8-4e8c-a2c0-f7091cfab770-activity-stack.png
swebench-verified/3966994b-8ff8-4e8c-a2c0-f7091cfab770-sad.png
swebench-verified/3966994b-8ff8-4e8c-a2c0-f7091cfab770-visual-debugger-full.png
swebench-verified/dbc0ee36-4010-4f6c-83ce-9edae031e423-activity-stack.png
swebench-verified/dbc0ee36-4010-4f6c-83ce-9edae031e423-happy.png
swebench-verified/dbc0ee36-4010-4f6c-83ce-9edae031e423-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-125.

researchrubrics/3885b030-b1be-48f1-9c8d-36ae146bd0e5-activity-stack.png
researchrubrics/3885b030-b1be-48f1-9c8d-36ae146bd0e5-happy.png
researchrubrics/3885b030-b1be-48f1-9c8d-36ae146bd0e5-visual-debugger-full.png
researchrubrics/5dec0876-329c-4186-9892-dea43a764c46-activity-stack.png
researchrubrics/5dec0876-329c-4186-9892-dea43a764c46-happy.png
researchrubrics/5dec0876-329c-4186-9892-dea43a764c46-visual-debugger-full.png
researchrubrics/e9515db0-7cf4-4aeb-b085-daa1695c05f5-activity-stack.png
researchrubrics/e9515db0-7cf4-4aeb-b085-daa1695c05f5-sad.png
researchrubrics/e9515db0-7cf4-4aeb-b085-daa1695c05f5-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-125.

minif2f/3a7eda1a-a3d3-4a60-9cf8-34a9c836186f-activity-stack.png
minif2f/3a7eda1a-a3d3-4a60-9cf8-34a9c836186f-sad.png
minif2f/3a7eda1a-a3d3-4a60-9cf8-34a9c836186f-visual-debugger-full.png
minif2f/72eff0d8-eb37-41f9-9553-4a3dc2a7cd0f-activity-stack.png
minif2f/72eff0d8-eb37-41f9-9553-4a3dc2a7cd0f-happy.png
minif2f/72eff0d8-eb37-41f9-9553-4a3dc2a7cd0f-visual-debugger-full.png
minif2f/bf547d38-5082-4202-9f55-895aa003c7a8-activity-stack.png
minif2f/bf547d38-5082-4202-9f55-895aa003c7a8-happy.png
minif2f/bf547d38-5082-4202-9f55-895aa003c7a8-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-125.

minif2f/6a4a9cc6-a936-418e-a8b1-7be41c2d58e6-activity-stack.png
minif2f/6a4a9cc6-a936-418e-a8b1-7be41c2d58e6-happy.png
minif2f/6a4a9cc6-a936-418e-a8b1-7be41c2d58e6-visual-debugger-full.png
minif2f/d7df7bb6-a3cc-4875-b3d8-91a1fa9863b2-activity-stack.png
minif2f/d7df7bb6-a3cc-4875-b3d8-91a1fa9863b2-happy.png
minif2f/d7df7bb6-a3cc-4875-b3d8-91a1fa9863b2-visual-debugger-full.png
minif2f/fd8f6929-c90a-4cfb-8e5a-1f9ed0ed8e9a-activity-stack.png
minif2f/fd8f6929-c90a-4cfb-8e5a-1f9ed0ed8e9a-sad.png
minif2f/fd8f6929-c90a-4cfb-8e5a-1f9ed0ed8e9a-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-125.

swebench-verified/1a941cff-7c99-4533-98b4-f0adb2ae2728-activity-stack.png
swebench-verified/1a941cff-7c99-4533-98b4-f0adb2ae2728-sad.png
swebench-verified/1a941cff-7c99-4533-98b4-f0adb2ae2728-visual-debugger-full.png
swebench-verified/72dbe98a-85c7-49dd-8d15-9ce299210757-activity-stack.png
swebench-verified/72dbe98a-85c7-49dd-8d15-9ce299210757-happy.png
swebench-verified/72dbe98a-85c7-49dd-8d15-9ce299210757-visual-debugger-full.png
swebench-verified/d73190d5-55f1-4975-a3ed-28c04eed8a47-activity-stack.png
swebench-verified/d73190d5-55f1-4975-a3ed-28c04eed8a47-happy.png
swebench-verified/d73190d5-55f1-4975-a3ed-28c04eed8a47-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-125.

researchrubrics/23dda204-5464-4c7a-9c1a-2a3bb636ad37-activity-stack.png
researchrubrics/23dda204-5464-4c7a-9c1a-2a3bb636ad37-sad.png
researchrubrics/23dda204-5464-4c7a-9c1a-2a3bb636ad37-visual-debugger-full.png
researchrubrics/9297a257-205e-4be9-8462-5c9d6ec1c4b0-activity-stack.png
researchrubrics/9297a257-205e-4be9-8462-5c9d6ec1c4b0-happy.png
researchrubrics/9297a257-205e-4be9-8462-5c9d6ec1c4b0-visual-debugger-full.png
researchrubrics/94cef78c-5672-4269-93f7-2c2da27398c5-activity-stack.png
researchrubrics/94cef78c-5672-4269-93f7-2c2da27398c5-happy.png
researchrubrics/94cef78c-5672-4269-93f7-2c2da27398c5-visual-debugger-full.png

@cm2435
cm2435 force-pushed the codex/episode-sample-pr05-submit-materialize branch from 35dc75c to 5e5adf6 Compare May 26, 2026 23:10
@cm2435
cm2435 force-pushed the codex/episode-sample-pr06-builtin-environments branch from 525ec9e to 620b8b1 Compare May 26, 2026 23:11
@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-125.

minif2f/8b2638c6-87b7-44a5-b991-33dcf67f3d43-activity-stack.png
minif2f/8b2638c6-87b7-44a5-b991-33dcf67f3d43-sad.png
minif2f/8b2638c6-87b7-44a5-b991-33dcf67f3d43-visual-debugger-full.png
minif2f/8c3fb4c7-3a5f-45bc-8242-1dea42aa4160-activity-stack.png
minif2f/8c3fb4c7-3a5f-45bc-8242-1dea42aa4160-happy.png
minif2f/8c3fb4c7-3a5f-45bc-8242-1dea42aa4160-visual-debugger-full.png
minif2f/b8b5e7b7-c5ee-4ca9-8280-f9a575e4d6cf-activity-stack.png
minif2f/b8b5e7b7-c5ee-4ca9-8280-f9a575e4d6cf-happy.png
minif2f/b8b5e7b7-c5ee-4ca9-8280-f9a575e4d6cf-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-125.

swebench-verified/6bd24afc-b8f7-4c9f-8187-c735f32bf623-activity-stack.png
swebench-verified/6bd24afc-b8f7-4c9f-8187-c735f32bf623-sad.png
swebench-verified/6bd24afc-b8f7-4c9f-8187-c735f32bf623-visual-debugger-full.png
swebench-verified/8aa716f1-2cd3-4777-8163-318d1ffb46cf-activity-stack.png
swebench-verified/8aa716f1-2cd3-4777-8163-318d1ffb46cf-happy.png
swebench-verified/8aa716f1-2cd3-4777-8163-318d1ffb46cf-visual-debugger-full.png
swebench-verified/f57c49a0-75af-4fcf-836a-542195bb40d3-activity-stack.png
swebench-verified/f57c49a0-75af-4fcf-836a-542195bb40d3-happy.png
swebench-verified/f57c49a0-75af-4fcf-836a-542195bb40d3-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-125.

researchrubrics/50ff8a3f-2b8d-4224-a49d-dcc3f45bc9d6-activity-stack.png
researchrubrics/50ff8a3f-2b8d-4224-a49d-dcc3f45bc9d6-happy.png
researchrubrics/50ff8a3f-2b8d-4224-a49d-dcc3f45bc9d6-visual-debugger-full.png
researchrubrics/7b016ebe-89f8-48a2-b772-7bce50ca993f-activity-stack.png
researchrubrics/7b016ebe-89f8-48a2-b772-7bce50ca993f-happy.png
researchrubrics/7b016ebe-89f8-48a2-b772-7bce50ca993f-visual-debugger-full.png
researchrubrics/f8da26d9-735d-4eb5-b352-5d0e5d59afb7-activity-stack.png
researchrubrics/f8da26d9-735d-4eb5-b352-5d0e5d59afb7-sad.png
researchrubrics/f8da26d9-735d-4eb5-b352-5d0e5d59afb7-visual-debugger-full.png

@cm2435
cm2435 force-pushed the codex/episode-sample-pr06-builtin-environments branch from 620b8b1 to 4c959b2 Compare May 27, 2026 14:00
@cm2435
cm2435 force-pushed the codex/episode-sample-pr05-submit-materialize branch from 5e5adf6 to 261f3bb Compare May 27, 2026 14:00
@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

No PNG screenshots were uploaded for this leg. See screenshots/pr-125 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

No PNG screenshots were uploaded for this leg. See screenshots/pr-125 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

No PNG screenshots were uploaded for this leg. See screenshots/pr-125 for the uploaded placeholder.

1 similar comment
@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

No PNG screenshots were uploaded for this leg. See screenshots/pr-125 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

No PNG screenshots were uploaded for this leg. See screenshots/pr-125 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

No PNG screenshots were uploaded for this leg. See screenshots/pr-125 for the uploaded placeholder.

@cm2435
cm2435 force-pushed the codex/episode-sample-pr05-submit-materialize branch from 261f3bb to faa0aae Compare May 27, 2026 14:44
@cm2435
cm2435 force-pushed the codex/episode-sample-pr06-builtin-environments branch from 4c959b2 to 77f8226 Compare May 27, 2026 14:44
@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

No PNG screenshots were uploaded for this leg. See screenshots/pr-125 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

No PNG screenshots were uploaded for this leg. See screenshots/pr-125 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

No PNG screenshots were uploaded for this leg. See screenshots/pr-125 for the uploaded placeholder.

@cm2435

cm2435 commented Jun 7, 2026

Copy link
Copy Markdown
Contributor Author

Closed after landing the cumulative episode/sample stack snapshot through PR132 directly onto dev in merge commit 1c22130. This PR's changes are included in dev.

@cm2435 cm2435 closed this Jun 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant