Skip to content

Replace RL trajectory extraction with episode views - #132

Merged
cm2435 merged 76 commits into
devfrom
codex/episode-sample-pr13-rl-views-stack
Jun 7, 2026
Merged

Replace RL trajectory extraction with episode views#132
cm2435 merged 76 commits into
devfrom
codex/episode-sample-pr13-rl-views-stack

Conversation

@cm2435

@cm2435 cm2435 commented May 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Replace legacy RL trajectory extraction with sample-scoped RL episode/read views that derive task/spawn lineage from the task tree and worker bindings.
  • Add actor-state, episode, and projection services for RL consumers, with minimal DTOs shared across the view layer.
  • Change rollout polling to emit training records instead of legacy trajectories, and update the TRL adapter/example to demonstrate single-agent TRL where subagent spawns are treated as environment stochasticity.
  • Regenerate REST OpenAPI/contracts for the new rollout response shape.

Stack-shaped review map

PR1: RL view contracts and actor state

  • Adds ergon_core.core.views.rl.models plus actor-state read support.
  • Tests: ergon_core/tests/unit/views/rl/test_actor_state.py.

PR2: Episode tree reconstruction

  • Adds RlEpisodeReadService for sample-level episode trees, attempts, steps, task parentage, worker binding, actor identity, and experiment/environment lineage.
  • Tests: ergon_core/tests/unit/views/rl/test_episode_view.py and ergon_core/tests/integration/rl/test_rl_episode_view_roundtrip.py.

PR3: RL training projections

  • Adds RlProjectionService that normalizes sample-level reward and emits per-attempt training records from the episode view.
  • Tests: ergon_core/tests/unit/views/rl/test_projections.py.

PR4: Rollout API integration

  • Deletes core/rl/extraction.py and routes rollout polling through the episode/projection services.
  • Updates generated REST contracts to expose trainingRecords.
  • Tests: ergon_core/tests/unit/rl/test_rollout_training_records.py, ergon_core/tests/unit/rest_api/test_rollouts_di.py, and architecture guard updates.

PR5: TRL example adapter

  • Updates ergon_infra.adapters.trl_http for training-record payloads and moves the runnable TRL script into examples/training.
  • Leaves ergon_infra.training.trl_runner as a deprecation shim only.
  • Tests: ergon_infra/tests/unit/test_training_experiment_submission.py and tests/examples/test_trl_single_agent_stochastic_imports.py.

Local verification

  • uv run pytest ergon_core/tests/unit/views/rl ergon_core/tests/integration/rl ergon_core/tests/unit/rl ergon_core/tests/unit/rest_api/test_rollouts_di.py ergon_core/tests/unit/architecture/test_no_legacy_compatibility.py ergon_infra/tests/unit/test_training_experiment_submission.py tests/examples/test_trl_single_agent_stochastic_imports.py -q -> 40 passed
  • uv run ergon test full unit -- --cov=ergon_core --cov=ergon_builtins --cov-report=xml:coverage.xml -> 933 passed, 1 skipped
  • uv run ruff check ergon_core ergon_builtins ergon_cli ergon_infra ergon_ingestion tests -> passed
  • uv run ruff format --check ergon_core ergon_builtins ergon_cli ergon_infra ergon_ingestion tests -> passed
  • uv run ty check ergon_core/ergon_core ergon_builtins/ergon_builtins ergon_cli/ergon_cli ergon_infra ergon_ingestion/ergon_ingestion -> passed
  • uv run slopcop --warn-only ergon_core/ergon_core ergon_builtins/ergon_builtins ergon_cli/ergon_cli ergon_infra ergon_ingestion/ergon_ingestion -> passed with warnings only
  • uv run python -m ergon_core.core.application.testing.suppression_budget -> passed
  • uv run xenon --max-absolute F --max-modules E --max-average C ergon_core ergon_builtins ergon_cli ergon_infra ergon_ingestion -> passed
  • pnpm -C ergon-dashboard run generate:contracts:rest -> passed
  • pnpm -C ergon-dashboard run generate:contracts -> passed
  • pnpm -C ergon-dashboard run lint -> passed
  • pnpm -C ergon-dashboard run typecheck -> passed
  • pnpm -C ergon-dashboard run test:unit -> 135 passed

Notes

  • Broader tests/examples -q still includes existing failures outside this stack; the new TRL example import test passes in isolation and with the focused stack suite.

cm2435 added 30 commits May 27, 2026 12:46
@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-132.

minif2f/11f60dd2-5a3c-45f2-a5e5-6e387fe17e7a-activity-stack.png
minif2f/11f60dd2-5a3c-45f2-a5e5-6e387fe17e7a-happy.png
minif2f/11f60dd2-5a3c-45f2-a5e5-6e387fe17e7a-visual-debugger-full.png
minif2f/5a9a24ff-9de3-4bdc-a605-ca7f5473b261-activity-stack.png
minif2f/5a9a24ff-9de3-4bdc-a605-ca7f5473b261-happy.png
minif2f/5a9a24ff-9de3-4bdc-a605-ca7f5473b261-visual-debugger-full.png
minif2f/c654765b-e7ee-4a8d-a976-8deceb887544-activity-stack.png
minif2f/c654765b-e7ee-4a8d-a976-8deceb887544-sad.png
minif2f/c654765b-e7ee-4a8d-a976-8deceb887544-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-132.

swebench-verified/402cbd17-1871-47cd-8410-3c82059a0de3-activity-stack.png
swebench-verified/402cbd17-1871-47cd-8410-3c82059a0de3-happy.png
swebench-verified/402cbd17-1871-47cd-8410-3c82059a0de3-visual-debugger-full.png
swebench-verified/c0df4ef5-7eb4-4702-b47b-9939c2571f18-activity-stack.png
swebench-verified/c0df4ef5-7eb4-4702-b47b-9939c2571f18-happy.png
swebench-verified/c0df4ef5-7eb4-4702-b47b-9939c2571f18-visual-debugger-full.png
swebench-verified/f6c0aa64-ae8a-4239-8ee8-eda64daad283-activity-stack.png
swebench-verified/f6c0aa64-ae8a-4239-8ee8-eda64daad283-sad.png
swebench-verified/f6c0aa64-ae8a-4239-8ee8-eda64daad283-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-132.

researchrubrics/1303a1a1-09fa-47c5-af1d-fd24c5796bc8-activity-stack.png
researchrubrics/1303a1a1-09fa-47c5-af1d-fd24c5796bc8-sad.png
researchrubrics/1303a1a1-09fa-47c5-af1d-fd24c5796bc8-visual-debugger-full.png
researchrubrics/a351a4a4-b980-4483-8761-ef47e9606057-activity-stack.png
researchrubrics/a351a4a4-b980-4483-8761-ef47e9606057-happy.png
researchrubrics/a351a4a4-b980-4483-8761-ef47e9606057-visual-debugger-full.png
researchrubrics/e0355a70-87ef-4f59-ac03-921e25d93e9c-activity-stack.png
researchrubrics/e0355a70-87ef-4f59-ac03-921e25d93e9c-happy.png
researchrubrics/e0355a70-87ef-4f59-ac03-921e25d93e9c-visual-debugger-full.png

@cm2435
cm2435 marked this pull request as ready for review May 28, 2026 19:03
@cm2435
cm2435 changed the base branch from codex/episode-sample-pr12-trainer-adapter to dev June 7, 2026 18:37
@cm2435
cm2435 merged commit 1c22130 into dev Jun 7, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant