Skip to content

[codex] Use sample memberships for rollout batches#121

Merged
cm2435 merged 14 commits into
devfrom
codex/episode-sample-pr02-rollout-batch-samples
Jun 7, 2026
Merged

[codex] Use sample memberships for rollout batches#121
cm2435 merged 14 commits into
devfrom
codex/episode-sample-pr02-rollout-batch-samples

Conversation

@cm2435

@cm2435 cm2435 commented May 25, 2026

Copy link
Copy Markdown
Contributor

Review Guide

Stack position: PR 02, base codex/episode-sample-pr01-typed-wal, head codex/episode-sample-pr02-rollout-batch-samples.

Implements RFC Docs

  • 03a-persistence-schemas.md: rollout tables.
  • 03b-implementation-churn.md: prereq PR 02, rollout batch samples.
  • 03c-postgres-table-inventory.md: rollout batch table inventory.
  • 09-implementation-plan/03-pr-02-rollout-batch-samples.md.

Why This PR Exists

Trainer batch membership needs to point at concrete samples, not old run/definition membership. This creates the batch/sample join shape that later sampler and candidate-pool work can use directly.

What Actually Changes

  • RolloutBatch drops definition_id and gains nullable experiment_id and sampler_invocation_id.
  • Replaces RolloutBatchRun with RolloutBatchSampleMembership.
  • Adds membership table rollout_batch_sample_memberships with composite primary key (batch_id, sample_id), plus ordinal, nullable environment_id, and nullable pool_entry_id.
  • Updates RolloutService to write memberships, read sample IDs via _batch_sample_ids(...), preserve order by (ordinal, sample_id), and expose create/get helpers.
  • Adds RolloutBatchSummary DTO.
  • Adds GET /rollouts/batches/{batch_id}.

Gotchas / Review Risks

  • The RFC text says rollout_batch_samples; the implementation uses rollout_batch_sample_memberships. Worth reviewing the table name before downstream PRs depend on it.
  • RolloutService.submit(...) still accepts definition_id and creates definition-backed SampleRecords. This PR removes RolloutBatch.definition_id, not the old trainer submit input.
  • experiment_id and sampler_invocation_id are nullable bridges here. Real sampler/candidate-pool linkage lands later.
  • There is no rollout-specific standalone Alembic revision in this slice; existing-DB migration expectations should be read in light of the stack’s destructive migration posture.
  • If GitHub shows inherited stack churn in this PR, the rollout-specific review surface is the rollout service, DTO, telemetry model, route, and tests.

Cleaned Later

  • PR 04 adds persisted experiment candidate pools and sampler invocations.
  • PR 12 connects trainer rollout submission to persisted experiment candidate pools.
  • PR 11 removes remaining definition-backed runtime launch paths.

Validation Notes

Key coverage includes test_rollout_batch_sample_memberships.py, rollout membership architecture guards, rollout service/API tests, ordered sample_ids, nullable experiment/sampler provenance, and /rollouts/batches/{batch_id} responses.

@cm2435
cm2435 force-pushed the codex/episode-sample-pr01-typed-wal branch from 4f37674 to c569c09 Compare May 25, 2026 20:47
@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@cm2435
cm2435 force-pushed the codex/episode-sample-pr02-rollout-batch-samples branch 2 times, most recently from 3c9a161 to 55ced39 Compare May 25, 2026 20:52
@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@cm2435
cm2435 force-pushed the codex/episode-sample-pr01-typed-wal branch from 5e77817 to 86b810f Compare May 25, 2026 20:54
@cm2435
cm2435 force-pushed the codex/episode-sample-pr02-rollout-batch-samples branch from 55ced39 to 36a78ab Compare May 25, 2026 20:56
@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@cm2435
cm2435 force-pushed the codex/episode-sample-pr02-rollout-batch-samples branch from 36a78ab to 13dc990 Compare May 25, 2026 21:02
@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@cm2435
cm2435 force-pushed the codex/episode-sample-pr01-typed-wal branch from d8705b1 to 8cd4955 Compare May 25, 2026 21:08
@cm2435
cm2435 force-pushed the codex/episode-sample-pr02-rollout-batch-samples branch 2 times, most recently from 38d3963 to 17c3850 Compare May 25, 2026 21:13
@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@cm2435
cm2435 force-pushed the codex/episode-sample-pr01-typed-wal branch from 3d96912 to b048da1 Compare May 25, 2026 21:16
@cm2435
cm2435 force-pushed the codex/episode-sample-pr02-rollout-batch-samples branch from 17c3850 to d124fd5 Compare May 25, 2026 21:16
@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

No PNG screenshots were uploaded for this leg. See screenshots/pr-121 for the uploaded placeholder.

@cm2435
cm2435 force-pushed the codex/episode-sample-pr02-rollout-batch-samples branch from b59496f to 87c0c5a Compare May 25, 2026 21:48
@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-121.

minif2f/717e184a-20f5-4958-b404-5bb075549016-activity-stack.png
minif2f/717e184a-20f5-4958-b404-5bb075549016-happy.png
minif2f/717e184a-20f5-4958-b404-5bb075549016-visual-debugger-full.png
minif2f/773c9863-9a8d-4c46-b580-cdfe59307719-activity-stack.png
minif2f/773c9863-9a8d-4c46-b580-cdfe59307719-happy.png
minif2f/773c9863-9a8d-4c46-b580-cdfe59307719-visual-debugger-full.png
minif2f/f4f1e047-8f0e-481c-8fbc-87287105a71d-activity-stack.png
minif2f/f4f1e047-8f0e-481c-8fbc-87287105a71d-sad.png
minif2f/f4f1e047-8f0e-481c-8fbc-87287105a71d-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-121.

swebench-verified/3beee8d6-30f7-4939-9241-b83d375b1191-activity-stack.png
swebench-verified/3beee8d6-30f7-4939-9241-b83d375b1191-happy.png
swebench-verified/3beee8d6-30f7-4939-9241-b83d375b1191-visual-debugger-full.png
swebench-verified/8c9c6421-c85f-4bc8-b270-6ccf5a4df130-activity-stack.png
swebench-verified/8c9c6421-c85f-4bc8-b270-6ccf5a4df130-sad.png
swebench-verified/8c9c6421-c85f-4bc8-b270-6ccf5a4df130-visual-debugger-full.png
swebench-verified/d769c167-1c68-4151-8a4e-54d626eb8e62-activity-stack.png
swebench-verified/d769c167-1c68-4151-8a4e-54d626eb8e62-happy.png
swebench-verified/d769c167-1c68-4151-8a4e-54d626eb8e62-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-121.

researchrubrics/a2b447dc-0d5e-4ea0-8cf3-48913eafb152-activity-stack.png
researchrubrics/a2b447dc-0d5e-4ea0-8cf3-48913eafb152-happy.png
researchrubrics/a2b447dc-0d5e-4ea0-8cf3-48913eafb152-visual-debugger-full.png
researchrubrics/b18b2276-87e5-48eb-a253-2f997d0f37c5-activity-stack.png
researchrubrics/b18b2276-87e5-48eb-a253-2f997d0f37c5-sad.png
researchrubrics/b18b2276-87e5-48eb-a253-2f997d0f37c5-visual-debugger-full.png
researchrubrics/cdab7938-fc3c-4a27-a497-547cd9f49d6b-activity-stack.png
researchrubrics/cdab7938-fc3c-4a27-a497-547cd9f49d6b-happy.png
researchrubrics/cdab7938-fc3c-4a27-a497-547cd9f49d6b-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-121.

swebench-verified/11111eac-09d8-4e72-bfcc-1e2932d94fa9-activity-stack.png
swebench-verified/11111eac-09d8-4e72-bfcc-1e2932d94fa9-sad.png
swebench-verified/11111eac-09d8-4e72-bfcc-1e2932d94fa9-visual-debugger-full.png
swebench-verified/23e79d9e-496e-4062-b1c9-bb6e84a971cc-activity-stack.png
swebench-verified/23e79d9e-496e-4062-b1c9-bb6e84a971cc-happy.png
swebench-verified/23e79d9e-496e-4062-b1c9-bb6e84a971cc-visual-debugger-full.png
swebench-verified/41297e2a-040d-45a7-9298-7c6f2a356bfa-activity-stack.png
swebench-verified/41297e2a-040d-45a7-9298-7c6f2a356bfa-happy.png
swebench-verified/41297e2a-040d-45a7-9298-7c6f2a356bfa-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-121.

minif2f/0c74ad0f-1f7b-4761-acff-935a775522b4-activity-stack.png
minif2f/0c74ad0f-1f7b-4761-acff-935a775522b4-sad.png
minif2f/0c74ad0f-1f7b-4761-acff-935a775522b4-visual-debugger-full.png
minif2f/38973f85-1191-4977-927f-b0828e82e10d-activity-stack.png
minif2f/38973f85-1191-4977-927f-b0828e82e10d-happy.png
minif2f/38973f85-1191-4977-927f-b0828e82e10d-visual-debugger-full.png
minif2f/4a7360cc-ddcf-4189-9128-7bcf203f789d-activity-stack.png
minif2f/4a7360cc-ddcf-4189-9128-7bcf203f789d-happy.png
minif2f/4a7360cc-ddcf-4189-9128-7bcf203f789d-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-121.

researchrubrics/09cd7fe9-65ef-4e3e-84a9-b96188bfb373-activity-stack.png
researchrubrics/09cd7fe9-65ef-4e3e-84a9-b96188bfb373-happy.png
researchrubrics/09cd7fe9-65ef-4e3e-84a9-b96188bfb373-visual-debugger-full.png
researchrubrics/f17f0bac-de0d-4b7f-b2e2-1993e7c1b8f1-activity-stack.png
researchrubrics/f17f0bac-de0d-4b7f-b2e2-1993e7c1b8f1-sad.png
researchrubrics/f17f0bac-de0d-4b7f-b2e2-1993e7c1b8f1-visual-debugger-full.png
researchrubrics/ffcb645d-0657-41b8-9f0e-e739e78f09a8-activity-stack.png
researchrubrics/ffcb645d-0657-41b8-9f0e-e739e78f09a8-happy.png
researchrubrics/ffcb645d-0657-41b8-9f0e-e739e78f09a8-visual-debugger-full.png

@cm2435
cm2435 force-pushed the codex/episode-sample-pr02-rollout-batch-samples branch from 3566a4d to 248542a Compare May 27, 2026 14:00
@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

Screenshots pushed to screenshots/pr-121.

swebench-verified/1b2c25af-0963-4900-8afc-8841a1cc2358-activity-stack.png
swebench-verified/1b2c25af-0963-4900-8afc-8841a1cc2358-happy.png
swebench-verified/1b2c25af-0963-4900-8afc-8841a1cc2358-visual-debugger-full.png
swebench-verified/5df68898-89ad-4c89-8297-54b94dbfea70-activity-stack.png
swebench-verified/5df68898-89ad-4c89-8297-54b94dbfea70-happy.png
swebench-verified/5df68898-89ad-4c89-8297-54b94dbfea70-visual-debugger-full.png
swebench-verified/63e3c817-57dd-4d70-94d3-be90076f0bc7-activity-stack.png
swebench-verified/63e3c817-57dd-4d70-94d3-be90076f0bc7-sad.png
swebench-verified/63e3c817-57dd-4d70-94d3-be90076f0bc7-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

Screenshots pushed to screenshots/pr-121.

researchrubrics/1918f169-085b-4b18-a39c-c9f5a81b498d-activity-stack.png
researchrubrics/1918f169-085b-4b18-a39c-c9f5a81b498d-happy.png
researchrubrics/1918f169-085b-4b18-a39c-c9f5a81b498d-visual-debugger-full.png
researchrubrics/8b482961-34af-454d-abdb-a8915c03b47c-activity-stack.png
researchrubrics/8b482961-34af-454d-abdb-a8915c03b47c-happy.png
researchrubrics/8b482961-34af-454d-abdb-a8915c03b47c-visual-debugger-full.png
researchrubrics/b7bc7d27-e577-4279-b073-b1b58a96c92d-activity-stack.png
researchrubrics/b7bc7d27-e577-4279-b073-b1b58a96c92d-sad.png
researchrubrics/b7bc7d27-e577-4279-b073-b1b58a96c92d-visual-debugger-full.png

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

Screenshots pushed to screenshots/pr-121.

minif2f/06396278-589e-47f5-ba22-57145b84315d-activity-stack.png
minif2f/06396278-589e-47f5-ba22-57145b84315d-happy.png
minif2f/06396278-589e-47f5-ba22-57145b84315d-visual-debugger-full.png
minif2f/92c8d306-105a-44e3-b1ef-480ff43260bc-activity-stack.png
minif2f/92c8d306-105a-44e3-b1ef-480ff43260bc-happy.png
minif2f/92c8d306-105a-44e3-b1ef-480ff43260bc-visual-debugger-full.png
minif2f/d6b5c184-4081-40cc-8a1a-1cd0e8f0e9bc-activity-stack.png
minif2f/d6b5c184-4081-40cc-8a1a-1cd0e8f0e9bc-sad.png
minif2f/d6b5c184-4081-40cc-8a1a-1cd0e8f0e9bc-visual-debugger-full.png

@cm2435
cm2435 changed the base branch from codex/episode-sample-pr01-typed-wal to dev June 7, 2026 18:36
@cm2435
cm2435 marked this pull request as ready for review June 7, 2026 18:36
@cm2435
cm2435 merged commit 248542a into dev Jun 7, 2026
6 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant