Skip to content

Run canonical smoke tests with real benchmark sandboxes#133

Closed
cm2435 wants to merge 3 commits into
codex/episode-sample-pr13-rl-views-stackfrom
codex/episode-sample-pr14-real-sandbox-smoke
Closed

Run canonical smoke tests with real benchmark sandboxes#133
cm2435 wants to merge 3 commits into
codex/episode-sample-pr13-rl-views-stackfrom
codex/episode-sample-pr14-real-sandbox-smoke

Conversation

@cm2435

@cm2435 cm2435 commented May 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • switches canonical smoke fixtures from fake SmokePublicSandbox to each environment's real sandbox class
  • renames root/dynamic smoke tasks to realistic semantic slugs and updates dashboard smoke expectations
  • adds seeded source docs, resource handoff, coordination-thread checks, synthetic logprob telemetry, RL episode-view assertions, and one dynamic-subtask evaluator
  • exercises production builtin toolkit surfaces from deterministic smoke workers
  • adds a fast unit posture guard that rejects fake/test sandboxes in canonical smoke task snapshots
  • marks live E2E smoke drivers xfail for real bugs exposed by the tighter posture rather than softening implementation logic

Known xfails

  • private E2B templates may be unavailable in the executing account
  • public object-bound sandbox calls do not currently emit command WAL like the old smoke-only InstrumentedSandbox path
  • the current E2B detach path calls an SDK close() method not present on the locally installed AsyncSandbox object

Verification

  • uv run ruff check tests/unit/smoke_base/test_smoke_real_sandbox_posture.py tests/e2e/_asserts.py tests/e2e/test_researchrubrics_smoke.py tests/e2e/test_minif2f_smoke.py tests/e2e/test_swebench_smoke.py tests/fixtures/smoke_components/smoke_base tests/fixtures/smoke_components/workers tests/fixtures/smoke_components/criteria tests/fixtures/smoke_components/benchmarks.py
  • uv run ruff format --check tests/e2e/_asserts.py tests/fixtures/smoke_components/smoke_base/dynamic_tasks.py tests/fixtures/smoke_components/smoke_base/handoff_evaluator.py tests/unit/smoke_base/test_smoke_real_sandbox_posture.py
  • uv run pytest tests/unit/smoke_base/test_smoke_real_sandbox_posture.py ergon_core/tests/unit/architecture/test_package_test_layout.py -q
  • ERGON_DATABASE_URL=postgresql://ergon:ergon_dev@localhost:5433/ergon DATABASE_URL=postgresql://ergon:ergon_dev@localhost:5433/ergon INNGEST_API_BASE_URL=http://localhost:8289 ERGON_API_BASE_URL=http://127.0.0.1:9000 uv run pytest tests/e2e/test_researchrubrics_smoke.py::test_smoke_experiment_group tests/e2e/test_minif2f_smoke.py::test_smoke_experiment_group tests/e2e/test_swebench_smoke.py::test_smoke_experiment_group -q # 3 xfailed

@cm2435
cm2435 force-pushed the codex/episode-sample-pr14-real-sandbox-smoke branch from 8430a16 to d93f04c Compare May 29, 2026 10:55
@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

No PNG screenshots were uploaded for this leg. See screenshots/pr-133 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

No PNG screenshots were uploaded for this leg. See screenshots/pr-133 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

No PNG screenshots were uploaded for this leg. See screenshots/pr-133 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — swebench-verified

No PNG screenshots were uploaded for this leg. See screenshots/pr-133 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — minif2f

No PNG screenshots were uploaded for this leg. See screenshots/pr-133 for the uploaded placeholder.

@github-actions

Copy link
Copy Markdown

E2E smoke — researchrubrics

No PNG screenshots were uploaded for this leg. See screenshots/pr-133 for the uploaded placeholder.

@github-actions

github-actions Bot commented Jun 1, 2026

Copy link
Copy Markdown

E2E smoke — researchrubrics

No PNG screenshots were uploaded for this leg. See screenshots/pr-133 for the uploaded placeholder.

@github-actions

github-actions Bot commented Jun 1, 2026

Copy link
Copy Markdown

E2E smoke — swebench-verified

No PNG screenshots were uploaded for this leg. See screenshots/pr-133 for the uploaded placeholder.

@github-actions

github-actions Bot commented Jun 1, 2026

Copy link
Copy Markdown

E2E smoke — minif2f

No PNG screenshots were uploaded for this leg. See screenshots/pr-133 for the uploaded placeholder.

@cm2435

cm2435 commented Jun 7, 2026

Copy link
Copy Markdown
Contributor Author

Closed as superseded by the new testing-posture cleanup stack: PR0 public sandbox cleanup plus PR1 stricter smoke/typed-WAL/RL assertions, to be rebased on the updated dev stack snapshot.

@cm2435 cm2435 closed this Jun 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant