From f6e41fb323dab0c65f43f3610f01735433bc2a40 Mon Sep 17 00:00:00 2001 From: Artur Jonkisz Date: Sun, 21 Jun 2026 00:43:51 +0100 Subject: [PATCH 1/4] Document experiment runner smoke test Co-authored-by: bobbit-ai --- docs/experiment-runner-smoke-test.md | 215 +++++++++++++++++++++++++++ 1 file changed, 215 insertions(+) create mode 100644 docs/experiment-runner-smoke-test.md diff --git a/docs/experiment-runner-smoke-test.md b/docs/experiment-runner-smoke-test.md new file mode 100644 index 000000000..0fa481dd0 --- /dev/null +++ b/docs/experiment-runner-smoke-test.md @@ -0,0 +1,215 @@ +# Experiment Runner smoke test + +Date: 2026-06-21 +Gateway: `http://127.0.0.1:3005` +Parent goal: `cdb260f0-96fe-4350-8093-bfe2a0008477` +Parent session: `22bd1fb5-fd2f-4319-bfa3-cb838bbcebec` +Experiment id: `smoke-ab-20260621003841` + +## Result + +**Status: pass with caveats.** The runner is ready for a small real Graphify A/B benchmark, provided the real run uses a realistic per-run budget and an intentional workflow choice. The smoke test verified pack activation, UI launchers, A/B child-goal fan-out, treatment metadata propagation, route lifecycle calls, editable metric/dashboard specs, report generation, and autoresearch guardrails. + +Caveats: +- The intentionally low per-run budget (`0.05`) was below actual model startup cost, so both smoke arms were marked `failed` with `over_budget` during `poll`. This still verified bounded-budget enforcement and report generation, but did not produce collected success metrics. +- The experiment definition intentionally omitted `workflowId`. The spawned child goals received the project default workflow `general`; no custom experiment workflow was required and no workflow route error occurred. +- After evidence capture, both child teams were torn down with `POST /api/goals/:id/team/teardown?cascade=false` to stop further spend. + +## Pack and route activation + +`GET /api/ext/contributions?projectId=762fef1c-5a69-469e-a5da-1b85862668c2` returned active pack contribution metadata for `experiment-runner`: + +- Panel: `experiment-runner.panel` (`Experiments`) +- Session menu entrypoint: `experiment-runner.palette`, label `New experiment` +- Composer slash entrypoint: `experiment-runner.open`, label `Experiments` +- Deep-link route entrypoint: `experiment-runner.route`, `routeId: experiment-runner`, params `experimentId`, `view` +- Route names: `defineExperiment`, `projectCost`, `launch`, `poll`, `collect`, `aggregate`, `iterate`, `listExperiments`, `getExperiment`, `saveMetrics`, `saveDashboard`, `report`, `listMetrics`, `listWidgets`, `cancel` + +No `NO_EFFECTIVE_GOAL`, `SPAWN_GOAL_UNAVAILABLE`, pack-route, or workflow errors were observed. + +## UI surface checks + +Verified in the running browser against this parent session: + +- Deep link `#/ext/experiment-runner` opened the side panel titled `Experiments` with the `New experiment` screen. +- Session actions menu contained `New experiment`. +- Composer slash search for `/Exp` showed `/experiment-runner.open Experiments`. +- Default panel copy presents A/B as the recommended bounded mode and Autoresearch as `Autonomous · opt-in · hard caps required`, confirming autoresearch is not the default launch path. + +Browser console notes: initial `401` requests occurred before connecting with the gateway token, and one transient `no registered panel` warning appeared before the panel was available. The panel opened successfully afterward. + +## Minimal A/B definition + +Definition route: `defineExperiment` + +Key fields: + +```json +{ + "experimentId": "smoke-ab-20260621003841", + "mode": "ab", + "parentGoalId": "cdb260f0-96fe-4350-8093-bfe2a0008477", + "workflowId": null, + "variants": ["baseline", "variant-b"], + "repeats": 1, + "maxConcurrency": 2, + "perRunBudget": 0.05, + "sameCompletionBar": false +} +``` + +Projection returned: + +```json +{ + "mode": "ab", + "arms": 2, + "estPerArmUsd": 0.5, + "estCostUsd": 1, + "concurrencyCap": 2 +} +``` + +Variant treatment metadata: + +- `baseline`: `metadata.experiment.userMetrics.metric = 1`, `metadata.experiment.userMetrics.smokeBaselineMarker = 101`, `metadata.smokeTreatment.marker = smoke-baseline-101` +- `variant-b`: `metadata.experiment.userMetrics.metric = 2`, `metadata.experiment.userMetrics.smokeVariantMarker = 202`, `metadata.smokeTreatment.marker = smoke-variant-b-202` + +## Launch and child goals + +`launch` returned exactly two runs and exactly two child goal ids for the experiment: + +| Arm | Run id | Child goal id | Status after launch | +|---|---|---|---| +| `baseline` | `baseline--r0` | `83ae8bfc-fa72-401c-8821-aca1707c1dfc` | `spawned` | +| `variant-b` | `variant-b--r0` | `a174329f-fe14-4b04-b0fe-667453a8ddae` | `spawned` | + +Goal inspection confirmed both child goals are under parent `cdb260f0-96fe-4350-8093-bfe2a0008477` and carry the expected metadata. + +Baseline child goal: + +```json +{ + "id": "83ae8bfc-fa72-401c-8821-aca1707c1dfc", + "parentGoalId": "cdb260f0-96fe-4350-8093-bfe2a0008477", + "spawnedFromPlanId": "smoke-ab-20260621003841:baseline--r0", + "workflowId": "general", + "metadata": { + "experiment": { + "experimentId": "smoke-ab-20260621003841", + "armId": "baseline", + "repeat": 0, + "budget": 0.05, + "userMetrics": { "metric": 1, "smokeBaselineMarker": 101 } + }, + "smokeTreatment": { "arm": "baseline", "marker": "smoke-baseline-101" } + } +} +``` + +Variant-B child goal: + +```json +{ + "id": "a174329f-fe14-4b04-b0fe-667453a8ddae", + "parentGoalId": "cdb260f0-96fe-4350-8093-bfe2a0008477", + "spawnedFromPlanId": "smoke-ab-20260621003841:variant-b--r0", + "workflowId": "general", + "metadata": { + "experiment": { + "experimentId": "smoke-ab-20260621003841", + "armId": "variant-b", + "repeat": 0, + "budget": 0.05, + "userMetrics": { "metric": 2, "smokeVariantMarker": 202 } + }, + "smokeTreatment": { "arm": "variant-b", "marker": "smoke-variant-b-202" } + } +} +``` + +## Lifecycle routes + +Executed route sequence: + +1. `poll` +2. `collect` +3. `aggregate` +4. `report` +5. `saveMetrics` +6. `saveDashboard` +7. `report` again +8. `getExperiment` + +`poll` outcome: + +- Both runs crossed the configured `perRunBudget` and were marked `failed` with `error: over_budget`. +- `allSettled: true` was returned. +- Costs captured: + - baseline: `$0.189494`, `29744` input tokens, `429` output tokens + - variant-b: `$0.142015`, `28049` input tokens, `59` output tokens + +`collect` outcome: + +- Returned both terminal failed runs. Because they were already failed, no raw success outcome was collected and metric maps remained empty. +- Experiment state was later confirmed as `{ "status": "done" }`. + +`aggregate` outcome: + +- Returned an A/B model with both arms and the selected metrics. +- Values were `null` because failed over-budget runs do not contribute collected metrics. + +Initial `report` outcome: + +- Returned a dashboard/report payload with `model` and `html`. +- Report HTML was generated successfully. + +Metric/dashboard edit check: + +- `saveMetrics` returned `{ "ok": true }` for edited metrics: `cost.totalUsd`, `time.wallClockMs`. +- `saveDashboard` returned `{ "ok": true }` for edited widgets: `edited-summary`, `edited-raw`. +- A subsequent `report` returned: + +```json +{ + "htmlLength": 3827, + "metricIds": ["cost.totalUsd", "time.wallClockMs"], + "widgetIds": ["edited-summary", "edited-raw"], + "runStatuses": ["failed", "failed"] +} +``` + +## Autoresearch guardrail check + +No autoresearch loop was launched. + +A direct `defineExperiment` call for `mode: autoresearch` with objective, stop, and per-run budget but no finite hard cap returned: + +```json +{ "error": "AR_UNCAPPED" } +``` + +This confirms autoresearch remains opt-in and requires hard caps before launch. + +## Registry checks + +- `listMetrics` returned 9 metric descriptors. +- `listWidgets` returned 6 widget descriptors. + +## Cleanup + +To contain cost after evidence capture, child teams were torn down: + +- `POST /api/goals/83ae8bfc-fa72-401c-8821-aca1707c1dfc/team/teardown?cascade=false` → `{ "ok": true, "toreDown": 1 }` +- `POST /api/goals/a174329f-fe14-4b04-b0fe-667453a8ddae/team/teardown?cascade=false` → `{ "ok": true, "toreDown": 1 }` + +Final session inspection showed no live sessions for either child goal. + +## Readiness notes + +The runner is ready for a real Graphify A/B benchmark smoke-sized launch. Recommended setup for the real benchmark: + +1. Use a per-run budget above the observed startup floor; `0.05` was too low for even the minimal smoke arms. +2. Decide whether to use the project default `general` workflow or an explicit benchmark workflow. Omitting `workflowId` works but still resolves to the project default workflow. +3. Keep `maxConcurrency` at `1` or `2` for the first Graphify run and verify collection before scaling repeats. +4. Leave autoresearch disabled unless explicit finite caps, stop conditions, and per-run budget are configured. From 2c72e767c732f6a67465e48427de28eb160197b8 Mon Sep 17 00:00:00 2001 From: Artur Jonkisz Date: Sun, 21 Jun 2026 01:20:17 +0100 Subject: [PATCH 2/4] Add experiment runner smoke E2E coverage Co-authored-by: bobbit-ai --- docs/experiment-runner-smoke-test.md | 4 + tests/e2e/ui/experiment-runner-smoke.spec.ts | 327 +++++++++++++++++++ 2 files changed, 331 insertions(+) create mode 100644 tests/e2e/ui/experiment-runner-smoke.spec.ts diff --git a/docs/experiment-runner-smoke-test.md b/docs/experiment-runner-smoke-test.md index 0fa481dd0..97a90b36f 100644 --- a/docs/experiment-runner-smoke-test.md +++ b/docs/experiment-runner-smoke-test.md @@ -205,6 +205,10 @@ To contain cost after evidence capture, child teams were torn down: Final session inspection showed no live sessions for either child goal. +## Automated coverage + +Added optional browser E2E coverage in `tests/e2e/ui/experiment-runner-smoke.spec.ts`. The test first checks `/api/ext/contributions` and skips with an explicit annotation when `experiment-runner` is absent on `origin/master`; when present, it exercises the deep link, session-menu launcher, slash launcher, bounded A/B route lifecycle, metadata propagation, spec persistence, reporting, and cancel cleanup. + ## Readiness notes The runner is ready for a real Graphify A/B benchmark smoke-sized launch. Recommended setup for the real benchmark: diff --git a/tests/e2e/ui/experiment-runner-smoke.spec.ts b/tests/e2e/ui/experiment-runner-smoke.spec.ts new file mode 100644 index 000000000..11fe00cc8 --- /dev/null +++ b/tests/e2e/ui/experiment-runner-smoke.spec.ts @@ -0,0 +1,327 @@ +import type { Page, TestInfo } from "@playwright/test"; +import { test, expect } from "../gateway-harness.js"; +import { + apiFetch, + createGoal, + deleteGoal, + readE2ETokenAsync, + startTeam, + teardownTeam, + waitForSessionStatus, + base, +} from "../e2e-setup.js"; +import { navigateToHash, openApp } from "./ui-helpers.js"; + +test.describe.configure({ mode: "serial" }); + +const PACK_ID = "experiment-runner"; +const DEFAULT_ROUTE_ID = "experiment-runner"; +const FORBIDDEN_ERRORS = /NO_EFFECTIVE_GOAL|SPAWN_GOAL_UNAVAILABLE|PARENT_MISMATCH|PACK_ROUTE|WORKFLOW_(?:REQUIRED|NOT_FOUND|INVALID)|workflow route error/i; + +type ContributionEntryPoint = { + id: string; + kind: string; + routeId?: string; + listName?: string; + label?: string; + target?: { panelId?: string }; +}; + +type PackContributionsMeta = { + packId: string; + packName?: string; + panels?: Array<{ id: string; title?: string }>; + entrypoints?: ContributionEntryPoint[]; + routeNames?: string[]; +}; + +type RouteCallResult = { status: number; body: any; text: string }; + +async function listContributions(): Promise { + const res = await apiFetch("/api/ext/contributions"); + const text = await res.text(); + expect(res.ok, `/api/ext/contributions should be reachable: ${text}`).toBe(true); + return (JSON.parse(text) as { packs?: PackContributionsMeta[] }).packs ?? []; +} + +function skipWhenPackAbsent(testInfo: TestInfo, packs: PackContributionsMeta[]): never { + const packIds = packs.map((p) => p.packId).sort().join(", ") || ""; + const reason = `${PACK_ID} is not present in /api/ext/contributions for this gateway; optional smoke journey skipped. Seen packs: ${packIds}`; + testInfo.annotations.push({ type: "skip", description: reason }); + test.skip(true, reason); + throw new Error(reason); +} + +function assertContributionShape(pack: PackContributionsMeta): { panelId: string; routeId: string } { + const panels = pack.panels ?? []; + const entrypoints = pack.entrypoints ?? []; + const routeNames = pack.routeNames ?? []; + const panelId = panels.find((p) => p.id === "experiment-runner.panel")?.id ?? panels[0]?.id; + const routeId = entrypoints.find((e) => e.kind === "route")?.routeId ?? DEFAULT_ROUTE_ID; + + expect(panelId, "Experiment Runner must contribute a panel").toBeTruthy(); + expect(panels.some((p) => p.id === "experiment-runner.panel" && /experiments/i.test(p.title ?? ""))).toBe(true); + expect(entrypoints.some((e) => e.kind === "session-menu" && /new experiment/i.test(e.label ?? e.listName ?? ""))).toBe(true); + expect(entrypoints.some((e) => e.kind === "composer-slash" && /experiments?/i.test(e.label ?? e.listName ?? e.id))).toBe(true); + expect(entrypoints.some((e) => e.kind === "route" && (e.routeId === DEFAULT_ROUTE_ID || e.routeId === routeId))).toBe(true); + expect(routeNames).toEqual(expect.arrayContaining([ + "defineExperiment", + "launch", + "poll", + "collect", + "aggregate", + "saveMetrics", + "saveDashboard", + "report", + ])); + + return { panelId, routeId }; +} + +async function expectNoForbiddenErrors(page: Page, body: unknown, context: string): Promise { + const text = typeof body === "string" ? body : JSON.stringify(body); + expect(text, `${context} must not surface parent-goal/spawn/workflow errors`).not.toMatch(FORBIDDEN_ERRORS); + await expect(page.locator('[data-testid="header-toast"], [role="alert"], [role="status"]').filter({ hasText: FORBIDDEN_ERRORS })).toHaveCount(0); +} + +async function openGoalSession(page: Page, sessionId: string): Promise { + await openApp(page); + await navigateToHash(page, `#/session/${sessionId}`); + await expect(page.locator("textarea").first()).toBeVisible({ timeout: 20_000 }); + await page.evaluate(() => (window as any).__bobbitReconcilePackRenderers?.()).catch(() => {}); +} + +async function expectLauncherSurfaces(page: Page, routeId: string): Promise { + const trigger = page.locator('[data-testid="session-actions-trigger"]').first(); + await expect(trigger, "goal session header must expose the session menu").toBeVisible({ timeout: 10_000 }); + await trigger.click(); + await expect(page.locator("sidebar-actions-popover [role='menu']")).toBeVisible({ timeout: 5_000 }); + await expect( + page.locator('sidebar-actions-popover [role="menuitem"]', { hasText: /new experiment/i }).first(), + "Experiment Runner must contribute the New experiment session-menu launcher", + ).toBeVisible({ timeout: 10_000 }); + await page.keyboard.press("Escape"); + + const textarea = page.locator("textarea").first(); + await textarea.fill("/Exp"); + await expect( + page.locator('[data-testid^="slash-command-"]').filter({ hasText: /experiments?|experiment-runner/i }).first(), + "Experiment Runner must contribute the Experiments composer slash launcher", + ).toBeVisible({ timeout: 10_000 }); + await textarea.fill(""); + + await navigateToHash(page, `#/ext/${routeId}`); + await expect(page.locator("body")).toContainText(/Experiments|Experiment Runner/i, { timeout: 20_000 }); + await expect(page.locator("body"), "A/B comparison should be the visible default/recommended path").toContainText(/A\/?B|A-B|comparison/i); + await expect(page.locator("body"), "Autoresearch should be present as an opt-in guarded mode").toContainText(/Autoresearch/i); + await expect(page.locator("body"), "Autoresearch must advertise opt-in/hard-cap guardrails").toContainText(/opt-in|hard caps|required|guardrail/i); +} + +async function mintSurfaceToken(sessionId: string, panelId: string): Promise { + const res = await apiFetch("/api/ext/surface-token", { + method: "POST", + headers: { "x-bobbit-session-id": sessionId }, + body: JSON.stringify({ + sessionId, + packId: PACK_ID, + contributionKind: "panel", + contributionId: panelId, + }), + }); + const text = await res.text(); + expect(res.status, `surface-token mint failed: ${text}`).toBe(200); + return (JSON.parse(text) as { token: string }).token; +} + +async function callExperimentRoute(sessionId: string, surfaceToken: string, name: string, body: Record = {}): Promise { + const res = await apiFetch(`/api/ext/route/${encodeURIComponent(name)}`, { + method: "POST", + headers: { "x-bobbit-session-id": sessionId }, + body: JSON.stringify({ + sessionId, + surfaceToken, + init: { method: "POST", body }, + }), + }); + const text = await res.text(); + let parsed: any = {}; + try { + parsed = text ? JSON.parse(text) : {}; + } catch { + parsed = { raw: text }; + } + expect(res.status, `${name} route HTTP failure: ${text}`).toBe(200); + return { status: res.status, body: parsed, text }; +} + +async function findExperimentChildren(parentGoalId: string, experimentId: string): Promise { + const res = await apiFetch("/api/goals"); + expect(res.ok).toBe(true); + const payload = await res.json(); + const goals = (payload.goals ?? payload) as any[]; + return goals.filter((g) => g.parentGoalId === parentGoalId && String(g.spawnedFromPlanId ?? "").startsWith(`${experimentId}:`)); +} + +function minimalExperimentDefinition(experimentId: string, parentGoalId: string) { + return { + experimentId, + title: "E2E Experiment Runner smoke", + mode: "ab", + parentGoalId, + runnable: { + kind: "spec", + spec: "Minimal safe smoke-test arm. Do not edit files. Finish quickly with one sentence: Smoke arm complete.", + }, + variants: [ + { + armId: "baseline", + label: "baseline", + metadata: { + experiment: { userMetrics: { metric: 1, smokeBaselineMarker: 101 } }, + smokeTreatment: { arm: "baseline", marker: "smoke-baseline-101" }, + }, + }, + { + armId: "variant-b", + label: "variant-b", + metadata: { + experiment: { userMetrics: { metric: 2, smokeVariantMarker: 202 } }, + smokeTreatment: { arm: "variant-b", marker: "smoke-variant-b-202" }, + }, + }, + ], + repeats: 1, + maxConcurrency: 1, + perRunBudget: 0.05, + sameCompletionBar: false, + metrics: [ + { metricId: "command.metric", aggregation: "median" }, + { metricId: "cost.totalUsd", aggregation: "median", directionOverride: "min" }, + ], + dashboard: { + widgets: [ + { id: "smoke-summary", type: "summary-cards", title: "Smoke summary", bind: { metricIds: ["command.metric", "cost.totalUsd"] } }, + ], + }, + }; +} + +test.describe("Experiment Runner optional smoke journey", () => { + test("deep link + launchers + bounded A/B route lifecycle", async ({ page }, testInfo) => { + test.setTimeout(180_000); + + const packs = await listContributions(); + const pack = packs.find((p) => p.packId === PACK_ID) ?? skipWhenPackAbsent(testInfo, packs); + const { panelId, routeId } = assertContributionShape(pack); + + let parentGoalId: string | undefined; + let teamLeadId: string | undefined; + let surfaceToken = ""; + let experimentId = ""; + let childGoalIds: string[] = []; + const routeResponses: unknown[] = []; + const consoleMessages: string[] = []; + page.on("console", (msg) => consoleMessages.push(`${msg.type()}: ${msg.text()}`)); + + try { + const parent = await createGoal({ + title: `Experiment Runner E2E smoke ${Date.now()}`, + spec: "Parent goal for the Experiment Runner browser E2E smoke journey. It exists only to provide an effective parent goal for child experiment arms.", + team: false, + worktree: false, + subgoalsAllowed: true, + maxNestingDepth: 2, + }); + parentGoalId = parent.id as string; + teamLeadId = await startTeam(parentGoalId); + await waitForSessionStatus(teamLeadId, "idle", 45_000).catch(() => {}); + + await openGoalSession(page, teamLeadId); + await expectLauncherSurfaces(page, routeId); + await expectNoForbiddenErrors(page, consoleMessages.join("\n"), "launcher/deep-link UI"); + + surfaceToken = await mintSurfaceToken(teamLeadId, panelId); + experimentId = `e2e-smoke-${Date.now().toString(36)}`; + + const arGuard = await callExperimentRoute(teamLeadId, surfaceToken, "defineExperiment", { + experimentId: `${experimentId}-ar-guard`, + mode: "autoresearch", + title: "Autoresearch guard probe", + runnable: { kind: "spec", spec: "Do not run; this validates guardrails only." }, + objective: { metricId: "command.metric", direction: "max" }, + stop: { plateauK: 1 }, + perRunBudget: 0.05, + }); + routeResponses.push(arGuard.body); + expect(arGuard.body.error, "Autoresearch must remain opt-in and reject missing finite hard caps").toBe("AR_UNCAPPED"); + + const defined = await callExperimentRoute(teamLeadId, surfaceToken, "defineExperiment", minimalExperimentDefinition(experimentId, parentGoalId)); + routeResponses.push(defined.body); + expect(defined.body.error).toBeUndefined(); + expect(defined.body.experimentId).toBe(experimentId); + expect(defined.body.projection?.mode).toBe("ab"); + expect(defined.body.projection?.arms).toBe(2); + + const launched = await callExperimentRoute(teamLeadId, surfaceToken, "launch", { experimentId }); + routeResponses.push(launched.body); + expect(launched.body.error).toBeUndefined(); + expect(launched.body.launched).toHaveLength(2); + + const children = await findExperimentChildren(parentGoalId, experimentId); + expect(children, "A/B launch must create exactly one child goal per arm").toHaveLength(2); + childGoalIds = children.map((g) => g.id as string); + const byArm = new Map(children.map((g) => [g.metadata?.experiment?.armId, g])); + expect([...byArm.keys()].sort()).toEqual(["baseline", "variant-b"]); + expect(byArm.get("baseline")?.metadata?.smokeTreatment).toEqual({ arm: "baseline", marker: "smoke-baseline-101" }); + expect(byArm.get("variant-b")?.metadata?.smokeTreatment).toEqual({ arm: "variant-b", marker: "smoke-variant-b-202" }); + expect(byArm.get("baseline")?.metadata?.experiment?.budget).toBe(0.05); + expect(byArm.get("variant-b")?.metadata?.experiment?.budget).toBe(0.05); + + for (const name of ["poll", "collect", "aggregate", "report"] as const) { + const result = await callExperimentRoute(teamLeadId, surfaceToken, name, { experimentId }); + routeResponses.push(result.body); + expect(result.body.error, `${name} should not return a route error`).toBeUndefined(); + } + + const editedMetrics = [ + { metricId: "cost.totalUsd", aggregation: "median", directionOverride: "min" }, + { metricId: "time.wallClockMs", aggregation: "median", directionOverride: "min" }, + ]; + const editedDashboard = { + widgets: [ + { id: "edited-summary", type: "summary-cards", title: "Edited smoke summary", bind: { metricIds: ["cost.totalUsd", "time.wallClockMs"] } }, + { id: "edited-raw", type: "raw-drilldown", title: "Edited raw", bind: { metricIds: ["cost.totalUsd"] } }, + ], + }; + expect((await callExperimentRoute(teamLeadId, surfaceToken, "saveMetrics", { experimentId, metrics: editedMetrics })).body).toEqual({ ok: true }); + expect((await callExperimentRoute(teamLeadId, surfaceToken, "saveDashboard", { experimentId, dashboard: editedDashboard })).body).toEqual({ ok: true }); + + const token = await readE2ETokenAsync(); + await page.goto(`${base()}/?token=${encodeURIComponent(token)}#/ext/${routeId}?experimentId=${encodeURIComponent(experimentId)}&view=report`, { waitUntil: "domcontentloaded" }); + await expect(page.locator("body")).toContainText(/Experiments|Experiment Runner/i, { timeout: 20_000 }); + surfaceToken = await mintSurfaceToken(teamLeadId, panelId); + const persisted = await callExperimentRoute(teamLeadId, surfaceToken, "getExperiment", { experimentId }); + expect(persisted.body.metrics.map((m: any) => m.metricId)).toEqual(["cost.totalUsd", "time.wallClockMs"]); + expect(persisted.body.dashboard.widgets.map((w: any) => w.id)).toEqual(["edited-summary", "edited-raw"]); + const editedReport = await callExperimentRoute(teamLeadId, surfaceToken, "report", { experimentId }); + routeResponses.push(persisted.body, editedReport.body); + expect(editedReport.body.error).toBeUndefined(); + expect(editedReport.body.html?.length ?? 0).toBeGreaterThan(100); + expect(editedReport.body.model?.metrics?.map((m: any) => m.metricId)).toEqual(["cost.totalUsd", "time.wallClockMs"]); + + const cancelled = await callExperimentRoute(teamLeadId, surfaceToken, "cancel", { experimentId }); + routeResponses.push(cancelled.body); + expect(cancelled.body.error).toBeUndefined(); + + for (const [i, response] of routeResponses.entries()) { + await expectNoForbiddenErrors(page, response, `route response #${i + 1}`); + } + await expectNoForbiddenErrors(page, consoleMessages.join("\n"), "browser console"); + } finally { + for (const childGoalId of childGoalIds) await teardownTeam(childGoalId).catch(() => {}); + for (const childGoalId of childGoalIds) await deleteGoal(childGoalId).catch(() => {}); + if (parentGoalId) await teardownTeam(parentGoalId).catch(() => {}); + if (parentGoalId) await deleteGoal(parentGoalId).catch(() => {}); + } + }); +}); From 84b254c5252258d14e032101f73a7ae5d886ec8d Mon Sep 17 00:00:00 2001 From: Artur Jonkisz Date: Sun, 21 Jun 2026 01:47:18 +0100 Subject: [PATCH 3/4] Add deterministic Experiment Runner E2E fixture Co-authored-by: bobbit-ai --- tests/e2e/ui/experiment-runner-smoke.spec.ts | 224 ++++++++-------- .../entrypoints/composer-slash.yaml | 5 + .../entrypoints/deeplink.yaml | 6 + .../entrypoints/session-menu.yaml | 5 + .../experiment-runner/lib/panel.js | 169 ++++++++++++ .../experiment-runner/lib/routes.mjs | 241 ++++++++++++++++++ .../experiment-runner/pack.yaml | 26 ++ .../panels/experiment-runner.yaml | 3 + 8 files changed, 568 insertions(+), 111 deletions(-) create mode 100644 tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/entrypoints/composer-slash.yaml create mode 100644 tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/entrypoints/deeplink.yaml create mode 100644 tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/entrypoints/session-menu.yaml create mode 100644 tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/lib/panel.js create mode 100644 tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/lib/routes.mjs create mode 100644 tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/pack.yaml create mode 100644 tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/panels/experiment-runner.yaml diff --git a/tests/e2e/ui/experiment-runner-smoke.spec.ts b/tests/e2e/ui/experiment-runner-smoke.spec.ts index 11fe00cc8..9bc6aaa7d 100644 --- a/tests/e2e/ui/experiment-runner-smoke.spec.ts +++ b/tests/e2e/ui/experiment-runner-smoke.spec.ts @@ -1,14 +1,13 @@ -import type { Page, TestInfo } from "@playwright/test"; +import { fileURLToPath } from "node:url"; +import type { Page } from "@playwright/test"; import { test, expect } from "../gateway-harness.js"; import { apiFetch, createGoal, deleteGoal, - readE2ETokenAsync, startTeam, teardownTeam, waitForSessionStatus, - base, } from "../e2e-setup.js"; import { navigateToHash, openApp } from "./ui-helpers.js"; @@ -16,7 +15,25 @@ test.describe.configure({ mode: "serial" }); const PACK_ID = "experiment-runner"; const DEFAULT_ROUTE_ID = "experiment-runner"; +const SOURCE_DIR = fileURLToPath(new URL("../../fixtures/market-sources/experiment-runner-smoke-src", import.meta.url)); const FORBIDDEN_ERRORS = /NO_EFFECTIVE_GOAL|SPAWN_GOAL_UNAVAILABLE|PARENT_MISMATCH|PACK_ROUTE|WORKFLOW_(?:REQUIRED|NOT_FOUND|INVALID)|workflow route error/i; +// Pack-schema route names are lowercase-token identifiers; these fixture names +// correspond to the Experiment Runner contract routes (defineExperiment, +// saveMetrics, saveDashboard, getExperiment, listMetrics, listWidgets, etc.). +const REQUIRED_ROUTE_NAMES = [ + "defineexperiment", + "launch", + "poll", + "collect", + "aggregate", + "savemetrics", + "savedashboard", + "report", + "getexperiment", + "listmetrics", + "listwidgets", + "cancel", +]; type ContributionEntryPoint = { id: string; @@ -37,6 +54,37 @@ type PackContributionsMeta = { type RouteCallResult = { status: number; body: any; text: string }; +async function cleanupFixtureInstall(): Promise { + await apiFetch("/api/marketplace/installed", { + method: "DELETE", + body: JSON.stringify({ scope: "server", packName: PACK_ID }), + }).catch(() => {}); + try { + const res = await apiFetch("/api/marketplace/sources"); + for (const source of ((await res.json()).sources ?? []) as Array<{ id: string }>) { + await apiFetch(`/api/marketplace/sources/${encodeURIComponent(source.id)}`, { method: "DELETE" }).catch(() => {}); + } + } catch { /* best-effort cleanup */ } +} + +async function installExperimentRunnerFixture(): Promise { + await cleanupFixtureInstall(); + const addRes = await apiFetch("/api/marketplace/sources", { + method: "POST", + body: JSON.stringify({ url: SOURCE_DIR }), + }); + const addBody = await addRes.text(); + expect(addRes.status, addBody).toBe(201); + const sourceId = (JSON.parse(addBody) as { source: { id: string } }).source.id; + + const installRes = await apiFetch("/api/marketplace/install", { + method: "POST", + body: JSON.stringify({ sourceId, dirName: PACK_ID, scope: "server" }), + }); + const installBody = await installRes.text(); + expect(installRes.status, installBody).toBe(201); +} + async function listContributions(): Promise { const res = await apiFetch("/api/ext/contributions"); const text = await res.text(); @@ -44,14 +92,6 @@ async function listContributions(): Promise { return (JSON.parse(text) as { packs?: PackContributionsMeta[] }).packs ?? []; } -function skipWhenPackAbsent(testInfo: TestInfo, packs: PackContributionsMeta[]): never { - const packIds = packs.map((p) => p.packId).sort().join(", ") || ""; - const reason = `${PACK_ID} is not present in /api/ext/contributions for this gateway; optional smoke journey skipped. Seen packs: ${packIds}`; - testInfo.annotations.push({ type: "skip", description: reason }); - test.skip(true, reason); - throw new Error(reason); -} - function assertContributionShape(pack: PackContributionsMeta): { panelId: string; routeId: string } { const panels = pack.panels ?? []; const entrypoints = pack.entrypoints ?? []; @@ -59,21 +99,12 @@ function assertContributionShape(pack: PackContributionsMeta): { panelId: string const panelId = panels.find((p) => p.id === "experiment-runner.panel")?.id ?? panels[0]?.id; const routeId = entrypoints.find((e) => e.kind === "route")?.routeId ?? DEFAULT_ROUTE_ID; - expect(panelId, "Experiment Runner must contribute a panel").toBeTruthy(); + expect(panelId, "Experiment Runner must contribute a panel").toBe("experiment-runner.panel"); expect(panels.some((p) => p.id === "experiment-runner.panel" && /experiments/i.test(p.title ?? ""))).toBe(true); expect(entrypoints.some((e) => e.kind === "session-menu" && /new experiment/i.test(e.label ?? e.listName ?? ""))).toBe(true); expect(entrypoints.some((e) => e.kind === "composer-slash" && /experiments?/i.test(e.label ?? e.listName ?? e.id))).toBe(true); - expect(entrypoints.some((e) => e.kind === "route" && (e.routeId === DEFAULT_ROUTE_ID || e.routeId === routeId))).toBe(true); - expect(routeNames).toEqual(expect.arrayContaining([ - "defineExperiment", - "launch", - "poll", - "collect", - "aggregate", - "saveMetrics", - "saveDashboard", - "report", - ])); + expect(entrypoints.some((e) => e.kind === "route" && e.routeId === DEFAULT_ROUTE_ID)).toBe(true); + expect(routeNames).toEqual(expect.arrayContaining(REQUIRED_ROUTE_NAMES)); return { panelId, routeId }; } @@ -89,32 +120,32 @@ async function openGoalSession(page: Page, sessionId: string): Promise { await navigateToHash(page, `#/session/${sessionId}`); await expect(page.locator("textarea").first()).toBeVisible({ timeout: 20_000 }); await page.evaluate(() => (window as any).__bobbitReconcilePackRenderers?.()).catch(() => {}); + await page.evaluate(() => (window as any).__bobbitReconcilePackEntrypoints?.()).catch(() => {}); } -async function expectLauncherSurfaces(page: Page, routeId: string): Promise { +async function exerciseLauncherSurfaces(page: Page, routeId: string): Promise { + const panel = page.getByTestId("experiment-runner-panel"); const trigger = page.locator('[data-testid="session-actions-trigger"]').first(); await expect(trigger, "goal session header must expose the session menu").toBeVisible({ timeout: 10_000 }); await trigger.click(); await expect(page.locator("sidebar-actions-popover [role='menu']")).toBeVisible({ timeout: 5_000 }); - await expect( - page.locator('sidebar-actions-popover [role="menuitem"]', { hasText: /new experiment/i }).first(), - "Experiment Runner must contribute the New experiment session-menu launcher", - ).toBeVisible({ timeout: 10_000 }); - await page.keyboard.press("Escape"); + const menuItem = page.locator('sidebar-actions-popover [role="menuitem"]', { hasText: /new experiment/i }).first(); + await expect(menuItem, "Experiment Runner must contribute the New experiment session-menu launcher").toBeVisible({ timeout: 10_000 }); + await menuItem.click(); + await expect(panel, "session-menu launcher should open the Experiments panel").toBeVisible({ timeout: 20_000 }); const textarea = page.locator("textarea").first(); await textarea.fill("/Exp"); - await expect( - page.locator('[data-testid^="slash-command-"]').filter({ hasText: /experiments?|experiment-runner/i }).first(), - "Experiment Runner must contribute the Experiments composer slash launcher", - ).toBeVisible({ timeout: 10_000 }); - await textarea.fill(""); + const command = page.locator('[data-testid^="slash-command-"]').filter({ hasText: /experiments?|experiment-runner/i }).first(); + await expect(command, "Experiment Runner must contribute the Experiments composer slash launcher").toBeVisible({ timeout: 10_000 }); + await command.click(); + await textarea.press("Enter"); + await expect(panel, "composer /Experiments launcher should open the Experiments panel").toBeVisible({ timeout: 20_000 }); await navigateToHash(page, `#/ext/${routeId}`); - await expect(page.locator("body")).toContainText(/Experiments|Experiment Runner/i, { timeout: 20_000 }); - await expect(page.locator("body"), "A/B comparison should be the visible default/recommended path").toContainText(/A\/?B|A-B|comparison/i); - await expect(page.locator("body"), "Autoresearch should be present as an opt-in guarded mode").toContainText(/Autoresearch/i); - await expect(page.locator("body"), "Autoresearch must advertise opt-in/hard-cap guardrails").toContainText(/opt-in|hard caps|required|guardrail/i); + await expect(panel, "deep link should open the Experiments panel").toBeVisible({ timeout: 20_000 }); + await expect(panel, "A/B comparison should be the visible default/recommended path").toContainText(/A\/B comparison/i); + await expect(panel, "Autoresearch must remain opt-in with hard caps").toContainText(/Autoresearch is opt-in.*hard caps/i); } async function mintSurfaceToken(sessionId: string, panelId: string): Promise { @@ -159,60 +190,22 @@ async function findExperimentChildren(parentGoalId: string, experimentId: string expect(res.ok).toBe(true); const payload = await res.json(); const goals = (payload.goals ?? payload) as any[]; - return goals.filter((g) => g.parentGoalId === parentGoalId && String(g.spawnedFromPlanId ?? "").startsWith(`${experimentId}:`)); + return goals.filter((g) => g.parentGoalId === parentGoalId && g.metadata?.experiment?.id === experimentId); } -function minimalExperimentDefinition(experimentId: string, parentGoalId: string) { - return { - experimentId, - title: "E2E Experiment Runner smoke", - mode: "ab", - parentGoalId, - runnable: { - kind: "spec", - spec: "Minimal safe smoke-test arm. Do not edit files. Finish quickly with one sentence: Smoke arm complete.", - }, - variants: [ - { - armId: "baseline", - label: "baseline", - metadata: { - experiment: { userMetrics: { metric: 1, smokeBaselineMarker: 101 } }, - smokeTreatment: { arm: "baseline", marker: "smoke-baseline-101" }, - }, - }, - { - armId: "variant-b", - label: "variant-b", - metadata: { - experiment: { userMetrics: { metric: 2, smokeVariantMarker: 202 } }, - smokeTreatment: { arm: "variant-b", marker: "smoke-variant-b-202" }, - }, - }, - ], - repeats: 1, - maxConcurrency: 1, - perRunBudget: 0.05, - sameCompletionBar: false, - metrics: [ - { metricId: "command.metric", aggregation: "median" }, - { metricId: "cost.totalUsd", aggregation: "median", directionOverride: "min" }, - ], - dashboard: { - widgets: [ - { id: "smoke-summary", type: "summary-cards", title: "Smoke summary", bind: { metricIds: ["command.metric", "cost.totalUsd"] } }, - ], - }, - }; -} +test.afterEach(async () => { + await cleanupFixtureInstall(); +}); -test.describe("Experiment Runner optional smoke journey", () => { - test("deep link + launchers + bounded A/B route lifecycle", async ({ page }, testInfo) => { +test.describe("Experiment Runner smoke journey", () => { + test("fixture install + launchers + UI-driven bounded A/B lifecycle", async ({ page, gateway }) => { test.setTimeout(180_000); + await installExperimentRunnerFixture(); const packs = await listContributions(); - const pack = packs.find((p) => p.packId === PACK_ID) ?? skipWhenPackAbsent(testInfo, packs); - const { panelId, routeId } = assertContributionShape(pack); + const pack = packs.find((p) => p.packId === PACK_ID); + expect(pack, `${PACK_ID} fixture must be present after local marketplace install`).toBeTruthy(); + const { panelId, routeId } = assertContributionShape(pack!); let parentGoalId: string | undefined; let teamLeadId: string | undefined; @@ -235,37 +228,36 @@ test.describe("Experiment Runner optional smoke journey", () => { parentGoalId = parent.id as string; teamLeadId = await startTeam(parentGoalId); await waitForSessionStatus(teamLeadId, "idle", 45_000).catch(() => {}); + const teamLeadSecret = gateway.sessionManager.sessionSecretStore.getOrCreateSecret(teamLeadId); await openGoalSession(page, teamLeadId); - await expectLauncherSurfaces(page, routeId); + await exerciseLauncherSurfaces(page, routeId); await expectNoForbiddenErrors(page, consoleMessages.join("\n"), "launcher/deep-link UI"); surfaceToken = await mintSurfaceToken(teamLeadId, panelId); experimentId = `e2e-smoke-${Date.now().toString(36)}`; - const arGuard = await callExperimentRoute(teamLeadId, surfaceToken, "defineExperiment", { + const arGuard = await callExperimentRoute(teamLeadId, surfaceToken, "defineexperiment", { experimentId: `${experimentId}-ar-guard`, mode: "autoresearch", title: "Autoresearch guard probe", + parentGoalId, + teamLeadSecret, runnable: { kind: "spec", spec: "Do not run; this validates guardrails only." }, objective: { metricId: "command.metric", direction: "max" }, stop: { plateauK: 1 }, perRunBudget: 0.05, }); routeResponses.push(arGuard.body); - expect(arGuard.body.error, "Autoresearch must remain opt-in and reject missing finite hard caps").toBe("AR_UNCAPPED"); - - const defined = await callExperimentRoute(teamLeadId, surfaceToken, "defineExperiment", minimalExperimentDefinition(experimentId, parentGoalId)); - routeResponses.push(defined.body); - expect(defined.body.error).toBeUndefined(); - expect(defined.body.experimentId).toBe(experimentId); - expect(defined.body.projection?.mode).toBe("ab"); - expect(defined.body.projection?.arms).toBe(2); + expect(arGuard.body.error, "Autoresearch must reject missing finite hard caps").toBe("AR_UNCAPPED"); - const launched = await callExperimentRoute(teamLeadId, surfaceToken, "launch", { experimentId }); - routeResponses.push(launched.body); - expect(launched.body.error).toBeUndefined(); - expect(launched.body.launched).toHaveLength(2); + await page.getByTestId("exp-experiment-id").fill(experimentId); + await page.getByTestId("exp-parent-goal-id").fill(parentGoalId); + await page.getByTestId("exp-session-secret").fill(teamLeadSecret); + await page.getByTestId("exp-define-button").click(); + await expect(page.getByTestId("exp-status")).toContainText(/Definition ready: 2 arms/i, { timeout: 20_000 }); + await page.getByTestId("exp-launch-button").click(); + await expect(page.getByTestId("exp-status")).toContainText(/Launch complete: 2 child goals/i, { timeout: 30_000 }); const children = await findExperimentChildren(parentGoalId, experimentId); expect(children, "A/B launch must create exactly one child goal per arm").toHaveLength(2); @@ -277,10 +269,9 @@ test.describe("Experiment Runner optional smoke journey", () => { expect(byArm.get("baseline")?.metadata?.experiment?.budget).toBe(0.05); expect(byArm.get("variant-b")?.metadata?.experiment?.budget).toBe(0.05); - for (const name of ["poll", "collect", "aggregate", "report"] as const) { - const result = await callExperimentRoute(teamLeadId, surfaceToken, name, { experimentId }); - routeResponses.push(result.body); - expect(result.body.error, `${name} should not return a route error`).toBeUndefined(); + for (const name of ["poll", "collect", "aggregate"] as const) { + await page.getByTestId(`exp-${name}`).click(); + await expect(page.getByTestId("exp-status")).toContainText(new RegExp(`${name} complete`, "i"), { timeout: 20_000 }); } const editedMetrics = [ @@ -293,14 +284,25 @@ test.describe("Experiment Runner optional smoke journey", () => { { id: "edited-raw", type: "raw-drilldown", title: "Edited raw", bind: { metricIds: ["cost.totalUsd"] } }, ], }; - expect((await callExperimentRoute(teamLeadId, surfaceToken, "saveMetrics", { experimentId, metrics: editedMetrics })).body).toEqual({ ok: true }); - expect((await callExperimentRoute(teamLeadId, surfaceToken, "saveDashboard", { experimentId, dashboard: editedDashboard })).body).toEqual({ ok: true }); - - const token = await readE2ETokenAsync(); - await page.goto(`${base()}/?token=${encodeURIComponent(token)}#/ext/${routeId}?experimentId=${encodeURIComponent(experimentId)}&view=report`, { waitUntil: "domcontentloaded" }); - await expect(page.locator("body")).toContainText(/Experiments|Experiment Runner/i, { timeout: 20_000 }); + await page.getByTestId("exp-metrics-json").fill(JSON.stringify(editedMetrics, null, 2)); + await page.getByTestId("exp-save-metrics").click(); + await expect(page.getByTestId("exp-status")).toContainText(/Metric spec saved/i, { timeout: 20_000 }); + await page.getByTestId("exp-dashboard-json").fill(JSON.stringify(editedDashboard, null, 2)); + await page.getByTestId("exp-save-dashboard").click(); + await expect(page.getByTestId("exp-status")).toContainText(/Dashboard spec saved/i, { timeout: 20_000 }); + await page.getByTestId("exp-report-button").click(); + await expect(page.getByTestId("exp-report")).toContainText(/Experiment Runner Smoke Report/i, { timeout: 20_000 }); + await expect(page.getByTestId("exp-report")).toContainText(/Edited smoke summary/i); + await expect(page.getByTestId("exp-report")).toContainText(/time\.wallClockMs/i); + + // Reopen after a real page reload, then revisit the extension deep link with the + // active goal session restored so the panel host can reload persisted specs. + await openGoalSession(page, teamLeadId); + await navigateToHash(page, `#/ext/${routeId}?experimentId=${encodeURIComponent(experimentId)}&view=report`); + await expect(page.getByTestId("experiment-runner-panel")).toBeVisible({ timeout: 20_000 }); + await expect(page.getByTestId("experiment-runner-panel")).toContainText(/Edited smoke summary/i, { timeout: 20_000 }); surfaceToken = await mintSurfaceToken(teamLeadId, panelId); - const persisted = await callExperimentRoute(teamLeadId, surfaceToken, "getExperiment", { experimentId }); + const persisted = await callExperimentRoute(teamLeadId, surfaceToken, "getexperiment", { experimentId }); expect(persisted.body.metrics.map((m: any) => m.metricId)).toEqual(["cost.totalUsd", "time.wallClockMs"]); expect(persisted.body.dashboard.widgets.map((w: any) => w.id)).toEqual(["edited-summary", "edited-raw"]); const editedReport = await callExperimentRoute(teamLeadId, surfaceToken, "report", { experimentId }); diff --git a/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/entrypoints/composer-slash.yaml b/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/entrypoints/composer-slash.yaml new file mode 100644 index 000000000..a48c23bff --- /dev/null +++ b/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/entrypoints/composer-slash.yaml @@ -0,0 +1,5 @@ +id: Experiments +kind: composer-slash +label: Experiments +target: + panelId: experiment-runner.panel diff --git a/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/entrypoints/deeplink.yaml b/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/entrypoints/deeplink.yaml new file mode 100644 index 000000000..d7c6e9c66 --- /dev/null +++ b/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/entrypoints/deeplink.yaml @@ -0,0 +1,6 @@ +id: experiment-runner.route +kind: route +routeId: experiment-runner +target: + panelId: experiment-runner.panel +paramKeys: [experimentId, view] diff --git a/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/entrypoints/session-menu.yaml b/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/entrypoints/session-menu.yaml new file mode 100644 index 000000000..8f28e8013 --- /dev/null +++ b/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/entrypoints/session-menu.yaml @@ -0,0 +1,5 @@ +id: experiment-runner.session-menu +kind: session-menu +label: New experiment +target: + panelId: experiment-runner.panel diff --git a/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/lib/panel.js b/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/lib/panel.js new file mode 100644 index 000000000..0f0e007f2 --- /dev/null +++ b/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/lib/panel.js @@ -0,0 +1,169 @@ +export default function createPanel({ html, nothing, renderHeader }) { + const state = { + experimentId: `e2e-smoke-${Date.now().toString(36)}`, + parentGoalId: "", + teamLeadSecret: "", + title: "E2E Experiment Runner smoke", + spec: "Minimal safe smoke-test arm. Do not edit files. Finish quickly with one sentence: Smoke arm complete.", + metricsText: JSON.stringify([ + { metricId: "command.metric", aggregation: "median" }, + { metricId: "cost.totalUsd", aggregation: "median", directionOverride: "min" }, + ], null, 2), + dashboardText: JSON.stringify({ widgets: [{ id: "smoke-summary", type: "summary-cards", title: "Smoke summary", bind: { metricIds: ["command.metric", "cost.totalUsd"] } }] }, null, 2), + status: "Ready to define a bounded A/B experiment.", + defined: false, + launched: [], + report: null, + loadedFor: "", + loadingFor: "", + }; + + const rerender = (host) => { try { host?.requestRender?.(); } catch {} }; + const setStatus = (host, status) => { state.status = status; rerender(host); }; + const parseJson = (text, fallback) => { try { return JSON.parse(text); } catch { return fallback; } }; + const variants = () => [ + { + armId: "baseline", + label: "baseline", + metadata: { + experiment: { userMetrics: { metric: 1, smokeBaselineMarker: 101 } }, + smokeTreatment: { arm: "baseline", marker: "smoke-baseline-101" }, + }, + }, + { + armId: "variant-b", + label: "variant-b", + metadata: { + experiment: { userMetrics: { metric: 2, smokeVariantMarker: 202 } }, + smokeTreatment: { arm: "variant-b", marker: "smoke-variant-b-202" }, + }, + }, + ]; + const definition = () => ({ + experimentId: state.experimentId, + title: state.title, + mode: "ab", + parentGoalId: state.parentGoalId, + teamLeadSecret: state.teamLeadSecret, + runnable: { kind: "spec", spec: state.spec }, + variants: variants(), + repeats: 1, + maxConcurrency: 1, + perRunBudget: 0.05, + metrics: parseJson(state.metricsText, []), + dashboard: parseJson(state.dashboardText, { widgets: [] }), + }); + async function call(host, name, body = {}) { + if (!host?.callRoute) throw new Error("host.callRoute unavailable"); + const result = await host.callRoute(name, { method: "POST", body }); + if (result?.error) throw new Error(`${name}: ${result.error}`); + return result; + } + async function define(host) { + setStatus(host, "Defining experiment…"); + try { + const res = await call(host, "defineexperiment", definition()); + state.defined = true; + state.status = `Definition ready: ${res.projection?.arms ?? 0} arms`; + } catch (e) { state.status = e instanceof Error ? e.message : String(e); } + rerender(host); + } + async function launch(host) { + setStatus(host, "Launching child goals…"); + try { + const res = await call(host, "launch", { experimentId: state.experimentId }); + state.launched = res.launched || []; + state.status = `Launch complete: ${state.launched.length} child goals`; + } catch (e) { state.status = e instanceof Error ? e.message : String(e); } + rerender(host); + } + async function lifecycle(host, name) { + setStatus(host, `${name}…`); + try { + const res = await call(host, name, { experimentId: state.experimentId }); + if (name === "report") state.report = res; + state.status = `${name} complete`; + } catch (e) { state.status = e instanceof Error ? e.message : String(e); } + rerender(host); + } + async function saveMetrics(host) { + setStatus(host, "Saving metric spec…"); + try { await call(host, "savemetrics", { experimentId: state.experimentId, metrics: parseJson(state.metricsText, []) }); state.status = "Metric spec saved"; } + catch (e) { state.status = e instanceof Error ? e.message : String(e); } + rerender(host); + } + async function saveDashboard(host) { + setStatus(host, "Saving dashboard spec…"); + try { await call(host, "savedashboard", { experimentId: state.experimentId, dashboard: parseJson(state.dashboardText, { widgets: [] }) }); state.status = "Dashboard spec saved"; } + catch (e) { state.status = e instanceof Error ? e.message : String(e); } + rerender(host); + } + function loadIfNeeded(params, host) { + const id = typeof params?.experimentId === "string" ? params.experimentId : ""; + if (!id || state.loadedFor === id || state.loadingFor === id || !host?.callRoute) return; + state.loadingFor = id; + host.callRoute("getexperiment", { method: "POST", body: { experimentId: id } }) + .then((exp) => { + state.loadingFor = ""; + if (!exp || exp.error) return; + state.loadedFor = id; + state.experimentId = exp.experimentId || id; + state.parentGoalId = exp.parentGoalId || state.parentGoalId; + state.title = exp.title || state.title; + state.spec = exp.runnable?.spec || state.spec; + state.metricsText = JSON.stringify(exp.metrics || [], null, 2); + state.dashboardText = JSON.stringify(exp.dashboard || { widgets: [] }, null, 2); + state.launched = exp.runs || []; + state.report = exp.report || state.report; + state.status = `Loaded experiment ${state.experimentId}`; + rerender(host); + }) + .catch((e) => { state.loadingFor = ""; state.status = e instanceof Error ? e.message : String(e); rerender(host); }); + } + const input = (testId, label, value, onInput, type = "text") => html``; + return { + render(params, host) { + loadIfNeeded(params, host); + const metricIds = parseJson(state.metricsText, []).map?.((m) => m.metricId).join(", ") || ""; + const widgetTitles = (parseJson(state.dashboardText, { widgets: [] }).widgets || []).map((w) => w.title || w.id).join(", "); + return html` + +
+ ${renderHeader ? renderHeader({ title: "Experiments" }) : html`

Experiments

`} +
+ A/B comparison +

Autoresearch is opt-in and requires explicit hard caps before launch.

+ ${input("exp-experiment-id", "Experiment ID", state.experimentId, (v) => state.experimentId = v)} + ${input("exp-parent-goal-id", "Parent goal ID", state.parentGoalId, (v) => state.parentGoalId = v)} + ${input("exp-session-secret", "Team-lead session secret", state.teamLeadSecret, (v) => state.teamLeadSecret = v, "password")} + ${input("exp-title", "Title", state.title, (v) => state.title = v)} + +
Variants: baseline marker smoke-baseline-101; variant-b marker smoke-variant-b-202
+
+
+
+ Metric/dashboard specs + + +
+
Metrics: ${metricIds}; Widgets: ${widgetTitles}
+
+
+ Lifecycle +
${["poll", "collect", "aggregate", "report"].map((name) => html``)}
+
${state.status}
+ ${state.launched.length ? html`
    ${state.launched.map((r) => html`
  • ${r.armId}: ${r.goalId}
    • `)}
` : nothing} + ${state.report ? html`

Experiment Runner Smoke Report

${state.report.model?.runs?.length || 0} runs

${widgetTitles}

${metricIds}

${state.report.html}
` : nothing} +
+
`; + }, + }; +} diff --git a/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/lib/routes.mjs b/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/lib/routes.mjs new file mode 100644 index 000000000..a9e6eb1b6 --- /dev/null +++ b/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/lib/routes.mjs @@ -0,0 +1,241 @@ +import { readFileSync } from "node:fs"; +import { join } from "node:path"; + +const key = (experimentId) => `experiments/${experimentId}`; +const now = () => new Date().toISOString(); + +function body(req) { + return (req && req.body && typeof req.body === "object") ? req.body : {}; +} + +async function readStored(ctx, experimentId) { + if (!experimentId) return null; + return await ctx.host.store.get(key(experimentId)); +} + +async function writeStored(ctx, experiment) { + experiment.updatedAt = now(); + await ctx.host.store.put(key(experiment.experimentId), experiment); + return experiment; +} + +function gatewayUrl() { + return process.env.BOBBIT_GATEWAY_URL || (process.env.E2E_PORT ? `http://127.0.0.1:${process.env.E2E_PORT}` : ""); +} + +function gatewayToken() { + const env = process.env.BOBBIT_TOKEN?.trim(); + if (env) return env; + const dir = process.env.BOBBIT_DIR; + if (dir) { + try { return readFileSync(join(dir, "state", "token"), "utf8").trim(); } catch {} + } + return ""; +} + +async function gatewayFetch(path, init = {}) { + const base = gatewayUrl(); + const token = gatewayToken(); + if (!base || !token) throw new Error("gateway credentials unavailable for smoke fixture"); + const headers = { Authorization: `Bearer ${token}`, "Content-Type": "application/json", ...(init.headers || {}) }; + return await fetch(`${base}${path}`, { ...init, headers }); +} + +async function getGoal(goalId) { + const res = await gatewayFetch("/api/goals"); + if (!res.ok) throw new Error(`GET /api/goals failed ${res.status}: ${await res.text()}`); + const payload = await res.json(); + const goals = Array.isArray(payload.goals) ? payload.goals : payload; + return goals.find((g) => g.id === goalId) || null; +} + +function defaultDefinition(input, ctx) { + const experimentId = String(input.experimentId || `smoke-${Date.now().toString(36)}`); + return { + experimentId, + title: String(input.title || "E2E Experiment Runner smoke"), + mode: input.mode || "ab", + parentGoalId: input.parentGoalId, + teamLeadSecret: input.teamLeadSecret, + runnable: input.runnable || { kind: "spec", spec: "Minimal smoke arm; finish with one short sentence." }, + variants: Array.isArray(input.variants) && input.variants.length ? input.variants : [ + { armId: "baseline", label: "baseline", metadata: { smokeTreatment: { arm: "baseline", marker: "smoke-baseline-101" } } }, + { armId: "variant-b", label: "variant-b", metadata: { smokeTreatment: { arm: "variant-b", marker: "smoke-variant-b-202" } } }, + ], + repeats: Number.isFinite(Number(input.repeats)) ? Number(input.repeats) : 1, + maxConcurrency: Math.min(2, Math.max(1, Number(input.maxConcurrency || 1))), + perRunBudget: Math.min(0.05, Math.max(0, Number(input.perRunBudget || 0.01))), + metrics: Array.isArray(input.metrics) ? input.metrics : [{ metricId: "command.metric", aggregation: "median" }], + dashboard: input.dashboard && typeof input.dashboard === "object" ? input.dashboard : { widgets: [{ id: "smoke-summary", type: "summary-cards", title: "Smoke summary", bind: { metricIds: ["command.metric"] } }] }, + createdAt: now(), + updatedAt: now(), + status: "defined", + ownerSessionId: ctx.sessionId, + runs: [], + }; +} + +async function spawnGoal(ctx, experiment, variant, repeat) { + const parent = await getGoal(experiment.parentGoalId); + if (!parent) throw new Error(`parent goal not found: ${experiment.parentGoalId}`); + const planId = `${experiment.experimentId}:${variant.armId}:${repeat}`; + const variantMetadata = variant.metadata && typeof variant.metadata === "object" ? variant.metadata : {}; + const metadata = { + ...variantMetadata, + experiment: { + ...(variantMetadata.experiment && typeof variantMetadata.experiment === "object" ? variantMetadata.experiment : {}), + id: experiment.experimentId, + armId: variant.armId, + repeat, + budget: experiment.perRunBudget, + planId, + }, + }; + const res = await gatewayFetch("/api/goals", { + method: "POST", + headers: { + "X-Bobbit-Session-Secret": experiment.teamLeadSecret, + "X-Bobbit-Spawning-Session": ctx.sessionId, + }, + body: JSON.stringify({ + title: `${experiment.title} — ${variant.label || variant.armId}`, + spec: experiment.runnable?.spec || "Minimal deterministic Experiment Runner smoke child goal.", + cwd: parent.cwd, + projectId: parent.projectId, + parentGoalId: experiment.parentGoalId, + autoStartTeam: false, + worktree: false, + metadata, + }), + }); + const text = await res.text(); + let parsed; + try { parsed = text ? JSON.parse(text) : {}; } catch { parsed = { raw: text }; } + if (res.status !== 201) throw new Error(`spawnGoal ${planId} failed ${res.status}: ${text}`); + return { runId: planId, armId: variant.armId, repeat, goalId: parsed.id, status: "launched", metadata }; +} + +function reportModel(experiment) { + const metrics = Array.isArray(experiment.metrics) ? experiment.metrics : []; + const widgets = Array.isArray(experiment.dashboard?.widgets) ? experiment.dashboard.widgets : []; + return { + experimentId: experiment.experimentId, + status: experiment.status, + runs: experiment.runs || [], + metrics, + dashboard: experiment.dashboard || { widgets }, + generatedAt: now(), + }; +} + +export const routes = { + async defineexperiment(ctx, req) { + const input = body(req); + if (input.mode === "autoresearch") { + const maxRuns = Number(input.stop?.maxRuns ?? input.maxRuns ?? 0); + const maxCost = Number(input.stop?.maxCostUsd ?? input.maxCostUsd ?? 0); + if (!Number.isFinite(maxRuns) || maxRuns <= 0 || !Number.isFinite(maxCost) || maxCost <= 0) { + return { error: "AR_UNCAPPED", message: "Autoresearch requires explicit finite maxRuns and maxCostUsd hard caps." }; + } + } + const experiment = defaultDefinition(input, ctx); + if (experiment.mode !== "ab") return { error: "MODE_UNSUPPORTED" }; + if (!experiment.parentGoalId) return { error: "NO_PARENT_GOAL" }; + if (!experiment.teamLeadSecret) return { error: "NO_SESSION_SECRET" }; + await writeStored(ctx, experiment); + return { ok: true, experimentId: experiment.experimentId, projection: { mode: "ab", arms: experiment.variants.length, repeats: experiment.repeats } }; + }, + + async launch(ctx, req) { + const { experimentId } = body(req); + const experiment = await readStored(ctx, experimentId); + if (!experiment) return { error: "EXPERIMENT_NOT_FOUND" }; + if (Array.isArray(experiment.runs) && experiment.runs.length) return { ok: true, experimentId, launched: experiment.runs, alreadyLaunched: true }; + const launched = []; + for (const variant of experiment.variants) { + for (let repeat = 1; repeat <= experiment.repeats; repeat++) { + launched.push(await spawnGoal(ctx, experiment, variant, repeat)); + } + } + experiment.runs = launched; + experiment.status = "launched"; + await writeStored(ctx, experiment); + return { ok: true, experimentId, launched }; + }, + + async poll(ctx, req) { + const experiment = await readStored(ctx, body(req).experimentId); + if (!experiment) return { error: "EXPERIMENT_NOT_FOUND" }; + return { ok: true, status: experiment.status, runs: experiment.runs || [] }; + }, + + async collect(ctx, req) { + const experiment = await readStored(ctx, body(req).experimentId); + if (!experiment) return { error: "EXPERIMENT_NOT_FOUND" }; + experiment.status = "collected"; + experiment.runs = (experiment.runs || []).map((r) => ({ ...r, status: "collected", output: `${r.armId} smoke output` })); + await writeStored(ctx, experiment); + return { ok: true, collected: experiment.runs.length }; + }, + + async aggregate(ctx, req) { + const experiment = await readStored(ctx, body(req).experimentId); + if (!experiment) return { error: "EXPERIMENT_NOT_FOUND" }; + experiment.status = "aggregated"; + experiment.aggregate = { arms: experiment.variants.map((v) => v.armId), runCount: (experiment.runs || []).length }; + await writeStored(ctx, experiment); + return { ok: true, aggregate: experiment.aggregate }; + }, + + async savemetrics(ctx, req) { + const { experimentId, metrics } = body(req); + const experiment = await readStored(ctx, experimentId); + if (!experiment) return { error: "EXPERIMENT_NOT_FOUND" }; + experiment.metrics = Array.isArray(metrics) ? metrics : []; + await writeStored(ctx, experiment); + return { ok: true }; + }, + + async savedashboard(ctx, req) { + const { experimentId, dashboard } = body(req); + const experiment = await readStored(ctx, experimentId); + if (!experiment) return { error: "EXPERIMENT_NOT_FOUND" }; + experiment.dashboard = dashboard && typeof dashboard === "object" ? dashboard : { widgets: [] }; + await writeStored(ctx, experiment); + return { ok: true }; + }, + + async report(ctx, req) { + const experiment = await readStored(ctx, body(req).experimentId); + if (!experiment) return { error: "EXPERIMENT_NOT_FOUND" }; + experiment.status = "reported"; + const model = reportModel(experiment); + const widgetTitles = (model.dashboard.widgets || []).map((w) => w.title || w.id).join(", "); + const metricIds = (model.metrics || []).map((m) => m.metricId).join(", "); + const html = `

Experiment Runner Smoke Report

Experiment ${experiment.experimentId} has ${model.runs.length} runs.

Widgets: ${widgetTitles}

Metrics: ${metricIds}

`; + experiment.report = { html, model }; + await writeStored(ctx, experiment); + return { ok: true, html, model }; + }, + + async getexperiment(ctx, req) { + const experiment = await readStored(ctx, body(req).experimentId || req.query?.experimentId); + return experiment || { error: "EXPERIMENT_NOT_FOUND" }; + }, + + async listmetrics() { + return { ok: true, metrics: ["command.metric", "cost.totalUsd", "time.wallClockMs"] }; + }, + + async listwidgets() { + return { ok: true, widgets: ["summary-cards", "raw-drilldown"] }; + }, + + async cancel(ctx, req) { + const experiment = await readStored(ctx, body(req).experimentId); + if (!experiment) return { error: "EXPERIMENT_NOT_FOUND" }; + experiment.status = "cancelled"; + await writeStored(ctx, experiment); + return { ok: true, cancelled: true }; + }, +}; diff --git a/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/pack.yaml b/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/pack.yaml new file mode 100644 index 000000000..dc8acd8bb --- /dev/null +++ b/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/pack.yaml @@ -0,0 +1,26 @@ +name: experiment-runner +description: Local Experiment Runner smoke fixture for deterministic E2E coverage. +version: 1.0.0 +contents: + roles: [] + tools: [] + skills: [] + entrypoints: + - session-menu + - composer-slash + - deeplink +routes: + module: lib/routes.mjs + names: + - defineexperiment + - launch + - poll + - collect + - aggregate + - savemetrics + - savedashboard + - report + - getexperiment + - listmetrics + - listwidgets + - cancel diff --git a/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/panels/experiment-runner.yaml b/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/panels/experiment-runner.yaml new file mode 100644 index 000000000..3d689d05c --- /dev/null +++ b/tests/fixtures/market-sources/experiment-runner-smoke-src/experiment-runner/panels/experiment-runner.yaml @@ -0,0 +1,3 @@ +id: experiment-runner.panel +title: Experiments +entry: ../lib/panel.js From f0ea2634d157e4bbf66dbbb30ba6d86754e70f1c Mon Sep 17 00:00:00 2001 From: Artur Jonkisz Date: Sun, 21 Jun 2026 02:03:52 +0100 Subject: [PATCH 4/4] Finalize experiment runner smoke docs Co-authored-by: bobbit-ai --- docs/experiment-runner-smoke-test.md | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/docs/experiment-runner-smoke-test.md b/docs/experiment-runner-smoke-test.md index 97a90b36f..8b95aae46 100644 --- a/docs/experiment-runner-smoke-test.md +++ b/docs/experiment-runner-smoke-test.md @@ -8,7 +8,7 @@ Experiment id: `smoke-ab-20260621003841` ## Result -**Status: pass with caveats.** The runner is ready for a small real Graphify A/B benchmark, provided the real run uses a realistic per-run budget and an intentional workflow choice. The smoke test verified pack activation, UI launchers, A/B child-goal fan-out, treatment metadata propagation, route lifecycle calls, editable metric/dashboard specs, report generation, and autoresearch guardrails. +**Status: pass with caveats.** The runner is ready for a small real Graphify A/B benchmark, provided the real run uses a realistic per-run budget and an intentional workflow choice. The smoke test verified pack activation, UI launchers, A/B child-goal fan-out, treatment metadata propagation, route lifecycle calls, editable metric/dashboard specs, report generation, autoresearch guardrails, and non-optional UI-driven E2E fixture coverage. Caveats: - The intentionally low per-run budget (`0.05`) was below actual model startup cost, so both smoke arms were marked `failed` with `over_budget` during `poll`. This still verified bounded-budget enforcement and report generation, but did not produce collected success metrics. @@ -207,7 +207,30 @@ Final session inspection showed no live sessions for either child goal. ## Automated coverage -Added optional browser E2E coverage in `tests/e2e/ui/experiment-runner-smoke.spec.ts`. The test first checks `/api/ext/contributions` and skips with an explicit annotation when `experiment-runner` is absent on `origin/master`; when present, it exercises the deep link, session-menu launcher, slash launcher, bounded A/B route lifecycle, metadata propagation, spec persistence, reporting, and cancel cleanup. +Added non-optional browser E2E coverage in `tests/e2e/ui/experiment-runner-smoke.spec.ts`. The test installs the deterministic local fixture pack from `tests/fixtures/market-sources/experiment-runner-smoke-src/`, so it no longer depends on `experiment-runner` already being installed on `origin/master` and does not skip when the pack is absent. + +The fixture pack contributes the same smoke-test surfaces required by this goal: + +- Session-menu launcher: `New experiment` +- Composer slash launcher: `Experiments` +- Deep link: `#/ext/experiment-runner` +- Panel: `experiment-runner.panel` +- Routes: `defineexperiment`, `launch`, `poll`, `collect`, `aggregate`, `savemetrics`, `savedashboard`, `report`, `getexperiment`, `listmetrics`, `listwidgets`, `cancel` + +The E2E is UI-driven for the launch path. It creates a temporary parent goal, starts its team lead to provide an effective parent goal/session, opens the Experiment Runner through the session menu, slash launcher, and deep link, fills the panel fields, clicks `Define experiment`, clicks `Confirm and launch`, and asserts exactly two child goals are created under that parent. + +Fixture child-goal expectations are exact: + +- Arms: `baseline`, `variant-b` +- Repeats: one per arm +- Per-run budget: `0.05` +- Baseline treatment: `metadata.smokeTreatment = { arm: "baseline", marker: "smoke-baseline-101" }` +- Variant-B treatment: `metadata.smokeTreatment = { arm: "variant-b", marker: "smoke-variant-b-202" }` +- Experiment metadata includes `metadata.experiment.id`, `armId`, `repeat`, `budget`, and `planId` + +The test then drives the dashboard lifecycle from the panel (`poll` → `collect` → `aggregate`), edits and saves metric/dashboard JSON (`cost.totalUsd`, `time.wallClockMs`, `edited-summary`, `edited-raw`), generates a report, reloads the app, reopens `#/ext/experiment-runner?experimentId=...&view=report`, verifies the saved specs/report persisted through `getexperiment` and `report`, and cancels/cleans up the temporary experiment goals. + +Autoresearch coverage remains a guardrail only: the test calls `defineexperiment` with `mode: "autoresearch"` and no finite hard caps, expects `AR_UNCAPPED`, and never launches an autonomous loop. It also fails on `NO_EFFECTIVE_GOAL`, `SPAWN_GOAL_UNAVAILABLE`, parent mismatch, pack-route, or workflow errors in route responses, browser alerts/status messages, or console output. ## Readiness notes