From 4e7fce1fab1b96f149a278b77924cb7bb2c03bd4 Mon Sep 17 00:00:00 2001 From: Artur Jonkisz Date: Wed, 10 Jun 2026 19:26:51 +0100 Subject: [PATCH 01/18] fable docs --- .../04-current-state-and-backlog.md | 917 ++++++++++++++++++ docs/design/sidebar-goal-nesting-audit.md | 433 +++++++++ 2 files changed, 1350 insertions(+) create mode 100644 docs/design/comms-stack/04-current-state-and-backlog.md create mode 100644 docs/design/sidebar-goal-nesting-audit.md diff --git a/docs/design/comms-stack/04-current-state-and-backlog.md b/docs/design/comms-stack/04-current-state-and-backlog.md new file mode 100644 index 000000000..9dcf68191 --- /dev/null +++ b/docs/design/comms-stack/04-current-state-and-backlog.md @@ -0,0 +1,917 @@ +# Bobbit Real-Time Comms Stack — Step-4: Current-State Verdict + Hand-off Task Backlog + +> **Date:** 2026-06-10 · **Baseline:** master @ `6ec8c8f9` · Follows +> [01-understanding.md](01-understanding.md) (S1–S46 seam table), +> [02-analysis.md](02-analysis.md) (verdicts, RC1–RC8, test-fidelity §4), +> [03-remediation-plan.md](03-remediation-plan.md) (WP0–WP11 / PR-0..D). +> +> **What this document is:** (§2) a seam-by-seam verdict on what actually shipped vs. what the +> plan promised; (§3) settlement of the open NEEDS-TRACE residues; (§4) a register of NEW +> defects found beyond the 46 catalogued seams; (§5) the updated invariant ledger with every +> unpinned invariant flagged; (§6) **the deliverable — an ordered backlog of self-contained +> hand-off tasks** (each executable by an agent who sees only that task's text), grouped into +> waves with acceptance gates. +> +> **Method:** every claim was re-verified against the current tree by parallel adversarial +> audits (file:line anchors below are current as of `6ec8c8f9`, not the drifted anchors in +> docs 01–03). SDK ground truth was read from the installed `pi-coding-agent`/`pi-agent-core` +> and cross-checked against the published 0.77.0 tarball. +> +> **Environment caveat (re-verify before implementing):** `package.json` declares +> `@earendil-works/pi-* 0.77.0` but this checkout's `node_modules` contains only +> `@mariozechner/pi-*` (pi-coding-agent at **0.67.5**). The SDK-behaviour claims below +> (steer expansion-before-queue, steering queue retained across abort, ack-after-compaction, +> `switch_session` no-replay, SDK-internal auto-retry default-on) were verified against the +> published 0.77.0 sources and agree across three independent reads — but implementers must +> `npm install` and re-confirm SDK anchors in their own tree. + +--- + +## 1. Executive summary + +**What shipped.** One PR (#674, `d520620f`, 2026-06-01) landed: WP0 (partial), WP1 (RC2 +images), WP2 (RC1 dedup, minus the prefixed-S17 leg), WP3 (S2 outbox + S31 caps), WP4 (RC3 +seq-less broadcasts, minus the snapshot-reconcile step), the S9 half of WP5, the S8 +grace-race + S40 halves of WP9/WP10, S42, S14, S3, and (earlier, separately) the WP11 +search-flush teardown fix. **What did not ship:** WP5's S25 leg, WP6 (S16/S38), WP7 +(S19/S20), WP8 except S14 (S32/S34/S35/S37), most WP10 standalones (S26/S33/S36/S43), S4, +S13, S29, two of the three WP11 flake fixes, and several promised WP0 harness legs (faithful +mock abort shape, `message_start`, steer-image threading, the behavioural auto-retry/status +harnesses). + +**Symptom-family status.** +- **(B) images** — substantially fixed (render-from-content + live enrichment, faithful + pins). Residuals: S26 steered images (data-loss, untouched), a **new deterministic + duplicate** on blank-text attachment-only sends (F10), document attachments still never + reach the model (S20). +- **(A) duplicates** — reducer-level holes closed and pinned. Residuals: the prefixed + error-recovery echo (S17 leg, deliberately deferred, still open), a raw-role keying edge + (F11a), S4 double-Enter (never shipped), and several **new server-side duplicate-turn + mechanisms** that dwarf the reducer class: SDK-vs-Bobbit double auto-retry (F5), + graceful-abort steer double-delivery (F3a), recovery-after-kill at-least-once re-dispatch + (F2b, F9). +- **(C) freeze / dead Stop / lost prompts** — Stop is now reliable (3s force-kill, seq'd + synthetic `agent_end`); the outbox closes the headline silent-loss. Residuals: S25 resume + stall (open), outbox loss on navigation-away and mid-flush (F14), steers parked dormant in + the SDK (F3b — deterministic, including across restart), prompts that never auto-drain + after `preparing`/restore (F7), **a gateway-crash bug** (F1), and a respawn-concurrency + cluster that can split-brain tabs and corrupt the status channel (F2). +- **(D) jank / teardown** — almost nothing shipped (only S14/S31). S19/S20/S32/S34/S35/S37 + all still open as catalogued. + +**The biggest single discovery of this pass:** the prior audit's queue/steer/respawn +*interaction surfaces* hide a cluster of HIGH-severity, mostly deterministic server-side +bugs (F1–F7) that none of the 46 catalogued seams covered. They share three root causes: +**(RC-A)** no lifecycle fencing of the old `SessionInfo` across in-place respawn; **(RC-B)** +Bobbit's `idle` status conflates "dispatchable" with two busy states the SDK actually has +(compacting, internal-retry backoff); **(RC-C)** steer delivery is neither at-most-once nor +at-least-once — the shadow ledger, the SDK steering queue, and the persisted prompt queue +each own the text at different moments with no reconciliation contract. + +--- + +## 2. Seam-by-seam current status (all 46 + plan packages) + +Verdicts: ✅ fixed+faithfully pinned · 🟡 partially fixed / fixed-with-open-edge · +❌ still open (unshipped) · ⚪ refuted/obsolete (no action). Anchors are current. + +| Seam | Verdict | Current state (anchors) | +|---|---|---| +| S1 slot | 🟡 | Image leg closed: slot demoted to one-shot fallback `remote-agent.ts:2562-2580`, live enrich `message-reducer.ts:241`. Open: document/PDF tiles still slot-dependent; fallback branch itself unpinned. | +| S2 send-drop | 🟡 | Outbox shipped (`remote-agent.ts:243-245,1435-1477`, flush on auth_ok `:749`; pinned by `tests/remote-agent-outbox.spec.ts`, real bundle). Open edges → **F14**: navigation-away discards a non-empty outbox (`src/app/session-manager.ts:930-941` caches only when `connected`); `_flushOutbox` doesn't re-push on failure (`:1472-1477`); `unsent:true` rendered by nothing; no browser E2E. | +| S3 IME | ✅ | `MessageEditor.ts:507-513` (isComposing + keyCode 229, before slash-menu). Pinned: `message-editor-ime.spec.ts` (real component). | +| S4 double-Enter | ❌ | Never shipped (assigned PR-B). Editor still clears after awaits (`AgentInterface.ts:1467-1491`), `onSend` un-awaited `:2113`, no re-entrancy flag; fixture clears synchronously (inverse of prod). → **CS-D9**. | +| S5 seq-less bypass | ✅ | All three frames via `emitSessionEvent` (`session-manager.ts:2642,2677,6166`). Pinned: `seqless-broadcast-exhaustive.test.ts` (source-walk, scoped to session-manager.ts only — caveat noted in ledger). Exhaustive re-scan found **no remaining S5-class hole**; the on-attach `compaction_start` unicast (`handler.ts:356`) and the `BOBBIT_E2E` replay shim are intentional/benign. | +| S6 image render gap | ✅ | `Messages.ts:246-267` rich-wins-else-content + `imageAttachmentsFromContent:789-809`. Pinned: `user-message-image-render.spec.ts` (real Lit bundle). | +| S7/S10 id-less empty-text dup | ✅ | `synth:seq:` stamp `message-reducer.ts:249-251`, `plainTextEquivKey:90-94`, empty-aware multisets `:393-399,441-451`. Pinned: `message-reducer-dedup.test.ts` (real reducer). Full-stack E2E leg still missing (mock abort shape unfaithful) → **CS-H1**. | +| S8 wedged Stop | 🟡 | Grace race fixed: listener-before-abort, un-awaited IIFE+catch, force-kill at ~3s (`session-manager.ts:6101-6136`), seq'd synthetic agent_end `:6166`. Watchdog deliberately dropped (plan §2.5.2). Residual: force-kill branch never clears `streamingStartedAt`/persisted `wasStreaming` → stale heartbeat payload + phantom continuation after a later hard crash → **CS-R9**. Pinned: `session-manager-force-abort-grace.test.ts` (real SessionManager). | +| S9 overflow stall | ✅ | Overflow sets `_seqInitialized=false` (`remote-agent.ts:1672-1683`); three recovery paths consistent. Pinned: `remote-agent-seq-overflow.spec.ts` (real bundled `handleServerMessage`). | +| S11 hand-off flicker | ⚪ | Settled by analysis: reducer mutates state before emit; container clear + list inclusion commit in the same task's microtask drain — no paintable blank frame. No action; pin only if the hand-off ever moves off the synchronous path. | +| S12 | ⚪ | Stays refuted (container self-heal pinned). | +| S13 30s ack double-dispatch | ❌ | Unchanged: `rpc-bridge.ts:445` 30s default, pending deleted on timeout `:454-457`, `compact()` gets 120s `:521`; recovery re-dispatches via `recoverPromptDispatch` (`session-manager.ts:2166-2209`). → **CS-R8**. | +| S14 UTF-8 split | ✅ | Persistent `StringDecoder` (`rpc-bridge.ts:213-214,374,380`). Pinned: `rpc-bridge-utf8-split.test.ts` — real decoder + real `handleData`, but the test mirrors the stdout wiring itself (a revert of `:374` would pass); decoders never `end()`-flushed / not re-created in `_attachProcessHandlers` (low — respawns build new bridges). Hardening pin → **CS-H2**. | +| S15 switch_session seq inflation | ⚪ | Closed from source (0.77.0 AND 0.67.5): `switchSession` rebuilds state with a single `session_start`, no per-message replay. The `restoring` guard comment at `session-manager.ts:3694-3704` is **stale and false** → cleanup in **CS-P3**. | +| S16 continuation prompt | ❌ | WP6 never shipped: `restoreSession:3766-3776` unconditional on `wasStreaming`; all grant modes + restartAgent + role-switch reach it via `_respawnAgentInPlace:3015`. → **CS-D6** (after CS-R2). | +| S17 skill-expand dup | 🟡 | Unprefixed client fallback shipped (`reconstructModelText`, `message-reducer.ts:102-117,282-288`) but is **inert in production** (expansion is server-side; optimistic rows never carry `skillExpansions`). The real defect — prefixed error-recovery echo missing the exact-match splice (`session-manager.ts:553-584` vs prefix at `:1941`) — is still open and now joined by the same-seam envelope leak (huntB C10). → **CS-D3**. Characterization pin at `message-reducer-dedup.test.ts:75`. | +| S18 optimistic survivor | 🟡 | Multiset consume shipped + over-dedup non-regressions pinned (`message-reducer.ts:485-499,301-312`). Open edge: `plainTextEquivKey` uses **raw role**, so a `user-with-attachments` optimistic vs `role:user` snapshot copy (documents / blank-text) never matches → duplicate (F11a) → **CS-D2**. | +| S19 queue base64 broadcast | ❌ | `broadcastQueue` still broadcasts+persists `toArray()` (`session-manager.ts:1782-1789`); a single ~1.5MB image can trip the 4MiB overflow-terminate on a slow client. → **CS-P1**. | +| S20 document bytes | ❌ | `remote-agent.ts:950-955` ships full attachments; `dispatchDirectPrompt` (`:2225`) forwards only `(text, images)`. → **CS-P1**. | +| S21 orphaned banner | 🟡 | Frames now seq'd+replayable (S5 fix). Open: snapshot handler never clears `_state.autoRetryPending` (WP4 step 4 unimplemented; only writers `remote-agent.ts:2374,2386,2401`); the `clients.size===0` cancel-skip (`session-manager.ts:2663-2679`) + forceAbort-idle path leaves a stuck banner for a later resume. → **CS-D7**. | +| S22 status mirror tests | ❌ | `session-manager-status.test.ts` inline mirrors and `fixtures/remote-agent-status.html` (omits `_maybeReplayGrant` + archived branch) still the only pins. → **CS-H2**. | +| S23 seq fixture mirrors | 🟡 | Overflow branch now real-code-pinned; plain dedup/reorder/baseline + `_advanceTopLevelSeq` + resume_gap still mirror-pinned (`remote-agent-seq-dedup.html`, `remote-agent-sequence-hole.html`). → **CS-H2**. | +| S24 | ⚪ | Stays refuted. | +| S25 perm resume hole | ❌ | `EventBuffer.pushFrame` still non-retaining (`event-buffer.ts:41-43`); resume `since()` has a permanent hole across a DENIED/TIMED-OUT perm; `event-buffer.test.ts:209-233` still pins the **pre-fix** contract. Severity now medium-low (S9 overflow self-heals after 500 events). → **CS-D8**. | +| S26 steered images | ❌ | Four drop points: `rpc-bridge.ts:479-481` (no images param), batch `session-manager.ts:2073,2089`, rollback `:2098-2100`, reconcile `:2136-2138`, drain batch keeps only `steered[0].images` `:2257-2259`. SDK supports steer images (rpc-mode `session.steer(message, images)`). → **CS-D4**. | +| S27 steer-echo expansion mismatch | **confirmed-latent** | Settled from source, no unobservable left: SDK `steer()` expands `/skill:`-prefixed and template-prefixed text **before** queueing (agent-session `_expandSkillCommand`/`expandPromptTemplate`); Bobbit's ledger holds raw text (`session-manager.ts:2073,2087`); `_consumeSteerEcho` exact-`indexOf` misses (`:2114-2123`, its own comment admits it) → stale entry re-dispatches on a later abort (skill runs twice). → folded into **CS-R3/CS-R4**. | +| S28 | ⚪ | Stays refuted. | +| S29 reconnect refetch | ❌ | `onReconnect` (`remote-agent.ts:755-771`) → 3 REST calls per reconnect (`src/app/session-manager.ts:1384-1390`); git-status deduped, bg-processes + annotations not. Mitigated by backoff. → **CS-P4**. | +| S30/S41/S43/S44/S45 | ⚪ | Stay refuted (S43's residual listener leak → **CS-P3**). | +| S31 payload caps | ✅ | `WS_MAX_PAYLOAD_BYTES=256MiB` (`server.ts:264,1346`); composer measures the **real serialized frame** (`MessageEditor.ts:56,62-73`) as the FIRST statement of `handleSend` (`:654-668`, draft preserved). Pinned: `ws-max-payload.test.ts` + `message-editor-size-guard.spec.ts` (real component). | +| S32 proposal scans | ❌ | `remote-agent.ts:2125-2170`, fresh RegExps per delta. → **CS-P2**. | +| S33 truncated flush | ❌ | `remote-agent.ts:2462-2468` plain `break`. → **CS-P3**. | +| S34 rect loop | ❌ | `AgentInterface.ts:672-684` via `_updateAndPin:723-733`. → **CS-P2**. | +| S35 sync base64 | ❌ | `attachment-utils.ts:88-95`. → **CS-P2**. | +| S36 unguarded handler broadcast | ❌ | `handler.ts:154-196`. → **CS-P3**. | +| S37 aigw header fork | ❌ | `aigw-manager.ts:381-389`; the two aigw tests **pin the defect**. → **CS-P2**. | +| S38 grant replay | 🟡 | Delivery leg fixed (replay rides the outbox). Open: fixed 200ms timer, still fired from the idempotent/heartbeat branch (`remote-agent.ts:1232-1241,1719-1721`); zero tests of any kind. → **CS-D6**. | +| S39 deferred-block a11y | ❌ | Unchanged; correctly re-scoped out of WP10 (own staged package; not in this backlog — UI-a11y, not comms). | +| S40 forceAbort cancels retry | ✅ | `session-manager.ts:6089-6093` before the `:6096` early-return. Pinned in the force-abort-grace test (real SessionManager). | +| S42 splice multiset | ✅ | `splice-inflight-message.ts:85-130` count-map. Pinned: `session-manager-getmessages-splice.test.ts:170-186`. | +| S46 mime mislabel | ✅/🟡 | MIME preserved end-to-end. Cosmetic residual: `image-N.png` filename hardcoded in both helpers — and post-WP1 the generic name now shows **live** too. Fold into any future image touch (noted, not tasked). | +| WP11 | 🟡 | search-flush ENOENT ✅ (awaited `closeAll`, `remove()` deliberately fire-and-forget per the docs deviation note); project-assistant-saved-state ✅; pre-compaction-history ❌ (no `__pinAgentSessionFile`); dynamic-chat-tabs ❌ (still samples the GET-mount hash, not `currentFilenameTab.state.contentHash`). → **CS-T1**. | +| repro-h3 case (A) | fixme | Still `test.fixme` (`repro-h3-snapshot-live-interleave.spec.ts:144`), correctly — owned by the separate snapshot-live-race goal; the deterministic gate (WP0 step 10) was never authored. → **CS-T2**. | + +--- + +## 3. NEEDS-TRACE residues — settled + +| Residue | Outcome | +|---|---| +| **S11** | **Refuted as a defect.** The state mutation precedes the emit; the container clear and the message-list inclusion are issued synchronously in one task and commit in its microtask drain — the browser cannot paint between them. The two-scheduler structure remains (re-pin only if the hand-off is ever made async). | +| **S13** | **Confirmed latent, fix unshipped.** Code path fully proven (ack emitted only after in-prompt auto-compaction; 30s vs the maintainers' own 120s compact budget). The wall-clock measurement is not decision-relevant. → **CS-R8**. | +| **S15** | **Closed.** No per-message replay in `switchSession` (verified in both 0.77.0 and the installed 0.67.5). The `restoring` guard's comment is stale/false → cleanup in **CS-P3**. | +| **S27** | **Settled — confirmed latent with an exact trigger** (no unobservable left): a live steer whose text starts with `/skill:` or `/` is expanded by the SDK before echo; the raw-text ledger entry never clears; a later abort in the same process lifetime re-dispatches it (skill runs twice). Severity low-medium; compounds with F4 (ledger never reconciled on respawn). → **CS-R3/CS-R4**. | +| **S29** | **Still open by design** (PR-D deferred); partially mitigated (reconnect backoff, git-status in-flight dedup, seq-resume instead of full refetch). → **CS-P4**. | + +--- + +## 4. New findings register (beyond the 46) + +Detailed diagnosis/repro/fix lives in the task cards (§6); this table maps finding → task. +Severity = (data-loss vs transient) × reachability. + +| ID | Finding (one line) | Severity | Task | +|---|---|---|---| +| **F1** | `drainQueue`'s bare synchronous `rpcClient.prompt()` throw inside the recovery `setTimeout(0)` is an `uncaughtException` → `process.exit(1)` — **the whole gateway dies** (window: in-flight dispatch ∩ in-place respawn) | **HIGH (total outage)** | CS-R1 | +| **F2** | No lifecycle fencing across `_respawnAgentInPlace`: stale-`SessionInfo` writers corrupt `statusVersion` past the carried frame-of-reference (permanently wrong status, heartbeat can't heal), clobber the persisted store (duplicate turn), leave the auto-retry timer alive (phantom turn into the new process); concurrent respawns / `addClient` dormant-revives spawn N children on one `.jsonl` (split-brain tabs, leaked processes, lost seq seed); grant `newTools` computed from role not `session.allowedTools` (double grant-respawn) | **HIGH** | CS-R2 | +| **F3** | Steer delivery semantics: (a) graceful abort + in-flight steer → text delivered **twice** (SDK retains it, Bobbit re-enqueues from the ledger); (b) the `agent_end` "safety net" (`session-manager.ts:2413-2418`) and the live-steer/turn-end race dispatch `steer()` to an **idle** SDK → text parks dormant (no echo, pill gone, phantom splice row, injected into an arbitrary future turn — or lost on restart) | **HIGH (duplicate model instruction / silent input loss)** | CS-R3 | +| **F4** | Steer ledger durability: in-memory only — in-place respawn and gateway crash silently destroy accepted-but-undelivered steers (bubble persists, model never sees it); `_dispatchSteer`'s failure rollback double-enqueues when a reconcile already drained the ledger; S27 expansion mismatch leaves permanent entries | **MEDIUM-HIGH (input loss)** | CS-R4 | +| **F5** | The SDK's **internal** auto-retry is enabled by default (2s base) and Bobbit's `maybeAutoRetryTransient` (~1s) races it: Bobbit's retry fires into the SDK's idle backoff window → duplicate user message / unsolicited turn / double spend; SDK-internal attempts also inflate `consecutiveErrorTurns` toward the human-gate park | **HIGH (duplicate turns, cost)** | CS-R5 | +| **F6** | No compaction gate on any dispatch path: `enqueuePrompt`/`drainQueue`/`retryLastPrompt`/implicit-unstick all dispatch into an in-flight compaction (status is `idle` during compaction); SDK `compact()` clobbers in-flight turn state, and **manual compact disconnects all agent listeners** → a turn started then streams into the void and skips `.jsonl` persistence (permanent message loss) | **HIGH (message loss / context corruption)** | CS-R6 | +| **F7** | Prompts accepted during `preparing`/`starting` and queues restored at boot **never auto-drain** (no idle-transition drains the queue); during a respawn window `enqueuePrompt` returns success-shaped `{status:"queued"}` without enqueuing | **MEDIUM (delivery stall / silent loss)** | CS-R7 | +| **F8** | = S13 (30s ack double-dispatch), confirmed unshipped | MEDIUM | CS-R8 | +| **F9** | `messageQueue` + `wasStreaming` ride the 1s store debounce (not in `RECOVERY_CRITICAL_FIELDS`): hard-kill windows yield a phantom continuation turn, a duplicate prompt (at-least-once re-dispatch of an accepted prompt), or a lost queued prompt; plus the WP9 residual (force-kill never clears `streamingStartedAt`/`wasStreaming`) | MEDIUM | CS-R9 | +| **F10** | Blank-text attachment-only send → **deterministic duplicate user bubble**: server synthesizes `ATTACHMENT_ONLY_TEXT` ("Attachments:"), optimistic text is `""`, no reconciliation path matches | MEDIUM (deterministic, single-tab) | CS-D1 | +| **F11** | Reducer hardening: (a) raw-role `plainTextEquivKey` defeats S18 dedup for document/blank optimistic rows; (b) live-echo artifact consume picks the EARLIEST same-text historical row in a no-optimistic tab (multi-tab row vanishes); (c) `synth:seq:` ids reuse after a gateway restart (resume_gap doesn't reset reducer state) → sub-second row morph | LOW-MEDIUM | CS-D2 | +| **F12** | = prefixed-S17 + huntB C10: error-recovery prefix breaks the skill-expansion splice AND leaks the orphaned envelope in `pendingSkillExpansions` (can mis-splice a future byte-identical message) | LOW-MEDIUM | CS-D3 | +| **F13** | = S26 steered images (4 drop points) | MEDIUM (data loss) | CS-D4 | +| **F14** | Outbox edges: navigation-away discards a non-empty outbox (silent loss with "queued" pill shown); `_flushOutbox` drops frames on mid-flush close/throw (no re-push); `unsent:true` has zero UI readers; outboxed `retry` frames invisible; no spawned-gateway E2E (the auth_ok→flush wiring is untested) | MEDIUM | CS-D5 | +| **F15** | = S16 + S38 (grant/restart continuation double-prompt; heartbeat-branch replay) | MEDIUM | CS-D6 | +| **F16** | = S21 residual (snapshot never clears `autoRetryPending`; zero-clients cancel orphan) | LOW | CS-D7 | +| **F17** | = S25 perm resume hole | MEDIUM-LOW | CS-D8 | +| **F18** | = S4 double-Enter | MEDIUM | CS-D9 | +| **F19** | Late RPC `response` frames after a `sendCommand` timeout are re-broadcast as seq'd session events (`rpc-bridge.ts:688-698`); a straggling compact response can synthesize a spurious `compaction_end{success:false}` client-side (`remote-agent.ts:2706-2718`) | LOW | CS-P3 | +| **F20** | `_advanceTopLevelSeq` gap path + skipped `compaction_end` → snapshot handler re-adds the compacting placeholder (stale "Compacting…" card). Narrow post-S5 | LOW | (documented; fold into any CS-D8 follow-up) | + +**NEEDS-TRACE (named unobservable + settling experiment):** +1. *Extension-command steer infinite redispatch* — a steer starting with a registered + extension command makes SDK `steer()` throw → rollback re-enqueues → re-fails at every + tool boundary, silently. Settle: steer `/login` mid-turn against the real agent; watch + `queue_update` churn. (If confirmed → fold into CS-R3 acceptance.) +2. *Stop during prompt preflight ghost bubble* — abort before the run starts → no + `agent_end` for the pending prompt → `recoverPromptDispatch` deliberately drops it; the + optimistic bubble may persist with no echo until the next snapshot. Settle: client-side + trace of the rendering outcome. (Arguably intended — user pressed Stop.) +3. *Compaction spinner stuck after `restart_agent` mid-compaction* — child killed before + `compaction_end`; handler flips `isCompacting` on the old object only. Settle: mock + harness — manual compact, restart mid-flight, assert the card reaches a terminal state. + (CS-R6's gate should make this window rarer; verify there.) +4. *`exit`-vs-buffered-stdout race* triggering F1 without a respawn (child prints + `agent_end` then exits; Node may deliver `exit` first). Settle: stub child. CS-R1's fix + covers it regardless. +5. *`deliverLiveSteer` cap-park doesn't cancel a pending auto-retry timer* + (`session-manager.ts:1994-2003` vs `enqueuePrompt:1880`) — product decision more than a + trace; note in CS-R5. + +--- + +## 5. Invariant ledger (current) — abridged to deltas + unpinned set + +The full 50-invariant ledger with per-invariant enforcement points and fidelity judgments +was compiled during this audit; the load-bearing summary: + +**Faithfully pinned (real production code under test):** IME guard; composer size guard +(first-statement ordering + draft retention); outbox enqueue/flush/pill (file:// real +bundle); overflow re-baseline (real `handleServerMessage`); seq-less-bypass-empty +(structural, scoped); EventBuffer push/seq/eviction/canResumeFrom; synth:seq stamping + +empty-text multisets + over-dedup non-regressions (real reducer); live==snapshot image +enrichment (real reducer) + tile render (real Lit bundle); streaming-row single-surface; +StreamingMessageContainer self-heal; forceAbort grace race + retry-cancel (real +SessionManager); splice multiset; UTF-8 decoder (object-level); queue persistence +exactly-once across restart (`steer-gateway-restart.spec.ts`); store durability +(tmp+fsync+rename + epoch latch); search-flush teardown. + +**UNPINNED or mirror-pinned (each is a gap a regression walks through silently):** +1. S4 double-Enter — unenforced and unpinned. (CS-D9) +2. Prompt slow-ack budget (S13) — unpinned; mock acks instantly. (CS-R8) +3. Steer shadow-ledger transitions (push/rollback/consume/reconcile incl. the S27 + miss-path) — zero unit coverage; steer E2Es are happy-path. (CS-R3/R4) +4. Steered-image forwarding (S26) — unpinned. (CS-D4) +5. `broadcastQueue` payload projection (S19) — no projection exists. (CS-P1) +6. `snapshot-clears-streaming-message.test.ts` is a **source-text regex scan** — behaviourally unpinned. (CS-H2) +7. Snapshot-authoritative `autoRetryPending` clear — missing invariant + code. (CS-D7) +8. Auto-retry timer fire/cancel behaviour — `auto-retry-policy.test.ts` is a pure-decision + mirror; `queue-dispatch.spec.ts` uses a fake timer object; the `status!=='idle'` fire + guard is untested. (CS-H2, then CS-R5) +9. Handler-local broadcast back-pressure (S36) — unguarded and unpinned. (CS-P3) +10. Grant replay (S38) — zero tests. (CS-D6) +11. Respawn continuation suppression (S16) — unenforced. (CS-D6) +12. Perm-hole resume (S25) — unpinned; `event-buffer.test.ts:209-233` pins the *defective* + contract. (CS-D8) +13. Client seq dedup/reorder baseline, `_advanceTopLevelSeq`, resume_gap, and the status + state machine — still pinned only by hand-copied HTML mirrors + (`remote-agent-seq-dedup.html`, `remote-agent-sequence-hole.html`, + `remote-agent-status.html` — the latter omits `_maybeReplayGrant` and the archived + branch); heartbeat/resync server halves are inline mirrors in + `session-manager-status.test.ts`. (CS-H2) +14. The `eventSeq` plumb feeding the synth:seq stamp (`remote-agent.ts:2365`) — no + real-RemoteAgent test; a frozen/zeroed seq would collapse all id-less rows to one id. (CS-H2) +15. `broadcastStatus` single-writer routing — convention + comment only, no structural pin. (CS-H2) +16. Respawn `_snapshotStreamingFrameOfReference` threading — the carry-over tests mirror + the threading inline rather than driving `_respawnAgentInPlace`. (CS-R2 acceptance) +17. statusVersion monotonicity **across concurrent stale writers** — violated today (F2a); + no pin. (CS-R2) +18. Steer-ledger / queue / `.jsonl` exactly-once delivery contract — does not exist as an + invariant today (F3/F4/F9). (CS-R3/R4/R9 define and pin it) + +--- + +## 6. THE BACKLOG — ordered waves of self-contained hand-off tasks + +Conventions for every task: **master stays green** (`npm run check`, `npm run test:unit`, +`npm run test:e2e` at/above current baseline); **no flaky tests** (a failure is a real +bug); **test-first** — author the red test before the fix (where master-red would violate +the green gate, use a characterization-green + `test.fixme` tagged with this task's ID); +minimal root-cause change; never start background servers from bash (use the harness); +all anchors verified at `6ec8c8f9` — re-verify before editing. Wave order is +(symptom impact × confidence × low risk) under dependencies; tasks within a wave are +parallel unless a dependency says otherwise. + +--- + +### WAVE 0 — Harness fidelity (unblocks faithful pins for waves 1–2) + +#### CS-H1 — Complete the mock-agent fidelity work (WP0 leftovers + steer semantics) +- **Serves:** symptom families A and C; prerequisite for CS-R3, CS-D1, CS-D4 E2E legs. +- **Targeted code:** `tests/e2e/mock-agent-core.mjs` (abort case ~`:2112-2165`, steer case + ~`:2057-2110`, prompt echo ~`:679-690`), `tests/e2e/in-process-mock-bridge.mjs`, + `src/ui/components/Messages.ts:500/748` (add `data-testid="aborted-banner"` only), + NEW `tests/mock-abort-shape.test.ts`. +- **Diagnosis:** the mock still cannot express three real wire behaviours, so whole defect + classes are untestable: (1) the DEFAULT abort emits only `agent_end`+idle — the real SDK + emits `message_start → message_end(role:'assistant', stopReason:'aborted', + content:[{type:'text',text:''}], errorMessage) → turn_end → agent_end → idle`; the + empty-text aborted row is exactly the S7/S10 dedup class. (2) `message_start` is never + emitted anywhere. (3) The mock converts a steer into an immediate prompt and models + abort-with-pending-steer as a *drop* — the real SDK **retains** the steering queue across + a graceful abort and replays it at the next run's initial steering poll; all five steer + E2Es currently pass against wrong semantics. Steer `images` are also discarded. +- **Steps:** (a) default abort → the full faithful sequence above (keep `MOCK_ABORT_AS_ERROR=1` + as the explicit error-shape override); audit every abort-consuming spec for the new DOM + banner row / message counts / first-message_end ordering. (b) emit + `{type:'message_start', message}` before every `message_end`. (c) steer: retain + steered texts across a graceful abort and replay them (own `message_start/end(user)`) + at the start of the next run; add a `MOCK_STEER_DROP_ON_ABORT=1` override for the old + behaviour; thread `msg.images` through the steer case. (d) `data-testid="aborted-banner"`. +- **Repro/red tests:** `mock-abort-shape.test.ts` (node:test over `MockAgentCore`) asserting + the exact sequence incl. `turn_end`; a steer-retention unit case (steer → abort → next + prompt → echo includes the retained steer exactly once). +- **Non-goals:** no production `src/` change beyond the `data-testid` attribute; do not + change default echo timing. +- **Acceptance:** `git diff src/` contains only the testid; all existing unit+e2e pass with + updated assertions; the two new node:test files green; every `message_end` in mock output + preceded by `message_start`. +- **Risk:** the abort-shape change perturbs abort-consuming specs (known: `abort-status-e2e`, + `steer-during-bash-tool`, error-gated-drain tests) — widen assertions deliberately, do not + silence. Rollback: revert the mock file. + +#### CS-H2 — Retire the mirror/inline pins; add the missing real-code harnesses +- **Serves:** invariant-ledger gaps #6, #8, #13–15; S22/S23. +- **Targeted code:** tests only. Replace/supplement: `tests/fixtures/remote-agent-status.html` + + `remote-agent-status.spec.ts` (mirror omits `_maybeReplayGrant` + archived branch); + `tests/fixtures/remote-agent-seq-dedup.html` / `remote-agent-sequence-hole.html` (dedup, + reorder, baseline, `_advanceTopLevelSeq`, resume_gap branches); inline mirrors in + `tests/session-manager-status.test.ts:107-159`; `tests/snapshot-clears-streaming-message.test.ts` + (source-regex → behavioural); NEW `tests/session-manager-auto-retry-fire-cancel.test.ts` + (real `SessionManager` + `t.mock.timers`: fire-while-idle, cancel-on-new-prompt, + forceAbort-cancels — last one is green already, keep as regression pin); extend the + bundled real-RemoteAgent seq spec (`tests/remote-agent-seq-overflow.spec.ts` pattern / + `tests/fixtures/remote-agent-seq-entry.ts`) with: (i) dedup/reorder/baseline/resume_gap + cases (retiring the HTML mirrors), (ii) **the eventSeq plumb pin** — two id-less + `message_end`s at seqs N, N+1 → two distinct `synth:seq:` ids in `_state.messages`, + (iii) a `session_status` case exercising `_maybeReplayGrant` coupling + archived branch. + Add a stderr-tail + wiring variant to `tests/rpc-bridge-utf8-split.test.ts` (spawned stub + CLI writing a split multibyte char, so a revert of `rpc-bridge.ts:374` goes red). + Optional structural pin: every `session.status` write routes through `broadcastStatus` + (source-walk, mirroring `seqless-broadcast-exhaustive.test.ts`). +- **Acceptance:** each retired mirror either deleted or reduced to a comment pointing at the + real-code pin; new tests green; no production change. +- **Dependencies:** none; parallel with CS-H1. Low risk (test-only). + +**Gate 0:** `git diff src/` empty bar the testid; suites green; the mock can express the +real abort/steer/image shapes; the seq/status/auto-retry real-code harnesses exist. + +--- + +### WAVE 1 — Server reliability: crash, duplicate turns, lost input (the new F1–F7 cluster) + +#### CS-R1 — Gateway crash: make `drainQueue` un-crashable (F1) +- **Serves:** symptom C (and total-outage prevention). Root cause RC-A edge. +- **Targeted code:** `src/server/agent/rpc-bridge.ts:445-448` (`sendCommand` throws + synchronously when `!this.process?.stdin`); `src/server/agent/session-manager.ts:2289` + (bare `session.rpcClient.prompt(...)` in `drainQueue`), `:2208` (recovery + `setTimeout(0)` redrain), `src/server/cli.ts:280-296` (uncaughtException handler exits 1 + for non-EPIPE). +- **Diagnosis:** trigger — (1) a prompt dispatch is in flight (long window: >30s + auto-compaction before the ack); (2) any in-place respawn starts: `_respawnAgentInPlace` + calls `unsubscribe()` **before** `stop()` (`:3022-3024`), so `process_exit` never reaches + `handleAgentLifecycle` and status never becomes `terminated`; (3) the pending prompt + rejects "Agent process exited…" → `recoverPromptDispatch` passes its status gate + (`:2172-2176`, status still `streaming`) → re-enqueue + `setTimeout(0) drainQueue(oldSession)`; + (4) the timer fires with `oldSession.rpcClient.process === null` → **synchronous throw + inside a timer → uncaughtException → `process.exit(1)`** — every session dies. +- **Repro:** mock bridge whose `prompt()` never acks; `enqueuePrompt` (idle) then + `restartAgent()` while the dispatch is pending; assert the tick-0 redrain does not throw + out of the timer (today it does — assert via an injected uncaughtException listener in a + child-process harness, or unit-assert `sendCommand` returns a rejected promise). +- **Fix direction:** (a) make `sendCommand` never throw synchronously — return + `Promise.reject` for the no-process case; (b) belt-and-braces: `drainQueue` checks + `session.rpcClient.running` and wraps the dispatch in try/catch routing to + `recoverPromptDispatch`. **Non-goals:** the broader respawn fencing (CS-R2) — this task + only removes the crash. +- **Acceptance:** new red→green unit test (real `RpcBridge`): `sendCommand` with no child + rejects (never throws); a `drainQueue` against a stopped bridge recovers instead of + throwing. Existing `rpc-bridge-lifecycle.test.ts` green. +- **Risk:** callers relying on the synchronous throw (grep `sendCommand` call sites) — + verify each handles rejection. Rollback: two small reverts. + +#### CS-R2 — Respawn lifecycle fencing: mutex + old-`SessionInfo` neutralization (F2) +- **Serves:** symptom C (frozen status, split-brain tabs, duplicate turns, leaked + processes). Root cause **RC-A** — fixes F2a–f once. +- **Targeted code:** `src/server/agent/session-manager.ts` — `_respawnAgentInPlace` + (`:3015-3045`), `restoreSession` (`:3600-3780`), `restartAgent` (`:3052`), + `_restartSessionWithUpdatedRole` (`:2957`), `grantToolPermission` (`:2804-2870`), + `addClient` dormant revive (`:6028-6046`), `recoverSandboxSessions`, `ensureSessionAlive` + (`:5221`), forceAbort respawn (`:6226-6273`), `recoverPromptDispatch` (`:2166-2209`), + `maybeAutoRetryTransient`/timer fire (`:2594-2649`), `cancelPendingAutoRetry` call-site + audit, `drainQueue` (`:2240-2310`). +- **Diagnosis (five legs, one root cause — no fencing of the old object):** + (a) **statusVersion corruption:** the frame-of-reference is snapshotted at `:3023`, but + in-flight `recoverPromptDispatch`/redrain then broadcast idle/streaming **on the old + object**, bumping its statusVersion past the snapshot and delivering those versions to + still-attached clients; the new session re-issues the consumed numbers with different + statuses → the client's `v<=last` gate drops them; the heartbeat never bumps versions → + **permanently wrong status** (e.g. stuck "streaming"/Stop on an idle session). + (b) **store clobber → duplicate turn:** the stale redrain persists `wasStreaming:true` + and a `messageQueue` containing a row the old child may already have accepted into the + `.jsonl` (the kill ate the ack, not the prompt) while `restoreSession` consumes the same + `ps` → continuation prompt + re-dispatch = the same user turn runs twice. + (c) **auto-retry timer survives every respawn:** `cancelPendingAutoRetry` is called at + `:1880,:2734,:5329,:6093,:6320` — never on any respawn path; the timer's guards read the + OLD object (`sessions.has(id)` true via the new session; old `status` frozen `idle`) → + fires `retryLastPrompt` into the fresh process → phantom "[SYSTEM: The model API returned + an error…]" turn. + (d) **concurrent respawns:** no per-session mutex; `sessions.delete` happens after the + first await (child shutdown up to 3s), so a second trigger (double-click Restart, grant + click, `recoverSandboxSessions`, `ensureSessionAlive`, `addClient` revive) starts a second + full `restoreSession` → **two pi children appending to one `.jsonl`**, loser child leaks + (cost double-count); the first run's `finally` deletes `ps._restartFrameOfReference` so + the second restore can seed `EventBuffer` at seq 1 → every post-respawn frame silently + dropped by attached clients. + (e) **`addClient` dormant revive:** the dormant entry stays `terminated` in the map for + the whole restore; every attach in that window fires another `restoreSession`; each + revive's `.then()` registers its ws on whatever the map holds at that moment → tabs split + across two `SessionInfo`s. Also (f): `grantToolPermission` computes `newTools` from the + *role* rather than `session.allowedTools` (`:2804-2831`), so a second grant during the + first's respawn also respawns. +- **Repro:** (d/e) delaying `RpcBridgeFactory` (2s start); dormant session; `addClient` + twice 100ms apart → factory must be invoked once (today twice). (c) errored turn arms the + timer; `restartAgent`; advance mock timers → `retryLastPrompt` must not dispatch. (a/b) + never-acking bridge + `restartAgent` mid-dispatch → assert no statusVersion regression on + the new session and no duplicate row in the restored queue. +- **Fix direction:** a per-session **respawn generation + in-flight restore promise**: + every entry point (`addClient` revive, `restartAgent`, `_restartSessionWithUpdatedRole`, + forceAbort respawn, `recoverSandboxSessions`, `ensureSessionAlive`) joins the in-flight + promise instead of starting a second restore. A **neutralization epilogue** on the old + object at the top of `_respawnAgentInPlace`: mark it terminated/fenced, clear + `clients`, `cancelPendingAutoRetry(session)`, and stamp a generation that + `recoverPromptDispatch`, `drainQueue`, and the retry-timer closure check before acting + (stale generation → no-op). Fix (f) by computing `newTools` from `session.allowedTools` + for session-scoped grants. **Non-goals:** WP6's continuation-prompt suppression (CS-D6 — + but note both edit `restoreSession`/`_respawnAgentInPlace`: land CS-R2 first, rebase + CS-D6); prompt idempotency keys (CS-R9). +- **Acceptance:** red→green real-`SessionManager` tests for (c), (d/e), plus a + generation-fencing unit (stale recover/redrain no-ops); the in-place-respawn carry-over + pins (`restart-preserves-streaming-frame.test.ts`, `sandbox-recovery-*`) stay green and + are **extended to drive the real `_respawnAgentInPlace`** (closing ledger gap #16); no + statusVersion regression observable by a fake client across a respawn-with-inflight-dispatch. +- **Risk:** the highest-touch task in the backlog (one file, many functions). Blast radius: + every respawn path incl. sandbox recovery. Mitigate by keeping the fence read-only for + all paths except the five entry points; rollback = revert the epilogue + generation + checks. + +#### CS-R3 — Steer delivery semantics: exactly-once between Bobbit and the SDK (F3 + S27 trigger) +- **Serves:** symptom A (duplicate steer) + C (silently lost steer). Root cause **RC-C**. +- **Targeted code:** `src/server/agent/session-manager.ts` — the `agent_end` "safety net" + (`:2413-2418`), `_dispatchSteer` (`:2069-2105`), `_consumeSteerEcho` (`:2114-2123`), + `_reconcileAfterAbort` (`:2133-2141`) + its call sites (agent_end-while-aborting ~`:2427`, + forceAbort `:6160`), `deliverLiveSteer` (`:1994-2050`), drainQueue steered batch + (`:2254-2259`); `src/server/agent/rpc-bridge.ts` (steer `:479-481`; optionally a + `clear_queue` passthrough — SDK rpc supports it). +- **Diagnosis:** + (a) **Graceful abort double-delivery:** `_dispatchSteer` pushes the ledger and the SDK + enqueues the steer; user clicks Stop; the **graceful** settle path leaves the child alive + — the SDK retains the steer in its steering queue, but `_reconcileAfterAbort` also + re-enqueues the ledger text and the post-abort drain dispatches it via `prompt` → the + next run echoes the prompt AND the initial steering poll injects the retained original → + the steer appears twice in the transcript and twice in model context. The reconcile is + only correct for force-kill (process dead). + (b) **Dormant steer:** `rpcClient.steer()` on a non-running agent only enqueues — it + never starts a run. Two paths do this: the agent_end safety net (deterministic — steers + still queued at a non-tool-final `agent_end` are dispatched via `steer()` *after* the run + ended, strictly before `drainQueue` would have dispatched them correctly via `prompt`), + and the live-steer-vs-turn-end race (incl. team-manager nudges, `team-manager.ts:1907,1952`). + Result: pill removed, no echo, ledger entry never clears (phantom `inflight-steer:` row in + every snapshot), text invisible to the model until it pops into an arbitrary future turn — + or is permanently lost on gateway restart (ledger is memory-only, row already removed from + the persisted queue). + (c) **S27:** the SDK expands `/skill:`/template steers before queueing; the echo carries + expanded text; the raw-text ledger entry never matches → permanent entry → duplicate + dispatch on a later abort. +- **Repro:** (a) real-`SessionManager` + fake bridge that retains steer texts across + `abort()` and replays them on the next `prompt()` (CS-H1's mock semantics): streaming → + steer("X") → graceful abort → assert exactly one user-message "X" (today two). (b) fake + bridge that parks steers when no run is active: streaming text-only turn → + `steerQueued(id)` → `agent_end` → assert the text was dispatched via `prompt` or still + queued (today: `steer` RPC after agent_end, queue empty, ledger stuck). (c) steer text + `/skill:x`, echo the expanded body, `forceAbort` → assert `inFlightSteerTexts` empty and + no re-dispatch. +- **Fix direction:** (1) delete the agent_end safety net — leftover steered rows flow into + `drainQueue`'s existing steered-batch `prompt` path; (2) make `_reconcileAfterAbort` + bridge-aware: on graceful abort (process alive) do NOT re-enqueue — the SDK still owns + the text; keep the ledger entry so the splice keeps the bubble, and let the next run's + steering poll deliver it (or expose `clear_queue` over RPC and use it to make abort + at-most-once, then re-enqueue uniformly — pick one model and pin it); re-enqueue only on + force-kill; (3) `_dispatchSteer`'s catch: skip the row re-enqueue when the ledger splice + missed (another path owns the rows — fixes the delayed-failure double-enqueue); (4) make + `_consumeSteerEcho` expansion-tolerant (consume when the echo *starts with* / maps to a + ledger entry after `/skill:`-expansion, or track a dispatch id). **Non-goals:** ledger + persistence (CS-R4); steer images (CS-D4). +- **Acceptance:** the three repro tests red→green; all five steer E2Es green **against the + CS-H1 retain-and-replay mock** (they currently pass against drop semantics — they must be + re-validated, not just re-run); a new invariant test: after any abort path, for every + ledger entry there is exactly one of {SDK-queued copy, Bobbit-queued row} (the + exactly-once contract). +- **Dependencies:** CS-H1 (mock steer semantics). Coordinate with CS-R2 (same file; disjoint + functions except forceAbort — rebase order CS-R2 → CS-R3). +- **Risk:** steer UX semantics change subtly (a steer landing at turn-end becomes a queued + prompt — that is the *correct* documented semantics). Rollback per sub-fix. + +#### CS-R4 — Steer ledger durability: persist + reconcile on respawn (F4) +- **Serves:** symptom C (user input silently lost). Root cause RC-C persistence leg. +- **Targeted code:** `src/server/agent/session-manager.ts` `_dispatchSteer` (`:2074-2087` — + row removal persisted before the RPC; ledger push in-memory only), `_respawnAgentInPlace` + (no reconcile call), `restoreSession`; `src/server/agent/session-store.ts:33-118` + (`PersistedSession` — `inFlightSteerTexts` is not a field), `RECOVERY_CRITICAL_FIELDS` + (`:513-522`); `src/server/agent/splice-inflight-message.ts` (snapshot splice reads the + in-memory ledger). +- **Diagnosis:** a steer the SDK accepted mid-turn is destroyed by any in-place respawn + (grant click, role switch, Restart, sandbox recovery): queue row already removed (and the + removal persisted), ledger discarded with the old object, the `wasStreaming` continuation + prompt resumes the turn **without the user's redirection** — and on reload even the + bubble vanishes (splice reads an empty ledger). Across a gateway crash the same holds. +- **Repro:** real-`SessionManager`: dispatch a live steer (fake bridge holds it), then + `_respawnAgentInPlace` → assert the steer text is re-enqueued as a steered row (today: + gone). Crash leg: persist-and-reload cycle via the store, assert the ledger text survives + as a queued row. +- **Fix direction:** persist the ledger (a `pendingSteerTexts` field next to + `messageQueue`, synchronous-flush class); `restoreSession` re-enqueues persisted ledger + entries as steered rows at front; `_respawnAgentInPlace` calls the (CS-R3 + bridge-aware) reconcile before tearing down. **Non-goals:** changing live-steer dispatch. +- **Acceptance:** repro tests red→green; `steer-gateway-restart.spec.ts` extended with a + mid-turn live steer surviving the restart exactly once; no double-delivery when combined + with CS-R3 (the exactly-once invariant test covers the join). +- **Dependencies:** CS-R3 (defines the reconcile semantics). Risk: store schema addition — + forward-compatible (absent field = empty). + +#### CS-R5 — Disable the double auto-retry layer (F5) +- **Serves:** symptom A/C (duplicate turns, unsolicited turns, premature error-park, cost). + Root cause **RC-B**. +- **Targeted code:** `src/server/agent/rpc-bridge.ts` (bridge start — send + `{type:"set_auto_retry", enabled:false}`; the rpc command exists, rpc-mode `:410-413`) OR + `src/server/agent/session-manager.ts` `maybeAutoRetryTransient` (`:2594-2649`) + + `handleAgentLifecycle` (`consecutiveErrorTurns` increment `:2377`); the SDK's internal + retry is **enabled by default** (settings-manager `enabled ?? true`, maxRetries 3, base + 2000ms) and Bobbit never calls `set_auto_retry` (zero hits in `src/`). +- **Diagnosis:** on a retryable provider error the SDK emits `agent_end` first, then sleeps + 2s·2ⁿ⁻¹ and resumes via `agent.continue()` (no new user message). Bobbit sees the errored + `agent_end` → arms its own timer (~1s first attempt — earlier than the SDK's 2s) → fires + while status is idle (the SDK's backoff window is by construction an idle-status window) + → `retryLastPrompt` re-sends the prompt as a **new user message**; the SDK's own continue + then hits a busy agent and is swallowed. Duplicate user message / unsolicited "[SYSTEM: + continue]" turn / double spend; SDK-internal attempts also increment + `consecutiveErrorTurns` toward the human-gate park during an episode the SDK was + handling. +- **Repro:** mock agent emits `message_end{stopReason:'error', errorMessage:'overloaded_error'}` + + `agent_end`, then accepts the next prompt; assert exactly one driving dispatch. +- **Fix direction (pick one layer, recommend Bobbit-owned):** send `set_auto_retry + enabled:false` at every bridge start so Bobbit's policy (with its UI banner, cancel paths, + and S40 integration) is the only retry layer. Alternative: observe the SDK's + `auto_retry_start`/`auto_retry_end` events (currently unhandled) and suppress/cancel + Bobbit's timer — more moving parts. Also decide NEEDS-TRACE #5 (cap-park in + `deliverLiveSteer` doesn't cancel the timer) here. **Non-goals:** changing backoff + schedules. +- **Acceptance:** red→green unit (real SessionManager + mock bridge capturing + `set_auto_retry`); the existing auto-retry banner E2E green; a regression pin that an + SDK-style internal-retry sequence does not double-dispatch. +- **Risk:** behaviour change for users relying on SDK silent retry — Bobbit's banner+timer + replaces it 1:1. Rollback: drop the command. + +#### CS-R6 — Compaction dispatch gate (F6) +- **Serves:** symptom C (vanished messages, corrupted turns). Root cause **RC-B**. +- **Targeted code:** `src/server/agent/session-manager.ts` — `enqueuePrompt` direct branch + (`:1949-1968`), `drainQueue` (`:2240+`), `retryLastPrompt` (`:2734+`), the + implicit-unstick branch, the `compaction_start`/`compaction_end` handling + (`session.isCompacting` set `:2476/:2490`, read today only by `handler.ts:737`). +- **Diagnosis:** during compaction the SDK is not streaming and Bobbit's status is `idle`; + `enqueuePrompt`'s only gate is `status==='idle'` → prompts/steers/retries dispatch + straight into an in-flight compaction. SDK `compact()` then clobbers + `agent.state.messages`; **manual compact disconnects all agent event listeners for its + whole duration** — a turn started in that window streams into the void and skips `.jsonl` + persistence (permanent message loss). Reachable without exotic timing: queued prompt + + `/compact`; threshold auto-compaction at `agent_end` racing the same `drainQueue`; a + prompt typed during the visible compaction card. +- **Repro:** mock harness: start a (mock) compaction, `enqueuePrompt` → assert the prompt + is parked and dispatches only after `compaction_end` (today: direct dispatch). +- **Fix direction:** treat `session.isCompacting` as busy at every dispatch site (park in + `promptQueue`); drain on `compaction_end`. This single server-side gate also removes the + SDK's concurrent-compaction and manual-disconnect hazards from production reach. + **Non-goals:** touching SDK compaction internals; the composer disable is optional polish. +- **Acceptance:** red→green real-`SessionManager` test; `/compact`-then-prompt E2E ordering + test; NEEDS-TRACE #3 (restart mid-compaction spinner) re-checked under the gate. +- **Dependencies:** none, but **must land before or with CS-R7** (drain-on-idle widens this + hole if the gate is absent — the agent_end→drain path would dispatch into the + threshold auto-compaction window). + +#### CS-R7 — Drain the queue on every transition into idle (F7) +- **Serves:** symptom C (queued prompt stuck forever). +- **Targeted code:** `src/server/agent/session-manager.ts` — `drainQueue` call sites + (today: agent_end `:2456`, enqueue-while-idle `:1968`, recovery timer `:2208`, forceAbort + respawn `:6273`); the idle broadcasts that drain nothing: `session-setup.ts:1036`, + `restoreSession:3746`, `_respawnAgentInPlace:3041`, `assignRole:5083`; the + respawn-window enqueue bug `:1833-1834` (returns `{status:"queued"}` without enqueuing); + `ws/handler.ts:489-510` (the comment promising "they'll drain when the session becomes + idle" — currently false). +- **Diagnosis:** prompts accepted during `preparing`/`starting`, and queues restored at + boot with `wasStreaming:false`, sit as pills forever — no `starting→idle` transition + drains. Internal callers (team prompts, triggers) hitting the respawn window get a + success-shaped return with the message dropped. +- **Repro:** enqueue during `preparing` (mock setup delay); assert dispatch on the idle + transition (today: parked). Boot-restore with a persisted queue; assert drain. +- **Fix direction:** a single drain hook on every transition into `idle` (either inside + `broadcastStatus` when `newStatus==='idle'` or an epilogue at the four call sites), + gated on CS-R6's compaction check; make the respawn-window `enqueuePrompt` either await + the in-flight restore (CS-R2's promise) or return an explicit error. +- **Acceptance:** red→green tests for both repro legs; the handler comment becomes true; + abort-status-e2e (post-abort drain) stays green. +- **Dependencies:** CS-R6 (ordering constraint above); CS-R2 (restore promise, soft). + +#### CS-R8 — S13: prompt ack must survive auto-compaction (F8) +- **Serves:** symptom A (duplicate turn after a long pause). +- **Targeted code:** `src/server/agent/rpc-bridge.ts:445` (30s default), `:454-457` + (pending deleted on timeout — late ack orphaned at `:688-691`), `:466-476` (`prompt()`), + `:521-522` (compact's 120s budget); `session-manager.ts:2166-2209` + (`recoverPromptDispatch` re-dispatch). +- **Diagnosis:** the SDK acks a prompt only after in-prompt auto-compaction; >30s + compaction → timeout → recovery re-enqueues → the next `agent_end` (i.e. the successful + completion of the very prompt that "timed out") re-dispatches it — the same user message + runs twice. +- **Repro:** stub CLI whose prompt handler delays its success line past an injectable + lowered ack timeout; assert the prompt resolves once and exactly one `prompt` command is + ever written (today: two). +- **Fix direction:** keep the pending entry alive past the timeout so a late ack resolves + it and cancels recovery (preferred — preserves a liveness signal), or raise the + prompt-specific budget ≥120s; rely on the `process_exit` path (`:407-432`) for genuine + death. **Non-goals:** changing the busy-guard recovery machinery (correct for genuine + rejections). +- **Acceptance:** the stub-CLI test red→green; `rpc-bridge-lifecycle.test.ts` green; + enqueuePrompt/REST error mapping unchanged for genuine failures. + +#### CS-R9 — Crash-window durability for `messageQueue`/`wasStreaming` + force-kill bookkeeping (F9 + WP9 residual) +- **Serves:** symptom A/C after hard crashes. +- **Targeted code:** `src/server/agent/session-store.ts:513-539` + (`RECOVERY_CRITICAL_FIELDS` + 1s debounce); `session-manager.ts` force-kill branch + (~`:6136-6167` — never clears `streamingStartedAt` or persists `wasStreaming:false`), + agent_end clears (`:2445-2447`), heartbeat payload (`:879-885`), boot continuation + (`:3766-3776`). +- **Diagnosis:** hard-kill inside the debounce window → disk keeps `wasStreaming:true` + after a completed turn (phantom continuation on boot), or keeps a dequeued row + (duplicate turn: continuation + re-dispatch of a prompt already in the `.jsonl`), or + loses a freshly queued prompt entirely. Separately, the force-kill branch leaves + in-memory `streamingStartedAt` set (heartbeat ships a stale value) and persisted + `wasStreaming:true` (a later hard crash then fires a continuation for a turn the user + deliberately killed). +- **Repro:** store-level: mutate queue/wasStreaming, simulate kill before the debounce + flush, reload, assert the documented post-state. Force-kill: extend + `session-manager-force-abort-grace.test.ts` to assert the store update. +- **Fix direction:** promote `messageQueue` + `wasStreaming` to the synchronous-flush set + (both mutate at low frequency — turn boundaries and queue mutations); clear + `streamingStartedAt` + persist `wasStreaming:false` in the force-kill branch. The full + idempotency-key design (dedupe a re-dispatched prompt against the `.jsonl` tail) is a + **non-goal** here — document it as the residual at-least-once edge. +- **Acceptance:** red→green store tests; force-abort test extended; restart E2Es green. + +**Gate 1:** all Wave-1 reds green; a soak of the steer/abort/respawn E2E suite (incl. the +re-validated steer specs) 20/20 under `--repeat-each=20`; no statusVersion regression +observable across a respawn-with-inflight-dispatch; the exactly-once steer invariant test +green. + +--- + +### WAVE 2 — Client-side duplicates, losses, and the deferred RC1/RC5 edges + +#### CS-D1 — Blank-text attachment-only duplicate bubble (F10) +- **Targeted code:** `src/server/agent/rpc-bridge.ts:152-162` (`ATTACHMENT_ONLY_TEXT` + synthesis, applied at `session-manager.ts:1847`); `src/app/remote-agent.ts:908-955` + (prompt build) or `src/app/message-reducer.ts:264-300` (fallback candidate set). +- **Diagnosis:** attach an image, send with an empty composer (allowed: + `MessageEditor.ts:564`, `AgentInterface.ts:1423`). Optimistic text `""`; server + synthesizes "Attachments:"; echo text ≠ `""` → id-fallback, text-fallback and + `reconstructModelText` all miss; snapshot keys (`…|EMPTY|` vs `…|Attachments:`) never + match → **two user bubbles every time**, healing only on reload. Pre-existing, unlisted + in the seam table; it is RC1's no-correlation-id root cause via a third text-divergence + source. +- **Repro (real reducer, no harness change):** `optimistic-prompt {role:'user-with-attachments', + content:[{text:''}], attachments:[img]}` → live `message_end {role:'user', + content:[{text:'Attachments:'},{type:'image',…}]}` → assert 1 row (today 2). +- **Fix direction:** export the synthesis rule (pure constant/function) and mirror it + client-side when building the optimistic row's candidate text set — or carry the + optimistic id through the prompt frame (the principled RC1 fix; bigger). **Non-goal:** + changing the server-side synthesis. +- **Acceptance:** reducer pin red→green; an E2E once CS-H1's auto-image-echo lands + (blank-text + image → one bubble live and after reload). + +#### CS-D2 — Reducer hardening bundle (F11 a/b/c) +- **Targeted code:** `src/app/message-reducer.ts:90-94` (`plainTextEquivKey` — raw role), + `:301-312` (live-echo artifact consume — earliest-match), `:249-251` (synth:seq stamp); + `src/app/remote-agent.ts:1697-1706` (resume_gap — no reducer epoch reset). +- **Diagnosis:** (a) `user-with-attachments` optimistic vs `role:'user'` snapshot copy + (documents / blank-text; images are enriched back and safe) never key-match → duplicate + until the next snapshot. (b) In a tab with no optimistic row, the artifact consume + `findIndex` eats the EARLIEST id-less same-text user row — a *historical* row vanishes + transiently when the same text is sent again from another tab. (c) After a gateway + restart, `EventBuffer` restarts at seq 1; `resume_gap` re-baselines `_highestSeq` without + clearing reducer rows → a new `synth:seq:N` replaces an unrelated old `synth:seq:N` row + for the sub-second window before the snapshot. +- **Repro:** three reducer-unit cases (the agent reports give exact shapes); all + expressible today. +- **Fix direction:** (a) normalise `user-with-attachments`→`user` inside + `plainTextEquivKey`; (b) prefer the LAST matching artifact (`findLastIndex`) or scope to + the snapshot tail; (c) include a connection-epoch nonce in the synth id + (`synth:seq::`) or clear positive-`_order` synth ids on `resume_gap`. +- **Acceptance:** three pins red→green; the existing dedup + over-dedup suite stays green. + +#### CS-D3 — Prefixed error-recovery splice (S17 leg + F12 envelope leak) +- **Targeted code:** `src/server/agent/session-manager.ts:553-584` + (`spliceSkillExpansionsIntoEvent`, exact `p.modelText === body` match `:575`), `:1861-1867` + (envelope stashed keyed on the UNprefixed modelText), `:1941-1942` (the + `buildErrorRecoveryPrefix(...)` dispatch; structure `[SYSTEM:…]\n\n${userText}` — + commit 52f11884 only reworded the snippet); characterization pin + `tests/message-reducer-dedup.test.ts:75`. +- **Diagnosis:** errored prior turn + slash-skill prompt → dispatch is prefixed → the echo + body never exact-matches the envelope → the user sees `/foo` AND a "[SYSTEM: previous + turn failed…]\n\n" wall of text, persistent until reload; the orphaned + envelope stays in `pendingSkillExpansions` and can mis-splice a future byte-identical + message. +- **Repro:** real-`SessionManager` unit — stash an envelope, dispatch prefixed, feed a + synthetic prefixed echo through `emitSessionEvent`, assert the splice rewrites it (today: + miss). Expressible now, no WP0 dependency. +- **Fix direction:** stash the envelope keyed on the **prefixed** text at the `:1941` call + site (smallest), or make the splice match a prefix-stripped body. Then flip the + characterization test at `message-reducer-dedup.test.ts:75` to the fixed expectation. + **Non-goal:** the inert client-side `reconstructModelText` path (leave as harmless + defence; optionally annotate it as production-inert). +- **Acceptance:** unit pin red→green; characterization test flipped; an errored-turn + + slash-skill E2E asserting exactly one user row. + +#### CS-D4 — S26: steered images end-to-end (F13) +- **Targeted code:** `src/server/agent/rpc-bridge.ts:479-481` (+ interface `:109`) — add + `images?`; `session-manager.ts:2073/2089` (`_dispatchSteer` union + `rows.flatMap(r=>r.images??[])`), `:2098-2100` (rollback re-enqueue with images), + `:2136-2138` (reconcile with images), `:2257-2259` (drain batch union); + `tests/e2e/mock-agent-core.mjs` steer-images threading (CS-H1). +- **Diagnosis:** four drop points lose steered images silently; the SDK supports steer + images (`session.steer(message, images)` in rpc-mode). User symptom: "look at this + screenshot" steers the model demonstrably never saw. +- **Repro:** unit — steer RPC payload contains `images`; real-`SessionManager` — a steered + row with images reaches `rpcClient.steer(text, images)` and survives rollback/reconcile; + batch — two steered rows with distinct images dispatch the union. +- **Fix direction:** as targeted above. **Verify first** that the installed SDK's steer RPC + accepts images (env caveat in the header); if not, reject image steers with a visible + error instead of silently threading. **Non-goals:** document attachments on steers. +- **Acceptance:** unit pins red→green; E2E with CS-H1's image echo (steered image renders + AND reaches the mock's prompt/steer command args). +- **Dependencies:** CS-H1; coordinate with CS-R3/CS-R4 (same functions — land after, rebase). + +#### CS-D5 — Outbox hardening (F14) +- **Targeted code:** `src/app/session-manager.ts:930-941` (navigation-away caches only when + `connected`; else `remote.disconnect()` discards the instance + outbox); + `src/app/remote-agent.ts:1464-1477` (`_flushOutbox` — empties up front, skips non-OPEN, + swallows throws, no re-push), `:217` (`unsent:true` — zero UI readers), `:749` (auth_ok + flush wiring — untested); NEW `tests/e2e/ui/send-outbox.spec.ts` (spawned gateway). +- **Diagnosis:** (a) VPN flap → prompt queued (pill shown) → user switches session → + switches back → new `RemoteAgent`, outbox and pill gone — silent loss of typed intent + exactly when the outbox was supposed to protect it. (b) A socket closing mid-flush drops + the remaining frames (the guard's own failure recreates S2). (c) The promised + "waiting to send" affordance doesn't exist (pills look ordinary; `retry` frames have no + pill at all). (d) The auth_ok→flush wiring has no test — deleting the `_flushOutbox()` + call passes the suite. +- **Repro:** (a) bundled-agent spec: non-OPEN ws, `prompt('x')`, simulate the + session-switch path, assert the outbox survives (today: discarded). (b) flush with a + socket that closes after frame 1 of 3 → assert frames 2–3 re-queued. (d) spawned-gateway + E2E: kill the socket, prompt, reconnect, assert exactly one server copy. +- **Fix direction:** cache the outgoing agent even when reconnecting (it self-reconnects); + re-push unsent remainder in `_flushOutbox`; render `unsent:true` distinctly in the pill + strip (plan §2.5.2's chosen UX); decide whether `retry` belongs in the pill channel. + **Non-goals:** cross-reload outbox persistence (accepted loss, no false "sent" state); + ack-based delivery (S13's territory). +- **Acceptance:** the three reds green; the E2E pins auth_ok→flush; exactly-once on flush + (no duplicate when the server also queued via another tab — document the multi-tab + duplicate-intent caveat). + +#### CS-D6 — WP6: respawn continuation suppression + grant-replay gating (F15 = S16+S38) +- **Targeted code:** `src/server/agent/session-manager.ts:3766-3776` (continuation gate), + `:3032-3035` (`_respawnAgentInPlace` finally — delete the transient flag), `restartAgent` + (`:3052+` mutatePs) and the **role-switch** caller of `_restartSessionWithUpdatedRole` + (`:2957` — the three grant modes `:2850/:2856/:2870` must NOT set the flag); + `src/app/remote-agent.ts:1719-1721` (remove the heartbeat-branch `_maybeReplayGrant`), + `:1232-1241` (replay timer — keep). +- **Diagnosis:** `restoreSession` fires "[SYSTEM: …continue where you left off…]" whenever + persisted `wasStreaming` — true for every deliberate respawn (grant pauses the turn + before agent_end). Grant path: server continuation + client 200ms replay = two driving + prompts. Plan review-option (i) is adopted: **keep** the continuation for the GRANT path + (it is the robust driver; `switch_session` does not auto-continue), suppress only for + `restartAgent` and role-switch; drop only the redundant heartbeat-branch client replay. +- **Repro:** real-`SessionManager`: persisted `wasStreaming:true` + `restartAgent` → assert + zero continuation prompts (today 1); genuine boot restore → still 1. Client: heartbeat + idle with a pending grant replay → no send; transition idle → one send (bundled-agent + spec — also closes ledger gap #10). +- **Fix direction:** `_suppressContinuationPrompt` transient on `ps` per the WP6 spec + (suppressed branch still clears `wasStreaming`); negative pin that + `recoverSandboxSessions` does NOT set it. **Non-goals:** the 200ms timer redesign; + suppressing for sandbox recovery. +- **Acceptance:** WP6's acceptance list verbatim (03-remediation §WP6): no continuation + bubble on restart/role-switch; grant still driven; boot/Docker restores keep the prompt; + `restart-preserves-streaming-frame`, `sandbox-recovery-*`, grant E2Es green. +- **Dependencies:** **after CS-R2** (same functions; CS-R2's fencing changes + `_respawnAgentInPlace`/`restoreSession` — rebase this on top). + +#### CS-D7 — Snapshot-authoritative auto-retry banner (F16, WP4 step 4) +- **Targeted code:** `src/app/remote-agent.ts` `case "messages"` snapshot branch + (~`:1546-1638`, near the streaming clear `:1574/:1588`) — add + `this._state.autoRetryPending = null;` (NOT in `case "state"`); + `tests/e2e/ui/auto-retry-banner.spec.ts` (extend). +- **Diagnosis:** the only banner clears are live events (`:2374,2386,2401`); a cancel + emitted with zero clients (`session-manager.ts:2663-2679` skip) or evicted from the ring + leaves a reconnecting tab showing "Retrying in Xs…" forever (reachable via + forceAbort-idle on a detached session, e.g. the team-abort route). +- **Repro:** inject `auto_retry_pending` via the window hook; drive a `messages` snapshot + WITHOUT a cancel; assert the banner is gone (today: stays). +- **Fix direction & acceptance:** exactly WP4 step 4 + its spec extension; the now-seq'd + pending replays after the snapshot if a retry is genuinely pending, so no flicker. + +#### CS-D8 — S25: perm-frame resume hole (F17) +- **Targeted code:** `src/server/agent/event-buffer.ts:41-43` (`pushFrame` retain a benign + `{type:'noop'}` seq-holder); `tests/event-buffer.test.ts:209-233` (BOTH tests pin the + pre-fix contract — update to size 3 / `[1,2,3]`); NEW `tests/perm-frame-resume-hole.test.ts`; + `perm-frame-late-joiner-seq-gap.test.ts` must stay green (still one `pushFrame` callsite + `session-manager.ts:2909`; on-attach still via `getPendingToolPermission`, + `handler.ts:436-444`). +- **Diagnosis:** a perm consumes seq N, never retained; resolved-while-disconnected + (DENY `:2943-2949` / 5-min timeout `:2913-2916` / implicit deny `:2896-2900`) → resume + `since(fromSeq)` replays around the hole → the client gap-buffers everything behind N. + Self-heals only after 500 buffered events (S9 overflow). Do NOT retain the full payload: + the resume loop routes ring entries into `handleAgentEvent`, which has no + `tool_permission_needed` case — the card renders only from the top-level frame; a full + payload would spuriously emit and never render. +- **Repro:** `buf.push(1..3); buf.pushFrame(); buf.push(5,6)`; feed `since(3)` into the + FakeClient sequencer from the late-joiner test → today seqs 5–6 strand. +- **Fix direction & acceptance:** WP5 steps 2–4 + both test edits; eviction-window test + re-verified (pushFrame now counts toward the 1000 cap); the retained holder never paints + a card or a `_state.messages` row. + +#### CS-D9 — S4: double-Enter sends twice (F18) +- **Targeted code:** `src/ui/components/MessageEditor.ts` `handleSend` (~`:654-679` — no + clear, no re-entrancy flag); `src/ui/components/AgentInterface.ts:1467-1491` (clear after + awaits), `:2113` (un-awaited onSend); fixture `tests/message-editor-send.html` clears + synchronously (the inverse of production — replace per the CS-H2 real-bundle pattern). +- **Diagnosis:** Enter #1 fires `onSend` without clearing; `sendMessage` suspends on the + IndexedDB `providerKeys.get` await; Enter #2 (auto-repeat / fast double) re-reads the + same value → two identical prompts, two turns (each optimistic id unique, so no + collapse). +- **Repro:** real-component bundle: macrotask-resolving storage stub, two `keydown{Enter}` + in a tick → assert one send (today two). +- **Fix direction:** clear `this.value`/`this.attachments` synchronously in `handleSend` + BEFORE invoking `onSend`, restoring on the abort/early-return paths (note: the size guard + from S31 is already the first statement — keep it first), OR a synchronous `_sending` + flag. **Non-goal:** touching the await ordering in `sendMessage`. +- **Acceptance:** red→green on the real bundle; size-guard + IME + draft specs stay green. + +**Gate 2:** all Wave-2 reds green; the duplicate-bubble class has zero known reproducible +triggers (blank-text, role-keying, prefixed-skill, double-Enter all pinned); outbox +survives navigation and mid-flush failure; banner reconciles on snapshot; perm resume +hole-free. + +--- + +### WAVE 3 — Payload, hot-path, perf (the unshipped PR-D) + +These four tasks are specified in detail in [03-remediation-plan.md](03-remediation-plan.md) +(WP7, WP8, WP10) — the specs there remain accurate (anchors drifted; current ones below). + +#### CS-P1 — WP7 payload slimming (S19+S20) +- `session-manager.ts:1782-1789` (`broadcastQueue` → `toBroadcastArray()`/`toPersistArray()` + via a new pure `queue-projection.ts`; persist keeps `images[].data` for the restart + re-dispatch invariant); `remote-agent.ts:950-955` (+ pure `stripAttachmentsForWire` in + `attachment-utils.ts` — documents lose `content`/`preview` on the wire, images' bytes ride + `images[].data` once); defensive ingestion strip sparing `msg.images`. Acceptance: WP7's + list; key risk — UI pill strip must render metadata-only rows; verify document chips + render from metadata after reload. Independent. + +#### CS-P2 — WP8 hot-path bundle (S32, S34, S35, S37) +- S32: `remote-agent.ts:2125-2170` — `textHasProposalTag` indexOf precheck + cached + regexes; byte-identical matched behaviour. S34: `AgentInterface.ts:672-684/723-733` — + rAF-coalesce ONLY `_updateAndPin`'s jump-button call (the shared wrapper has ~11 callers + incl. the scroll handler `:1198/:1207` — keep those synchronous); cancel in + `disconnectedCallback`; the scroll-spec suite is the blast radius. S35: + `attachment-utils.ts:88-95` — async `arrayBufferToBase64` yielding every ~1MB. S37: + `aigw-manager.ts:381-389` — `"${BOBBIT_SESSION_ID}"` template; **update the two + literal-pinning tests** (`aigw-headers.test.ts`, `aigw-header-resolver.test.ts`) to the + throw-equivalence invariant — they currently pin the defect. Commit each seam separately. + +#### CS-P3 — Standalone hygiene bundle (S33, S36, S43, F19, S15-comment) +- S33: `remote-agent.ts:2462-2468` trailing-edge flush (`_truncatedFlushTimer`; clear on + `message_end`/reset; the no-stale-post-finalize assertion is the load-bearing red test). +- S36: extract `guardedBroadcast` (owns the WeakSets, diag label as param), behaviour + pinned unchanged for `session-manager.broadcast`, then point `handler.ts:154-196` at it. +- S43: `rpc-bridge.ts:312-316` — `removeAllListeners()` before kill (+ drop the dead + `.toString()`); spy-pin in `rpc-bridge-lifecycle.test.ts`. +- F19: filter `parsed.type === "response"` from the event fan-out at + `rpc-bridge.ts:688-698` (late post-timeout responses must not become seq'd session + events); client guard at `remote-agent.ts:2706-2718` optional. +- S15: rewrite the stale `restoring` comment at `session-manager.ts:3694-3704` (or simplify + the guard) — the "replays every message" claim is false on pi ≥0.67. + +#### CS-P4 — S29 reconnect hydration coalescing +- `src/app/session-manager.ts:1384-1390` + `:2730-2740`: add in-flight dedup for + bg-processes and annotation-store refreshes (mirror the git-status guard `:2855-2857`); + optionally debounce `onReconnect` bursts. Low risk; perf-only. + +**Gate 3:** PR-D acceptance (03-plan Gate 4) holds: `queue_update` frames carry no base64; +no document bytes on the wire; hot-path pins green; all scroll/jump specs green within the +existing `settleFrames` budget. + +--- + +### WAVE 4 — Test-debt mop-up + +#### CS-T1 — WP11 leftovers (two flaky specs) +- pre-compaction-history: implement the test-only `__pinAgentSessionFile` hook + (`session-manager.ts:4531-4541` is the production overwrite site) + the forced-`getState` + prerequisite assertion, per WP11 steps 4–5 (UX decision #29 already chose the hook). +- dynamic-chat-tabs: sample the collapse hash from the field production reads — + `currentFilenameTab.state.contentHash` via `page.evaluate` poll (decision at + `src/app/preview-panel.ts:216-218`) — not the GET-mount response. Acceptance: 20/20 under + `--repeat-each=20`; no quarantine/retries/timeout bumps. + +#### CS-T2 — repro-h3 case (A) deterministic gate +- Author the deterministic snapshot↔in-flight gate (reuse the shipped `USER_ECHO_DELAY` + knob; escalate to a `SNAPSHOT_GATE` trigger only if flaky) but **leave case A + `test.fixme`** — the quarantine comment binds it to the separate snapshot-live-race goal + (docs/design/snapshot-live-race-fix.md), which flips it in its own PR. This task only + removes the harness excuse. + +**Gate 4 (done):** symptoms A–D have no known reproducible trigger; every invariant in §5's +unpinned list is either pinned or explicitly accepted with a documented owner; suites green +with zero flakes. + +--- + +## 7. Dependency / merge map + +``` +CS-H1 ──┬─→ CS-R3 ──→ CS-R4 + ├─→ CS-D4 (also after CS-R3/R4 — same functions) + └─→ CS-D1 (E2E leg only) +CS-H2 (independent) +CS-R1 (independent, land first — trivial + total-outage fix) +CS-R2 ──→ CS-D6 (same functions; rebase) +CS-R6 ──→ CS-R7 (ordering constraint: gate before drain-on-idle) +CS-R5, CS-R8, CS-R9, CS-D1..D3, CS-D5, CS-D7..D9, CS-P1..P4, CS-T1..T2: parallel +File-conflict hotspots: session-manager.ts (CS-R1..R9, CS-D3, CS-D6, CS-P3 — + serialize merges in task order), remote-agent.ts (CS-D2/D5/D7, CS-P2/P3 — distinct + functions, merge cleanly), rpc-bridge.ts (CS-R1, CS-R8, CS-D4, CS-P3). +``` + +Recommended landing order: **CS-R1 → CS-H1‖CS-H2 → CS-R2 → CS-R5‖CS-R6 → CS-R3 → CS-R7 → +CS-R4 → CS-R8‖CS-R9 → Wave 2 (D1–D9, D6 after R2) → Wave 3 → Wave 4.** diff --git a/docs/design/sidebar-goal-nesting-audit.md b/docs/design/sidebar-goal-nesting-audit.md new file mode 100644 index 000000000..4e0d32994 --- /dev/null +++ b/docs/design/sidebar-goal-nesting-audit.md @@ -0,0 +1,433 @@ +# Sidebar & Goal-Nesting Audit — Findings + Ordered Task Backlog + +> **Date:** 2026-06-10 · **Baseline:** master @ `6ec8c8f9` · Companion to the comms-stack +> audit ([comms-stack/04-current-state-and-backlog.md](comms-stack/04-current-state-and-backlog.md)) +> but a distinct surface: the left navigation pane (projects/goals/sessions) and the +> goal/sub-goal nesting model. All file:line anchors verified against the current tree. +> +> **Scope:** race conditions and redraw correctness in the sidebar's data flow; whether +> goal/subgoal nesting is modelled and rendered optimally; reuse/composability defects. +> Each backlog task (§4) is self-contained for hand-off to an implementer agent. + +--- + +## 1. Executive summary + +The sidebar is fed by a **5s REST poll (`refreshSessions`) + ~25 scattered ad-hoc +`refreshSessions()` call sites + WS-pushed point patches**, with **no in-flight coalescing +and no monotonic-generation acceptance guard** — every "X briefly reverts/reappears" +symptom traces to this one hole (D1–D5). The nesting UI maintains **two parallel render +paths** for one parent/child relation: Path A (spawned children rendered inside a +team-lead's expanded block, `render-helpers.ts::renderTeamGroup`) and Path B (the pure +forest, `sidebar-nesting.ts::buildNestedGoalForest`), kept disjoint by a third function +(`computeSpawnedClaim`) whose doc-comment literally promises to "mirror render-helpers +exactly". That mirror discipline has already failed once in the wild (commit `62755c3d`, +"don't render spawned sub-goals at top level") and still contains live disagreement windows +— invisible goals (D10), duplicate rows (D9), three different badge semantics (D6). Mobile +is a third, drifted near-copy with **no forest path at all** (D7). The same goal tree is +independently re-derived in at least **six places** (S2). Test coverage is dominated by +**hand-copied mirror fixtures that have already drifted from source**, and the core forest +builder has **zero direct tests** (S5). + +The structural moves that remove these bug classes by construction: (1) a single sync +engine with epoch discipline, (2) a single memoized goal-graph module, (3) one recursive +tree renderer shared by desktop/mobile/archived — deleting the claim/mirror machinery. + +What is already sound (verified, no action): the 5cfc0306 "Loading… blanking" fix is a real +root-cause fix; rAF render coalescing is respected by the sidebar; expansion/scroll state +survives re-renders; gate-status/PR caches already implement the correct in-flight/token +patterns (the model for T2); goal-dashboard descendant fetches are guarded; unread/read +mark race is defended; cross-tab project reorder is sound. + +--- + +## 2. Confirmed defects + +### D1 — `refreshSessions` has no in-flight dedup and no monotonic-generation guard (root-cause class) +- **Anchors:** `src/app/api.ts:299-498` (no reentry guard; unconditional + `state.gatewaySessions = newSessions` at `:367`; `state.sessionsGeneration = + sessionsData.generation` at `:388-390`; goals merge `:397-443`). Contrast with the guards + the codebase already built elsewhere: `_prRefreshInFlight` (`api.ts:1119-1122`), + gate-status tokens (`api.ts:1084-1115`). +- **Trigger:** any two overlapping invocations — trivially reachable: the 5s poll + (`api.ts:273-280`), `visibilitychange` (`main.ts:986-992`), hashchange route handlers + (`main.ts:293-543`), and the WS `goal_state_changed`/`goal_child_spawned`/`cost_changed`/ + `goal_spec_changed` handlers (`remote-agent.ts:1778-1800`) all call it un-coordinated. A + `goal_child_spawned` burst during plan execution fires N concurrent fetches. +- **Symptom:** last-**arriving** (not last-issued) response wins. A stale snapshot can + (a) regress `sessionsGeneration`, (b) revert a WS-applied status/title for ≤5s + (flickering sprite/unread pulse, spurious notification beep via `_prevSessionStatus`, + `api.ts:351-365`), (c) resurrect removed rows. Self-heals on the next poll tick — unless + the tab is hidden (polling is visibility-gated), in which case stale state persists until + refocus. +- **Severity:** medium (transient wrong UI, spurious beeps; high reachability). +- **Why tests miss it:** `tests/sidebar-loading-flash.test.ts` pins only the initial-load + flag; nothing pins ordering across overlapping fetches. +- **Repro:** stub `gatewayFetch` with two controllable promises; call `refreshSessions()` + twice; resolve the second (newer generation) first, then the first; assert state keeps + the newer generation. Fails today. +- **Fix:** coalesce (return the in-flight promise) + accept a response only if + `generation >= state.sessionsGeneration` (sessions and goals independently), mirroring + the gate-status token pattern. **Non-goal:** moving to WS-pushed lists. + +### D2 — WS point-updates race in-flight poll snapshots (ghost session after `session_removed`) +- **Anchors:** `src/app/session-manager.ts:1396-1412` (`onSessionRemoved` filters the list + immediately), `remote-agent.ts:1951-1960`; resurrect path = D1's `api.ts:367`. Same class + for `updateLocalSessionStatus`/`updateLocalSessionTitle` (`api.ts:251-271`). +- **Trigger:** poll issued at T0; session archived at T1 (WS removes the row); poll + response (snapshot from T0) applies at T2 → the row reappears for ≤5s; clicking it + 404s/reconnects to a dead session. +- **Severity:** medium. **Fix:** subsumed by D1's epoch guard (removal bumps the server + generation, so the stale snapshot is dropped). + +### D3 — Goal archived by an agent/another tab vanishes entirely when "See Archived" is on +- **Anchors:** goals merge rule `api.ts:403-421` — the live payload is authoritative; a + goal newly archived **server-side** is absent from the live payload and not in + `preservedArchived` (the local copy isn't flagged archived) → dropped from `state.goals` + outright. The archived list refreshes only on toggle/search/load-more/initial + (`sidebar.ts:1090-1101`, `api.ts:493-497`); the WS `goal_state_changed` handler only + calls `refreshSessions()` (`remote-agent.ts:1778-1789`), never the archived page. +- **Trigger:** showArchived ON; a team-lead calls `goal_archive_child` (or a second browser + archives the goal). Sidebar: the goal disappears from the live section and does **not** + appear in Archived until the user toggles See-Archived off/on. +- **Severity:** medium-high perceived ("my goal is gone"); no data loss. +- **Fix:** when a previously-live goal id disappears from the live payload while + `state.showArchived && archivedGoalsLoaded()`, trigger a debounced first-page archived + re-fetch; or have the server include `archivedSince` deltas. **Non-goal:** keeping + archived pages fully live. + +### D4 — Duplicate concurrent archived-goals fetches; inconsistent loaded-flag discipline +- **Anchors:** `api.ts:871-879` — `_archivedGoalsLoaded = true` is set **after** the await + (`fetchArchivedSessions` sets its flag **before**, `api.ts:529-537`). + `_ensureArchivedForSearch` (`sidebar.ts:1090-1101`) checks the flag per keystroke → every + keystroke during the in-flight window issues another fetch; each first-page response + **replaces** the archived slice (`api.ts:887-891`), racing a concurrent "Load more" + append (`api.ts:882-886`). +- **Severity:** low-medium (wasted fetches; pagination cursor/content mismatch under fast + typing + load-more). +- **Fix:** set the flag pre-await + in-flight promise dedup, same shape as sessions. + +### D5 — Optimistic archive can flash back to live +- **Anchors:** `eagerMarkArchived` (`api.ts:1425-1434`) then `await refreshSessions()` + (`api.ts:1364, :1391`). An unrelated poll issued pre-DELETE that lands after the awaited + refresh reverts the goal to live for one tick (D1 class). +- **Severity:** low (cosmetic flash). Fixed by D1's epoch guard. + +### D6 — Descendant-count badge has three different meanings (and is often missing) +- **Anchors:** Path B badge = transitive descendant count over the **claim-filtered** + forest input (`sidebar.ts:1377-1384` excludes claimed children before + `buildNestedGoalForest`; count from `sidebar-nesting.ts:133-159`). Path A badge = + **direct non-archived children only** (`render-helpers.ts:862`). Mobile = no badge at all + (`render.ts:549` passes no opts). +- **Symptom:** a parent whose children are all claimed by its team-lead shows **no badge** + (its forest node has zero children) even though docs/nested-goals.md promises one; a + spawned child's badge undercounts grandchildren. +- **Severity:** cosmetic-medium, always visible to subgoal users. +- **Fix:** compute the badge from the **full unfiltered goal graph** (one shared + `descendants(goalId)`), independent of which render path draws the row. (→ T4/T5.) + +### D7 — Mobile has no forest path; divergent near-copy of desktop bucketing +- **Anchors:** `render.ts:461-549`: top-level goals render flat via `renderGoalGroup(goal)` + — no `buildNestedGoalForest`, no indentation, no truncation row, no + archived-into-live-forest folding, no title-suffix pass. Additionally mobile drops goals + without `projectId` (`render.ts:485`) while desktop resolves via the parent-chain walk + (`sidebar.ts:1599-1611`) — a spawned child that inherits its project from its parent + renders on desktop but is **silently missing on mobile**. +- **Trigger:** a child goal not claimed by Path A (parent's lead terminated/purged, or + unstamped) renders as an un-indented top-level row on mobile; a child with inherited + projectId is missing on mobile entirely. +- **Severity:** medium. Commit `62755c3d` fixed only the duplicate-row symptom of this + divergence, not the divergence. (→ T6.) + +### D8 — Role-picker popover renders once per project section +- **Anchors:** `${renderRolePickerDropdown()}` inside the per-project Sessions header loop — + `sidebar.ts:1433` (desktop) and `render.ts:577` (mobile). With N projects and + `state.rolePickerOpen`, N identical `position:fixed` popovers stack. The capture-phase + keyboard handler and `_focusCwdIfNeeded` target + `document.querySelector(".cwd-combobox input")` — the **first** copy in DOM order + (`sidebar.ts:637-639, :786-789`), which can sit *under* the painted copy: focus/caret + land on an occluded element. +- **Severity:** low (state-driven copies stay in sync; caret/focus artifacts + N× DOM and + listeners). **Fix:** render the popover exactly once at the sidebar root (it is + `position:fixed`-anchored anyway). (→ T8.) + +### D9 — Unstamped children render under multiple leads (duplicate rows) +- **Anchors:** `selectSpawnedChildren`'s unstamped branch matches whenever + `leadId === parentLeadId` (`sidebar-spawned-children.ts:63-68`), and both call sites pass + `leadId` as its own `parentLeadId`: the live lead (`render-helpers.ts:1489-1495`) **and + each archived lead** (`render-helpers.ts:1620-1621`, rendered via `renderLeadWithMembers` + `:1627-1650` alongside the live branch). +- **Trigger:** showArchived ON; the goal has a live team-lead **and** ≥1 archived team-lead + (team restarted); a child with `parentGoalId` set but no `spawnedBySessionId` (created + via `createGoal(..., parentGoalId)` from the proposal modal's subgoal prefill, or legacy + data). The child renders under the live lead AND under each archived lead. +- **Severity:** medium — the exact bug class the dedupe E2E was written for, but + `tests/e2e/ui/sidebar-spawned-children-dedupe.spec.ts` only covers stamped same-title + siblings, and `tests/sidebar-spawned-children.test.ts` tests one call at a time (no + global at-most-once assertion). The browser fixture + (`tests/sidebar-spawned-children.html`) is a drifted mirror without the `parentLeadId` + parameter. +- **Fix (tactical):** unstamped children attach to exactly one lead (live if present, else + most-recent archived). **Strategic:** T5 removes the per-lead enumeration entirely. + +### D10 — Claim/Path-A lead-choice disagreement can hide a goal completely +- **Anchors:** `computeSpawnedClaim` picks `liveLeadCandidates[0]` in **input order**; its + own comment concedes the order may disagree with Path A's createdAt sort and asserts the + consequence is harmless (`sidebar-spawned-children.ts:176-188`). It is not: if a stamped + child's `spawnedBySessionId` is lead₂ but Path A picks lead₁ (`render-helpers.ts:1347` + sort + `:1446` find), the claim excludes the child from the forest **and** Path A never + renders it → the goal is invisible in the sidebar. +- **Trigger:** two live `team-lead` sessions for one goal (respawn/recovery) with server + list order ≠ createdAt order. +- **Severity:** medium impact, low-medium reachability. Fixed by construction under T5. + +### D11 — Archived-forest exclusion filter is global, not tree-scoped +- **Anchors:** `render-helpers.ts:887-894` — an archived goal is excluded from the bottom + Archived forest if its `spawnedBySessionId` matches **any** archived team-lead in + `state.archivedSessions`. If that lead's parent goal is purged/not rendered in this + section, the goal renders **nowhere**. Placement is also pagination-dependent: before the + lead's archived page loads the goal shows in the forest; after "Load more" it silently + relocates under the lead. +- **Severity:** low-medium (invisible/jumping archived goals). +- **Fix:** exclude only when the spawning lead will actually be rendered in this project's + archived section. (→ T9, or by construction under T5.) + +### D12 — Detached parent-cycles vanish silently +- **Anchors:** `sidebar-nesting.ts:186-195` — root detection is "no parent or parent + absent"; the `visited` cycle guard (`:135-150`) fires only for cycles reachable from a + root. A detached A↔B cycle yields no root → neither renders, no warning. +- **Severity:** very-low reachability; silent-data-hiding class. **Fix:** after root + collection, emit unvisited goals as degraded roots + `console.warn`. (→ T9.) + +### D13 — Minor confirmed items +- **Auto-expand misses team goals:** `api.ts:429` checks only `s.goalId === g.id`, not + `teamGoalId` — newly discovered team goals don't auto-expand; the dedupe E2E's helper + comment ("expansion timing-sensitive") works around this symptom + (`sidebar-spawned-children-dedupe.spec.ts:26-31`). +- **Render-time fetch staleness:** the on-demand team-agents fetch inside `renderGoalGroup` + (`render-helpers.ts:1375-1403`) can push into `state.archivedSessions` *after* + `clearArchivedSessionsState()` emptied it (toggle-off race) — no staleness token. +- **Unbounded maps:** `_prevSessionStatus` (api.ts), `gateStatusCache`/`prStatusCache` + never pruned for deleted goals; `expandedGoals` localStorage accretes dead ids (only + archived ones are pruned, `state.ts:650-655`). +- **Stale comment claiming a nonexistent optimization:** `computeSpawnedClaim` says "Avoid + an O(parents × sessions) scan: bucket sessions by parent goal id once" + (`sidebar-spawned-children.ts:160-163`) — the code below does exactly the + O(parents × sessions) filter-per-parent it claims to avoid, per project per render frame + (again on mobile). + +--- + +## 3. Structural findings (the defect factories) + +### S1 — Two render paths + a hand-maintained mirror "claim" function for one relation +Path A: `renderGoalGroup → renderTeamGroup → selectSpawnedChildren` +(`render-helpers.ts:1450-1545` + archived-leads branch `:1594-1652`). Path B: +`buildNestedGoalForest → renderNestedNode` (`sidebar.ts:1310-1400`). Disjointness is +enforced by `computeSpawnedClaim`, documented as a mirror of render-helpers. This produced +`62755c3d`, D6, D9, D10, D11 — and two different expansion semantics for the same child +(Path A children hide under `isTeamLeadExpanded(lead)`, Path B children under +`expandedGoals.has(parent)`; the same goal switches regimes when its lead's session is +purged). +**Refactor:** one pure, memoized per-project tree build that assigns every goal exactly one +display anchor: `{ parent: goalId | null, slot: "forest" | { teamLeadId } }`. One recursive +renderer consumes it for live, archived, desktop, mobile, collapsed. `computeSpawnedClaim` +is deleted; the double-render and invisible-goal classes become unrepresentable. +**Migration risk:** medium — placement edge cases (unstamped, archived-lead, pagination); +must land after the pinning tests (T1) and behind the existing E2E suite. + +### S2 — Six independent goal-tree derivations +`buildNestedGoalForest` (sidebar-nesting.ts); `selectSpawnedChildren`/`computeSpawnedClaim` +(sidebar-spawned-children.ts); `countDescendants` (goal-descendants-count.ts) + +`collectGoalIdsFor` (`api.ts:1399-1417`); the `projectIdForGoal` chain-walk +(`sidebar.ts:1599-1611`); `buildChildSummaries` (`goal-dashboard-children-tab.ts:35-57`); +the plan-tab direct-children filter (`goal-dashboard-plan-tab.ts:98-113`). Each +re-implements parent indexing, archived rules, and cycle guards with subtle differences. +**Refactor:** `src/app/goal-graph.ts` — one memoized index (`byId`, `childrenByParent`, +`descendants()`, `directChildren()`, `depthOf()`, `projectOf()` with chain fallback, +cycle-safe), invalidated on `goalsGeneration` change. Removes D7's mobile projectId drop +and the badge inconsistency by construction. + +### S3 — Data-flow: ad-hoc pull, no epoch discipline +`refreshSessions` is the de-facto store update but is called from ~25 sites with no +coordination (D1–D5). Gate-status and PR caches already demonstrate the correct patterns in +this codebase — sessions/goals, the most important lists, are the only ones without them. +**Refactor:** a `requestSync()` façade: coalesce concurrent callers onto one in-flight +promise, debounce WS-event bursts (~100ms), apply responses only when the generation is +monotonic. All call sites switch mechanically. + +### S4 — Desktop/mobile/collapsed are three drifting copies +Project bucketing + section rendering exists 3×: `renderSidebar`/`renderProjectContent` +(`sidebar.ts:1458-1724`), `renderMobileLanding` (`render.ts:461-622`), +`renderCollapsedSidebar` (`sidebar.ts:1730-1887`). Mobile has drifted twice already +(`62755c3d`; D7). Shared pieces (`renderGoalGroup`, `renderProjectArchivedSection`, +`renderStaffSidebarSection`) prove the parameterized pattern works. +**Refactor:** extract `bucketSidebarDataByProject()` (pure, using S2's `projectOf`) + a +variant-parameterized project-section renderer (`"desktop" | "mobile"`). Collapsed stays +bespoke but consumes the same buckets. + +### S5 — Test fidelity: mirrors with confirmed drift; the core builder untested +`tests/sidebar-spawned-children.html` reimplements `selectSpawnedChildren` **without** the +`parentLeadId` fallback and the archived-first sort — it pins a stale version. +`tests/sidebar-hierarchy.html` and `tests/sidebar-goal-rendering.html` self-describe as +mirrors. `buildNestedGoalForest` has **no direct tests at all**. The real renderer is +exercised only by spawned-gateway E2Es. +**Refactor:** the pure modules are already DOM-free with test overrides +(`_setSubgoalsEnabledForTesting`) — pin them with node:test against the **real source**; +retire mirror fixtures or rebuild them on bundled real modules. + +### S6 — Unkeyed list rendering +No `repeat()` anywhere in the sidebar (only MessageList/tab code uses it). All rows are +`.map()` → positional DOM reuse: when a row is inserted/re-sorted (new session, archived +bucket boundary moves), DOM nodes rebind to different entities — keyboard focus and running +CSS animations (`bobbit-unread-pulse`, spinners) silently transfer to a different +session/goal. Expansion state is module-state (survives correctly), so this is the +remaining redraw-correctness gap. +**Refactor:** `repeat(items, x => x.id, render)` for project sections, goal nodes, session +rows. Low risk, mechanical. + +--- + +## 4. Ordered task backlog (hand-off ready) + +Conventions: master stays green (`npm run check`, `npm run test:unit`, `npm run test:e2e`); +test-first (red on master where expressible); no flaky tests; anchors verified at +`6ec8c8f9` — re-verify before editing. + +### T1 — Pin the pure tree builders with real-source tests *(no deps; prerequisite for T5/T6)* +- **Files:** NEW `tests/sidebar-nesting.test.ts`; extend + `tests/sidebar-spawned-children.test.ts`; delete-or-rebuild + `tests/sidebar-spawned-children.html` (drifted mirror: missing `parentLeadId` + the + archived-first sort). +- **Diagnosis:** `buildNestedGoalForest` (the core tree builder) has zero direct tests; the + existing browser fixture pins a stale mirror (S5); nothing asserts global at-most-once + emission of a child across the live-lead + archived-lead branches (D9's gap). +- **Fix:** node:test suites importing the real TS (`_setSubgoalsEnabledForTesting(true)`), + covering: orphan promotion, depth cap + `truncatedChildrenCount`, archived-first sort, + title suffixes (root + nested), cycle stubs, flag-off flattening; plus a cross-branch + test asserting **global at-most-once emission** of an unstamped child across live-lead + + archived-lead `selectSpawnedChildren` calls — this test documents D9 red. +- **Acceptance:** tests run in the unit phase; the D9 case is either fixed inline (tactical + single-anchor rule) or committed as a tracked known-failure pin for T5. + +### T2 — `refreshSessions` concurrency hardening *(no deps)* +- **Files:** `src/app/api.ts:299-498`; NEW `tests/refresh-sessions-race.test.ts` (extract + the apply/merge step into a pure helper, the `session-load-state.ts` pattern). +- **Diagnosis:** D1/D2/D5 — no in-flight coalescing, no monotonic-generation acceptance; + last-arriving stale snapshot clobbers WS-fresh state (ghost rows, status flicker, + spurious beeps). +- **Fix:** module-level in-flight promise returned to concurrent callers; drop session/goal + payloads whose `generation` is below current (independently for sessions and goals); keep + the archived-preserve merge rule. +- **Acceptance:** interleaved-resolution unit test (old response after new → state keeps + new) red→green; `tests/sidebar-loading-flash.test.ts` stays green; no E2E regressions. + +### T3 — Archived freshness + fetch dedup *(after T2)* +- **Files:** `src/app/api.ts:403-421, :871-912`; `src/app/sidebar.ts:1090-1101`. +- **Diagnosis:** D3 (goal archived elsewhere vanishes while showArchived is on), D4 + (loaded-flag set post-await; replace-vs-append race under typing + load-more). +- **Fix:** set `_archivedGoalsLoaded` pre-await + in-flight promise; in the goals merge, + detect ids that disappeared from the live payload while + `showArchived && archivedGoalsLoaded()` and debounce a first-page archived re-fetch. +- **Acceptance:** unit test for the disappearance→refetch trigger; rapid-typing search + issues exactly one archived-goals fetch; NEW E2E: archive a goal via API while + showArchived is on → it appears in the Archived section without toggling. + +### T4 — Single `goal-graph.ts` module *(no hard deps; before T5)* +- **Files:** NEW `src/app/goal-graph.ts`; migrate `goal-descendants-count.ts`, + `api.ts::collectGoalIdsFor`, `sidebar.ts::projectIdForGoal`, + `goal-dashboard-children-tab.ts::buildChildSummaries` candidates, + `goal-dashboard-plan-tab.ts` candidate filter. +- **Diagnosis:** S2 — six divergent derivations of one tree. +- **Fix:** memoized id/parent indices + `descendants()`, `directChildren()`, `projectOf()` + (chain walk), `depthOf()`, all cycle-guarded; invalidate on `state.goalsGeneration`. +- **Acceptance:** behaviour-preserving (existing tests green); new unit tests for the + `projectOf` chain fallback and walk-through-archived descendant counting. + +### T5 — Unify nesting into one render path *(after T1, T4; the big one)* +- **Files:** `src/app/sidebar-nesting.ts` (extend the node model with team-lead display + slots); `src/app/render-helpers.ts` (delete `renderTeamGroup`'s spawned-children + + archived-lead `spawnedSubGoalsOf` blocks; `renderGoalGroup` becomes pure row+sessions); + `src/app/sidebar.ts:1346-1456`; delete `computeSpawnedClaim` from + `src/app/sidebar-spawned-children.ts`; update `src/app/render.ts` mobile exclusion. +- **Diagnosis:** S1/D6/D9/D10/D11 — dual paths + the mirror claim function. +- **Fix:** the forest assigns each goal exactly one display anchor (under the parent's + team-lead slot when a stamped/attributable lead exists, else a plain forest child); the + badge always = transitive descendants from the full graph (T4); one expansion semantic + (child visibility = parent goal expanded AND, when slotted, lead expanded — pick one rule + and pin it). +- **Acceptance:** T1's placement table-tests green (incl. unstamped-child at-most-once); + `tests/e2e/ui/sidebar-spawned-children-dedupe.spec.ts`, `sidebar-child-loading.spec.ts`, + archived-layout specs green; NEW E2E asserting the descendant badge on a parent whose + children are claimed. +- **Dependencies:** T1, T4. Risk: medium (placement edges); rollback = revert to the dual + path (keep the new tests as documentation of intent). + +### T6 — Mobile/desktop reuse *(after T5)* +- **Files:** `src/app/render.ts:461-622`, `src/app/sidebar.ts`, NEW shared + `bucketSidebarDataByProject` helper. +- **Diagnosis:** D7 — mobile flat-renders unclaimed children, silently drops + inherited-project goals, no badges/truncation. +- **Fix:** mobile consumes the same forest renderer with `variant: "mobile"` (the + `renderProjectArchivedSection` pattern). +- **Acceptance:** mobile E2E (pattern: `sidebar-mobile-archived-per-project.spec.ts`) + showing a nested child indented under its parent on mobile AND a child with inherited + projectId visible. + +### T7 — Keyed sidebar lists *(after T5, to avoid churn)* +- **Files:** `src/app/sidebar.ts`, `src/app/render-helpers.ts`, `src/app/render.ts`. +- **Diagnosis:** S6 — positional DOM reuse transfers focus/animations between entities on + insertions/re-sorts. +- **Fix:** `repeat()` keyed by entity id for project sections, goal nodes, session rows. +- **Acceptance:** type-check + existing browser E2Es; a focused row's action button still + targets the same session after a new session is inserted above it (new E2E or fixture). + +### T8 — Single-instance role-picker popover *(independent)* +- **Files:** `src/app/sidebar.ts:1433, :485-615`; `src/app/render.ts:577`. +- **Diagnosis:** D8 — N popovers for N projects; keyboard/focus targets the first DOM copy, + which can be occluded. +- **Fix:** render `renderRolePickerDropdown()` once at the sidebar root; keep per-project + anchor-rect state. +- **Acceptance:** with ≥2 projects and the picker open, exactly one `.cwd-combobox` exists; + keyboard nav focuses the visible input. + +### T9 — Cleanups bundle *(independent, low risk)* +- **Files/items:** `src/app/api.ts:429` (add `teamGoalId` to auto-expand — also lets the + dedupe E2E drop its timing workaround); prune `_prevSessionStatus` + gate/PR caches on + goal/session removal; `src/app/sidebar-nesting.ts` (emit detached-cycle nodes as degraded + roots + warn, D12); `src/app/render-helpers.ts:887-894` (tree-scope the archived-forest + exclusion, D11 — skip if T5 already subsumed it); staleness token for the on-demand + team-agents fetch (`render-helpers.ts:1375-1403`); delete the false "bucket once" comment + in `sidebar-spawned-children.ts:160-163` (moot after T5). +- **Acceptance:** unit tests per item; an archived goal with a purged parent + archived + lead renders in the Archived section. + +**Suggested order:** T1 ‖ T2 → T3 → T4 → T5 → T6 ‖ T7 → T8 ‖ T9. T2/T3 (data-flow) and +T1/T4/T5 (tree model) are independent tracks until T5; everything visual lands after T5 to +avoid double churn. + +--- + +## 5. Checked and sound (no action) + +- **The 5cfc0306 "Loading… blanking" fix is a real root-cause fix**: initial-load keyed off + `sessionsGeneration` in a pure, real-source-tested helper (`session-load-state.ts`, + `tests/sidebar-loading-flash.test.ts`). +- **rAF coalescing** (`state.ts:687-712`) is respected by the sidebar; drag suppression + buffers exactly one render; the streaming bypass (PATH B) is confined to `AgentInterface` + as documented. +- **Expansion/scroll state survives re-renders** (module/state-backed with localStorage) — + poll ticks do not collapse accordions. +- **Gate-status and PR caches** have correct last-writer-wins tokens / in-flight guards + (`api.ts:1084-1161`) — the model T2 copies. +- **Goal-dashboard descendant fetches** are guarded (in-flight flags + + `currentGoalId === goalId` staleness checks, `goal-dashboard.ts:306-327`); no N+1 per + nesting level (single `/descendants` call). +- **Unread/read-state:** `markSessionVisited`'s `_readMirror` max() defends against the + poll racing the mark-read POST (`render-helpers.ts:226-275`); `updateLocalSessionStatus` + deliberately never touches `lastActivity` (pinned by `tests/spurious-idle-unread.spec.ts`). +- **`buildNestedGoalForest` internals** are well-built (Map dedupe, deterministic sort, + per-root cycle guard, orphan promotion, depth-cap truncation) — it lacks tests and a + second-consumer discipline, not quality. +- **Cross-tab project reorder** is sound (optimistic apply + echo + `projects_changed` + broadcast + equality gate). From 5a72202c829f381fce6c67d57d5e0a2995ed262b Mon Sep 17 00:00:00 2001 From: Artur Jonkisz Date: Wed, 10 Jun 2026 20:28:27 +0100 Subject: [PATCH 02/18] Added docs/design/extension-platform.md --- docs/design/extension-platform.md | 735 ++++++++++++++++++++++++++++++ 1 file changed, 735 insertions(+) create mode 100644 docs/design/extension-platform.md diff --git a/docs/design/extension-platform.md b/docs/design/extension-platform.md new file mode 100644 index 000000000..7dd9c84cb --- /dev/null +++ b/docs/design/extension-platform.md @@ -0,0 +1,735 @@ +# Bobbit Extension Platform — pack-first providers, hooks, runtimes, and ecosystem compat + +Status: design / implementation blueprint. Each phase in §13 is written to convert 1:1 into a +mission goal with acceptance criteria. + +--- + +## 1. Mission & principles + +Bobbit should stop growing core features and become an **extension platform**: memory, context +providers, skills, roles/personas, workflow templates, MCP servers, lifecycle hooks, managed +services, and UI panels all arrive as **marketplace packs** — installable, updatable, +per-entity toggleable, and uninstallable — instead of as core code. + +Principles (binding on every phase): + +1. **Pack-first from day one.** New machinery (providers, hooks, runtimes, capabilities) ships + as pack entity types immediately. The reference implementations — `session-memory` and + `hindsight` — are real packs in `market-packs/`, with zero feature code in core. Core grows + only *platform* code: loaders, dispatchers, supervisors, policy. +2. **Host-controlled compilation.** Packs and selector LLMs *propose*; Bobbit validates, + budgets, fences, records provenance, and applies at safe lifecycle boundaries. No pack gets + raw system-prompt rewrites or unbounded injection. +3. **Everything inspectable.** Every injected context block carries source, provider, reason, + and token estimate, and is visible in the existing prompt-sections inspector. Invisible + context selection cannot be trusted or debugged. +4. **Failures are non-fatal.** A dead memory service, a hung provider, a crashed worker — the + session always proceeds; the degradation is surfaced, never silent. +5. **Great defaults, changeable settings.** A pack that needs a service (Hindsight + Postgres) + must work from a single Enable click with generated secrets, a preconfigured cheap model, + and sane budgets — while exposing every knob for power users. +6. **One UX for memory.** The agent "just remembers". Which engine produced a memory (built-in + session search vs Hindsight) is provenance detail, not user burden. +7. **Trusted code, honest framing.** Packs are trusted code with resource/crash isolation only + (the deliberate stance of the Extension Host — see `docs/design/extension-host.md` and the + isolation simplification in #732). We disclose capabilities at install/enable; we do not + pretend to sandbox. + +Compatibility targets in scope: **Claude Code plugin format** (plugin marketplaces install as +packs), **MCP servers as pack content**, **pi extensions as pack content**, and +**Claude-style lifecycle hooks**. + +--- + +## 2. Substrate inventory (what we build on — verified anchors) + +| Substrate | Where | What we reuse | +|---|---|---| +| Pack resolver | `src/server/agent/pack-types.ts:26` (`EntityType`, `EntityLoader` at `:138`), `pack-resolver.ts`, `pack-list.ts` | One ordered pack list, scope order `builtin < server < global-user < project`, `pack_order`, `pack_activation` per-entity disable, conflict reporting, synchronous cache invalidation. New entity type = new loader. | +| Manifest validation | `src/server/agent/pack-manifest.ts:71-138` | Schema V1; unknown top-level keys ignored (forward compat); `contents.mcp` hard-rejected at `:95-97` (lifted for `schema: 2`). | +| Marketplace install | `src/server/agent/marketplace-install.ts`, `marketplace-source-store.ts`, REST in `server.ts` (§ marketplace routes) | Git/local/builtin sources, atomic staging install, `.pack-meta.yaml` provenance, update/uninstall. The staging step is the seam for the Claude-plugin format adapter. | +| Built-in first-party band | `src/server/agent/builtin-packs.ts`, `scripts/copy-builtin-packs.mjs`, `BOBBIT_BUILTIN_PACKS_DIR` | Pre-installed, resolve-in-place packs (e.g. `pr-walkthrough`): the delivery vehicle for `session-memory` and `hindsight`, and the test seam for fixture packs. | +| Extension Host execution tier | `src/server/extension-host/module-host-worker.ts`, `route-dispatcher.ts`, `action-dispatcher.ts`; contract in `src/shared/extension-host/host-api.ts` | Confined `worker_threads` execution of trusted pack server modules: per-call timeout (terminate-on-timeout), concurrency cap, per-session rate limit, import containment, pack-scoped stores, surface-binding tokens. Providers run on this tier. | +| Generated pi extensions | `src/server/agent/tool-guard-extension.ts:31` (codegen, `pi.on("tool_call")` + HTTP long-poll via `BOBBIT_GATEWAY_URL`/`BOBBIT_TOKEN`), `tool-activation.ts` (`writeMcpProxyExtensions`, `resolveExtensionPath`) | The proven gateway⇄agent bridge pattern. The provider-bridge extension (§5.4) and hook bridge (§13 P6) are new instances of it. | +| pi extension surface | `node_modules/@…/pi-coding-agent/dist/core/extensions/types.d.ts`, `system-prompt.d.ts` | Events: `before_agent_start` (systemPrompt replace), `context`, `tool_call`/`tool_result`, `turn_start/end`, `agent_start/end`, `session_before_compact`, `session_shutdown`, `input`; `pi.registerTool`; `BuildSystemPromptOptions`. | +| Prompt assembly + inspector | `src/server/agent/system-prompt.ts:294-448` (`PromptParts` → `assembleSystemPrompt`), `persistPromptSections` `:654`, `GET /api/sessions/:id/prompt-sections` (`server.ts:11354`) | Ordered prompt sections with byte budgets (`skillsCatalogBudget` pattern) and a shipping inspector. Provider blocks land here. | +| Session search | `src/server/search/flex-store.ts:155` (`FlexSearchStore`, BM25 + recency boost, chunking), `search-service.ts`, `indexer.ts`, `GET /api/search`; `transcript-reader.ts` + the `read_session` tool | Goals/sessions/messages/staff indexed per project; transcripts kept indefinitely in `.bobbit/state/sessions/.jsonl`. This is the `session-memory` provider's substrate. Missing today: automatic prompt-time recall — exactly the gap providers fill. | +| Docker supervision patterns | `src/server/agent/project-sandbox.ts` (DOCKER_BIN/execFile discipline, MSYS env handling), `sandbox-manager.ts` (idempotent ensure + in-flight dedupe), `aigw-manager.ts` (supervise external HTTP service) | The shapes the pack-runtime supervisor mirrors (not reuses — sandboxes are per-project repo isolation; runtimes are server-scoped services). | +| Secrets | `SecretsStore` (state-dir, gitignored) | At-rest storage for runtime secrets. | +| Stores | `RoleStore`/`RoleManager` (`Role{promptTemplate, accessory, toolPolicies, model, thinkingLevel}`), `WorkflowStore`/`WorkflowManager` (project.yaml-scoped, goal snapshots), `McpManager` (9 discovery locations incl. Claude's), `slash-skills.ts` (incl. `.claude/skills`/`.claude/commands`), `model-registry.ts` | Selectors and workflow templates operate **through** these stores, never around them. | +| Exemplar packs | `market-packs/artifacts` (tool+renderer+panel litmus), `market-packs/pr-walkthrough` (routes/panels/entrypoints, built-in band) | The authoring patterns and the test-litmus convention every new entity type follows. | + +--- + +## 3. Entity model + +Six new pack entity types, plus a capability registry. They stay distinct because they have +three genuinely different execution substrates and two authoring models: + +| Entity | Declared at | Code runs | Purpose | +|---|---|---|---| +| `providers` | `providers/.yaml` + module | **Gateway**, Extension Host worker tier | Typed lifecycle participants: contribute context blocks, react to turns/compaction/shutdown, make proposals (selector kind). | +| `hooks` | `hooks/.yaml` (or Claude `hooks/hooks.json`) | Gateway-spawned child processes | Claude-style declarative command hooks (JSON stdin/stdout); adapt onto the same dispatcher as providers. | +| `mcp` | `mcp/.yaml` | n/a (config) | MCP server configs merged into `McpManager` discovery with pack provenance. | +| `pi-extensions` | `pi-extensions/.yaml` → module | **Agent subprocess** (`--extension`) | Escape hatch + pi-ecosystem compat. Hooks pi events without registering tools. | +| `runtimes` | `runtimes/.yaml` + compose file | **Docker**, gateway-supervised | Managed services (health, ports, volumes, secrets) a pack depends on. | +| `workflows` | `workflows/.yaml` | n/a (templates) | Workflow **templates** instantiated into the project store at project/goal setup. `project.yaml` remains the source of truth; goal snapshotting is unchanged. | + +Notes on the boundaries: + +- **A provider is host-side, not a pi extension.** Prompt assembly happens in the gateway, so + before-prompt contribution with provenance is only possible host-side; recall/retain then + costs no per-call subprocess hop (the gateway already sees every prompt and every + `turn_end`); secrets (e.g. Hindsight keys) never enter the agent process; and the worker + tier's timeout/terminate machinery gives failure isolation for free. The agent-side reach a + provider occasionally needs is delivered by one Bobbit-generated bridge extension (§5.4), + not by pack code in the agent. +- **Provider tools are just tools.** `hindsight_recall` ships as a normal pack tool group with + the existing `bobbit-extension` provider type (the `defaults/tools/shell/` pattern), whose + pi extension calls back to the pack's own routes. No new "provider tool" concept; role + `toolPolicies`, the tool guard, and `pack_activation` apply unchanged. +- **`pi-extensions` is not a new trust tier.** Pack tools already run arbitrary code in the + agent process; this entity just admits it for event-only extensions. Both are gated by the + same acknowledgment (§8). + +### 3.1 Capability registry (inter-pack composition) + +Packs can publish and consume named capabilities: + +```yaml +# pack.yaml fragments +provides: [model-selector] # this pack implements the capability +requires: [model-selector] # this pack calls it (install-time check: warn + suggest) +``` + +A capability is implemented by a pack route or provider method registered under the +capability name. Consumers call it through the host: + +```ts +const res = await ctx.capabilities.call("model-selector", { + task: "implementation", spec, candidates, constraints: { maxCostTier: 2 }, +}); +``` + +The call is host-mediated: routed to the providing pack's module on the worker tier, budgeted +(timeout, payload caps), and recorded in the trace. This is how a persona selector consults a +"best model selector", how a workflow template asks for stage-model assignment, and how future +packs compose without importing each other's code. Conflicts (two packs providing the same +capability) resolve by pack precedence, same as entities. + +--- + +## 4. pack.yaml schema v2 + +Additive; `schema: 2` marks packs using the new keys. v1 validators ignore unknown keys, so a +v2 pack on an old server degrades to its v1 subset (roles/tools/skills/entrypoints load; the +rest is ignored). v2 servers warn when `schema` is newer than supported. + +```yaml +schema: 2 +name: hindsight +description: Persistent agent memory backed by Hindsight (recall/retain/reflect, per-project banks). +version: 1.0.0 +contents: + roles: [] + tools: [hindsight] # existing: tool GROUP dirs under tools/ + skills: [] + entrypoints: [hindsight-open, hindsight-route] + providers: [memory] # NEW → providers/memory.yaml + hooks: [] # NEW → hooks/.yaml (or hooks/hooks.json, Claude layout) + mcp: [] # NEW → mcp/.yaml (v1 rejection lifted when schema >= 2) + pi-extensions: [] # NEW → pi-extensions/.yaml + runtimes: [hindsight] # NEW → runtimes/hindsight.yaml + workflows: [] # NEW → workflows/.yaml (templates) +provides: [] # NEW: capability names this pack implements +requires: [] # NEW: capability names this pack consumes +routes: # existing (pack-level routes) + module: lib/routes.mjs + names: [status, recall, retain, reflect, banks, config] +``` + +Per-entity declaration files: + +```yaml +# providers/memory.yaml +id: hindsight-memory +kind: memory # memory | selector | generic +module: ../lib/provider.mjs # contained in pack root (existing path-guard discipline) +hooks: [sessionSetup, beforePrompt, afterTurn, beforeCompact, sessionShutdown] +runtime: hindsight # optional: binds to runtimes/; host injects {baseUrl, headers} +budget: { maxTokens: 1200, timeoutMs: 1500 } +defaultEnabled: true +config: # typed settings surface, rendered in the pack's settings UI + mode: { type: enum, values: [managed, external], default: managed } + externalUrl: { type: string, optional: true } + apiKey: { type: secret, optional: true } + autoRecall: { type: boolean, default: true } + autoRetain: { type: boolean, default: true } + recallBudget:{ type: number, default: 1200 } +``` + +```yaml +# runtimes/hindsight.yaml +id: hindsight +kind: docker-compose +compose: ../runtime/compose.yaml # images pinned by digest in the compose file +services: [hindsight, postgres] +healthcheck: { service: hindsight, path: /health, intervalMs: 5000, startupTimeoutMs: 120000 } +ports: + hindsight: { container: 8888, host: auto } # host port allocated + persisted by supervisor +volumes: [pg-data] # named volumes; survive pack update +secrets: + POSTGRES_PASSWORD: { generate: true } # crypto-random at first start +startPolicy: on-enable # never auto-start without a user action +``` + +```yaml +# workflows/multi-model-delivery.yaml (template) +id: multi-model-delivery +name: Multi-model delivery +description: Plan with a frontier model, implement cheap+precise, QA and review with independent models. +gates: [...] # same shape WorkflowStore validates today +``` + +Validator deltas (`pack-manifest.ts`): accept the six new `contents` keys (string arrays, +`isSafeBasename`-guarded), accept `provides`/`requires` (string arrays), lift the +`contents.mcp` rejection when `schema >= 2` (keep rejecting for v1 manifests). New +`EntityType` members + one `EntityLoader` per type (`pack-types.ts:26/:138` — the seam was +designed for this). `pack_activation` extends to all six types so per-entity disable works on +day one. + +--- + +## 5. The Lifecycle Hub + +One new gateway component — `src/server/agent/lifecycle-hub.ts` — the minimal host-controlled +orchestrator. It resolves enabled providers/hooks via the PackResolver, dispatches lifecycle +events to them on the Extension Host worker tier, validates and budgets their output, and +records provenance. + +### 5.1 Hook points + +| Hook | Fires from | May contribute | On failure/timeout | +|---|---|---|---| +| `sessionSetup` | New pipeline step in session setup, between tool resolution and `resolvePrompt` (`session-setup.ts`) | `ContextBlock[]` → new `PromptParts.dynamicContext` → rendered as a **"Dynamic Context"** prompt section, persisted by `persistPromptSections` (visible in the existing inspector) | Provider skipped; diagnostic recorded; session proceeds | +| `beforePrompt` | Gateway prompt-submit path, before `RpcBridge.prompt()` (and per-turn via the bridge, §5.4) | `ContextBlock[]` → fenced `` blocks prepended to the outgoing prompt | Hard timeout (default 1500ms) ⇒ prompt sent without blocks; diagnostic recorded | +| `afterTurn` | Existing event subscription (`turn_end`/`agent_end` in the session manager's lifecycle handler) | Nothing (fire-and-forget; receives the turn transcript slice) | Logged | +| `beforeCompact` | Compaction detection (`session_before_compact` via the bridge) | Nothing in P1 (notification: flush/retain before context loss) | Logged | +| `sessionShutdown` | Session archive/stop path | Nothing (flush) | Logged | +| `beforeGoalCreate` / `beforeSessionSpawn` | Goal-creation and session-spawn paths (P8) | Typed **proposals** (§12) — never direct mutation | Proposal dropped; deterministic defaults | + +Deferred deliberately: mid-turn `context` message mutation, compact-output mutation, +`tool_call`/`tool_result` interception for packs (P6 generalizes the existing tool-guard +long-poll for hooks), `before_provider_request` telemetry. + +### 5.2 Context blocks, budgets, provenance + +```ts +export interface ContextBlock { + id: string; // ":" + title: string; // "Relevant past work" + providerId: string; + authority: "memory" | "skill" | "tool" | "workflow" | "role" | "generic"; + content: string; // fenced by the HOST, never raw-injected + reason: string; // why this was included — shown in the inspector + priority: number; + tokenEstimate: number; // host-computed (~4 chars/token heuristic, as PromptSection.tokens) +} +``` + +Budgets: per-provider `budget.maxTokens` (default 1200–1600) and a global dynamic-context cap +(default 4000), enforced by priority-ordered truncation — the same approach as +`SKILLS_CATALOG_BUDGET` (`system-prompt.ts`). All blocks are wrapped by the host: + +```xml + +... + +``` + +Trace: `sessionSetup` blocks appear in prompt-sections (free inspector win). Per-turn +`beforePrompt` blocks and all diagnostics (timeouts, skips, budget omissions) append to +`.bobbit/state/session-context-trace/.jsonl`, exposed at +`GET /api/sessions/:id/context-trace`. + +### 5.3 Provider module contract + +```ts +// lib/provider.mjs — runs on the Extension Host worker tier (trusted, confined) +export default { + async sessionSetup(ctx) { return { blocks: [...] }; }, + async beforePrompt(ctx) { return { blocks: [...] }; }, // ctx.prompt, ctx.budget + async afterTurn(ctx) { /* fire-and-forget */ }, + async beforeCompact(ctx) { /* flush */ }, + async sessionShutdown(ctx){ /* flush */ }, +}; +// ctx: { sessionId, projectId?, scope: "project" | "global", cwd, goalId?, roleName?, +// prompt?, turn?, budget, config, // values from providers/.yaml config +// runtime?: { baseUrl, headers, status }, // when bound to a runtime +// store, // pack-scoped KV (existing pack-store) +// capabilities: { call(name, input) }, // §3.1 +// log } +``` + +Providers are trusted ambient code (fetch/fs available — consistent with routes.mjs). The host +validates *output* (schema, budget, fencing), not the code. + +### 5.4 The provider-bridge pi extension + +One Bobbit-generated extension per session (codegen modeled on +`generateToolGuardExtension`, `tool-guard-extension.ts:31`), wired in the same session-setup +step that writes the tool guard: + +- `before_agent_start` → `POST /api/sessions/:id/provider-hooks/before-prompt` with the + pending user input; the gateway runs `beforePrompt` across enabled providers (bounded), and + the extension appends the returned fenced text via pi's systemPrompt/append mechanism. The + gateway also refreshes the persisted prompt-sections snapshot so the inspector stays + truthful per turn. +- `turn_end` → fire-and-forget `POST …/after-turn`. +- `session_before_compact` → blocking-with-timeout `POST …/before-compact`. +- `session_shutdown` → `POST …/shutdown`. + +Auth and transport identical to the tool guard (`BOBBIT_GATEWAY_URL` + `BOBBIT_TOKEN`). + +### 5.5 Claude-hook adapter mapping + +The `hooks` entity registers command hooks onto the same Hub: + +| Claude hook | Hub point | Semantics | +|---|---|---| +| `SessionStart` | `sessionSetup` | stdout `additionalContext` → ContextBlock | +| `UserPromptSubmit` | `beforePrompt` | stdout-injected context (matches Claude semantics) | +| `Stop` | `afterTurn` | notification | +| `PreCompact` | `beforeCompact` | notification | +| `SessionEnd` | `sessionShutdown` | notification | +| `PreToolUse` / `PostToolUse` | tool-guard generalization (P6) | block/mutate via the existing long-poll endpoint | + +Spawned via `execFile`, JSON on stdin, timeout-bound, non-fatal. + +--- + +## 6. Memory architecture — two tiers, one UX + +The promise: **Bobbit gives the impression of infinite memory.** Two engines deliver it behind +one surface; neither is special-cased in core. + +- **Tier 1 — `session-memory` (built-in pack, default-on, zero dependencies).** Uses the + existing FlexSearch index and transcripts: `beforePrompt` queries BM25+recency over past + sessions/goals/messages in the project and contributes top-K bounded "Relevant past work" + blocks (each linking its source sessionId); `sessionSetup` recalls against the goal/task + spec. Every Bobbit user gets automatic recall with no Docker, no service, no setup — and + this pack is the litmus implementation that proves the provider API (P1). +- **Tier 2 — `hindsight` (pack, pre-installed dormant).** Semantic memory: consolidation, + reflect, mental models, cross-session synthesis — the upgrade. Detailed in §11. +- **One UX.** Both emit the same ContextBlocks under one budget; provenance disambiguates in + the inspector. Default priority: `hindsight > session-memory`; cross-engine dedupe is a P4 + refinement. The agent-facing story is simply: the agent remembers. + +**Scope-awareness (designed in from day one).** Provider contexts carry +`scope: "project" | "global"`. Memory bank routing: + +- Project sessions → bank `bobbit-proj-` **and** a shared `bobbit-global` bank, + recalled in parallel under one budget. +- The `hindsight_recall` tool and the panel accept `bank: current | global | all` + (`all` = fan-out across banks). +- A future gateway-level "Mission Control" surface (global sessions/staff above projects — + see §14) slots into `scope: "global"` without rework. + +Retains are tagged `sessionId` / `goalId` / `roleName` so the panel can filter per +session/goal. + +--- + +## 7. Managed runtimes + +New core: `src/server/runtimes/pack-runtime-supervisor.ts`, mirroring the discipline of +`project-sandbox.ts` (DOCKER_BIN, execFile wrapper, MSYS env handling) and `sandbox-manager.ts` +(lazy idempotent `ensureRuntime(id)` with in-flight dedupe), plus the +supervise-an-HTTP-service flavor of `aigw-manager.ts`. + +- **Compose project** `bobbit-pack-` — stable across pack updates, so named volumes (and + therefore data) survive updates; image digests pinned in the pack's compose file mean an + update recreates the app container against the same volume. +- **Env/secrets**: generated `.env` (mode 0600) under `~/.bobbit/state/pack-runtimes//`; + `generate: true` secrets created with `crypto.randomBytes` on first start; values stored via + `SecretsStore`; secrets never sent to panels (status only). +- **Ports**: `host: auto` allocates an ephemeral host port, persists it, re-validates on boot; + the bound provider reads the resolved `baseUrl` from its ctx — never hardcodes. +- **Status machine**: `docker-unavailable | stopped | starting | running | unhealthy`, driven + by the declared HTTP healthcheck. +- **REST**: `GET /api/pack-runtimes`, `POST /api/pack-runtimes/:id/{start|stop|restart}`, + `GET /api/pack-runtimes/:id/logs?tail=`. +- **Lifecycle**: `startPolicy: on-enable` — and a pinned platform rule: **a pre-installed pack + never starts containers without a deliberate user action.** Disable ⇒ `compose stop` (data + kept). Uninstall ⇒ `compose down` with **volumes preserved by default**; an explicit + "Delete data" confirmation runs `down -v` and removes the state dir. Reinstall reattaches + existing volumes. +- **Fallback**: when Docker is unavailable, the runtime reports `docker-unavailable` and the + consuming provider's `external` mode (user-supplied URL/key) is the first-class path, not an + error state. + +--- + +## 8. Trust & consent + +Packs are **trusted code**. The platform inherits the Extension Host stance: worker confinement +is resource/crash isolation and import hygiene, not a security sandbox; pack tools and +pi-extensions run with ambient access in the agent process, as tools always have. What v2 adds +is **disclosure granularity and deliberate activation**: + +1. **Capability summary at install/enable**, computed from the manifest: "runs code in the + gateway" (providers/hooks), "runs code in the agent" (tools/pi-extensions — already true + today, now stated), "connects MCP servers" (command/URL listed per server), "runs Docker + services" (images, ports, volumes), "requests secrets" (names + prompts), "provides/requires + capabilities". Shown inline on the install/enable card; no separate consent ceremony. +2. **Per-entity activation** — `pack_activation` covers all six new entity types. Disabling a + provider unregisters its hooks synchronously (`invalidateResolverCaches()` + Hub + re-resolve); disabling a runtime stops its containers. +3. **Hooks and pi-extensions need a one-time per-pack acknowledgment** (arbitrary commands on + the gateway host / arbitrary code in the agent process), default-deny until acknowledged. + Claude-converted packs additionally show the exact hook command lines. +4. **Memory consent posture** (owner decision): enabling a memory pack is the consent — + auto-recall and auto-retain start at Enable, with the enable card disclosing inline what is + stored (per-project bank), which model forms memories, and a cost note. Switching to + `external` mode (content leaves the machine) shows a one-line notice. Per-project opt-out + rides `pack_activation`. + +--- + +## 9. Claude Code plugin compatibility + +A **format adapter at install time**, not a parallel install system. The atomic-staging step in +`marketplace-install.ts` is the seam: + +- `MarketplaceSourceStore` sources gain `format: "bobbit" | "claude-plugin"`, auto-detected at + sync (presence of `.claude-plugin/marketplace.json`); browse lists plugins as packs. +- Install conversion into the staging dir: + +| Claude plugin piece | Bobbit pack entity | +|---|---| +| `commands/*.md` | skills (the `commands-flat` layout already exists, `pack-list.ts`) | +| `skills/` | skills | +| `agents/*.md` | roles (frontmatter name/description/model → `Role{name, promptTemplate, model}`; tool restrictions best-effort → `toolPolicies`) | +| `hooks/hooks.json` | `hooks` entities (§5.5 mapping) | +| `.mcp.json` | `mcp` entities | +| `scripts/`, assets | copied verbatim; `${CLAUDE_PLUGIN_ROOT}` rewritten to the pack root | + +- A `pack.yaml` (`schema: 2`) is synthesized; `.pack-meta.yaml` records + `sourceFormat: claude-plugin`. +- **Unsupported features are reported, never silently dropped**: statusline, output styles, + permission/sandbox settings, MCP transports `McpManager` can't handle → structured + `skipped: [{feature, reason}]` in the install response, rendered in the UI. +- Full fidelity depends on the `mcp` (P5) and `hooks` (P6) phases; the adapter phase (P7) is + sequenced after them to avoid shipping a degraded story. + +--- + +## 10. Reference pack: `session-memory` + +``` +market-packs/session-memory/ + pack.yaml # schema 2; contents.providers: [recall] + providers/recall.yaml # kind: memory; hooks: [sessionSetup, beforePrompt]; budget 1200/800ms + lib/provider.mjs # queries GET /api/search (or the search service via a host route) + panels/ + entrypoints/ # optional: small "what was recalled" panel (can defer to inspector) +``` + +Behavior: `beforePrompt` → FlexSearch query from the user prompt (BM25 + recency, project +scope, archived included) → top-K results formatted as compact blocks with session links; +`sessionSetup` → recall keyed on goal/task spec. Deterministic, no LLM, no network beyond the +gateway. Ships in the built-in band, **default-on** (it only reads data Bobbit already has). +Config: K, budget, source types (sessions/goals/messages), include-archived. + +This pack is also the platform's litmus: its tests pin the provider loader, Hub dispatch, +budget clamp, timeout-skip, provenance, and activation toggles against a real pack. + +## 11. Reference pack: `hindsight` + +``` +market-packs/hindsight/ + pack.yaml # §4 example + providers/memory.yaml # hooks: sessionSetup, beforePrompt, afterTurn, beforeCompact, sessionShutdown + runtimes/hindsight.yaml # §4 example + runtime/compose.yaml # hindsight + pgvector postgres, digest-pinned + tools/hindsight/ + hindsight_recall.yaml # provider: { type: bobbit-extension, extension: extension.ts } + hindsight_reflect.yaml + hindsight_retain.yaml + extension.ts # pi.registerTool ×3 → POST pack routes (tool-guard auth pattern) + panels/hindsight-memory.yaml # native panel + entrypoints/hindsight-open.yaml, hindsight-route.yaml + lib/provider.mjs, routes.mjs, HindsightPanel.js, hindsight-client.mjs # one shared REST client + src/ # TS sources, built like the other market packs +``` + +- **Provider flow**: `sessionSetup` → recall vs goal/task spec → "Relevant memory" section; + `beforePrompt` → recall vs user prompt (parallel project + global banks, 1500ms timeout ⇒ + skip); `afterTurn` → **async** retain of the turn (failures queue in the pack store, retried + next turn); `beforeCompact` → synchronous retain of salient facts before context loss; + `sessionShutdown` → flush. +- **Tools**: explicit `hindsight_recall` / `hindsight_reflect` / `hindsight_retain` for the + model, with `bank: current | global | all`. (Hindsight's own MCP server remains usable via + normal MCP discovery, orthogonally.) +- **Panel** (native, same-origin): status + mode setup (managed vs external), memory search, + recent retains, retain-queue/operations view, runtime health/logs links, settings (model for + memory formation, budgets, toggles). +- **Modes**: `managed` (default when Docker present — Enable → supervisor up → healthy → + works) | `external` (URL + API key). **Great defaults**: cheap memory-formation model + preconfigured (reusing the user's existing configured key by default), generated Postgres + password, sane recall budget — power users can change all of it, but nobody has to. +- **Distribution**: built-in first-party band, **dormant** until enabled (no containers, no + hooks active). +- **Failure modes (all non-fatal)**: Docker absent → setup card offers external mode; service + unhealthy/timeout → recall skipped + diagnostic in trace + status badge; retain failure → + queued with counter in panel; provider crash → worker isolation, provider marked errored for + the session. + +--- + +## 12. Selectors & composition + +The "something clever": after each prompt — or before creating a goal / spawning a session — a +selector decides which personas, workflows, skills, MCP servers, and models fit best. In this +platform, **selectors are themselves provider packs** (`kind: selector`), and the things they +select over are the existing stores. + +- **Decision points**: `beforeGoalCreate`, `beforeSessionSpawn`, `beforePrompt` (selector + flavor). The Hub hands selectors typed summaries — available roles (`RoleManager`), workflow + templates + project workflows (`WorkflowManager`), skills catalog (`slash-skills`), MCP + servers (`McpManager`), models (`model-registry`) — within an input budget. +- **Typed proposals, never mutation**: + +```ts +interface SelectorProposal { + role?: { roleName: string; model?: string; thinkingLevel?: string; + personaPatch?: string /* bounded, additive */; confidence: number; reason: string }; + workflow?: { action: "reuse" | "instantiate-template" | "ask-project-assistant"; + workflowId?: string; templateId?: string; confidence: number; reason: string }; + skills?: { include: string[]; reason: string }; + mcp?: { servers: string[]; reason: string }; + model?: { model: string; thinkingLevel?: string; reason: string }; +} +``` + +- **Approval policy (host-enforced)** — auto-apply when: the decision is pre-session/pre-goal, + the selected role does not expand tool grants beyond the default, the model is + available/not-more-expensive than policy allows, persona/AGENTS.md changes are additive and + bounded, and confidence clears threshold. Otherwise ask the user (mid-session role/model + change, tool expansion, new/cloned workflows, expensive optional steps, low confidence). + Workflow *creation* always goes through the existing WorkflowManager APIs and user approval. +- **LLM-backed selectors**: strict-JSON schema output, hard timeout, deterministic fallback + (invalid output ignored). The selector may call models via its own config or via the + `model-selector` capability. +- **Capability composition (the model-selector example)**: a `model-selector` pack + `provides: [model-selector]` — input `{task, spec, candidates, constraints}` → output + `{model, thinkingLevel, reason}`. The persona selector, workflow templates, and verification + steps all `requires: [model-selector]` and call it via `ctx.capabilities.call`. +- **Flagship composition — the `multi-model-delivery` pack**: roles + (`planner` pinned to a frontier model at high thinking, `implementer` pinned to a cheap + precise model, `qa` and `reviewer` pinned to independent strong models — each gate carrying + the initial spec) + a workflow template wiring plan → implement → QA → review with a + ralph-loop (re-run until green) step + `requires: [model-selector]` so stage models can be + chosen dynamically per goal. Installing one pack gives a project a complete multi-model + delivery pipeline; the selector offers it when a goal's prompt fits. + +--- + +## 13. Phases + +Each phase is one mission goal: independently shippable, master-green +(`npm run check`, `test:unit`, `test:e2e` at/above baseline), test-first per AGENTS.md (every +user-facing feature gets unit + browser E2E; real Docker only in `tests/manual-integration/`). + +### P1 — Schema v2 + providers entity + Lifecycle Hub + `session-memory` pack +- **Outcome**: packs can ship lifecycle providers; Bobbit dispatches + `sessionSetup/beforePrompt/afterTurn/beforeCompact/sessionShutdown` with budgets, fencing, + and provenance; the built-in `session-memory` pack gives every user automatic recall of past + work (the infinite-memory impression) with zero dependencies. +- **Scope**: `pack-types.ts` (+`"providers"`), `pack-manifest.ts` (schema 2 keys), new + provider loader (pattern: `pack-contributions.ts`), `src/server/agent/lifecycle-hub.ts`, + provider dispatch on the worker tier (pattern: `route-dispatcher.ts`), session-setup + pipeline step + provider-bridge codegen (pattern: `tool-guard-extension.ts`), + `system-prompt.ts` (`PromptParts.dynamicContext`), `server.ts` + (`/api/sessions/:id/provider-hooks/*`, `/api/sessions/:id/context-trace`), Market UI + provider listing + activation toggle, `market-packs/session-memory/`, plus a deterministic + `provider-demo` fixture pack (loaded via `BOBBIT_BUILTIN_PACKS_DIR`) recording hook + invocations to its pack store. +- **Acceptance**: manifest validation accepts/round-trips schema-2 `contents.providers`; + budget clamp + timeout-skip + non-fatal failure pinned by unit tests; bridge codegen + snapshot test; API E2E — session with provider-demo shows its block in prompt-sections with + provenance, afterTurn recorded, disabled-via-activation ⇒ no block; session-memory E2E — a + second session in a project recalls content from the first (block visible in inspector and + in the outgoing prompt), toggle off ⇒ none; browser E2E — inspector renders the Dynamic + Context section; Market UI toggle persists across reload. +- **Non-goals**: runtimes/Docker; Hindsight; selectors; mid-turn `context` mutation; + hooks/mcp/pi-extensions/workflows entities. +- **Dependencies**: none. + +### P2 — `hindsight` pack v1 (external-URL mode) +- **Outcome**: the Hindsight pack works end-to-end against a user-supplied Hindsight URL: + scope-aware banks (project + global), auto-recall, async auto-retain with retry queue, + the three tools, routes, and the native panel. +- **Scope**: `market-packs/hindsight/` (everything in §11 except the runtime), + build-script entry, provider config surface. +- **Acceptance**: unit — bank-id derivation incl. global fan-out, block formatting/fencing, + retain-queue retry (REST client mocked); API E2E against an **in-process stub Hindsight + server** — sessionSetup + beforePrompt blocks injected, turn_end retains with correct bank + + tags, stub down ⇒ session unaffected + diagnostic, `hindsight_recall` tool round-trips with + `bank: all`; browser E2E — panel configures external URL, shows status, search returns stub + results; per-project disable ⇒ no injection; persists across reload. +- **Non-goals**: managed Docker; mental-models UI; cross-engine dedupe. +- **Dependencies**: P1. + +### P3 — Managed runtimes + Hindsight zero-step enable +- **Outcome**: `contents.runtimes` supported; the supervisor brings up Hindsight + Postgres + with generated secrets, allocated port, and healthcheck; **Enable → working memory with no + manual steps**; uninstall preserves data unless explicitly purged. +- **Scope**: runtimes entity (manifest/loader), `src/server/runtimes/pack-runtime-supervisor.ts` + + runtime-manifest parser, REST endpoints, `.env`/secrets handling, uninstall hook in + `marketplace-install.ts`, panel status/logs/start-stop wiring, `runtime/compose.yaml` in the + pack. +- **Acceptance**: unit — manifest validation, port allocation/persistence, secret-generation + idempotence, compose arg construction (execFile mocked), keep-vs-purge; API E2E — mocked + docker binary walks stopped→starting→running on health 200; docker-unavailable surfaces the + external-mode setup card path; browser E2E — runtime status card states + logs view + (mocked); **manual-integration** (real Docker) — enable → compose up → healthy → + recall/retain round-trip → disable stops; volume survives a pack update. Pinned platform + rule: pre-installed band never auto-starts containers. +- **Non-goals**: arbitrary third-party runtime marketplace policy; Kubernetes; remote Docker. +- **Dependencies**: P1, P2. + +### P4 — Memory depth & polish +- **Outcome**: compaction never loses memory-worthy content; users can browse/manage memories; + the two memory tiers compose cleanly. +- **Scope**: beforeCompact sync-retain ordering (bridge + pack), memory-browser panel v2 + (browse/filter by session/goal, delete, retain-queue + operations view, reflect surface), + hindsight↔session-memory priority/dedupe under the shared budget, cost surfacing for memory + formation. +- **Acceptance**: API E2E — forced compaction triggers retain *before* compaction completes + (stub asserts ordering); browser E2E — memory browser lists/filters/deletes and survives + reload; unit — dedupe/priority under budget. +- **Non-goals**: mental-model auto-refresh scheduling; multi-bank admin UI. +- **Dependencies**: P2 (P3 for managed-mode paths). + +### P5 — MCP as pack content +- **Outcome**: packs ship MCP server configs that flow into existing discovery, meta-tools, + proxy extensions, and policy — with pack provenance and per-entity activation. +- **Scope**: lift `pack-manifest.ts:95-97` rejection for schema 2; `mcp` loader; a pack-sourced + discovery band in `mcp-manager.ts` (below user configs — a pack must never shadow a user's + own MCP config); conflict reporting; Market UI provenance badge; capability summary entry. +- **Acceptance**: unit — manifest + loader + precedence-vs-user-config; API E2E — pack-shipped + MCP server produces meta-tools via the existing `writeMcpProxyExtensions` path and is + removed on uninstall; browser E2E — pack-provenance badge in the tools UI. +- **Non-goals**: packs shipping MCP server *binaries*; new secret mechanisms for MCP. +- **Dependencies**: P1 (manifest plumbing precedent). + +### P6 — Hooks + pi-extensions entities +- **Outcome**: packs ship Claude-style command hooks (mapped per §5.5, including + PreToolUse/PostToolUse via a generalized tool-guard long-poll) and raw pi extension modules + (added to the session `--extension` list) — both behind a one-time per-pack trust + acknowledgment. +- **Scope**: `hooks` + `pi-extensions` loaders; hook dispatcher (execFile, JSON protocol, + timeouts); tool-guard endpoint generalization; extension-list assembly in session setup; + trust-grant store + consent UI. +- **Acceptance**: unit — hook mapping table, default-deny gate; API E2E — fixture pack's + PreToolUse hook blocks a tool call; un-acknowledged pack contributes no hooks/extensions; + browser E2E — acknowledgment flow, revoke works after reload. +- **Non-goals**: sandboxing claims; hook-driven prompt rewrites beyond the fenced path. +- **Dependencies**: P1. + +### P7 — Claude Code plugin marketplace adapter +- **Outcome**: a Claude plugin marketplace adds as a source; its plugins browse and install as + packs with full mapping (skills/commands/agents/hooks/mcp) and an explicit skipped-features + report. +- **Scope**: format detection in source sync; `claude-plugin-adapter.ts` conversion at the + staging seam; `${CLAUDE_PLUGIN_ROOT}` rewrite; UI source badge + skipped report rendering. +- **Acceptance**: unit — conversion mapping table incl. skip cases; API E2E — fixture plugin + repo installs, skills resolve with correct precedence, hooks/mcp entities materialize; + browser E2E — install a real public Claude marketplace plugin from the Market UI, slash + command appears in the composer, skipped report shown. +- **Non-goals**: bidirectional export (Bobbit packs → Claude plugins). +- **Dependencies**: P5, P6 (full fidelity). + +### P8 — Capability registry + selector framework +- **Outcome**: packs compose via `provides`/`requires` + `ctx.capabilities.call`; selector + providers make validated, policy-gated proposals at `beforeGoalCreate` / + `beforeSessionSpawn` / `beforePrompt`. +- **Scope**: manifest `provides`/`requires` + install-time dependency check; capability + dispatch on the worker tier; Hub decision points wired into goal-creation and session-spawn + paths; proposal types + approval policy (§12); proposal UI (accept/decline affordance); + fixture selector pack. +- **Acceptance**: unit — registry resolution incl. precedence + missing-dependency warning; + proposal validation (over-grant ⇒ requiresApproval; invalid JSON ⇒ deterministic fallback); + API E2E — fixture selector proposes role+workflow at goal creation, auto-applies under safe + policy, asks otherwise; browser E2E — proposal surfaced at goal creation, accept/decline + round-trips. +- **Non-goals**: codegraph provider; mid-session automatic role changes. +- **Dependencies**: P1. + +### P9 — Workflow-template entity + flagship selector packs +- **Outcome**: packs ship workflow templates; first-party `model-selector` capability pack and + `multi-model-delivery` pack deliver the multi-model pipeline (frontier-model planning → + cheap precise implementation → independent QA → independent review, ralph-loop) as a single + install. +- **Scope**: `workflows` loader (templates → instantiation through existing + `WorkflowManager`/`WorkflowStore` APIs at project setup or goal creation; project.yaml stays + the source of truth; goal snapshotting unchanged); `market-packs/model-selector/`; + `market-packs/multi-model-delivery/` (roles + template + `requires: [model-selector]`). +- **Acceptance**: unit — template instantiation + validation reuse (`workflow-validator`); + API E2E — installing the pack offers the template, instantiation creates a valid project + workflow, goal snapshot carries stage roles with pinned models, selector swaps a stage model + via the capability; browser E2E — template offered in goal-creation UI; per-gate role/model + visible. +- **Non-goals**: workflow auto-creation by LLM without approval; cross-project workflow sync. +- **Dependencies**: P8. + +--- + +## 14. Later (recorded, not goals) + +- **Mission Control as an extension** — global sessions/staff above projects ("talk to the + gateway before/instead of a project"). Needs one core enabler: a gateway-level pseudo-scope + for sessions/staff plus cross-project service APIs (search-all-projects, spawn-goal-in- + project). The memory architecture is already global-ready (`scope: "global"`, the + `bobbit-global` bank, `bank: all` fan-out); the surface itself (panel + routes + tools) fits + the Extension Host. Sketch only; not a phase. +- Codegraph/LSP context provider; `POST /api/context/compile-preview`; mid-turn pi `context` + message mutation; dynamic `setActiveTools` selection; `before_provider_request` telemetry; + per-conflict capability pinning. The ContextBlock schema keeps + `reason/priority/authority/tokenEstimate` so these slot in without breaking changes. + +## 15. Risks & open questions + +1. **Docker availability/Windows**: Docker Desktop/WSL2; `docker compose` v2 vs legacy + binary detection; MSYS path handling (reuse `project-sandbox.ts` discipline). External mode + must remain a first-class path, not an error state. +2. **Hindsight OCI images**: confirm a pullable, digest-pinned image (amd64+arm64). If none is + published, the pack needs a build step — decide before P3. +3. **Memory-formation LLM key**: default = reuse the user's configured key with a cheap model + (zero-step), documented on the enable card with a cost note. A separate-key option exists + in settings. Trade-off: memory ops bill to the main key. +4. **beforePrompt latency**: up to `timeoutMs` (1500ms default) added per turn when memory is + enabled. Mitigations: timeout-skip (pinned), prompt-hash recall cache, prefetch-at-turn-end + for the next turn. Measure in P2 and record the budget. +5. **Ports & multi-server**: persisted auto-allocated ports must re-validate on boot; two + Bobbit servers on one machine sharing a compose project name need a server-identity suffix — + decide in P3. +6. **Postgres migrations on pack update**: minor updates ride volume persistence + app-run + migrations; a schema-breaking Hindsight update needs a documented path (backup-before- + update?). Open. +7. **Secrets at rest**: 0600 `.env` + `SecretsStore`; no OS keychain in v1. Threat model: + localhost-bound ports + bearer keys; document it. +8. **Bank lifecycle**: project deletion → prompt to delete the bank (default keep); projectId + reuse across servers sharing one Hindsight could collide — consider a server-salt in the + bank id. +9. **CI has no Docker**: the stub-Hindsight E2E is the real gate; managed-mode regressions + surface only in `tests/manual-integration/` — consider a scheduled weekly manual run. +10. **Schema-v2 on v1 servers**: silent degradation (new keys ignored). v2 servers warn on + newer-than-supported `schema`; packs requiring v2 should say so in `description`. +11. **Selector trust creep**: selectors must stay proposal-only; the approval policy is the + single enforcement point — pin it with tests (over-grant ⇒ approval required) before any + selector pack ships. +12. **Token estimation**: the ~4 chars/token heuristic (already used by `PromptSection.tokens`) + is approximate; fine for budgeting, recorded as a known approximation. From 4a93680763233ed3f7e3efd891647a818715ab1e Mon Sep 17 00:00:00 2001 From: Artur Jonkisz Date: Wed, 10 Jun 2026 20:47:19 +0100 Subject: [PATCH 03/18] Added docs/design/extension-platform-implementation-plan.md --- .../extension-platform-implementation-plan.md | 760 ++++++++++++++++++ 1 file changed, 760 insertions(+) create mode 100644 docs/design/extension-platform-implementation-plan.md diff --git a/docs/design/extension-platform-implementation-plan.md b/docs/design/extension-platform-implementation-plan.md new file mode 100644 index 000000000..8c9503e0f --- /dev/null +++ b/docs/design/extension-platform-implementation-plan.md @@ -0,0 +1,760 @@ +# Extension Platform — Implementation Plan (hand-off) + +Companion to [extension-platform.md](extension-platform.md) (the WHAT/WHY — read its §3–§12 +before implementing anything). This document is the HOW: a tree of goals, each independently +mergeable and verifiable, written so an implementer agent **that has not seen any other goal** +can execute its goal from this text alone. + +> **Anchor baseline:** master @ `6ec8c8f9` (2026-06-10). Line numbers WILL drift — always +> locate by **symbol name** (function/class/field), and treat line numbers as hints. If a +> named symbol does not exist where described, STOP and re-derive from the pattern file cited +> next to it; do not improvise a parallel mechanism. +> +> **Precision policy:** Goals G1–G3 are specified to file/function level. G4–G9 are specified +> to contract level (their substrate is created by G1–G3 and will have evolved); their +> implementers must re-verify every anchor first and follow the stated contracts exactly. + +--- + +## 0. Universal rules — every goal's definition of done + +These apply to EVERY sub-goal below. A sub-goal is not done until all of them hold. + +1. **Read before editing.** Run `rg "" docs/ tests/ src/` for each seam you touch and + read the hits (AGENTS.md rule). Pinning tests are the invariants — if your change breaks + one, fix your change, not the test. +2. **Test-first.** Author the goal's listed tests BEFORE the implementation; confirm each is + RED on the unmodified tree where the spec says so, then flip GREEN. New invariants you + introduce must each get a pinning test. +3. **Gates:** `npm run check` clean · `npm run test:unit` green · `npm run test:e2e` green + (run the suites relevant to your change per AGENTS.md: UI-only → unit; server → unit+e2e). + Real Docker tests go ONLY in `tests/manual-integration/`. +4. **Every user-facing feature needs a browser E2E** (navigation, happy path, persistence + across reload) — pattern: `tests/e2e/ui/settings.spec.ts`. +5. **No flaky tests.** No `test.skip`/quarantine/retries/timeout bumps. Deterministic fixtures + only. +6. **Master stays green; minimal change.** Do not refactor neighbouring code, do not touch + seams owned by other goals (each goal lists its owned files). +7. **Conventions:** LF line endings; lowercase-kebab file names; new server code under + `src/server/…` mirrors sibling style; packs built like `market-packs/pr-walkthrough` + (TS in `src/`, built bundles in `lib/`, wired into `scripts/build-market-packs.mjs` and — + for built-in band packs — `scripts/copy-builtin-packs.mjs` `FIRST_PARTY_PACKS`). +8. **Coordination note:** `src/server/agent/session-manager.ts` and `session-setup.ts` are + also touched by the comms-stack reliability backlog + ([comms-stack/04-current-state-and-backlog.md](comms-stack/04-current-state-and-backlog.md)). + Keep edits confined to the named functions; rebase rather than resolve broad conflicts. + +### 0.1 Patterns library (copy these, don't invent) + +| Need | Copy from | +|---|---| +| Pack-scoped contribution loader (yaml per file, validation, containment) | `src/server/agent/pack-contributions.ts` (`loadPackContributions`, `PanelContribution`) | +| Executing pack module code with timeout/isolation | `src/server/extension-host/module-host-worker.ts` (`ModuleHost.invoke(InvokeRequest, timeoutMs)`) + `route-dispatcher.ts` (`RouteRegistry.resolve`) | +| Generated pi extension calling the gateway | `src/server/agent/tool-guard-extension.ts` (`generateToolGuardExtension`, invoked from `session-setup.ts::resolveToolActivation`, appended as `--extension` to `plan.bridgeOptions.args`) | +| Prompt section with byte budget | `src/server/agent/system-prompt.ts` (`buildSkillsCatalogSection`, `skillsCatalogBudget`, `getPromptSections`, `persistPromptSections`) | +| Docker execFile discipline | `src/server/agent/project-sandbox.ts` (DOCKER_BIN, MSYS env handling) | +| Idempotent ensure-with-inflight-dedupe supervisor | `src/server/agent/sandbox-manager.ts` | +| Marketplace REST + atomic install | `src/server/agent/marketplace-install.ts` (`installPack`/`uninstallPack`/`updatePack`) | +| Per-entity activation | `src/server/agent/project-config-store.ts` (`DisabledRefs`, `getPackActivation`/`setPackActivation`) + `pack-resolver.ts` `ActivationFilter` | +| API E2E boot | `tests/e2e/e2e-setup.ts` (`base()`, `bobbitDir()`, `gitCwd()`) | +| Marketplace browser E2E | `tests/marketplace-active-project.spec.ts` | +| Exemplar packs | `market-packs/artifacts` (tool+renderer+panel), `market-packs/pr-walkthrough` (routes/panels/entrypoints + built-in band) | + +### 0.2 One architectural refinement vs the design doc + +`extension-platform.md` §4 sketches the new entity types as `EntityLoader` entities. **The +implementation uses the pack-contributions path instead** (the path panels/entrypoints/routes +already use), because the new types are **pack-scoped** — two packs may each ship a provider +with id `memory` and BOTH must be active; `EntityLoader` name-merging (shadowing by name +across packs) is the wrong semantics. Concretely: providers, hooks, mcp, pi-extensions, +runtimes, and workflow templates are loaded by `pack-contributions.ts`-style loaders into the +`PackContributionRegistry`, keyed `(packId, contributionId)`, with `contents.` lists as +the activation catalogue and `DisabledRefs.` as the per-entity toggles — exactly the +existing entrypoints model. `EntityType` in `pack-types.ts` is NOT extended. Record this in +any code comment that would otherwise cite the design doc's sketch. + +--- + +## 1. Goal map + +``` +G1 Providers + Lifecycle Hub (P1) + ├─ G1.1 Manifest schema 2 + activation plumbing (no deps) + ├─ G1.2 Lifecycle Hub core + worker dispatch + trace (after G1.1) + ├─ G1.3 sessionSetup hook → Dynamic Context prompt section (after G1.2) + ├─ G1.4 Per-turn hooks: gateway events + provider-bridge (after G1.3) + ├─ G1.5 Market UI: provider listing + toggles (after G1.1; parallel with G1.2-4) + └─ G1.6 session-memory pack (after G1.4) +G2 Hindsight pack, external mode (P2) + ├─ G2.1 REST client + stub-server harness (no deps; parallel with G1) + ├─ G2.2 Pack core: provider + routes + config + banks (after G1.4, G2.1) + └─ G2.3 Tools + panel + entrypoints (after G2.2) +G3 Managed runtimes (P3) + ├─ G3.1 Runtime manifest + secrets/env/ports (pure) (after G1.1) + ├─ G3.2 PackRuntimeSupervisor + REST (docker mocked) (after G3.1) + └─ G3.3 Wire-up: lifecycle hooks, hindsight managed mode, (after G3.2, G2.3) + consent line, manual-integration test +G4 Memory depth (P4): G4.1 beforeCompact ordering · G4.2 memory browser v2 · + G4.3 dual-provider dedupe/priority + cost (after G2.x; G4.* parallel) +G5 MCP as pack content (P5): G5.1 entity+discovery band+UI (after G1.1) +G6 Hooks + pi-extensions (P6): + ├─ G6.1 hooks entity + command dispatcher + Hub mapping (after G1.4) + ├─ G6.2 PreToolUse/PostToolUse via tool-guard generalization (after G6.1) + └─ G6.3 pi-extensions entity + trust acknowledgment (after G1.1) +G7 Claude plugin adapter (P7): G7.1 detect+browse · G7.2 convert+skipped report · + G7.3 UI + real-marketplace E2E (after G5, G6) +G8 Capabilities + selectors (P8): + ├─ G8.1 Capability registry (provides/requires/call) (after G1.2) + ├─ G8.2 Selector decision points + proposals + policy (after G8.1) + └─ G8.3 Proposal UI + fixture selector pack (after G8.2) +G9 Workflow templates + flagship packs (P9): + ├─ G9.1 workflows entity + instantiation (after G1.1) + ├─ G9.2 model-selector capability pack (after G8.1) + └─ G9.3 multi-model-delivery pack (after G9.1, G9.2) +``` + +**Merge lanes (parallelizable):** Lane A: G1.1→G1.2→G1.3→G1.4→G1.6 · Lane B: G1.5 · +Lane C: G2.1 (immediately) then G2.2→G2.3 · Lane D: G3.1→G3.2 (after G1.1) · +Lane E: G5, G6.3, G9.1 (after G1.1). **File-conflict hotspots:** `pack-contributions.ts` + +`pack-contribution-registry.ts` (G1.1, G3.1, G5, G6, G9.1 — serialize merges in goal order); +`session-setup.ts` (G1.3, G1.4, G6.3); `server.ts` route block (most goals — keep each goal's +routes in one contiguous block); `marketplace-install.ts` (G3.3, G7.2). + +--- + +## G1 — Providers + Lifecycle Hub + +### G1.1 Manifest schema 2 + provider contribution + activation plumbing + +**Outcome.** `pack.yaml` accepts `schema: 2` with six new `contents` keys and +`provides`/`requires`; `providers/.yaml` files load into the contribution registry; +per-provider activation toggles work end-to-end through the existing REST. No runtime behavior +yet (nothing dispatches providers) — independently mergeable because everything added is inert +until G1.2. + +**Owned files.** `src/server/agent/pack-manifest.ts`, `pack-types.ts` (types only), +`pack-contributions.ts`, `src/server/extension-host/pack-contribution-registry.ts`, +`project-config-store.ts` (DisabledRefs), the pack-activation REST handlers. + +**Context.** `validateManifest` (`pack-manifest.ts:71-138`) currently: requires +`contents.{roles,tools,skills}` arrays, optional `contents.entrypoints` (safe basenames), +**rejects `contents.mcp`** at `:95-97`, ignores unknown top-level keys. +`loadPackContributions(packRoot, manifest)` (`pack-contributions.ts:111`) loads +panels (auto-discovered), entrypoints (filtered by `contents.entrypoints`), routes. +`DisabledRefs` (`project-config-store.ts`) = `{roles?, tools?, skills?, entrypoints?}`; +accessors `getPackActivation`/`setPackActivation` (`:855/:868`). + +**Steps.** +1. `pack-types.ts`: add to `PackManifest`: + `schema?: number` (default 1), `provides?: string[]`, `requires?: string[]`, and + `contents` gains `providers?/hooks?/mcp?/piExtensions?/runtimes?/workflows?: string[]` + (YAML keys: `pi-extensions` maps to `piExtensions`). Do NOT touch `EntityType` (§0.2). +2. `pack-manifest.ts::validateManifest`: + - parse `schema` (must be a positive integer if present; default 1); + - for each new contents key: if present, must be `string[]` of safe basenames (reuse the + entrypoints validation helper); default `[]`; + - `contents.mcp` present **and** `schema >= 2` → accepted; `schema < 2` → keep the existing + rejection message verbatim; + - `provides`/`requires`: optional `string[]`, entries match `/^[a-z0-9][a-z0-9-]*$/`; + - if `schema > 2`: log one warning (`pack.yaml: schema 3 is newer than supported (2)`), + still load the v2 subset. +3. `pack-contributions.ts`: add `ProviderContribution` + `{ id, kind: "memory"|"selector"|"generic", module, hooks: string[], runtime?, budget: + {maxTokens:number, timeoutMs:number}, defaultEnabled: boolean, config?, listName, + sourceFile, packRoot }` + `loadProviders(packRoot, manifest)` mirroring + `loadEntrypoints` exactly: only files named in `contents.providers` load + (`providers/.yaml`); `module` resolved relative to the yaml and containment-checked + against packRoot (same guard the routes module uses); `hooks` entries must be from + `["sessionSetup","beforePrompt","afterTurn","beforeCompact","sessionShutdown"]` + (unknown → load error for THAT provider, pack still loads); budget defaults + `{maxTokens: 1600, timeoutMs: 1500}` clamped to `maxTokens∈[64,8192]`, + `timeoutMs∈[100,10000]`. Duplicate provider id within a pack → `PackContributionError` + (same as panels). +4. `pack-contribution-registry.ts`: index providers like entrypoints (winning-pack collapse, + activation filtering via `DisabledRefs.providers` ↔ `listName`); add + `listProviders(projectId): ProviderContribution[]` returning only installed+active+enabled. +5. `project-config-store.ts`: extend `DisabledRefs` with + `providers?/hooks?/mcp?/piExtensions?/runtimes?/workflows?: string[]`; the pack-activation + REST catalogue response must include the new types (find the handler via + `rg "pack-activation" src/server` and extend the catalogue builder symmetrically). +6. Cache invalidation: confirm `invalidateResolverCaches()` already drops + `packContributionRegistry` (it does — verify, don't re-add). + +**Tests (author first).** +- Extend the existing pack-manifest unit test file (find via `rg "validateManifest" tests/`): + schema-2 manifest with all six keys parses; `contents.mcp` rejected at schema 1 (existing + test stays green) and accepted at schema 2 (RED before step 2); bad `provides` entry + rejected; `schema: 3` loads with warning. +- New `tests/pack-providers-loader.test.ts`: fixture pack dir (tmp) with two providers — + valid one loads with clamped budget; unknown hook name → that provider errored, other loads; + module outside pack root → error; duplicate id → `PackContributionError`; provider NOT in + `contents.providers` → not loaded. +- Extend the registry test (find via `rg "PackContributionRegistry" tests/`): disabled ref in + `DisabledRefs.providers` hides the provider; re-enabling restores it. +- API E2E: `PUT /api/marketplace/pack-activation` round-trips a `providers` disabled ref. + +**Acceptance.** All above green; `git grep "EntityType" | grep -i provider` → no hits (the +refinement honored); zero behavior change for v1 packs (run the full existing marketplace +test set). + +**Non-goals.** Dispatching providers (G1.2); loaders for hooks/mcp/pi-extensions/runtimes/ +workflows (G5/G6/G3/G9 own those — but the *manifest keys* are accepted now). + +--- + +### G1.2 Lifecycle Hub core + worker dispatch + budgets + trace + +**Outcome.** A `LifecycleHub` that resolves enabled providers and dispatches a named hook to +each on the Extension Host worker tier with per-provider timeout, collects `ContextBlock[]`, +applies budgets, fences content, and records a trace. Pure server core + fixture-driven tests; +nothing calls it yet from session paths (mergeable independently). + +**Owned files (new).** `src/server/agent/context-blocks.ts`, +`src/server/agent/lifecycle-hub.ts`, `src/server/agent/context-trace-store.ts`. **Edited:** +`src/server/extension-host/module-host-worker.ts` (one union member). + +**Steps.** +1. `context-blocks.ts`: + ```ts + export interface ContextBlock { + id: string; title: string; providerId: string; + authority: "memory"|"skill"|"tool"|"workflow"|"role"|"generic"; + content: string; reason: string; priority: number; tokenEstimate: number; + } + export function estimateTokens(s: string): number; // ceil(s.length / 4) — same heuristic as PromptSection.tokens + export function fenceBlock(b: ContextBlock): string; + // `\n{content}\n` + // reason/title attribute values: strip newlines + escape double quotes. + export function applyBudgets(blocks: ContextBlock[], perProviderMax: Map, + globalMax: number): { kept: ContextBlock[]; omitted: {block: ContextBlock; why: string}[] }; + // sort by priority desc then provider order; truncate the first over-budget block's + // content to fit (append "…[truncated]"), drop the rest; never partially emit a block + // smaller than 32 tokens — drop instead. + ``` +2. `module-host-worker.ts`: extend `InvokeRequest.exportKind` union with `"providers"`; the + worker resolves the member on the module's **default export object** for that kind (routes + use named exports — mirror whichever branch handles member lookup and add the + default-export branch). No other worker changes. +3. `lifecycle-hub.ts`: + ```ts + export type LifecycleHook = "sessionSetup"|"beforePrompt"|"afterTurn"|"beforeCompact"|"sessionShutdown"; + export interface HookCtx { sessionId: string; projectId?: string; scope: "project"|"global"; + cwd: string; goalId?: string; roleName?: string; prompt?: string; + turn?: { index: number }; budget: { maxTokens: number }; + config: Record; // provider config values + runtime?: { baseUrl: string; headers: Record; status: string }; + gateway: { baseUrl: string; token: string }; // loopback REST for trusted provider code + } + export class LifecycleHub { + constructor(deps: { registry: PackContributionRegistry; moduleHost: ModuleHost; + trace: ContextTraceStore; gatewayInfo: () => {baseUrl:string;token:string}; + globalMaxTokens?: number /* default 4000 */ }); + async dispatch(hook: LifecycleHook, base: Omit): + Promise<{ blocks: ContextBlock[]; diagnostics: HubDiagnostic[] }>; + } + ``` + `dispatch`: resolve providers for `projectId` whose `hooks` include `hook`; for each, + `ModuleHost.invoke({ exportKind: "providers", member: hook, url: , + packRoot, ctx-as-arg })` with `timeoutMs = provider.budget.timeoutMs`. Per-provider + failure/timeout ⇒ diagnostic `{providerId, hook, error|"timeout", ms}`, continue. Validate + each returned block (shape + string fields + `providerId` forced to the provider's id + + `tokenEstimate` recomputed host-side); then `applyBudgets`. Record one trace entry per + dispatch: `{ts, hook, sessionId, providers: [{id, ms, blocks, omitted, error?}]}`. +4. `context-trace-store.ts`: `appendTrace(sessionId, entry)` / + `readTrace(sessionId, limit?)` over JSONL at + `/session-context-trace/.jsonl` (atomic append; create dir lazily; + cap file at 2MB by dropping oldest — simple rewrite). + +**Tests (author first).** +- `tests/context-blocks.test.ts`: fencing escapes quotes/newlines in attributes; budget: 3 + blocks/2 fit → third dropped with reason; oversize first block truncated; <32-token + remainder dropped not truncated; global cap binds before per-provider headroom. +- `tests/lifecycle-hub.test.ts` (node:test, real `ModuleHost`, fixture modules in tmp dir): + (a) two fixture providers return blocks → merged, budgeted, provenance forced; + (b) provider that `await new Promise(r=>setTimeout(r, 5000))` with `timeoutMs: 200` → + diagnostic `timeout`, other provider unaffected, dispatch total time < 1s; + (c) provider throws → diagnostic, no crash; (d) provider returning malformed blocks → + blocks dropped with diagnostic; (e) trace file contains the dispatch record. +- `tests/context-trace-store.test.ts`: append/read round-trip; 2MB cap behavior. + +**Acceptance.** All green; `ModuleHost` route/action tests untouched and green; no session +code calls the hub yet (grep). + +**Non-goals.** Prompt integration (G1.3); endpoints (G1.4); selector hooks (G8). + +--- + +### G1.3 sessionSetup hook → "Dynamic Context" prompt section + +**Outcome.** New sessions dispatch `sessionSetup` through the Hub; returned blocks render as a +**Dynamic Context** section in the system prompt and appear in the existing prompt-sections +inspector with provenance. Proven by a deterministic fixture pack. + +**Owned files.** `src/server/agent/system-prompt.ts`, `session-setup.ts`, +`session-manager.ts` (construction/injection of the Hub only), `tests/fixtures/packs/provider-demo/`. + +**Context.** Pipeline steps run in order `resolveBridgeOptions(:331) → resolveGoalExtensions(:389) +→ resolveTools(:414) → resolvePrompt(:451) → resolveToolActivation(:606)`, driven by +`executePlan` (insertion point ~line 699, before `resolvePrompt`) and `executeWorktreeAsync` +(~line 887). `PromptParts` (`system-prompt.ts:294`), `getPromptSections(:543)`, +`persistPromptSections(:654)` called from `session-manager.ts:1645`. + +**Steps.** +1. `system-prompt.ts`: add `dynamicContext?: ContextBlock[]` to `PromptParts`. In + `getPromptSections` (and the parallel branch in `assembleSystemPrompt` — note both + call sites of `buildSkillsCatalogSection` at `:471` and `:600`; mirror that duality), when + `dynamicContext?.length`, append a final section + `{label: "Dynamic Context", source: "providers", content: blocks.map(fenceBlock).join("\n\n")}` + AFTER all existing sections (it is the freshest, lowest-authority content; placement + pinned by test). +2. `session-setup.ts`: new async step `resolveDynamicContext(plan, ctx)` inserted before + `resolvePrompt` in BOTH `executePlan` and `executeWorktreeAsync`: if `ctx.lifecycleHub` is + set, `const {blocks} = await ctx.lifecycleHub.dispatch("sessionSetup", {...ids, scope: + plan.projectId ? "project" : "global", prompt: plan.initialPromptText ?? undefined})`; + stash on the plan so `resolvePrompt` copies them into `parts.dynamicContext`. Entire step + wrapped in try/catch → log + proceed (provider failures never block a spawn). + Add `lifecycleHub?: LifecycleHub` to `PipelineContext`. +3. `session-manager.ts`: construct one `LifecycleHub` at manager init (deps: the existing + contribution registry + a `ModuleHost` instance + trace store + gateway info from the + same source the tool guard uses) and pass it through the pipeline context. Keep this the + ONLY new session-manager edit. +4. Fixture pack `tests/fixtures/packs/provider-demo/`: + `pack.yaml` (schema 2, `contents.providers: [demo]`), `providers/demo.yaml` + (hooks: all five; budget 512/1000), `lib/provider.mjs` — deterministic: `sessionSetup` + returns one block `{id:"demo:setup", title:"Demo", content:"DEMO_SETUP_BLOCK ", + reason:"fixture", priority:10}`; every hook also POSTs `{hook, sessionId}` to + `ctx.gateway.baseUrl + "/api/ext/route/record"`? — NO: simpler and dependency-free, append + to a file under `ctx.cwd + "/.provider-demo-log"` (the worker has fs; the test reads the + file). Loading: verify `buildPackList` supports a `BOBBIT_BUILTIN_PACKS_DIR` override; if + it does not exist on the current tree, add it (env var consulted where the builtin packs + dir is resolved; production default unchanged) — that override is this fixture's loader + and G-wide test seam. + +**Tests (author first).** +- Unit `tests/dynamic-context-section.test.ts`: `getPromptSections` with two blocks → last + section is "Dynamic Context", fenced content, token count populated; absent when no blocks. +- API E2E `tests/e2e/provider-session-setup.spec.ts` (in-process gateway, + `BOBBIT_BUILTIN_PACKS_DIR` → fixtures): create session → `GET + /api/sessions/:id/prompt-sections` contains the Dynamic Context section with + `DEMO_SETUP_BLOCK`; disable the provider via pack-activation → new session has no section; + fixture log shows exactly one `sessionSetup`. +- Codegen-free; no browser E2E here (inspector UI already renders sections — verify by the + existing prompt-sections UI spec if one exists; otherwise G1.5 covers UI). + +**Acceptance.** Sections appear with provenance; spawn latency without providers unchanged +(no Hub configured ⇒ zero-cost path); failure injection (fixture module throws) still spawns. + +**Non-goals.** Per-turn dispatch (G1.4); UI (G1.5). + +--- + +### G1.4 Per-turn hooks: gateway-event hooks + provider-bridge extension + endpoints + +**Outcome.** `afterTurn`/`sessionShutdown` fire from the gateway's existing event stream; +`beforePrompt` (per turn) and `beforeCompact` fire via a Bobbit-generated **provider-bridge pi +extension** with blocking semantics; per-turn blocks are traceable via +`GET /api/sessions/:id/context-trace`. + +**Owned files (new).** `src/server/agent/provider-bridge-extension.ts`. **Edited:** +`session-setup.ts::resolveToolActivation` (append one more `--extension`), `server.ts` +(4 endpoints), `session-manager.ts` (two event-handler call sites). + +**Design decisions an implementer must follow exactly:** +- **Never mutate the user's message text.** Injecting recall into the outgoing prompt text + would corrupt the transcript echo and re-open the optimistic-reconciliation duplicate class + (see comms-stack docs). Per-turn injection goes through the **system prompt tail**: + the bridge handles pi's `before_agent_start`; inspect the event payload type in + `node_modules/@…/pi-coding-agent/dist/core/extensions/types.d.ts` + (`BeforeAgentStartEvent` / `BeforeAgentStartEventResult`): if the event exposes the current + system prompt, return `{ systemPrompt: stripPreviousTail(current) + tail }`; the tail is + delimited `\n…` so + replacement is idempotent turn-over-turn. If the event does NOT expose the current prompt, + maintain the full prompt client-side: the gateway endpoint returns + `{ tail, fullSystemPrompt }` (gateway knows the assembled prompt — it built it) and the + extension returns the recomposed prompt. Pick whichever the types support; pin with the + codegen snapshot test. +- **Bridge transport** = the tool-guard pattern verbatim: generated TS reads + `BOBBIT_GATEWAY_URL` + `BOBBIT_TOKEN`, `fetch` with an AbortController timeout (2500ms + beforePrompt / 5000ms beforeCompact); on ANY failure return undefined (turn proceeds + without dynamic context — non-fatal is the platform invariant). + +**Steps.** +1. Endpoints (one contiguous block in `server.ts`, bearer-authed like the tool-grant + endpoints): + - `POST /api/sessions/:id/provider-hooks/before-prompt` body `{prompt}` → + `hub.dispatch("beforePrompt", …)` → `{ tail: string, blocks: }` (tail = fenced + blocks joined inside the delimiters; empty string when none). + - `POST …/before-compact` → dispatch, returns `{}` when all flushes settle (bounded by per- + provider timeouts). + - `GET /api/sessions/:id/context-trace?limit=` → trace store read. + - (afterTurn/shutdown have NO endpoint — gateway-internal, next step.) +2. `session-manager.ts`: in the existing agent-lifecycle event handling, on `turn_end` (or + `agent_end` when turns aren't granular) fire-and-forget + `hub.dispatch("afterTurn", {…, turn:{index}})`; on session archive/stop path dispatch + `sessionShutdown`. Locate by `rg "turn_end|handleAgentLifecycle" src/server/agent/session-manager.ts` + — add ONLY the dispatch calls. +3. `provider-bridge-extension.ts`: `generateProviderBridgeExtension(sessionId): string | + undefined` — returns a written tmp file path (mirror `writeToolGuardExtension`'s file + handling); generated source subscribes `before_agent_start` and `session_before_compact` + per the design decisions above. Generate ONLY when at least one enabled provider declares + `beforePrompt` or `beforeCompact` (registry query) — zero overhead otherwise. +4. `session-setup.ts::resolveToolActivation`: after the tool-guard block (`:622-629` pattern), + call the generator and push `--extension ` when defined. + +**Tests (author first).** +- `tests/provider-bridge-extension.test.ts`: codegen snapshot (delimiters present; gateway URL + read; abort timeout present); not generated when no provider wants the hooks. +- API E2E `tests/e2e/provider-turn-hooks.spec.ts` (provider-demo fixture): drive a turn via + the mock agent → fixture log contains `beforePrompt` then `afterTurn`; `GET + …/context-trace` lists both dispatches with timing; kill switch — disable provider, next + turn logs nothing; before-prompt endpoint with a hanging provider responds within budget + with empty tail (RED until timeout handling correct). +- Idempotent-tail unit: applying the tail twice yields one delimited region. + +**Acceptance.** Per-turn injection visible in the mock agent's received system prompt +(in-process mock exposes it — verify via the bridge spy or transcript); a turn with the +provider down completes normally; trace endpoint paginates. + +**Non-goals.** Tool-call hooks (G6.2); compaction-content mutation; UI. + +--- + +### G1.5 Market UI: provider listing + activation toggle + +**Outcome.** The Marketplace pack-detail UI lists a pack's providers (id, kind, hooks, budget, +origin) with working per-provider enable/disable that takes effect synchronously. + +**Owned files.** The marketplace UI module(s) — locate with +`rg -l "pack-activation|packActivation" src/app src/ui` — plus the catalogue REST already +extended in G1.1. + +**Steps.** Mirror exactly how entrypoint toggles render and persist today (same component, +new section "Providers"); show `kind` and the hook list as chips; disabled state persists via +the existing `PUT /api/marketplace/pack-activation`. + +**Tests.** Browser E2E `tests/e2e/ui/marketplace-providers.spec.ts` (model: +`tests/marketplace-active-project.spec.ts`): provider row visible for the fixture pack; toggle +off → persists across reload → (combined with G1.3's API assertion) new sessions skip it. + +**Non-goals.** Provider settings forms (each pack's panel owns its config UI); trace UI. + +--- + +### G1.6 `session-memory` pack + +**Outcome.** Built-in, default-on memory: before each turn the agent receives bounded +"Relevant past work" blocks recalled from the project's existing search index; new sessions +recall against the goal/task spec. Zero external dependencies. This is the platform's flagship +litmus. + +**Owned files (new).** `market-packs/session-memory/` (pack.yaml schema 2; +`providers/recall.yaml` hooks `[sessionSetup, beforePrompt]`, budget `{1200, 800}`; +`src/provider.ts` → built `lib/provider.mjs`); entries in `scripts/build-market-packs.mjs` +(`PACKS` const, line ~91) and `scripts/copy-builtin-packs.mjs` (`FIRST_PARTY_PACKS`, line 23). + +**Provider behavior (spec — implement exactly):** +1. `beforePrompt(ctx)`: skip when `!ctx.prompt` or prompt < 8 chars. Query + `GET {ctx.gateway.baseUrl}/api/search?q=&projectId= + &type=all&limit=8` with bearer `ctx.gateway.token`, 600ms fetch timeout. +2. Filter results: drop rows where `sessionId === ctx.sessionId` (never recall the ongoing + conversation); drop score below a floor (tune with the fixture; start 0.05); keep top K=3. +3. One block per result: + `{ id: "session-memory:"+result.id, title: "Relevant past work", authority: "memory", + priority: 5, reason: "BM25 recall for current prompt", + content: "[] (session <sessionId>, <ISO date>)\n<snippet with <b> tags stripped>" }`. +4. `sessionSetup(ctx)`: same flow keyed on the goal/task spec text when present (the Hub + passes it as `ctx.prompt` for sessionSetup — G1.3 wired `plan.initialPromptText`; extend + to goal spec if available on the plan). +5. Config (in `providers/recall.yaml` `config`): `k` (default 3), `minScore`, + `includeArchived` (default true), `sources` (default all). Read from `ctx.config`. + +**Tests (author first).** +- Unit `tests/session-memory-provider.test.ts` (drive the BUILT `lib/provider.mjs` directly + with a stubbed `fetch` injected via ctx or global): current-session rows filtered; K cap; + score floor; snippet `<b>` stripping; fetch timeout ⇒ returns `{blocks: []}` (never throws). +- API E2E `tests/e2e/session-memory-recall.spec.ts`: seed project with one session containing + a distinctive token (drive the mock agent to emit it, let the indexer flush); create a + second session and send a prompt containing that token; assert the Dynamic Context tail + delivered to the agent (or the trace) contains a `session-memory:` block referencing the + first session; toggle the pack off ⇒ no block. +- Browser E2E: rely on G1.5's toggle spec + extend it to assert this pack's row exists + (it ships in the built-in band, so it's present by default). + +**Acceptance.** Recall works on a fresh dev install with no configuration; total added +latency per turn ≤ the 800ms provider budget (pinned by the timeout test); disabling the pack +returns the system to byte-identical prompts (assert via prompt-sections diff in the E2E). + +**Non-goals.** Embeddings/semantic ranking; summarization artifacts; cross-project recall; +panel UI. + +--- + +## G2 — Hindsight pack (external-URL mode) + +> Implementer note: verify the real Hindsight REST paths against the targeted Hindsight +> release BEFORE coding the client; the contract below is Bobbit's internal client interface — +> the stub mirrors THIS contract, and `hindsight-client` adapts it to upstream paths. + +### G2.1 REST client + stub-server harness *(parallel with all of G1)* + +**Outcome.** `market-packs/hindsight/src/hindsight-client.ts` (built to +`lib/hindsight-client.mjs`) + an in-process stub server usable by every later test. + +**Client contract.** +```ts +export interface HindsightClient { + health(): Promise<{ok: boolean}>; + recall(bank: string, query: string, opts?: {maxTokens?: number}): Promise<{memories: {text: string; score?: number; id?: string}[]}>; + retain(bank: string, content: string, opts?: {tags?: Record<string,string>; sync?: boolean}): Promise<void>; + reflect(bank: string, prompt: string): Promise<{text: string}>; + listBanks(): Promise<{banks: string[]}>; +} +export function createClient(cfg: {baseUrl: string; apiKey?: string; timeoutMs?: number}): HindsightClient; +``` +Every method: AbortController timeout (default 1500ms), throws typed `HindsightError` +(`kind: "timeout"|"http"|"network"`, status?). + +**Stub** `tests/e2e/hindsight-stub.mjs`: `startHindsightStub({port?: 0})` → +`{url, calls: RecordedCall[], setHealthy(bool), seedMemories(bank, mem[]), close()}` — +an `http.createServer` with canned JSON for the five operations, recording every call +(method, path, bank, body) for assertions. + +**Tests.** `tests/hindsight-client.test.ts` against the stub: round-trips; timeout ⇒ +`HindsightError{kind:"timeout"}` within budget; 500 ⇒ http error; auth header sent when +apiKey set. + +### G2.2 Pack core: provider + routes + config + bank scoping + +**Outcome.** `market-packs/hindsight/` with provider (`providers/memory.yaml`, hooks: all +five), pack routes (`status, recall, retain, reflect, banks, config`), config persistence in +the pack store, and scope-aware banks — fully working against an external URL. + +**Spec highlights (implement exactly; full shapes in extension-platform.md §11):** +- Bank ids: `bobbit-proj-<projectId>` / `bobbit-global`; `beforePrompt` recalls BOTH in + parallel (`Promise.allSettled`, one shared deadline = provider `timeoutMs`), merges by + score, caps to budget. `sessionSetup` recalls vs goal/task spec. +- `afterTurn`: build a compact turn summary (user text + final assistant text, capped 2000 + chars) → `retain(projectBank, …, {tags:{sessionId, goalId, roleName}})`, **async**; on error + push `{content, tags, ts}` onto a retry queue in the pack store (`host.store` from routes / + `ctx.store` from provider — same packId namespace), retry queue head on every later + afterTurn; cap queue at 100 (drop oldest, count surfaced via `status` route). +- `beforeCompact`: `retain(…, {sync: true})` of the about-to-be-lost span summary. +- `sessionShutdown`: drain queue best-effort (one pass). +- Config route persists `{mode: "external"|"managed", externalUrl, apiKeyRef, autoRecall, + autoRetain, recallBudget}` in the pack store keyed per scope; provider reads it via + `ctx.config` (G1.1 loader merges store-config over yaml defaults — add that merge here if + G1.1 shipped static config only; keep the merge in the loader, not the provider). +- Dormancy: if neither a healthy runtime (G3) nor `externalUrl` is configured, every hook + returns immediately (`{blocks: []}`) — the pack is installed-but-dormant. + +**Tests.** Unit (bank derivation incl. global fan-out; retain-queue retry incl. cap; +dormancy). API E2E `tests/e2e/hindsight-external.spec.ts` with the stub: configure +externalUrl via the config route → sessionSetup+beforePrompt blocks appear (trace/prompt +sections), turn_end produces a retain with correct bank+tags; `setHealthy(false)` ⇒ session +unaffected + diagnostic recorded + status route reports unhealthy; stub recovers ⇒ queued +retain flushes. + +### G2.3 Tools + panel + entrypoints + +**Outcome.** `hindsight_recall` / `hindsight_reflect` / `hindsight_retain` agent tools +(tool group `tools/hindsight/`, `provider: {type: bobbit-extension, extension: extension.ts}` +— copy the `defaults/tools/shell/` + tool-guard auth pattern; tools POST the pack's routes), +each accepting `bank: "current"|"global"|"all"`; native panel +(`panels/hindsight-memory.yaml` + built `lib/HindsightPanel.js`) with status card, mode/URL +settings, memory search, recent retains + queue counter; two entrypoints (command palette + +deep link) — copy `market-packs/pr-walkthrough` shapes. + +**Tests.** API E2E: `hindsight_recall` invoked by the mock agent round-trips through the +route to the stub with the resolved bank; per-project pack disable ⇒ tools absent from the +session's tool list. Browser E2E `tests/e2e/ui/hindsight-panel.spec.ts`: open panel via +command palette; configure external URL; status flips to connected (stub); search renders +seeded memories; settings persist across reload. + +--- + +## G3 — Managed runtimes + +### G3.1 Runtime manifest + secrets/env/ports (pure, no Docker) + +**Outcome.** `runtimes/<id>.yaml` parsing/validation (`RuntimeContribution`: compose path +containment-checked, services, healthcheck, ports `host: auto`, volumes, secrets +`generate|prompt`, `startPolicy: on-enable|on-demand`), loaded via the contributions path +(G1.1 pattern, `contents.runtimes`); secret generation (`crypto.randomBytes(24).toString +("base64url")`, stored via `SecretsStore` under `runtime.<pack>.<id>.<NAME>`, idempotent); +`.env` rendering (0600) under `~/.bobbit/state/pack-runtimes/<pack>/`; host-port allocation +(bind `net.Server` to 0, record, persist in a state JSON, re-validate on load). All pure + +unit-tested; nothing executes Docker. + +**Tests.** `tests/runtime-manifest.test.ts` (validation incl. compose-path escape rejection); +`tests/runtime-secrets-env.test.ts` (idempotent generation; env file mode 0600; `${secret:X}` +interpolation); `tests/runtime-ports.test.ts` (allocation, persistence, revalidation when the +recorded port is taken). + +### G3.2 PackRuntimeSupervisor + REST (Docker mocked) + +**Outcome.** `src/server/runtimes/pack-runtime-supervisor.ts`: +`ensureRuntime(packName, id)` (idempotent, in-flight dedupe — copy `sandbox-manager.ts`), +`start/stop/restart/status/logs`; compose invocation +`docker compose -p bobbit-pack-<pack> -f <pack>/<compose> --env-file <env> up -d` via the +`project-sandbox.ts` execFile discipline (DOCKER_BIN, MSYS env); health = poll declared HTTP +path until `startupTimeoutMs`; status machine +`docker-unavailable|stopped|starting|running|unhealthy`. REST: `GET /api/pack-runtimes`, +`POST /api/pack-runtimes/:id/{start|stop|restart}`, `GET /api/pack-runtimes/:id/logs?tail=`. +Docker binary injectable (constructor `execFileImpl`) so all tests mock it. + +**Tests.** `tests/pack-runtime-supervisor.test.ts`: mocked exec walks +stopped→starting→running on health 200; health timeout ⇒ unhealthy; missing docker ⇒ +docker-unavailable (exec ENOENT); concurrent `ensureRuntime` ⇒ one compose invocation; stop +issues `compose stop` (not down). API E2E for the three REST routes against the mocked +supervisor. + +### G3.3 Wire-up: lifecycle, Hindsight managed mode, consent, manual-integration + +**Outcome.** Enable-with-runtime = zero-step Hindsight: pack enable (activation) with +`startPolicy: on-enable` starts the runtime **only from an explicit user action** (the enable +click handler — pinned: no auto-start on boot/install); provider `ctx.runtime` injected +(`{baseUrl: http://127.0.0.1:<port>, headers, status}`) and `managed` mode preferred when +healthy; disable ⇒ `compose stop`; `uninstallPack` (pre-delete seam at the function head, +`marketplace-install.ts::uninstallPack`) ⇒ `compose down`, volumes preserved; explicit purge +flag ⇒ `down -v` + state-dir removal. Hindsight pack gains `runtimes/hindsight.yaml` + +`runtime/compose.yaml` (digest-pinned images; default memory-formation model preconfigured +per design §11). Enable card shows the capability summary line for runtimes (images, ports, +volumes) + the memory disclosure (design §8.4). Manual-integration test +(`tests/manual-integration/hindsight-runtime.test.ts`): real compose up → health → +recall/retain round-trip → disable stops → volume survives `updatePack`. + +**Tests.** Unit: keep-vs-purge logic; no-auto-start pin (boot with enabled pack + stopped +runtime ⇒ supervisor not invoked). API E2E (mocked docker): enable → start invoked; disable → +stop; uninstall default keeps volumes (no `-v` in argv), purge adds it. Browser E2E: runtime +status card states + logs view (mocked), enable-card disclosure text present. + +--- + +## G4 — Memory depth *(contract level — re-verify anchors; substrate = G1/G2/G3)* + +- **G4.1 beforeCompact ordering.** Pin that the bridge's `session_before_compact` POST + completes (or times out) BEFORE compaction proceeds, and Hindsight's sync retain lands + before the compacted span is dropped. Test: stub asserts retain arrives before the mock + agent emits `compaction_end` (ordering assertion via recorded timestamps). +- **G4.2 Memory browser v2.** Panel: list/filter (session/goal tags) /delete memories via new + pack routes wrapping `listMemories`/`delete`; retain-queue and operations views. Browser + E2E: browse/filter/delete survives reload. +- **G4.3 Dual-provider compose.** When both memory providers contribute: stable order + (hindsight priority > session-memory), near-duplicate suppression (normalized-text overlap + ≥0.8 ⇒ keep higher-priority block), cost line (memory-formation token estimate) in the + panel. Unit tests on the dedupe; E2E with both enabled asserting one merged Dynamic Context + section under the global budget. + +## G5 — MCP as pack content *(single goal)* + +Lift is already manifest-side (G1.1 accepts `contents.mcp` at schema 2). Add: `mcp/<name>.yaml` +loader (`{name, command|url, args?, env?}` → `McpServerConfig`), a **pack band** in +`mcp-manager.ts::discoverServers` (`:125-170`) inserted at the LOWEST priority position +(packs must never shadow user/project MCP configs — priority below "custom directories"); +provenance (`originPackName`) carried into the tool-info summaries; uninstall ⇒ configs gone +on next discovery (registry-driven, no persistence). Consent: capability summary line lists +each server's command/URL (G3.3 card machinery; if G3.3 hasn't merged, add the line to the +install response only). Tests: unit (loader + precedence: same server name in user config and +pack ⇒ user wins); API E2E (pack-shipped stdio MCP fixture server reachable via existing +`mcp_<server>` meta-tool; uninstall removes); browser E2E (provenance badge in tools UI). + +## G6 — Hooks + pi-extensions + +- **G6.1 hooks entity + dispatcher.** `hooks/<id>.yaml` + (`{event: SessionStart|UserPromptSubmit|Stop|PreCompact|SessionEnd|PreToolUse|PostToolUse, + command, timeoutMs?, matcher?}`) + Claude-layout `hooks/hooks.json` accepted as an + alternative source (one listName per file). Command dispatcher: `execFile` with JSON event + on stdin, capture stdout JSON; map onto the Hub per design §5.5 (SessionStart/ + UserPromptSubmit stdout `additionalContext` → ContextBlock). Non-tool events only in this + goal. Tests: mapping-table unit; E2E fixture pack hook injects context visible in the tail. +- **G6.2 PreToolUse/PostToolUse.** Generalize the tool-guard long-poll endpoint so hook + verdicts (`{block, reason}` / input mutation) ride the SAME generated guard extension + (extend `generateToolGuardExtension` inputs rather than adding a second tool_call + subscriber). Tests: fixture hook blocks a named tool (E2E); allow-path latency unchanged + (no hook ⇒ no extra HTTP round-trip — pin by codegen snapshot). +- **G6.3 pi-extensions entity + trust gate.** `pi-extensions/<name>.yaml` → module path; + appended as `--extension` in `resolveToolActivation` ONLY when the pack has a recorded + acknowledgment. Trust store: `~/.bobbit/state/pack-trust.json` + (`{packName, scope, acknowledgedAt, capabilities}`); default-deny; Market UI acknowledgment + flow + revoke. Hooks (G6.1) consume the same gate. Tests: default-deny pin (un-acked pack + contributes nothing); ack → extension in spawn args (inspect `plan.bridgeOptions.args` via + a unit on the pipeline step); browser E2E ack + revoke across reload. + +## G7 — Claude plugin adapter + +- **G7.1 Detect + browse.** Source sync detects `.claude-plugin/marketplace.json`; source row + gains `format`; browse lists plugins (name/description/version from each `plugin.json`). +- **G7.2 Convert + skipped report.** `claude-plugin-adapter.ts` invoked inside `installPack` + between staging copy and `writeMeta`/rename (seam at `marketplace-install.ts:514-529`): + conversion table per design §9 (`commands/`→skills commands-flat, `skills/`→skills, + `agents/*.md`→roles, `hooks/hooks.json`→hooks, `.mcp.json`→mcp, `${CLAUDE_PLUGIN_ROOT}` + rewrite, synthesized `pack.yaml schema: 2`); returns `skipped: [{feature, reason}]` + embedded in the install response and `.pack-meta.yaml` (`sourceFormat: claude-plugin`). + Tests: fixture plugin repo → converted pack resolves skills/roles through the normal + cascade; every unsupported feature in the fixture appears in `skipped`; nothing silently + dropped (fixture includes a statusline + an unsupported MCP transport). +- **G7.3 UI + real-marketplace E2E.** Source-format badge; skipped report rendered + post-install; browser E2E installs a pinned-commit public Claude marketplace plugin and its + slash command appears in the composer. + +## G8 — Capabilities + selectors + +- **G8.1 Registry.** `provides`/`requires` (already parsed in G1.1) → capability index in the + contribution registry (a capability implementation = a named pack route or provider member + declared `capability: <name>` in its yaml); `ctx.capabilities.call(name, input)` available + to providers and pack routes — host resolves by pack precedence, dispatches on the worker + tier with a 5s default timeout, records to the trace. Install-time check: missing required + capability ⇒ warning in install response + Market UI hint. Tests: precedence; missing-dep + warning; call round-trip provider→capability across two fixture packs; timeout isolation. +- **G8.2 Decision points + proposals + policy.** Hub gains `beforeGoalCreate` / + `beforeSessionSpawn` dispatch points wired into goal-creation and session-spawn paths + (locate via `rg "createGoal|createSession" src/server` and insert PRE-validation, never + post). Selector providers (`kind: selector`) receive typed summaries (roles via + `RoleManager.listRoles`, workflows via `WorkflowManager`, skills catalog, MCP servers, + models via `model-registry`) and return the `SelectorProposal` shape (design §12) — + schema-validated; invalid ⇒ dropped + diagnostic. Approval policy module + (`selector-policy.ts`, pure, fully unit-tested): auto-apply iff pre-session ∧ no tool-grant + expansion vs the default role ∧ model available ∧ persona patch additive-and-bounded + (≤1200 chars, append-only) ∧ confidence ≥ 0.75; else `requiresApproval`. Application goes + through existing stores ONLY (role assignment, workflow selection APIs). Tests: policy + truth-table unit (every clause has a RED case); E2E fixture selector auto-applies a safe + role and flags an unsafe one. +- **G8.3 Proposal UI + fixture pack.** Goal-creation UI surfaces the proposal + (accept/decline, reason shown); decline ⇒ deterministic defaults. Browser E2E both paths. + +## G9 — Workflow templates + flagship packs + +- **G9.1 workflows entity.** `workflows/<id>.yaml` (same gate schema `workflow-validator` + validates today) loaded as TEMPLATES into the registry; instantiation API + (`POST /api/projects/:id/workflows/instantiate {packName, templateId}`) copies through + `WorkflowManager.createWorkflow` (validation reused; collision ⇒ suffix `-2`); goal-creation + UI offers templates alongside project workflows. Project.yaml remains the only source of + truth; goal snapshotting untouched (pin: instantiate → mutate template pack → existing goal + unchanged). Tests: loader/validation reuse; instantiation E2E; snapshot-immunity pin. +- **G9.2 model-selector pack.** `provides: [model-selector]`; route implementing + `{task, spec?, candidates, constraints?} → {model, thinkingLevel?, reason}` with a + deterministic rule table first (cost/capability tiers from `model-registry` data) and an + optional LLM refinement behind config (strict JSON, timeout, fallback to the table). Tests: + rule-table unit; capability call E2E. +- **G9.3 multi-model-delivery pack.** Roles (`planner` frontier+xhigh, `implementer` cheap + precise, `qa`, `reviewer` — each promptTemplate carries the initial-spec handoff stanza), + workflow template plan→implement→qa→review with a ralph-loop verify step (re-run until + green, bounded iterations), `requires: [model-selector]`. Tests: instantiation E2E; goal + snapshot carries per-gate roles with pinned models; selector swap path (capability mocked). + +--- + +## 2. Verification matrix (run per goal; full sweep before each parent-goal close) + +| Check | Command | +|---|---| +| Types | `npm run check` | +| Unit phase | `npm run test:unit` | +| E2E phase | `npm run test:e2e` | +| Real Docker (G3.3, G4 managed paths only) | `npm run test:manual` | +| Pack builds | `node scripts/build-market-packs.mjs && node scripts/copy-builtin-packs.mjs` (then re-run e2e) | +| No quarantines added | `rg "test.skip|fixme" tests/ --new-only` vs base | + +Parent-goal close-out additionally requires: the phase's acceptance criteria in +[extension-platform.md §13](extension-platform.md) all demonstrably true, and a short +goal-report noting any anchor drift corrected (so later goals inherit fresh anchors). From 261d5b8f39921692f5b2d88e9cbf4d864a28398f Mon Sep 17 00:00:00 2001 From: Artur Jonkisz <ajonkisz@gmail.com> Date: Wed, 10 Jun 2026 21:26:56 +0100 Subject: [PATCH 04/18] Added docs/design/token-cost-efficiency.md --- docs/design/token-cost-efficiency.md | 615 +++++++++++++++++++++++++++ 1 file changed, 615 insertions(+) create mode 100644 docs/design/token-cost-efficiency.md diff --git a/docs/design/token-cost-efficiency.md b/docs/design/token-cost-efficiency.md new file mode 100644 index 000000000..74f090792 --- /dev/null +++ b/docs/design/token-cost-efficiency.md @@ -0,0 +1,615 @@ +# Token & Cost Efficiency — audit, findings, and remediation plan + +Status: **investigation complete, remediation not started** · Audited: 2026-06-10 against +master @ `4e7fce1f` and live state on the primary dev machine. +Companion docs: [extension-platform-implementation-plan.md](extension-platform-implementation-plan.md) +(several goals here ride its pack surfaces), [comms-stack/04-current-state-and-backlog.md](comms-stack/04-current-state-and-backlog.md) +(overlapping retry findings). + +**The question this answers:** Bobbit *feels* more expensive per unit of work than Claude Code +or hermes-agent. Is it, why, and what do we change? + +**The one-paragraph answer:** Prompt caching is working (only 0.008% of input tokens were +uncached across 246 sessions), so the spend is not a caching bug. The spend is +**resident-context size × API-request count × model price**: every tool-call round re-reads the +session's full context at cache-read price, Bobbit's resident context is large (18k–55k tokens +of system prompt before any conversation), tool results accumulate unbounded (no spill/truncation +tier), orchestration multiplies turns (idle nudges, verifier gates, repeated commands), and +every agent — lead, worker, reviewer — runs the same frontier model. Claude Code and Hermes are +cheaper because they cap tool output, spill large results to disk, keep resident prompts lean, +and (Hermes) summarize with a cheap auxiliary model. All of those are adoptable. The plan below +is instrumentation-first: we build the measurement/benchmark harness *before* the +behavior-affecting cuts, so every change is validated on data, not vibes. + +--- + +## §0 Universal rules for executing this plan + +Same rules as the extension-platform plan; restated because goals here will be handed off +independently: + +1. **Test-first.** Every sub-goal lists pinning tests; write them RED before the fix where the + spec says so. No flaky tests — a flaky test is a real bug. +2. **Locate code by symbol name, not line number.** Line anchors below were verified on + master @ `4e7fce1f` and will drift. If a symbol is missing, STOP and re-derive from the + pattern files, don't guess. +3. **Gates.** `npm run check` + `npm run test:unit` for everything; `npm run test:e2e` for + server changes; `npm run test:manual` for session-lifecycle changes. Browser E2E for every + user-facing surface (pattern: `tests/e2e/ui/settings.spec.ts`). +4. **Measurement gate (specific to this plan).** Goals marked **[BENCH-GATED]** change agent + behavior (prompts, models, truncation). They may not merge on code review alone: run the + benchmark suite (CE-G0.3) before and after on the same task set and attach the comparison to + the PR. A regression in task success rate blocks the merge regardless of cost savings. +5. **Environment caveat.** Several SDK-layer findings below were verified against the *stale* + `node_modules` (`@mariozechner/pi-* 0.67.5`) while `package.json` declares + `@earendil-works/pi-* 0.77.0` and the fork's latest is **0.79.1**. CE-G1.0 re-verifies every + SDK claim against the version actually adopted before any SDK-dependent goal starts. + +--- + +## §1 Where the money actually goes (empirical) + +All numbers measured 2026-06-10 from this machine. Reproduction recipes in §7. The owner notes +the codebase moves fast — re-measure before relying on any specific number; the *shapes* are the +durable findings. + +### 1.1 Aggregate (`.bobbit/state/session-costs.json`, 246 sessions) + +| Metric | Value | +|---|---| +| Total recorded spend | **$2,768.90** | +| Uncached input tokens | 284,670 (**0.008%** of input) | +| Cache-read tokens | **3,570,141,959** (3.57B) | +| Cache-write tokens | 125,505,474 | +| Output tokens | 8,097,011 | +| Median session cost | $4.37 | +| Top-10 sessions' share | **$1,420.50 (51%)** | + +Verdict: **caching works**. The bill is dominated by cache *reads* — i.e., how big the resident +context is and how many API requests re-read it — plus output. Cache-read is ~10× cheaper than +fresh input, and we still spent ~$2.7k. Without working caching this history would have cost +five figures; with it, the levers left are context size, request count, and model price. + +### 1.2 The outliers + +| Session | Cost | cacheRead | cacheWrite | Output | What it is | +|---|---|---|---|---|---| +| `0dbda4f4` "Team Lead: Meg Abyte" | **$901.99** | 1,021M | 57.6M | 1.23M | team-lead orchestrator | +| `acce11b7` "Team Lead: Banksy" | $222.26 | 261M | 13.7M | 229k | team-lead orchestrator | +| `llm-review-69d1e3e3` | $46.73 | 77M | 1.0M | 77k | workflow gate verifier | +| `llm-review-0cd4b16b` | $41.14 | 63M | 1.1M | 90k | workflow gate verifier | +| (next 6, all llm-review or team) | $29–40 each | 37–64M | ~1M | 80–103k | — | + +Two shapes account for the tail: + +- **Long-lived team-leads.** The 35KB team-lead role + goal + AGENTS.md + tools is resident for + the session's whole life; every coordination turn (nudge, status steer, gate event) re-reads + it. 1.02B cache-read tokens ≈ thousands of API requests × a six-figure-token context. +- **`llm-review` verifiers** (`verification-harness.ts::runLlmReviewStep`, spawn at + `verification-harness.ts:2542`). One-shot reviewers that re-ingest role + AGENTS.md + goal + spec + the change under review, do tens of tool-call rounds each (each round = full + context re-read), with up to 3 bounded retries — at $29–47 a pop in the tail, these run on + every gated workflow step. + +### 1.3 Resident prompt anatomy (`.bobbit/state/session-prompts/*-prompt.json`) + +Recent sessions in this repo (tokens, from the persisted inspector snapshots): + +| Section | Recent typical | Worst observed | Source | +|---|---|---|---| +| Role | 6,254 (code-reviewer) | **20,180** (older review session) | `defaults/roles/*.yaml` | +| Project AGENTS.md | **9,021** | 17,309 | `readAllAgentFiles()` cascade — *project-dependent*: the 9k example is another project's hand-written playbook; this repo's AGENTS.md is ~1.3k tokens | +| Tools | 5,600 | 6,565 | `toolManager.getToolDocsForPrompt()` | +| System Prompt | 3,855 | 3,855 | `.bobbit/config/system-prompt.md` (15.5KB) overriding `defaults/system-prompt.md` (18.8KB) | +| Goal | 650–2,100 | 7,856 | user-authored goal spec | +| Available Skills | 150 | ~2,000 | budget-capped (`SKILLS_CATALOG_BUDGET` 16KB) | +| Working Directory | 206 | 206 | template | +| **Total resident** | **~18k–26k** | **~55k** | before any conversation history | + +Role files on disk: `team-lead.yaml` **35,491 chars** (~8.9k tokens — 40% of the whole +`defaults/roles/` corpus), `qa-tester` 9.9KB, `docs-writer` 6.7KB, `code-reviewer` 5.6KB. + +Key ratio: **Role + AGENTS.md = 60–68% of resident prompt tokens** in the measured sessions. +The base system prompt everyone assumes is the problem is only ~7%. + +### 1.4 Transcript anatomy (`~/.bobbit/agent/sessions/`, 971MB total; `.bobbit/state` 258MB) + +Three representative sessions: + +| Session | Size | Assistant msgs | Shape | +|---|---|---|---| +| `audit-subg-…` (goal work) | 18.0MB | 1,771 | bash = **56%** of 1,727 tool calls; `read` 233 calls / 747KB returned (max single read 41KB); grep up to 48KB per result | +| `nested-goa-…` (team) | 14.5MB | 2,644 | **30+ idle-agent nudge turns**; `git push … \| tail -3` executed **17× identically**; `npm run check` 9×; team_spawn/task churn | +| `session-_pool-…` (images) | 7.2MB | 19 | 3 `generate_image` results = **5.1MB of base64 (73% of the transcript)** | + +These three patterns — uncapped tool results accumulating in history, coordination turns that +each re-read full context, and multi-MB base64 images living in context — are the per-turn +multiplier on top of the resident-prompt floor. + +### 1.5 The SDK version situation + +| Layer | Version | Note | +|---|---|---| +| `package.json` / lockfile | `@earendil-works/pi-* `**0.77.0** | the fork is already adopted on paper | +| `node_modules` (this machine) | `@mariozechner/pi-*` **0.67.5** | stale install — `src/` still imports `@mariozechner/*` paths in places (`rg "@mariozechner/pi" src/`) | +| Fork latest | **0.79.1** (2026-06-09) | github.com/earendil-works/pi | +| Upstream latest | 0.73.1 (2026-05-07) | effectively superseded by the fork | + +Fork releases since 0.67 ship several fixes we currently pay for not having: + +- **0.76.0** — `excludeFromContext` on the bash RPC command (keep command output out of model + context); `retry.provider.maxRetries` honored with **SDK retries defaulting to 0** (this is + the fix for the double-retry stack — Bobbit's ~1s retry racing the SDK's hidden 2s retry — + catalogued as F5 in the comms-stack audit); better context-overflow detection. +- **0.77.0** — `--exclude-tools` (drop unused built-ins per session); session disposal aborts + in-flight compaction/retry/bash; `streamingBehavior` on input events. +- **0.78.1** — `ctx.getSystemPromptOptions()` for extensions. +- **0.79.0** — **prompt cache-hit rate surfaced** (`CH` in pi's footer — the measurement we + want, already computed SDK-side); compaction summarization prompt fix. +- **0.79.1** — Claude Fable 5 + adaptive thinking/`xhigh`. + +--- + +## §2 The cost model (how to think about every change) + +Per API request: `cost ≈ resident_tokens × p_cacheRead + new_tokens × p_cacheWrite + output_tokens × p_output` + +Per session: `cost ≈ Σ_turns Σ_toolRounds(turn) [above]` — **each tool-call round inside a turn +is its own API request that re-reads everything**. + +So there are exactly four levers, and every goal in §6 maps to one: + +| Lever | Goals | +|---|---| +| **L1 — shrink resident context** (prompt diet, tool-result budgets, image-by-reference, compaction) | CE-G2, CE-G4, CE-G6 | +| **L2 — cut request count** (turn economy: nudges, repeated commands, verifier scoping, discovery efficiency) | CE-G3, CE-G7 | +| **L3 — cheaper price per token** (model/thinking tiering per role/stage) | CE-G5 | +| **L4 — stop paying twice** (retry dedup, SDK upgrade hygiene) | CE-G1 | + +Worked example: an llm-review session with 63M cache-read tokens at ~150k resident context ≈ +~420 API requests. Halving the resident context (L1) *and* running it on a model with ⅓ the +price (L3) is a ~6× cost reduction on that session class with zero reduction in rounds. + +--- + +## §3 Findings (ranked, with evidence and difficulty) + +Severity = estimated share of current spend addressable. Difficulty = engineering risk +including behavior change. Anchors verified on master @ `4e7fce1f`. + +### F1 — No tool-result budget tier (HIGH, difficulty M) +Tool results enter history verbatim and stay there for the session's life; every later request +re-reads them. Largest observed: 48KB grep results, 41KB single reads, 13–51KB bash logs. +`truncateLargeToolContent()` exists (`src/server/agent/truncate-large-content.ts`, referenced at +`session-manager.ts:45`) but is applied opportunistically, not as a systematic policy. Neither +per-result spill-to-disk nor a per-turn aggregate budget exists. Compare Hermes' three-layer +system (§5). *Caveat: verify against pi 0.79.1 — fork may have added caps (CE-G1.0).* + +### F2 — Resident prompt floor is 18k–55k tokens, dominated by Role + AGENTS.md (HIGH, difficulty M–L, behavior-affecting) +§1.3. `team-lead.yaml` alone is 8.9k tokens of always-resident prose, much of it recipe-style +content that could live in skills/docs loaded on demand. The AGENTS.md cascade +(`system-prompt.ts::readAllAgentFiles`, recursive `@ref` resolution, depth 5) has **no token +budget** — a verbose project playbook silently costs its tokens × every request of every +session in that project. + +### F3 — Orchestration turn multiplication (HIGH for team sessions, difficulty M) +The two most expensive sessions ever are team-leads (§1.2). Contributors: idle-agent nudges +(30+ observed in one session; templates in `team-manager.ts` ~1165–1254, debounce exists — +`WORKER_IDLE_NUDGE_DEBOUNCE_MS = 5s`, exponential backoff for lead idle — but each nudge is +still a full turn at full resident context); worker status steers per task transition; gate +inspections (5 × ~13KB observed). The nudge *text* is small (~85 tokens); the **turn it forces +is not** — at a 100k-token resident context, every nudge turn costs ~$0.03–0.15 in cache reads +alone before the model does anything. + +### F4 — Uniform frontier model everywhere (HIGH, difficulty S–M) +Workers, verifiers, and assistants inherit the lead's/session default model and thinking level. +The override *mechanism* exists (`review-model-override.ts::applyReviewModelOverrides`, +`clampReviewThinking` in `verification-harness.ts`; `initialModel` pin in +`session-setup.ts::resolveBridgeOptions`) but there are no tiered **defaults**: nothing says +"llm-review verifiers default to a cheap model" or "coder workers run one tier below the lead". +The owner already wants multi-model pipelines (extension-platform P8/P9 model-selector); this +finding is the static-default version of that. + +### F5 — llm-review economics (MED-HIGH, difficulty M) +$29–47 per review at the tail (§1.2), and reviews run per gated workflow step with up to 3 +bounded retry attempts (`verification-harness.ts:~2550`). Each reviewer re-ingests the full +resident stack even when the reviewed artifact is a small diff. No diff-scoped context, no +cheap-model default, no cap on tool-call rounds. + +### F6 — Images as base64 in context (MED overall, EXTREME per-incident, difficulty S) +One `generate_image` result = 2.4–2.7MB of base64 in the transcript and in context; 73% of a +$10.75 session was image bytes. Results should be written to disk and returned by reference +(path + dimensions + thumbnail for the UI), never inlined into model context. + +### F7 — Repeated identical commands (MED, difficulty S) +`git push … | tail -3` ×17, `npm run check` ×9 in single sessions. Each repeat is a full +request round. No guard, hint, or memoization exists. + +### F8 — Stale SDK + double retry billing (MED, difficulty S) +§1.5. Running 0.67.5 forfeits `excludeFromContext`, `--exclude-tools`, retry control, and +cache-hit telemetry. The SDK's hidden default-on retry (~2s) underneath Bobbit's own auto-retry +(`session-manager.ts:2594–2656`) can double-bill failed turns — fixed by fork 0.76.0's +`retry.provider.maxRetries` honoring, **if we upgrade and set it**. + +### F9 — No per-turn cost instrumentation (BLOCKER for everything else, difficulty S–M) +`session-costs.json` (`cost-tracker.ts`, recorded from `message_end` usage at +`session-manager.ts:~3103`) is **cumulative per session only**. No per-turn deltas, no cache-hit +ratio surfaced in Bobbit's UI, no per-goal/per-project rollup, no compaction-cost attribution. +We cannot currently answer "what did this nudge/turn/verifier cost" without jq archaeology — +which is why this audit needed one. + +### F10 — Discovery inefficiency: bash is 54–56% of all tool calls (MED, difficulty S–M) +Agents shell out for grep/find/cat instead of using capped tools with offsets; whole-file reads +of 41KB; no "grep -l first, then targeted read with offset/limit" discipline in prompts or +skills. Each inefficient discovery step is both a bigger result (L1) and often an extra round +(L2). pi's built-ins do include `grep`/`find`/`read` — the gap is output caps + guidance, not +tool existence. *(pi's `edit` is also reported as whole-file-rewrite at 0.67.5 — re-verify on +0.79.1; if still true it inflates output tokens on large files.)* + +### F11 — Compaction is a cost black box (LOW-MED, difficulty S) +Compaction summarization calls aren't attributed in cost tracking, and Bobbit doesn't configure +a proactive policy or a cheap summarizer model (pi's `compaction-sidecar` defaults apply). + +### F12 — Storage bloat (cosmetic, difficulty S) +112MB of `session-prompts/` snapshots and 971MB of transcripts — not an API-cost issue, but +prompt snapshots duplicate identical AGENTS.md/role content per session; worth content-hashing +during G0 work since the same code is being touched. + +**Verified non-problems** (don't spend effort here): prompt caching (works, §1.1); per-turn +system-prompt mutation (none — written once at spawn, `--system-prompt` path, cache survives); +skills catalog (150 tokens, budget-capped); tool description budgets (pinned by +`tests/tool-description-budget.test.ts`); goal-tree serialization (only nesting stanzas ≤1.5k); +MCP tool docs (off-wire pointers already). + +--- + +## §4 What Claude Code and Hermes do that we don't + +From source study of `/Users/aj/Documents/dev/claude-code/src` and +`/Users/aj/Documents/dev/hermes-agent` (local checkouts, 2026-06-10): + +| Technique | Claude Code | Hermes | Bobbit today | Adopting goal | +|---|---|---|---|---| +| Stable cache prefix w/ explicit static/dynamic boundary | `SYSTEM_PROMPT_DYNAMIC_BOUNDARY`, `cacheBreak` flags, system-reminders injected user-side | system prompt memoized; ephemeral prompt injected at API-time, outside cache | ✅ already good (prefix stable) | — | +| Read caps + offsets | 25k-token read cap, 256KB pre-read stat check, offset/limit params | `read_file` unbounded but spills | partial (no enforced cap) | CE-G2.1 | +| **Tool-result spill-to-disk** | success/error only for write/edit | **3-layer budget: per-tool cap → >100KB results persisted to disk w/ 1.5KB preview + path → 200KB per-turn aggregate spill** | ❌ | CE-G2.1 | +| Deferred tool loading | `defer_loading` + ToolSearch when tool docs >10% of context | ❌ (67 tools inline) | partial (MCP docs off-wire) | CE-G1.2 (`--exclude-tools`) | +| **Cheap auxiliary model for summarization** | ❌ (manual /compact) | **auto-compress at 50% context via cheap model, 20% summary ratio, last-20-messages protected, tool-result pruning pre-pass** | ❌ | CE-G6.1 | +| Lazy skills | partial | `skill_list`/`skill_view` split — full skill body loaded only on use | ✅ catalog is names+descriptions | — | +| Sub-agent economics | minimal task prompt, summary-only return | parent context compressed before sub-agent call | ❌ full resident stack per sub-agent | CE-G3.3, CE-G5 | +| File-read dedup | `fileStateCache` (path+mtime) | — | ❌ | CE-G7.1 (guidance) / later | +| Image budgeting | downsampling + flat 1,600-token estimate | flat per-image budget | ❌ base64 inline | CE-G2.2 | +| Cost attribution | per-request workload headers, cost tracker | per-call usage accumulation | per-session cumulative only | CE-G0.1 | + +The headline gap is **Hermes' three-layer tool-result budget** and **auxiliary-model +compression** on the L1 lever, and Claude Code's **read caps/deferral** on L1/L2. Nothing +either tool does on caching is missing in Bobbit — confirming §1.1. + +--- + +## §5 Design positions (settled — do not re-litigate during execution) + +1. **Instrumentation before surgery.** CE-G0 lands first. Behavior-affecting goals are + BENCH-GATED (§0 rule 4). The owner explicitly wants data-driven before/after validation, + "not guessing". +2. **The benchmark harness is a product feature, not a dev script.** It ships as a first-class + Bobbit capability ("run this task suite, compare cost/outcome between two configurations"), + designed so the run-suite/report surfaces can later be packaged as an extension-platform + pack (panel + provider) — but the core runner lives in the server like the verification + harness does. This is the owner's "test/validation harness as an extension would be a + massive win". +3. **Adopt the pi fork at latest (0.79.x), don't patch blind.** `@earendil-works/pi` is our + patchable fork already declared in package.json; first move is *upgrade + re-verify*, and + only then patch the fork where gaps remain (tool-result caps, edit semantics) — upstreaming + into the fork rather than working around it in Bobbit when the fix is SDK-shaped. +4. **Spill, don't destroy.** Truncation policies always preserve the full artifact on disk with + a preview + path in context, so the agent can re-read on demand. Never silently drop + information (mirrors the "no silent caps" culture). +5. **Tiering is defaults, not hard-coding.** Model/thinking tiers live in role/project config + with sane defaults; the extension-platform model-selector (P8/P9) later makes them dynamic. + This plan must not conflict with that — it defines the *static* layer the selector overrides. +6. **Real measured numbers stay in this doc** with their date and machine context; re-measure + rather than trust them after substantial changes. + +--- + +## §6 Remediation plan — goals + +Hand-off format matches `extension-platform-implementation-plan.md`: each sub-goal is +independently mergeable, master-green, with tests and acceptance criteria. Lanes: G0 and G1 are +parallel day-one lanes; G2/G3 depend on G1.0's re-verification; G4/G5 are BENCH-GATED on G0.3. + +**Conflict hotspots:** `session-manager.ts` and `session-setup.ts` are also touched by the +comms-stack backlog (CS-R*) — coordinate merge order; `verification-harness.ts` is touched by +CE-G3.3 and CE-G5.2 (sequence them). + +--- + +### CE-G0 — Instrumentation & benchmark harness *(lane A — start immediately)* + +**Outcome:** every token and dollar is attributable (per turn, per session, per goal, per +project), visible in the UI, and comparable across runs of the same task. + +#### CE-G0.1 Per-turn usage ledger (S) +- On every `message_end` with usage (existing hook: `trackCostFromEvent`, + `session-manager.ts:~3103`), append a record to + `.bobbit/state/session-usage/<sessionId>.jsonl`: + `{ts, turnIndex, model, inputTokens, outputTokens, cacheReadTokens, cacheWriteTokens, costUsd, contextEstimate, trigger}` — + `trigger` distinguishes `user-prompt | steer | nudge | auto-retry | verification | compaction` + (plumb from the enqueue site; nudges/steers already flow through known call sites in + `team-manager.ts` / `session-manager.ts::enqueuePrompt`). +- Attribute compaction summarization usage to the session (closes F11's blind spot). +- REST: `GET /api/sessions/:id/usage` (paginated), `GET /api/projects/:id/usage/rollup` + (per-goal, per-role, per-trigger aggregates). +- **Tests:** unit — ledger append/rotation, trigger attribution table-driven; API E2E — rollup + math over a fixture ledger. Pin: cumulative ledger total equals `session-costs.json` entry + (±1 token) for a replayed fixture. + +#### CE-G0.2 Cost lens UI (M) +- Session view: per-turn cost strip (cost, cache-hit %, trigger badge), session total with + cacheR/cacheW/output breakdown. Goal/project view: rollup table (cost by role, by trigger + class — "coordination" vs "work" vs "verification"), top-N expensive sessions. +- Surface the SDK's cache-hit rate once on ≥0.79 (CE-G1.1) instead of recomputing. +- **Tests:** browser E2E (navigation, rendering from fixture ledger, persistence across + reload). UI-only → `test:unit` + the E2E. + +#### CE-G0.3 Benchmark harness (L) — the BENCH gate +- New server module (pattern: `verification-harness.ts`): a **bench suite** is a YAML file of + repeatable tasks — `{id, fixtureRepo (tests/fixtures or a pinned git ref), goalSpec, role, + successCheck (command or llm-review rubric), maxCost, maxTurns}`. +- `POST /api/bench/run {suiteId, label, overrides}` runs each task as a normal (sandboxed, + worktree-isolated) session, records outcome + the CE-G0.1 ledger, and produces a run report: + `{task, success, costUsd, turns, toolCalls, wallClock, residentPromptTokens}`. +- `GET /api/bench/compare?a=<runId>&b=<runId>` → per-task and aggregate deltas; this comparison + JSON (or its rendered table) is what BENCH-GATED PRs attach. +- CI never runs real-LLM benches (`tests/manual-integration/` is the only gate-exempt path); + unit/E2E cover the runner with a stubbed agent. +- Seed suite `bench/suites/core-dev.yaml`: (1) scoped bugfix in a fixture repo, (2) small + feature + tests, (3) code-review of a prepared diff, (4) a team goal with 2 workers — + deliberately covering worker, reviewer, and orchestration cost shapes. +- **Extension hook (deliberate):** the report store and compare endpoint are plain REST + files + so a future pack (extension-platform P1 surfaces: panel + entrypoint) can ship "Cost Lab" UI + without core changes. Record this seam in the code. +- **Tests:** unit — suite parsing, report math, compare; API E2E — stubbed-agent run end to + end; manual-integration — one real run of the smallest task. + +--- + +### CE-G1 — SDK adoption & config hygiene *(lane B — start immediately)* + +#### CE-G1.0 Upgrade to `@earendil-works/pi-*` latest 0.79.x and re-verify claims (M) +- `npm install` against latest fork; migrate any remaining `@mariozechner/pi*` imports in `src/` + (`rg "@mariozechner/pi" src/` — UI and server hits listed in the audit); fix breakages; + full gate run incl. `test:manual`. +- **Re-verify and document** (append a dated note to this doc's §1.5): does 0.79.x truncate + tool results? what are `read`/`grep`/`bash` output caps? is `edit` still whole-file-rewrite? + what does compaction default to? This report re-scopes CE-G2/G6/G7 before they start. +- Set `retry.provider.maxRetries` explicitly and align with Bobbit's auto-retry so exactly one + layer retries (cross-ref comms-stack F5/CS backlog — same fix, coordinate). +- **Tests:** existing suites are the regression net; add a pinning test that Bobbit's spawn + config sets the retry setting (no silent SDK-default regression on future upgrades). + +#### CE-G1.1 Cache-hit telemetry adoption (S, after G1.0) +- Ingest the SDK's cache-hit rate / usage detail into the CE-G0.1 ledger instead of deriving it. + +#### CE-G1.2 Context-exclusion adoption (S–M, after G1.0) +- Use `excludeFromContext` for every Bobbit-initiated bash whose output the model doesn't need + (health probes, bookkeeping, the verification harness's own setup commands). +- Use `--exclude-tools` from role tool policy so sessions don't carry built-ins their role + forbids (today the guard extension denies at call-time but the definitions still ship). +- **Tests:** unit — spawn args include exclusions per role fixture; E2E — excluded tool absent + from the session's advertised tools. + +--- + +### CE-G2 — Tool-output budget tier *(L1; after CE-G1.0 re-verification)* + +#### CE-G2.1 Three-layer result budget with spill-to-disk (M–L) +Adopt Hermes' model, scoped to whatever 0.79.x doesn't already do: +- **Layer 1** per-tool caps (defaults, config-overridable): bash 20KB (head+tail split), grep + 20KB, read 50KB (with offset/limit guidance — see CE-G7.1), default 32KB. +- **Layer 2** any result over its cap → write full output to + `.bobbit/state/tool-spill/<sessionId>/<n>.txt`, return preview (first ~1.5KB to a line + boundary) + `[full output: <path>, <N> bytes — read with offset/limit if needed]`. +- **Layer 3** per-turn aggregate budget (~200KB): if a turn's results exceed it, spill the + largest non-spilled results. +- Implementation seam: prefer the SDK layer (fork patch — it owns tool execution) with a + Bobbit-side fallback in the tool wrappers; decide per CE-G1.0's report. Spill files are + session-scoped and cleaned with session deletion. +- **Tests (write RED first):** unit — cap/preview/spill matrix per layer, path cleanup; E2E — + oversized fixture command produces preview + readable spill file; pin that previews always + carry the path (no silent truncation). + +#### CE-G2.2 Images by reference (S) +- Any tool producing images (incl. `generate_image`) writes to disk and returns + `{path, width, height, bytes}` + a one-line description in model context; the UI renderer + loads from the path. Transcript stores the reference, not the base64. +- **Tests:** unit — renderer + reducer handle reference form; E2E — image tool round-trip + shows the image in UI with context payload under 1KB. Expected effect: the §1.4 image + session's shape ($10.75, 73% image bytes) becomes impossible. + +--- + +### CE-G3 — Turn economy *(L2)* + +#### CE-G3.1 Nudge & status batching (M) +- Coalesce per-agent status changes into one steer per quiet window (extend the existing 5s + debounce to batch *content*, not just delay it); suppress idle nudges that carry no new + actionable delta since the last one (track last-nudge state hash); keep the existing + exponential backoff. +- Add `trigger` attribution (CE-G0.1) so the coordination share of team-session cost is a + dashboard number; target: coordination turns <15% of team-session spend. +- **Tests:** unit — batching/suppression state machine (table-driven); E2E — fixture team + session emits N status events → ≤1 steer per window. + +#### CE-G3.2 Repeated-command guard (S) +- In the tool-guard path (`tool-guard-extension.ts` pattern — it already sees every + `tool_call`), detect an identical read-only bash command (allowlisted prefixes: `git status`, + `npm run check`, build/test commands) repeated within the same turn-window with no + intervening file changes, and prepend a short note to the result: + `[identical command run <N> ago — output unchanged? consider reusing]` — informational, never + blocking (the 17× `git push` case had legitimate retries in it; we hint, not forbid). +- **Tests:** unit — detection window logic; fixture E2E — note appears on 3rd identical call. + +#### CE-G3.3 llm-review scoping (M) **[BENCH-GATED]** +- Verifier sessions get a *review-scoped* context: the diff/artifact under review + acceptance + criteria + a slim reviewer role — not the full AGENTS.md cascade by default (config flag to + restore it for projects whose review genuinely needs it); cap tool-call rounds per attempt + (default ~20, configurable per gate) with an explicit "out of budget → fail with reason" + result instead of unbounded exploration. +- **Tests:** unit — context assembly for review sessions; E2E — round cap produces the + structured failure; BENCH — review-task success rate unchanged on the bench suite's review + task while cost drops. + +--- + +### CE-G4 — Prompt diet *(L1; BENCH-GATED; the behavior-affecting tier)* + +#### CE-G4.1 team-lead role rewrite: 35.5KB → ≤12KB (M–L, **highest-risk/highest-value**) **[BENCH-GATED]** +- Restructure `defaults/roles/team-lead.yaml`: keep identity, decision rules, and invariants + resident; move recipes/walkthroughs (the bulk) into skills or `docs/` loaded on demand via + the existing skills mechanism. Mechanical rule: anything the lead needs *every* turn stays; + anything needed in specific situations becomes a discoverable skill. +- Run the bench team-task before/after (N=3 runs each, same model); ship only if success and + coordination-quality metrics hold. +- **Tests:** pin a role-size budget test (like the tool-description budget) so it can't silently + regrow: `defaults/roles/*.yaml` each ≤ a per-role budget, sum ≤ corpus budget. + +#### CE-G4.2 AGENTS.md cascade budget (M) **[BENCH-GATED]** +- Token-budget `readAllAgentFiles()` output (default ~6k tokens, project-overridable): + priority order (project AGENTS.md > global > referenced docs), truncate at section boundaries + with an explicit `[truncated — full file at <path>; read on demand]` tail (spill rule, §5.4). +- Surface the measured size in prompt-sections (already visible) and warn in the UI when a + project's cascade exceeds budget — make the cost of a 17k-token playbook *visible* to its + author instead of silent. +- **Tests:** unit — budget/truncation boundary cases; pin the truncation-marker invariant. + +#### CE-G4.3 Role corpus diet + lazy role docs (S–M) **[BENCH-GATED]** +- Apply the G4.1 treatment to `qa-tester` (9.9KB) and `docs-writer` (6.7KB); add the role-size + budget test for all roles (delivered in G4.1). + +--- + +### CE-G5 — Model & thinking tiering *(L3)* + +#### CE-G5.1 Per-role model/thinking defaults (M) **[BENCH-GATED]** +- Extend role schema (role files + `role-store.ts`) with optional `model:` and `thinking:`; + precedence: explicit session pin > role default > project default > server default. + Resolution lives in `session-setup.ts::resolveBridgeOptions` (the pin mechanism exists; + this adds the role/project layers). +- Ship conservative defaults: leads = project default (frontier); workers = one tier below + unless role says otherwise; assistants = mid-tier. Show the resolved model per session in the + UI (it's a cost-relevant fact users should see). +- Forward-compat: this is the static layer the extension-platform P8 model-selector proposes + *overrides* to — keep the resolution function pure and injectable. +- **Tests:** unit — precedence matrix (table-driven); E2E — role fixture resolves expected + model in spawn args; BENCH — worker-task success holds on the cheaper tier. + +#### CE-G5.2 llm-review default model (S) **[BENCH-GATED]** +- Default verifier model = configurable `verification.reviewModel`, shipped as a mid-tier model; + wire through the existing `applyReviewModelOverrides` path. With CE-G3.3, targets the + $29–47 review tail directly. +- **Tests:** unit — override precedence incl. the new default; BENCH — review task quality holds. + +--- + +### CE-G6 — Compaction & history hygiene *(L1)* + +#### CE-G6.1 Proactive compaction policy + cost attribution (M) +- Configure pi's compaction proactively (threshold ~60–70% of context window — confirm exact + knobs in CE-G1.0's report) with a **cheap summarizer model** if the SDK supports a separate + compaction model (if not: fork patch, Hermes-style auxiliary model). +- Attribute compaction usage in the CE-G0.1 ledger (`trigger: compaction`). +- Coordinate with comms-stack F6 (compaction dispatch gate) — same code region, land the safety + fix first. +- **Tests:** manual-integration — long fixture session compacts proactively, session continues + correctly, ledger shows attributed compaction cost. + +--- + +### CE-G7 — Discovery efficiency *(L2; cheap, do early alongside G2)* + +#### CE-G7.1 Discovery guidance + skills (S) +- Add a concise (≤600 token) "efficient code discovery" section to the base system prompt / + relevant roles: grep with `-l` first → targeted read with offset/limit → never `cat` whole + files via bash; prefer built-in capped tools over bash for search; don't re-read unchanged + files. Ship a `code-discovery` skill with the longer playbook (lazy-loaded). +- Measure: bash share of tool calls (CE-G0.1 ledger dimension) — target <40% from 56%. +- **Tests:** prompt-section snapshot test; BENCH comparison is the real validation (bundle with + a G2 bench run to share runs). + +#### CE-G7.2 (Later, recorded) Code-graph/LSP pack +- AST/LSP-backed "find usages / go to definition" as an extension-platform pack — already in + that plan's Later section; cost rationale recorded here: replaces multi-round grep cascades + with single tool calls. + +--- + +### Expected impact (estimates — the bench harness replaces these with measurements) + +| Goal | Lever | Est. effect on affected session class | +|---|---|---| +| CE-G2 | L1 | 10–30% (work sessions; eliminates image-session blowups entirely) | +| CE-G3 | L2 | 15–30% of team/verifier session cost | +| CE-G4 | L1 | 20–40% of resident-context cost (≈ proportional cut in cache-reads) | +| CE-G5 | L3 | 30–60% on worker/verifier spend (price-sheet arithmetic) | +| CE-G1 | L4 | small steady-state; prevents double-billed retries; unlocks G2/G6 | +| CE-G6/G7 | L1/L2 | 5–15% each, mostly on long sessions | + +These do not multiply cleanly (overlapping bases), but a realistic compounded outcome on the +expensive session classes (team-leads, llm-reviews — 51%+ of spend) is **2–4×** cheaper at +equal task success, which is the gap the owner perceives vs Claude Code/Hermes. + +--- + +## §7 Reproduction recipes + +```bash +# Aggregate + cache ratio +jq -r '[.[] | {i:(.inputTokens//0), cr:(.cacheReadTokens//0), cw:(.cacheWriteTokens//0), o:(.outputTokens//0), c:(.totalCost//0)}] | + "sessions=\(length) uncachedIn=\([.[].i]|add) cacheRead=\([.[].cr]|add) cacheWrite=\([.[].cw]|add) out=\([.[].o]|add) totalCost=$\([.[].c]|add)"' \ + .bobbit/state/session-costs.json + +# Top sessions by cost +jq -r 'to_entries | sort_by(-(.value.totalCost//0)) | .[0:10][] | + "\(.key)\t$\(.value.totalCost)\tcacheR=\(.value.cacheReadTokens)\tout=\(.value.outputTokens)"' \ + .bobbit/state/session-costs.json + +# Prompt-section sizes for recent sessions +for f in $(ls -t .bobbit/state/session-prompts/*-prompt.json | head -5); do + jq -r '[.sections[] | "\(.label): \(.tokens)"] | join("\n")' "$f"; echo ---; done + +# Tool-call distribution in a transcript +jq -s '[.[] | select(.type=="message" and .message.role=="assistant") | .message.content[]? | + select(.type=="toolCall") | .name] | group_by(.) | map({tool:.[0],n:length}) | sort_by(-.n)' \ + ~/.bobbit/agent/sessions/<file>.jsonl + +# Repeated identical bash commands +jq -s '[.[] | select(.type=="message" and .message.role=="assistant") | .message.content[]? | + select(.type=="toolCall" and .name=="bash") | .arguments.command] | group_by(.) | + map({cmd:.[0],reps:length}) | map(select(.reps>1)) | sort_by(-.reps)' \ + ~/.bobbit/agent/sessions/<file>.jsonl + +# Largest tool results +jq -s '[.[] | select(.type=="message" and .message.role=="toolResult") | + {tool:.message.toolName, bytes:(.message.content|tostring|length)}] | sort_by(-.bytes) | .[0:20]' \ + ~/.bobbit/agent/sessions/<file>.jsonl +``` + +## §8 Risks & open questions + +1. **Prompt-diet regressions** (G4) — the team-lead role encodes hard-won behavior; the bench + gate + role-budget pin are the mitigations, but the bench suite must include a real team + task or G4.1 is being validated on the wrong distribution. +2. **Truncation hiding needed info** (G2) — mitigated by spill+preview+path and the read-on- + demand loop; watch for agents that never learn to follow the path (add guidance in G7.1). +3. **Fork divergence** — patching `earendil-works/pi` (G2 SDK-side, G6 auxiliary model) carries + maintenance cost; prefer upstreamable patches with tests in the fork repo. +4. **Bench cost & flakiness** — real-LLM benches cost real money and have run-to-run variance; + N=3 minimum, compare medians, and keep suites small. Bench runs are manual-only, never CI. +5. **Cheaper-model failure modes** (G5) — a worker that fails and retries on a cheap model can + cost more than succeeding once on a frontier model; the ledger's per-trigger attribution + (retry share by model) is the early-warning metric. +6. **Data drift** — every number in §1 is a 2026-06-10 snapshot of one machine spanning + multiple projects and weeks of model-price changes; CE-G0 exists so this document never + needs to be hand-reproduced again. From 5144a208147a4a6a904198a4abbb81a956ffdd52 Mon Sep 17 00:00:00 2001 From: Claude <noreply@anthropic.com> Date: Wed, 10 Jun 2026 22:00:24 +0000 Subject: [PATCH 05/18] Add four design docs: Mission Control, autoimprovement, client battery, harness gaps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - docs/design/mission-control.md — global scope at the top of the sidebar: global sessions/staff, the `bobbit` REST meta-tool, Global System Staff (Caretaker/Archivist/Improver/Observer), flight-recorder audit feed, phased core-vs-pack implementation plan. - docs/design/autoimprovement.md — recursive self-improvement with earned autonomy: Hermes-style per-goal review + curator, Claude-style dreaming gates, confidence-scored trust ladder with shadow-mode calibration, auto-approval with full audit + revert + automatic demotion. - docs/design/client-performance-battery.md — instrumentation-first battery plan: ambient infinite animations, redundant polling, full-tree render cadence; measurement harness, tiered fixes, battery-saver mode. - docs/design/harness-gap-analysis.md — feature/UX gaps vs openclaw, hermes (masters) and the Claude Code mirror, with a prioritized easy-wins roadmap. https://claude.ai/code/session_01LBW68Z2KvA3EhQMq3c2wrR --- docs/design/autoimprovement.md | 311 +++++++++++++++++++++ docs/design/client-performance-battery.md | 320 +++++++++++++++++++++ docs/design/harness-gap-analysis.md | 273 ++++++++++++++++++ docs/design/mission-control.md | 321 ++++++++++++++++++++++ 4 files changed, 1225 insertions(+) create mode 100644 docs/design/autoimprovement.md create mode 100644 docs/design/client-performance-battery.md create mode 100644 docs/design/harness-gap-analysis.md create mode 100644 docs/design/mission-control.md diff --git a/docs/design/autoimprovement.md b/docs/design/autoimprovement.md new file mode 100644 index 000000000..1857b552d --- /dev/null +++ b/docs/design/autoimprovement.md @@ -0,0 +1,311 @@ +# Autoimprovement — recursive self-improvement with earned autonomy + +Status: proposal / implementation blueprint. Each phase in §7 converts 1:1 into a mission +goal with acceptance criteria; steps name exact files and reuse points so they can be +executed without re-deriving this analysis. + +**What this is:** Bobbit learning from its own usage — drafting new skills after notable +sessions, patching skills that misfired, proposing AGENTS.md/prompt/config improvements, +consolidating memory — under a trust model where **autonomy is earned, never assumed, and +nothing happens off the record**. + +**Stance (binding):** + +1. **No self-modification without a human-visible artifact.** Every change — even an + auto-approved one — exists as a reviewed-or-reviewable proposal with a diff, an evidence + bundle, and a revert affordance. +2. **Human oversight is the *starting* posture, not the permanent one.** A confidence-scored + ladder (§5) moves low-risk change classes to auto-approval once the system's judgment has + been calibrated against the human's — and demotes them automatically on regression. +3. **Visibility is king.** Approved, auto-approved, rejected, and reverted changes all land + in the same audit surface (the Mission Control flight recorder, + [mission-control.md](mission-control.md) §6). +4. **Apply only through existing channels.** Skills/roles/tools land via the pack system and + config cascade; prompt context via the prompt-section assembly. Never raw file writes + outside those paths. + +Owner at runtime: the **Improver** system staff defined in +[mission-control.md](mission-control.md) §5 (this doc specifies *what* it runs; that doc +specifies *where it lives*). + +--- + +## §1 What the peers do (source study, 2026-06-10; checkouts per [harness-gap-analysis.md](harness-gap-analysis.md) §1) + +### Hermes — the reference self-improvement loop + +Two cooperating mechanisms: + +**(a) Per-turn background review** (`agent/background_review.py`). After every turn the agent +*forks itself* on a daemon thread — same provider/model/credentials/prompt-cache prefix — and +replays the conversation snapshot with two prompts: a **memory review** ("did the user reveal +persona/preferences/expectations worth remembering?") and a **skill review**. The fork runs +with a **tool whitelist limited to memory and skill management tools**; the main conversation +and its cache are never touched. The skill-review prompt is the crown jewel — its heuristics: + +- *Be active*: "most sessions produce at least one skill update… a pass that does nothing is + a missed learning opportunity". +- *User corrections are first-class skill signals* — "stop doing X", "too verbose", "you + always do Y and I hate it" get encoded into the governing skill, not just memory. +- *Target shape*: a small library of **class-level umbrella skills**, each with a rich + SKILL.md plus `references/` for session-specific detail — not "a long flat list of narrow + one-session-one-skill entries". +- *Patch-first preference order*: (1) update a skill that was actually loaded this session → + (2) update an existing umbrella → (3) add a support file under an umbrella → (4) only then + create a new skill. + +**(b) The curator** (`agent/curator.py`, +`website/docs/user-guide/features/curator.md`) — background maintenance so self-created +skills "don't pile up forever". Inactivity-triggered (interval ≥7 days **and** agent idle ≥2 h +— no cron daemon); runs on a configurable **auxiliary model slot** (`auxiliary.curator`). A +run = deterministic lifecycle transitions (`active → stale` after 30 d unused, `→ archived` +after 90 d) then one bounded LLM pass (max 8 iterations) that keeps / patches / consolidates / +archives. Safety furniture worth copying wholesale: **never auto-deletes** (archive is +recoverable); **pinned skills are untouchable**; **tar.gz snapshot before every pass** with +one-command rollback (and rollback-of-rollback); **`--dry-run`** produces the report with no +mutations; **first-run grace** — a fresh install defers the first real pass by one full +interval so the user can pin or opt out first. + +### Claude Code — gate discipline and away-summaries + +`services/autoDream/autoDream.ts` (background memory consolidation, "dreaming") fires a +forked subagent only after three gates pass **cheapest-first**: time gate (hours since +`lastConsolidatedAt`, one stat) → session-count gate (transcripts touched since) → a +consolidation **lock** (no concurrent dreamer), plus a 10-minute scan throttle so a passing +time-gate doesn't re-scan every turn. The dream runs as a first-class background task type +(`tasks/DreamTask/`) with a restricted `canUseTool`, visible in the task UI — dreaming is +*observable*, not hidden. Sibling services: `extractMemories` (per-session), `SessionMemory`, +and `awaySummary.ts` (small-fast-model catch-up — adopted via gap-analysis R4). +*(Mirror is ~2 months stale — verify details against current behavior before leaning on +specifics.)* + +### OpenClaw — authority, not learning + +No skill-learning loop; its contribution here is the **standing-orders convention** +(`docs/automation/standing-orders.md`): Scope / Trigger / Approval gates / Escalation rules as +the written form of delegated authority. Adopted as the authoring format for the Improver's +own charter (gap-analysis G4) — the trust ladder in §5 is, in effect, a machine-enforced +standing order. + +### Adopt / adapt / reject + +| Mechanism | Verdict | +|---|---| +| Per-turn review fork | **Adapt** — per-*session/goal* review instead of per-turn (Bobbit turns are expensive team workflows, not chat turns); tool-whitelisted; cheap aux model | +| Skill-review heuristics (active, corrections-first, umbrella shape, patch-first) | **Adopt verbatim** into the Improver's review prompt | +| Direct writes from the review fork (Hermes writes stores immediately) | **Reject** — violates stance #1; Bobbit's equivalent emits *proposals* | +| Curator lifecycle + snapshots + pin + dry-run + first-run grace + never-delete | **Adopt** | +| autoDream gate ordering + lock + visible task | **Adopt** for the dreaming schedule | +| Aux-model slots for review passes | **Adopt** (gap-analysis G11) | + +--- + +## §2 What can be improved (change classes) + +Every proposal carries exactly one **change class**; class determines blast radius, evidence +requirements, and the autonomy ceiling (§5). + +| Class | Examples | Blast radius | Autonomy ceiling | +|---|---|---|---| +| `memory` | user-profile entries, project facts | additive, bounded | L1 | +| `skill-new` | draft a new staged skill | additive (staged ≠ active) | L1 | +| `skill-patch` | patch/extend an existing **agent-created** skill | mutating, snapshot-backed | L2 | +| `skill-lifecycle` | stale/archive/consolidate (curator) | mutating, recoverable | L2 | +| `prompt-context` | AGENTS.md edits, role prompt tweaks, tool-description fixes | mutating, behavior-wide | L0 only (human always) | +| `tool-new` / `config` | new tool group, config cascade changes, pack updates | mutating–destructive | L0 only | + +Human-authored skills, pinned skills, and anything under `defaults/` or budget-pinned by tests +(e.g. `tests/tool-description-budget.test.ts`, `tests/agents-md-budget.test.ts`) are +**off-limits to auto-apply at every level** — proposals against them always require a human. + +## §3 The improvement loop + +```mermaid +graph LR + O[Observe<br/>outcomes, corrections,<br/>gate failures, cost] --> P[Propose<br/>diff + evidence bundle<br/>+ confidence score] + P --> R{Review} + R -->|"L0 / over-ceiling"| H[Human: proposal panel<br/>approve / edit / reject] + R -->|"auto: conf ≥ threshold<br/>(class at L1/L2)"| A + H --> A[Apply<br/>via packs / config cascade<br/>+ snapshot] + A --> F[Flight recorder<br/>every outcome recorded] + A --> M[Measure<br/>did it help?] + M -->|regression| D[Demote class<br/>+ auto-revert option] + M -->|calibration data| R +``` + +- **Observe.** Signals, cheapest first: goal/gate outcomes (`gate_failed`, retries), + human-signoff rejections, user-correction heuristics in transcripts ("stop doing X", + "too verbose" — the Hermes signal list verbatim), skill load-but-failed events, cost ledger + outliers (token-cost CE-G0.1 when it lands). Transcripts are already retained + (`.bobbit/state/sessions/<id>.jsonl`) and indexed (`src/server/search/flex-store.ts`). +- **Propose.** The Improver emits a structured proposal: change class, unified diff (or new + file tree), **evidence bundle** (transcript excerpts + signal that triggered it + expected + impact + rollback plan), and **confidence score**. Scoring is a *separate* cheap-model judge + pass against a per-class rubric — the proposer never grades its own homework. +- **Review.** Routed by class + ladder level (§5). Human review uses the existing proposal + panel with inline editing ([editable-proposals.md](editable-proposals.md), + [proposal-inline-comments.md](proposal-inline-comments.md)) — approve / edit-then-approve / + reject(+reason). Rejections with reasons are calibration gold (§5). +- **Apply.** Staged-skill writes go to the improvement pack (§4); config/prompt changes go + through the config cascade. Mutating classes snapshot first (curator-style tar.gz of the + affected tree under `.bobbit/state/improvement/snapshots/`). +- **Record.** Every transition (proposed, approved-by-human, approved-by-policy, rejected, + applied, reverted, demoted) appends to the flight recorder with actor + evidence link. +- **Measure.** §6. + +## §4 Skill lifecycle (where learned skills live) + +Learned skills are **pack-delivered, agent-created, and visibly second-class until promoted**: + +- A server-scoped **`learned-skills` pack** (created on first proposal; resolved via the + normal `PackResolver` scope order) holds them — *not* `.claude/skills/` (human-authored + space) and *not* `defaults/` (shipped space). Provenance is therefore structural: you can + always tell what the machine wrote, disable any of it per-entity via `pack_activation`, or + nuke the whole experiment by removing one pack. +- States: `draft` (proposal pending) → `staged` (approved; catalog-visible to agents with a + `[learned — staged]` marker; usage tracked) → `active` (promoted after N successful uses + with no negative signal) → `stale` → `archived` (curator transitions, Hermes timings as + defaults: 30 d / 90 d). `pinned` exempts from all auto-transitions. Archive = + `.bobbit/state/improvement/skill-archive/`, never deletion. +- Shape discipline: the Improver's prompt enforces Hermes' umbrella rule — patch-first, few + class-level skills with `references/`, no one-session confetti. +- Dedup: before drafting, the Improver must search existing skills/packs + (`slash-skills.ts` catalog + marketplace) and prefer `skill-patch`. + +## §5 Trust ladder — confidence-scored, calibrated, demotable + +Autonomy is granted **per change class**, not globally: + +| Level | Meaning | +|---|---| +| **L0** | Every proposal human-reviewed. The starting posture for all classes. | +| **L0.5 shadow** | Still human-reviewed, but the system *records* "would have auto-approved at threshold θ" alongside each human decision. After ≥20 decisions in a class, the calibration report shows: at θ, X% of auto-approvals would have matched the human, Y false-approves. The user promotes a class only when shown that evidence. | +| **L1** | Auto-approve `memory` / `skill-new` (additive classes) when `confidence ≥ θ_class`. Applied immediately; flight-recorder entry reads "approved by policy (0.93 ≥ 0.85)" with one-click revert; a digest of auto-approvals lands in the user's inbox (visibility without interruption). | +| **L2** | Adds mutating classes (`skill-patch`, `skill-lifecycle`) with mandatory pre-apply snapshot + auto-generated revert proposal attached. | + +Guard rails (all enforced in code, not prompt): + +- `prompt-context`, `tool-new`, `config` classes **cannot** be promoted past L0 (table §2). +- **Kill switch**: one settings toggle drops everything to L0 instantly; the Improver staff + pausing (existing staff machinery) halts the loop entirely. +- **Automatic demotion**: a human revert of an auto-approved change, or a measured regression + (§6), drops that class one level and requires re-calibration to climb back. +- Thresholds `θ_class` are user-tunable in settings; defaults conservative (0.85 additive, + 0.9 mutating). + +## §6 Measurement (closing the loop) + +Each applied change records its *expected impact* (from the evidence bundle) and is evaluated +against observed signals over a trailing window: skill usage count + post-use outcome (goal +gate pass/fail in sessions that loaded it), correction-signal recurrence (did "too verbose" +complaints stop?), and cost deltas once CE-G0.1 lands. Outcomes: `helped` / `neutral` / +`regressed` → flight recorder; `regressed` → demotion + revert proposal. Honest scope: v1 +measurement is heuristic, not causal — the point is a feedback signal and demotion trigger, +not a science experiment. + +## §7 Phased implementation plan + +Run the review/judge/dream passes on named aux-model slots from day one +(gap-analysis G11; config shape mirrors [per-role-model-overrides.md](per-role-model-overrides.md)). +Phases assume mission-control.md P0 (global scope) + P4 (flight recorder) exist; P1–P2 below +can land before Mission Control's UI by writing flight-recorder entries only. + +### AI-P1 — Proposal substrate + signal plumbing + +1. Define `ImprovementProposal` (class, diff/files, evidence bundle, confidence, status) in a + new `src/server/agent/improvement-store.ts` (JSONL under `.bobbit/state/improvement/`, + crash-safe write pattern per `session-store-crash-safety.md`). +2. New proposal tool `propose_improvement` under `defaults/tools/proposals/` following the + `propose_staff` pattern; render in the proposal panel via the existing proposal registry + (`src/app/proposal-registry.ts`, parsers in `proposal-parsers.ts`) with diff view + inline + edit + approve/reject(+reason). +3. Push triggers `gate_failed` / `session_errored` (shared with gap-analysis R2) so the + Improver can subscribe. +4. REST: `GET/POST /api/improvements`, `POST /api/improvements/:id/decision`. +5. Tests: unit (store round-trip, class validation), API e2e (decision flow), browser e2e + (panel renders evidence + diff; reject persists reason). + +Acceptance: a hand-fed proposal flows draft → human decision → recorded; nothing auto-applies. + +### AI-P2 — The Improver staff + post-goal review (Hermes loop, proposal-shaped) + +1. Improver ships in the mission-control pack roster (mission-control.md §5): role prompt = + Hermes review heuristics (§1a verbatim: active, corrections-first, umbrella shape, + patch-first) + standing-orders charter; triggers: `goal_archived`, `gate_failed`, weekly + `schedule`. +2. Review run = tool-whitelisted session (read_session/search + `propose_improvement` only — + enforce via role `toolPolicies`). +3. `learned-skills` pack bootstrap (§4): server-scoped pack created on first approved + `skill-new`; staged-skill marker in the skills catalog; usage events recorded (hook the + skill-load path in `slash-skills.ts`). +4. Judge pass: separate cheap-model scoring call with per-class rubric prompts (stored beside + the role in the pack, not hardcoded). +5. Tests: e2e — archive a goal with a planted correction signal ⇒ a `skill-new` proposal with + evidence appears; approving it makes the skill resolvable with staged marker. + +Acceptance: end-to-end learn-from-session path works with a human in the loop at every step. + +### AI-P3 — Curator + dreaming + +1. Curator run (Hermes semantics): deterministic lifecycle transitions + bounded LLM pass + emitting `skill-lifecycle`/`skill-patch` proposals; tar.gz snapshot before any apply; + `pin`/`restore`/`dry-run` via REST + staff-page UI; first-run grace (seed `lastRunAt` on + creation). +2. Dreaming = scheduled Archivist/Improver job with autoDream's gate ordering (time → new + transcript count since last dream → lock file under `.bobbit/state/improvement/`), mining + transcripts for `memory` + `skill-new` proposals; runs as a visible session (Bobbit's + equivalent of DreamTask — it's just a staff wake, already visible in the sidebar). +3. Tests: lifecycle transition unit tests with fake clocks; snapshot/rollback round-trip; + dream gates (no run when gates closed; lock prevents concurrency). + +Acceptance: an unused staged skill ages to stale/archived and back via restore; dreams run +only when gates pass and appear in the flight recorder. + +### AI-P4 — Shadow mode + calibration (L0.5) + +1. Judge runs on every proposal; store `(confidence, wouldAutoApprove@θ, humanDecision)`. +2. Calibration report: `GET /api/improvements/calibration?class=` → agreement matrix per + threshold; surfaced on the Improver staff page ("at θ=0.85: 94% agreement, 1 + false-approve / 31"). +3. Tests: calibration math unit tests; e2e for report endpoint. + +Acceptance: after seeded decisions, the report shows per-class agreement; still zero +auto-applies. + +### AI-P5 — Graduated autonomy (L1/L2) + +1. Settings: per-class level + threshold (enforced server-side against §2 ceilings); global + kill switch. +2. Auto-approval path: policy check → apply → flight-recorder "approved by policy" entry → + inbox digest entry → revert affordance (`POST /api/improvements/:id/revert`, snapshot + restore). +3. Automatic demotion on revert/regression. +4. Tests: e2e — class at L1 + high confidence ⇒ auto-applied + recorded + revertible; revert + ⇒ class demoted; ceiling classes can never be set to L1 (server rejects). + +Acceptance: autonomy on, fully audited, reversible, demotable; kill switch verified. + +### AI-P6 — Measurement loop + +Outcome evaluation per §6 wired into a scheduled Improver pass; `helped/neutral/regressed` +recorded; regression triggers demotion + revert proposal. Acceptance: planted regression +(skill that makes a gate fail) is detected and demoted within one evaluation window. + +--- + +## §8 Open questions (for review, not blockers) + +- Should staged skills be visible to *all* sessions or only sessions in the project where the + signal originated? (Lean: global, since the learned-skills pack is server-scoped; per-entity + `pack_activation` already gives opt-out.) +- Per-turn review (Hermes) vs per-goal review (this doc): per-goal is the right starting + cadence for cost reasons, but chat-style staff sessions may deserve a per-session review on + sleep. Revisit with AI-P2 data. +- Judge model quality: if cheap-model confidence proves noisy in shadow mode, calibration + will show it before any autonomy is granted — that's the point of L0.5. + +Cross-references: [mission-control.md](mission-control.md) (Improver/Archivist staff, flight +recorder, inbox digests) · [harness-gap-analysis.md](harness-gap-analysis.md) G1/G2/G11 · +[token-cost-efficiency.md](token-cost-efficiency.md) (cost ledger CE-G0.1, aux summarizer) · +[skill-ux-and-autonomous-activation.md](skill-ux-and-autonomous-activation.md) · +[editable-proposals.md](editable-proposals.md) · [human-signoff-gates.md](human-signoff-gates.md). diff --git a/docs/design/client-performance-battery.md b/docs/design/client-performance-battery.md new file mode 100644 index 000000000..90fe40541 --- /dev/null +++ b/docs/design/client-performance-battery.md @@ -0,0 +1,320 @@ +# Client performance & battery — analysis and implementation plan + +Status: proposal / implementation blueprint. Each phase in §6 is written to convert 1:1 into a +mission goal with acceptance criteria, and each step is explicit enough to execute without +re-deriving the analysis. + +**The symptom this answers:** Bobbit visibly drains laptop battery — noticeably worse than +leaving a typical web app open. The goal: an *idle* Bobbit tab should cost ≈0% CPU with zero +continuous repaints, and an *active* one should stay smooth even on machines without dedicated +GPU acceleration (soft constraint, not a hard gate). + +**The one-paragraph answer:** The server side was already fixed (see the +[reduce-server-cpu series](reduce-server-cpu-performance-report.md) — idle gateway ≈ 0% CPU), +so the drain is client-side, and it is structural, not one bug: **the UI is never idle.** +Dozens of `infinite` CSS animations (sprites, accessories, pulses, shimmers) keep the +compositor — and on non-GPU machines, the CPU rasterizer — awake 24/7 even when nothing is +happening; two of the pulse animations animate `box-shadow`, which repaints on the main thread +every frame; a 5-second REST poll re-fetches the session list the WebSocket already pushes; +a 1-second timer re-renders the *entire app* while any verification runs; and every WebSocket +streaming delta schedules a full root re-render (rAF-coalesced, so up to ~60 full lit-html +tree evaluations per second during streaming). None of these is individually fatal; together +they mean the browser never reaches its power-saving render-idle state. The plan below is +**instrumentation-first** (same discipline as +[token-cost-efficiency.md](token-cost-efficiency.md)): we build a small repeatable measurement +harness, capture a baseline, then land tiered fixes — each behind a +[`perf-flags.ts`](../../src/app/perf-flags.ts)-style kill switch and re-measured against the +baseline. + +Related: [docs/perf/defer-offscreen-render.md](../perf/defer-offscreen-render.md) (prior art: +measured, flag-gated, default-on), [docs/perf/bundle-profile.md](../perf/bundle-profile.md), +[docs/design/shrink-initial-bundle.md](shrink-initial-bundle.md). + +--- + +## §1 Why "battery" ≠ "CPU benchmark" + +Battery drain on modern laptops is dominated by how often the machine is *prevented from +sleeping its silicon*, not by average CPU%. Three distinct mechanisms matter here: + +1. **Render-loop liveness.** Any running CSS animation — even a compositor-only + `transform`/`opacity` one — forces the browser to produce frames continuously. The GPU (or + CPU rasterizer without GPU acceleration) never idles, and the display pipeline never drops + to its low-power state. Cheap per frame × 60 fps × every hour the tab is open = the single + biggest battery line item for "decorative" animation. +2. **Main-thread paint.** Animating paint-affecting properties (`box-shadow`, `background`, + layout properties) repaints on the main thread every frame. Strictly worse than (1). +3. **Timer wakeups.** Every `setInterval` tick wakes the process, runs JS, possibly fetches + over the network (radio/Wi-Fi power-up), and possibly re-renders. Browsers throttle timers + in *hidden* tabs but not in visible-but-idle ones. + +The fix plan maps 1:1 onto these: stop animating when nothing is happening (§6 P1), never +animate paint properties (§6 P1), and make timers event-driven or visibility/idle-gated +(§6 P2). + +--- + +## §2 Findings (verified in source, 2026-06-10) + +### F1 — Ambient `infinite` animations never stop (HIGH, the headline) + +Inventory (grep `animation:.*infinite` across `src/**/*.css`): + +- **`src/ui/app.css` (sprite/chat layer)** — ~25 infinite rules: `blob-busy-shadow` (10s), + `blob-crown-idle-sleep-breathe` / `blob-bandana-idle-sleep-breathe` (3.6s), + `blob-compact-squash` / `blob-compact-squash-rigid` (3s, several marked `!important`), + `magnifier-depth-idle` (10s `steps(1)`), `wand-sparkle-a/b`, `flask-bubbles`, + `ask-heartbeat` (1.8s), two `shimmer` variants. +- **`src/app/app.css` (shell layer)** — `bobbit-bob`, `bobbit-breathe`, `bobbit-eyes` (+ `-s` + sidebar variants), `status-pulse` (2s), `unseen-dot-pulse` (1.8s), `gentle-float` (4s), + `bobbit-unread-tap` (1.6s), `bobbit-sidebar-unread-blink` (3.4s `steps(1)`). + +Two important nuances (so the fix targets the right thing): + +- The sprite keyframes themselves are **well-built**: `bobbit-eyes`, `blob-busy-shadow` etc. + animate `transform`/`opacity` only (verified at `src/app/app.css:439`, + `src/ui/app.css:2693`) — compositor-friendly, no per-frame paint. The cost is **liveness** + (§1.1): they run `infinite`, on *every* visible sprite (sidebar shows one per session), at + all times — including when every agent is idle and the user is just reading. +- `status-pulse` and `unseen-dot-pulse` **do animate `box-shadow`** (§1.2): main-thread + repaint every frame, forever, for each status dot / unseen indicator on screen. + +`prefers-reduced-motion` is honored in only **9** CSS sites — e.g. `.unseen-dot` and +`.bobbit-unread-pulse` opt out, but the sprite system, shimmers, and pulses largely don't. + +### F2 — Polling alongside a push channel (MEDIUM-HIGH) + +- **Session list poll** — `src/app/api.ts::startSessionPolling()` (`api.ts:273`) runs + `refreshSessions()` every **5 s** whenever the app is authenticated and visible. The + gateway *already pushes* `session_status` frames over the WebSocket (the same file handles + them ~20 lines above). So in steady state the poll is pure redundancy: a REST round-trip + + state diff + `renderApp()` every 5 s, all day. It exists as a safety net for missed frames — + the right shape is "slow fallback, fast on reconnect" (§6 P2.1). +- **Live verification timer** — `src/app/goal-dashboard.ts:1090` `startLiveVerifTimer()`: + `setInterval(() => renderApp(), 1000)` while any verification step is `running`. A full app + re-render every second, to advance an elapsed-time label. (It does stop when no step is + running — `stopLiveVerifTimerIfDone()` — but verifications can run for many minutes.) +- **Relative-clock timer** — `src/app/render-helpers.ts:338`: `renderApp()` every 60 s. Fine + in isolation; included in the wakeup budget. +- **Good pattern already in-tree** — the git-status poll + (`src/app/session-manager.ts:2821`) is the model citizen: 30 s tick, visibility-gated, + coalesced against event-driven refreshes (`gitStatusLastRefreshAt`, 10 s min spacing), + self-cancelling when the session changes or the repo is known-absent, with a + `visibilitychange` catch-up refresh. P2 below converges the other pollers on this shape. +- ~92 `setInterval`/`setTimeout`-loop sites exist across `src/app` (top files: + `goal-dashboard.ts` ×24, `dialogs.ts` ×16, `session-manager.ts` ×11, `api.ts` ×10). Most + are benign one-shots; the audit in P0 classifies them so we fix the loops, not the noise. + +### F3 — Every event is a full-tree render (MEDIUM) + +`renderApp()` (`src/app/state.ts:687`) is rAF-coalesced (good — multiple calls per frame +collapse to one) but each invocation re-evaluates the **entire root lit-html template** +(sidebar + chat + panels). There are **646** `renderApp()` call sites; during agent streaming, +every `message_update` WS frame schedules one, so a streaming session runs a full-tree +evaluation at up to the frame rate, continuously, for minutes. lit-html diffing makes each +pass cheap-ish, but "cheap × 60/s × whole tree" is exactly the §1.1 problem in JS form. The +server-side coalescing experiment was rejected +([reduce-server-cpu-experiment-stream-coalescing.md](reduce-server-cpu-experiment-stream-coalescing.md)) +— the client is the right layer: batch *visual* updates without delaying state updates. + +### F4 — Dev mode amplifies everything (LOW, but affects the owner's perception) + +`npm run dev:harness` runs vite dev (esbuild transforms, HMR socket, unminified lit) plus the +restart harness. The baseline (§4) must be captured on a **production build** (`npm run +build` + gateway) *and* dev mode, separately, so we know how much of the perceived drain is +dev-only and don't chase ghosts. + +### F5 — What was ruled out + +- Gateway CPU: already measured ≈0% idle (`reduce-server-cpu-baseline.md`); not the drain. +- `deferOffscreenRender` already prevents offscreen transcript paint on activation; long + transcripts are not the idle-drain cause. + +--- + +## §3 Targets (acceptance bar for the whole effort) + +Measured on a production build, mid-range laptop, all four states from §4: + +| State | Target | +|---|---| +| Idle-foreground (agents idle, user reading) | 0 continuous animations running; <2 paint events/s; <8 timer wakeups/min; CPU ≈0% | +| Hidden tab | 0 animations, 0 renders, ≤1 network poll/min | +| Streaming (one active agent) | visual update cadence ~10 Hz (not 60); no long tasks >50 ms from render | +| Dashboard open, verification running | scoped second-tick (no full-tree render/s) | + +"Battery-saver mode" (§6 P3.3) tightens these further on demand. + +--- + +## §4 Phase P0 — Measurement harness (build BEFORE any fix) + +Goal: a repeatable, scriptable protocol that outputs one comparison table per run. Pattern: +the server-side benchmark harness (`reduce-server-cpu-benchmark.md`) and `boot-timing.ts`. + +Implementation steps: + +1. **Add `src/app/perf-monitor.ts`** (debug-only, loaded when + `localStorage.bobbitPerfFlags` contains `+perfMonitor` — reuse the + `isPerfFlagEnabled`/`perf-flags.ts` mechanism). It records, over a sampling window: + - renders/s: increment a counter inside `renderApp()`'s rAF callback (one-line hook, + guarded by the flag); + - timer wakeups/s: in monitor mode only, wrap `window.setInterval`/`setTimeout` to count + scheduled-callback executions by call-site stack bucket; + - long tasks: `PerformanceObserver({ type: "longtask" })`; + - running animations: `document.getAnimations().length` sampled 1/s (this is the direct + "is the page render-idle" probe); + - WS frames/s by frame type (hook the existing WS message dispatch). + Expose `window.__bobbitPerf.report()` returning JSON; the + [`client-debug` skill](../../.claude/skills/client-debug/SKILL.md) is the retrieval path. +2. **Define the four states** as a manual-but-scripted protocol (a new + `docs/perf/battery-protocol.md` section or appendix in this doc): idle-foreground, + hidden, streaming (one mock-agent session — reuse the E2E mock agent), dashboard+verification. + 60 s sample each, three runs, production build and dev build. +3. **Record the baseline table** into this doc (§7 placeholder) — same convention as the + shared baseline table in `reduce-server-cpu-performance-report.md`. +4. OS-level cross-check (manual, documented): macOS `powermetrics`/Activity Monitor "Energy + Impact" or Windows Task Manager power column for the browser process, idle-foreground vs + after P1. This is the number the owner actually feels. + +Acceptance: `window.__bobbitPerf.report()` returns all five metric families in all four +states; baseline table committed. No behavior change when the flag is off (pinned by a unit +test asserting `perf-monitor` is not imported in the default boot path — follow the +`tests/`-side pattern used for `boot-timing`). + +--- + +## §5 Fix inventory (what we change and why it's safe) + +Numbered for reference from the phases: + +- **FX1 Ambient-animation gating.** Introduce one CSS class contract: every *ambient* + (infinite, decorative/status) animation rule gains the guard + `html.anims-paused & { animation-play-state: paused; }` (or is collected under a shared + selector). A new tiny module `src/app/animation-power.ts` toggles `anims-paused` on + `<html>` when: tab hidden (`visibilitychange`), OR no session is in a working state + (derive from existing session status state — the sidebar already knows), OR battery-saver + is on (P3.3). One-shot transition animations (card-slide-in, banner-slide-down) are NOT + ambient and are excluded — pausing those would freeze mid-transition. +- **FX2 Sprites as status, not decoration** (UX improvement, not just a fix): a bobbit + animates **only while its agent is actually working**; idle/sleeping agents show a static + frame (or the existing sleep pose, frozen). Animation regains meaning as an at-a-glance + status signal, and N-sessions × ambient-loops disappears. Implemented as CSS: the + bob/breathe/eyes loops apply only under the existing busy/working status classes + (verify exact class names in `src/ui/bobbit-render.ts` / `app.css` during impl). +- **FX3 Kill paint-property animation.** Rewrite `status-pulse` and `unseen-dot-pulse` to + animate `opacity`/`transform` on a pseudo-element halo instead of `box-shadow` (visually + identical: a pre-rendered shadow layer fading/scaling). `pr-conflict-pulse` same treatment. +- **FX4 Global `prefers-reduced-motion` sweep.** One blanket rule disabling all ambient + animations under reduced-motion (keep opacity cross-fades), replacing the 9 piecemeal + opt-outs. +- **FX5 Session-poll demotion.** `startSessionPolling()` becomes the fallback it was meant to + be: interval 5 s → 60 s while the WS is connected and healthy; immediate `refreshSessions()` + on WS reconnect and on `visibilitychange`→visible (catch-up), mirroring the git-status + pattern. Keep 5 s only while the WS is down. +- **FX6 Scoped second-tick.** Replace `liveVerifTimer`'s `renderApp()` with a scoped update: + the elapsed-time labels become self-updating (a small component/`setInterval` that writes + `textContent` on the specific nodes, or a CSS counter). Full-tree render is reserved for + actual status transitions (which already arrive via WS events). +- **FX7 Streaming render cadence.** In the WS dispatch path for high-frequency frame types + (`message_update` deltas): apply state immediately (never delay data), but schedule the + visual render through a 100 ms trailing-edge throttle instead of per-frame rAF. Terminal + frames (`message_complete`, status changes) flush immediately. Build on the existing + `_renderScheduled` machinery in `state.ts` — add a `renderAppThrottled(ms)` next to + `renderApp()`; only the streaming call sites switch to it. +- **FX8 Timer audit & visibility gates.** From the P0 wakeup-by-callsite report, every + surviving interval ≥1/min gets the git-status-pattern treatment (visibility gate + + event-coalescing + self-cancel). `dialogs.ts`/`goal-dashboard.ts` loops are the known + suspects; fix what the data convicts. +- **FX9 Battery-saver mode.** A settings toggle (and auto-*suggestion* via + `navigator.getBattery()` when discharging — suggest, never silently switch): forces + `anims-paused`, stretches FX5/FX7 cadences (poll 120 s, render throttle 250 ms), disables + shimmer placeholders. State lives with the other UI settings; surfaced in the settings page + and the `bobbitPerfFlags` escape hatch. + +--- + +## §6 Phased implementation plan + +Each phase = one mission goal; land in order; every behavior change ships behind a perf-flag +kill switch (default-on once verified, like `deferOffscreenRender`) and is re-measured with +the P0 harness before the flag's removal cycle. + +### P1 — Stop animating at idle *(FX1–FX4; the battery headline)* + +Steps, in order: + +1. Build `src/app/animation-power.ts`: exports `initAnimationPower()` (called from + `main.ts`) — registers `visibilitychange` + subscribes to session-status state changes; + computes `shouldPause = hidden || noSessionWorking || batterySaver || reducedMotion`; + toggles `anims-paused` class on `document.documentElement`. Perf-flag: + `-pauseAmbientAnims` restores current behavior. +2. CSS sweep #1: tag every `infinite` animation rule listed in F1 with the + `html.anims-paused` pause guard. Mechanical change; keep one shared comment block in each + CSS file pointing back to this doc ("never reintroduce un-gated infinite animations"). +3. CSS sweep #2 (FX2): move the sprite work/idle loops under the busy/working status + selectors so idle sprites are static. Confirm the sleep pose stays as a static frame. +4. Rewrite `status-pulse` / `unseen-dot-pulse` / `pr-conflict-pulse` per FX3. +5. Add the blanket reduced-motion rule (FX4); delete the now-redundant piecemeal opt-outs. +6. Tests: browser E2E (`tests/e2e/ui/animation-power.spec.ts`) asserting (a) idle state ⇒ + `html.anims-paused` present and `document.getAnimations().length === 0`; (b) mock agent + working ⇒ its sprite animates; (c) `visibilitychange` to hidden pauses; (d) perf-flag + opt-out restores animations. Unit test pinning that no `infinite` animation in the two CSS + files lacks the pause guard (regex over the CSS — the "missing test IS the bug" rule). + +Acceptance: idle-foreground `getAnimations()` = 0; paint events/s <2; the E2E suite above +green; visual QA confirms sprites still bob while agents work. + +### P2 — Timer & render hygiene *(FX5–FX8)* + +1. FX5 session-poll demotion in `api.ts` (+ unit test on the interval-selection logic; the + spurious-unread pinning test `tests/spurious-idle-unread.spec.ts` must stay green). +2. FX6 scoped verification tick in `goal-dashboard.ts` (+ E2E: elapsed label advances while + `renderApp` counter — exposed via perf monitor — does not tick per second). +3. FX7 `renderAppThrottled()` in `state.ts` + switch streaming dispatch call sites; E2E: + streaming a long mock response produces ≤15 renders/s (perf-monitor counter) with no + visible degradation (follow-tail still pins to bottom — `follow-tail` tests stay green). +4. FX8 audit: commit the wakeup-by-callsite table to this doc, fix convicted loops on the + git-status pattern. + +Acceptance: idle-foreground wakeups <8/min; hidden-tab network ≤1 req/min; streaming render +cadence ~10 Hz; all existing unit + e2e phases green. + +### P3 — Product polish + +1. **P3.1 Remove temporary flags** that have soaked (per the `defer-offscreen-render.md` + removal convention), keep `anims-paused` as the permanent mechanism. +2. **P3.2 Re-baseline & publish**: re-run the §4 protocol, fill the after-table in §7, include + the OS-level energy reading. If streaming long-tasks remain, only then evaluate the heavier + options (scoped/component-level rendering for chat; sprite-sheet/canvas sprite engine) — + they are deliberately out of scope until data convicts them. +3. **P3.3 Battery-saver mode** (FX9): settings toggle + `getBattery()` suggestion banner + + docs. Browser E2E: toggle persists across reload (the settings E2E pattern, + `tests/e2e/ui/settings.spec.ts`). + +--- + +## §7 Baseline / after (filled by P0 / P3.2) + +| State | Metric | Baseline (dev) | Baseline (prod) | After P1 | After P2 | +|---|---|---|---|---|---| +| *(populated by the harness — renders/s, wakeups/min, animations running, long tasks/min, paint events/s, OS energy)* | | | | | | + +--- + +## §8 Non-goals / explicitly rejected + +- **Server-side stream coalescing** — already tried and rejected for UX latency reasons + (`reduce-server-cpu-experiment-stream-coalescing.md`); FX7 does it client-side at the + visual layer only. +- **Removing the bobbit sprite system** — it is the product's personality + ([bobbit-sprites.md](../bobbit-sprites.md)). FX2 makes it *more* meaningful, not less + present. +- **A WebGL/canvas renderer rewrite** — not justified by any current evidence; revisit only + if P3.2 data demands it. + +Cross-references: [token-cost-efficiency.md](token-cost-efficiency.md) (the sibling +"instrumentation-first" effort, server/cost side), [mission-control.md](mission-control.md) +(the Observer staff can watch for perf regressions once the harness exists), +[harness-gap-analysis.md](harness-gap-analysis.md) §battery for how peer harnesses avoid this +class of cost (mostly by being TUIs — Bobbit's bar is higher because it ships a real UI). diff --git a/docs/design/harness-gap-analysis.md b/docs/design/harness-gap-analysis.md new file mode 100644 index 000000000..722c25e0b --- /dev/null +++ b/docs/design/harness-gap-analysis.md @@ -0,0 +1,273 @@ +# Harness gap analysis — OpenClaw, Hermes, Claude Code vs Bobbit + +Status: research + prioritized roadmap. Feature/UX gaps only — **cost/token efficiency is +explicitly out of scope** here because [token-cost-efficiency.md](token-cost-efficiency.md) +§4 already covers the Claude Code / Hermes comparison on that axis. + +**The question this answers:** what do the strongest peer harnesses have that Bobbit lacks, +which of those Bobbit actually wants given its direction (the best harness out there — +extensible and fun for coders, researchers, and small-business owners alike), and which are +easy wins on substrate that already exists? + +**The one-paragraph answer:** Bobbit's core — goals/workflows with verification gates, team +orchestration, worktree/sandbox isolation, packs/marketplace, and a real web UI — is ahead of +all three peers; nothing in their orchestration stories needs copying. The real gaps cluster +in four places: **(1) memory** — all three peers remember things across sessions +automatically; Bobbit only has per-staff memory and a search index nobody consults at prompt +time; **(2) the self-improvement loop** — Hermes reviews every turn for skill/memory updates +and runs a background curator; Claude Code ships background "dreaming" memory consolidation; +Bobbit has neither (→ [autoimprovement.md](autoimprovement.md)); **(3) reach** — OpenClaw +meets users in 30+ messaging channels with companion apps while Bobbit is a localhost web +app; **(4) ambient automation UX** — peers have one-shot reminders, away summaries, and +"standing orders" conventions on top of cron, where Bobbit has (good) staff triggers but +nothing above them. Several gaps are genuinely cheap because the substrate (staff, triggers, +inbox, packs, session search) already exists — the roadmap in §5 sequences the easy wins +first. + +--- + +## §1 Method + +Source study of three checkouts (2026-06-10/11): + +| Harness | Repo / commit | Freshness | +|---|---|---| +| OpenClaw | `github.com/openclaw/openclaw` @ `2d404f1b` (2026-06-11) | current master | +| Hermes | `github.com/NousResearch/hermes-agent` @ `a8f404b` (2026-06-10) | current master | +| Claude Code | leaked-source mirror `github.com/yasasbanukaofficial/claude-code` @ `a371abb` (2026-04-05) | **~2 months stale** — findings cross-checked against public docs/changelog; anything load-bearing is flagged *(verify against current behavior)* | + +File citations below are repo-relative within those checkouts. "Easy win" means: high user +value, ≤ a few days of work, and lands on an existing Bobbit substrate without new +architecture. + +Bobbit baseline assumed known: goals/workflows/gates (`docs/goals-workflows-tasks.md`), +staff + triggers + inbox (`docs/staff-agents.md`, `docs/staff-triggers.md`), packs/marketplace +(`docs/marketplace.md`), session search (`src/server/search/`), sandbox/worktrees, the +planned extension platform ([extension-platform.md](extension-platform.md)). + +--- + +## §2 Comparison matrix + +✅ strong · 🟡 partial · ❌ absent. The right column says where the gap is handled. + +| Axis | OpenClaw | Hermes | Claude Code | Bobbit | Disposition | +|---|---|---|---|---|---| +| Goal/workflow orchestration, verification gates | 🟡 Task Flow | 🟡 kanban/delegate | 🟡 coordinator mode, subagents | ✅ | Bobbit ahead — keep | +| Sandbox & worktree isolation | 🟡 | 🟡 Docker | 🟡 | ✅ | Bobbit ahead — keep | +| Plugin/extension ecosystem | ✅ 143 extensions + ClawHub | 🟡 skills + agentskills.io hub | 🟡 plugins | 🟡 packs (platform designed, landing) | Covered by extension-platform.md | +| Cross-session memory (automatic) | ✅ memory extensions + workspace files | ✅ MEMORY/USER.md + per-turn review | ✅ memdir + autoDream | ❌ (staff memory only; search unused at prompt time) | **G1** | +| Self-improving skills | 🟡 (static skills, hub) | ✅ per-turn review + curator | 🟡 autoDream (memory only) | ❌ | **G2** → autoimprovement.md | +| Scheduled automation | ✅ cron + heartbeat | ✅ cron jobs w/ skills | 🟡 | ✅ staff `schedule` triggers | parity | +| One-shot "remind me at" | ✅ `--at` | ✅ one-shot schedules | 🟡 | ❌ | **G3** (easy) | +| Ambient/idle automation (heartbeat, commitments) | ✅ heartbeat + inferred commitments | 🟡 idle-triggered curator | 🟡 autoDream gates | ❌ | **G3** | +| Standing orders / persistent authority | ✅ first-class docs/convention | 🟡 SOUL.md | 🟡 CLAUDE.md | 🟡 role/staff prompts (no convention) | **G4** (easy, docs+templates) | +| Away summary / catch-up | 🟡 tasks audit | ❌ | ✅ `awaySummary` (small-fast model) | ❌ (unread dots only) | **G5** (easy) → mission-control.md briefing | +| Messaging channels (Telegram/WhatsApp/…) | ✅ 30+ channels, pairing, access groups | 🟡 Discord/telephony tools | ❌ (terminal/IDE/web) | ❌ | **G6** (strategic) | +| Companion / fun layer | 🟡 | ❌ | ✅ `buddy` (species/rarities/stats) | ✅ bobbit sprites | parity; see §G7 | +| Config doctor / migrations | ✅ `openclaw doctor --fix` | 🟡 | 🟡 | ❌ | **G8** (medium) | +| Cross-harness import | ✅ `migrate-claude`, `migrate-hermes` extensions | ✅ openclaw skill imports | ❌ | 🟡 Claude-plugin compat planned | **G9** (easy, extends planned work) | +| Onboarding for non-coders | ✅ companion apps, first-run UX priority | 🟡 | ✅ `projectOnboardingState`, MagicDocs, PromptSuggestion | 🟡 add-project flow | **G10** → mission-control.md | +| Multi-model / aux-model routing | ✅ per-extension providers | ✅ auxiliary slots (curator, vision, compression…) | ✅ small-fast-model pattern | 🟡 per-role models | **G11** (covered partly by token-cost CE-G6) | +| Trajectory datagen for training | ❌ | ✅ batch runner + `trajectory_compressor.py` | ❌ | ❌ | out of scope (note only) | +| Voice / TTS | ✅ | 🟡 | ❌ | ❌ | future; rides G6 channels | + +--- + +## §3 Per-gap analysis + +### G1 — Automatic cross-session memory (HIGH value, M effort) + +What peers do: + +- **Hermes**: two bounded files — `MEMORY.md` (2,200 chars) + `USER.md` (1,375 chars) — + injected as a frozen snapshot at session start (cache-safe), managed by the agent via a + `memory` tool with add/replace/remove, and **fed by a per-turn background review fork** + (`agent/background_review.py`) that asks "did the user reveal preferences/expectations + worth saving?". Bounded size forces consolidation; usage % is shown in the header + (`website/docs/user-guide/features/memory.md`). +- **Claude Code**: `memdir` auto-memory, `services/extractMemories` (per-session extraction), + `services/SessionMemory`, and `services/autoDream` consolidation (G2). +- **OpenClaw**: workspace bootstrap files auto-injected every session (`AGENTS.md`, `SOUL.md`, + `USER.md`, `MEMORY.md`, `IDENTITY.md`…) plus pluggable memory backends + (`extensions/memory-core`, `memory-lancedb`, `memory-wiki`, `active-memory`). + +Bobbit today: staff agents have pinned memory; sessions have nothing; the FlexSearch session +index exists but **nothing consults it at prompt time** (already identified as the provider +gap in extension-platform.md §2). + +Disposition: the *retrieval* half is the extension platform's `session-memory` reference pack +— don't duplicate. The cheap, immediately-valuable half is **the Hermes-style bounded user +profile**: a global `USER.md`-equivalent (server-scoped, ~500 tokens, agent-managed via a +small tool, injected as a stable prompt section through the existing `PromptParts` assembly +in `src/server/agent/system-prompt.ts`). Frozen-snapshot semantics preserve the cache prefix +(matches Bobbit's existing cache discipline per token-cost-efficiency §1). The *writing* half +(when to save) belongs to the autoimprovement loop (→ autoimprovement.md §loop, "observe"). + +### G2 — Self-improving skills (HIGH value; the subject of its own doc) + +Hermes is the reference implementation: per-turn background review with a tool whitelist +(`agent/background_review.py` — "most sessions produce at least one skill update"; patch-first +preference order; class-level umbrella skills) plus the idle-triggered **curator** +(`agent/curator.py`, `website/docs/user-guide/features/curator.md`): lifecycle +`active → stale → archived`, never-delete, pin/rollback/dry-run, tar.gz snapshots before every +pass, aux-model review. Claude Code's `autoDream` shows the gate discipline (time gate → +session-count gate → lock, forked subagent, first-class `DreamTask`). Full adopt/adapt/reject +analysis and Bobbit's design: **[autoimprovement.md](autoimprovement.md)**. + +### G3 — One-shot triggers + ambient automation (easy win) + +OpenClaw's automation taxonomy (`docs/automation/index.md`) distinguishes exact-time cron, +one-shot `--at`, **heartbeat** (flexible periodic batched checks with main-session context), +background-task ledger (`openclaw tasks list/audit`), and **inferred commitments** ("user +mentioned an interview → check in after"). Hermes cron jobs attach skills and persist per-job +output dirs (`cron/jobs.py`). + +Bobbit has the staff `schedule` (cron) trigger and the inbox. Missing, in increasing effort: + +1. **One-shot trigger** — `{ type: "schedule", config: { at: ISO } }` self-disabling after + fire. Small change to `staff-trigger-engine.ts` + trigger editor UI. Unlocks "remind me / + check on X at 5pm" via any staff. *(easy win)* +2. **Goal-event coverage** — triggers already cover `goal_created`/`goal_archived`; add + `gate_failed` and `session_errored` push triggers (same dispatcher pattern, + `goal-trigger-dispatcher.ts`) so system staff can react to failures. *(easy win)* +3. **Heartbeat-style digest** — not a new mechanism: a cron-triggered system staff whose + prompt says "scan what changed; act only if needed" (→ mission-control.md Observer). + Inferred commitments are deliberately **not** adopted as machinery — they're a prompt + pattern for Global Staff, documented in the standing-orders template (G4). + +### G4 — Standing orders convention (easy win, mostly docs) + +OpenClaw's `docs/automation/standing-orders.md` formalizes *permanent operating authority*: +each program defines **Scope / Trigger / Approval gates / Escalation rules**, lives in +auto-injected workspace files, and pairs with cron for enforcement ("cron says when, standing +orders say what you may do"). This is precisely the right authoring convention for Bobbit +staff system prompts — and it's what makes the autoimprovement trust ladder legible. + +Disposition: adopt as **convention + template, zero machinery**: a documented standing-order +template in `docs/staff-agents.md` (or a new `docs/standing-orders.md`), the staff-creation +assistant (`src/server/agent/staff-assistant.ts`) nudged to structure proposed prompts that +way, and the mission-control system-staff roster (mission-control.md §5) written in that +format from day one. + +### G5 — Away summary / catch-up (easy win) + +Claude Code's `services/awaySummary.ts` + `hooks/useAwaySummary.ts`: when the user returns +after a gap, a **small/fast model** (`getSmallFastModel()`) summarizes what happened while +they were away, seeded with session memory. Bobbit's equivalent surface already exists — the +staff inbox and unread dots — but nothing *synthesizes*. Disposition: the **Morning briefing / +Observer staff** in mission-control.md §"Added ideas" is the strategic home; a tactical +slice (per-session "while you were away" header when unread count is high, generated by a +cheap model) can ship independently and reuses the compaction-summary plumbing. *(easy win +for the tactical slice)* + +### G6 — Channels (strategic; the "anyone, not just coders" play) + +OpenClaw's moat: 30+ channel docs (`docs/channels/` — WhatsApp, Telegram, Signal, Discord, +Slack, iMessage, Matrix, SMS, Teams…), device pairing, per-group access controls, broadcast +groups, bot-loop protection. For Bobbit's small-business-owner audience this is the +difference between "a dev tool I open" and "an assistant I text". + +Disposition: **channels as packs**, riding the extension platform — a channel is exactly a +pack `runtime` (long-lived supervised service) + `provider` (session bridge) per +extension-platform.md §3. Do not build bespoke channel code in core. Sequencing: after +extension-platform P5 (runtimes). First channel: **Telegram** (simplest bot API, no phone +pairing). The flight-recorder/notification-policy work in mission-control.md defines what +gets *pushed* to a channel. Not an easy win — listed for the roadmap because of strategic +weight. + +### G7 — Fun/companion layer (parity, one borrowed idea) + +Claude Code ships a full companion pet (`src/buddy/` — species, hats, eyes, **rarities with +weights, named stats**) — gamified identity for what Bobbit already does better with live +status sprites. Borrowed idea worth a note in mission-control.md: Bobbit accessories already +exist per staff/role; a light "collection" surface (which accessories your staff have earned) +is cheap delight. No goal proposed; tracked as an idea only. + +### G8 — `bobbit doctor` (medium) + +OpenClaw treats config drift as a first-class product problem: breaking config changes ship +with a doctor migration; `openclaw doctor --fix` detects old shapes, explains, backs up, +rewrites (VISION.md "Configuration compatibility"). Bobbit's config cascade +(`docs/internals.md`) has no equivalent; stale `.bobbit/config` shapes surface as runtime +errors. Disposition: a `bobbit doctor` CLI subcommand (checks: config schema, orphaned staff, +stale worktrees, pack integrity, state-dir locks) — natural pairing with the Caretaker staff +(mission-control.md), which can *run* doctor checks on schedule and inbox the findings. + +### G9 — Cross-harness import (easy win, extends planned work) + +OpenClaw ships `extensions/migrate-claude` and `extensions/migrate-hermes`; Hermes auto-uses +skills imported from OpenClaw (`~/.hermes/skills/openclaw-imports/`, +`tests/skills/test_openclaw_migration.py` shows hardening). Bobbit already plans Claude-plugin +compat in the marketplace ([pack-based-marketplace.md](pack-based-marketplace.md)); extend the +same staging-adapter seam (`marketplace-install.ts`) with **import adapters for Hermes skills +and OpenClaw workspace skills** (both are SKILL.md-shaped already). One adapter module per +format + fixture-pack tests. Grows the pack catalog for free. + +### G10 — Onboarding (handled in mission-control.md) + +Claude Code invests heavily here (`projectOnboardingState.ts`, `services/MagicDocs`, +`services/PromptSuggestion`, `outputStyles`); OpenClaw makes first-run UX a stated top +priority. Bobbit's answer is the Mission Control first-run flow (mission-control.md §1): +land in a chat that can set everything up. Output styles (tone/verbosity presets per +audience) are noted as a cheap follow-on — a role-prompt preset, no machinery. + +### G11 — Auxiliary-model routing (note) + +Hermes routes every side task (curator, vision, compression, session search) through named +`auxiliary.<task>` model slots with one picker UI. Bobbit has per-role models and the +token-cost plan adds a cheap summarizer (CE-G6.1). When autoimprovement lands its review/judge +passes, give them named aux slots from day one rather than hardcoding models — one config +surface, matching `per-role-model-overrides.md` conventions. + +--- + +## §4 Things examined and deliberately not adopted + +- **Inferred commitments as machinery** (OpenClaw) — prompt-pattern value without the + false-positive machinery; folded into standing-orders templates (G4). +- **Trajectory datagen / model training** (Hermes `batch_runner.py`, + `trajectory_compressor.py`) — out of charter for a harness; transcripts are already + retained if this ever changes. +- **TUI parity** — peers are terminal-first; Bobbit's web UI is the differentiator. The cost + is the battery discipline documented in + [client-performance-battery.md](client-performance-battery.md). +- **Bespoke channel integrations in core** — channels arrive as packs or not at all (G6). + +--- + +## §5 Prioritized roadmap + +Easy wins first; each row names the owning doc/goal so this table stays a router, not a +second backlog. + +| # | Item | Gap | Effort | Owner | +|---|---|---|---|---| +| R1 | Animation/timer battery fixes | — | S–M | [client-performance-battery.md](client-performance-battery.md) P1–P2 | +| R2 | One-shot (`at`) staff trigger + `gate_failed`/`session_errored` push triggers | G3 | S | new goal off this doc | +| R3 | Standing-orders template + staff-assistant nudge | G4 | S (docs) | new goal off this doc | +| R4 | "While you were away" summary slice (cheap model) | G5 | S–M | tactical slice; strategic home is mission-control.md briefing | +| R5 | Bounded global user-profile memory (Hermes USER.md pattern) | G1 | M | new goal; prompt-section + tool; coordinate with extension-platform `session-memory` | +| R6 | Hermes/OpenClaw skill import adapters | G9 | M | extends marketplace adapter seam | +| R7 | Mission Control core (global scope, meta-tool, system staff) | G3/G5/G10 | L | [mission-control.md](mission-control.md) | +| R8 | Autoimprovement loop (review, curator, trust ladder) | G2 | L | [autoimprovement.md](autoimprovement.md) | +| R9 | `bobbit doctor` | G8 | M | new goal; pairs with Caretaker staff | +| R10 | Telegram channel pack (first channel) | G6 | L | after extension-platform runtimes phase | +| R11 | Aux-model slots for review/judge passes | G11 | S (when R8 lands) | folded into autoimprovement phases | + +Implementation notes for R2–R4 (the goals this doc owns directly, written to be executable +without re-deriving context): + +- **R2**: extend `StaffTrigger` config union in `src/server/agent/staff-store.ts` with + `{ at: string }` alongside `{ cron }`; `staff-trigger-engine.ts` fires once when + `now >= at && !lastFired`, then sets `enabled: false`; trigger-editor UI on the staff edit + page gains a datetime field; e2e test beside the existing trigger specs. For the push + triggers, mirror the `goal_created` wiring in `goal-trigger-dispatcher.ts` from the gate / + session-status mutation points, required-prompt rule included (`docs/staff-triggers.md`). +- **R3**: add the Scope/Trigger/Approval/Escalation template to `docs/staff-agents.md`; update + the staff assistant's proposal guidance in `src/server/agent/staff-assistant.ts`; seed the + mission-control pack roster (mission-control.md §5) in that format. +- **R4**: server: on session activation with unread count above threshold, run a + cheap-model summarization over messages since `lastSeen` (reuse the compaction-summary + path; aux-model slot per G11); client: collapsible "while you were away" banner above the + transcript tail. Behind a setting; browser E2E for banner render + dismissal persistence. diff --git a/docs/design/mission-control.md b/docs/design/mission-control.md new file mode 100644 index 000000000..c05e48309 --- /dev/null +++ b/docs/design/mission-control.md @@ -0,0 +1,321 @@ +# Mission Control — the global scope, global staff, and the flight recorder + +Status: proposal / implementation blueprint. Each phase in §8 converts 1:1 into a mission +goal with acceptance criteria; steps name exact files, functions, and reuse points so they +can be executed without re-deriving this analysis. + +**What this is:** a first-class **global scope** at the top of the sidebar — the main entry +point for using Bobbit. Mission Control sessions can set up new projects, talk to any agent +in any project, create goals (and goals-of-goals) anywhere, and manage everything Bobbit can +do — because everything is an API call. It is home to **Global Staff** (the user's personal +assistants) and **Global System Staff** (Bobbit's own housekeeping crew: cleanup, +memory/housekeeping scheduling, dreaming, observability, and the autoimprovement loop from +[autoimprovement.md](autoimprovement.md)). And it is where the **flight recorder** lives — +the audit surface that makes growing agent autonomy trustworthy. + +**Why now:** Bobbit's destination is "the best harness out there, usable by anyone — coders, +researchers, small-business owners". Today the entry point is a *project*, which presumes the +user already thinks in repos. Mission Control inverts that: you land in one conversation that +can do everything, and projects become things it sets up *for* you +([harness-gap-analysis.md](harness-gap-analysis.md) G10). Peers point the same direction — +OpenClaw's assistant-first framing and standing-orders authority model, Claude Code's +away-summaries and background dream tasks — but none of them has Bobbit's goals/staff/packs +substrate to anchor it. + +--- + +## §1 UX — one place that runs everything + +- **Placement:** a pinned **Mission Control** entry at the very top of the left sidebar, + above all projects — rendered by `renderSidebar()` (`src/app/sidebar.ts:1458`) and the + mobile shell (`render.ts::renderSidebarShellMobile`), expandable like a project section: + its sessions, its staff (Global Staff and System Staff sub-grouped), and an **Activity** + entry (the flight recorder, §6). It does not participate in project reordering + (`docs/sidebar-project-reorder.md` — hidden/system projects already don't). +- **First run:** instead of today's empty-state dead end, a new user lands in a Mission + Control chat that interviews them and sets up their first project(s) via the `bobbit` + meta-tool (§3) — reusing the existing add-project preflight and project-onboarding flows + (`docs/add-project-preflight.md`, `docs/design/project-onboarding-ux.md`) rather than + replacing them: Mission Control *drives* the same proposals the UI drives. +- **Daily rhythm:** the **morning briefing** (§7.1) — overnight activity, costs, pending + proposals, stuck goals — arrives as an inbox digest; the user reads it in the Mission + Control chat and acts by replying. +- **Tone:** Mission Control speaks plain language. A small-business owner should be able to + say "set up a folder for my invoices and remind me every Friday to send them" and get a + project + a scheduled Global Staff trigger, never seeing the words "worktree" or "cron". + +## §2 The global scope (core) + +The synthetic hidden project already exists: `SYSTEM_PROJECT_ID = "system"` +(`src/server/agent/project-registry.ts:41`). Today it is a quarantine zone — staff stored +under it are "orphans" to re-home (`docs/staff-agents.md` §Legacy staff records). Mission +Control promotes it to a first-class, still-synthetic, un-deletable scope: + +- **Visibility:** keep it out of `visibleProjects()` (`project-registry.ts:117`) — it is not + a project row; the sidebar renders it as the dedicated top entry instead. No drag-reorder, + no archive, no delete. +- **Sessions:** global sessions attach to the system project. Projectless session handling + already exists (`tests/e2e/sessions-projectless.spec.ts` pins current behavior — extend, + don't fork). **Cwd:** a dedicated `~/.bobbit/mission-control/` workspace dir (a real, + writable directory so file tools work), never a project root and never + `config.defaultCwd`. +- **Staff:** `staff-store.ts` / `staff-manager.ts` accept `projectId: "system"` as a *valid* + owner: project-anchoring validation (staff-agents.md §Project and cwd anchoring) gains an + explicit system-scope branch (cwd must be inside the mission-control workspace; worktree + mode is always off — there is no repo). The orphan detector + (`GET /api/staff/orphaned`) distinguishes **legacy orphans** (no `projectId`) from + **intentionally global** staff (`projectId === "system"` *and* a new `global: true` field + on `PersistedStaff`) so the orphan banner stops flagging them. Legacy records without the + flag remain orphans — no silent migration. +- **Goals:** global-scope goals are **not** introduced. Goals stay project-owned (worktrees, + branches, and gates are repo concepts). Mission Control creates goals *in* target projects + via the meta-tool — which already covers "goals with goals" since the nested-goals API is + project-scoped (`docs/nested-goals.md`). + +## §3 The `bobbit` meta-tool group (core) + +"Manage anything Bobbit-related — everything is an API call." Rather than one tool per REST +endpoint (the context-bloat failure mode), reuse the shipped MCP meta-tool aggregation +pattern (`docs/mcp-meta-tools.md`, `docs/design/mcp-meta-tool-aggregation.md`) against +Bobbit's own gateway: + +- New tool group `defaults/tools/bobbit/` exposing two tools: + - `bobbit(operation, args)` — `operation` constrained by a typo-proof enum; + - `bobbit_describe(operation?)` — returns the JSON schema + doc string for an operation + (lazy discovery, same shape as `mcp_describe`). +- The **operation catalog is a curated allowlist**, not a blind mirror of + `server.ts::handleApiRoute()`: each entry maps `operation → {method, path template, schema, + tier}` in one registry module (`src/server/mcp/bobbit-meta-tool.ts` or sibling — follow + where the MCP meta-tool implementation lives). Initial catalog: projects + (list/add/preflight), sessions (list/create/send-message/archive), goals + (list/create/nest/archive), staff (list/create-proposal/wake/inbox-enqueue), worktrees + (list/cleanup), packs (list/install from registered sources), config (read), costs (read), + activity (read, §6). +- **Safety tiers** enforced in the registry, not the prompt: `read` (free) · `mutate` + (allowed but every call lands in the flight recorder) · `sensitive` (delete project, + bulk-archive, pack install — requires the existing ask/blocking-tool confirmation flow, + `docs/blocking-tools.md` / `docs/non-blocking-ask.md`). Role `toolPolicies` and the tool + guard apply unchanged — this is just a tool group. +- **Auth:** calls execute in-process against the same handlers REST uses (no self-HTTP + loop), attributed to the calling session in the flight recorder. +- **Availability:** in the default tool policy for Mission Control sessions/staff; available + to project sessions only by explicit role/tool-policy opt-in. + +Cross-project communication falls out of the catalog: `sessions.send-message` (delivers into +any session's prompt queue, `docs/prompt-queue.md`) and `staff.inbox-enqueue` (the existing +inbox contract, `docs/staff-inbox.md`) — no new channel machinery. + +## §4 Global Staff (user-facing) + +Plain staff agents owned by the global scope: personal assistant, researcher, email/invoice +chaser — anything not tied to a repo. They get the existing machinery unchanged (roles, +accessory, memory, cron/manual triggers, inbox) plus the `bobbit` meta-tool so they can act +across projects. Authoring convention: **standing orders** (Scope / Trigger / Approval gates / +Escalation — gap-analysis G4) baked into the staff-creation assistant's guidance. + +## §5 Global System Staff (Bobbit's own crew) + +Bobbit-discretionary housekeeping staff, **defined as data** (roles + skills + trigger +templates) and shipped as the **mission-control built-in pack** (§7 split). They are real +staff records the user can inspect, pause, retune, or delete — discretion with a leash. Every +run writes to the flight recorder. Roster v1: + +| Staff | Charter (standing-order form) | Triggers | +|---|---|---| +| **Caretaker** | Sweep stale worktrees/branches (builds on `docs/design/orphan-remote-branch-cleanup.md` and the worktree pool), stale archived sessions, dangling sandbox containers. *Approval gate:* deletion lists over N items → inbox proposal first. | weekly `schedule`, `goal_archived` | +| **Archivist** | Transcript/index hygiene; memory consolidation scheduling; **dreaming** — gate-disciplined idle reflection over recent transcripts producing memory/skill proposals (mechanism in autoimprovement.md AI-P3). | nightly `schedule` (gates inside) | +| **Improver** | Runs the autoimprovement loop (autoimprovement.md §7). | `goal_archived`, `gate_failed`, weekly | +| **Observer** | Observability digests: cost/burn-rate, failure patterns, flaky behaviors, perf-flag regressions; assembles the **morning briefing** (§7.1). *Escalation:* anomalies (cost spike, repeated gate failures) → immediate inbox entry, not the next briefing. | daily `schedule`, `session_errored` | + +All four use existing trigger machinery (`docs/staff-triggers.md`) plus the two new push +triggers (`gate_failed`, `session_errored`) shared with gap-analysis R2 and autoimprovement +AI-P1. The pack ships triggers **disabled-by-default except Observer**; enabling the crew is +a one-click, per-staff choice during Mission Control onboarding — discretion is granted, not +presumed. + +## §6 The flight recorder (core) + +Visibility is king: every autonomous action is on the record, queryable, and (where +applicable) revertible. + +- **Store:** append-only JSONL `.bobbit/state/activity.jsonl` (crash-safe append; daily + rotation with size cap), entries: + `{ts, actor: {kind: session|staff|policy|user, id}, action, target, tier, evidence?, + revert?: {kind, ref}, projectId?}`. +- **Writers:** the `bobbit` meta-tool (every `mutate`/`sensitive` call), system-staff wake + results, autoimprovement decisions (autoimprovement.md §3 "Record"), Caretaker sweeps. + One `recordActivity()` helper in the server; writers call it explicitly — no magic + middleware pass that would record noise. +- **Read surface:** `GET /api/activity?since&actor&tier&projectId` (paginated) + a WS event + for live tail; rendered as the **Activity** panel under the Mission Control sidebar entry — + filterable timeline, each row: actor chip, action, target link, evidence link, revert + button when `revert` present. +- **Retention:** keep the JSONL indefinitely (it's small text); the panel reads windowed. + +## §7 Added product ideas riding the same substrate + +1. **Morning briefing** (Observer): daily digest — what ran overnight, cost/burn vs budget, + pending proposals, stuck goals (notification-policy "idle stuck" rule feeds this), + calibration status from autoimprovement shadow mode. Delivered as a staff inbox digest; + later deliverable to external channels when channel packs exist (gap-analysis G6). +2. **Global cost guardrails**: per-project and per-staff soft budgets (config), surfaced in + the briefing and as an Observer escalation on breach; builds on per-session cost tracking + (`docs/session-cost.md`) and the planned CE-G0.1 ledger. Soft-stop only in v1 (warn + + require confirmation for new expensive sessions), never hard-kill mid-goal. +3. **First-run onboarding** (§1): Mission Control chat as the guided setup path. + +## §8 Architectural split — what is core, what is pack + +Decision (user-confirmed): **core with pack-shaped seams**, leaning on the extension platform +([extension-platform.md](extension-platform.md)) where it genuinely fits. Pack panels, +renderers, launchers, and deep-link routes already work today (`src/app/pack-panels.ts`, +`pack-entrypoints.ts`, `#/ext/<routeId>`, the pr-walkthrough built-in pack) — so the split +is honest, not aspirational: + +| Piece | Lives | Why | +|---|---|---| +| Global scope legitimization, staff `global` flag, cwd rules | **core** | project-registry/staff-store invariants; nothing else can own them | +| Sidebar top entry + Activity navigation | **core** | the sidebar shell is core UI; packs cannot (and should not) claim the top slot | +| `bobbit` meta-tool group + operation registry + tiers | **core** | security-sensitive chokepoint; must be budget-pinned and test-pinned like other `defaults/tools/` | +| Flight recorder store + REST/WS | **core** | platform-level audit primitive; packs *write into* it (a natural future Extension-Host API) | +| **mission-control pack** (built-in, first-party — `market-packs/mission-control/`): system-staff roles, skills, standing-order prompts, trigger templates; the Mission Control **dashboard panel** (briefing view, crew status, activity timeline UI) at `#/ext/mission-control` | **pack** | pure content + panel — exactly what `builtin-packs.ts` + `copy-builtin-packs.mjs` + pack panels deliver today (pr-walkthrough precedent) | +| Staff-roster *instantiation* from pack data | core loader seam | v1: a small core bootstrap reads the pack's staff templates ("create crew" action); migrates to the extension platform's `workflows`/provider entities when those phases land — the seam is the template format, which stays pack-owned either way | + +Dependency on extension-platform phases: **none for MC-P0…P4** (panels/routes exist); +the staff-template entity and any provider-based enrichment ride later platform phases +without rework because the data already lives in the pack. + +## §9 Phased implementation plan + +Order matters: P0→P2 are the spine; P3/P4 can proceed in parallel after P1. + +```mermaid +graph LR + P0[MC-P0 global scope] --> P1[MC-P1 sessions + sidebar] + P1 --> P2[MC-P2 bobbit meta-tool] + P1 --> P3[MC-P3 flight recorder] + P2 --> P4[MC-P4 mission-control pack + system staff] + P3 --> P4 + P4 --> P5[MC-P5 briefing + onboarding + budgets] + P4 -.-> AI[autoimprovement.md AI-P2+] +``` + +### MC-P0 — Global scope legitimization + +1. `project-registry.ts`: keep `SYSTEM_PROJECT_ID` hidden from `visibleProjects()`; add + `isSystemScope(id)` helper; ensure the system project's anchor dir is the + `~/.bobbit/mission-control/` workspace (created lazily). +2. `staff-store.ts`: add `global?: boolean` to `PersistedStaff` (loader normalises missing → + `false`); `staff-manager.ts` accepts `projectId: "system"` + `global: true` with + system-scope cwd validation (inside the mission-control workspace) and worktree forced + off. +3. Orphan logic: `GET /api/staff/orphaned` excludes `global: true` records; orphan banner + unchanged for legacy records. Update `docs/staff-agents.md` orphan section. +4. Tests: unit (store normalisation, scope validation); API e2e (create global staff; orphan + endpoint excludes it; legacy orphan still flagged — extend + `tests/staff-orphan-reassign.test.ts` / `tests/e2e/staff-patch-reassign.spec.ts` + patterns). + +Acceptance: a global staff can be created, woken, and edited; zero orphan-banner noise; all +existing staff/project suites green. + +### MC-P1 — Mission Control sessions + sidebar entry + +1. Session creation accepts the system scope (extend the projectless path pinned by + `tests/e2e/sessions-projectless.spec.ts`); cwd = mission-control workspace. +2. Sidebar: pinned top entry in `renderSidebar()` + mobile shell; expandable section listing + global sessions and staff (System Staff sub-grouped); keyboard-nav per + `docs/sidebar-keyboard-navigation.md`; excluded from project reorder. +3. "New Mission Control chat" affordance in the entry header. +4. Tests: browser E2E `tests/e2e/ui/mission-control.spec.ts` — entry visible at top, create + chat, persists across reload, archived session disappears (navigation/happy + path/persistence/cleanup per the AGENTS.md E2E rule). + +Acceptance: a user can converse in a Mission Control session that survives gateway restart +(sessions.json) and reload. + +### MC-P2 — `bobbit` meta-tool + +1. Operation registry module (catalog table in §3): typed entries, tier, schema; unit-test + pinning that every entry resolves to a real route handler and that the enum matches the + registry (the "missing test IS the bug" rule). +2. Tool group `defaults/tools/bobbit/` (two tools; description budget — extend + `tests/tool-description-budget.test.ts` pins). +3. In-process dispatch into the same handler path as REST; tier enforcement: `sensitive` → + blocking-ask flow; `mutate`/`sensitive` → `recordActivity()` (stub until MC-P3, then + real). +4. Default tool policy wiring for system-scope sessions/staff. +5. Tests: API e2e — meta-tool session creates a project, creates a nested goal in it, sends + a message to another session, enqueues a staff inbox entry; `sensitive` op blocks on ask; + describe returns schemas. + +Acceptance: a Mission Control session performs the §3 catalog end-to-end against a temp +project; no operation outside the allowlist is callable. + +### MC-P3 — Flight recorder + +1. `recordActivity()` + JSONL store (+ rotation) in `src/server/agent/activity-store.ts`; + REST `GET /api/activity` with filters; WS live-tail event (scoped fanout — follow the + goal-fanout lesson from `reduce-server-cpu-experiment-goal-fanout.md`: only explicitly + subscribed sockets). +2. Activity panel UI under the Mission Control sidebar entry (core view; the pack panel in + MC-P4 may embed the same component). +3. Writers: meta-tool (MC-P2 stub flips on), staff wake completion summaries. +4. Tests: unit (rotation, filter math); browser E2E (action appears live in panel; filters; + revert button only when `revert` present). + +Acceptance: every `mutate` meta-tool call from MC-P2's e2e is visible in the panel with +actor attribution. + +### MC-P4 — mission-control pack: system staff + dashboard panel + +1. `market-packs/mission-control/`: roles + skills + standing-order prompts + trigger + templates for the §5 roster; panel + route (`#/ext/mission-control`) following the + pr-walkthrough pack layout; ships via `scripts/copy-builtin-packs.mjs` + + `builtin-packs.ts`. +2. Core "create crew" bootstrap: instantiate staff records from pack templates (one REST + action, idempotent, user-invoked from the panel; Observer trigger enabled, others + disabled per §5). +3. New push triggers `gate_failed` / `session_errored` in `goal-trigger-dispatcher.ts` + (shared dependency with autoimprovement AI-P1 and gap-analysis R2 — land once, here or + there, whichever goal runs first). +4. Tests: pack-litmus tests per the `market-packs/artifacts` convention; e2e — create crew ⇒ + four staff exist under the global scope with correct triggers; Caretaker dry-run sweep + writes a flight-recorder entry; panel deep-link survives reload. + +Acceptance: one click creates the crew; a scheduled Observer tick produces an inbox digest; +everything visible in the activity panel. + +### MC-P5 — Briefing, onboarding, budgets + +1. Observer briefing prompt + digest formatting (inbox entry kind reused); "stuck goals" + sourced from the notification-policy predicate. +2. First-run flow: when no projects are registered, land on Mission Control with the + onboarding prompt (drives the existing add-project proposal flow). +3. Budgets: config keys (per-project/per-staff monthly soft budget), Observer breach + escalation, briefing line item. +4. Tests: browser E2E for first-run path (fresh state dir ⇒ Mission Control onboarding ⇒ + project created via chat proposal); unit for budget math; e2e for breach escalation + entry. + +Acceptance: the §1 "small-business owner" sentence works end-to-end on a clean install. + +--- + +## §10 Open questions (for review, not blockers) + +- Sidebar naming: "Mission Control" vs something shorter ("HQ", "Home"). UX copy decision; + the plan uses Mission Control throughout. +- Should project sessions be able to *see* the activity feed (read-only) via the meta-tool + `read` tier? Lean yes — visibility is king cuts both ways. +- Crew model defaults: system staff should likely default to a cheaper model via role + `model` overrides (`per-role-model-overrides.md`) — decide per-staff during MC-P4 with + token-cost guidance. + +Cross-references: [autoimprovement.md](autoimprovement.md) · +[harness-gap-analysis.md](harness-gap-analysis.md) (G3/G4/G5/G6/G10) · +[client-performance-battery.md](client-performance-battery.md) (Observer watches the perf +baseline once the harness exists) · [extension-platform.md](extension-platform.md) · +[token-cost-efficiency.md](token-cost-efficiency.md) (budgets, aux models) · +`docs/staff-agents.md` · `docs/staff-triggers.md` · `docs/staff-inbox.md` · +`docs/mcp-meta-tools.md` · `docs/rest-api.md`. From 1df92968a6b5c9546e1526febc4e11eae24786c3 Mon Sep 17 00:00:00 2001 From: Claude <noreply@anthropic.com> Date: Wed, 10 Jun 2026 22:21:37 +0000 Subject: [PATCH 06/18] Add implementation contracts + program execution plan to fable design docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - docs/design/fable-program-execution-plan.md (new) — single sequencing source of truth across all fable-docs workstreams: binding rules (reuses extension-platform-implementation-plan §0 as universal DoD), doc-is-the-spec deviation protocol, shared-seam coordination, lane dependency graph, full PR slicing with effort + mergeability rationale, and a master checklist for the orchestrating session. - mission-control.md Appendix A — exact constants/storage layout, PersistedStaff.global validation matrix, complete bobbit meta-tool operation catalog v1 (op → tier → REST), ActivityEntry interface + REST/WS contract, mission-control pack layout, owned files per phase. - autoimprovement.md Appendix A — ImprovementProposal/ChangeClass types, JSONL/snapshot/lock storage layout, REST table, tool + rubric placement (pack-owned, never hardcoded), settings keys with ceilings, owned files per phase. - client-performance-battery.md Appendix A — complete classified inventory of every infinite animation (incl. goal-dashboard.css), animation-power.ts contract, perf report shape, poll-site checklist, PR boundaries. - harness-gap-analysis.md — concrete R2 trigger schema (one-shot `at`, gate_failed/session_errored) so trigger work cannot drift. https://claude.ai/code/session_01LBW68Z2KvA3EhQMq3c2wrR --- docs/design/autoimprovement.md | 99 ++++++++++++++ docs/design/client-performance-battery.md | 99 ++++++++++++++ docs/design/fable-program-execution-plan.md | 143 ++++++++++++++++++++ docs/design/harness-gap-analysis.md | 15 ++ docs/design/mission-control.md | 136 +++++++++++++++++++ 5 files changed, 492 insertions(+) create mode 100644 docs/design/fable-program-execution-plan.md diff --git a/docs/design/autoimprovement.md b/docs/design/autoimprovement.md index 1857b552d..00fe6615a 100644 --- a/docs/design/autoimprovement.md +++ b/docs/design/autoimprovement.md @@ -309,3 +309,102 @@ recorder, inbox digests) · [harness-gap-analysis.md](harness-gap-analysis.md) G [token-cost-efficiency.md](token-cost-efficiency.md) (cost ledger CE-G0.1, aux summarizer) · [skill-ux-and-autonomous-activation.md](skill-ux-and-autonomous-activation.md) · [editable-proposals.md](editable-proposals.md) · [human-signoff-gates.md](human-signoff-gates.md). +Execution tracking: [fable-program-execution-plan.md](fable-program-execution-plan.md). + +--- + +## Appendix A — Implementation contracts (definite lists; do not re-derive) + +Universal definition-of-done: +[extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) +binds every AI phase. + +### A.1 Types (AI-P1, `src/server/agent/improvement-store.ts`) + +```ts +export type ChangeClass = + | "memory" | "skill-new" | "skill-patch" | "skill-lifecycle" + | "prompt-context" | "tool-new" | "config"; // §2 table is the law for ceilings + +export type ProposalStatus = + | "draft" | "approved-human" | "approved-policy" | "rejected" + | "applied" | "reverted" | "superseded"; + +export interface ImprovementProposal { + id: string; // nanoid, same generator as session ids + ts: string; // ISO-8601 UTC + changeClass: ChangeClass; + title: string; // one line, imperative + diff?: string; // unified diff for patches + files?: { path: string; content: string }[]; // for skill-new (paths relative to the learned-skills pack root) + evidence: { + signal: string; // which observer signal fired (e.g. "user-correction", "gate_failed") + sessionIds: string[]; + excerpts: string[]; // verbatim transcript quotes, ≤ 500 chars each, ≤ 5 + expectedImpact: string; + rollbackPlan: string; + }; + confidence?: number; // 0..1, judge-assigned; absent until judged + judge?: { model: string; rubricVersion: string; wouldAutoApproveAt: number[] }; // thresholds it clears + status: ProposalStatus; + decision?: { by: "human" | "policy"; ts: string; reason?: string; editedBeforeApprove?: boolean }; + applied?: { ts: string; snapshotRef?: string }; + outcome?: "helped" | "neutral" | "regressed"; // AI-P6 +} +``` + +Storage: `.bobbit/state/improvement/proposals.jsonl` (append-only; status changes append a +new full record with same `id` — last-writer-wins on load, crash-safe per +[session-store-crash-safety.md](session-store-crash-safety.md)). Snapshots: +`.bobbit/state/improvement/snapshots/<iso>/…tar.gz`. Skill archive: +`.bobbit/state/improvement/skill-archive/`. Dream lock: +`.bobbit/state/improvement/dream.lock` (mtime = lastDreamAt; same gate semantics as +autoDream §1). + +### A.2 REST (AI-P1/P4/P5) + +| Route | Notes | +|---|---| +| `GET /api/improvements?status&class&limit` | newest-first | +| `POST /api/improvements` | from the `propose_improvement` tool path only | +| `POST /api/improvements/:id/decision` | body `{approve: boolean, reason?, editedFiles?}` | +| `POST /api/improvements/:id/revert` | restores snapshot, appends `reverted`, demotes class | +| `GET /api/improvements/calibration?class` | `{class, decisions: n, perThreshold: [{theta, agreement, falseApproves}]}` | + +### A.3 Tool + prompts + +`defaults/tools/proposals/propose_improvement.yaml` — params: +`change_class, title, diff?, files?, evidence` (mirror the `propose_staff` YAML shape; +provider `bobbit-extension`). The Improver's review prompt, judge rubrics (one per +`ChangeClass`, versioned `rubricVersion: "1"`), and the dream prompt live in the +mission-control pack: `market-packs/mission-control/skills/improver/…` — **never hardcoded +in server source**, so prompt iteration is a pack update, not a release. + +### A.4 Settings keys (AI-P5; config cascade) + +```yaml +autoimprovement: + enabled: true # global kill switch — false drops everything to L0 behavior + levels: # server clamps to §2 ceilings regardless of config + memory: 0 # 0=L0, 0.5=shadow, 1=L1, 2=L2 + skill-new: 0 + skill-patch: 0 + skill-lifecycle: 0 + thresholds: { memory: 0.85, skill-new: 0.85, skill-patch: 0.9, skill-lifecycle: 0.9 } + curator: { intervalHours: 168, minIdleHours: 2, staleAfterDays: 30, archiveAfterDays: 90 } + dream: { minHours: 24, minSessions: 5 } +auxiliary: # gap-analysis G11; resolution mirrors per-role-model-overrides.md + improver-judge: { model: auto } + dreamer: { model: auto } +``` + +### A.5 Owned files per phase (PR boundaries) + +| Phase | New | Modified | +|---|---|---| +| AI-P1 | `improvement-store.ts`, `defaults/tools/proposals/propose_improvement.yaml` (+extension wiring), proposal-panel parser/view for kind `improvement` | `server.ts` (routes), `proposal-registry.ts`, `proposal-parsers.ts`, `goal-trigger-dispatcher.ts` (push triggers — coordinate with MC-P4/GA-R2, land once) | +| AI-P2 | improver role/skills in mission-control pack; judge module `src/server/agent/improvement-judge.ts`; learned-skills pack bootstrap in `src/server/agent/learned-skills-pack.ts` | skill-load usage hook in `slash-skills.ts` | +| AI-P3 | curator module `src/server/agent/improvement-curator.ts` (+snapshot/rollback helpers) | archivist role (pack), staff page curator controls | +| AI-P4 | calibration math in `improvement-store.ts` + report endpoint | Improver staff page section | +| AI-P5 | — | settings page, `improvement-store.ts` (policy path), `activity-store.ts` writers, inbox digest | +| AI-P6 | outcome evaluator in `improvement-curator.ts` | — | diff --git a/docs/design/client-performance-battery.md b/docs/design/client-performance-battery.md index 90fe40541..fae0e3bfd 100644 --- a/docs/design/client-performance-battery.md +++ b/docs/design/client-performance-battery.md @@ -318,3 +318,102 @@ Cross-references: [token-cost-efficiency.md](token-cost-efficiency.md) (the sibl (the Observer staff can watch for perf regressions once the harness exists), [harness-gap-analysis.md](harness-gap-analysis.md) §battery for how peer harnesses avoid this class of cost (mostly by being TUIs — Bobbit's bar is higher because it ships a real UI). +Execution tracking: [fable-program-execution-plan.md](fable-program-execution-plan.md). + +--- + +## Appendix A — Implementation contracts (definite lists; do not re-derive) + +The universal definition-of-done in +[extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) +applies to every phase here. This appendix removes the remaining judgment calls. + +### A.1 Complete `infinite`-animation inventory and classification + +Source of truth: `grep -rn "animation:.*infinite" src --include=*.css` (re-run before +starting; if new rules appeared, classify them with the same three buckets and extend this +table in the same PR). + +Buckets: **ambient** = pause under `html.anims-paused` (P1 step 2) · **status** = must run +*only* under a working/busy state selector (P1 step 3) · **keep** = semantically must keep +running (rare; justify inline). + +| File:line | Keyframes | Bucket | Note | +|---|---|---|---| +| `src/app/app.css:108` | `status-pulse` | status | also FX3: rewrite off `box-shadow` | +| `src/app/app.css:478` | `gentle-float` | ambient | empty-state decoration | +| `src/app/app.css:943` | `unseen-dot-pulse` | ambient | FX3 rewrite; already has reduced-motion opt-out — keep | +| `src/app/app.css:967` | `bobbit-unread-tap` | ambient | | +| `src/app/app.css:985` | `bobbit-sidebar-unread-blink` | ambient | `steps()` — cheap but still wakes compositor | +| `src/app/app.css:1007` | `sidebar-heartbeat` | status | | +| `src/app/app.css:1026` | `sidebar-conic-spin` | status | spinner — runs only while loading; verify it unmounts | +| `src/app/app.css:1064,1075` | `gate-wave`, `gate-blink` | status | gate in-progress only | +| `src/app/app.css:1102` | `title-wave` | ambient | | +| `src/app/app.css:1111` | `pr-conflict-pulse` | ambient | FX3 rewrite | +| `src/app/goal-dashboard.css:781,1039` | `phase-glow`, `phase-glow-rejected` | status | dashboard visible + phase active only | +| `src/app/goal-dashboard.css:793,1070` | `phase-dot-pulse` | status | | +| `src/app/goal-dashboard.css:796` | `status-indicator-pulse` | status | | +| `src/app/goal-dashboard.css:944` | `gate-dot-pulse` | status | | +| `src/ui/app.css:1117,1140` | `shimmer` ×2 | ambient | loading placeholder; disabled entirely in battery-saver | +| `src/ui/app.css:1166` | `ask-heartbeat` | status | pending-ask only | +| `src/ui/app.css:1239,1411,1412,1521,1586+` | `blob-busy-*` family (move/eyes/shadow) | status | already gated on the busy blob — verify the element unmounts when idle | +| `src/ui/app.css:1526,1669` | `blob-*-idle-sleep-breathe` | **ambient** | the idle/sleep "breathing" loops — these are the FX2 headline: idle sprites must be static | +| `src/ui/app.css:1555,1700,1775,…` | `blob-compact-squash(-rigid)` (all ~10 sites) | ambient | several use `!important` — remove the `!important` when adding the gate | +| `src/ui/app.css:1755–2089` | `magnifier-depth-idle` (×7), `wand-depth-idle` | ambient | accessory idle loops | +| `src/ui/app.css:2041,2050` | `flask-bubbles` ×2 | ambient | | +| `src/ui/app.css:2154,2165` | `wand-sparkle-a/b` | ambient | | +| `src/app/app.css:359–474` | `bobbit-bob`, `bobbit-breathe`, `bobbit-eyes(-s)`, squish family | status | sidebar/chat sprites: working state only (FX2) | + +### A.2 `animation-power.ts` contract + +```ts +// src/app/animation-power.ts +export function initAnimationPower(): void; // called once from main.ts after state init +// Internally: +// shouldPause = document.hidden +// || !anySessionWorking() // derive from state.gatewaySessions statuses +// || isBatterySaverOn() // P3.3; returns false until that phase lands +// || matchMedia("(prefers-reduced-motion: reduce)").matches +// → document.documentElement.classList.toggle("anims-paused", shouldPause) +// Inputs: "visibilitychange" listener + a state subscription invoked from the same +// place session status updates already call renderApp(). No polling. No timers. +// Perf-flag: if isPerfFlagEnabled("-pauseAmbientAnims") → never add the class. +``` + +CSS gate (one block per file, appended at end): +`html.anims-paused :is(<every ambient selector from A.1>) { animation-play-state: paused; }` +Never use a bare `html.anims-paused *` rule — it would freeze one-shot transitions mid-flight. + +### A.3 `perf-monitor` report shape + +```ts +// window.__bobbitPerf.report() → +{ windowMs: number, + rendersPerSec: number, // renderApp rAF executions + timerWakeups: { perSec: number, byCallsite: Record<string, number> }, + longTasks: { count: number, totalMs: number }, + runningAnimations: number, // document.getAnimations().length, last sample + wsFramesPerSec: Record<string, number> } // keyed by frame type +``` + +### A.4 Poll-site checklist (the loops; one-shots are out of scope) + +| Site | Today | Required end state | +|---|---|---| +| `src/app/api.ts:273` `startSessionPolling` | 5 s, visibility-gated | FX5: 60 s while WS healthy; 5 s only while WS down; refresh on reconnect + visible | +| `src/app/goal-dashboard.ts:1090` `startLiveVerifTimer` | 1 s full `renderApp()` | FX6: scoped elapsed-label update only | +| `src/app/render-helpers.ts:338` | 60 s `renderApp()` | keep (within budget) | +| `src/app/session-manager.ts:2821` git-status poll | 30 s, gated, coalesced | keep — this is the reference pattern | +| remaining `setInterval` loops in `goal-dashboard.ts` / `dialogs.ts` | unaudited | P2.4: classify from the P0 wakeup report; loops get the git-status pattern | + +### A.5 Owned files per phase (PR boundaries) + +- **PB-P0**: new `src/app/perf-monitor.ts`; one guarded hook line in `state.ts::renderApp`; + baseline table in this doc. No behavior change. +- **PB-P1**: new `src/app/animation-power.ts`; `main.ts` (one init call); the four CSS files + in A.1; new `tests/e2e/ui/animation-power.spec.ts`; CSS-pinning unit test + `tests/animation-gate.test.ts` (regex: every `infinite` rule in the four files is either + bucket-listed here or fails). +- **PB-P2**: `api.ts`, `goal-dashboard.ts`, `state.ts` (`renderAppThrottled`), streaming + dispatch call sites in `session-manager.ts`/`message-reducer.ts`; tests beside each. +- **PB-P3**: settings page + `animation-power.ts` (battery-saver input); docs. diff --git a/docs/design/fable-program-execution-plan.md b/docs/design/fable-program-execution-plan.md new file mode 100644 index 000000000..f5fb4eabc --- /dev/null +++ b/docs/design/fable-program-execution-plan.md @@ -0,0 +1,143 @@ +# Fable program — execution plan (hand-off for the main Bobbit session) + +Status: living tracking document. This is the **single sequencing source of truth** for the +work proposed across the fable-docs design set. Hand this file to the orchestrating Bobbit +session; it should create goals from §3, in order, and tick the checklist in §4 as PRs merge. + +The design docs in scope (each owns its *content*; this doc owns *ordering and slicing*): + +| Doc | Workstream prefix | +|---|---| +| [extension-platform.md](extension-platform.md) + [extension-platform-implementation-plan.md](extension-platform-implementation-plan.md) | EP (G1–G9 there) | +| [token-cost-efficiency.md](token-cost-efficiency.md) §6 | CE (CE-G0–G7 there) | +| [client-performance-battery.md](client-performance-battery.md) | PB (P0–P3) | +| [mission-control.md](mission-control.md) | MC (P0–P5) | +| [autoimprovement.md](autoimprovement.md) | AI (P1–P6) | +| [harness-gap-analysis.md](harness-gap-analysis.md) §5 | GA (R2–R6, R9; R7/R8/R1 alias MC/AI/PB) | + +--- + +## §1 Binding rules (no exceptions, no re-litigating) + +1. **Universal definition-of-done**: + [extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) + applies verbatim to every PR in every workstream — read-before-edit, tests authored first + and shown RED where the spec says so, `npm run check` + `test:unit` + relevant `test:e2e` + green, browser E2E for every user-facing feature, no flaky tests, minimal change, master + stays green. Its §0.1 patterns library is the copy-from list for all workstreams. +2. **The docs are the spec.** Implement what the owning doc's phase says, using its + Appendix A contracts (types, file layouts, catalogs) **as written**. If reality forces a + deviation (an anchor moved, an API differs), amend the owning doc *in the same PR* with a + short `> Deviation:` note at the affected section — the doc and the code must never + disagree after a merge. Do not redesign; if a deviation is architectural, stop and + escalate to a human instead. +3. **One phase = one goal; one goal = 1–3 PRs.** Never batch phases. Every PR must leave the + product working and mergeable on its own (feature-flagged or additive when incomplete). +4. **Shared-seam coordination.** Three seams are touched by multiple workstreams — land them + **once**, in whichever goal runs first, and the later goal consumes: + - `goal-trigger-dispatcher.ts` push triggers `gate_failed`/`session_errored` + (GA-R2 ∩ MC-P4 ∩ AI-P1); + - `recordActivity()` / `activity-store.ts` (MC-P3; AI-P5 and the meta-tool write into it); + - aux-model slot config (`auxiliary.*` — GA-R11 ∩ AI-P2 ∩ CE-G6.1; one config shape, + [per-role-model-overrides.md](per-role-model-overrides.md) conventions). + Also respect the impl-plan §0.8 note: `session-manager.ts`/`session-setup.ts` are shared + with the comms-stack backlog — confine edits to named functions. +5. **Tracking discipline.** When a PR merges: tick its row in §4 *in that PR*; when a goal + completes, update the owning doc's `Status:` header line. The orchestrating session + should treat an unticked-but-merged PR as a bug. + +## §2 Dependency graph (what can start when) + +```mermaid +graph TD + subgraph independent now + PB0[PB-P0 perf harness] --> PB1[PB-P1 animations] --> PB2[PB-P2 timers/renders] --> PB3[PB-P3 battery saver] + CE0[CE-G0 cost instrumentation] --> CErest[CE-G1…G7 per its own plan] + GA2[GA-R2 triggers] + GA3[GA-R3 standing-orders docs] + EP[EP G1…G9 per its own plan] + end + MC0[MC-P0 global scope] --> MC1[MC-P1 sessions+sidebar] --> MC2[MC-P2 bobbit meta-tool] + MC1 --> MC3[MC-P3 flight recorder] + MC2 --> MC4[MC-P4 pack + crew] + MC3 --> MC4 --> MC5[MC-P5 briefing/onboarding/budgets] + GA2 --> MC4 + MC0 --> AI1[AI-P1 proposal substrate] + GA2 --> AI1 + AI1 --> AI2[AI-P2 Improver + review] --> AI3[AI-P3 curator + dreaming] + MC3 --> AI5 + AI2 --> AI4[AI-P4 shadow calibration] --> AI5[AI-P5 graduated autonomy] --> AI6[AI-P6 measurement] + MC4 --> AI2 + GA4[GA-R4 away summary] -.aux slots.- AI2 + GA5[GA-R5 user profile memory] -.coordinate.- EP + GA6[GA-R6 skill import adapters] --> EP +``` + +Three lanes can run in parallel from day one: **(A)** PB (perf), **(B)** CE + EP (their own +plans), **(C)** GA quick wins → MC spine → AI. Lane C is strictly ordered as drawn. + +## §3 PR slicing (every planned PR, in recommended order within each lane) + +Effort: S ≤ 1 day · M ≤ 3 days · L ≈ a week. "Mergeable because" is the standalone-value +test from §1.3. + +### Lane A — performance/battery ([client-performance-battery.md](client-performance-battery.md), contracts in its Appendix A) + +| PR | Scope | Effort | Mergeable because | +|---|---|---|---| +| PB-P0 | `perf-monitor.ts` + baseline table committed to the doc | S | flag-gated, zero behavior change | +| PB-P1a | `animation-power.ts` + CSS pause gates (Appendix A.1 table) + pinning test | M | default-on flag, kill switch `-pauseAmbientAnims` | +| PB-P1b | FX3 box-shadow→opacity pulse rewrites + blanket reduced-motion rule | S | pure CSS, visually identical | +| PB-P2a | FX5 session-poll demotion | S | behavior identical while WS down | +| PB-P2b | FX6 scoped verification tick | S | dashboard-local | +| PB-P2c | FX7 `renderAppThrottled` + streaming call sites | M | flag-gated | +| PB-P2d | FX8 timer audit table → fixes for convicted loops | M | per-loop, independent | +| PB-P3 | battery-saver mode + flag-soak removals + after-table | M | additive setting | + +### Lane B — cost + extension platform + +Follow their own plans verbatim: CE-G0 → (CE lanes per token-cost §6); EP G1→G9 per +extension-platform-implementation-plan §1 goal map. Nothing here re-slices them; this doc +only adds: CE-G6.1's aux-summarizer config must use the shared `auxiliary.*` shape (§1.4). + +### Lane C — quick wins → Mission Control → Autoimprovement + +| PR | Scope | Effort | Mergeable because | +|---|---|---|---| +| GA-R2 | `at` one-shot trigger + `gate_failed`/`session_errored` push triggers (contract in gap-analysis §5) | M | additive trigger types, immediately useful to any staff | +| GA-R3 | standing-orders template in docs + staff-assistant guidance | S | docs+prompt only | +| MC-P0 | global scope + `PersistedStaff.global` (MC Appendix A.1–A.2) | M | invisible until MC-P1; orphan behavior pinned by tests | +| MC-P1 | global sessions + sidebar top entry + E2E | M | complete user-visible feature | +| MC-P2a | `bobbit-meta-tool.ts` registry + catalog pinning test (MC Appendix A.3) | M | server-only, no tool exposure yet | +| MC-P2b | `defaults/tools/bobbit/` group + tier enforcement + policy wiring + e2e | M | tool ships complete with guards | +| MC-P3 | `activity-store.ts` + REST/WS + Activity panel (MC Appendix A.4) | M | useful immediately for meta-tool audit | +| MC-P4a | mission-control pack: roles/skills/templates + panel skeleton (A.5) | M | pack installs; crew not yet instantiable | +| MC-P4b | "create crew" bootstrap + crew E2E + Caretaker dry-run sweep | M | completes the crew feature | +| AI-P1 | improvement store + `propose_improvement` + panel kind (AI Appendix A.1–A.3) | L | proposals usable manually before any automation | +| AI-P2a | judge module + learned-skills pack bootstrap + skill-usage hook | M | inert until Improver exists | +| AI-P2b | Improver role/skills in pack + post-goal review wiring + e2e | M | completes human-in-loop learning | +| AI-P3a | curator (lifecycle, snapshots, pin, dry-run, CLI-less REST controls) | M | maintenance value standalone | +| AI-P3b | dreaming job (gates per AI Appendix A.1 lock semantics) + Archivist wiring | M | flag: `autoimprovement.dream` config | +| AI-P4 | shadow-mode recording + calibration endpoint + staff-page report | M | zero autonomy granted | +| AI-P5 | levels/thresholds settings + policy auto-approve path + revert + demotion + kill switch | L | ships OFF (all levels 0) | +| AI-P6 | outcome evaluator + regression demotion | M | completes the loop | +| MC-P5 | briefing + first-run onboarding + budgets | L | three separable sub-PRs if needed | +| GA-R4 | away-summary tactical slice | M | independent; share `auxiliary.*` | +| GA-R5 | bounded user-profile memory | M | coordinate with EP `session-memory` to avoid double-injection | +| GA-R6 | hermes/openclaw skill import adapters | M | marketplace adapter seam | +| GA-R9 | `bobbit doctor` | M | standalone CLI; Caretaker integration optional later | + +## §4 Master checklist (tick in the same PR that merges) + +- [ ] PB-P0 · [ ] PB-P1a · [ ] PB-P1b · [ ] PB-P2a · [ ] PB-P2b · [ ] PB-P2c · [ ] PB-P2d · [ ] PB-P3 +- [ ] GA-R2 · [ ] GA-R3 · [ ] GA-R4 · [ ] GA-R5 · [ ] GA-R6 · [ ] GA-R9 +- [ ] MC-P0 · [ ] MC-P1 · [ ] MC-P2a · [ ] MC-P2b · [ ] MC-P3 · [ ] MC-P4a · [ ] MC-P4b · [ ] MC-P5 +- [ ] AI-P1 · [ ] AI-P2a · [ ] AI-P2b · [ ] AI-P3a · [ ] AI-P3b · [ ] AI-P4 · [ ] AI-P5 · [ ] AI-P6 +- CE and EP: tracked inside their own plans; mirror parent-goal completion here: + [ ] CE program · [ ] EP program + +## §5 Known pre-existing issue (not owned by any lane — fix first) + +`tests/agents-md-budget.test.ts` currently fails on the branch base: AGENTS.md is 6,233 +bytes against the 6,144 budget. Trim AGENTS.md (move detail into `docs/`) in a standalone +S-size PR before starting lanes, so every lane starts from a green suite. diff --git a/docs/design/harness-gap-analysis.md b/docs/design/harness-gap-analysis.md index 722c25e0b..07363da54 100644 --- a/docs/design/harness-gap-analysis.md +++ b/docs/design/harness-gap-analysis.md @@ -271,3 +271,18 @@ without re-deriving context): cheap-model summarization over messages since `lastSeen` (reuse the compaction-summary path; aux-model slot per G11); client: collapsible "while you were away" banner above the transcript tail. Behind a setting; browser E2E for banner render + dismissal persistence. + +R2 trigger contract (so implementation cannot drift): extend the `StaffTrigger` union with + +```ts +{ type: "schedule"; config: { cron: string; timezone?: string } | { at: string /* ISO-8601 */ } } +{ type: "gate_failed"; config: {}; prompt: string } // required prompt, like goal_created +{ type: "session_errored"; config: {}; prompt: string } +``` + +`at` semantics: fire once when `now >= at && !lastFired`, then set `enabled: false` (record +kept for audit). Push triggers dispatch from the gate-failure and session-error mutation +points via `goal-trigger-dispatcher.ts`, mirroring `goal_created` exactly (including the +required-prompt validation in `docs/staff-triggers.md`). + +Execution tracking for every row above: [fable-program-execution-plan.md](fable-program-execution-plan.md). diff --git a/docs/design/mission-control.md b/docs/design/mission-control.md index c05e48309..9b5c7724b 100644 --- a/docs/design/mission-control.md +++ b/docs/design/mission-control.md @@ -319,3 +319,139 @@ baseline once the harness exists) · [extension-platform.md](extension-platform. [token-cost-efficiency.md](token-cost-efficiency.md) (budgets, aux models) · `docs/staff-agents.md` · `docs/staff-triggers.md` · `docs/staff-inbox.md` · `docs/mcp-meta-tools.md` · `docs/rest-api.md`. +Execution tracking: [fable-program-execution-plan.md](fable-program-execution-plan.md). + +--- + +## Appendix A — Implementation contracts (definite lists; do not re-derive) + +The universal definition-of-done in +[extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) +(read-before-edit, test-first RED→GREEN, gates, browser E2E, no flake, minimal change) is +binding for every MC phase. This appendix fixes the contracts a weaker implementer would +otherwise have to invent. + +### A.1 Constants and storage layout + +```ts +// project-registry.ts (exists): SYSTEM_PROJECT_ID = "system" +// NEW in project-registry.ts: +export function isSystemScope(projectId: string): boolean; // === SYSTEM_PROJECT_ID +export function missionControlDir(): string; // path.join(stateDir, "mission-control"); mkdir lazily +``` + +State additions under `.bobbit/state/`: +`mission-control/` (cwd workspace for global sessions/staff) · `activity.jsonl` (+ rotated +`activity-YYYY-MM-DD.jsonl`, rotate when current file > 8 MB). + +### A.2 `PersistedStaff` change (MC-P0) + +```ts +// staff-store.ts — add ONE field; loader normalises missing → false: +global?: boolean; // true ⇒ projectId === "system" is legitimate, not an orphan +// Validation matrix (staff-manager.ts): +// global:true ⇒ projectId must be "system"; cwd inside missionControlDir(); worktree +// forced off; sandboxed allowed (plain boolean, unchanged semantics) +// global:false ⇒ existing rules unchanged (real project required) +// GET /api/staff/orphaned: exclude records with global === true. Everything else unchanged. +``` + +### A.3 `bobbit` meta-tool — file layout and operation catalog v1 + +Tool-group anatomy copies `defaults/tools/team/` exactly: one YAML per tool + +`extension.ts`, `provider: { type: bobbit-extension, extension: extension.ts }`, +`group: Bobbit`. Files: `defaults/tools/bobbit/bobbit.yaml`, `bobbit_describe.yaml`, +`extension.ts`. The operation registry lives server-side in a new +`src/server/agent/bobbit-meta-tool.ts`; the extension calls the gateway over the existing +tool-extension HTTP bridge (`tool-guard-extension.ts` pattern) — never spawning a second +HTTP client against localhost REST. + +```ts +export type OpTier = "read" | "mutate" | "sensitive"; +export interface BobbitOp { + op: string; // enum value exposed to the model + tier: OpTier; + method: "GET" | "POST" | "PATCH" | "PUT" | "DELETE"; + path: string; // REST path template, e.g. "/api/projects/:id/goals" + argsSchema: object; // JSON schema; bobbit_describe returns it verbatim + doc: string; // one-liner; bobbit_describe returns it +} +``` + +Catalog v1 (complete; adding an op later = registry row + pinning-test row, nothing else): + +| op | tier | REST | +|---|---|---| +| `projects.list` | read | `GET /api/projects` | +| `projects.preflight` | read | the add-project preflight route (`docs/add-project-preflight.md`) | +| `projects.add` | sensitive | the add-project route used by the UI proposal flow | +| `sessions.list` | read | `GET /api/sessions` | +| `sessions.create` | mutate | `POST /api/sessions` | +| `sessions.send_message` | mutate | the prompt-queue enqueue route (`docs/prompt-queue.md`) | +| `sessions.archive` | sensitive | existing archive route | +| `goals.list` | read | `GET /api/goals` (per project) | +| `goals.tree` | read | the nested-goals tree route (`docs/nested-goals.md`) | +| `goals.create` | mutate | existing goal-create route (supports `parentId` for nesting) | +| `goals.archive` | sensitive | existing archive route | +| `staff.list` | read | `GET /api/staff` | +| `staff.wake` | mutate | existing wake route | +| `staff.inbox_enqueue` | mutate | `POST /api/staff/:id/inbox` | +| `staff.propose` | mutate | drives the staff-proposal flow (returns proposal for human accept — never creates directly) | +| `worktrees.list` | read | worktree-pool route | +| `worktrees.cleanup` | sensitive | existing cleanup route | +| `packs.list` | read | marketplace list route | +| `packs.install` | sensitive | `marketplace-install` route, registered sources only | +| `config.read` | read | config-cascade read route | +| `costs.read` | read | session-cost routes (`docs/session-cost.md`) | +| `activity.read` | read | `GET /api/activity` (MC-P3) | + +Resolve each "existing route" by reading `docs/rest-api.md` + the matching `server.ts` +handler **at implementation time**; the pinning test (`tests/bobbit-meta-tool.test.ts`) +asserts every catalog row dispatches to a registered handler, so a renamed route fails the +test rather than drifting silently. Tier enforcement: `sensitive` ⇒ route through the +blocking-ask flow (`docs/blocking-tools.md`); `mutate`+`sensitive` ⇒ `recordActivity()`. + +### A.4 Flight recorder (MC-P3) + +```ts +// src/server/agent/activity-store.ts +export interface ActivityEntry { + ts: string; // ISO-8601 UTC + actor: { kind: "session" | "staff" | "policy" | "user"; id: string; label?: string }; + action: string; // e.g. "bobbit.goals.create", "improvement.auto-approved" + target?: { kind: string; id: string; projectId?: string }; + tier: "read" | "mutate" | "sensitive"; // reads are recorded only for `sensitive`-adjacent audits — default: don't record reads + evidence?: string; // link/path, e.g. session id or proposal id + revert?: { kind: "improvement" | "api"; ref: string }; +} +export function recordActivity(e: ActivityEntry): void; // append + size-based rotation +// REST: GET /api/activity?since=<iso>&actor=<id>&tier=<t>&projectId=<id>&limit=<n (default 100, max 500)> +// WS: event type "activity_entry", delivered ONLY to sockets that sent +// {type:"subscribe_activity"} (goal-fanout lesson — no broadcast). +``` + +### A.5 mission-control pack layout (MC-P4) + +Copy `market-packs/pr-walkthrough/` structure; built-in band via +`scripts/copy-builtin-packs.mjs` `FIRST_PARTY_PACKS`. + +``` +market-packs/mission-control/ + pack.yaml # name, schema, contents: panels, entrypoints, roles, skills + roles/{caretaker,archivist,improver,observer}.yaml # standing-order-format prompts (§5) + skills/… # crew skills (briefing format, sweep checklists) + staff-templates/crew.yaml # name/role/accessory/triggers per §5 table; consumed by the + # core "create crew" REST action (idempotent by staff name) + src/panel.ts → lib/ # dashboard panel at #/ext/mission-control +``` + +### A.6 Owned files per phase (PR boundaries) + +| Phase | New files | Modified files | +|---|---|---| +| MC-P0 | — | `project-registry.ts`, `staff-store.ts`, `staff-manager.ts`, `server.ts` (orphan route), `docs/staff-agents.md` | +| MC-P1 | `tests/e2e/ui/mission-control.spec.ts` | session-create path (`server.ts`/`session-setup.ts`), `src/app/sidebar.ts`, `src/app/render.ts` (mobile shell) | +| MC-P2 | `defaults/tools/bobbit/*`, `src/server/agent/bobbit-meta-tool.ts`, `tests/bobbit-meta-tool.test.ts` | tool-activation default policy for system scope, `tests/tool-description-budget.test.ts` | +| MC-P3 | `src/server/agent/activity-store.ts`, activity panel module in `src/app/` | `server.ts` (REST+WS), `sidebar.ts` (Activity nav) | +| MC-P4 | `market-packs/mission-control/*` | `copy-builtin-packs.mjs`, `goal-trigger-dispatcher.ts` (new push triggers, if not landed via AI-P1/GA-R2 first — coordinate, land once) | +| MC-P5 | — | Observer role prompt (pack), first-run routing in `src/app/`, config keys + budget check, briefing tests | From 5d7e7d8c9612119b122c802066f10b4faedee1f8 Mon Sep 17 00:00:00 2001 From: Claude <noreply@anthropic.com> Date: Wed, 10 Jun 2026 22:56:42 +0000 Subject: [PATCH 07/18] Unify all fable-docs workstreams under the execution plan; format parity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - fable-program-execution-plan.md — expanded to cover ALL eight workstream docs on this branch, not just the four newest: adds the comms-stack (CS) and sidebar/goal-nesting (SN) backlogs as lanes, enumerates every EP and CE sub-goal in the master checklist, adds a shared-seam serialization table (notably: SN-T2/T3 before PB-P2a on api.ts refreshSessions; SN-T5 before PB-P2c on the sidebar render paths; CS merge-map governs session-manager.ts), a Day-1 parallel starter set, and a §6 doc-format contract that defines parity for any future design doc. - extension-platform.md — banner: implementation plan is execution authority, incl. its §0.2 entity-loading refinement superseding §4. - extension-platform-implementation-plan.md, token-cost-efficiency.md, sidebar-goal-nesting-audit.md, comms-stack/04 — Status + workstream pointers to the execution plan; SN gains explicit seam-overlap warnings against the battery plan. https://claude.ai/code/session_01LBW68Z2KvA3EhQMq3c2wrR --- .../04-current-state-and-backlog.md | 5 + .../extension-platform-implementation-plan.md | 4 + docs/design/extension-platform.md | 7 + docs/design/fable-program-execution-plan.md | 259 ++++++++++-------- docs/design/sidebar-goal-nesting-audit.md | 7 + docs/design/token-cost-efficiency.md | 4 +- 6 files changed, 169 insertions(+), 117 deletions(-) diff --git a/docs/design/comms-stack/04-current-state-and-backlog.md b/docs/design/comms-stack/04-current-state-and-backlog.md index 9dcf68191..900882f0a 100644 --- a/docs/design/comms-stack/04-current-state-and-backlog.md +++ b/docs/design/comms-stack/04-current-state-and-backlog.md @@ -12,6 +12,11 @@ > hand-off tasks** (each executable by an agent who sees only that task's text), grouped into > waves with acceptance gates. > +> **Status:** backlog not started. Workstream **CS** in +> [../fable-program-execution-plan.md](../fable-program-execution-plan.md) (program +> sequencing + master checklist). Land CS tasks in the §7 merge-map order; `session-manager.ts` +> is also touched by CE-G3 and EP — serialize per the execution plan's shared-seam table. +> > **Method:** every claim was re-verified against the current tree by parallel adversarial > audits (file:line anchors below are current as of `6ec8c8f9`, not the drifted anchors in > docs 01–03). SDK ground truth was read from the installed `pi-coding-agent`/`pi-agent-core` diff --git a/docs/design/extension-platform-implementation-plan.md b/docs/design/extension-platform-implementation-plan.md index 8c9503e0f..3fc5c90ec 100644 --- a/docs/design/extension-platform-implementation-plan.md +++ b/docs/design/extension-platform-implementation-plan.md @@ -1,5 +1,9 @@ # Extension Platform — Implementation Plan (hand-off) +Status: ready for execution, not started. Workstream **EP** in +[fable-program-execution-plan.md](fable-program-execution-plan.md) (program sequencing + +master checklist; its §0-equivalent rules ARE this document's §0). + Companion to [extension-platform.md](extension-platform.md) (the WHAT/WHY — read its §3–§12 before implementing anything). This document is the HOW: a tree of goals, each independently mergeable and verifiable, written so an implementer agent **that has not seen any other goal** diff --git a/docs/design/extension-platform.md b/docs/design/extension-platform.md index 7dd9c84cb..dd4ca7931 100644 --- a/docs/design/extension-platform.md +++ b/docs/design/extension-platform.md @@ -3,6 +3,13 @@ Status: design / implementation blueprint. Each phase in §13 is written to convert 1:1 into a mission goal with acceptance criteria. +> **Execution authority:** implement from +> [extension-platform-implementation-plan.md](extension-platform-implementation-plan.md) +> (goal map, owned files, RED→GREEN tests) — including its **§0.2 refinement** (new entity +> types load via the pack-contributions path, NOT new `EntityLoader` entities; that +> supersedes the §4 sketch below). Program-wide sequencing + master checklist: +> [fable-program-execution-plan.md](fable-program-execution-plan.md). + --- ## 1. Mission & principles diff --git a/docs/design/fable-program-execution-plan.md b/docs/design/fable-program-execution-plan.md index f5fb4eabc..0e7de16de 100644 --- a/docs/design/fable-program-execution-plan.md +++ b/docs/design/fable-program-execution-plan.md @@ -1,19 +1,22 @@ # Fable program — execution plan (hand-off for the main Bobbit session) -Status: living tracking document. This is the **single sequencing source of truth** for the -work proposed across the fable-docs design set. Hand this file to the orchestrating Bobbit -session; it should create goals from §3, in order, and tick the checklist in §4 as PRs merge. - -The design docs in scope (each owns its *content*; this doc owns *ordering and slicing*): - -| Doc | Workstream prefix | -|---|---| -| [extension-platform.md](extension-platform.md) + [extension-platform-implementation-plan.md](extension-platform-implementation-plan.md) | EP (G1–G9 there) | -| [token-cost-efficiency.md](token-cost-efficiency.md) §6 | CE (CE-G0–G7 there) | -| [client-performance-battery.md](client-performance-battery.md) | PB (P0–P3) | -| [mission-control.md](mission-control.md) | MC (P0–P5) | -| [autoimprovement.md](autoimprovement.md) | AI (P1–P6) | -| [harness-gap-analysis.md](harness-gap-analysis.md) §5 | GA (R2–R6, R9; R7/R8/R1 alias MC/AI/PB) | +Status: living tracking document. This is the **single sequencing source of truth** for all +work proposed on the fable-docs branch. Hand this file to the orchestrating Bobbit session; +it creates goals from §3, in lane order, and ticks §4 as PRs merge. + +The workstreams (each doc owns its *content* — tasks, contracts, acceptance; this doc owns +*ordering, slicing, and the checklist*): + +| Workstream | Doc | Granularity there | +|---|---|---| +| **EP** extension platform | [extension-platform.md](extension-platform.md) (design) + [extension-platform-implementation-plan.md](extension-platform-implementation-plan.md) (execution authority) | G1.1–G9, owned files, RED→GREEN tests | +| **CE** token/cost efficiency | [token-cost-efficiency.md](token-cost-efficiency.md) §6 | CE-G0.1–G7.2, lanes, BENCH gates | +| **CS** comms-stack reliability | [comms-stack/04-current-state-and-backlog.md](comms-stack/04-current-state-and-backlog.md) §6–§7 | CS-R/H/D/P/T waves + merge map | +| **SN** sidebar & goal-nesting | [sidebar-goal-nesting-audit.md](sidebar-goal-nesting-audit.md) §4 | T1–T9, files/diagnosis/fix/acceptance | +| **PB** client perf/battery | [client-performance-battery.md](client-performance-battery.md) + Appendix A | P0–P3, FX1–FX9, owned files | +| **MC** Mission Control | [mission-control.md](mission-control.md) + Appendix A | P0–P5, contracts, owned files | +| **AI** autoimprovement | [autoimprovement.md](autoimprovement.md) + Appendix A | P1–P6, contracts, owned files | +| **GA** gap-analysis easy wins | [harness-gap-analysis.md](harness-gap-analysis.md) §5 | R2–R6, R9 (R1/R7/R8 alias PB/MC/AI) | --- @@ -21,123 +24,147 @@ The design docs in scope (each owns its *content*; this doc owns *ordering and s 1. **Universal definition-of-done**: [extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) - applies verbatim to every PR in every workstream — read-before-edit, tests authored first - and shown RED where the spec says so, `npm run check` + `test:unit` + relevant `test:e2e` - green, browser E2E for every user-facing feature, no flaky tests, minimal change, master - stays green. Its §0.1 patterns library is the copy-from list for all workstreams. -2. **The docs are the spec.** Implement what the owning doc's phase says, using its - Appendix A contracts (types, file layouts, catalogs) **as written**. If reality forces a - deviation (an anchor moved, an API differs), amend the owning doc *in the same PR* with a - short `> Deviation:` note at the affected section — the doc and the code must never - disagree after a merge. Do not redesign; if a deviation is architectural, stop and - escalate to a human instead. -3. **One phase = one goal; one goal = 1–3 PRs.** Never batch phases. Every PR must leave the - product working and mergeable on its own (feature-flagged or additive when incomplete). -4. **Shared-seam coordination.** Three seams are touched by multiple workstreams — land them - **once**, in whichever goal runs first, and the later goal consumes: - - `goal-trigger-dispatcher.ts` push triggers `gate_failed`/`session_errored` - (GA-R2 ∩ MC-P4 ∩ AI-P1); - - `recordActivity()` / `activity-store.ts` (MC-P3; AI-P5 and the meta-tool write into it); - - aux-model slot config (`auxiliary.*` — GA-R11 ∩ AI-P2 ∩ CE-G6.1; one config shape, - [per-role-model-overrides.md](per-role-model-overrides.md) conventions). - Also respect the impl-plan §0.8 note: `session-manager.ts`/`session-setup.ts` are shared - with the comms-stack backlog — confine edits to named functions. -5. **Tracking discipline.** When a PR merges: tick its row in §4 *in that PR*; when a goal - completes, update the owning doc's `Status:` header line. The orchestrating session - should treat an unticked-but-merged PR as a bug. - -## §2 Dependency graph (what can start when) + applies verbatim to every PR in every workstream — read-before-edit (`rg` the symbol in + docs/tests/src first), tests authored first and shown RED where expressible, `npm run + check` + `test:unit` + relevant `test:e2e` green, browser E2E for every user-facing + feature, no flaky tests, minimal change, master stays green, anchors located by **symbol + name** (line numbers are hints; if a named symbol is missing, STOP and re-derive from the + cited pattern file — never improvise a parallel mechanism). Its §0.1 patterns library is + the copy-from list for all workstreams. +2. **The docs are the spec.** Implement exactly what the owning doc's task/phase says, using + its appendix contracts (types, catalogs, file layouts) as written. If reality forces a + deviation, amend the owning doc **in the same PR** with a `> Deviation:` note at the + affected section — doc and code must never disagree after a merge. Architectural + deviation ⇒ stop and escalate to a human. +3. **One task/phase = one goal; one goal = 1–3 PRs.** Never batch. Every PR leaves the + product working and mergeable on its own (flag-gated or additive when incomplete). +4. **Shared-seam serialization.** These seams are touched by multiple workstreams. The first + goal to land owns the change; later goals rebase onto it — never parallel-edit a seam: + + | Seam | Touched by | Order | + |---|---|---| + | `api.ts` `refreshSessions`/`startSessionPolling` | SN-T2/T3 · PB-P2a (FX5) | **SN-T2 → SN-T3 → PB-P2a** | + | sidebar render paths (`sidebar.ts`, `render-helpers.ts`, nesting) | SN-T5/T6/T7 · PB-P2c (FX7) | SN first; PB-P2c after or rebased | + | `session-manager.ts` / `session-setup.ts` | CS-R\*/D\*/P3 · CE-G3 · EP G1.3/G1.4 | CS merge-map order governs; CE/EP confine edits to named functions and rebase | + | `goal-trigger-dispatcher.ts` push triggers (`gate_failed`/`session_errored`) | GA-R2 · MC-P4 · AI-P1 | land once in GA-R2 | + | `activity-store.ts` / `recordActivity()` | MC-P3 · MC-P2 · AI-P5 | land in MC-P3; earlier callers stub | + | `auxiliary.*` model-slot config | AI-P2 · GA-R4 · CE-G6.1 | one shape ([per-role-model-overrides.md](per-role-model-overrides.md)); first lander defines it | + | `verification-harness.ts` | CE-G3.3 · CE-G5.2 | sequence per CE doc | +5. **Tracking discipline.** Tick the §4 row **in the merging PR**; update the owning doc's + `Status:` line when a workstream's parent goal completes. The orchestrating session + treats merged-but-unticked as a bug. + +## §2 Dependency graph ```mermaid graph TD - subgraph independent now - PB0[PB-P0 perf harness] --> PB1[PB-P1 animations] --> PB2[PB-P2 timers/renders] --> PB3[PB-P3 battery saver] - CE0[CE-G0 cost instrumentation] --> CErest[CE-G1…G7 per its own plan] - GA2[GA-R2 triggers] - GA3[GA-R3 standing-orders docs] - EP[EP G1…G9 per its own plan] + subgraph "independent lanes — start day one" + CS[CS waves R1→…→T2<br/>per its §7 merge map] + SN1[SN-T1/T2/T4/T8/T9] --> SN2[SN-T3, T5] --> SN3[SN-T6, T7] + CE[CE-G0 → CE lanes per its §6] + EP[EP G1→G9 per its goal map] + PB0[PB-P0 harness] --> PB1[PB-P1a/b animations] + GA2[GA-R2 triggers]; GA3[GA-R3 standing orders] end - MC0[MC-P0 global scope] --> MC1[MC-P1 sessions+sidebar] --> MC2[MC-P2 bobbit meta-tool] + SN2 --> PB2[PB-P2a-d timers/renders] --> PB3[PB-P3 battery saver] + PB1 --> PB2 + MC0[MC-P0 global scope] --> MC1[MC-P1 sessions+sidebar] --> MC2[MC-P2a/b meta-tool] MC1 --> MC3[MC-P3 flight recorder] - MC2 --> MC4[MC-P4 pack + crew] - MC3 --> MC4 --> MC5[MC-P5 briefing/onboarding/budgets] - GA2 --> MC4 - MC0 --> AI1[AI-P1 proposal substrate] - GA2 --> AI1 - AI1 --> AI2[AI-P2 Improver + review] --> AI3[AI-P3 curator + dreaming] - MC3 --> AI5 - AI2 --> AI4[AI-P4 shadow calibration] --> AI5[AI-P5 graduated autonomy] --> AI6[AI-P6 measurement] - MC4 --> AI2 - GA4[GA-R4 away summary] -.aux slots.- AI2 - GA5[GA-R5 user profile memory] -.coordinate.- EP - GA6[GA-R6 skill import adapters] --> EP + MC2 --> MC4[MC-P4a/b pack+crew]; MC3 --> MC4; GA2 --> MC4 + MC4 --> MC5[MC-P5 briefing/onboarding/budgets] + MC0 --> AI1[AI-P1 substrate]; GA2 --> AI1 + AI1 --> AI2[AI-P2a/b Improver] --> AI3[AI-P3a/b curator+dreaming] + MC4 --> AI2; MC3 --> AI5 + AI2 --> AI4[AI-P4 shadow] --> AI5[AI-P5 autonomy] --> AI6[AI-P6 measurement] + GA6[GA-R6 importers] --> EP + GA5[GA-R5 user profile] -.coordinate.- EP ``` -Three lanes can run in parallel from day one: **(A)** PB (perf), **(B)** CE + EP (their own -plans), **(C)** GA quick wins → MC spine → AI. Lane C is strictly ordered as drawn. +**Day-1 starter set** (parallel, no collisions): AGENTS.md trim (§5) · CS-R1 · SN-T1 · +PB-P0 · CE-G0.1 · EP G1.1 · GA-R2 · GA-R3. + +## §3 Lanes and PR slicing -## §3 PR slicing (every planned PR, in recommended order within each lane) +Effort: S ≤ 1 day · M ≤ 3 days · L ≈ a week. EP, CE, and CS are **not re-sliced here** — +their own docs already slice to mergeable units; follow them verbatim (EP goal map order; +CE lanes with BENCH gates; CS §7 merge-map order). The lanes below slice the remaining +workstreams: -Effort: S ≤ 1 day · M ≤ 3 days · L ≈ a week. "Mergeable because" is the standalone-value -test from §1.3. +### Lane SN — sidebar/nesting ([sidebar-goal-nesting-audit.md](sidebar-goal-nesting-audit.md) §4; one PR per task) -### Lane A — performance/battery ([client-performance-battery.md](client-performance-battery.md), contracts in its Appendix A) +T1 (M, no deps) → T2 (M) → T3 (S) → T4 (M) → T5 (L, after T1+T4) → T6 (M) → T7 (S) → +T8 (S, independent) → T9 (S, independent). T2/T3/T5/T7 gate PB-P2 (§1.4). + +### Lane PB — perf/battery (contracts in its Appendix A) | PR | Scope | Effort | Mergeable because | |---|---|---|---| -| PB-P0 | `perf-monitor.ts` + baseline table committed to the doc | S | flag-gated, zero behavior change | -| PB-P1a | `animation-power.ts` + CSS pause gates (Appendix A.1 table) + pinning test | M | default-on flag, kill switch `-pauseAmbientAnims` | -| PB-P1b | FX3 box-shadow→opacity pulse rewrites + blanket reduced-motion rule | S | pure CSS, visually identical | -| PB-P2a | FX5 session-poll demotion | S | behavior identical while WS down | +| PB-P0 | `perf-monitor.ts` + baseline table | S | flag-gated, zero behavior change | +| PB-P1a | `animation-power.ts` + CSS gates (A.1 table) + pinning test | M | default-on flag, kill switch | +| PB-P1b | FX3 box-shadow→opacity rewrites + blanket reduced-motion | S | pure CSS, visually identical | +| PB-P2a | FX5 poll demotion *(after SN-T2/T3)* | S | behavior identical while WS down | | PB-P2b | FX6 scoped verification tick | S | dashboard-local | -| PB-P2c | FX7 `renderAppThrottled` + streaming call sites | M | flag-gated | -| PB-P2d | FX8 timer audit table → fixes for convicted loops | M | per-loop, independent | -| PB-P3 | battery-saver mode + flag-soak removals + after-table | M | additive setting | - -### Lane B — cost + extension platform - -Follow their own plans verbatim: CE-G0 → (CE lanes per token-cost §6); EP G1→G9 per -extension-platform-implementation-plan §1 goal map. Nothing here re-slices them; this doc -only adds: CE-G6.1's aux-summarizer config must use the shared `auxiliary.*` shape (§1.4). +| PB-P2c | FX7 `renderAppThrottled` + streaming sites *(after/rebased on SN-T5)* | M | flag-gated | +| PB-P2d | FX8 timer audit + convicted-loop fixes | M | per-loop independent | +| PB-P3 | battery-saver mode + flag-soak removal + after-table | M | additive setting | -### Lane C — quick wins → Mission Control → Autoimprovement +### Lane MC→AI — Mission Control then autoimprovement (contracts in their Appendices A) | PR | Scope | Effort | Mergeable because | |---|---|---|---| -| GA-R2 | `at` one-shot trigger + `gate_failed`/`session_errored` push triggers (contract in gap-analysis §5) | M | additive trigger types, immediately useful to any staff | -| GA-R3 | standing-orders template in docs + staff-assistant guidance | S | docs+prompt only | -| MC-P0 | global scope + `PersistedStaff.global` (MC Appendix A.1–A.2) | M | invisible until MC-P1; orphan behavior pinned by tests | -| MC-P1 | global sessions + sidebar top entry + E2E | M | complete user-visible feature | -| MC-P2a | `bobbit-meta-tool.ts` registry + catalog pinning test (MC Appendix A.3) | M | server-only, no tool exposure yet | -| MC-P2b | `defaults/tools/bobbit/` group + tier enforcement + policy wiring + e2e | M | tool ships complete with guards | -| MC-P3 | `activity-store.ts` + REST/WS + Activity panel (MC Appendix A.4) | M | useful immediately for meta-tool audit | -| MC-P4a | mission-control pack: roles/skills/templates + panel skeleton (A.5) | M | pack installs; crew not yet instantiable | -| MC-P4b | "create crew" bootstrap + crew E2E + Caretaker dry-run sweep | M | completes the crew feature | -| AI-P1 | improvement store + `propose_improvement` + panel kind (AI Appendix A.1–A.3) | L | proposals usable manually before any automation | -| AI-P2a | judge module + learned-skills pack bootstrap + skill-usage hook | M | inert until Improver exists | -| AI-P2b | Improver role/skills in pack + post-goal review wiring + e2e | M | completes human-in-loop learning | -| AI-P3a | curator (lifecycle, snapshots, pin, dry-run, CLI-less REST controls) | M | maintenance value standalone | -| AI-P3b | dreaming job (gates per AI Appendix A.1 lock semantics) + Archivist wiring | M | flag: `autoimprovement.dream` config | -| AI-P4 | shadow-mode recording + calibration endpoint + staff-page report | M | zero autonomy granted | -| AI-P5 | levels/thresholds settings + policy auto-approve path + revert + demotion + kill switch | L | ships OFF (all levels 0) | -| AI-P6 | outcome evaluator + regression demotion | M | completes the loop | -| MC-P5 | briefing + first-run onboarding + budgets | L | three separable sub-PRs if needed | -| GA-R4 | away-summary tactical slice | M | independent; share `auxiliary.*` | -| GA-R5 | bounded user-profile memory | M | coordinate with EP `session-memory` to avoid double-injection | -| GA-R6 | hermes/openclaw skill import adapters | M | marketplace adapter seam | -| GA-R9 | `bobbit doctor` | M | standalone CLI; Caretaker integration optional later | - -## §4 Master checklist (tick in the same PR that merges) - -- [ ] PB-P0 · [ ] PB-P1a · [ ] PB-P1b · [ ] PB-P2a · [ ] PB-P2b · [ ] PB-P2c · [ ] PB-P2d · [ ] PB-P3 -- [ ] GA-R2 · [ ] GA-R3 · [ ] GA-R4 · [ ] GA-R5 · [ ] GA-R6 · [ ] GA-R9 -- [ ] MC-P0 · [ ] MC-P1 · [ ] MC-P2a · [ ] MC-P2b · [ ] MC-P3 · [ ] MC-P4a · [ ] MC-P4b · [ ] MC-P5 -- [ ] AI-P1 · [ ] AI-P2a · [ ] AI-P2b · [ ] AI-P3a · [ ] AI-P3b · [ ] AI-P4 · [ ] AI-P5 · [ ] AI-P6 -- CE and EP: tracked inside their own plans; mirror parent-goal completion here: - [ ] CE program · [ ] EP program - -## §5 Known pre-existing issue (not owned by any lane — fix first) - -`tests/agents-md-budget.test.ts` currently fails on the branch base: AGENTS.md is 6,233 -bytes against the 6,144 budget. Trim AGENTS.md (move detail into `docs/`) in a standalone -S-size PR before starting lanes, so every lane starts from a green suite. +| GA-R2 | `at` one-shot + `gate_failed`/`session_errored` triggers (schema in gap doc §5) | M | additive trigger types | +| GA-R3 | standing-orders template + staff-assistant guidance | S | docs+prompt only | +| MC-P0 | global scope + `PersistedStaff.global` (A.1–A.2) | M | invisible until MC-P1 | +| MC-P1 | global sessions + sidebar top entry + E2E | M | complete visible feature | +| MC-P2a | meta-tool registry + catalog pinning test (A.3) | M | server-only | +| MC-P2b | `defaults/tools/bobbit/` + tiers + policy wiring + e2e | M | ships complete with guards | +| MC-P3 | activity store + REST/WS + panel (A.4) | M | immediate audit value | +| MC-P4a | mission-control pack: roles/skills/templates + panel (A.5) | M | installs; crew inert | +| MC-P4b | "create crew" bootstrap + crew E2E + Caretaker dry-run | M | completes crew | +| AI-P1 | improvement store + `propose_improvement` + panel kind (A.1–A.3) | L | manual proposals useful alone | +| AI-P2a | judge + learned-skills pack bootstrap + usage hook | M | inert until Improver | +| AI-P2b | Improver role + post-goal review + e2e | M | human-in-loop learning complete | +| AI-P3a | curator (lifecycle/snapshots/pin/dry-run) | M | maintenance standalone | +| AI-P3b | dreaming job + Archivist wiring | M | config-gated | +| AI-P4 | shadow mode + calibration report | M | zero autonomy granted | +| AI-P5 | levels/thresholds + policy path + revert + demotion + kill switch | L | ships OFF (levels 0) | +| AI-P6 | outcome evaluator + regression demotion | M | completes loop | +| MC-P5 | briefing + onboarding + budgets (3 sub-PRs allowed) | L | each sub-PR standalone | +| GA-R4 | away-summary slice | M | independent; shared `auxiliary.*` | +| GA-R5 | bounded user-profile memory | M | coordinate w/ EP `session-memory` | +| GA-R6 | hermes/openclaw skill import adapters | M | marketplace seam | +| GA-R9 | `bobbit doctor` | M | standalone CLI | + +## §4 Master checklist (tick in the merging PR) + +- **Prereq**: [ ] AGENTS.md trim (§5) +- **CS** (its §7 order): [ ] R1 · [ ] H1 · [ ] H2 · [ ] R2 · [ ] R5 · [ ] R6 · [ ] R3 · [ ] R7 · [ ] R4 · [ ] R8 · [ ] R9 · [ ] D1 · [ ] D2 · [ ] D3 · [ ] D4 · [ ] D5 · [ ] D6 · [ ] D7 · [ ] D8 · [ ] D9 · [ ] P1 · [ ] P2 · [ ] P3 · [ ] P4 · [ ] T1 · [ ] T2 +- **SN**: [ ] T1 · [ ] T2 · [ ] T3 · [ ] T4 · [ ] T5 · [ ] T6 · [ ] T7 · [ ] T8 · [ ] T9 +- **PB**: [ ] P0 · [ ] P1a · [ ] P1b · [ ] P2a · [ ] P2b · [ ] P2c · [ ] P2d · [ ] P3 +- **CE**: [ ] G0.1 · [ ] G0.2 · [ ] G0.3 · [ ] G1.0 · [ ] G1.1 · [ ] G1.2 · [ ] G2.1 · [ ] G2.2 · [ ] G3.1 · [ ] G3.2 · [ ] G3.3 · [ ] G4.1 · [ ] G4.2 · [ ] G4.3 · [ ] G5.1 · [ ] G5.2 · [ ] G6.1 · [ ] G7.1 +- **EP**: [ ] G1.1 · [ ] G1.2 · [ ] G1.3 · [ ] G1.4 · [ ] G1.5 · [ ] G1.6 · [ ] G2.1 · [ ] G2.2 · [ ] G2.3 · [ ] G3.1 · [ ] G3.2 · [ ] G3.3 · [ ] G4 · [ ] G5 · [ ] G6 · [ ] G7 · [ ] G8 · [ ] G9 +- **GA**: [ ] R2 · [ ] R3 · [ ] R4 · [ ] R5 · [ ] R6 · [ ] R9 +- **MC**: [ ] P0 · [ ] P1 · [ ] P2a · [ ] P2b · [ ] P3 · [ ] P4a · [ ] P4b · [ ] P5 +- **AI**: [ ] P1 · [ ] P2a · [ ] P2b · [ ] P3a · [ ] P3b · [ ] P4 · [ ] P5 · [ ] P6 + +## §5 Prerequisite fix (before any lane) + +`tests/agents-md-budget.test.ts` fails on the branch base: AGENTS.md is 6,233 bytes vs the +6,144 budget. One S-size PR: move detail into `docs/`, get the suite green. Every lane +starts from green. + +## §6 Doc-format contract (parity rule for any future design doc in this program) + +Every workstream doc must carry, and these eight do: + +1. A `Status:` line (state + baseline commit where relevant) and a **workstream pointer to + this file**. +2. A hand-off backlog where each task names: owned files (NEW vs modified), diagnosis or + contract reference, the fix/steps, tests to author first, acceptance criteria. +3. Exact contracts for anything an implementer would otherwise invent (types, catalogs, + schemas, directory layouts) — in the doc body or an `Appendix A`. +4. Anchors by **symbol name** with a verified-at baseline, and the §1.1 stop-rule when an + anchor is missing. +5. Seam-overlap warnings pointing at §1.4 of this file. + +A new design doc that lacks any of these is not ready to enter §3/§4. diff --git a/docs/design/sidebar-goal-nesting-audit.md b/docs/design/sidebar-goal-nesting-audit.md index 4e0d32994..0d1f6c2b1 100644 --- a/docs/design/sidebar-goal-nesting-audit.md +++ b/docs/design/sidebar-goal-nesting-audit.md @@ -8,6 +8,13 @@ > **Scope:** race conditions and redraw correctness in the sidebar's data flow; whether > goal/subgoal nesting is modelled and rendered optimally; reuse/composability defects. > Each backlog task (§4) is self-contained for hand-off to an implementer agent. +> +> **Status:** audit complete, backlog (T1–T9) not started. Workstream **SN** in +> [fable-program-execution-plan.md](fable-program-execution-plan.md). ⚠️ Seam overlap with +> the battery plan ([client-performance-battery.md](client-performance-battery.md)): +> **T2/T3 and PB-FX5 both rewrite `api.ts` `refreshSessions`/`startSessionPolling`** — land +> T2 (concurrency hardening) first, then FX5 (poll demotion) on top; **T5/T7 reshape the +> sidebar render paths PB-FX7 throttles** — sequence per the execution plan, not ad hoc. --- diff --git a/docs/design/token-cost-efficiency.md b/docs/design/token-cost-efficiency.md index 74f090792..607f413bc 100644 --- a/docs/design/token-cost-efficiency.md +++ b/docs/design/token-cost-efficiency.md @@ -1,7 +1,9 @@ # Token & Cost Efficiency — audit, findings, and remediation plan Status: **investigation complete, remediation not started** · Audited: 2026-06-10 against -master @ `4e7fce1f` and live state on the primary dev machine. +master @ `4e7fce1f` and live state on the primary dev machine. Workstream **CE** in +[fable-program-execution-plan.md](fable-program-execution-plan.md) (program sequencing + +master checklist). Companion docs: [extension-platform-implementation-plan.md](extension-platform-implementation-plan.md) (several goals here ride its pack surfaces), [comms-stack/04-current-state-and-backlog.md](comms-stack/04-current-state-and-backlog.md) (overlapping retry findings). From 75c3dc03785271be41a09bf04d41cbbb63a6e2b3 Mon Sep 17 00:00:00 2001 From: Claude <noreply@anthropic.com> Date: Thu, 11 Jun 2026 07:21:38 +0000 Subject: [PATCH 08/18] Add hand-off implementation plans for MC/AI/PB/GA; wave-ordered execution plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brings every workstream to extension-platform-implementation-plan.md grade (the program's gold standard): per-goal Outcome / Owned files / symbol-anchored Steps / tests-authored-first with RED expectations / Acceptance, plus anchor-baseline stop-rules and precision policies. - mission-control-implementation-plan.md (new) — MC-P0..P5; P0–P3 at file/function level (PersistedStaff at staff-store.ts:60, participatesInVisibleOrder pin, renderSidebar insertion, meta-tool registry + catalog pinning tests), P4/P5 contract level. - autoimprovement-implementation-plan.md (new) — AI-P1..P6; proposal store/panel plugin (PROPOSAL_TYPE_REGISTRY), judge + learned-skills pack, Improver with tool-whitelist pins, curator/dreaming/shadow/ autonomy at contract level with ceiling clamps pinned in tests. - client-performance-battery-implementation-plan.md (new) — PB-P0..P3 all at file/function level, with measurement discipline (no behavior-affecting merge without before/after harness numbers) and SN seam guards. - gap-easy-wins-implementation-plan.md (new) — GA-R2/R3/R4/R5/R6/R9, incl. the shared-seam rule that GA-R2's triggers land exactly once. - fable-program-execution-plan.md — §2.1 THE ORDER: wave diagram (W0 prereq → W5 loop closes) with per-wave parallel sets, blocking rules, user-visible milestones per wave, and the orchestrator's rule of thumb; workstream table now has a design vs execution- authority split; lanes point at the new plans. - mission-control.md, autoimprovement.md, client-performance-battery.md, harness-gap-analysis.md — execution-authority banners (the extension-platform.md pattern). https://claude.ai/code/session_01LBW68Z2KvA3EhQMq3c2wrR --- .../autoimprovement-implementation-plan.md | 213 ++++++++++++++ docs/design/autoimprovement.md | 9 +- ...performance-battery-implementation-plan.md | 190 +++++++++++++ docs/design/client-performance-battery.md | 9 +- docs/design/fable-program-execution-plan.md | 74 ++++- .../gap-easy-wins-implementation-plan.md | 179 ++++++++++++ docs/design/harness-gap-analysis.md | 11 +- .../mission-control-implementation-plan.md | 268 ++++++++++++++++++ docs/design/mission-control.md | 9 +- 9 files changed, 939 insertions(+), 23 deletions(-) create mode 100644 docs/design/autoimprovement-implementation-plan.md create mode 100644 docs/design/client-performance-battery-implementation-plan.md create mode 100644 docs/design/gap-easy-wins-implementation-plan.md create mode 100644 docs/design/mission-control-implementation-plan.md diff --git a/docs/design/autoimprovement-implementation-plan.md b/docs/design/autoimprovement-implementation-plan.md new file mode 100644 index 000000000..2e6437290 --- /dev/null +++ b/docs/design/autoimprovement-implementation-plan.md @@ -0,0 +1,213 @@ +# Autoimprovement — Implementation Plan (hand-off) + +Status: ready for execution, not started. Workstream **AI** in +[fable-program-execution-plan.md](fable-program-execution-plan.md). + +Companion to [autoimprovement.md](autoimprovement.md) (the WHAT/WHY — read §1–§6 and +**Appendix A** before implementing; the §2 change-class table and §5 ladder guard rails are +LAW — enforce in code, not prompts). + +> **Anchor baseline:** fable-docs @ 2026-06-11 (master parent `6ec8c8f9`). Locate by symbol +> name; missing symbol ⇒ STOP and re-derive from the cited pattern, never improvise. +> +> **Precision policy:** AI-P1/P2 file/function level; AI-P3…P6 contract level (substrate +> created by P1/P2) — re-verify anchors first. +> +> **Universal rules:** [extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) +> + [fable-program-execution-plan.md §1](fable-program-execution-plan.md). Depends on: +> MC-P0 (global scope) for the Improver staff home; GA-R2 (push triggers — shared seam, +> land there); MC-P3 (flight recorder — stub until it lands, like MC-P2a does). + +--- + +## Goal map + +``` +AI-P1 substrate ─→ AI-P2a judge+pack ─→ AI-P2b Improver ─→ AI-P3a curator ─→ AI-P3b dreaming + AI-P2b ─→ AI-P4 shadow ─→ AI-P5 autonomy ─→ AI-P6 measurement +``` + +--- + +## AI-P1 — Proposal substrate + +**Outcome:** an `ImprovementProposal` can be created by a tool, rendered in the proposal +panel with diff + evidence, decided by a human, and queried over REST. Nothing automatic. + +**Owned files:** NEW `src/server/agent/improvement-store.ts`; NEW +`defaults/tools/proposals/propose_improvement.yaml` (+ wiring in that group's +`extension.ts`); `src/app/proposal-registry.ts` + the proposal panel render path; +`server.ts` (routes); NEW `tests/improvement-store.test.ts`, +`tests/e2e/improvements.spec.ts`, `tests/e2e/ui/improvement-proposal.spec.ts`. + +**Steps** + +1. `improvement-store.ts`: implement the types from autoimprovement.md Appendix A.1 + **verbatim** (copy the interfaces; do not rename fields). JSONL at + `.bobbit/state/improvement/proposals.jsonl`: append-only, status change = append full + record same `id`, last-writer-wins on load; crash-safe write per + [session-store-crash-safety.md](session-store-crash-safety.md). Export + `createProposal`, `decideProposal`, `listProposals`, `getProposal`. +2. Enforce the §2 ceilings **in `decideProposal`**: `decision.by === "policy"` is rejected + outright in this phase (autonomy is AI-P5); class validation rejects unknown + `ChangeClass`. +3. REST per Appendix A.2: `GET /api/improvements`, `POST /api/improvements`, + `POST /api/improvements/:id/decision`. (`/revert` and `/calibration` come later — do + not stub routes.) +4. Tool: `propose_improvement.yaml` mirroring `propose_staff`'s YAML shape in + `defaults/tools/proposals/` (params per Appendix A.3: + `change_class, title, diff?, files?, evidence`); provider `bobbit-extension` calling + `POST /api/improvements`. +5. Client: extend `ProposalType` union (`src/app/proposal-registry.ts:21`), add + `"improvement"` to `PROPOSAL_TYPES` (`:23`), and register a `ProposalTypePlugin` + (`:60`, registry at `PROPOSAL_TYPE_REGISTRY:277`) rendering: title, change-class badge, + evidence bundle (signal, session links, excerpts), unified diff (reuse the existing + proposal diff component — see `src/app/project-proposal-diff.ts`), Approve / Reject + (+required reason on reject). Inline editing rides the editable-proposals machinery — + reuse, don't fork ([editable-proposals.md](editable-proposals.md)). + +**Tests (author first; RED unless noted)** + +- Unit: store round-trip; status append/last-writer-wins; crash-mid-append load; unknown + class rejected; `by: "policy"` rejected. +- API E2E: create → list → decide(approve) → status `approved-human`; + decide(reject, no reason) → 400. +- Browser E2E: hand-POST a proposal; panel renders class badge + diff + evidence; reject + persists reason across reload. + +**Acceptance:** all green; proposal-panel suites for existing types green unmodified; +`tests/test-phase-invariant.test.ts` green (new tests land in the right phases). + +--- + +## AI-P2a — Judge + learned-skills pack (inert plumbing) + +**Outcome:** proposals get judged confidence scores; an approved `skill-new` lands as a +staged skill in a server-scoped `learned-skills` pack; skill usage is tracked. No staff yet. + +**Owned files:** NEW `src/server/agent/improvement-judge.ts`; NEW +`src/server/agent/learned-skills-pack.ts`; skill-usage hook in +`src/server/agent/slash-skills.ts` (one call site); NEW `tests/improvement-judge.test.ts`, +`tests/learned-skills-pack.test.ts`. + +**Steps** + +1. `improvement-judge.ts`: `judgeProposal(p) → {confidence, rubricVersion, model}` — one + non-streaming aux-model call. Model resolution: the `auxiliary.improver-judge` slot + (Appendix A.4); **first lander defines the `auxiliary.*` config shape** per execution + plan §1.4 — follow [per-role-model-overrides.md](per-role-model-overrides.md) + conventions for resolution + fallback-to-main-model on `auto`. Rubric text loaded from + the pack path (Appendix A.3), keyed by `ChangeClass`; missing rubric ⇒ judge returns + no score (never throws). Judge result appended onto the proposal record. +2. `learned-skills-pack.ts`: create-on-first-use server-scoped pack (resolver scope order + already supports it — see `pack-types.ts` scope order in EP plan §0.1); approved + `skill-new.files[]` written under it with a `staged: true` marker the skills catalog + renders as `[learned — staged]`; `pack_activation` per-entity disable must work on + these like any pack skill (pin it). +3. Usage hook: where `slash-skills.ts` resolves/loads a skill, append + `{skillName, sessionId, ts}` to `.bobbit/state/improvement/skill-usage.jsonl` + (cheap fire-and-forget; this feeds AI-P3 lifecycle and AI-P6). + +**Tests:** judge — rubric routing per class, `auto` fallback, missing-rubric no-throw +(mock the model client; no live LLM in unit/e2e). Pack — approve `skill-new` ⇒ skill +resolvable with staged marker; `pack_activation` disable works; usage hook appends once +per load. *(RED)* + +**Acceptance:** green; pack-resolver and skills suites green unmodified. + +## AI-P2b — The Improver (human-in-the-loop learning live) + +**Outcome:** archiving a goal with correction signals produces a judged +`propose_improvement` in the panel, end-to-end, with a human deciding. + +**Owned files:** Improver role + skills + rubrics inside +`market-packs/mission-control/` (coordinate with MC-P4a — if it hasn't landed, this PR +creates the pack skeleton per mission-control.md Appendix A.5 and MC-P4a extends it); +trigger wiring only via existing staff machinery; NEW `tests/e2e/improver-loop.spec.ts`. + +**Steps** + +1. Improver role prompt = autoimprovement.md §1a heuristics **verbatim** (active stance, + corrections-first, umbrella shape, patch-first order 1→4) + standing-orders charter + (GA-R3 format) + §3 evidence-bundle requirements. Lives in the pack, never in server + source. +2. Staff template: `global: true`, role above, triggers `goal_archived` + `gate_failed` + (from GA-R2) + weekly `schedule`, all **disabled by default** (mission-control.md §5). +3. Tool whitelist via role `toolPolicies`: read/search tools + `propose_improvement` + ONLY (the Hermes fork-whitelist discipline, §1a). Pin: Improver role cannot call + write/bash tools. +4. Wire judge: `POST /api/improvements` triggers `judgeProposal` asynchronously; panel + shows confidence when present. + +**Tests:** API E2E with a mock agent (existing mock-agent harness): seed a session +transcript containing a planted correction ("stop doing X"); archive the goal; assert an +inbox entry fires, the Improver session's proposal appears with evidence excerpts +referencing the planted text, judge score attached; approving creates the staged skill +(AI-P2a path). Tool-whitelist pin: Improver session denied a bash call. *(RED)* + +**Acceptance:** the full observe→propose→judge→human-approve→staged-skill path green with +zero live-LLM dependence in CI (mock agent + mocked judge). + +--- + +## AI-P3a — Curator · AI-P3b — Dreaming *(contract level — re-verify anchors)* + +Contracts: autoimprovement.md §1b (Hermes semantics: deterministic transitions, bounded +LLM pass, never-delete, pin, snapshot+rollback, dry-run, first-run grace), §4 lifecycle +states, Appendix A.1 (lock/snapshot paths), A.4 (config defaults), A.5 (owned files). + +- **P3a:** `improvement-curator.ts` — `runCurator({dryRun})`: phase 1 deterministic + (usage-file derived `active→stale→archived`, timings from config; `pinned` exempt; + archive = move under `skill-archive/`, never delete); phase 2 bounded aux-model pass + emitting `skill-lifecycle`/`skill-patch` **proposals** (Bobbit divergence from Hermes: + no direct writes — §1 adopt/adapt table). tar.gz snapshot before any apply; REST: + pin/unpin/restore/dry-run + snapshot list/rollback; staff-page controls. First-run + seeds `lastRunAt = now`. Tests: fake-clock transitions; snapshot/rollback round-trip + (incl. rollback-of-rollback); dry-run mutates nothing (tree hash compare); pinned + untouchable. +- **P3b:** dreaming = Archivist staff scheduled wake whose gate check + (`time → new-transcripts-count → dream.lock`, autoDream ordering from §1) runs + server-side **before** the staff is woken (cheap gates outside the LLM); the wake mines + recent transcripts (read_session/search whitelist) and emits `memory`/`skill-new` + proposals. Tests: gates closed ⇒ no wake; lock prevents concurrent dream; dream run + writes a flight-recorder entry. + +**Acceptance:** staged skill ages stale→archived→restored across fake-clock runs; dreams +appear in the activity panel; zero direct skill mutations anywhere in P3 paths. + +## AI-P4 — Shadow calibration *(contract level)* + +Contract: §5 L0.5 + Appendix A.2 calibration endpoint. Judge runs on every proposal +(AI-P2b already does); `decideProposal` stores +`(confidence, wouldAutoApproveAt: thresholds[], humanDecision)`; `GET +/api/improvements/calibration?class` computes the per-threshold agreement matrix; Improver +staff page renders it ("at θ=0.85: 94% agreement, 1 false-approve / 31"). Tests: matrix +math unit tests with seeded decisions; endpoint e2e. **Zero auto-applies — pin it** (a +test that a high-confidence proposal still requires a human decision at level 0.5). + +## AI-P5 — Graduated autonomy *(contract level)* + +Contracts: §5 ladder + guard rails, Appendix A.4 keys, A.2 `/revert`. Server-side ceiling +clamp (config requesting L1 for `prompt-context` ⇒ rejected with 400 — pin); policy path +in `decideProposal` (`by: "policy"` now legal when class level + threshold + ceiling all +pass); apply with snapshot; flight-recorder entry `improvement.auto-approved` with +`revert` ref; inbox digest of auto-approvals; `POST /api/improvements/:id/revert` +restores snapshot + appends `reverted` + **demotes the class one level** (persisted in +config store); global kill switch (`autoimprovement.enabled: false` ⇒ everything behaves +as L0). Tests: e2e auto-apply happy path; revert demotes; ceiling clamp; kill switch; +every policy decision visible in activity feed. + +## AI-P6 — Measurement loop *(contract level)* + +Contract: §6. Scheduled evaluator pass (inside the curator run) computes +`helped/neutral/regressed` per applied change from: skill-usage + post-use goal/gate +outcomes, recurrence of the originating correction signal, cost deltas when CE-G0.1's +ledger exists (omit cleanly when absent). `regressed` ⇒ flight-recorder entry + automatic +demotion + an auto-generated revert proposal (NOT auto-revert). Test: planted regression +fixture (staged skill whose sessions then fail a gate) detected and demoted within one +evaluation window (fake clock). + +**Program acceptance (AI complete):** with levels at defaults (0) the system only ever +proposes; raising `skill-new` to 1 after shadow calibration auto-applies high-confidence +drafts, every one visible and revertible in the activity panel; killing the switch stops +all autonomy instantly. All suites green; no flaky tests. diff --git a/docs/design/autoimprovement.md b/docs/design/autoimprovement.md index 00fe6615a..57db21621 100644 --- a/docs/design/autoimprovement.md +++ b/docs/design/autoimprovement.md @@ -1,8 +1,11 @@ # Autoimprovement — recursive self-improvement with earned autonomy -Status: proposal / implementation blueprint. Each phase in §7 converts 1:1 into a mission -goal with acceptance criteria; steps name exact files and reuse points so they can be -executed without re-deriving this analysis. +Status: design accepted, not started. This is the WHAT/WHY + contracts (Appendix A). + +> **Execution authority:** implement from +> [autoimprovement-implementation-plan.md](autoimprovement-implementation-plan.md) +> (per-goal owned files, symbol anchors, RED→GREEN tests, acceptance). Sequencing: +> [fable-program-execution-plan.md](fable-program-execution-plan.md). **What this is:** Bobbit learning from its own usage — drafting new skills after notable sessions, patching skills that misfired, proposing AGENTS.md/prompt/config improvements, diff --git a/docs/design/client-performance-battery-implementation-plan.md b/docs/design/client-performance-battery-implementation-plan.md new file mode 100644 index 000000000..9dd9f5c5f --- /dev/null +++ b/docs/design/client-performance-battery-implementation-plan.md @@ -0,0 +1,190 @@ +# Client Performance & Battery — Implementation Plan (hand-off) + +Status: ready for execution, not started. Workstream **PB** in +[fable-program-execution-plan.md](fable-program-execution-plan.md). + +Companion to [client-performance-battery.md](client-performance-battery.md) (the WHAT/WHY — +read §1–§6 and **Appendix A**; its A.1 animation table and A.4 poll table are the definite +work lists). + +> **Anchor baseline:** fable-docs @ 2026-06-11 (master parent `6ec8c8f9`). Locate by symbol +> name; missing symbol ⇒ STOP, re-derive from the cited pattern. +> +> **Precision policy:** all goals file/function level (this workstream touches few files). +> +> **Universal rules:** [extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) +> + [fable-program-execution-plan.md §1](fable-program-execution-plan.md). **Seam guard:** +> PB-P2a must land AFTER SN-T2/T3, and PB-P2c after (or rebased onto) SN-T5 — see the +> execution plan §1.4 table; check the §4 checklist before starting those goals. +> +> **Measurement discipline:** no behavior-affecting goal merges without before/after +> numbers from the PB-P0 harness in the PR description, and the doc's §7 table updated in +> the same PR. + +--- + +## Goal map + +``` +PB-P0 harness ─→ PB-P1a anim gating ─→ PB-P1b paint-prop rewrites + └─→ PB-P2a poll demotion (after SN-T2/T3) · PB-P2b scoped tick · + PB-P2c render throttle (after SN-T5) · PB-P2d timer audit +PB-P1 + PB-P2 ─→ PB-P3 battery saver + flag soak + re-baseline +``` + +--- + +## PB-P0 — Measurement harness + +**Outcome:** `window.__bobbitPerf.report()` returns the Appendix A.3 shape in all four §4 +protocol states; baseline table committed. Zero default-path behavior change. + +**Owned files:** NEW `src/app/perf-monitor.ts`; `src/app/state.ts` (ONE guarded line in +`renderApp()`, symbol at `state.ts:687`); `src/app/perf-flags.ts` (flag registration only); +doc §7 table; NEW `tests/perf-monitor.spec.ts` (browser fixture). + +**Steps** + +1. `perf-monitor.ts` per Appendix A.3: render counter (incremented from the guarded hook + inside `renderApp`'s rAF callback — read how `_renderScheduled` works first), timer + wrap (monitor mode only: wrap `window.setInterval`/`setTimeout`, bucket by + `new Error().stack` top app frame), `PerformanceObserver longtask`, + `document.getAnimations().length` sampled 1/s, WS frames/s by type (hook the client WS + dispatch — find the single switch over frame types in `src/app/`). +2. Activation: only when `bobbitPerfFlags` contains `+perfMonitor` + (`isPerfFlagEnabled` — read `perf-flags.ts` and `docs/perf/defer-offscreen-render.md` + for the conventions). Module must not be imported on the default boot path — lazy + `import()` behind the flag check in `main.ts`. +3. Run the §4 protocol (4 states × 60 s × 3 runs × dev+prod) **manually**, fill the doc §7 + baseline columns, include the OS-level energy reading (§4 step 4). + +**Tests:** browser fixture — flag off ⇒ `window.__bobbitPerf` undefined *(GREEN pins the +default path)*; flag on ⇒ report returns every A.3 field with sane types *(RED)*. + +**Acceptance:** baseline table filled in the same PR; `npm run test:unit` green. + +--- + +## PB-P1a — Ambient animation gating + +**Outcome:** idle-or-hidden Bobbit runs **zero** animations +(`document.getAnimations().length === 0`); working agents still animate. + +**Owned files:** NEW `src/app/animation-power.ts`; `src/app/main.ts` (one init call); +`src/app/app.css`, `src/app/goal-dashboard.css`, `src/ui/app.css` (gates per the A.1 +table); NEW `tests/e2e/ui/animation-power.spec.ts`, `tests/animation-gate.test.ts`. + +**Steps** + +1. Re-run the A.1 grep; reconcile drift (new rules ⇒ classify + extend the table in this + PR — execution plan §1.2). +2. `animation-power.ts` per Appendix A.2 **exactly**: inputs are `visibilitychange`, the + session-status update path (find where `state.gatewaySessions` statuses are written — + `api.ts` `updateLocalSessionStatus` and the `refreshSessions` apply step — and call a + re-evaluate from there), `prefers-reduced-motion` media query listener, and (later) + battery-saver. Output: toggle `anims-paused` on `document.documentElement`. No timers. +3. CSS: for every **ambient** row in A.1, add the pause guard + (`html.anims-paused <selector> { animation-play-state: paused; }` grouped in one block + per file); for every **status** row, move/confirm the animation under its + working-state selector so it doesn't run at idle (the `blob-*-idle-sleep-breathe` + loops are the flagship: idle/sleep sprites become static frames). Remove `!important` + from `blob-compact-squash*` while adding gates. NEVER a bare `html.anims-paused *` + rule (A.2 freeze hazard). +4. Flag: `-pauseAmbientAnims` restores current behavior (gate the class-toggle, not the + CSS). + +**Tests (author first):** + +- `tests/animation-gate.test.ts` (node): parse the three CSS files; every + `animation:.*infinite` rule must match an A.1 bucket: ambient ⇒ a pause guard exists + for its selector; status ⇒ selector includes a state class; else FAIL ("the missing + test IS the bug" — this is the never-reintroduce pin). *(RED)* +- Browser E2E: all-idle fixture ⇒ class present + `getAnimations()` = 0; mock agent + working ⇒ its sprite animates and count > 0; tab hidden ⇒ 0; flag opt-out restores. + *(RED)* + +**Acceptance:** E2E + pin green; harness shows idle-foreground animations 0 and paint +events <2/s; visual check: working sprites unchanged (record before/after GIF in PR). + +## PB-P1b — Paint-property rewrites + reduced-motion sweep + +**Owned files:** `src/app/app.css` only. **Steps:** rewrite `status-pulse`, +`unseen-dot-pulse`, `pr-conflict-pulse` to opacity/transform on a pseudo-element halo +(FX3 — visually equivalent: pre-rendered shadow layer scaling/fading); add the blanket +reduced-motion rule (FX4) and delete the now-redundant piecemeal opt-outs (grep +`prefers-reduced-motion`, currently 9 sites). **Tests:** extend `animation-gate.test.ts`: +no `@keyframes` in the three files may animate `box-shadow` *(RED until rewritten)*. +**Acceptance:** pin green; screenshots before/after in PR. + +--- + +## PB-P2a — Session-poll demotion *(after SN-T2/T3 — verify §4 checklist first)* + +**Owned files:** `src/app/api.ts` (`startSessionPolling`, symbol near `api.ts:273`, plus +the WS-health signal); NEW `tests/session-poll-policy.test.ts`. + +**Steps:** extract interval selection into a pure +`sessionPollIntervalMs({wsConnected, visible})` → 60 000 when WS healthy, 5 000 otherwise +(read how the client tracks WS connection state first — reuse, don't add a tracker); +immediate `refreshSessions()` on WS reconnect and on `visibilitychange`→visible (the +git-status catch-up pattern, `session-manager.ts` near `startGitStatusPoll`). Rebase onto +SN-T2's coalescing/generation guard — do not reimplement any part of it. + +**Tests:** pure-function unit table *(RED)*; existing +`tests/spurious-idle-unread.spec.ts` green unmodified. **Acceptance:** harness: +idle-foreground network ≤1 sessions fetch/min with WS up; status updates still arrive +(WS push path) — E2E asserts a status change renders without a poll tick (fake timers). + +## PB-P2b — Scoped verification tick + +**Owned files:** `src/app/goal-dashboard.ts` (`startLiveVerifTimer`, symbol near `:1090`). +**Steps:** the 1 s `setInterval(() => renderApp(), 1000)` becomes a scoped updater writing +`textContent` on the elapsed-time node(s) only (give them a stable +`data-testid="verif-elapsed"`); status *transitions* still arrive via the existing WS/event +path and call `renderApp()` as today; keep `stopLiveVerifTimerIfDone` semantics. +**Tests:** browser E2E — while a mock verification runs, elapsed label advances ≥2 s while +the perf-monitor render counter advances 0 in the same window; timer stops when the run +ends *(RED)*. **Acceptance:** dashboard suites green unmodified. + +## PB-P2c — Streaming render throttle *(after/rebased on SN-T5)* + +**Owned files:** `src/app/state.ts` (add `renderAppThrottled(ms = 100)` beside `renderApp`, +reusing `_renderScheduled`/suppression machinery — trailing edge, max one render per +window); the streaming dispatch call sites (`message_update`-class frames in the client WS +switch; terminal frames keep plain `renderApp()`). Flag `-throttleStreamRender`. +**Tests:** unit — burst of N calls in 100 ms ⇒ 1 render, trailing call lands; terminal +frame flushes immediately *(RED)*. Browser E2E — long mock stream: perf-counter ≤15 +renders/s; `follow-tail` suites green unmodified. **Acceptance:** harness streaming state +~10 Hz; no visible degradation (manual check noted in PR). + +## PB-P2d — Timer audit + +**Owned files:** convicted loops only (expect `goal-dashboard.ts`, `dialogs.ts`). +**Steps:** run PB-P0's wakeup-by-callsite report in the idle + dashboard states; commit the +table into the doc (§F2 update, same PR); every surviving loop ≥1 tick/min gets the +git-status pattern (visibility gate + event coalescing + self-cancel). One PR may fix +several loops if each has its own test. **Acceptance:** idle-foreground wakeups <8/min on +the harness; doc table updated. + +--- + +## PB-P3 — Battery saver + flag soak + re-baseline + +**Owned files:** settings page (pattern `tests/e2e/ui/settings.spec.ts`), +`animation-power.ts` (battery-saver input), `api.ts`/`state.ts` (cadence overrides), doc +§7 after-table; flag removals per `docs/perf/defer-offscreen-render.md` §"when to remove". + +**Steps:** (1) `batterySaver` setting (persisted with other UI settings): forces +`anims-paused`, poll 120 s, throttle 250 ms, shimmer off. (2) `navigator.getBattery()` +suggestion banner when discharging — suggest only, never auto-enable; dismissal persists. +(3) Remove soaked temporary flags (keep `anims-paused` as mechanism). (4) Re-run the full +§4 protocol; fill §7 after-columns + OS energy reading; only then evaluate whether any §3 +target is still missed (heavier options stay out of scope unless data convicts — doc §8). + +**Tests:** browser E2E — toggle persists across reload; saver state pauses animations +while a mock agent works (overrides status animations); banner dismissal persists. *(RED)* + +**Acceptance:** §3 targets table met on the harness (idle-foreground: 0 animations, +<2 paint/s, <8 wakeups/min, ≈0% CPU; hidden: 0 renders, ≤1 poll/min; streaming ~10 Hz, +no >50 ms render long-tasks; dashboard: no full-tree render/s) — or a doc deviation note +explaining the miss and the follow-up goal. diff --git a/docs/design/client-performance-battery.md b/docs/design/client-performance-battery.md index fae0e3bfd..eeb774455 100644 --- a/docs/design/client-performance-battery.md +++ b/docs/design/client-performance-battery.md @@ -1,8 +1,11 @@ # Client performance & battery — analysis and implementation plan -Status: proposal / implementation blueprint. Each phase in §6 is written to convert 1:1 into a -mission goal with acceptance criteria, and each step is explicit enough to execute without -re-deriving the analysis. +Status: design accepted, not started. This is the WHAT/WHY + contracts (Appendix A). + +> **Execution authority:** implement from +> [client-performance-battery-implementation-plan.md](client-performance-battery-implementation-plan.md) +> (per-goal owned files, symbol anchors, RED→GREEN tests, measurement discipline). +> Sequencing: [fable-program-execution-plan.md](fable-program-execution-plan.md). **The symptom this answers:** Bobbit visibly drains laptop battery — noticeably worse than leaving a typical web app open. The goal: an *idle* Bobbit tab should cost ≈0% CPU with zero diff --git a/docs/design/fable-program-execution-plan.md b/docs/design/fable-program-execution-plan.md index 0e7de16de..1c4e129ca 100644 --- a/docs/design/fable-program-execution-plan.md +++ b/docs/design/fable-program-execution-plan.md @@ -7,16 +7,19 @@ it creates goals from §3, in lane order, and ticks §4 as PRs merge. The workstreams (each doc owns its *content* — tasks, contracts, acceptance; this doc owns *ordering, slicing, and the checklist*): -| Workstream | Doc | Granularity there | +| Workstream | Design (WHAT/WHY) | Execution authority (HOW) | |---|---|---| -| **EP** extension platform | [extension-platform.md](extension-platform.md) (design) + [extension-platform-implementation-plan.md](extension-platform-implementation-plan.md) (execution authority) | G1.1–G9, owned files, RED→GREEN tests | -| **CE** token/cost efficiency | [token-cost-efficiency.md](token-cost-efficiency.md) §6 | CE-G0.1–G7.2, lanes, BENCH gates | -| **CS** comms-stack reliability | [comms-stack/04-current-state-and-backlog.md](comms-stack/04-current-state-and-backlog.md) §6–§7 | CS-R/H/D/P/T waves + merge map | -| **SN** sidebar & goal-nesting | [sidebar-goal-nesting-audit.md](sidebar-goal-nesting-audit.md) §4 | T1–T9, files/diagnosis/fix/acceptance | -| **PB** client perf/battery | [client-performance-battery.md](client-performance-battery.md) + Appendix A | P0–P3, FX1–FX9, owned files | -| **MC** Mission Control | [mission-control.md](mission-control.md) + Appendix A | P0–P5, contracts, owned files | -| **AI** autoimprovement | [autoimprovement.md](autoimprovement.md) + Appendix A | P1–P6, contracts, owned files | -| **GA** gap-analysis easy wins | [harness-gap-analysis.md](harness-gap-analysis.md) §5 | R2–R6, R9 (R1/R7/R8 alias PB/MC/AI) | +| **EP** extension platform | [extension-platform.md](extension-platform.md) | [extension-platform-implementation-plan.md](extension-platform-implementation-plan.md) (G1.1–G9) | +| **CE** token/cost efficiency | [token-cost-efficiency.md](token-cost-efficiency.md) §1–§5 | same doc, §6 (CE-G0.1–G7.2, BENCH gates) | +| **CS** comms-stack reliability | [comms-stack/01–03](comms-stack/01-understanding.md) | [comms-stack/04-current-state-and-backlog.md](comms-stack/04-current-state-and-backlog.md) §6–§7 (waves + merge map) | +| **SN** sidebar & goal-nesting | [sidebar-goal-nesting-audit.md](sidebar-goal-nesting-audit.md) §1–§3 | same doc, §4 (T1–T9) | +| **PB** client perf/battery | [client-performance-battery.md](client-performance-battery.md) | [client-performance-battery-implementation-plan.md](client-performance-battery-implementation-plan.md) | +| **MC** Mission Control | [mission-control.md](mission-control.md) | [mission-control-implementation-plan.md](mission-control-implementation-plan.md) | +| **AI** autoimprovement | [autoimprovement.md](autoimprovement.md) | [autoimprovement-implementation-plan.md](autoimprovement-implementation-plan.md) | +| **GA** gap-analysis easy wins | [harness-gap-analysis.md](harness-gap-analysis.md) | [gap-easy-wins-implementation-plan.md](gap-easy-wins-implementation-plan.md) | + +**An implementer agent receives exactly one goal ID and reads: the execution-authority doc's +goal section → the contracts it cites → §1 below. Nothing else is required.** --- @@ -83,6 +86,55 @@ graph TD **Day-1 starter set** (parallel, no collisions): AGENTS.md trim (§5) · CS-R1 · SN-T1 · PB-P0 · CE-G0.1 · EP G1.1 · GA-R2 · GA-R3. +## §2.1 THE ORDER — waves (the operational answer to "what do we work on now?") + +The §2 graph is the truth about *dependencies*; this section is the truth about *sequence*. +A wave starts only when its blocking entries from the previous wave are merged (non-blocking +stragglers from an earlier wave may continue in parallel). Within a wave, everything is +parallel-safe **provided the §1.4 seam table is respected**. + +```mermaid +flowchart TD + W0["WAVE 0 — prereq (hours) + AGENTS.md trim → suite green"] + W1["WAVE 1 — foundations, all parallel + CS-R1 (outage fix) · SN-T1 (pin tree builders) + PB-P0 (perf baseline) · CE-G0.1 (cost ledger) + EP G1.1 (manifest v2) · GA-R2 (triggers) · GA-R3 (standing orders)"] + W2["WAVE 2 — hygiene + scope + CS: H1‖H2 → R2 → R5‖R6 (its §7 order) + SN: T2 → T3, T4 · PB: P1a → P1b + CE: G0.2, G0.3, G1.0 · EP: G1.2 → G1.3 + MC: P0 → P1 · GA: R9, R6 (anytime)"] + W3["WAVE 3 — spines + CS: R3 → R7 → R4 → R8‖R9 · SN: T5 (the big one), T8, T9 + PB: P2a (after SN-T2/T3) → P2b → P2d + CE: post-G1.0 lanes (G2, G7; BENCH-gated G4/G5 wait for G0.3) + EP: G1.4 → G1.5 → G1.6 · MC: P2a → P2b, P3 · AI: P1"] + W4["WAVE 4 — features on the spines + SN: T6, T7 · PB: P2c (after SN-T5) → P3 + CS: waves 2–4 (D*, P*, T*) · EP: G2 → G3 + MC: P4a → P4b · AI: P2a → P2b · GA: R4, R5 (R5 prefers EP G1.6)"] + W5["WAVE 5 — the loop closes + AI: P3a → P3b → P4 → P5 → P6 · MC: P5 + EP: G4…G9 · CE: remaining BENCH-gated goals"] + W0 --> W1 --> W2 --> W3 --> W4 --> W5 +``` + +Milestones the waves deliver (what the user can feel): + +| After | The product visibly gains | +|---|---| +| Wave 1 | green suite, cost + perf both measurable, staff can fire one-shot/`gate_failed` triggers | +| Wave 2 | battery: idle tab stops animating; sidebar races hardened; Mission Control entry exists | +| Wave 3 | `bobbit` meta-tool live (chat can run everything); flight recorder visible; one nesting render path | +| Wave 4 | system-staff crew + dashboard; human-in-the-loop self-improvement proposing skills | +| Wave 5 | dreaming, calibrated autonomy with revert/demotion, briefing + onboarding; EP ecosystem complete | + +**Rule of thumb for the orchestrator:** at any moment, prefer (1) the lowest-numbered +unfinished wave, (2) within it, the entry that unblocks the most §2 edges, (3) never two +in-flight goals on the same §1.4 seam. + ## §3 Lanes and PR slicing Effort: S ≤ 1 day · M ≤ 3 days · L ≈ a week. EP, CE, and CS are **not re-sliced here** — @@ -95,7 +147,7 @@ workstreams: T1 (M, no deps) → T2 (M) → T3 (S) → T4 (M) → T5 (L, after T1+T4) → T6 (M) → T7 (S) → T8 (S, independent) → T9 (S, independent). T2/T3/T5/T7 gate PB-P2 (§1.4). -### Lane PB — perf/battery (contracts in its Appendix A) +### Lane PB — perf/battery (execute from [client-performance-battery-implementation-plan.md](client-performance-battery-implementation-plan.md)) | PR | Scope | Effort | Mergeable because | |---|---|---|---| @@ -108,7 +160,7 @@ T8 (S, independent) → T9 (S, independent). T2/T3/T5/T7 gate PB-P2 (§1.4). | PB-P2d | FX8 timer audit + convicted-loop fixes | M | per-loop independent | | PB-P3 | battery-saver mode + flag-soak removal + after-table | M | additive setting | -### Lane MC→AI — Mission Control then autoimprovement (contracts in their Appendices A) +### Lane MC→AI — Mission Control then autoimprovement (execute from [mission-control-implementation-plan.md](mission-control-implementation-plan.md), [autoimprovement-implementation-plan.md](autoimprovement-implementation-plan.md), [gap-easy-wins-implementation-plan.md](gap-easy-wins-implementation-plan.md)) | PR | Scope | Effort | Mergeable because | |---|---|---|---| diff --git a/docs/design/gap-easy-wins-implementation-plan.md b/docs/design/gap-easy-wins-implementation-plan.md new file mode 100644 index 000000000..1738ccc2d --- /dev/null +++ b/docs/design/gap-easy-wins-implementation-plan.md @@ -0,0 +1,179 @@ +# Gap-Analysis Easy Wins — Implementation Plan (hand-off) + +Status: ready for execution, not started. Workstream **GA** in +[fable-program-execution-plan.md](fable-program-execution-plan.md). + +Companion to [harness-gap-analysis.md](harness-gap-analysis.md) (the WHAT/WHY — read the +owning G-section for each goal; §5 holds the R2 trigger schema, which is LAW). + +> **Anchor baseline:** fable-docs @ 2026-06-11 (master parent `6ec8c8f9`). Locate by symbol +> name; missing symbol ⇒ STOP, re-derive from the cited pattern. +> +> **Universal rules:** [extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) +> + [fable-program-execution-plan.md §1](fable-program-execution-plan.md). GA-R2 is a +> **shared seam** (MC-P4 and AI-P1 consume its triggers) — it lands here, exactly once. + +Goals are independent unless stated; R2 and R3 are Day-1 starters. + +--- + +## GA-R2 — One-shot `at` trigger + `gate_failed`/`session_errored` push triggers + +**Outcome:** any staff can fire once at an ISO time, and react to gate failures / session +errors, per the schema in harness-gap-analysis.md §5 (verbatim — field names included). + +**Owned files:** `src/server/agent/staff-store.ts` (`StaffTrigger` config union), +`src/server/agent/staff-trigger-engine.ts` (`TriggerEngine`, schedule eval near `:220`), +`src/server/agent/goal-trigger-dispatcher.ts` (`GoalTriggerDispatcher`, symbol at `:30`), +the gate-failure + session-error mutation points (located in step 3), staff edit page +trigger editor (`src/app/staff-page.ts`); NEW `tests/staff-trigger-at.test.ts`, +`tests/staff-trigger-push.test.ts`; extend the staff-triggers e2e suite; +`docs/staff-triggers.md` (table rows). + +**Steps** + +1. **`at` one-shot.** Extend the `schedule` trigger config union with `{ at: string }` + (ISO-8601). In `TriggerEngine`'s schedule evaluation (the branch that currently + requires `trigger.config.cron` — `if (!trigger.config.cron) return false;` near + `:220`): when `at` is present instead, fire iff `now >= at && !lastFired`, then set + `enabled: false` on the trigger record (kept for audit, per the §5 contract). Invalid + `at` (unparseable / cron+at both set) ⇒ reject at PUT-validation time, not silently at + eval time. +2. **Trigger editor UI:** the staff edit page's schedule row gains a cron-vs-datetime + choice (native `datetime-local`, serialized to ISO UTC). Disabled-after-fire state + renders with a "fired <ts>" hint. +3. **Push triggers.** Read how `GoalTriggerDispatcher` wires `onGoalCreated`/ + `onGoalArchived` from `GoalStore.put`/`archive` (`docs/staff-triggers.md` §"Why push, + not poll"), then mirror it exactly for: (a) `gate_failed` — from the gate + verification-result mutation point (find the single place a gate step transitions to + failed in the verification harness; the human-signoff doc §1 names the step types); + (b) `session_errored` — from the session-status write where status becomes `errored` + (**`session-manager.ts` is a §1.4 shared seam** — confine to the status-transition + function, coordinate with CS merge order). Both types require a `prompt` + (the `goal_created` required-prompt rule — same validation path). +4. Docs: add both rows + the `at` variant to the `docs/staff-triggers.md` type table. + +**Tests (author first; RED):** `at` fires once with a fake clock, then `enabled === false` +and never re-fires; `at` in the past at creation fires on next tick exactly once; +cron+at rejected; push: gate-fail fixture enqueues one inbox entry with the trigger's +prompt; re-failing the same gate re-fires (each transition is an event); prompt-less push +trigger rejected at PUT. E2E: editor round-trip (create `at` trigger → reload → fired +state visible). + +**Acceptance:** new + existing trigger/inbox suites green; `docs/staff-triggers.md` +updated in the same PR. + +--- + +## GA-R3 — Standing-orders convention (docs + assistant guidance) + +**Outcome:** a documented Scope/Trigger/Approval-gates/Escalation template, and the staff +creation assistant proposes prompts in that shape. + +**Owned files:** `docs/staff-agents.md` (new §"Standing orders"); the staff assistant's +guidance text in `src/server/agent/staff-assistant.ts`; NEW assertion in the staff +assistant's existing test coverage. + +**Steps:** (1) add the template + one filled example (the gap doc §G4 anatomy, adapted to +a Bobbit staff prompt) to `docs/staff-agents.md`; (2) extend the assistant's +system-prompt guidance: proposed `systemPrompt`s for recurring/triggered staff SHOULD +follow the four-section template (do not hard-fail free-form prompts); (3) the +mission-control pack roster (MC-P4a) consumes this format — note it there, build nothing +for it here. + +**Tests:** assistant prompt-text pin (the guidance mentions the four section names); +docs link-check. **Acceptance:** S-size PR, no behavior change beyond assistant guidance. + +--- + +## GA-R4 — "While you were away" summary slice + +**Outcome:** opening a session with a large unread backlog shows a collapsible +cheap-model summary banner above the transcript tail. Off by default (setting). + +**Owned files:** server summarizer endpoint (reuse the compaction-summary generation path +— locate via `docs/compaction.md` / `compaction-history.md` before writing anything new); +`auxiliary.away-summary` model slot (**shared `auxiliary.*` seam — §1.4: reuse the shape +the first lander defined; if none landed yet, this PR defines it per +[per-role-model-overrides.md](per-role-model-overrides.md)**); client banner component in +the chat header area; settings toggle; NEW `tests/e2e/ui/away-summary.spec.ts`. + +**Steps:** (1) `GET /api/sessions/:id/away-summary?since=<lastSeen>` — summarizes messages +since `since` with the aux model, returns `{summary, messageCount}`; hard cap input via +the existing truncation helper (`truncate-large-content.ts`); (2) client: when the +setting is on and unread count ≥ threshold (default 10), fetch + render collapsible +banner; dismissal stores per-session; (3) never block transcript render on the fetch. + +**Tests:** API e2e with mock model (deterministic stub summary); browser E2E — banner +renders over threshold, absent under, dismissal survives reload, setting-off ⇒ no fetch +(network spy). **Acceptance:** zero added latency to session open (banner is async); +suites green. + +--- + +## GA-R5 — Bounded global user-profile memory *(coordinate with EP `session-memory`)* + +**Outcome:** a Hermes-style bounded user profile (~500 tokens) injected as a stable prompt +section into every session, agent-managed via one tool. + +**Owned files:** NEW `src/server/agent/user-profile-store.ts` +(`.bobbit/state/user-profile.md`, char-capped 1,375 like Hermes USER.md); prompt section +via `system-prompt.ts` `PromptParts` (the byte-budget pattern — +`buildSkillsCatalogSection`/`skillsCatalogBudget`, §0.1 patterns library); NEW +`defaults/tools/memory/user_profile.yaml` (+ group extension) with +`add/replace/remove` actions (substring matching, Hermes semantics per gap doc §G1); +NEW tests (store cap/round-trip; prompt-section budget pin; tool e2e). + +**Steps:** (1) store with frozen-snapshot semantics — section text captured at session +start, mid-session writes persist but don't mutate the live prompt (cache discipline, +gap doc §G1); (2) section renders with usage % header exactly like the Hermes format so +the agent knows capacity; (3) **coordination rule:** if EP G1.6 (`session-memory` pack) +has landed, this section must register through the provider path instead of a hardcoded +`PromptParts` entry — check the EP checklist first; the store + tool are identical either +way. **Acceptance:** profile survives restart; over-cap add returns a "consolidate first" +error to the tool, never truncates silently; prompt-sections inspector shows the block +with provenance. + +--- + +## GA-R6 — Hermes/OpenClaw skill import adapters + +**Outcome:** `bobbit` can import a Hermes skill dir (`~/.hermes/skills/...`) or an +OpenClaw workspace skill as a pack, via the marketplace's existing staging seam. + +**Owned files:** the staging-adapter seam in +`src/server/agent/marketplace-install.ts` (the same insertion point the Claude-plugin +adapter uses per [pack-based-marketplace.md](pack-based-marketplace.md) — read that first); +NEW `src/server/agent/skill-import-adapters.ts` (two pure functions: +`adaptHermesSkill(dir) → staged pack tree`, `adaptOpenclawSkill(dir) → staged pack tree`); +fixture skills under `tests/fixtures/`; NEW `tests/skill-import-adapters.test.ts` + one +marketplace e2e install case per format. + +**Steps:** both formats are SKILL.md-shaped (gap doc §G9); map: SKILL.md → pack skill +entity; `references/`/`scripts/`/`templates/` → carried verbatim; name collisions → +standard pack-precedence rules (no special-casing). Reject (with a clear error) skills +whose SKILL.md frontmatter is missing required fields rather than guessing. +**Acceptance:** fixture imports install/uninstall cleanly; imported skill resolvable and +`pack_activation`-toggleable; malformed fixture rejected with actionable message. + +--- + +## GA-R9 — `bobbit doctor` + +**Outcome:** a CLI subcommand that detects-and-explains (default) and repairs (`--fix`) +known state/config drift, OpenClaw-doctor style (gap doc §G8). + +**Owned files:** NEW `src/server/cli/doctor.ts` wired into the existing CLI entry (find +how `run`/`run.cmd` dispatch subcommands first); NEW `tests/doctor.test.ts`. + +**Steps:** v1 checks (each = `{id, severity, explain, canFix}`): (1) orphaned staff +records (reuse the orphan-detector logic — import, don't duplicate); (2) stale worktrees +(worktree-pool list vs git state, reuse `orphan-remote-branch-cleanup` machinery's +read side); (3) pack integrity (`.pack-meta.yaml` vs source manifest); (4) config keys +unknown to the current schema (report-only, never delete); (5) state-dir lock files older +than 24 h. `--fix` applies only `canFix` items, **always backing up the touched file to +`.bobbit/state/doctor-backups/<ts>/` first**; `--dry-run` = default behavior. Output: +human table + `--json`. **Tests:** one seeded-breakage fixture per check, assert +detect → fix → re-run clean → backup exists. **Acceptance:** `bobbit doctor` on a healthy +state dir exits 0 with "no findings"; Caretaker integration is explicitly OUT of scope +(future MC follow-up). diff --git a/docs/design/harness-gap-analysis.md b/docs/design/harness-gap-analysis.md index 07363da54..64b89af1a 100644 --- a/docs/design/harness-gap-analysis.md +++ b/docs/design/harness-gap-analysis.md @@ -1,8 +1,13 @@ # Harness gap analysis — OpenClaw, Hermes, Claude Code vs Bobbit -Status: research + prioritized roadmap. Feature/UX gaps only — **cost/token efficiency is -explicitly out of scope** here because [token-cost-efficiency.md](token-cost-efficiency.md) -§4 already covers the Claude Code / Hermes comparison on that axis. +Status: research complete; roadmap not started. Feature/UX gaps only — **cost/token +efficiency is explicitly out of scope** here because +[token-cost-efficiency.md](token-cost-efficiency.md) §4 already covers the Claude Code / +Hermes comparison on that axis. + +> **Execution authority** for the §5 quick wins (R2–R6, R9): +> [gap-easy-wins-implementation-plan.md](gap-easy-wins-implementation-plan.md). +> Sequencing: [fable-program-execution-plan.md](fable-program-execution-plan.md). **The question this answers:** what do the strongest peer harnesses have that Bobbit lacks, which of those Bobbit actually wants given its direction (the best harness out there — diff --git a/docs/design/mission-control-implementation-plan.md b/docs/design/mission-control-implementation-plan.md new file mode 100644 index 000000000..a0e78118e --- /dev/null +++ b/docs/design/mission-control-implementation-plan.md @@ -0,0 +1,268 @@ +# Mission Control — Implementation Plan (hand-off) + +Status: ready for execution, not started. Workstream **MC** in +[fable-program-execution-plan.md](fable-program-execution-plan.md). + +Companion to [mission-control.md](mission-control.md) (the WHAT/WHY — read its §1–§8 and +**Appendix A contracts** before implementing anything). This document is the HOW: goals +written so an implementer agent that has not seen any other goal can execute its goal from +this text plus the cited contracts alone. + +> **Anchor baseline:** fable-docs @ 2026-06-11 (master parent `6ec8c8f9`). Line numbers WILL +> drift — locate by **symbol name**; if a named symbol does not exist where described, STOP +> and re-derive from the cited pattern file; do not improvise a parallel mechanism. +> +> **Precision policy:** MC-P0…P3 are specified to file/function level. MC-P4/P5 are +> contract level (their substrate is created by P0–P3); re-verify every anchor first. +> +> **Universal rules:** [extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) +> (definition of done, patterns library) + [fable-program-execution-plan.md §1](fable-program-execution-plan.md) +> (doc-is-the-spec deviation protocol, shared-seam serialization) bind every goal below. + +--- + +## Goal map + +``` +MC-P0 global scope ──→ MC-P1 sessions+sidebar ──┬─→ MC-P2a registry ─→ MC-P2b tool group + └─→ MC-P3 flight recorder +MC-P2b + MC-P3 + GA-R2 ──→ MC-P4a pack ─→ MC-P4b crew ──→ MC-P5 briefing/onboarding/budgets +``` + +--- + +## MC-P0 — Global scope legitimization + +**Outcome:** `projectId: "system"` + `global: true` is a valid, first-class home for staff; +zero orphan-banner noise; nothing user-visible yet. + +**Owned files:** `src/server/agent/project-registry.ts`, `staff-store.ts`, +`staff-manager.ts`, `server.ts` (orphaned-staff route only), `docs/staff-agents.md`; +NEW `tests/global-staff-scope.test.ts`. + +**Steps** + +1. `project-registry.ts`: next to `SYSTEM_PROJECT_ID` (top of file), add + `export function isSystemScope(id: string): boolean` and + `export function missionControlDir(): string` per mission-control.md Appendix A.1 + (`path.join(stateDir, "mission-control")`, `mkdirSync` recursive, lazily). Do NOT touch + `participatesInVisibleOrder()` — the system project stays out of the visible order; + add a one-line pinning test for that instead. +2. `staff-store.ts`: extend `interface PersistedStaff` (symbol at `:60`) with + `global?: boolean`; in the store's load path (inside `class StaffStore`, the same + normalisation pass that runs `normalizeStaffAccessory`), normalise missing/non-boolean + `global` → `false`. Persist round-trip unchanged otherwise. +3. `staff-manager.ts`: in the create/edit validation that currently resolves a real project + (the path documented in `docs/staff-agents.md` §"Project and cwd anchoring"), add the + explicit branch from Appendix A.2: `global === true` ⇒ require + `projectId === SYSTEM_PROJECT_ID`, cwd inside `missionControlDir()` (default to it when + blank), force worktree off (ignore `worktree` flag; never set `branch`/`worktreePath`), + leave `sandboxed` semantics untouched. `global === false` ⇒ existing rules byte-for-byte. +4. Orphan route: in `server.ts`, locate the `GET /api/staff/orphaned` handler; exclude + records with `global === true`. Legacy records (no `projectId`, or system-project + without `global`) remain orphans — **no migration**. +5. Update `docs/staff-agents.md` §"Legacy staff records" with one paragraph distinguishing + intentionally-global staff (and a pointer to mission-control.md). + +**Tests (author first; RED on unmodified tree where marked)** + +- `tests/global-staff-scope.test.ts` (node:test, store + manager level — follow + `tests/staff-orphan-reassign.test.ts` for fixtures): + (a) load normalisation `global` missing → `false`; + (b) create global staff with blank cwd → cwd = missionControlDir() *(RED)*; + (c) global staff with cwd outside missionControlDir → rejected *(RED)*; + (d) global + real projectId → rejected *(RED)*; + (e) worktree flag ignored for global staff *(RED)*; + (f) orphan listing excludes `global: true`, still includes legacy orphan *(RED)*; + (g) pinning: system project never appears in visible order (GREEN today — pins step 1). + +**Acceptance:** all new tests green; full staff suites +(`tests/staff-*.test.ts`, `tests/e2e/staff*.spec.ts`) green unmodified. + +--- + +## MC-P1 — Mission Control sessions + sidebar entry + +**Outcome:** a pinned Mission Control entry at the very top of the sidebar (desktop + +mobile); the user can create/use/archive a global chat session that survives reload and +gateway restart. + +**Owned files:** session-create path (`server.ts` + `src/server/agent/session-setup.ts` — +**confine to the project-resolution branch**; this file is a §1.4 shared seam), +`src/app/sidebar.ts`, `src/app/render.ts` (mobile shell), `src/app/state.ts` (nav id only); +NEW `tests/e2e/ui/mission-control.spec.ts`. + +**Steps** + +1. Server: allow session creation with `projectId: "system"`. Find where session creation + resolves the project (the same resolution the projectless path in + `tests/e2e/sessions-projectless.spec.ts` pins — read that spec first); when + `isSystemScope(projectId)`, set cwd = `missionControlDir()`, skip worktree/git setup + entirely (same code path as non-git projects), and tag the session record so the client + can group it (reuse the existing `projectId` field — no new field). +2. Sidebar: in `renderSidebar()` (`src/app/sidebar.ts`, symbol `renderSidebar`), render a + Mission Control block **before** the project loop: header row (nav id + `mission-control-header`), expandable; lists sessions with + `projectId === SYSTEM_PROJECT_ID` and staff with `global === true` (staff sub-grouped + System/Global by a `roleId`-independent flag — for now: pack-roster names, see MC-P4; + until then one flat Staff group). Add "+ New chat" button posting a system-scope + session. Reuse `renderStaffSidebarSection(filteredList, projectId)` with the system + scope rather than forking it. +3. Mobile: mirror in `render.ts::renderSidebarShellMobile` (same data, existing mobile + row components). +4. Exclusions: Mission Control does not participate in project reorder + (`renderProjectReorderHandle` must not render for it) and is not draggable; keyboard + nav follows `docs/sidebar-keyboard-navigation.md` (its nav id participates in the + ordered nav list). + +**Tests (author first)** + +- `tests/e2e/ui/mission-control.spec.ts` (browser E2E, pattern + `tests/e2e/ui/settings.spec.ts`): entry visible at top (strictly above first project); + create chat → session opens; reload → entry + session persist; archive session → + disappears; keyboard navigation reaches the header. *(all RED)* +- Extend `tests/e2e/sessions-projectless.spec.ts` with one system-scope creation case + asserting cwd = missionControlDir and no worktree. *(RED)* + +**Acceptance:** E2E green incl. restart-persistence (sessions.json round-trip — assert via +reload case); existing sidebar suites (`tests/e2e/ui/side-panel-tabs.spec.ts`, +project-reorder specs) green unmodified. + +--- + +## MC-P2a — `bobbit` meta-tool registry (server only) + +**Outcome:** the operation registry exists, is pinned by tests, and dispatches in-process — +no tool exposed to any model yet. + +**Owned files:** NEW `src/server/agent/bobbit-meta-tool.ts`; NEW +`tests/bobbit-meta-tool.test.ts`. + +**Steps** + +1. Implement `BobbitOp`/`OpTier` and the **complete catalog v1 table** from + mission-control.md Appendix A.3 — every row, no additions, no omissions. For each row, + resolve the cited REST route to its `server.ts` handler **by reading + `docs/rest-api.md` + the handler source now**; record the resolved path template in the + registry entry. +2. `export async function dispatchBobbitOp(op, args, ctx)`: validates `args` against the + row's `argsSchema`, then invokes the same internal handler function REST uses (extract a + shared function where the route body is inline — smallest possible extraction, no + behavior change). `sensitive` rows: return a structured `requiresConfirmation` marker — + the ask-flow wiring is MC-P2b's job. `mutate`/`sensitive`: call `recordActivity()` — + until MC-P3 lands, a no-op stub exported from the same module (one TODO referencing + MC-P3). +3. `export function describeBobbitOp(op?)` returning `{op, tier, doc, argsSchema}` rows. + +**Tests (author first)** + +- `tests/bobbit-meta-tool.test.ts`: (a) **catalog pinning** — every registry row resolves + to a callable handler (no 404-shaped dispatch), and the exported op enum equals the + registry keys exactly *(RED)*; (b) schema validation rejects malformed args per op + *(RED)*; (c) `sensitive` ops return the confirmation marker, never execute directly + *(RED)*; (d) read ops do not invoke `recordActivity`, mutate ops do (spy on the stub) + *(RED)*. + +**Acceptance:** new tests green; `npm run check` clean; zero diff outside owned files +except minimal handler extractions. + +## MC-P2b — `bobbit` tool group + tiers + policy + +**Outcome:** Mission Control sessions/staff can call `bobbit`/`bobbit_describe` +end-to-end with ask-gating on sensitive ops. + +**Owned files:** NEW `defaults/tools/bobbit/{bobbit.yaml,bobbit_describe.yaml,extension.ts}`; +tool-policy default for system scope (locate where role/tool policies resolve per scope — +`docs/staff-agents.md` §roleId notes `session-setup.ts` applies them); extend +`tests/tool-description-budget.test.ts`; NEW `tests/e2e/bobbit-meta-tool.spec.ts`. + +**Steps** + +1. Copy `defaults/tools/team/` anatomy exactly (YAML fields: `name`, `description`, + `summary`, `params`, `provider: {type: bobbit-extension, extension: extension.ts}`, + `group: Bobbit`, `docs`, `detail_docs`). `extension.ts` bridges to the gateway via the + `tool-guard-extension.ts` HTTP long-poll pattern (§0.1 patterns library) and calls + `dispatchBobbitOp`. +2. Sensitive ops: wire the `requiresConfirmation` marker into the existing blocking-ask + flow (`docs/blocking-tools.md` / `docs/non-blocking-ask.md` — find the tool that + currently blocks on user confirmation and copy its server round-trip). +3. Default availability: system-scope sessions/staff get the group enabled; project + sessions get it only via explicit role tool-policy opt-in. Pin both directions. + +**Tests:** budget rows for both YAML descriptions; API E2E (`tests/e2e/`, in-process +gateway): a system-scope session calls `projects.list`, creates a temp project +(`projects.add` behind a scripted confirm), creates a nested goal in it, sends a message to +a second session, enqueues a staff inbox entry; asserts a project session WITHOUT opt-in +gets a tool-not-available error. *(all RED)* + +**Acceptance:** E2E green; `tool-description-budget` green; tool-guard/policy suites green +unmodified. + +--- + +## MC-P3 — Flight recorder + +**Outcome:** `recordActivity()` is real; `GET /api/activity` + WS live tail + Activity +panel under the Mission Control sidebar entry. + +**Owned files:** NEW `src/server/agent/activity-store.ts`; `server.ts` (one GET route + WS +subscribe handling in `src/server/ws/`); NEW `src/app/activity-panel.ts`; `sidebar.ts` +(Activity nav row); NEW `tests/activity-store.test.ts`, `tests/e2e/ui/activity-panel.spec.ts`. + +**Steps** + +1. Implement `ActivityEntry` + `recordActivity()` exactly per mission-control.md Appendix + A.4 (JSONL append, crash-safe pattern from + [session-store-crash-safety.md](session-store-crash-safety.md); rotate at 8 MB to + `activity-YYYY-MM-DD.jsonl`). Replace the MC-P2a stub (delete the TODO). +2. REST: `GET /api/activity` with `since/actor/tier/projectId/limit` filters (default 100, + max 500, newest-first). +3. WS: `subscribe_activity` opt-in frame; fan out `activity_entry` ONLY to subscribed + sockets (goal-fanout precedent — + [reduce-server-cpu-experiment-goal-fanout.md](reduce-server-cpu-experiment-goal-fanout.md)). +4. UI: Activity nav row in the Mission Control block → panel listing entries (actor chip, + action, target link, evidence link, revert button only when `revert` present — + revert wiring itself lands with AI-P5; render disabled with tooltip until then), filter + controls, live-tail via the subscription. + +**Tests:** unit — rotation boundary, filter math, crash-safe append (kill mid-write +fixture); browser E2E — meta-tool mutate call appears in panel without reload (live tail), +filters narrow, read ops absent. *(RED)* + +**Acceptance:** every `mutate`/`sensitive` call from MC-P2b's E2E visible with correct +actor attribution; WS fanout test proves non-subscribed sockets receive nothing. + +--- + +## MC-P4a/P4b — mission-control pack + crew *(contract level — re-verify anchors)* + +Contracts: mission-control.md §5 (roster + trigger table + disabled-by-default rule), +Appendix A.5 (pack layout), A.6 (owned files). Depends on GA-R2's push triggers (shared +seam — if GA-R2 hasn't landed, land the triggers there first, never here). + +- **P4a:** pack skeleton copying `market-packs/pr-walkthrough/` (build wiring: + `scripts/build-market-packs.mjs` + `copy-builtin-packs.mjs` `FIRST_PARTY_PACKS`); roles + written in the standing-orders format (GA-R3); panel route `#/ext/mission-control`. + Litmus tests per the `market-packs/artifacts` convention. +- **P4b:** idempotent `POST /api/mission-control/crew` instantiating staff records from + `staff-templates/crew.yaml` (idempotency key: staff name + global flag); Observer + triggers enabled, others disabled (§5); panel "Create crew" button. E2E: create crew ⇒ + 4 global staff with correct triggers; re-POST ⇒ no duplicates; Caretaker manual wake + dry-run sweep ⇒ flight-recorder entry; panel deep-link survives reload. + +**Acceptance:** pack installs/uninstalls cleanly; crew E2E green; activity panel shows the +dry-run entry. + +## MC-P5 — Briefing, onboarding, budgets *(contract level; 3 separable PRs)* + +Contracts: mission-control.md §7. (1) Observer briefing = inbox digest assembled from: +flight recorder window, cost routes (`docs/session-cost.md`), notification-policy stuck +predicate, AI calibration endpoint when present (omit section when absent). (2) First-run: +zero registered projects ⇒ client routes to a Mission Control chat seeded with the +onboarding prompt (drives the existing add-project proposal flow — do not build a new +wizard). (3) Budgets: `budgets.{project,staff}.monthlyUsd` config keys; Observer breach +escalation entry + briefing line. Browser E2E per feature (first-run on a fresh state dir; +briefing digest renders; budget breach escalates). + +**Acceptance:** mission-control.md §1's small-business-owner sentence works on a clean +install: project created + Friday `schedule` trigger attached via chat alone. diff --git a/docs/design/mission-control.md b/docs/design/mission-control.md index 9b5c7724b..1a98c5cd6 100644 --- a/docs/design/mission-control.md +++ b/docs/design/mission-control.md @@ -1,8 +1,11 @@ # Mission Control — the global scope, global staff, and the flight recorder -Status: proposal / implementation blueprint. Each phase in §8 converts 1:1 into a mission -goal with acceptance criteria; steps name exact files, functions, and reuse points so they -can be executed without re-deriving this analysis. +Status: design accepted, not started. This is the WHAT/WHY + contracts (Appendix A). + +> **Execution authority:** implement from +> [mission-control-implementation-plan.md](mission-control-implementation-plan.md) +> (per-goal owned files, symbol anchors, RED→GREEN tests, acceptance). Sequencing: +> [fable-program-execution-plan.md](fable-program-execution-plan.md). **What this is:** a first-class **global scope** at the top of the sidebar — the main entry point for using Bobbit. Mission Control sessions can set up new projects, talk to any agent From 835f6bd000cc268a779e871869929effc8cbfb39 Mon Sep 17 00:00:00 2001 From: Claude <noreply@anthropic.com> Date: Thu, 11 Jun 2026 08:36:10 +0000 Subject: [PATCH 09/18] Add CI workstream: code intelligence (LSP, ast-grep, repo map, diagnostics) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - code-intelligence.md (new) — layer model (lexical/structural/symbol/ spatial/truth/temporal), design stance from owner interview: native first-party tool groups over MCP surfaces; everything behind swappable extension-platform capability names; per-worktree derived state with a relative-paths pin (the graphify cross-worktree lesson); language servers as autodiscovered per-language packs; graphify as optional visualization pack, in-house budgeted code_map for the prompt path; context-services chip answering "what's running, where" from chat. - code-intelligence-implementation-plan.md (new) — CI-1 ast_search/ ast_rewrite (binaries/ shipping, preview-diff rewrites), CI-2 code_check structured diagnostics, CI-3 LSP supervisor + code_* tools with graceful-degradation invariants, CI-4 lsp-<lang> descriptor packs + autodiscovery UX, CI-5 budgeted repo map, CI-6 services chip + graphify pack, CI-7 capability shim → EP G8 swap + BENCH exit gate. - fable-program-execution-plan.md — CI workstream row, wave placement (W3: CI-1/2 · W4: CI-3/5 · W5: CI-4/6/7), checklist incl. 7 language- pack cards, new shared-seam rows (session-setup tool activation, EP G8 shim). https://claude.ai/code/session_01LBW68Z2KvA3EhQMq3c2wrR --- .../code-intelligence-implementation-plan.md | 165 ++++++++++++++++++ docs/design/code-intelligence.md | 156 +++++++++++++++++ docs/design/fable-program-execution-plan.md | 16 +- 3 files changed, 333 insertions(+), 4 deletions(-) create mode 100644 docs/design/code-intelligence-implementation-plan.md create mode 100644 docs/design/code-intelligence.md diff --git a/docs/design/code-intelligence-implementation-plan.md b/docs/design/code-intelligence-implementation-plan.md new file mode 100644 index 000000000..0de8be1b1 --- /dev/null +++ b/docs/design/code-intelligence-implementation-plan.md @@ -0,0 +1,165 @@ +# Code Intelligence — Implementation Plan (hand-off) + +Status: ready for execution, not started. Workstream **CI** in +[fable-program-execution-plan.md](fable-program-execution-plan.md). + +Companion to [code-intelligence.md](code-intelligence.md) (WHAT/WHY + the layer model and +design stance — its §7 worktree contract and §1 capability names are LAW). + +> **Anchor baseline:** fable-docs @ 2026-06-11 (master parent `6ec8c8f9`). Locate by symbol +> name; missing symbol ⇒ STOP, re-derive from the cited pattern — never improvise. +> +> **Precision policy:** CI-1/CI-2 file/function level; CI-3…CI-7 contract level (re-verify +> anchors; CI-3's supervisor is the largest new component in the program — its goal text +> includes extra invariants). +> +> **Universal rules:** [extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) +> + [fable-program-execution-plan.md §1](fable-program-execution-plan.md). **Seams:** +> tool-activation in `session-setup.ts` (shared — confine to named functions); `binaries/` +> shipping (`binaries.versions.json`, checksums — copy the existing binary's wiring); +> EP G8 capabilities (CI consumes when it lands; until then a thin local registry shim, +> see CI-7). + +--- + +## Goal map + +``` +CI-1 ast-grep tools ──→ CI-5 repo map (reuses CI-1 engine) +CI-2 structured diagnostics (independent) +CI-3 LSP supervisor + code_* tools (TS+Py) ──→ CI-4 language packs + autodiscovery UX +CI-1..CI-5 ──→ CI-6 services chip + graphify viz pack ──→ CI-7 capability swap + BENCH +``` + +## CI-1 — `ast_search` / `ast_rewrite` tool group + +**Outcome:** structural search and preview-diff rewrites across the worktree; regex/sed +edits become obsolete. + +**Owned files:** NEW `defaults/tools/ast/{ast_search.yaml,ast_rewrite.yaml,extension.ts}`; +ast-grep binary wiring in `binaries/` + `binaries.versions.json` + checksums (copy an +existing entry's pattern end-to-end, incl. `scripts/` download/verify step); NEW +`src/server/agent/ast-grep-runner.ts`; NEW `tests/ast-grep-runner.test.ts`, +`tests/e2e/ast-tools.spec.ts`; budget rows in `tests/tool-description-budget.test.ts`. + +**Steps** + +1. Runner: `runAstGrep({pattern, rewrite?, lang?, paths?, cwd})` → execFile the pinned + binary with `--json`; map results to `{file, line, column, text}[]` (relative paths + ONLY — pin per design §7); output cap + spill via `truncateLargeToolContent` + (`src/server/agent/truncate-large-content.ts`). +2. `ast_search`: tool YAML per the `defaults/tools/team/` anatomy; renderer lists matches + as clickable `file:line` rows (copy a search-style renderer from + `src/ui/tools/renderers/`). +3. `ast_rewrite`: runner in diff mode → unified diff returned as the tool result rendered + by the existing diff renderer; `apply: true` arg performs the write through the normal + file-edit path (so git-status/review flows observe it). Default is preview. +4. Prompt guidance: one paragraph in the CE-G7.1 discovery section (if landed; else the + tool's `docs:` field carries it): "prefer ast_search over grep for code shapes; prefer + ast_rewrite over sed always". + +**Tests (author first; RED):** runner unit (json mapping, relative-path pin, cap+spill, +unknown lang error); e2e — search finds a planted pattern; rewrite previews a diff without +mutating; `apply: true` mutates and git-status reflects it; budget rows. + +**Acceptance:** suites green; manual: an agent in a scratch project completes a +rename-shaped change via ast_rewrite preview→apply with zero bash sed. + +## CI-2 — `code_check` structured diagnostics + +**Outcome:** one tool returns typed diagnostics from the project's checkers; gates can +consume the same parsers. + +**Owned files:** NEW `defaults/tools/code/code_check.yaml` (group seeds the `code` group +early; CI-3 adds siblings) + extension wiring; NEW `src/server/agent/diagnostics/` +(`runner.ts` + `parsers/{tsc,eslint,pytest,go}.ts`); NEW unit tests per parser with +captured-output fixtures; e2e. + +**Steps:** checker set resolution (project config override → autodiscovery by manifest: +`tsconfig.json`⇒tsc+eslint, `pyproject.toml`⇒pytest/ruff if present, `go.mod`⇒go vet); +each parser: raw output → `{tool, file, line, code?, severity, message}[]`; result = +summary counts + first N (default 50) diagnostics + spill ref; unknown checker ⇒ capped +raw fallback, never an error. Renderer: grouped-by-file list. Verification-harness +consumption is a recorded follow-up (do NOT touch `verification-harness.ts` here — CE-G3.3 +/ CE-G5.2 own it per §1.4). + +**Tests:** parser fixtures (each: real captured output → exact typed rows); resolution +matrix unit test; e2e on a fixture project with a planted type error → structured row. + +**Acceptance:** a failing `tsc` run produces ≤2 KB of tool result instead of the raw log; +ledger note recorded in the doc. + +## CI-3 — LSP supervisor + `code_*` symbol tools *(contract level; the big one)* + +Contracts: design §2. **Owned (new):** `src/server/agent/lsp/` +(`lsp-supervisor.ts`, `lsp-client.ts` JSON-RPC, `language-registry.ts` detection rules, +descriptor schema); `defaults/tools/code/{code_definition,code_references,code_symbols, +code_hover,code_rename,code_diagnostics}.yaml`; descriptor data for TypeScript + Python. + +Invariants (each gets a pinning test): + +1. Supervisor = idempotent ensure with in-flight dedupe (`sandbox-manager.ts` shape); + key `(worktreeRoot, languageId)`; idle-shutdown default 10 min; max concurrent servers + (default 4, config); crash ⇒ restart with backoff, surfaced non-fatally (tool returns + "language server unavailable — falling back" guidance, never hangs the turn). +2. Tools degrade gracefully: no descriptor for the file's language ⇒ structured "not + covered; detected language X; install lsp-X pack" result (feeds CI-4's UX), never an + exception. +3. `code_rename` is preview-diff → apply through the file-edit path (CI-1 step 3 pattern). +4. All positions/paths relative; budget + spill on every result. +5. Lifecycle: worktree removal (goal cleanup) disposes servers + caches — hook the + existing worktree-cleanup path; Caretaker sweep (MC) catches leaks. + +Tests: supervisor unit with a fake LS binary (spawn/dedupe/idle/crash); e2e against +typescript-language-server on a fixture project: definition/references/rename round-trip; +uncovered-language fallback; worktree-dispose kills the server (process poll). + +**Acceptance:** on the Bobbit repo itself, `code_references` on a core symbol returns in +<2 s warm, and the supervisor shows 0 servers after idle timeout. + +## CI-4 — Language packs + autodiscovery UX *(contract level)* + +Descriptor-as-pack: `market-packs/lsp-<lang>/` containing one `descriptor.yaml` +(server binary acquisition: npm/pip/download+checksum; args; init options; health probe; +file-detection globs) — data only, no code. Bundle `lsp-typescript`, `lsp-python` as +built-ins (band per `builtin-packs.ts`); **cards** (checklist rows, not yet specced): +`lsp-go`, `lsp-rust`, `lsp-csharp` (Roslyn), `lsp-fsharp` (FsAutoComplete), `lsp-jvm` +(JDT-LS), `lsp-clangd`, `lsp-kotlin`. Autodiscovery surfaces uncovered detected languages +through the CI-6 chip + a marketplace deep-link. Acceptance: installing `lsp-go` on a Go +fixture makes `code_definition` work with zero config; uninstalling degrades gracefully. + +## CI-5 — `code_map` budgeted repo map *(contract level; after CI-1)* + +`code_map(path?, depth?)`: directory → file → public symbols/signatures from ast-grep +parse output; content-hash cache per worktree under the design-§7 cache dir; hard result +budget (default 2 KB, arg-raisable to 8 KB); relative paths pin. When EP G1.3 (prompt +sections via providers) is merged, add the provider that contributes a "Repo Map" section +under the same budget — the tool stays for on-demand deeper maps. Acceptance: on the +Bobbit repo, default call ≤2 KB and a cold→warm rebuild is ≥10× faster (cache test). + +## CI-6 — Context-services chip + graphify visualization pack *(contract level)* + +- **Chip** (core UI): chat-header widget next to `<git-status-widget>` showing session + scope (global/project/worktree) + active code-intel services from a new generic + `GET /api/sessions/:id/services` (supervisor + pack-runtime sourced — generic shape so + EP runtimes plug in later). Popover: status/restart/open-panel/install-suggestion. + Browser E2E: chip renders per scope; restart works; suggestion deep-links marketplace. +- **Graphify pack** (marketplace, NOT built-in): wraps graphify per worktree into + `graphify-out/` (gitignored), panel at `#/ext/graphify` rendering its `graph.html`; + install-time doc notes the absolute-path caveat and pins output as derived cache + (never committed; design §4). Pack-litmus tests only — graphify itself is not under test. + +## CI-7 — Capability seam + BENCH validation *(contract level)* + +Until EP G8 lands: a 30-line local registry +(`src/server/agent/code-intel-capabilities.ts`) mapping the four capability names → +implementing module; tools resolve through it (pin: no direct cross-module imports of +engines from tool extensions). When EP G8 merges, swap the shim for pack capabilities +(one PR, the shim's API is G8's `ctx.capabilities.call` shape). Then: run the CE BENCH +suite (CE-G0.3) with CI tools on vs off; record token/turn deltas in +code-intelligence.md §8; this is the workstream's exit gate. + +--- + +Checklist rows (mirrored in the execution plan §4): CI-1 · CI-2 · CI-3 · CI-4 (+7 language +cards) · CI-5 · CI-6 · CI-7. diff --git a/docs/design/code-intelligence.md b/docs/design/code-intelligence.md new file mode 100644 index 000000000..014629d3b --- /dev/null +++ b/docs/design/code-intelligence.md @@ -0,0 +1,156 @@ +# Code Intelligence — giving agents IDE superpowers (find, edit, verify) + +Status: design accepted, not started. Workstream **CI** in +[fable-program-execution-plan.md](fable-program-execution-plan.md). + +> **Execution authority:** implement from +> [code-intelligence-implementation-plan.md](code-intelligence-implementation-plan.md). + +**The problem:** Bobbit agents are "an LLM with grep". 56% of tool calls are bash +(token-cost-efficiency.md §3), discovery is multi-round grep cascades, edits are text +patches whose correctness is only discovered at gate time, and nothing understands symbols, +types, or structure. Peers are converging on the same answer (Serena, Octocode, Aider's +repo-map, graphify): pair fast lexical search with **structural (AST), semantic (LSP), +spatial (repo map), and truth (diagnostics) layers**. + +**Design stance (from owner interview, 2026-06-11):** + +1. **Native first-party tool groups, not MCP-first.** Observed reality: agents underuse + generic MCP tools. First-party groups get budgeted outputs, custom renderers, tool-guard + policies, and prompt guidance — that's why `team_*` gets used and random MCP servers + don't. External engines may sit *behind* a tool group; they are not the model-facing + surface. +2. **Best solution first, swappable forever.** Every layer is addressed through an + extension-platform **capability name** (extension-platform.md §3.1): + `code.structural-search`, `code.symbol-nav`, `code.map`, `code.diagnostics`. Reference + implementations ship as built-in packs that *provide* those capabilities; tools and + prompt sections *consume* capabilities, never concrete packs. Swapping engines later = + installing a different provider pack — zero tool-surface change. +3. **Per-worktree everything.** Indexes, caches, and language-server instances are keyed by + worktree root, stored as derived gitignored state, rebuilt per checkout. Nothing derived + is ever committed; nothing stores absolute paths (the exact failure class graphify hit + across worktrees). +4. **Cost is a feature.** Every tool here exists to replace a more expensive pattern + (grep cascade → one `code_references`; 50 KB test log → 12 structured diagnostics; + read-five-files orientation → one budgeted map section). Each lands with a CE-ledger + measurement expectation (CE-G7.2's rationale, now executed). +5. **Verifiability is a feature.** Structural rewrites preview as diffs through the + existing review flow before applying; every edit path ends in `code_check`; nothing + mutates silently. + +--- + +## §1 The layer model (what exists, what's new) + +| Layer | Question it answers | Bobbit today | This workstream adds | Capability | +|---|---|---|---|---| +| Lexical | "where is this string?" | bash grep/rg (uncapped, 56% of calls) | nothing new — CE-G2 caps it, CE-G7.1 steers it | — | +| **Structural** | "find/replace this syntax shape" | ❌ regex/sed via bash | `ast_search` / `ast_rewrite` tools (ast-grep engine) | `code.structural-search` | +| **Symbol/semantic** | "definition? references? rename? type?" | ❌ | `code_*` LSP tool group + gateway LS supervisor + per-language packs | `code.symbol-nav` | +| **Spatial** | "what's here and how does it connect?" | ❌ (session search ≠ code) | budgeted repo-map (tool now, prompt section when EP providers land); graphify pack for *visualization* | `code.map` | +| **Truth** | "did my change actually work?" | gates run commands, agents read raw logs | `code_check` structured diagnostics (typed results, gate-consumable) | `code.diagnostics` | +| Temporal | "what did we learn before?" | staff memory; FlexSearch sessions | **already planned elsewhere**: Hindsight pack = EP G2; session-memory = EP G1.6; profile = GA-R5 | `memory` (EP) | + +Graphify-vs-Hindsight, settled: **different layers, complementary, both already placed.** +Graphify-shaped tools are spatial (structure of *this checkout, now*); Hindsight is temporal +(what the agent/team *learned over time*) and is literally EP G2. CI adds the spatial layer; +it does not duplicate the temporal one. + +## §2 Symbol layer — native LSP tool group + +- **Surface:** `defaults/tools/code/` — `code_definition`, `code_references`, + `code_symbols` (document/workspace), `code_hover`, `code_rename` (preview-diff → + review-flow apply), `code_diagnostics`. Budgeted outputs (cap + spill per CE-G2), + renderers that show locations as clickable `file:line`. +- **Engine:** a gateway-side **LSP supervisor** (`lsp-supervisor.ts`): spawns language + servers per `(worktree, language)`, idle-shutdown (default 10 min), capped concurrent + servers, health-restart — the `sandbox-manager.ts` idempotent-ensure shape. Servers are + *processes the gateway owns*; agents never talk LSP directly. +- **Language autodiscovery:** a detection registry maps manifest/file signals → + language-server descriptor (`tsconfig.json`/`package.json` → typescript-language-server; + `pyproject.toml` → pyright; `go.mod` → gopls; `Cargo.toml` → rust-analyzer; `.csproj`/ + `.fsproj` → Roslyn/FsAutoComplete; `pom.xml`/`build.gradle*` → JDT-LS; `CMakeLists.txt`/ + compile-commands → clangd). Descriptors ship as **per-language packs** + (`lsp-typescript`, `lsp-python`, …): the descriptor is data (binary acquisition, args, + init options, health probe), so adding a fleet language = a pack, not core code. v1 + bundles TypeScript + Python; the rest are cards (CI-4) — autodiscovery *detects* an + uncovered language and surfaces "install the lsp-go pack" in the UI instead of failing. +- **Why not Serena/MCP as the surface:** same capability, but token-opaque outputs, no + renderer, no tool-guard tiers, Python runtime baggage per worktree, and the observed + MCP-underuse problem. Serena remains the design reference and a fallback pack if the + supervisor proves harder than planned (the capability seam makes that swap cheap). + +## §3 Structural layer — ast-grep tools + +`ast_search(pattern, lang?, paths?)` and `ast_rewrite(pattern, rewrite, …)` over the +**ast-grep** single static binary (shipped via the existing `binaries/` mechanism — +`binaries.versions.json` + checksums; per-call execution, nothing to supervise, ~25 +languages). `ast_rewrite` never writes directly: it returns a unified diff rendered through +the existing diff renderer; applying goes through the normal edit path so review-pane and +git-status flows see it. This is the single biggest verifiability upgrade over regex/sed — +and the first goal to land (CI-1). + +## §4 Spatial layer — repo map + visualization + +Two deliberately separate concerns: + +- **Prompt/orientation path (in-house, small):** `code_map` — an Aider-style budgeted map: + per-directory → file → public symbols/signatures, built from ast-grep's parse output + (no new parser dependency), content-hash cached per worktree, **hard token budget** + (default 2 KB tool result; later an EP provider contributes a prompt section under the + same budget machinery as the skills catalog). Owner asked "isn't in-house overkill?" — + no, because this reuses the ast-grep engine from §3 and the existing budget/prompt-section + patterns; the *graph database* part of graphify is precisely what we don't need for + prompts. Deterministic, relative-path-only, worktree-safe by construction. +- **Visualization path (reuse, optional):** a `graphify` **marketplace pack** (not built-in) + wrapping graphify per worktree: output to a gitignored `graphify-out/`, rendered in a pack + panel (`#/ext/graphify`). Known risk to verify at install: absolute-path leakage in + `graph.json` across worktrees (reported upstream; verify against current release — + treat the graph as per-worktree derived cache, never committed, which sidesteps the + team-workflow/merge-driver features entirely). + +## §5 Truth layer — structured diagnostics + +`code_check(scope)` runs the project's known checkers (from project config; autodiscovered +defaults: `tsc --noEmit`, eslint, pytest, `go vet`/`go test`, `cargo check`) and returns +**typed results**: `{tool, file, line, code, severity, message}[]` + summary counts — +never raw logs (cap + spill for the rare overflow). Verification gates can consume the same +parser output, and the team-lead's "did it work" loop becomes data-driven. Parsers are +per-tool modules; unknown checkers fall back to capped raw output (never block). + +## §6 UX — "what's running, where?" (owner requirement) + +Code-intel services are invisible infrastructure; the user must be able to see and steer +them from where they live — the chat: + +- **Context services chip** in the chat header (next to the git-status widget): shows the + active session's scope (global / project / worktree) and the live code-intel services for + it (e.g. "TS ✓ · Py ✓ · map 4.2k"). Click → popover: per-service status, restart, open + pack panel (graphify viz, map inspector), install-suggestion for detected-but-uncovered + languages. +- Pack panels remain the deep-dive surface (`#/ext/…`); the chip is the discoverability + bridge the platform currently lacks — designed here, generalized for all packs with + runtimes/services as a follow-up card under EP (the chip reads a generic + "active pack services for this session" endpoint, not CI-specific state). + +## §7 Worktree semantics (the contract) + +Everything in CI obeys: keyed by worktree root · derived state in +`<worktree>/.bobbit-cache/` or the server cache dir (both gitignored) · relative paths only +in any artifact · LS instances per worktree with pooling/idle-shutdown · goal worktree +cleanup also disposes CI state (Caretaker sweep covers leaks). A pinning test greps CI +artifacts for absolute paths — the graphify lesson, enforced. + +## §8 Cost & verifiability acceptance (program-level) + +- CE ledger (CE-G0.1) dimensions for CI tools; success = bash share of discovery calls + falls materially (CE-G7.1 targets <40%; CI should push further), grep-cascade turns + replaced by single-call lookups on the BENCH suite (CE-G0.3) at equal task success. +- Every mutating surface (`ast_rewrite`, `code_rename`) previews before applying; every + CI tool result is budgeted; `code_check` closes every edit loop. + +Cross-references: [token-cost-efficiency.md](token-cost-efficiency.md) (CE-G2/G7), +[extension-platform.md](extension-platform.md) (capabilities §3.1, providers, runtimes), +[harness-gap-analysis.md](harness-gap-analysis.md) (peer evidence), +[mission-control.md](mission-control.md) (Caretaker sweep, flight recorder for service +lifecycle events). diff --git a/docs/design/fable-program-execution-plan.md b/docs/design/fable-program-execution-plan.md index 1c4e129ca..4af404974 100644 --- a/docs/design/fable-program-execution-plan.md +++ b/docs/design/fable-program-execution-plan.md @@ -17,6 +17,7 @@ The workstreams (each doc owns its *content* — tasks, contracts, acceptance; t | **MC** Mission Control | [mission-control.md](mission-control.md) | [mission-control-implementation-plan.md](mission-control-implementation-plan.md) | | **AI** autoimprovement | [autoimprovement.md](autoimprovement.md) | [autoimprovement-implementation-plan.md](autoimprovement-implementation-plan.md) | | **GA** gap-analysis easy wins | [harness-gap-analysis.md](harness-gap-analysis.md) | [gap-easy-wins-implementation-plan.md](gap-easy-wins-implementation-plan.md) | +| **CI** code intelligence | [code-intelligence.md](code-intelligence.md) | [code-intelligence-implementation-plan.md](code-intelligence-implementation-plan.md) | **An implementer agent receives exactly one goal ID and reads: the execution-authority doc's goal section → the contracts it cites → §1 below. Nothing else is required.** @@ -52,7 +53,9 @@ goal section → the contracts it cites → §1 below. Nothing else is required. | `goal-trigger-dispatcher.ts` push triggers (`gate_failed`/`session_errored`) | GA-R2 · MC-P4 · AI-P1 | land once in GA-R2 | | `activity-store.ts` / `recordActivity()` | MC-P3 · MC-P2 · AI-P5 | land in MC-P3; earlier callers stub | | `auxiliary.*` model-slot config | AI-P2 · GA-R4 · CE-G6.1 | one shape ([per-role-model-overrides.md](per-role-model-overrides.md)); first lander defines it | - | `verification-harness.ts` | CE-G3.3 · CE-G5.2 | sequence per CE doc | + | `verification-harness.ts` | CE-G3.3 · CE-G5.2 (CI-2's gate-consumption is a recorded follow-up, NOT in CI-2) | sequence per CE doc | + | tool-activation in `session-setup.ts` | CI-1/2/3 · EP G1.4 · CS | confine to named functions; rebase order per wave | + | EP G8 capability registry | CI-7 consumes | CI ships a local shim until G8 merges; swap PR after | 5. **Tracking discipline.** Tick the §4 row **in the merging PR**; update the owning doc's `Status:` line when a workstream's parent goal completes. The orchestrating session treats merged-but-unticked as a bug. @@ -110,14 +113,17 @@ flowchart TD CS: R3 → R7 → R4 → R8‖R9 · SN: T5 (the big one), T8, T9 PB: P2a (after SN-T2/T3) → P2b → P2d CE: post-G1.0 lanes (G2, G7; BENCH-gated G4/G5 wait for G0.3) - EP: G1.4 → G1.5 → G1.6 · MC: P2a → P2b, P3 · AI: P1"] + EP: G1.4 → G1.5 → G1.6 · MC: P2a → P2b, P3 · AI: P1 + CI: CI-1 (ast tools) · CI-2 (diagnostics)"] W4["WAVE 4 — features on the spines SN: T6, T7 · PB: P2c (after SN-T5) → P3 CS: waves 2–4 (D*, P*, T*) · EP: G2 → G3 - MC: P4a → P4b · AI: P2a → P2b · GA: R4, R5 (R5 prefers EP G1.6)"] + MC: P4a → P4b · AI: P2a → P2b · GA: R4, R5 (R5 prefers EP G1.6) + CI: CI-3 (LSP supervisor) → CI-5 (repo map)"] W5["WAVE 5 — the loop closes AI: P3a → P3b → P4 → P5 → P6 · MC: P5 - EP: G4…G9 · CE: remaining BENCH-gated goals"] + EP: G4…G9 · CE: remaining BENCH-gated goals + CI: CI-4 (language packs) · CI-6 (services chip + graphify viz) · CI-7 (capability swap + BENCH)"] W0 --> W1 --> W2 --> W3 --> W4 --> W5 ``` @@ -198,6 +204,8 @@ T8 (S, independent) → T9 (S, independent). T2/T3/T5/T7 gate PB-P2 (§1.4). - **GA**: [ ] R2 · [ ] R3 · [ ] R4 · [ ] R5 · [ ] R6 · [ ] R9 - **MC**: [ ] P0 · [ ] P1 · [ ] P2a · [ ] P2b · [ ] P3 · [ ] P4a · [ ] P4b · [ ] P5 - **AI**: [ ] P1 · [ ] P2a · [ ] P2b · [ ] P3a · [ ] P3b · [ ] P4 · [ ] P5 · [ ] P6 +- **CI**: [ ] CI-1 · [ ] CI-2 · [ ] CI-3 · [ ] CI-4 · [ ] CI-5 · [ ] CI-6 · [ ] CI-7 · + language-pack cards: [ ] lsp-go · [ ] lsp-rust · [ ] lsp-csharp · [ ] lsp-fsharp · [ ] lsp-jvm · [ ] lsp-clangd · [ ] lsp-kotlin ## §5 Prerequisite fix (before any lane) From be913e3c87c3a024a762c164f37bb627d6deeb77 Mon Sep 17 00:00:00 2001 From: Claude <noreply@anthropic.com> Date: Thu, 11 Jun 2026 09:54:02 +0000 Subject: [PATCH 10/18] CI: add research annex (alternatives + evidence), upgrade picks from research MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New code-intelligence-alternatives.md — cited research record (June 2026): market validation (Claude Code native LSP v2.0.74, Copilot CLI LSP, OpenCode; Codex/Gemini still without), LSP engine bake-off (vscode-jsonrpc/-protocol vs Serena/solidlsp vs multilspy vs MCP bridges), per-language server matrix with operational caveats from issue trackers, structural-engine bake-off (ast-grep vs Semgrep/Opengrep, comby, GritQL, srgn, raw tree-sitter), repo-map evidence (Aider algorithm, RepoGraph ICLR'25, stack-graphs archived, embeddings contested), diagnostics formats (per-checker JSON matrix, SARIF-as-input, rdjson, errorformat), and a memory-layer check that validates the EP G2 Hindsight bet with a file-first framing. Applied to the CI docs: - CI-3 engine named (vscode-jsonrpc + vscode-languageserver-protocol; coc.nvim reference; solidlsp cache + multilspy auto-download as ported designs) and invariants expanded to 9 with pinning tests: handshake order, processId + reap, position-encoding negotiation, didOpen/didChange tracker, watched-file forwarding, readiness gating per server, post-edit diagnostics push, gitignore-filtered references, per-worktree CARGO_TARGET_DIR. - Server picks corrected: vtsls (not typescript-language-server), basedpyright (not pyright), csharp-ls before Roslyn LS, kotlin-lsp deferred (alpha). - CI-1 ships @ast-grep/cli as npm dependency (--json=stream) instead of the custom binaries/ route. - CI-2 schema = flattened rdjsonl; per-tool parser matrix; SARIF adapter; pytest via junitxml; bundled errorformat fallback binary. - CI-5 upgraded from flat symbol map to ranked def/ref-graph map with personalized PageRank seeded by session state (Aider mechanism; RepoGraph evidence), tags.scm-vs-napi spike called out. https://claude.ai/code/session_01LBW68Z2KvA3EhQMq3c2wrR --- docs/design/code-intelligence-alternatives.md | 174 ++++++++++++++++++ .../code-intelligence-implementation-plan.md | 128 +++++++++---- docs/design/code-intelligence.md | 79 +++++--- 3 files changed, 316 insertions(+), 65 deletions(-) create mode 100644 docs/design/code-intelligence-alternatives.md diff --git a/docs/design/code-intelligence-alternatives.md b/docs/design/code-intelligence-alternatives.md new file mode 100644 index 000000000..fe8a2c333 --- /dev/null +++ b/docs/design/code-intelligence-alternatives.md @@ -0,0 +1,174 @@ +# Code Intelligence — alternatives considered & evidence (research annex) + +Status: research record, 2026-06-11. Companion to [code-intelligence.md](code-intelligence.md) +(design) and [code-intelligence-implementation-plan.md](code-intelligence-implementation-plan.md) +(execution). This doc exists so future maintainers know **what else was on the table, why the +proposed option won, and which facts that judgment rests on** — re-verify the dated facts +before overturning a decision. + +--- + +## §1 The market moved our way (validation of the core stance) + +The design's bet — *first-party LSP-backed tools in the harness, not MCP add-ons* — is now +the shipped pattern across terminal agents: + +- **Claude Code** shipped a native LSP tool in v2.0.74 (2025-12-19): go-to-definition, + references, hover, workspace symbols, call hierarchies, **automatic post-edit diagnostics**; + servers configured via plugins (11 official language plugins), binaries installed separately + ([changelog](https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md), + [tools reference](https://code.claude.com/docs/en/tools-reference)). +- **GitHub Copilot CLI** shipped LSP support (8 operations incl. rename + call hierarchy) with + repo/plugin/user JSON config and an "LSP Setup skill" for ~14 languages + ([blog, 2026-06-10](https://github.blog/ai-and-ml/github-copilot/give-github-copilot-cli-real-code-intelligence-with-language-servers/)). +- **OpenCode** preconfigures LSP for 18+ languages. **Codex CLI and Gemini CLI** still lack it + — their top-voted open feature requests ([codex#8745](https://github.com/openai/codex/issues/8745)). +- IDE agents (Cursor, Windsurf, Zed) expose at most a *diagnostics* tool to the model and lean + on retrieval indexes — Cursor 2.0 even "simplified the agent to use the LSP less." + +Implications adopted: (a) config-not-binaries plugin model = exactly our `lsp-<lang>` +descriptor packs; (b) Claude Code's early bugs (zombie servers, stale pre-edit diagnostics, +gitignored files in references, missing `didOpen`, command injection in binary detection) are +a free checklist — each becomes a CI-3 invariant/test; (c) **post-edit auto-diagnostics** is +table stakes — added to CI-3 scope; (d) differentiation is real: nobody ships a budgeted +ranked repo map, typed multi-checker `code_check`, worktree-keyed supervision, or +capability-swappable engines. + +## §2 LSP engine: build on `vscode-jsonrpc` — vs Serena, bridges, Python libs + +| Option | Verdict | Decisive facts | +|---|---|---| +| **vscode-jsonrpc 9 + vscode-languageserver-protocol 3.18** (npm) — **CHOSEN** | Build the supervisor on these | Microsoft-maintained, republished 2026-06-03, ~10M+ downloads/wk each; protocol README: "tool independent… usable in any node application". coc.nvim (~25k★) is the proven reference for the lifecycle layer. Protocol plumbing ≈ 500–1000 lines / 1–2 wks; the hard part (lifecycle) we own under any option | +| `vscode-languageclient` | Rejected | `engines.vscode` — runs only inside the VS Code extension host (verified package.json) | +| **Serena** (oraios/serena, MIT, ~25k★) | Design reference + fallback pack, not the engine | Python 3.13 + uv runtime per workspace; MCP indirection; live reliability battle in its tracker: 30 GB RAM ([#944](https://github.com/oraios/serena/issues/944)), init hangs ([#1390](https://github.com/oraios/serena/issues/1390), [#937](https://github.com/oraios/serena/issues/937)), stdio init races ([#900](https://github.com/oraios/serena/issues/900)), LS termination loops ([#634](https://github.com/oraios/serena/issues/634)). **Port its designs**: solidlsp's synchronous wrapper + two-tier symbol cache (100–500 ms → <10 ms), auto-download of server binaries | +| multilspy (Microsoft) | N/A directly | Python-only research library (13 langs); its 2026 fix history (server hangs, orphaned processes) confirms lifecycle is where the bodies are buried | +| MCP bridges (mcp-language-server 1.5k★, lsmcp, cclsp, lsproxy) | Rejected | All single-maintainer; most dormant ≥10 months (mcp-language-server since 2025-06, lsmcp since 2025-08); single-root designs; none do worktree-aware pooling. lsproxy (REST, Rust, multi-server pool) is the closest design reference | + +## §3 Per-language servers (research-corrected picks) + +Two picks **changed** from the original draft: TS/JS `typescript-language-server` → **vtsls**; +Python `pyright` → **basedpyright**. + +| Language | Pick | Why / acquisition | Caveats to engineer for | +|---|---|---|---| +| TS/JS | **vtsls** | Wraps VS Code's TS extension (feature parity); Zed's default ([zed#13140](https://github.com/zed-industries/zed/pull/13140)); npm install | tsserver under the hood: plan multi-GB RSS, set max old-space (Zed uses 8 GiB) | +| Python | **basedpyright** | Pyright fork with Pylance-only features (find-implementations, semantic tokens) reimplemented in the open server ([docs](https://docs.basedpyright.com/dev/benefits-over-pyright/pylance-features/)); self-contained pip wheel, no Node needed | `openFilesOnly` default limits analysis — configure workspace-wide; references can be incomplete for unopened files | +| Go | gopls | Canonical; `go install` | Needs Go toolchain + `go.mod`; note gopls ≥0.20 has a built-in experimental MCP server — possible shortcut, evaluate at CI-4 | +| Rust | rust-analyzer | Prebuilt single binaries on GH releases (rustup shim caveat: [rustup#3846](https://github.com/rust-lang/rustup/issues/3846)) | **Worktrees**: set `CARGO_TARGET_DIR` per worktree + `rust-analyzer.cargo.targetDir` or instances serialize on cargo locks ([#10684](https://github.com/rust-lang/rust-analyzer/issues/10684)); weakest multi-root — one instance per cargo workspace; watcher walks all worktrees ([#16534](https://github.com/rust-lang/rust-analyzer/issues/16534)) | +| C# | **razzmatazz/csharp-language-server first**; Microsoft Roslyn LS later | csharp-ls: `dotnet tool install -g csharp-ls`, standard stdio LSP, active. Roslyn LS is what VS Code uses but: Azure-DevOps-feed acquisition, nonstandard named-pipe handshake, custom `workspace/projectInitializationComplete` readiness signal | OmniSharp effectively dead — do not use | +| F# | FsAutoComplete | `dotnet tool install fsautocomplete`; Ionide-maintained | Needs restorable solution (MSBuild project load) | +| Java | JDT-LS | Standard; tarball + **JDK 21+** | Heaviest warm-up (Maven/Gradle import; progress sticks); per-project `-data` workspace dirs; 1–2 GB+ heap | +| Kotlin | JetBrains kotlin-lsp — **defer (alpha)** | Official, but alpha, partially closed-source, IntelliJ-class memory, init timeouts on large projects; community fwcd server deprecated | Ship the card last; document degraded support | +| C/C++ | clangd | Prebuilt zips on GH releases | Practically requires `compile_commands.json` — descriptor should auto-suggest `cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON` / `bear` | + +**Operational invariants the supervisor must encode** (each cited in research; all promoted to +CI-3 pinning tests): strict `initialize`→`initialized` ordering; advertise `workspaceFolders` +capability *and* still pass rootUri (claude-code [#27220](https://github.com/anthropics/claude-code/issues/27220)); +**didOpen-before-query document tracker** (typescript-ls returns empty without it, +[#89](https://github.com/typescript-language-server/typescript-language-server/issues/89)) and +didChange/didClose on disk changes (server must not re-read opened docs); **position encoding** +— negotiate `utf-8` via `general.positionEncodings`, else convert UTF-16 code units (the +"bottom emoji breaks rust-analyzer" class); **readiness gating** — no standard signal exists; +consume `$/progress` + per-server signals (rust-analyzer `experimental/serverStatus.quiescent`, +Roslyn `projectInitializationComplete`) with per-descriptor `startupTimeout`; **client-side +file watching** — honor `workspace/didChangeWatchedFiles` registrations or servers go stale on +git checkouts (gopls depends on it); pass `processId` AND reap on our side (servers commonly +ignore parent-death); `shutdown` → `exit` → wait → SIGKILL; **diagnostics settle** before +`code_diagnostics` returns (push model is async); memory caps + idle eviction + orphan sweep +(the oh-my-opencode jdtls-leak system-freeze failure mode). + +## §4 Structural engine: ast-grep — vs Semgrep, comby, GritQL, srgn, raw tree-sitter + +| Engine | Verdict | Decisive facts | +|---|---|---| +| **ast-grep** — **CHOSEN** | Ship `@ast-grep/cli` as npm dep | MIT, ~14.5k★, very active (sole-maintainer risk noted; mitigated by capability seam). 25–31 langs + custom grammars. `--json=stream` NDJSON with metavariable captures; rewrite with diff preview; YAML rule system. **npm-native**: `@ast-grep/cli` (~569k dl/wk) resolves prebuilt platform binaries via optionalDependencies — no custom shipping; `@ast-grep/napi` (~2M dl/wk) for in-process use (repo map). De-facto agent choice (official MCP + Claude skill; Codemod jssg; oh-my-opencode) | +| Semgrep CE / Opengrep | Rejected for embedding | Python front-end + OCaml core subprocess — needs a Python runtime; seconds of per-invocation startup; rules relicensed restrictively Dec 2024 (engine still LGPL); security-lint-shaped, not edit-shaped. Opengrep fork (Jan 2025, 10+ vendors) is the safer governance if ever needed | +| comby | Rejected | Right interface (single binary, `-json-lines`) but last release June 2022; "v2 prep" commits June 2026 are a flicker, not a maintenance signal | +| GritQL | Rejected | Survived Honeycomb acquisition only by donation to Biome (Dec 2025); volunteer-paced; lives on as Biome's plugin language | +| srgn | Rejected | MIT single binary but 7 languages, no JSON output, beta CLI churn. (Claims that Codex bundles it: **unverified/likely false** — codex-universal ships ripgrep/fd/universal-ctags) | +| raw web-tree-sitter | Used for the repo map, not the tool | It's a parsing primitive — we'd rebuild matching/rewrite/CLI ergonomics ast-grep already has | + +## §5 Repo map: Aider-style ranked map — vs ctags, SCIP, stack-graphs, packers, embeddings + +**Upgrade adopted from research:** the map is not just "files → symbols"; it is a **ranked** +map — defs+refs graph with personalized PageRank, which is what makes 2–8 KB budgets effective. + +- **Aider's mechanism** (the model): tree-sitter `tags.scm` queries → def/ref tags per file → + `MultiDiGraph` (referencer file → definer file, weight `√refs` × multipliers: ×10 chat-mentioned + idents, ×10 well-named long idents, ×0.1 private `_`/defined-in->5-files, ×50 in-chat + referencing file) → personalized PageRank seeded by chat files → rank split across edges to + definitions → binary-search the tag count to the token budget (default 1k, up to 4k on + large-context models; ~2–8× expansion when no files in chat) → render signature lines only; + mtime-keyed disk cache ([repomap blog](https://aider.chat/2023/10/22/repomap.html), + [repomap.py](https://github.com/Aider-AI/aider/blob/main/aider/repomap.py)). +- **Evidence it pays**: RepoGraph (ICLR'25, [arXiv:2410.14684](https://arxiv.org/abs/2410.14684)) + — def/ref graph plug-in lifts SWE-bench-Lite resolve rates across four harnesses (≈+2–2.7 pts + absolute, ≈+33% relative avg); Agentless ([arXiv:2407.01489](https://arxiv.org/abs/2407.01489)) + shows cheap structural repo views rival full agentic exploration; CodePlan (FSE'24) likewise. + Aider itself publishes no ablation — the academic record carries the claim. +- **Node implementation path**: tag extraction via **web-tree-sitter + `tree-sitter-wasms` + prebuilt grammars** reusing Aider's vendored `tags.scm` queries (MIT; battle-tested across + 100+ languages), or `@ast-grep/napi` kind-queries where grammars overlap — decide at CI-5 + with a spike; ranking via `graphology` PageRank; cache by content hash per worktree. +- Rejected: **universal-ctags** (defs only — no refs ⇒ no ranking graph; the reason Aider left + it), **SCIP** (compiler-accurate but per-language indexers + build required; scip-typescript + last released Oct 2024), **stack-graphs** (archived 2025-09-09 — dead; GitHub itself fell + back to tree-sitter tag search), **CodeQL** (license bars closed-source analysis; minutes of + DB build), **repomix/packers** (one-shot dumps, unranked, token cost scales with repo). +- **Embeddings**: genuinely contested. Cursor's eval claims +12.5% avg agent accuracy from + semantic search; Anthropic found agentic search "outperformed RAG by a lot" and ships none; + continue.dev deprecated its embeddings `@codebase`. Both headline numbers are + vendor-interested. Decision: **not in v1** (infra + staleness + privacy cost; weaker fit for + our worktree-keyed model); the `code.map`/search capability seam leaves room for an optional + embeddings pack later if BENCH shows orientation gaps on very large repos. + +## §6 Diagnostics: custom rdjson-shaped schema + errorformat fallback — vs SARIF-internal + +- **Native structured output exists for most of the fleet**: ESLint `--format json`; Biome + `--reporter=rdjson|sarif`; ruff `--output-format json|rdjson|sarif` (best in class); mypy + `--output=json` (≥1.11); `go vet -json` / `go test -json` (compile errors arrive as text in + Output events — caveat); cargo `--message-format=json` (gold standard, includes fixes); + dotnet via `-p:ErrorLog=out.sarif`. **tsc has no JSON** ([TS#46340](https://github.com/microsoft/TypeScript/issues/46340)) + — but `--pretty false` output has been regex-stable for a decade. **pytest**: use built-in + `--junitxml`; the json-report plugin is third-party/lightly maintained. **javac/gradle/maven**: + weakest — text parsing territory. +- **Canonical internal schema: our own minimal record** `{tool,file,line,endLine?,col?,code, + severity,message,fix?}` — essentially a flattened **rdjsonl** (reviewdog diagnostic format). + ruff and Biome already emit rdjson natively, validating the shape. **SARIF is an input, not + the model**: deeply nested, 90% of its surface irrelevant to an agent loop; one SARIF→internal + adapter unlocks dotnet, semgrep, clang, CodeQL at once. +- **Bundle reviewdog's `errorformat`** (single static Go binary, ~50+ preset parsers + user + `-efm` patterns, emits rdjsonl): the universal fallback that covers javac/gradle and the + long tail without writing parsers. This one *does* ship via the `binaries/` mechanism. +- Peer check: Claude Code feeds raw hook stderr text; Cursor pipes LSP diagnostics (uneven by + language); Codex/Devin read raw logs. A typed multi-checker `code_check` is ahead of all of + them; LSP pull-diagnostics is a later fast path (CLI checkers stay source-of-truth because + they reflect project task orchestration — nx/turbo configs, generated code). + +## §7 Memory layer (branch-level check, settles the EP G2 bet) + +Researched because the PR branch commits to Hindsight (EP G2). **Bet validated, framing +sharpened**: vectorize-io/hindsight is MIT, very active (v0.8.1 June 2026), Node SDK, hybrid +retrieval (semantic+BM25+graph+temporal), Postgres+pgvector, and the Nous Hermes agent +integration proves the exact daemon-autostart-pack pattern. Alternatives are worse fits: mem0 +(cloud-leaning, disputed benchmarks), Zep (self-hosted CE discontinued), cognee (no TS SDK), +Letta (a competing harness). **But** the 2026 production trend is text-first (Claude Code +CLAUDE.md/auto-memory, Devin Knowledge, Cursor memories) — so: **markdown staff memory stays +authoritative** for procedural knowledge; Hindsight is the *optional* episodic-memory pack, +gated on Docker/Postgres availability, an LLM key per `retain` priced in. This nuance is +recorded against EP G2 here rather than rewriting the EP doc. + +## §8 Net changes applied to the CI docs after this research + +1. CI-3 engine named: `vscode-jsonrpc` + `vscode-languageserver-protocol`; coc.nvim as + lifecycle reference; solidlsp's sync+cache and multilspy's binary auto-download as ported + designs; §3's operational invariants promoted into CI-3 tests; **post-edit auto-diagnostics** + added to CI-3 scope (match Claude Code). +2. Server picks: **vtsls**, **basedpyright**, **csharp-ls-first**, kotlin deferred (alpha); + per-worktree `CARGO_TARGET_DIR`; clangd compile-commands hint in descriptor. +3. CI-1 shipping: `@ast-grep/cli` npm dependency (platform optionalDependencies) instead of + the custom `binaries/` route; `--json=stream`. +4. CI-2: rdjsonl-shaped internal schema; `errorformat` bundled as fallback; SARIF adapter; + pytest via junitxml; tsc regex parser. +5. CI-5: upgraded from flat symbol listing to **ranked** def/ref-graph map (personalized + PageRank, session-aware seeds), with the RepoGraph/Agentless evidence as rationale. diff --git a/docs/design/code-intelligence-implementation-plan.md b/docs/design/code-intelligence-implementation-plan.md index 0de8be1b1..2f8e224fd 100644 --- a/docs/design/code-intelligence-implementation-plan.md +++ b/docs/design/code-intelligence-implementation-plan.md @@ -4,7 +4,10 @@ Status: ready for execution, not started. Workstream **CI** in [fable-program-execution-plan.md](fable-program-execution-plan.md). Companion to [code-intelligence.md](code-intelligence.md) (WHAT/WHY + the layer model and -design stance — its §7 worktree contract and §1 capability names are LAW). +design stance — its §7 worktree contract and §1 capability names are LAW) and +[code-intelligence-alternatives.md](code-intelligence-alternatives.md) (the research annex: +why each engine/library below beat its alternatives, with citations and the operational +invariants their issue trackers taught us — **read its §3 before implementing CI-3**). > **Anchor baseline:** fable-docs @ 2026-06-11 (master parent `6ec8c8f9`). Locate by symbol > name; missing symbol ⇒ STOP, re-derive from the cited pattern — never improvise. @@ -37,15 +40,17 @@ CI-1..CI-5 ──→ CI-6 services chip + graphify viz pack ──→ CI-7 capab edits become obsolete. **Owned files:** NEW `defaults/tools/ast/{ast_search.yaml,ast_rewrite.yaml,extension.ts}`; -ast-grep binary wiring in `binaries/` + `binaries.versions.json` + checksums (copy an -existing entry's pattern end-to-end, incl. `scripts/` download/verify step); NEW -`src/server/agent/ast-grep-runner.ts`; NEW `tests/ast-grep-runner.test.ts`, -`tests/e2e/ast-tools.spec.ts`; budget rows in `tests/tool-description-budget.test.ts`. +`@ast-grep/cli` added as a pinned npm dependency (prebuilt platform binaries via +optionalDependencies — no `binaries/` wiring needed; resolve the binary path via +`require.resolve`); NEW `src/server/agent/ast-grep-runner.ts`; NEW +`tests/ast-grep-runner.test.ts`, `tests/e2e/ast-tools.spec.ts`; budget rows in +`tests/tool-description-budget.test.ts`. **Steps** -1. Runner: `runAstGrep({pattern, rewrite?, lang?, paths?, cwd})` → execFile the pinned - binary with `--json`; map results to `{file, line, column, text}[]` (relative paths +1. Runner: `runAstGrep({pattern, rewrite?, lang?, paths?, cwd})` → execFile the resolved + binary with `--json=stream` (NDJSON); map results to `{file, line, column, text}[]` + plus metavariable captures (relative paths ONLY — pin per design §7); output cap + spill via `truncateLargeToolContent` (`src/server/agent/truncate-large-content.ts`). 2. `ast_search`: tool YAML per the `defaults/tools/team/` anatomy; renderer lists matches @@ -77,9 +82,16 @@ captured-output fixtures; e2e. **Steps:** checker set resolution (project config override → autodiscovery by manifest: `tsconfig.json`⇒tsc+eslint, `pyproject.toml`⇒pytest/ruff if present, `go.mod`⇒go vet); -each parser: raw output → `{tool, file, line, code?, severity, message}[]`; result = -summary counts + first N (default 50) diagnostics + spill ref; unknown checker ⇒ capped -raw fallback, never an error. Renderer: grouped-by-file list. Verification-harness +schema = flattened rdjsonl: `{tool, file, line, endLine?, col?, code?, severity, message, +fix?}[]`; per-tool strategy (annex §6): native JSON for eslint / ruff‑rdjson / mypy +`--output=json` / cargo `--message-format=json` / `go vet -json` (NB compile errors arrive +as text inside `go test -json` Output events); stable regex for `tsc --pretty false` +(`^path(line,col): error TSnnnn: msg`); `--junitxml` for pytest (not the json plugin); +one SARIF→internal adapter (covers dotnet `-p:ErrorLog=…sarif`, semgrep, clang); bundled +reviewdog **`errorformat`** static binary via the `binaries/` mechanism as universal +fallback (preset parsers + user `-efm` from project config). Result = summary counts + +first N (default 50) diagnostics + spill ref; unknown checker ⇒ capped raw fallback, never +an error. Renderer: grouped-by-file list. Verification-harness consumption is a recorded follow-up (do NOT touch `verification-harness.ts` here — CE-G3.3 / CE-G5.2 own it per §1.4). @@ -91,28 +103,53 @@ ledger note recorded in the doc. ## CI-3 — LSP supervisor + `code_*` symbol tools *(contract level; the big one)* -Contracts: design §2. **Owned (new):** `src/server/agent/lsp/` -(`lsp-supervisor.ts`, `lsp-client.ts` JSON-RPC, `language-registry.ts` detection rules, +Contracts: design §2; **annex §§2–3 are required reading** (engine rationale + the +operational invariants harvested from rust-analyzer/gopls/tsserver/Serena/Claude Code +issue trackers). **Owned (new):** `src/server/agent/lsp/` (`lsp-supervisor.ts`; +`lsp-client.ts` on **`vscode-jsonrpc` + `vscode-languageserver-protocol`** — coc.nvim's +client layer is the reference implementation; `language-registry.ts` detection rules; descriptor schema); `defaults/tools/code/{code_definition,code_references,code_symbols, -code_hover,code_rename,code_diagnostics}.yaml`; descriptor data for TypeScript + Python. +code_hover,code_rename,code_diagnostics}.yaml`; descriptor data for TypeScript (**vtsls**) ++ Python (**basedpyright**, workspace-wide analysis configured, not `openFilesOnly`). Invariants (each gets a pinning test): 1. Supervisor = idempotent ensure with in-flight dedupe (`sandbox-manager.ts` shape); key `(worktreeRoot, languageId)`; idle-shutdown default 10 min; max concurrent servers - (default 4, config); crash ⇒ restart with backoff, surfaced non-fatally (tool returns - "language server unavailable — falling back" guidance, never hangs the turn). -2. Tools degrade gracefully: no descriptor for the file's language ⇒ structured "not + (default 4, config) + per-server memory ceilings; crash ⇒ restart with backoff, + surfaced non-fatally (tool returns "language server unavailable — falling back" + guidance, never hangs the turn). +2. Protocol correctness: strict `initialize`→`initialized` ordering; advertise + `workspaceFolders` *and* pass `rootUri`; pass `processId` AND reap our side + (`shutdown`→`exit`→wait→SIGKILL); negotiate `general.positionEncodings: utf-8`, else + convert UTF-16 code units at the client boundary. +3. Document lifecycle tracker: `didOpen` before any query against a file (tsserver returns + empty otherwise); `didChange`/`didClose` when our edit tools or git change disk; honor + server `didChangeWatchedFiles` registrations (forward our watcher events) or servers go + stale on checkouts. +4. Readiness gating: per-descriptor warm-up signal (`$/progress` settle, rust-analyzer + `experimental/serverStatus.quiescent`, Roslyn `projectInitializationComplete`) + + `startupTimeout`; `code_diagnostics` waits for the push-diagnostics settle window. +5. Tools degrade gracefully: no descriptor for the file's language ⇒ structured "not covered; detected language X; install lsp-X pack" result (feeds CI-4's UX), never an exception. -3. `code_rename` is preview-diff → apply through the file-edit path (CI-1 step 3 pattern). -4. All positions/paths relative; budget + spill on every result. -5. Lifecycle: worktree removal (goal cleanup) disposes servers + caches — hook the - existing worktree-cleanup path; Caretaker sweep (MC) catches leaks. - -Tests: supervisor unit with a fake LS binary (spawn/dedupe/idle/crash); e2e against -typescript-language-server on a fixture project: definition/references/rename round-trip; -uncovered-language fallback; worktree-dispose kills the server (process poll). +6. `code_rename` is preview-diff → apply through the file-edit path (CI-1 step 3 pattern); + references results exclude gitignored paths (Claude Code regression #26051 class). +7. All positions/paths relative; budget + spill on every result. +8. Lifecycle: worktree removal (goal cleanup) disposes servers + caches — hook the + existing worktree-cleanup path; Caretaker sweep (MC) catches leaks/orphans (the + jdtls-leak failure mode). Per-worktree env isolation lives in the descriptor (e.g. + rust: `CARGO_TARGET_DIR=<worktree>/.bobbit-cache/target`). +9. **Post-edit diagnostics push**: after our edit tools touch a covered file, fresh (not + stale pre-edit) diagnostics are injected into the turn, capped; per-descriptor + `diagnostics: false` opt-out. Port Serena's solidlsp symbol cache (two-tier, content- + keyed) for hot `code_symbols`/`code_definition` paths. + +Tests: supervisor unit with a fake LS binary (spawn/dedupe/idle/crash/handshake-order/ +encoding negotiation); e2e against vtsls on a fixture project: definition/references/ +rename round-trip + post-edit diagnostics; uncovered-language fallback; worktree-dispose +kills the server (process poll); stale-diagnostics pin (edit → old diagnostics never +delivered after new edit). **Acceptance:** on the Bobbit repo itself, `code_references` on a core symbol returns in <2 s warm, and the supervisor shows 0 servers after idle timeout. @@ -120,22 +157,33 @@ uncovered-language fallback; worktree-dispose kills the server (process poll). ## CI-4 — Language packs + autodiscovery UX *(contract level)* Descriptor-as-pack: `market-packs/lsp-<lang>/` containing one `descriptor.yaml` -(server binary acquisition: npm/pip/download+checksum; args; init options; health probe; -file-detection globs) — data only, no code. Bundle `lsp-typescript`, `lsp-python` as -built-ins (band per `builtin-packs.ts`); **cards** (checklist rows, not yet specced): -`lsp-go`, `lsp-rust`, `lsp-csharp` (Roslyn), `lsp-fsharp` (FsAutoComplete), `lsp-jvm` -(JDT-LS), `lsp-clangd`, `lsp-kotlin`. Autodiscovery surfaces uncovered detected languages -through the CI-6 chip + a marketplace deep-link. Acceptance: installing `lsp-go` on a Go -fixture makes `code_definition` work with zero config; uninstalling degrades gracefully. - -## CI-5 — `code_map` budgeted repo map *(contract level; after CI-1)* - -`code_map(path?, depth?)`: directory → file → public symbols/signatures from ast-grep -parse output; content-hash cache per worktree under the design-§7 cache dir; hard result -budget (default 2 KB, arg-raisable to 8 KB); relative paths pin. When EP G1.3 (prompt -sections via providers) is merged, add the provider that contributes a "Repo Map" section -under the same budget — the tool stays for on-demand deeper maps. Acceptance: on the -Bobbit repo, default call ≤2 KB and a cold→warm rebuild is ≥10× faster (cache test). +(server binary acquisition: npm/pip/download+checksum; args; init options; readiness +signal; health probe; file-detection globs; per-worktree env) — data only, no code. +Bundle `lsp-typescript` (vtsls), `lsp-python` (basedpyright) as built-ins (band per +`builtin-packs.ts`); **cards** (checklist rows, not yet specced), server picks per annex +§3: `lsp-go` (gopls — evaluate its built-in MCP server as a shortcut), `lsp-rust` +(rust-analyzer + CARGO_TARGET_DIR isolation), `lsp-csharp` (**csharp-ls first**, Microsoft +Roslyn LS as a later upgrade — nonstandard handshake), `lsp-fsharp` (FsAutoComplete), +`lsp-jvm` (JDT-LS, JDK 21+, per-project `-data` dirs), `lsp-clangd` (descriptor suggests +generating `compile_commands.json`), `lsp-kotlin` (**last** — kotlin-lsp is alpha). +Autodiscovery surfaces uncovered detected languages through the CI-6 chip + a marketplace +deep-link. Acceptance: installing `lsp-go` on a Go fixture makes `code_definition` work +with zero config; uninstalling degrades gracefully. + +## CI-5 — `code_map` budgeted **ranked** repo map *(contract level; after CI-1)* + +Aider's algorithm, in Node (annex §5 has the full mechanism + evidence): tag extraction +(defs **and** refs) via web-tree-sitter + `tree-sitter-wasms` prebuilt grammars reusing +Aider's MIT `tags.scm` queries — spike vs `@ast-grep/napi` kind-queries first, pick one — +→ def/ref file graph (edge weight `√refs` × multipliers: boost session-mentioned idents +and in-context files, damp `_`-private and defined-everywhere idents) → personalized +PageRank (`graphology`) seeded by session state → signature-only rendering, binary-search +tag count to budget. `code_map(path?, depth?, budget?)`: hard result budget (default 2 KB, +max 8 KB); content-hash cache per worktree under the design-§7 cache dir; relative paths +pin. When EP G1.3 (prompt sections via providers) is merged, add the provider that +contributes a "Repo Map" section under the same budget — the tool stays for on-demand +deeper maps. Acceptance: on the Bobbit repo, default call ≤2 KB, ranking test (a hub +module outranks a leaf util), and cold→warm rebuild ≥10× faster (cache test). ## CI-6 — Context-services chip + graphify visualization pack *(contract level)* diff --git a/docs/design/code-intelligence.md b/docs/design/code-intelligence.md index 014629d3b..d3ee33c91 100644 --- a/docs/design/code-intelligence.md +++ b/docs/design/code-intelligence.md @@ -5,6 +5,10 @@ Status: design accepted, not started. Workstream **CI** in > **Execution authority:** implement from > [code-intelligence-implementation-plan.md](code-intelligence-implementation-plan.md). +> **Alternatives & evidence:** every engine/library pick below is justified against its +> competitors, with citations, in +> [code-intelligence-alternatives.md](code-intelligence-alternatives.md) — read it before +> swapping any component. **The problem:** Bobbit agents are "an LLM with grep". 56% of tool calls are bash (token-cost-efficiency.md §3), discovery is multi-round grep cascades, edits are text @@ -62,19 +66,29 @@ it does not duplicate the temporal one. `code_symbols` (document/workspace), `code_hover`, `code_rename` (preview-diff → review-flow apply), `code_diagnostics`. Budgeted outputs (cap + spill per CE-G2), renderers that show locations as clickable `file:line`. -- **Engine:** a gateway-side **LSP supervisor** (`lsp-supervisor.ts`): spawns language - servers per `(worktree, language)`, idle-shutdown (default 10 min), capped concurrent - servers, health-restart — the `sandbox-manager.ts` idempotent-ensure shape. Servers are - *processes the gateway owns*; agents never talk LSP directly. +- **Engine:** a gateway-side **LSP supervisor** (`lsp-supervisor.ts`) built on + `vscode-jsonrpc` + `vscode-languageserver-protocol` (Microsoft-maintained, standalone- + capable; coc.nvim is the reference lifecycle implementation): spawns language servers per + `(worktree, language)`, idle-shutdown (default 10 min), capped concurrent servers, + health-restart — the `sandbox-manager.ts` idempotent-ensure shape. Servers are *processes + the gateway owns*; agents never talk LSP directly. The supervisor also pushes **post-edit + diagnostics** into the turn (the Claude Code v2.0.74 pattern — fix without a separate + build step). The operational invariants (didOpen tracking, position-encoding negotiation, + readiness gating, client-side file watching, orphan reaping…) are enumerated in + [code-intelligence-alternatives.md §3](code-intelligence-alternatives.md) and are CI-3 + pinning tests. - **Language autodiscovery:** a detection registry maps manifest/file signals → - language-server descriptor (`tsconfig.json`/`package.json` → typescript-language-server; - `pyproject.toml` → pyright; `go.mod` → gopls; `Cargo.toml` → rust-analyzer; `.csproj`/ - `.fsproj` → Roslyn/FsAutoComplete; `pom.xml`/`build.gradle*` → JDT-LS; `CMakeLists.txt`/ - compile-commands → clangd). Descriptors ship as **per-language packs** + language-server descriptor (`tsconfig.json`/`package.json` → **vtsls**; + `pyproject.toml` → **basedpyright**; `go.mod` → gopls; `Cargo.toml` → rust-analyzer + (per-worktree `CARGO_TARGET_DIR`); `.csproj` → csharp-ls; `.fsproj` → FsAutoComplete; + `pom.xml`/`build.gradle*` → JDT-LS; `CMakeLists.txt`/compile-commands → clangd; Kotlin + deferred — kotlin-lsp is alpha). Descriptors ship as **per-language packs** (`lsp-typescript`, `lsp-python`, …): the descriptor is data (binary acquisition, args, init options, health probe), so adding a fleet language = a pack, not core code. v1 bundles TypeScript + Python; the rest are cards (CI-4) — autodiscovery *detects* an uncovered language and surfaces "install the lsp-go pack" in the UI instead of failing. + This config-not-binaries pack model is the same shape Claude Code (LSP plugins) and + Copilot CLI (`.github/lsp.json`) converged on independently. - **Why not Serena/MCP as the surface:** same capability, but token-opaque outputs, no renderer, no tool-guard tiers, Python runtime baggage per worktree, and the observed MCP-underuse problem. Serena remains the design reference and a fallback pack if the @@ -82,10 +96,11 @@ it does not duplicate the temporal one. ## §3 Structural layer — ast-grep tools -`ast_search(pattern, lang?, paths?)` and `ast_rewrite(pattern, rewrite, …)` over the -**ast-grep** single static binary (shipped via the existing `binaries/` mechanism — -`binaries.versions.json` + checksums; per-call execution, nothing to supervise, ~25 -languages). `ast_rewrite` never writes directly: it returns a unified diff rendered through +`ast_search(pattern, lang?, paths?)` and `ast_rewrite(pattern, rewrite, …)` over +**ast-grep** (MIT, the de-facto structural engine for agents), shipped as the +`@ast-grep/cli` npm dependency — prebuilt per-platform binaries via optionalDependencies, +no custom shipping; per-call execution with `--json=stream`, nothing to supervise, ~25–31 +languages + custom tree-sitter grammars. `ast_rewrite` never writes directly: it returns a unified diff rendered through the existing diff renderer; applying goes through the normal edit path so review-pane and git-status flows see it. This is the single biggest verifiability upgrade over regex/sed — and the first goal to land (CI-1). @@ -94,14 +109,20 @@ and the first goal to land (CI-1). Two deliberately separate concerns: -- **Prompt/orientation path (in-house, small):** `code_map` — an Aider-style budgeted map: - per-directory → file → public symbols/signatures, built from ast-grep's parse output - (no new parser dependency), content-hash cached per worktree, **hard token budget** - (default 2 KB tool result; later an EP provider contributes a prompt section under the - same budget machinery as the skills catalog). Owner asked "isn't in-house overkill?" — - no, because this reuses the ast-grep engine from §3 and the existing budget/prompt-section - patterns; the *graph database* part of graphify is precisely what we don't need for - prompts. Deterministic, relative-path-only, worktree-safe by construction. +- **Prompt/orientation path (in-house, small):** `code_map` — an Aider-style budgeted + **ranked** map: tree-sitter tag extraction (defs **and** refs) → file/symbol graph → + personalized PageRank seeded by session state (files in context, identifiers mentioned in + the goal) → signature-only rendering, binary-searched down to a **hard token budget** + (default 2 KB tool result, arg-raisable to 8 KB; later an EP provider contributes a + prompt section under the same budget machinery as the skills catalog). Content-hash + cached per worktree. The ranking is what makes a 2 KB map useful — flat symbol dumps + aren't (evidence: RepoGraph ICLR'25 ≈+33% relative on SWE-bench-Lite; details and the + Aider algorithm in [code-intelligence-alternatives.md §5](code-intelligence-alternatives.md)). + Owner asked "isn't in-house overkill?" — no: extraction reuses tree-sitter/ast-grep + machinery and Aider's MIT `tags.scm` queries; the *graph database* part of graphify is + precisely what we don't need for prompts. Deterministic, relative-path-only, + worktree-safe by construction. Embeddings indexing: deliberately not in v1 (contested + evidence, real infra cost — annex §5); the capability seam leaves room for a pack. - **Visualization path (reuse, optional):** a `graphify` **marketplace pack** (not built-in) wrapping graphify per worktree: output to a gitignored `graphify-out/`, rendered in a pack panel (`#/ext/graphify`). Known risk to verify at install: absolute-path leakage in @@ -113,10 +134,16 @@ Two deliberately separate concerns: `code_check(scope)` runs the project's known checkers (from project config; autodiscovered defaults: `tsc --noEmit`, eslint, pytest, `go vet`/`go test`, `cargo check`) and returns -**typed results**: `{tool, file, line, code, severity, message}[]` + summary counts — -never raw logs (cap + spill for the rare overflow). Verification gates can consume the same -parser output, and the team-lead's "did it work" loop becomes data-driven. Parsers are -per-tool modules; unknown checkers fall back to capped raw output (never block). +**typed results** — a flattened-rdjsonl record `{tool, file, line, endLine?, col?, code, +severity, message, fix?}[]` + summary counts — never raw logs (cap + spill for the rare +overflow). Parser strategy: native JSON where it exists (eslint, ruff/Biome rdjson, mypy, +cargo, go), stable-regex for `tsc --pretty false`, JUnit XML for pytest, a SARIF→internal +adapter (unlocks dotnet/semgrep/clang in one converter), and the bundled reviewdog +**`errorformat`** static binary as the universal fallback (~50 preset parsers + user `-efm` +patterns) — that one ships via the existing `binaries/` mechanism. Verification gates can +consume the same parser output, and the team-lead's "did it work" loop becomes data-driven. +Unknown checkers fall back to capped raw output (never block). No shipping agent harness +has this layer typed today — Claude Code feeds raw hook text, Codex/Devin read raw logs. ## §6 UX — "what's running, where?" (owner requirement) @@ -149,7 +176,9 @@ artifacts for absolute paths — the graphify lesson, enforced. - Every mutating surface (`ast_rewrite`, `code_rename`) previews before applying; every CI tool result is budgeted; `code_check` closes every edit loop. -Cross-references: [token-cost-efficiency.md](token-cost-efficiency.md) (CE-G2/G7), +Cross-references: [code-intelligence-alternatives.md](code-intelligence-alternatives.md) +(alternatives, citations, research record), +[token-cost-efficiency.md](token-cost-efficiency.md) (CE-G2/G7), [extension-platform.md](extension-platform.md) (capabilities §3.1, providers, runtimes), [harness-gap-analysis.md](harness-gap-analysis.md) (peer evidence), [mission-control.md](mission-control.md) (Caretaker sweep, flight recorder for service From 04b406f4520f8c33d6455e13c8291ab35b746bf1 Mon Sep 17 00:00:00 2001 From: Claude <noreply@anthropic.com> Date: Thu, 11 Jun 2026 10:46:45 +0000 Subject: [PATCH 11/18] =?UTF-8?q?CI=20annex:=20record=20the=20full=20Seren?= =?UTF-8?q?a-vs-native-supervisor=20argument=20(=C2=A72.1)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Captures the owner Q&A in the research annex so the decision can't be re-litigated without the evidence: stack comparison diagram, the four-part argument (Serena's valuable layer is the lifecycle we'd rewrite anyway — per-worktree pooling is absent from its architecture; the protocol layer is the cheap part via vscode-jsonrpc; the MCP surface is the observed underuse failure mode a wrapper can't fix; Serena stays as ported designs + the ~one-week escape-hatch pack behind code.symbol-nav), and the one-sentence verdict. Design doc's "why not Serena" bullet now links to it. https://claude.ai/code/session_01LBW68Z2KvA3EhQMq3c2wrR --- docs/design/code-intelligence-alternatives.md | 36 +++++++++++++++++++ docs/design/code-intelligence.md | 6 ++-- 2 files changed, 40 insertions(+), 2 deletions(-) diff --git a/docs/design/code-intelligence-alternatives.md b/docs/design/code-intelligence-alternatives.md index fe8a2c333..4f63e4344 100644 --- a/docs/design/code-intelligence-alternatives.md +++ b/docs/design/code-intelligence-alternatives.md @@ -44,6 +44,42 @@ capability-swappable engines. | multilspy (Microsoft) | N/A directly | Python-only research library (13 langs); its 2026 fix history (server hangs, orphaned processes) confirms lifecycle is where the bodies are buried | | MCP bridges (mcp-language-server 1.5k★, lsmcp, cclsp, lsproxy) | Rejected | All single-maintainer; most dormant ≥10 months (mcp-language-server since 2025-06, lsmcp since 2025-08); single-root designs; none do worktree-aware pooling. lsproxy (REST, Rust, multi-server pool) is the closest design reference | +### §2.1 The Serena question, in full ("why not one LSP to rule them all, wrapped and fixed?") + +Owner asked this directly (2026-06-11); recording the complete argument since it's the most +tempting future re-litigation. Compare the two stacks: + +``` +Wrap Serena: agents → MCP protocol → Serena (Python 3.13 + uv sidecar) → solidlsp → language servers +Proposed: agents → first-party code_* tools → LSP supervisor (in the gateway, TS) → language servers +``` + +1. **The valuable part of Serena is the part we'd rewrite anyway.** Its worth is the bottom + layer — server lifecycle + symbol caching — and its tracker shows that battle ongoing + (§2 table: #944 30 GB RAM, #1390/#937 init hangs, #900 stdio races, #634 re-init loops). + The fixes Bobbit needs most — **per-(worktree, language) pooling, idle eviction, disposal + wired into goal-worktree cleanup** — are absent from Serena's one-project-one-session + architecture. That's not a wrapper-sized change; it's the design. +2. **The protocol layer — the part that looks scary — is the cheap part.** Microsoft's + `vscode-jsonrpc` + `vscode-languageserver-protocol` give the full typed client off the + shelf; the lifecycle layer is ~500–1000 lines with coc.nvim as reference. So "buy Serena" + saves the *easy* part while charging a Python-3.13/uv runtime, a sidecar process per + workspace, and a second supervisor-in-a-different-language to debug when it wedges. +3. **The MCP surface is the already-observed failure mode.** Generic MCP tools = no token + budgets, no `file:line` renderers, no tool-guard tiers, no prompt guidance — and Serena's + rename writes files directly, bypassing the preview-diff → review flow. The model-facing + UX, where tool adoption is won or lost, is precisely what a wrapper can't fix. +4. **Serena still pays us twice.** We port its two best designs (synchronous two-tier symbol + cache; auto-download of server binaries, both also in multilspy), and because everything + sits behind the `code.symbol-nav` capability, a Serena-backed pack remains a legitimate + community alternative — and the **escape hatch**: if the CI-3 supervisor stalls in its + wave, wrapping Serena behind the same capability is roughly a week's work, not a rewrite. + +One-sentence verdict: *Serena is the right idea delivered as the wrong dependency — keep the +idea (supervised language servers behind symbol tools), delete the Python/MCP middle, build +the thin lifecycle layer natively where Bobbit's budgets, renderers, review flow, and +worktree model already live.* + ## §3 Per-language servers (research-corrected picks) Two picks **changed** from the original draft: TS/JS `typescript-language-server` → **vtsls**; diff --git a/docs/design/code-intelligence.md b/docs/design/code-intelligence.md index d3ee33c91..b87b10342 100644 --- a/docs/design/code-intelligence.md +++ b/docs/design/code-intelligence.md @@ -91,8 +91,10 @@ it does not duplicate the temporal one. Copilot CLI (`.github/lsp.json`) converged on independently. - **Why not Serena/MCP as the surface:** same capability, but token-opaque outputs, no renderer, no tool-guard tiers, Python runtime baggage per worktree, and the observed - MCP-underuse problem. Serena remains the design reference and a fallback pack if the - supervisor proves harder than planned (the capability seam makes that swap cheap). + MCP-underuse problem. Serena remains the design reference and the escape hatch — a + Serena-backed pack behind the same `code.symbol-nav` capability is ~a week's work if the + supervisor stalls. Full argument: + [code-intelligence-alternatives.md §2.1](code-intelligence-alternatives.md). ## §3 Structural layer — ast-grep tools From dca6145fd84b22c06046b9f44a29bf6174990ce5 Mon Sep 17 00:00:00 2001 From: Claude <noreply@anthropic.com> Date: Thu, 11 Jun 2026 13:04:42 +0000 Subject: [PATCH 12/18] Add agent-memory.md decision record; fix EP G2 bank topology from verified facts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit agent-memory.md (new): three-layer model (local transcripts = system of record, stay; markdown staff memory = authoritative/curated; Hindsight = optional episodic recall) with the distill-then-prune Caretaker retention invariant; full Hindsight-vs-alternatives comparison with reasons (mem0, Letta, Zep/Graphiti, cognee, LangMem, file-only); bank-topology decision — one shared tag-scoped bank with pack-side auto-tagging (project:/agent:/goal:/kind:) and scope: project|global|all on recall. Grounded in verified Hindsight docs (2026-06-11): cross-bank search is NOT supported (all ops target one bank), tags ARE the filtering mechanism with strict modes, shared-bank-with-tags is an endorsed pattern. This corrects EP G2's earlier per-project-banks sketch and the bank: current|global|all tool arg, which could not be implemented as written: - extension-platform.md: manifest description + provider flow/tools updated (scope→tag filters, auto-tagging, link to the decision record). - extension-platform-implementation-plan.md G2: two upstream verification checkpoints added (tag-filtered strict recall; delete-by-tag for project offboarding, with per-project-bank fallback if absent). - execution plan: Caretaker retention-consolidation card (after EP G2+MC P3). - CI research annex §7 now points to the full record. https://claude.ai/code/session_01LBW68Z2KvA3EhQMq3c2wrR --- docs/design/agent-memory.md | 95 +++++++++++++++++++ docs/design/code-intelligence-alternatives.md | 5 +- .../extension-platform-implementation-plan.md | 7 ++ docs/design/extension-platform.md | 16 ++-- docs/design/fable-program-execution-plan.md | 1 + 5 files changed, 117 insertions(+), 7 deletions(-) create mode 100644 docs/design/agent-memory.md diff --git a/docs/design/agent-memory.md b/docs/design/agent-memory.md new file mode 100644 index 000000000..87fa9cf9e --- /dev/null +++ b/docs/design/agent-memory.md @@ -0,0 +1,95 @@ +# Agent Memory — architecture decision record + +Status: accepted 2026-06-11 (owner interview + research). Governs the memory layers across +workstreams: EP G2/G3.3 (Hindsight pack), EP G1.6 (session-memory provider), GA-R5 +(profile memory), MC Caretaker (retention). Research base: +[code-intelligence-alternatives.md §7](code-intelligence-alternatives.md) + Hindsight +docs/best-practices (verified 2026-06-11). + +## §1 Three layers, three jobs — sessions stay local + +| Layer | What it is | Job | Lifetime | +|---|---|---|---| +| **Session transcripts** (local JSONL + FlexSearch) | the *system of record* | replay, debugging, flight recorder (MC), audit, exact-quote search | prunable by policy, never silently | +| **Markdown staff memory** (per-role/project files) | *curated, authoritative* procedural knowledge — conventions, gotchas, preferences | always-loaded context; human-auditable and editable | permanent, small | +| **Hindsight pack** (optional, EP G2) | *episodic recall* — distilled facts/experiences/entities/beliefs over high-volume history | "what did we decide about X in March", cross-session/cross-agent recall, temporal queries | grows; managed by the daemon | + +**Do we still need local sessions if Hindsight is enabled? Yes — they are different +artifacts.** Hindsight stores *LLM-extracted distillations* (an LLM call per retain), not +transcripts; it is lossy by design and cannot replay a session, settle a "what exactly did +the agent run" dispute, or feed the MC flight recorder. The industry pattern matches: +Claude Code keeps transcripts AND auto-memory; Devin keeps sessions AND Knowledge. Memory +systems answer questions; transcripts prove them. + +**Pruning (owner's instinct, adopted):** transcript retention becomes config +(`sessions.retention`: age/size/per-project caps), and the **Caretaker (MC) runs a +consolidation pass before pruning**: for transcripts past the threshold — ensure the +Hindsight retain happened (if pack enabled), write/refresh the markdown summary, then +prune. Order is the invariant: **distill → then delete; never silently drop unretained +history.** This lands as a Caretaker card under MC once EP G2 exists. + +## §2 Is Hindsight the best daemon? Yes — comparison and reasons + +(Condensed from the research annex §7; URLs there.) + +| System | Verdict | Decisive reasons | +|---|---|---| +| **Hindsight** (vectorize-io) — **CHOSEN** | First-party memory pack | MIT; very active (v0.8.1 June 2026, 60+ releases); **official Node/TS SDK + REST + MCP**; hybrid recall (semantic + BM25 + graph + temporal) with reranking; self-hosted on Postgres+pgvector (no external vector-DB dependency); peer-reviewed design (arXiv:2512.12818); the Nous Hermes coding agent ships it as a native memory provider with daemon autostart + 5-min idle stop — the exact pack pattern EP G3.3 plans | +| mem0 | Rejected | Cloud-leaning OSS; benchmark claims disputed by third parties (LoCoMo rebuttals); weaker self-host story | +| Letta (MemGPT) | Rejected | It's a competing *agent harness* with memory inside, not a memory service for ours | +| Zep / Graphiti | Rejected | Zep discontinued its self-hosted CE; only the Graphiti library remains (bring-your-own graph DB + service layer = we'd build the daemon ourselves) | +| cognee | Rejected | Lightest self-host but Python-only SDK — wrong language for the gateway | +| LangMem | Rejected | Little value outside LangChain/LangGraph | +| File-based only | **Kept — as the default** | Zero ops, auditable, in-context; the 2026 production trend (Claude Code, Cursor, Devin are all text-first). It is layer 2, not a rejected option — Hindsight must *earn* its ops cost (Docker/Postgres, LLM call per retain), which is why the pack is optional and file memory stays authoritative for procedural knowledge | + +Caveat recorded: many "Hindsight is #1" comparisons are published by vectorize.io itself; +the independent signals are the license, activity, SDK fit, and the Hermes adoption. + +## §3 Bank topology: one shared bank, mandatory tags (owner instinct confirmed) + +**Verified Hindsight facts (best-practices docs, 2026-06-11):** banks are fully isolated; +**all operations (retain/recall/reflect) target a single bank — cross-bank search is not +supported**; **tags are the filtering mechanism** (metadata is explicitly NOT filterable); +strict modes (`any_strict`/`all_strict`) exist to prevent cross-scope leakage; their docs +endorse "a shared bank with tags" as a standard pattern; banks carry mission/directives/ +disposition (bank-level reasoning identity). + +**Decision: one shared `bobbit` bank for the whole installation, tag-scoped.** + +- Every retain auto-tagged by the pack from session context — agents never hand-tag: + `project:<id>`, `agent:<role>`, `goal:<id>`, `kind:<decision|gotcha|preference|outcome>`. + Worktrees get **no** tag dimension (ephemeral; their learnings belong to the project). +- Default recall = strict-scoped to `project:<current>` (+ untagged/org-wide). Explicit + wider recall = the tool's `scope: project | global | all` arg mapped to tag filters — + so "have we solved this anywhere before?" is **one native query**, which is the decisive + argument: with per-project banks that query is impossible (no cross-bank search) short of + pack-side fan-out across N banks with hand-rolled merging, and reflect could never span + projects at all. +- One bank also means **one entity graph** (the same staff member, library, or convention + resolves to one entity instead of N copies) and one mission/disposition to configure. +- **Costs, with mitigations:** (a) reflect/mental-models consolidate bank-wide, so beliefs + can blend projects — mitigate with `observation_scopes` and project-scoped recall + defaults; (b) project offboarding needs **delete-by-tag** — verify against the targeted + Hindsight version at G2 implementation (checkpoint added to the EP plan's G2 notes; if + absent, fall back to per-project banks for deletion-sensitive installs); (c) multi-tenant + leakage — not our shape (single-operator installations), but strict tag matching is on + by default anyway. +- **Escape hatch kept:** isolation-critical projects (client work under NDA) may opt into + a dedicated bank via project config, accepting invisibility to `scope: all`. Default + remains shared. + +This **changes EP G2's earlier sketch** ("per-project banks", tool arg +`bank: current | global | all`): the manifest description and provider flow now say +*shared tag-scoped bank*, and the tool arg becomes `scope:` backed by tag filters, not +bank switching. Updated in [extension-platform.md](extension-platform.md) (manifest example, +Hindsight pack section). + +## §4 Interaction with the other memory-adjacent plans + +- **EP G1.6 session-memory provider** (FlexSearch over transcripts) is unaffected — it + searches layer 1 and works with zero infrastructure; it is also the graceful degradation + when the Hindsight pack is disabled. +- **GA-R5 profile memory** writes to layer 2 (markdown), optionally mirrored into Hindsight + as `kind:preference` retains when the pack is on. +- **CI repo map / code intelligence** is spatial, not temporal — no overlap (layer table in + [code-intelligence.md §1](code-intelligence.md)). diff --git a/docs/design/code-intelligence-alternatives.md b/docs/design/code-intelligence-alternatives.md index 4f63e4344..9c90c0734 100644 --- a/docs/design/code-intelligence-alternatives.md +++ b/docs/design/code-intelligence-alternatives.md @@ -192,7 +192,10 @@ Letta (a competing harness). **But** the 2026 production trend is text-first (Cl CLAUDE.md/auto-memory, Devin Knowledge, Cursor memories) — so: **markdown staff memory stays authoritative** for procedural knowledge; Hindsight is the *optional* episodic-memory pack, gated on Docker/Postgres availability, an LLM key per `retain` priced in. This nuance is -recorded against EP G2 here rather than rewriting the EP doc. +recorded against EP G2 here rather than rewriting the EP doc. **Superseded for depth:** the +full memory decision record — sessions-vs-Hindsight layering, the comparison table with +reasons, and the bank-topology decision (one shared tag-scoped bank) — now lives in +[agent-memory.md](agent-memory.md). ## §8 Net changes applied to the CI docs after this research diff --git a/docs/design/extension-platform-implementation-plan.md b/docs/design/extension-platform-implementation-plan.md index 3fc5c90ec..e8fd28071 100644 --- a/docs/design/extension-platform-implementation-plan.md +++ b/docs/design/extension-platform-implementation-plan.md @@ -498,6 +498,13 @@ panel UI. > Implementer note: verify the real Hindsight REST paths against the targeted Hindsight > release BEFORE coding the client; the contract below is Bobbit's internal client interface — > the stub mirrors THIS contract, and `hindsight-client` adapts it to upstream paths. +> +> Bank topology is decided: **one shared tag-scoped bank**, tools take `scope:` mapped to +> tag filters (NOT `bank:` switching) — rationale + auto-tag taxonomy in +> [agent-memory.md §3](agent-memory.md). Two upstream checks at this goal: (1) confirm +> tag-filtered recall with strict matching via REST/SDK; (2) confirm **delete-by-tag** +> exists for project offboarding — if absent, record the per-project-bank fallback for +> deletion-sensitive installs in the pack README. ### G2.1 REST client + stub-server harness *(parallel with all of G1)* diff --git a/docs/design/extension-platform.md b/docs/design/extension-platform.md index dd4ca7931..5a6f30847 100644 --- a/docs/design/extension-platform.md +++ b/docs/design/extension-platform.md @@ -136,7 +136,7 @@ rest is ignored). v2 servers warn when `schema` is newer than supported. ```yaml schema: 2 name: hindsight -description: Persistent agent memory backed by Hindsight (recall/retain/reflect, per-project banks). +description: Persistent agent memory backed by Hindsight (recall/retain/reflect, shared tag-scoped bank — see docs/design/agent-memory.md). version: 1.0.0 contents: roles: [] @@ -472,12 +472,16 @@ market-packs/hindsight/ ``` - **Provider flow**: `sessionSetup` → recall vs goal/task spec → "Relevant memory" section; - `beforePrompt` → recall vs user prompt (parallel project + global banks, 1500ms timeout ⇒ - skip); `afterTurn` → **async** retain of the turn (failures queue in the pack store, retried - next turn); `beforeCompact` → synchronous retain of salient facts before context loss; - `sessionShutdown` → flush. + `beforePrompt` → recall vs user prompt (project-scoped + org-wide tag filters in one + query, 1500ms timeout ⇒ skip); `afterTurn` → **async** retain of the turn, auto-tagged + `project:/agent:/goal:/kind:` from session context (failures queue in the pack store, + retried next turn); `beforeCompact` → synchronous retain of salient facts before context + loss; `sessionShutdown` → flush. - **Tools**: explicit `hindsight_recall` / `hindsight_reflect` / `hindsight_retain` for the - model, with `bank: current | global | all`. (Hindsight's own MCP server remains usable via + model, with `scope: project | global | all` mapped to tag filters on the shared bank — + NOT bank switching; Hindsight has no cross-bank search, which is why the topology is one + shared tag-scoped bank (decision + verified facts: + [agent-memory.md §3](agent-memory.md)). (Hindsight's own MCP server remains usable via normal MCP discovery, orthogonally.) - **Panel** (native, same-origin): status + mode setup (managed vs external), memory search, recent retains, retain-queue/operations view, runtime health/logs links, settings (model for diff --git a/docs/design/fable-program-execution-plan.md b/docs/design/fable-program-execution-plan.md index 4af404974..ba740f95f 100644 --- a/docs/design/fable-program-execution-plan.md +++ b/docs/design/fable-program-execution-plan.md @@ -206,6 +206,7 @@ T8 (S, independent) → T9 (S, independent). T2/T3/T5/T7 gate PB-P2 (§1.4). - **AI**: [ ] P1 · [ ] P2a · [ ] P2b · [ ] P3a · [ ] P3b · [ ] P4 · [ ] P5 · [ ] P6 - **CI**: [ ] CI-1 · [ ] CI-2 · [ ] CI-3 · [ ] CI-4 · [ ] CI-5 · [ ] CI-6 · [ ] CI-7 · language-pack cards: [ ] lsp-go · [ ] lsp-rust · [ ] lsp-csharp · [ ] lsp-fsharp · [ ] lsp-jvm · [ ] lsp-clangd · [ ] lsp-kotlin +- **Cards** (post-dependency, owner-accepted): [ ] Caretaker transcript-retention consolidation pass (distill→prune; after EP G2 + MC P3; [agent-memory.md §1](agent-memory.md)) ## §5 Prerequisite fix (before any lane) From 7f18a8ed6712e2a420f464da57b20cac8174fa79 Mon Sep 17 00:00:00 2001 From: Claude <noreply@anthropic.com> Date: Thu, 11 Jun 2026 14:18:25 +0000 Subject: [PATCH 13/18] =?UTF-8?q?Execution=20plan:=20plain-English=20narra?= =?UTF-8?q?tives=20under=20both=20diagrams;=20CI=20lane=20in=20=C2=A72=20g?= =?UTF-8?q?raph?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - §2 dependency graph: add the missing CI lane nodes (ast tools/diagnostics → LSP supervisor → packs/map/chip → capability swap, dotted edge to EP G8) and a prose translation naming every goal: the six day-one-parallel lanes, the sidebar-before-performance coupling, the Mission Control chain, the autoimprovement ladder, the CI lane, and the two soft couplings. - §2.1 waves: prose walkthrough of waves 0-5 in goal names rather than IDs — what each wave does, what runs in parallel inside it, and why each cross-wave gate exists (BENCH, SN-T5 rebase, EP G8 shim swap). https://claude.ai/code/session_01LBW68Z2KvA3EhQMq3c2wrR --- docs/design/fable-program-execution-plan.md | 83 +++++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/docs/design/fable-program-execution-plan.md b/docs/design/fable-program-execution-plan.md index ba740f95f..c67a839d4 100644 --- a/docs/design/fable-program-execution-plan.md +++ b/docs/design/fable-program-execution-plan.md @@ -82,10 +82,46 @@ graph TD AI1 --> AI2[AI-P2a/b Improver] --> AI3[AI-P3a/b curator+dreaming] MC4 --> AI2; MC3 --> AI5 AI2 --> AI4[AI-P4 shadow] --> AI5[AI-P5 autonomy] --> AI6[AI-P6 measurement] + CI12[CI-1 ast tools, CI-2 diagnostics] --> CI3[CI-3 LSP supervisor] --> CI456[CI-4 lang packs, CI-5 repo map, CI-6 services chip] + CI456 --> CI7[CI-7 capability swap + BENCH] + EP -.G8 capabilities.- CI7 GA6[GA-R6 importers] --> EP GA5[GA-R5 user profile] -.coordinate.- EP ``` +**The same graph in plain English.** Read "·" as "can run at the same time". + +- **Six lanes need nothing from anyone and can all start on day one, in parallel:** + the comms-stack reliability backlog (starting with making the gateway's message-queue + drain un-crashable, then respawn fencing, steer delivery, and the rest of its own + ordered waves); the sidebar-nesting cleanup (pin the pure tree builders with tests + first — that's the safety net for everything after); the cost-efficiency lane (per-turn + usage ledger first, because every later claim is measured against it); the extension + platform (manifest schema 2 → lifecycle hub → prompt-section providers, in order); + the client-performance lane (measurement harness first, then animation gating); and + the two small gap wins (one-shot/`gate_failed` triggers, standing-orders docs). +- **Sidebar before performance-on-the-sidebar.** The performance lane's session-poll + demotion and render work wait for the sidebar's concurrency hardening + (refresh-dedup, archived freshness), and the streaming render throttle explicitly + rebases on the big nesting render unification. Battery saver comes last, after all of + it soaks behind a flag. +- **Mission Control is a strict chain:** legitimize the global scope → Mission Control + sessions + sidebar entry → then two parallel branches (the `bobbit` meta-tool; the + flight recorder) → which rejoin for the mission-control pack + system-staff crew (this + also wants the triggers win, so staff can be woken by events) → briefing, onboarding, + and budgets close it out. +- **Autoimprovement rides on Mission Control:** the proposal substrate needs the global + scope and triggers; the Improver needs the meta-tool + crew; then curator/dreaming, + shadow calibration, graduated autonomy (which also wants the flight recorder for its + evidence trail), and the measurement loop — strictly in that order. +- **Code intelligence is its own lane:** ast-grep tools and structured diagnostics first + (independent of everything); then the LSP supervisor; then language packs, the repo + map, and the services chip in parallel; the final capability-swap goal waits on the + extension platform's capability registry (a local shim covers the gap). +- **Two soft couplings:** the Hermes/OpenClaw skill importers feed the extension + platform's pack format, and the user-profile memory win should coordinate with (not + block on) the platform's session-memory pack. + **Day-1 starter set** (parallel, no collisions): AGENTS.md trim (§5) · CS-R1 · SN-T1 · PB-P0 · CE-G0.1 · EP G1.1 · GA-R2 · GA-R3. @@ -127,6 +163,53 @@ flowchart TD W0 --> W1 --> W2 --> W3 --> W4 --> W5 ``` +**The same waves in plain English** — what each wave actually does, and what runs in +parallel inside it. Everything listed within one wave can be in flight simultaneously +(different people/sessions) as long as no two touch the same §1.4 seam. + +- **Wave 0 — hours, not days.** Trim AGENTS.md and get the test suite green. Nothing else + starts until the suite is trustworthy, because every later goal proves itself with tests. +- **Wave 1 — foundations, seven things at once.** Fix the gateway crash (comms); pin the + sidebar's pure tree builders with real-source tests; stand up the performance + measurement harness; stand up the per-turn cost ledger; land manifest schema 2 for + packs; ship the one-shot/`gate_failed` triggers; write the standing-orders convention. + None of these share files; all seven can merge in any order. +- **Wave 2 — hygiene and scope.** Comms does its harness-fidelity work and then respawn + fencing + retry/compaction gating; sidebar hardens `refreshSessions` concurrency, then + archived-freshness and the goal-graph module; performance gates ambient animations and + rewrites paint properties; cost-efficiency adds the cost lens UI, the benchmark harness + (the BENCH gate everything risky waits for), and the SDK upgrade; the extension + platform builds the lifecycle hub and the sessionSetup → "Dynamic Context" section; + Mission Control legitimizes the global scope and gets its sidebar entry; `bobbit + doctor` and the skill importers can land any time here. +- **Wave 3 — the spines (the heaviest wave).** Comms lands steer exactly-once delivery, + idle-drain, and ledger durability; sidebar does the big one — unifying nesting into a + single render path — plus the role-picker and cleanup bundles; performance demotes + session polling and scopes the verification tick (now safe, because the sidebar + concurrency work merged in wave 2); cost-efficiency starts its post-upgrade lanes + (tool-output budgets, discovery guidance); the extension platform wires per-turn hooks, + market UI, and the session-memory pack; Mission Control ships the `bobbit` meta-tool + and the flight recorder; autoimprovement lays its proposal substrate; **code + intelligence starts here** with the ast-grep tools and structured diagnostics — both + independent, both immediately useful to every coding agent. +- **Wave 4 — features on the spines.** Sidebar finishes mobile reuse and keyed lists; + performance lands the streaming render throttle (rebased on the wave-3 render + unification) and then battery saver; comms clears its client-side duplicate/loss + backlog; the extension platform ships the Hindsight memory pack, then managed runtimes; + Mission Control assembles the pack + system-staff crew; autoimprovement turns on the + human-in-the-loop Improver; the "while you were away" summary and user-profile memory + land; code intelligence builds the LSP supervisor (TypeScript + Python first), then + the ranked repo map on top of the wave-3 ast tooling. +- **Wave 5 — the loop closes.** Autoimprovement runs curator → dreaming → shadow + calibration → graduated autonomy → the measurement loop, strictly in that order; + Mission Control finishes briefing/onboarding/budgets; the extension platform completes + its ecosystem (memory depth, MCP-as-pack, hooks, the Claude-plugin adapter, + capabilities + selectors, workflow templates); cost-efficiency executes its remaining + BENCH-gated diets (the team-lead role rewrite chief among them); code intelligence + fans out language packs (Go, Rust, C#, F#, JVM, clangd), ships the services chip + + graphify visualization pack, and swaps its local capability shim for the platform + registry, validating the whole workstream on BENCH. + Milestones the waves deliver (what the user can feel): | After | The product visibly gains | From 2d403fb04b52a22183b8106657145145165a8f04 Mon Sep 17 00:00:00 2001 From: Claude <noreply@anthropic.com> Date: Thu, 11 Jun 2026 14:26:41 +0000 Subject: [PATCH 14/18] =?UTF-8?q?MC=20doc:=20fix=20global-scope=20cwd=20in?= =?UTF-8?q?consistency=20(=C2=A72=20prose=20vs=20A.1=20spec)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §2 said the global-session cwd is "~/.bobbit/mission-control/" while A.1 (authoritative storage layout) defines missionControlDir() = path.join(stateDir, "mission-control"), and bobbit-dir.ts resolves the state dir to BOBBIT_DIR env → else <gateway root>/.bobbit — home only when the env points there. §2 now states the A.1 path and resolution rule. https://claude.ai/code/session_01LBW68Z2KvA3EhQMq3c2wrR --- docs/design/mission-control.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/design/mission-control.md b/docs/design/mission-control.md index 1a98c5cd6..cae42be5d 100644 --- a/docs/design/mission-control.md +++ b/docs/design/mission-control.md @@ -59,8 +59,11 @@ Control promotes it to a first-class, still-synthetic, un-deletable scope: no archive, no delete. - **Sessions:** global sessions attach to the system project. Projectless session handling already exists (`tests/e2e/sessions-projectless.spec.ts` pins current behavior — extend, - don't fork). **Cwd:** a dedicated `~/.bobbit/mission-control/` workspace dir (a real, - writable directory so file tools work), never a project root and never + don't fork). **Cwd:** a dedicated `mission-control/` workspace dir under the gateway's + state dir — `missionControlDir()` per A.1, i.e. `<bobbit-dir>/state/mission-control/`, + where `bobbitDir()` resolves `BOBBIT_DIR` env → else `<gateway root>/.bobbit` + (`src/server/bobbit-dir.ts`); it is `~/.bobbit/…` only when `BOBBIT_DIR` points at home. + A real, writable directory so file tools work; never a project root and never `config.defaultCwd`. - **Staff:** `staff-store.ts` / `staff-manager.ts` accept `projectId: "system"` as a *valid* owner: project-anchoring validation (staff-agents.md §Project and cwd anchoring) gains an @@ -207,7 +210,7 @@ graph LR 1. `project-registry.ts`: keep `SYSTEM_PROJECT_ID` hidden from `visibleProjects()`; add `isSystemScope(id)` helper; ensure the system project's anchor dir is the - `~/.bobbit/mission-control/` workspace (created lazily). + `missionControlDir()` workspace under the state dir (A.1; created lazily). 2. `staff-store.ts`: add `global?: boolean` to `PersistedStaff` (loader normalises missing → `false`); `staff-manager.ts` accepts `projectId: "system"` + `global: true` with system-scope cwd validation (inside the mission-control workspace) and worktree forced From 6b323aa438e523446f4642b0e711f04a98528c2c Mon Sep 17 00:00:00 2001 From: Artur Jonkisz <ajonkisz@gmail.com> Date: Fri, 19 Jun 2026 21:18:25 +0100 Subject: [PATCH 15/18] CE doc: add latency axis + process-per-agent root cause; rename to time-and-token-cost-efficiency MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Renames token-cost-efficiency.md -> time-and-token-cost-efficiency.md (updates all 6 cross-refs) and adds the wall-clock/time dimension the audit lacked: - New §9: latency axis + the inherent process-per-agent root cause. Measured (2026-06-19, 326 sessions): CC runs LARGER contexts per request (377k median) yet turns in ~4.4s; Bobbit's slowness is structural, not per-request. 1.34 verifiers per work session, median 2.8min, serialized on the critical path. - Root cause: every agent/verifier is a separate OS process (rpc-bridge spawn + MCP stdio + optional docker exec) that cold-writes its resident stack and can't inherit the lead's warm cache — vs claude-code's in-process forks + omitClaudeMd. - F5 refinement: verifier cache economics are INVERTED — write-bound (34% of cost is cache writes) not read-bound; re-scopes CE-G3.3. - New findings F13-F17 and CE-G8 latency implementation workstream (risk-proportional gates, slim sub-agent context, warm-cache reuse, latency instrumentation, solo-fast). - CE-G2.1: Hermes exact budget constants + the read_file=inf persist-loop gotcha. Co-authored-by: bobbit-ai <bobbit@bobbit.ai> --- ...token-cost-efficiency.md => time-and-token-cost-efficiency.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/design/{token-cost-efficiency.md => time-and-token-cost-efficiency.md} (100%) diff --git a/docs/design/token-cost-efficiency.md b/docs/design/time-and-token-cost-efficiency.md similarity index 100% rename from docs/design/token-cost-efficiency.md rename to docs/design/time-and-token-cost-efficiency.md From 9ec1a2859489407988a6472d0e6d72ea5d559ac8 Mon Sep 17 00:00:00 2001 From: Artur Jonkisz <ajonkisz@gmail.com> Date: Fri, 19 Jun 2026 21:18:46 +0100 Subject: [PATCH 16/18] CE doc: latency axis + process-per-agent root cause; cross-ref updates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content for the rename in the previous commit (only the bare mv landed there): - New §9: latency axis + inherent process-per-agent root cause. Measured 2026-06-19 (326 sessions): CC runs LARGER contexts per request (377k median) yet turns in ~4.4s; Bobbit slowness is structural. 1.34 verifiers per work session, median 2.8min, serialized on the critical path before team_complete. - Root cause: every agent/verifier is a separate OS process (rpc-bridge spawn + MCP stdio + optional docker exec) that cold-writes its resident stack and cannot inherit the lead's warm cache — vs claude-code in-process forks + omitClaudeMd. - F5 refinement: verifier cache economics INVERTED — write-bound (34% of cost is cache writes) not read-bound; re-scopes CE-G3.3. - F13-F17 + CE-G8 latency workstream (risk-proportional gates, slim sub-agent context, warm-cache reuse, latency instrumentation, solo-fast mode). - CE-G2.1: Hermes exact budget constants + read_file=inf persist-loop gotcha. - Update 6 cross-referencing docs + fable execution-plan CE row. Co-authored-by: bobbit-ai <bobbit@bobbit.ai> --- docs/design/autoimprovement.md | 2 +- docs/design/client-performance-battery.md | 4 +- docs/design/code-intelligence.md | 4 +- docs/design/fable-program-execution-plan.md | 2 +- docs/design/harness-gap-analysis.md | 4 +- docs/design/mission-control.md | 2 +- docs/design/time-and-token-cost-efficiency.md | 236 +++++++++++++++++- 7 files changed, 241 insertions(+), 13 deletions(-) diff --git a/docs/design/autoimprovement.md b/docs/design/autoimprovement.md index 57db21621..6c25d36de 100644 --- a/docs/design/autoimprovement.md +++ b/docs/design/autoimprovement.md @@ -309,7 +309,7 @@ recorded; regression triggers demotion + revert proposal. Acceptance: planted re Cross-references: [mission-control.md](mission-control.md) (Improver/Archivist staff, flight recorder, inbox digests) · [harness-gap-analysis.md](harness-gap-analysis.md) G1/G2/G11 · -[token-cost-efficiency.md](token-cost-efficiency.md) (cost ledger CE-G0.1, aux summarizer) · +[time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) (cost ledger CE-G0.1, aux summarizer) · [skill-ux-and-autonomous-activation.md](skill-ux-and-autonomous-activation.md) · [editable-proposals.md](editable-proposals.md) · [human-signoff-gates.md](human-signoff-gates.md). Execution tracking: [fable-program-execution-plan.md](fable-program-execution-plan.md). diff --git a/docs/design/client-performance-battery.md b/docs/design/client-performance-battery.md index eeb774455..5b126d8fe 100644 --- a/docs/design/client-performance-battery.md +++ b/docs/design/client-performance-battery.md @@ -24,7 +24,7 @@ streaming delta schedules a full root re-render (rAF-coalesced, so up to ~60 ful tree evaluations per second during streaming). None of these is individually fatal; together they mean the browser never reaches its power-saving render-idle state. The plan below is **instrumentation-first** (same discipline as -[token-cost-efficiency.md](token-cost-efficiency.md)): we build a small repeatable measurement +[time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md)): we build a small repeatable measurement harness, capture a baseline, then land tiered fixes — each behind a [`perf-flags.ts`](../../src/app/perf-flags.ts)-style kill switch and re-measured against the baseline. @@ -316,7 +316,7 @@ cadence ~10 Hz; all existing unit + e2e phases green. - **A WebGL/canvas renderer rewrite** — not justified by any current evidence; revisit only if P3.2 data demands it. -Cross-references: [token-cost-efficiency.md](token-cost-efficiency.md) (the sibling +Cross-references: [time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) (the sibling "instrumentation-first" effort, server/cost side), [mission-control.md](mission-control.md) (the Observer staff can watch for perf regressions once the harness exists), [harness-gap-analysis.md](harness-gap-analysis.md) §battery for how peer harnesses avoid this diff --git a/docs/design/code-intelligence.md b/docs/design/code-intelligence.md index b87b10342..1c79b3876 100644 --- a/docs/design/code-intelligence.md +++ b/docs/design/code-intelligence.md @@ -11,7 +11,7 @@ Status: design accepted, not started. Workstream **CI** in > swapping any component. **The problem:** Bobbit agents are "an LLM with grep". 56% of tool calls are bash -(token-cost-efficiency.md §3), discovery is multi-round grep cascades, edits are text +(time-and-token-cost-efficiency.md §3), discovery is multi-round grep cascades, edits are text patches whose correctness is only discovered at gate time, and nothing understands symbols, types, or structure. Peers are converging on the same answer (Serena, Octocode, Aider's repo-map, graphify): pair fast lexical search with **structural (AST), semantic (LSP), @@ -180,7 +180,7 @@ artifacts for absolute paths — the graphify lesson, enforced. Cross-references: [code-intelligence-alternatives.md](code-intelligence-alternatives.md) (alternatives, citations, research record), -[token-cost-efficiency.md](token-cost-efficiency.md) (CE-G2/G7), +[time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) (CE-G2/G7), [extension-platform.md](extension-platform.md) (capabilities §3.1, providers, runtimes), [harness-gap-analysis.md](harness-gap-analysis.md) (peer evidence), [mission-control.md](mission-control.md) (Caretaker sweep, flight recorder for service diff --git a/docs/design/fable-program-execution-plan.md b/docs/design/fable-program-execution-plan.md index c67a839d4..ac882ddd5 100644 --- a/docs/design/fable-program-execution-plan.md +++ b/docs/design/fable-program-execution-plan.md @@ -10,7 +10,7 @@ The workstreams (each doc owns its *content* — tasks, contracts, acceptance; t | Workstream | Design (WHAT/WHY) | Execution authority (HOW) | |---|---|---| | **EP** extension platform | [extension-platform.md](extension-platform.md) | [extension-platform-implementation-plan.md](extension-platform-implementation-plan.md) (G1.1–G9) | -| **CE** token/cost efficiency | [token-cost-efficiency.md](token-cost-efficiency.md) §1–§5 | same doc, §6 (CE-G0.1–G7.2, BENCH gates) | +| **CE** time & token-cost efficiency | [time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) §1–§5, **§9** (latency + arch root cause) | same doc, §6 (CE-G0.1–G7.2) + **§9.5 (CE-G8 latency)**, BENCH gates | | **CS** comms-stack reliability | [comms-stack/01–03](comms-stack/01-understanding.md) | [comms-stack/04-current-state-and-backlog.md](comms-stack/04-current-state-and-backlog.md) §6–§7 (waves + merge map) | | **SN** sidebar & goal-nesting | [sidebar-goal-nesting-audit.md](sidebar-goal-nesting-audit.md) §1–§3 | same doc, §4 (T1–T9) | | **PB** client perf/battery | [client-performance-battery.md](client-performance-battery.md) | [client-performance-battery-implementation-plan.md](client-performance-battery-implementation-plan.md) | diff --git a/docs/design/harness-gap-analysis.md b/docs/design/harness-gap-analysis.md index 64b89af1a..7bfb1435c 100644 --- a/docs/design/harness-gap-analysis.md +++ b/docs/design/harness-gap-analysis.md @@ -2,7 +2,7 @@ Status: research complete; roadmap not started. Feature/UX gaps only — **cost/token efficiency is explicitly out of scope** here because -[token-cost-efficiency.md](token-cost-efficiency.md) §4 already covers the Claude Code / +[time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) §4 already covers the Claude Code / Hermes comparison on that axis. > **Execution authority** for the §5 quick wins (R2–R6, R9): @@ -106,7 +106,7 @@ Disposition: the *retrieval* half is the extension platform's `session-memory` r profile**: a global `USER.md`-equivalent (server-scoped, ~500 tokens, agent-managed via a small tool, injected as a stable prompt section through the existing `PromptParts` assembly in `src/server/agent/system-prompt.ts`). Frozen-snapshot semantics preserve the cache prefix -(matches Bobbit's existing cache discipline per token-cost-efficiency §1). The *writing* half +(matches Bobbit's existing cache discipline per time-and-token-cost-efficiency §1). The *writing* half (when to save) belongs to the autoimprovement loop (→ autoimprovement.md §loop, "observe"). ### G2 — Self-improving skills (HIGH value; the subject of its own doc) diff --git a/docs/design/mission-control.md b/docs/design/mission-control.md index cae42be5d..09b5556a5 100644 --- a/docs/design/mission-control.md +++ b/docs/design/mission-control.md @@ -322,7 +322,7 @@ Cross-references: [autoimprovement.md](autoimprovement.md) · [harness-gap-analysis.md](harness-gap-analysis.md) (G3/G4/G5/G6/G10) · [client-performance-battery.md](client-performance-battery.md) (Observer watches the perf baseline once the harness exists) · [extension-platform.md](extension-platform.md) · -[token-cost-efficiency.md](token-cost-efficiency.md) (budgets, aux models) · +[time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) (budgets, aux models) · `docs/staff-agents.md` · `docs/staff-triggers.md` · `docs/staff-inbox.md` · `docs/mcp-meta-tools.md` · `docs/rest-api.md`. Execution tracking: [fable-program-execution-plan.md](fable-program-execution-plan.md). diff --git a/docs/design/time-and-token-cost-efficiency.md b/docs/design/time-and-token-cost-efficiency.md index 607f413bc..45b717445 100644 --- a/docs/design/time-and-token-cost-efficiency.md +++ b/docs/design/time-and-token-cost-efficiency.md @@ -1,15 +1,17 @@ -# Token & Cost Efficiency — audit, findings, and remediation plan +# Time & Token-Cost Efficiency — audit, findings, and remediation plan Status: **investigation complete, remediation not started** · Audited: 2026-06-10 against -master @ `4e7fce1f` and live state on the primary dev machine. Workstream **CE** in +master @ `4e7fce1f`; **latency axis + architecture root-cause added 2026-06-19** (§9, from a +second 326-session snapshot — shapes corroborate §1). Workstream **CE** in [fable-program-execution-plan.md](fable-program-execution-plan.md) (program sequencing + master checklist). Companion docs: [extension-platform-implementation-plan.md](extension-platform-implementation-plan.md) (several goals here ride its pack surfaces), [comms-stack/04-current-state-and-backlog.md](comms-stack/04-current-state-and-backlog.md) (overlapping retry findings). -**The question this answers:** Bobbit *feels* more expensive per unit of work than Claude Code -or hermes-agent. Is it, why, and what do we change? +**The question this answers:** Bobbit *feels* more expensive **and slower** per unit of work +than Claude Code or hermes-agent. Is it, why, and what do we change? (Cost: §1–§7. Wall-clock +latency and the inherent architectural root cause: **§9**.) **The one-paragraph answer:** Prompt caching is working (only 0.008% of input tokens were uncached across 246 sessions), so the spend is not a caching bug. The spend is @@ -218,6 +220,23 @@ bounded retry attempts (`verification-harness.ts:~2550`). Each reviewer re-inges resident stack even when the reviewed artifact is a small diff. No diff-scoped context, no cheap-model default, no cap on tool-call rounds. +**Refinement (2026-06-19): verifier cache economics are *inverted* — write-bound, not +read-bound.** §1.1's headline ("the bill is cache reads") is a global average that does **not** +hold for the `llm-review` class. Measured on the 2026-06-19 snapshot (189 verifier sessions), +cache *writes* are 40–48% of each verifier's (write+read) tokens, vs a 3% global write/read +ratio. Converting to base-input-equivalent units (write ×1.25, read ×0.1, fresh ×1.0): + +| Class | write share | read share | fresh share | +|---|---|---|---| +| Verifiers (`llm-review`, n=189) | **34%** | 51% | 15% | +| Work sessions (n=144) | 23% | 68% | 9% | + +The cause is structural (see §9): each verifier is a **separate, short-lived OS process** that +writes the full resident stack *cold* and dies before the cache write amortises — it can never +inherit the lead's warm cache. **Remediation implication:** for verifiers the dominant lever is +cold-**write** bytes (shrink the resident context per spawn — CE-G8.2) and/or warm-cache reuse +(CE-G8.4), **not** request-count reduction as CE-G3.3 frames it. Re-scope CE-G3.3 accordingly. + ### F6 — Images as base64 in context (MED overall, EXTREME per-incident, difficulty S) One `generate_image` result = 2.4–2.7MB of base64 in the transcript and in context; 73% of a $10.75 session was image bytes. Results should be written to disk and returned by reference @@ -321,6 +340,10 @@ Hand-off format matches `extension-platform-implementation-plan.md`: each sub-go independently mergeable, master-green, with tests and acceptance criteria. Lanes: G0 and G1 are parallel day-one lanes; G2/G3 depend on G1.0's re-verification; G4/G5 are BENCH-GATED on G0.3. +**Latency workstream (CE-G8)** lives in §9.5 — it targets *wall-clock*, not just dollars +(risk-proportional gates, slim sub-agent context, warm-cache reuse, latency instrumentation, +"solo fast" mode). CE-G8.1/G8.3 are the quick wins and can start day-one alongside G0/G1. + **Conflict hotspots:** `session-manager.ts` and `session-setup.ts` are also touched by the comms-stack backlog (CS-R*) — coordinate merge order; `verification-harness.ts` is touched by CE-G3.3 and CE-G5.2 (sequence them). @@ -418,6 +441,14 @@ Adopt Hermes' model, scoped to whatever 0.79.x doesn't already do: - Implementation seam: prefer the SDK layer (fork patch — it owns tool execution) with a Bobbit-side fallback in the tool wrappers; decide per CE-G1.0's report. Spill files are session-scoped and cleaned with session deletion. +- **Hermes' exact constants** (source: `hermes-agent/tools/budget_config.py`, verified + 2026-06-19) — adopt as starting defaults: `DEFAULT_RESULT_SIZE_CHARS = 100_000` (per-result), + `DEFAULT_TURN_BUDGET_CHARS = 200_000` (per-turn aggregate), `DEFAULT_PREVIEW_SIZE_CHARS = 1_500` + (inline snippet after spill). Resolution order: `pinned > tool_overrides > registry per-tool > + default`. **Non-obvious gotcha to copy:** Hermes *pins* `read_file = inf` in `PINNED_THRESHOLDS` + — a read tool that itself spills creates an infinite persist→read→persist loop, so the read path + must be exempt from the budget (caps belong on `read` via offset/limit guidance — CE-G7.1 — not + via spill). - **Tests (write RED first):** unit — cap/preview/spill matrix per layer, path cleanup; E2E — oversized fixture command produces preview + readable spill file; pin that previews always carry the path (no silent truncation). @@ -615,3 +646,200 @@ jq -s '[.[] | select(.type=="message" and .message.role=="toolResult") | 6. **Data drift** — every number in §1 is a 2026-06-10 snapshot of one machine spanning multiple projects and weeks of model-price changes; CE-G0 exists so this document never needs to be hand-reproduced again. + +--- + +## §9 Latency axis & the process-per-agent root cause *(added 2026-06-19)* + +§1–§8 optimise **dollars**. The owner's lived complaint is also **wall-clock**: "goals take a +*really* long time; Claude Code felt a lot faster." Cost and time are correlated but not the +same — a $1 verifier still makes you wait three minutes — so latency needs its own axis, and the +investigation surfaced an **inherent architectural root cause** that several §3 symptoms +(F2 resident floor, F5 verifier economics) trace back to. All numbers here are a 2026-06-19 +snapshot of `.bobbit/state` (326 sessions, $1,069, 1.16B cache-read) on the primary dev machine; +the *shapes* corroborate §1's 2026-06-10 snapshot. Repro recipes in §9.6. + +### 9.1 The surprise: per-request latency is *not* where Bobbit loses + +| Metric | Bobbit | Claude Code | Source | +|---|---|---|---| +| Context tokens per API request (median) | ~18k–55k resident + history | **377,590** | CC is *larger* and still fast | +| Per-API-round latency (median / mean / p90) | comparable per round | **4.4s / 15.1s / 26.5s** | CC `~/.claude/projects/*.jsonl` timestamps | + +Claude Code runs **bigger** contexts per request and still turns each round in ~4.4s median. So +Bobbit is **not** slow because individual requests are heavy. The wall-clock goes somewhere +structural — the orchestration model inserts machine work *between you and "done"* that, in +Claude Code, either doesn't happen or is you doing it instantly in real time. + +### 9.2 Where the wall-clock actually goes (measured) + +| Signal | Value | Why it's wall-clock you wait through | +|---|---|---| +| Verifier (`llm-review`) sessions vs work sessions | **189 vs 141 — 1.34 per unit of work** | Each gated step spawns a verifier that runs *after* the work, on the critical path | +| Verifier session duration | median **2.8 min**, max **9.9 min** (10h total, 175 traced) | `review-findings` blocks `team_complete`; serialized, not overlapped with work | +| Work session duration | median **20.5 min** | multi-turn, multi-tool | +| Per-spawn startup floor | process boot + N MCP stdio spawns + optional `docker exec` + role/AGENTS/tool-doc load | paid *before the first token*, every spawn | +| Nudge/backoff dead time | 5s worker-idle debounce, 5-min stuck-quiet, 30-min long-streaming suppression, exp backoff (cap 12h) | literal inserted sleeps in the coordination loop (`team-manager.ts`) | + +Claude Code has none of these: you are the orchestrator and reviewer, in real time, so there is +zero machine wall-clock between turns. **The slowness is the price of unattended autonomy + +automated quality gates** — the fix is making them *proportional to risk*, not removing them. + +### 9.3 Root cause: process-per-agent (inherent to the gateway architecture) + +Bobbit's deepest inherent tax is that **every agent — lead, worker, and each of the 189 +verifiers — is a separate OS process.** `rpc-bridge.ts` does +`spawn(process.execPath, [pi-cli, ...])` per session; each session connects its own **MCP stdio +child processes** (`mcp-client.ts::_connectStdio`); sandboxed goals add a `docker exec` into a +pool container on top (`rpc-bridge.ts:~658`). + +Claude Code does the opposite — sub-agents are **in-process forks** (`tools/AgentTool/runAgent.ts` +is an async generator; `utils/swarm/spawnInProcess.js`) that **inherit the parent's already-warm +context** via `forkContextMessages`, and it *deliberately* trims per-spawn context: the +read-only Explore/Plan agents drop CLAUDE.md — source comment: *"Dropping claudeMd here saves +~5–15 Gtok/week across 34M+ Explore spawns"* (`runAgent.ts:~387`). Hermes likewise runs tools +in-process against one resident context. + +This single choice explains the symptoms: +- **Verifier write-thrash (F5 refinement).** A separate process has its own prompt-cache + lifecycle, so it *cannot* inherit the lead's warm cache — it writes the full resident stack + **cold** and dies before the write amortises. Hence 34% of verifier cost is cache *writes* + (vs 3% global). +- **The per-agent latency floor (9.2).** Process boot + MCP stdio handshakes + optional + container exec + role/AGENTS/tool-doc load happen before the model emits a token, paid 189× + for verifiers alone. +- **Why the resident-prompt floor (F2) is so expensive.** It isn't just resident once — it's + re-written cold on every short-lived spawn. + +This is not a quick fix (it fights the gateway/session model), but naming it reframes the +priority order: **shrink what each cold spawn must write, and reuse warm cache where possible**, +before chasing request-count. + +### 9.4 New findings (continuing the §3 F-numbering) + +#### F13 — Verification on the critical path with no risk-proportionality (HIGH for latency, difficulty M) +`review-findings` is server-enforced before `team_complete`; a 2-line tooltip tweak triggers the +same full-context ~3-min verifier as a 500-line refactor. No diff-size threshold, no "skip gate +under N lines", no fast-path. This is the single biggest *wall-clock* tax. (Cost cousin: F5.) + +#### F14 — Cold-spawn write tax from process-per-agent (HIGH for cost+latency, difficulty L, **inherent**) +§9.3. Short-lived spawns (verifiers, small worker tasks) never amortise their cold cache write +and pay a process+MCP+container startup floor. Inherent to the architecture; mitigable by +context-diet (F15) now and warm-cache reuse (CE-G8.4) later. + +#### F15 — No slim sub-agent context profile (MED-HIGH, difficulty S–M) +Every spawned agent inherits the full AGENTS.md cascade + full role, including read-only +verifiers that never commit, lint, or open PRs and so cannot act on most of it. Claude Code's +`omitClaudeMd` for read-only sub-agents is the proven pattern. **This is the highest-leverage, +lowest-risk quick win** — attacks F5's 34% write share with zero behaviour risk. + +#### F16 — Coordination dead-time is invisible (MED for latency, difficulty S–M) +Nudge debounce/backoff and the per-spawn startup floor are pure wall-clock that **no token or +cost metric captures**, so CE-G0's ledger (dollars/tokens) would still not surface them. Latency +needs its own instrumentation dimension (wall-clock per turn, queue/startup/verify waits). + +#### F17 — No "solo fast" execution mode (MED, difficulty S) +Every goal pays the full team+gate orchestration even when the user would have just used a +single interactive agent. There is no lightweight single-agent, no-gate path for low-risk work — +so the cases most comparable to Claude Code pay the most overhead relative to their value. + +### 9.5 Remediation — latency workstream (CE-G8) + +Hand-off format matches §6. Lane independent of G0–G7 except where noted. **CE-G8.1 and CE-G8.3 +are the quick wins; do them first.** Behaviour-affecting items are BENCH-GATED on CE-G0.3 plus a +**wall-clock** comparison (extend the bench report with `wallClock`, `startupMs`, `verifyMs`). + +#### CE-G8.1 Risk-proportional verification gates (M) **[BENCH-GATED]** — *biggest latency win* +- Add a per-gate **fast-path**: skip or downgrade `review-findings` when the change under review + is below a configurable threshold (default ~30 changed lines, no non-doc/test files, + green `npm run check`) — config-overridable per project/workflow; never silently skip a gate a + workflow marks `required: strict`. +- Where not skipped, run a **lighter review** (cheap model + diff-scoped context, ties to + CE-G3.3/CE-G5.2) instead of the full verifier. +- **Tests:** unit — threshold/skip decision table (lines, file classes, strictness); E2E — a + sub-threshold gated step completes without spawning a full verifier; BENCH — review task + quality holds on the suite while median goal wall-clock drops. + +#### CE-G8.2 Verifier cold-write reduction (S–M) — *attacks F5's 34% write share* +- Pair with CE-G3.3/F15: verifiers spawn with a **diff-scoped, slim** context (artifact + + acceptance criteria + slim reviewer role), measured by cache-**write** tokens per verifier, + not just rounds. Target: halve verifier write share. +- **Tests:** unit — verifier context assembly excludes the AGENTS.md cascade by default; E2E — + spawned verifier's resident snapshot under a byte budget; **measure cacheWrite/verifier** in + the CE-G0.1 ledger before/after. + +#### CE-G8.3 Slim sub-agent / read-only context profile (S–M) — *quick win, zero behaviour risk* +- Adopt Claude Code's `omitClaudeMd` pattern: read-only roles (verifiers, explorers, reviewers) + get a context profile that **drops the project AGENTS.md cascade and commit/PR/lint rules** + they cannot act on. Resolution lives where role context is assembled (`session-setup.ts`). +- **Tests:** unit — profile resolution per role (read-only ⇒ no cascade); E2E — a read-only + spawn's prompt sections omit AGENTS.md; pin a per-profile resident-size budget (like the + tool-description budget) so it can't silently regrow. + +#### CE-G8.4 Warm-cache reuse for short-lived spawns (L, **inherent-architecture**, research-first) +- Investigate letting a verifier/sub-agent **reuse the parent's warm cache** instead of + cold-spawning a fresh process: options are (a) an in-process review path for non-sandboxed + goals, (b) a pooled warm agent process keyed by resident-prefix hash, (c) SDK/fork support for + prefix sharing across processes. Decide feasibility against the gateway model in a spike; + this is the real ceiling on F14 but must not regress sandbox isolation. +- **Tests:** spike + design note appended here before any code; if adopted, E2E that a reused + spawn records ~0 cold cache-write for the shared prefix. + +#### CE-G8.5 Latency instrumentation (S–M) — *extends CE-G0.1, closes F16* +- Add wall-clock dimensions to the CE-G0.1 ledger: per-turn `wallClockMs`, and per-spawn + `startupMs` (spawn→first-token), `queueMs` (enqueue→start), `verifyMs` (gate signal→verdict). + Surface a "time breakdown" alongside the cost lens (CE-G0.2): work vs coordination vs + verification vs startup. +- Extend the bench report (CE-G0.3) with `wallClock`, `startupMs`, `verifyMs` so CE-G8.1’s + BENCH gate can assert *time* not just dollars. +- **Tests:** unit — timing capture + attribution; API E2E — rollup math over a fixture ledger. + +#### CE-G8.6 "Solo fast" goal mode (S) — *closes F17* +- A goal option that runs a **single agent, no team, no gates** (or design-doc only) for + low-risk work — the Claude-Code-equivalent path — selectable at goal creation, with the + trade-off (no automated verification) made explicit in the UI. +- **Tests:** unit — mode disables team/gate machinery; E2E — a solo-fast goal completes without + spawning verifiers; browser E2E for the creation toggle + persisted choice. + +### Expected impact (latency — estimates; the bench harness replaces these) + +| Goal | Axis | Est. effect | +|---|---|---| +| CE-G8.1 | wall-clock | removes ~1.34× × ~3 min serialized verify from small goals — the dominant tax | +| CE-G8.2/G8.3 | cost+wall-clock | halves verifier cache-**write** share + cuts per-spawn load; zero behaviour risk (G8.3) | +| CE-G8.4 | cost+wall-clock | eliminates cold-write for reused prefixes (ceiling fix; highest risk) | +| CE-G8.5 | visibility | makes coordination/startup dead-time a dashboard number (blocker for tuning) | +| CE-G8.6 | wall-clock | low-risk goals skip orchestration entirely | + +### 9.6 Reproduction recipes (latency) + +```bash +# Verifier vs work session counts + cache-write share (the inverted economics) +python3 - <<'PY' +import json,statistics +d=json.load(open(".bobbit/state/session-costs.json")) +for lbl,pred in [("VERIFIER",lambda k:k.startswith("llm-review")),("WORK",lambda k:not k.startswith("llm-review"))]: + rows=[v for k,v in d.items() if pred(k)] + cw=sum(v.get("cacheWriteTokens",0) for v in rows); cr=sum(v.get("cacheReadTokens",0) for v in rows) + print(f"{lbl}: n={len(rows)} write/read={cw/max(1,cr):.3f}") +PY + +# Bobbit verifier wall-clock (sessionSetup→Shutdown) from context traces +for f in .bobbit/state/session-context-trace/llm-review-*.jsonl; do + python3 -c 'import json,sys; t=[json.loads(l)["ts"] for l in open(sys.argv[1])]; print((max(t)-min(t))/1000)' "$f" +done | sort -n | awk '{a[NR]=$1} END{print "median(s)=" a[int(NR/2)], "max(s)=" a[NR]}' + +# Claude Code per-API-round latency baseline +python3 - <<'PY' +import json,glob,datetime,statistics +def ms(t): + try: return datetime.datetime.fromisoformat(t.replace("Z","+00:00")).timestamp()*1000 + except: return None +gaps=[] +for f in glob.glob("~/.claude/projects/*/*.jsonl".replace("~",__import__("os").path.expanduser("~"))): + a=[ms(json.loads(l).get("timestamp")) for l in open(f) if '"assistant"' in l] + a=[x for x in a if x] + gaps+=[ (y-x)/1000 for x,y in zip(a,a[1:]) if 0<(y-x)/1000<600 ] +if gaps: print(f"CC per-round gap: median {statistics.median(gaps):.1f}s p90 {sorted(gaps)[int(len(gaps)*0.9)]:.1f}s") +PY +``` From ecbb060cfc9ef9b1c512bd22358d57d410297c61 Mon Sep 17 00:00:00 2001 From: Artur Jonkisz <ajonkisz@gmail.com> Date: Fri, 19 Jun 2026 21:37:14 +0100 Subject: [PATCH 17/18] Add SW workstream: agent swarms & reconciliation (parallelism as wall-clock lever) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New exploratory design doc agent-swarm-and-reconciliation.md, the only lever that attacks serialization itself (vs §9's levers that make serial steps cheaper): - Two shapes: fan-out swarm (Claude Code in-process swarm: spawnInProcess + forkContextMessages + omitClaudeMd) and council/reconciliation (karpathy/llm-council, hermes mixture_of_agents) — the latter folds part of the verification tax into the parallel wave as peer cross-review. - Composes on existing platform machinery (capability registry, Lifecycle Hub, model-selector pack EP G9.2, workflow templates) — pack+capability, not core surgery. - Hard dependency: cheap spawning. A swarm of cold OS processes amplifies the process-per-agent tax (§9 F14) by swarm width. Unlock = single-container Bobbit (per-agent sandboxing redundant → in-process/worker_thread members inheriting warm cache, CE-G8.4); later, federation across connected Bobbits via the gateway. - BENCH-gated phased plan SW-G0..G4 (gated on the sandbox change), reusing CE-G0.3 + CE-G8.5 wall-clock metrics. Wires in: time-and-token-cost-efficiency §9.6 (serial→parallel lever + pointer), fable execution-plan SW row. Co-authored-by: bobbit-ai <bobbit@bobbit.ai> --- docs/design/agent-swarm-and-reconciliation.md | 209 ++++++++++++++++++ docs/design/fable-program-execution-plan.md | 1 + docs/design/time-and-token-cost-efficiency.md | 30 ++- 3 files changed, 238 insertions(+), 2 deletions(-) create mode 100644 docs/design/agent-swarm-and-reconciliation.md diff --git a/docs/design/agent-swarm-and-reconciliation.md b/docs/design/agent-swarm-and-reconciliation.md new file mode 100644 index 000000000..93c86e1f9 --- /dev/null +++ b/docs/design/agent-swarm-and-reconciliation.md @@ -0,0 +1,209 @@ +# Agent Swarms & Reconciliation — parallelism as the wall-clock lever + +Status: **exploratory / vision** (no implementation) · Drafted 2026-06-19. Workstream **SW** in +[fable-program-execution-plan.md](fable-program-execution-plan.md). +Companion / dependency docs: [time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) +§9 (the latency axis + the process-per-agent root cause this builds on), +[extension-platform.md](extension-platform.md) (the capability/hook/lifecycle-hub substrate this +ships on), [extension-host.md](extension-host.md) (the worker tier), +[agent-memory.md](agent-memory.md) (shared context for swarm members). + +**The question this answers:** Claude Code now fans work out to *many* small specialised agents +(especially for research, but general work too) and runs them in parallel. If spawning an agent +becomes cheap — particularly once agents run **in-process** — can a fleet/swarm plus a +**reconciliation** step (à la [karpathy/llm-council](https://github.com/karpathy/llm-council), +or hermes' Mixture-of-Agents) cut Bobbit's wall-clock and/or raise quality? Where does it fit, +and what must land first? + +**One-paragraph answer.** Yes, and it is the single biggest *wall-clock* lever available, +because every other lever in §9 makes *serial* work cheaper while a swarm converts serial work +into **parallel** work: N independent sub-tasks at ~3 min each stop costing 3N min and start +costing ~3 min. Bobbit already has the parallel substrate (teams run up to `maxConcurrent` 12 +workers concurrently) — what's missing is (a) **cheap spawning** so fan-out of *many small* +agents is economical, (b) a **reconciliation/council** primitive to merge or arbitrate parallel +outputs, and (c) **risk-proportional, model-tiered** swarm members so you don't fan a frontier +model out 12×. All three compose out of machinery already on the roadmap — the capability +registry, the Lifecycle Hub, the model-selector pack, and workflow templates — so this is a +**pack + capability** play, not core surgery. The hard dependency is cost: a swarm of 12 *cold +OS processes* (today's model, §9.3) **amplifies** the process-per-agent tax 12× and would be a +net loss. The enabler is the move to running **full Bobbit in one container**, which makes +per-agent sandboxing unnecessary and lets swarm members run as cheap **in-process / worker_thread +forks that inherit the parent's warm cache** (§9.5 CE-G8.4). Swarm is therefore *gated on* the +sandbox change + in-process spawning; proposed here so the substrate work is shaped to allow it. + +--- + +## §1 Why parallelism is the lever the cost doc doesn't have + +[time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) §9 established that +Bobbit's slowness is **structural serialization**, not slow requests: CC runs *larger* contexts +per request (377k median) and still turns in ~4.4s; Bobbit's wall-clock goes into verifiers on +the critical path (1.34 per work session × ~3 min), multi-agent handoffs, and nudge dead-time. + +Every CE lever (L1 shrink context, L2 cut requests, L3 cheaper model, L4 stop double-paying) +makes a *serial* step cheaper. **None of them removes the serialization.** A swarm does exactly +that: + +| | Serial (today) | Swarm (proposed) | +|---|---|---| +| N independent research probes / file investigations | Σ tᵢ (one agent, N turns) | ~max(tᵢ) (N agents, 1 wave) | +| N independent sub-edits across modules | Σ tᵢ (or N team workers, but each a cold OS spawn) | ~max(tᵢ) on cheap in-process forks | +| Verification | a *downstream* gate (serial tax, §9 F13) | folded into the parallel wave as **peer cross-review** (council) | + +The third row matters most: reconciliation lets a fraction of the verification tax (§9 F5/F13) +move *into* the parallel phase as cross-checking among swarm members, instead of a separate +serial verifier gate after the fact. + +## §2 The two shapes (both already proven elsewhere) + +1. **Fan-out swarm (work/research).** A coordinator decomposes a goal into independent + sub-tasks, spawns one *small, specialised* agent per sub-task (explorer, grep-er, + doc-reader, single-file editor), runs them concurrently, and collects structured summaries — + not raw transcripts — back. This is Claude Code's `utils/swarm/spawnInProcess.js` / + `inProcessRunner.js` model: **in-process** teammates, parent context **inherited** via + `forkContextMessages`, read-only explorers shipped with a **slim context** (`omitClaudeMd`). + Bobbit's team machinery is the serial-process ancestor of this; the upgrade is cheap in-process + members + summary-only returns. + +2. **Council / reconciliation (decisions/quality).** K diverse agents (different + models/personas/prompts) answer the *same* question in parallel; a **reconciler** then + cross-reviews, ranks, and synthesises a single answer. This is + [karpathy/llm-council](https://github.com/karpathy/llm-council) (models answer → peer-rank → + chairman synthesises) and hermes' `tools/mixture_of_agents_tool.py` (reference models in + parallel → aggregator synthesis). It is a **quality** play (ensemble reduces variance on hard + reasoning) *and* a latency play (replaces a serial author→verify loop with parallel + propose+cross-check). Use it for design decisions, ambiguous specs, risky diffs, research + synthesis — not for every edit. + +The two compose: a fan-out swarm for breadth, a council for the decisions where being right the +first time avoids an expensive serial retry. + +## §3 How it composes on the existing platform (no core surgery) + +This is deliberately a **capability + workflow-template** feature on the +[extension-platform.md](extension-platform.md) substrate, honouring its "core grows only +*platform* code" principle. + +- **`swarm` capability** (`provides: [swarm]`). `ctx.capabilities.call("swarm", { tasks, + memberRole, model, reconcile, budget })` → host fans out members on the worker tier, enforces + per-member + aggregate budgets (the §5.2 budget/provenance machinery already specified), + returns structured summaries. Host-mediated, traced, non-fatal on member failure (principle 4). +- **`reconcile` capability** (`provides: [reconcile]`). The council/aggregator step as its own + pack method, so the merge policy (rank+synthesise, majority, chairman-model) is swappable. + hermes' MoA aggregator and llm-council's chairman are the reference designs. +- **`model-selector` capability (EP G9.2)** picks the member model tier and the reconciler tier — + this is exactly the **multi-model-delivery** pattern (EP G9.3: planner frontier+xhigh, + implementer cheap). Swarm members default to a *small/cheap* model; the reconciler gets the + frontier one. Fanning a frontier model out 12× is the anti-pattern the selector prevents. +- **Lifecycle Hub** (extension-platform §5) is the dispatch/budget/provenance host for members, + and `beforeSessionSpawn`/`beforeGoalCreate` proposal hooks (P8) are where a swarm member's + slim context + model tier are assigned without core changes. +- **Workflow templates as packs** (`workflows/*.yaml`): ship `research-swarm.yaml` and + `council-decision.yaml` so a goal can *opt into* the shape, same `WorkflowStore` gate schema + used today. +- **Shared context** via [agent-memory.md](agent-memory.md): members read the same recall bank + so the swarm doesn't re-derive shared facts N times. + +## §4 The hard dependency: cheap spawning (and why the sandbox change is the unlock) + +A swarm is only a win if a member is cheap to start and cheap in tokens. Today (§9.3) every agent +is a **separate OS process** (`rpc-bridge.ts` `spawn`), with its own MCP stdio children, and — +for sandboxed goals — its own `docker exec` into a pool container. A 12-member swarm in that model +pays the per-spawn startup floor **12×** and cold-writes its resident stack **12×** (§9 F14) — a +net wall-clock and cost *loss*. + +Two roadmap items remove the blocker, in order: + +1. **Full Bobbit in a single container (imminent).** Once the *whole* gateway runs inside one + Docker container, per-agent sandboxing is redundant — the trust boundary is the container, not + the agent. That removes the per-member `docker exec`/container tax and, more importantly, + makes it **safe** to run members as **in-process / `worker_threads`** units (the Extension + Host worker tier already runs pack modules this way: terminate-able workers with resource + caps — see [extension-host.md](extension-host.md)). *Per-agent sandboxing is the only reason + members must be separate processes today; collapse that and the cheap path opens.* +2. **Warm-cache reuse for short-lived spawns (§9.5 CE-G8.4).** In-process members can inherit the + coordinator's already-cached prefix (Claude Code's `forkContextMessages`) instead of cold- + writing — directly killing the §9 F14 cold-write tax that would otherwise scale with swarm + width. + +**Sequencing:** SW is gated on (1); it gets dramatically cheaper with (2). Both are independently +motivated in other workstreams — this doc's ask is only that they be **shaped to admit cheap +fan-out** (a pooled/forked member runner, summary-only returns, slim member context). + +## §5 Federation — swarms across multiple Bobbits + +Once multiple Bobbit instances connect through the **Bobbit gateway** (their projects, agents, +runtimes shared), the gateway becomes a natural **swarm scheduler across instances**: a fan-out +wave can place members on whichever Bobbit has spare capacity / the right project worktree / +the right local model, and the reconciler collects across the federation. This is out of scope +for v1 (single-instance, in-process swarm first) but the `swarm` capability's task/return +contract should be **transport-agnostic** (structured task in, structured summary out) so the +same call site works whether a member runs in-process or on a peer Bobbit. Record this seam now; +don't build it yet. + +## §6 Risks & open questions + +1. **Fan-out cost blow-up.** Width × model price can dwarf the wall-clock saving. Mitigation: + cheap-model members by default (model-selector), hard per-wave token/cost budget, and a + max-width cap (start ~4–6, not 12). BENCH-gated like all behaviour-affecting CE work. +2. **Reconciliation is not free.** The council/aggregator turn is itself a frontier call over K + answers; for trivial tasks it costs more than it saves. Reserve council for genuinely + ambiguous/risky decisions; default work to plain fan-out + the existing (risk-proportional, + §9 CE-G8.1) verification. +3. **Decomposition quality is the ceiling.** A swarm only helps when sub-tasks are genuinely + independent; bad decomposition serialises anyway (members blocking on each other) or + duplicates work. The coordinator's decomposition is the hard part and should be evaluated in + the bench suite (CE-G0.3) with a real multi-file task. +4. **Context coherence / merge conflicts.** Parallel editors on a shared worktree race; need + per-member worktrees or file-range leasing, then a reconcile/merge step. Read-only research + swarms avoid this and are the safe v1. +5. **Provenance & debuggability** (platform principle 3): every member's contribution and the + reconciler's rationale must be inspectable, or a swarm becomes an untrustable black box. +6. **Quality evidence.** MoA/llm-council quality gains are task-dependent; validate on the bench + suite before claiming them — don't ship council on faith. + +## §7 Phased plan (SW) + +All phases BENCH-gated (CE-G0.3) on **both** cost and wall-clock; gated on the single-container +sandbox change for the in-process tiers. + +- **SW-G0 — Bench shapes (S).** Add to the CE bench suite: (a) a decomposable multi-file research + task, (b) an ambiguous design-decision task. These are the measuring sticks for everything + below; without them SW is "vibes". Reuse CE-G0.3's runner + the wall-clock report fields + (CE-G8.5). +- **SW-G1 — `swarm` capability + read-only research fan-out (M).** Coordinator decomposes → + N read-only members (slim context, cheap model) → summary-only returns → coordinator + synthesises. Process-based members acceptable here (read-only, no worktree races) to ship value + before the container change; cap width ~4. Tests: capability contract, budget enforcement, + member-failure non-fatal, summary-shape pin. +- **SW-G2 — `reconcile` capability + `council-decision` workflow (M) [BENCH-GATED].** K-agent + parallel answer → reconciler synthesis (pluggable policy: rank+synthesise default). Wire + model-selector for member/reconciler tiers. BENCH: decision-quality on SW-G0(b) vs single-agent + baseline, at acceptable cost. +- **SW-G3 — In-process member runner (L, gated on single-container sandbox).** Run members as + worker_thread/in-process forks inheriting the coordinator's warm cache (CE-G8.4). This is what + makes fan-out *cheap* and unlocks wider swarms + parallel *editing* (with per-member worktrees + or file leasing + a merge reconcile). BENCH: wall-clock + cost vs SW-G1 process model. +- **SW-G4 — Federation-aware scheduling (later, recorded).** Transport-agnostic member placement + across connected Bobbits via the gateway. Design only until single-instance swarm is proven. + +### Expected impact (estimates — bench replaces these) + +| Phase | Axis | Est. effect on affected goal class | +|---|---|---| +| SW-G1 | wall-clock | research/discovery phase Σtᵢ → ~max(tᵢ); biggest on broad investigations | +| SW-G2 | quality (+ avoided serial retries) | fewer wrong-first-time decisions → fewer expensive serial redo loops | +| SW-G3 | wall-clock + cost | makes wide fan-out economical; enables parallel editing; kills cold-write width tax | +| SW-G4 | throughput | horizontal scale across instances | + +--- + +## §8 Relationship to the rest of the program + +- **Depends on:** EP G1.x (pack platform), EP G9.2 model-selector, EP §5 Lifecycle Hub + + Extension Host worker tier; the single-container sandbox change (removes per-agent sandbox → + in-process members); CE-G8.4 warm-cache reuse; CE-G0.3 bench + CE-G8.5 wall-clock metrics. +- **Reframes:** §9's verification tax — council folds cross-review into the parallel wave, + complementing CE-G8.1's risk-proportional gates rather than replacing them. +- **Does not change:** core orchestration; everything ships as packs + capabilities + + workflow templates per the extension-platform principles. diff --git a/docs/design/fable-program-execution-plan.md b/docs/design/fable-program-execution-plan.md index ac882ddd5..951258211 100644 --- a/docs/design/fable-program-execution-plan.md +++ b/docs/design/fable-program-execution-plan.md @@ -18,6 +18,7 @@ The workstreams (each doc owns its *content* — tasks, contracts, acceptance; t | **AI** autoimprovement | [autoimprovement.md](autoimprovement.md) | [autoimprovement-implementation-plan.md](autoimprovement-implementation-plan.md) | | **GA** gap-analysis easy wins | [harness-gap-analysis.md](harness-gap-analysis.md) | [gap-easy-wins-implementation-plan.md](gap-easy-wins-implementation-plan.md) | | **CI** code intelligence | [code-intelligence.md](code-intelligence.md) | [code-intelligence-implementation-plan.md](code-intelligence-implementation-plan.md) | +| **SW** agent swarms & reconciliation | [agent-swarm-and-reconciliation.md](agent-swarm-and-reconciliation.md) (exploratory) | same doc, §7 (SW-G0–G4; gated on single-container sandbox + CE-G8.4) | **An implementer agent receives exactly one goal ID and reads: the execution-authority doc's goal section → the contracts it cites → §1 below. Nothing else is required.** diff --git a/docs/design/time-and-token-cost-efficiency.md b/docs/design/time-and-token-cost-efficiency.md index 45b717445..e4a5727e9 100644 --- a/docs/design/time-and-token-cost-efficiency.md +++ b/docs/design/time-and-token-cost-efficiency.md @@ -657,7 +657,7 @@ same — a $1 verifier still makes you wait three minutes — so latency needs i investigation surfaced an **inherent architectural root cause** that several §3 symptoms (F2 resident floor, F5 verifier economics) trace back to. All numbers here are a 2026-06-19 snapshot of `.bobbit/state` (326 sessions, $1,069, 1.16B cache-read) on the primary dev machine; -the *shapes* corroborate §1's 2026-06-10 snapshot. Repro recipes in §9.6. +the *shapes* corroborate §1's 2026-06-10 snapshot. Repro recipes in §9.7. ### 9.1 The surprise: per-request latency is *not* where Bobbit loses @@ -811,7 +811,33 @@ are the quick wins; do them first.** Behaviour-affecting items are BENCH-GATED o | CE-G8.5 | visibility | makes coordination/startup dead-time a dashboard number (blocker for tuning) | | CE-G8.6 | wall-clock | low-risk goals skip orchestration entirely | -### 9.6 Reproduction recipes (latency) +### 9.6 Parallelism & swarms — the serial→parallel lever (forward-looking) + +Every lever in §6/§9.5 makes a *serial* step cheaper; none removes the **serialization** that +§9.2 identified as the real wall-clock tax. The one lever that does is **parallelism**: fan a +goal out to many small specialised agents (Claude Code's in-process swarm) and, for +decisions/quality, add a **reconciliation/council** step +([karpathy/llm-council](https://github.com/karpathy/llm-council); hermes' +`mixture_of_agents_tool.py`). N independent ~3-min sub-tasks then cost ~3 min, not 3N; and +peer cross-review folds part of the verification tax (F13/F5) *into* the parallel wave instead +of a serial downstream gate. + +The blocker is cost: a swarm of *cold OS processes* (§9.3) amplifies the process-per-agent tax +(F14) by the swarm width — a net loss. The unlock is the imminent **single-container Bobbit** +change (per-agent sandboxing becomes redundant → members run as cheap **in-process / +worker_thread forks** that inherit the parent's warm cache, CE-G8.4) and, later, **federation** +across connected Bobbits via the gateway. Because it composes from machinery already on the +roadmap (capability registry, Lifecycle Hub, model-selector pack, workflow templates), it's a +**pack + capability** feature, not core surgery. + +This is large enough to warrant its own doc — see +**[agent-swarm-and-reconciliation.md](agent-swarm-and-reconciliation.md)** (workstream **SW**: +fan-out swarm, council/reconciliation, the cheap-spawning dependency chain, federation, and a +BENCH-gated phased plan SW-G0–G4). It is *gated on* this doc's CE-G8.4 (warm-cache reuse) + +the sandbox change, and reuses CE-G0.3/CE-G8.5 (bench + wall-clock metrics) as its measuring +stick. + +### 9.7 Reproduction recipes (latency) ```bash # Verifier vs work session counts + cache-write share (the inverted economics) From e14b8d79691c543d52932bbbc1218d384112873a Mon Sep 17 00:00:00 2001 From: Artur Jonkisz <ajonkisz@gmail.com> Date: Sat, 20 Jun 2026 21:15:30 +0100 Subject: [PATCH 18/18] =?UTF-8?q?CE:=20verification-loop=20economics=20?= =?UTF-8?q?=E2=80=94=20the=20retry=20tax=20+=20trust-first=20right-sizing?= =?UTF-8?q?=20levers?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured 2026-06-20: 51% of gate signals are retries, 70% reviewer-driven, ~40h repeated impl verification. Adds CE-G8.7 (severity floor+round budget), CE-G8.8 (affected-only re-verify), CE-G8.9 (composed/right-sized workflows), CE-G8.10 (plan->cheap-exec->frontier-review + per-role thinking), CE-G8.11 (team-lead diet). Leads with trust-first north star (no lever may raise escaped-defect rate); cross-harness lessons from Claude Code + hermes-agent. Co-authored-by: bobbit-ai <bobbit@bobbit.ai> --- docs/design/fable-program-execution-plan.md | 2 +- docs/design/time-and-token-cost-efficiency.md | 10 +- docs/design/verification-loop-economics.md | 422 ++++++++++++++++++ 3 files changed, 432 insertions(+), 2 deletions(-) create mode 100644 docs/design/verification-loop-economics.md diff --git a/docs/design/fable-program-execution-plan.md b/docs/design/fable-program-execution-plan.md index 951258211..b23a1b2f3 100644 --- a/docs/design/fable-program-execution-plan.md +++ b/docs/design/fable-program-execution-plan.md @@ -10,7 +10,7 @@ The workstreams (each doc owns its *content* — tasks, contracts, acceptance; t | Workstream | Design (WHAT/WHY) | Execution authority (HOW) | |---|---|---| | **EP** extension platform | [extension-platform.md](extension-platform.md) | [extension-platform-implementation-plan.md](extension-platform-implementation-plan.md) (G1.1–G9) | -| **CE** time & token-cost efficiency | [time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) §1–§5, **§9** (latency + arch root cause) | same doc, §6 (CE-G0.1–G7.2) + **§9.5 (CE-G8 latency)**, BENCH gates | +| **CE** time & token-cost efficiency | [time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) §1–§5, **§9** (latency + arch root cause); [verification-loop-economics.md](verification-loop-economics.md) (verification-loop tax + right-sizing, trust-first) | same doc, §6 (CE-G0.1–G7.2) + **§9.5 (CE-G8 latency)** + **CE-G8.7–G8.11** (verification-loop-economics §4), BENCH gates | | **CS** comms-stack reliability | [comms-stack/01–03](comms-stack/01-understanding.md) | [comms-stack/04-current-state-and-backlog.md](comms-stack/04-current-state-and-backlog.md) §6–§7 (waves + merge map) | | **SN** sidebar & goal-nesting | [sidebar-goal-nesting-audit.md](sidebar-goal-nesting-audit.md) §1–§3 | same doc, §4 (T1–T9) | | **PB** client perf/battery | [client-performance-battery.md](client-performance-battery.md) | [client-performance-battery-implementation-plan.md](client-performance-battery-implementation-plan.md) | diff --git a/docs/design/time-and-token-cost-efficiency.md b/docs/design/time-and-token-cost-efficiency.md index e4a5727e9..124ca44ce 100644 --- a/docs/design/time-and-token-cost-efficiency.md +++ b/docs/design/time-and-token-cost-efficiency.md @@ -746,7 +746,15 @@ so the cases most comparable to Claude Code pay the most overhead relative to th ### 9.5 Remediation — latency workstream (CE-G8) Hand-off format matches §6. Lane independent of G0–G7 except where noted. **CE-G8.1 and CE-G8.3 -are the quick wins; do them first.** Behaviour-affecting items are BENCH-GATED on CE-G0.3 plus a +are the quick wins; do them first.** + +> **Deepened by measured loop data + new levers:** see +> [verification-loop-economics.md](verification-loop-economics.md) — a 2026-06-20 snapshot showing +> 51% of gate signals are retries (70% reviewer-driven, ~40 h of repeated impl verification) and +> adding goals **CE-G8.7** (severity floor + round budget), **CE-G8.8** (affected-only +> re-verification), **CE-G8.9** (goal-author-composed / agent-right-sized workflows), **CE-G8.10** +> (plan→cheap-exec→frontier-review + per-role thinking tiering), **CE-G8.11** (team-lead diet). +> It leads with the **trust-first north star**: no lever ships if it raises the escaped-defect rate. Behaviour-affecting items are BENCH-GATED on CE-G0.3 plus a **wall-clock** comparison (extend the bench report with `wallClock`, `startupMs`, `verifyMs`). #### CE-G8.1 Risk-proportional verification gates (M) **[BENCH-GATED]** — *biggest latency win* diff --git a/docs/design/verification-loop-economics.md b/docs/design/verification-loop-economics.md new file mode 100644 index 000000000..33de7bc24 --- /dev/null +++ b/docs/design/verification-loop-economics.md @@ -0,0 +1,422 @@ +# Verification-Loop Economics — the retry tax, and right-sizing the gate suite + +Status: **investigation complete, remediation not started** · Measured 2026-06-20 against a live +`.bobbit/state` snapshot on the primary dev machine (28 goals, 633 sessions, $2,184 tracked +spend, 240 gate signals, 379 verifier runs), cross-checked against 1,461 Claude Code sessions and +the `~/.hermes` council harness. Workstream **CE** in +[fable-program-execution-plan.md](fable-program-execution-plan.md) — this doc **deepens +CE-G8.1** (risk-proportional verification) with measured loop data and adds the goals +**CE-G8.7–G8.11**. +Companion docs: [time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) (§9 latency +axis + the process-per-agent root cause this builds on; the F5/F13/F14/F15 findings), +[agent-swarm-and-reconciliation.md](agent-swarm-and-reconciliation.md) (the parallel end-state +these levers converge toward), [extension-platform-implementation-plan.md](extension-platform-implementation-plan.md) +(model-selector G9.2/G9.3 — the substrate for model/thinking tiering), [agent-memory.md](agent-memory.md). + +**The question this answers:** goals run for hours and re-spawn fix/review agents long after the +work looks done. Where does that time actually go, is it the *agents* or the *machinery*, and +what is the cheapest change that bends the curve? + +**The one-paragraph answer.** The dominant tax is the **verification loop**, and it is driven by +**reviewer opinion, not broken code**. Half of all gate-signal events (123 of 240, **51%**) are +*retries* of a gate that was already signaled; 112 of those 123 retries follow a **failed** +verification, and **70% of failed signals were a reviewer returning FAIL while the build/tests +passed** — only ~9% were a real build/test failure. Each implementation re-signal re-runs the +*entire* phase suite (Build+Check+Unit+E2E + gap-analysis + code-review + security ≈ **27 min of +machine compute**, ~9–10 min wall) even when one nit changed, because there is **no +severity→verdict mapping** (any single subjective `[high]` blocks the gate) and **no +affected-only re-verification** (a re-signal cascade-resets and re-runs everything). Across the +dataset that is **≈40 h of repeated implementation-gate verification**. The fix is four +data-driven, mostly-infra-free levers — a **severity floor + round budget**, **affected-only +re-verification**, **goal-author-composed / agent-right-sized workflows**, and **model+thinking +tiering (plan→cheap→frontier-review)** — each validated by the experiment-runner before it ships, +with one hard guardrail learned from Hermes: a cheap tier is only safe behind an **independent +reviewer that checks ground truth and never self-certifies**. + +--- + +## North star — the invariant every lever bows to + +**Bobbit's defining value is that autonomous agent work is *trustworthy*: verified to a high bar +with minimal human babysitting. That is non-negotiable and ranks above every optimization in this +doc.** Speed, token cost, swarm width, thinking budget, and model tier are all **subordinate** — +no lever may ship if it raises the **escaped-defect rate**, which is why every behaviour-affecting +goal here is BENCH-gated on defect-escape, not dollars (§0.4, §6). + +What we are cutting is **waste, not assurance**: re-reviewing code that did not change, looping on +subjective nits, running heavyweight gates on trivial diffs, and re-reading bloated context. Every +lever either (a) holds quality constant while removing that waste, or (b) *strengthens* +verification — the **strengthen-free / weaken-gated asymmetry** (CE-G8.9), the +**reviewer-confirmed-bug floor** (CE-G8.7), the **non-removable build/security gate floor**, and +the **ground-truth reviewer guardrail** on any cheap tier (CE-G8.10, from Hermes). Gates stay +re-openable; nothing self-certifies. + +The end-state we are building toward — Fable-style **fan-out → synthesize → concrete plan → +execute → loop** across many agents (see [agent-swarm-and-reconciliation.md](agent-swarm-and-reconciliation.md)) +— is valuable *precisely because* it sits on this trust foundation. Parallelism multiplies output; +verification is what makes that output safe to trust at scale. The levers in this doc make +verification **cheaper and better-targeted**, never weaker, so that scaling up the agent count +later compounds quality instead of diluting it. + +--- + +## §0 Universal rules + +Same as the extension-platform / CE plans (restated because goals here are handed off +independently): + +1. **Test-first.** Every CE-G8.x goal below lists pinning tests; write them RED first. No flaky + tests. +2. **Locate code by symbol name, not line number.** Anchors here were verified on the + 2026-06-20 snapshot and will drift; if a symbol is missing, STOP and re-derive — don't guess. +3. **Gates.** `npm run check` + `npm run test:unit` for everything; `+ test:e2e` for + server/UI changes. +4. **BENCH-gated.** Every behaviour-affecting lever ships behind an experiment-runner A/B + (see §6) that measures retries, wall-clock, cost, **and** escaped-defect rate — never just + dollars. A latency/quality win on vibes is not a win. +5. **Shared-seam serialization.** `verification-logic.ts`, `gate-store.ts`, `team-manager.ts`, + `session-setup.ts` are hot seams (also touched by CE-G8.1–G8.6, CS, EP). `rg` the symbol and + serialize per execution-plan §1.4. + +--- + +## §1 The measured loop (the data) + +All figures from the 2026-06-20 snapshot; reproduction recipes in **Appendix A**. + +### 1.1 Half of all gate signals are do-overs + +| Quantity | Value | +|---|---| +| Distinct gates signaled ≥ once | 117 | +| Total signal events | 240 | +| **Re-signals (retries)** | **123 — 51% of all signals** | +| Gates that looped (≥ 1 retry) | 48 | +| Retries following a **FAILED** verification | 112 | +| Retries following an already-**PASSED** gate (cascade-reset re-run of green work) | 11 | +| Retries at a **new** commit (a fix landed) | 108 | +| Retries at the **same** commit (pure re-verification — flaky / metadata / reset) | 15 | + +Worst offenders are all the `implementation` gate: **11 retries** on *Hierarchical goal metadata* +and on *Hindsight setup & deploy modes*; 7 on *P3 modes consent* and *Experiment runner*; median +≈ 2.6 retries per implementation gate. + +### 1.2 Failures are reviewer verdicts, not broken builds + +Cause attribution of the 110 failed signals that carry step detail: + +| Cause | Count | +|---|---| +| **Reviewer (llm-review) said FAIL, build/tests green** | **70** | +| Mixed (reviewer + a command both failed) | 30 | +| Real build/test failure only | 10 | + +Which step actually failed the gate (count across all failed signals): + +| Failing step (role) | Times it failed the gate | +|---|---| +| **Code quality review** (`code-reviewer`) | **83** | +| Gap analysis (`spec-auditor`) | 40 | +| Security review (`security-reviewer`) | 31 | +| Unit tests *(real)* | 20 | +| E2E tests *(real)* | 19 | +| Regression / design / docs reviews | ~20 | + +`code-reviewer` alone failed more gates than **all build+test failures combined**. + +### 1.3 The findings are mostly subjective, yet any one blocks the gate + +Severity tags across **failed** review steps: `[critical] 18 · [high] 112 · [medium] 88 · +[low] 33`. There are only **18 critical findings in the entire dataset**. But the verdict is a +**binary pass/fail an LLM picks** (`verification_result(verdict)` in +`defaults/tools/tasks/verification_result.yaml`) with **no severity→verdict contract**: a single +`[high]` — and most "high" are subjective quality (AGENTS.md growth, UI inconsistency, "missing +test coverage"), not broken functionality — fails the gate. + +### 1.4 The cascade: one nit re-runs everything + +`gate-store.ts` cascade-resets all transitive downstream gates on a re-signal, and +`verification-logic.ts` re-runs the **whole** phase suite from scratch — there is no +"only re-run steps whose inputs changed". Measured per-step averages on the `implementation` +gate: Build 8 s · Check 14–22 s · Unit 45–76 s · **E2E 227–260 s** · Post-impl gap 145–208 s · +**Code quality 266 s** · Security 94–198 s. **One full re-run ≈ 27 min of machine compute** +(~9–10 min wall, since same-phase steps overlap). With **89 implementation re-signals** in the +dataset that is **≈ 40 h of repeated implementation-gate verification alone**, the bulk of it +re-checking code that did not change. + +### 1.5 Cost shape (corroborates CE §9) + +Spend by bucket (this snapshot): MAKER (coder/test/docs) **$968** · **TEAM-LEAD $570 across just +28 sessions** · VERIFIER steps **$549** (368) · other ~$98. The team-lead bucket is the +under-weighted sink — 28 long-lived leads cost more than all 368 verifier runs and ~60% of all +230 maker sessions, because a lead lives 30–66 h and re-reads its bloated coordination context on +every nudge (615M cache-read tokens across 28 leads). Cost falls out of wall-clock: every retry +is *both* minutes you wait *and* tokens you pay. + +--- + +## §2 Root causes (fishbone) + +``` + VERIFICATION DESIGN ORCHESTRATION MODEL STRATEGY + • no severity→verdict map • team-lead = context sink • one frontier tier for all + • binary pass/fail on opinion • 1 lead per (sub)goal • no plan→cheap→review split + • full suite re-runs on re-sig • verify serial, after work • reviewer not risk-tiered + \ | / + \ | / + ────────────────────────────────────────────────────────────────────────────────► GOALS TAKE HOURS & COST $$ + / \ + / \ + PROCESS MODEL CONTEXT + • process-per-agent (cold spawn) • full AGENTS.md on every agent + • no warm-cache reuse • no slim read-only profile + • MCP stdio per agent • re-read whole context each turn +``` + +The CE §9 doc owns the **process-model** and **context** bones (CE-G8.2/G8.3 + the inherent +F14). This doc owns the **verification-design**, **orchestration** (team-lead diet), and +**model-strategy** bones. + +--- + +## §3 What the other harnesses teach + +| Dimension | Claude Code | hermes-agent (council) | Bobbit today | +|---|---|---|---| +| Reviewer | human, inline / real-time | 1 independent `reviewer-opus`, re-verifies vs **ground truth** | 5–7 automated verifiers per gate, **re-run every re-signal** | +| Per-turn latency | ~4 s (warm in-process) | — | ~291 s (cold process-per-agent) | +| Sub-agents | opt-in; **97.6% of work uses none** | kanban tasks + MoA fan-out | always team + full gate suite | +| Gate loops | none (no gates) | FAIL → scoped fix (cards can't reopen) | 123 re-signals, full-suite re-run | +| Quality risk | human must catch it | **cheap models hallucinated PASS / "done" on RED CI; 48 FAIL vs 34 APPROVE** | nit-driven loops; non-deterministic reviewers | + +Two opposite lessons, both binding on the levers below: + +1. **Claude Code (the speed lesson).** Fast because the maker stays *warm and in-process* across + turns and fan-out is *opt-in* (97.6% of work uses no sub-agent). Portable: keep makers warm + (CE-G8.4), make sub-agents opt-in, and **don't re-run the full review suite on every + re-signal** (CE-G8.8). It has *no* automated assurance — that is the part Bobbit should keep. +2. **Hermes (the quality lesson).** Cheap workers **self-certified garbage**: claimed a clean + PASS while master CI was RED 8/8, fabricated commit SHAs, and "done" cards could not reopen so + wrong results persisted. It only stayed sane because **one independent `reviewer-opus` + re-verified against ground truth (CI, real SHAs)**. This is the **hard guardrail** on the + cheap-executor lever (CE-G8.10): a cheap tier is only safe behind a strong reviewer that + checks reality and never trusts the worker's self-report; and gates must remain + **re-openable** (Bobbit's already are — preserve that). + +--- + +## §4 The levers (hand-off backlog) + +Ordering rationale: Wave-1 (G8.7/G8.8) needs no new infra and removes roughly half of all gate +work (the retries). Wave-2 (G8.9/G8.10/G8.11) right-sizes *what* runs and on *which model/think +budget*. The parallel end-state lives in [agent-swarm-and-reconciliation.md](agent-swarm-and-reconciliation.md). + +### CE-G8.7 — Severity floor + review-round budget (S–M) **[BENCH-GATED]** — *biggest bang/effort* + +- **Diagnosis:** §1.2/§1.3. 70 of 110 failures are reviewer-only; the verdict has no severity + contract, so subjective `[high]`/`[medium]`/`[low]` block the gate and force a full re-run. +- **Contract:** make `verification_result` carry **structured findings** `{ severity, kind: + "bug"|"quality", file, line, note }`, and add a gate-level **block policy**: a gate FAILs only + on a finding that is `severity ∈ {critical, high}` **and** `kind == "bug"` (reviewer-confirmed, + reproducible). `quality`/`medium`/`low` findings are **recorded as advisory** on the gate and + surfaced to the team-lead, not auto-looped. Add a **round budget**: after N (default 2) auto + re-review rounds the gate escalates to a human decision instead of spawning another fix→re-sig + cycle. The exact rule is **a tunable**, decided by CE-G8.7-EXP (§6), defaulting to + strict-but-bug-only. +- **Owned files:** `verification_result.yaml` (+ its `extension.ts`); the reviewer role prompts + (`defaults/roles/code-reviewer.yaml`, `spec-auditor.yaml`, `security-reviewer.yaml`, + `reviewer.yaml`) to emit structured severities and the bug-vs-quality distinction; + `verification-logic.ts` (verdict aggregation → block decision); `gate-store.ts` (advisory + findings persisted, round counter). +- **Tests:** unit — block-decision table (severity × kind × strictness); a `[high] quality` + finding does NOT fail the gate, a `[high] bug` does; round-budget escalation after N. E2E — a + gate signaled with only advisory findings passes and records them; the (N+1)th retry escalates + rather than re-spawns. + +### CE-G8.8 — Affected-only re-verification (M) **[BENCH-GATED]** + +- **Diagnosis:** §1.4. A re-signal re-runs the entire phase suite even when one file changed; 89 + impl re-signals ≈ 40 h of recompute. +- **Contract:** on re-signal, compute the diff between the new commit and the previously-verified + commit; **re-run only the steps whose inputs intersect the changed files** (a step declares its + input scope: command steps by component/path globs, llm-review by the diff it reviews). Steps + whose inputs are unchanged **reuse the prior passed result** (extends the existing same-SHA + verification cache in `gate-store.ts` to a *changed-paths* cache). Cascade-reset still + invalidates downstream *content* gates, but their *unchanged* steps reuse cache. +- **Owned files:** `verification-logic.ts` (input-scope resolution + skip decision), `gate-store.ts` + (changed-paths cache key), workflow step schema (`component`/`run`/`prompt` gain an optional + `inputs:` glob list; absent ⇒ "always run", preserving today's behaviour). +- **Tests:** unit — changed-paths → step-skip table; a docs-only re-signal skips Build/Unit/E2E + and re-runs only Documentation. E2E — second signal at a new commit touching one file re-runs + only the affected steps; cache markers (`[reused — inputs unchanged]`) appear. +- **Caveat:** never skip a step a workflow marks `required: strict`; never reuse `human-signoff` + (already non-cacheable). + +### CE-G8.9 — Goal-author-composed / agent-right-sized workflows (M) **[BENCH-GATED]** + +- **Diagnosis:** today the workflow is a **frozen template picked at creation** and snapshotted + into the goal; a 2-line tooltip goal pays the same 5–7-verifier suite as a 500-line refactor. + The machinery to do better already exists (project assistant generates workflows; `propose_goal` + validates a workflow id; the workflow-editor schema; `WorkflowStore`). +- **Contract — two parts:** + 1. **Compose at creation.** A standardized step lets the goal-creating session **assemble the + gate set for *this* goal from a catalog** and record an **explicit rationale** per + included/excluded gate ("docs-only → no security/e2e", "concurrency fix → add soak gate", + "research → design+review only, no build"). Persist the rationale on the goal for audit. + 2. **Right-size in flight.** The team-lead may **propose** enabling/disabling gates after + seeing the real diff — reusing the **divergence-policy gradient + `mutation_pending` + approval card** (same machinery as plan/sub-goal mutation). **Asymmetry is the safety + model:** *strengthening* (ADD/upgrade a gate) is autonomous under `balanced`/`autonomous`; + *weakening* (disable/downgrade) is **gated**, with a **non-removable floor** — build, + typecheck, and security-on-code-changes are never agent-disableable (mirrors the bypass + human-only rule). Every toggle is justified, audit-logged, and reversible. +- **Owned files:** a `compose_workflow` proposal tool (or extend `propose_goal`); the + proposal-panel workflow surface (`src/app/proposal-panels.ts`); `WorkflowStore` snapshot path; + the gate-mutation route + `mutation_pending` plumbing (`team-manager.ts`, the policy/divergence + code); a `gateFloor` allow-list (non-disableable gate ids/kinds). +- **Tests:** unit — floor enforcement (cannot disable build/typecheck/security on a code goal); + asymmetry table (ADD autonomous, DISABLE gated by policy). E2E — a docs-only goal composed + without build/e2e completes without spawning those verifiers; an agent disable-security + proposal is held for human approval; browser E2E for the creation-time composer + persisted + rationale. +- **Why it beats a pure line-count threshold (CE-G8.1):** the author/lead reasons about the + *actual* change and records *why* — auditable and smarter than a heuristic. CE-G8.1's threshold + becomes the *default* the composer starts from. + +### CE-G8.10 — Model + thinking tiering: plan → cheap executor → frontier review (M) **[BENCH-GATED]** + +- **Diagnosis:** §1.5 — every role runs the same frontier model at the same thinking budget; + makers are 44% of spend. Front-loading reasoning into a precise plan lets a cheap/fast model + execute and a frontier model review — cutting cost *and* wall-clock. +- **Contract:** a **plan gate** whose deliverable is a *precise change-list* (exactly what to + change, where, with acceptance per item); a **cheap, low/no-thinking executor** role that + implements the change-list mechanically; a **frontier reviewer** (high thinking) that checks + plan + diff **against ground truth**. Tiers resolved by the **model-selector** (EP G9.2/G9.3); + thinking budget set **per role × task risk** (Bobbit already supports per-role `thinkingLevel` + in `session-setup.ts`) — executors default to no/low extended thinking, planners/reviewers to + high. +- **Hard guardrail (Hermes, §3.2):** the cheap tier is only enabled behind CE-G8.7's + reviewer-confirmed-bug floor and a reviewer that verifies reality (CI/tests/SHAs), never the + worker's self-report. If plan quality is the ceiling, a wrong plan faithfully produces wrong + code — so the plan gate is itself reviewed (frontier) before execution. +- **Owned files:** model-selector capability (EP); role definitions for `planner` / cheap + `executor`; `session-setup.ts` (per-role model + thinking resolution); workflow templates that + wire the plan→exec→review sequence. +- **Tests:** unit — tier resolution per role/risk; executor spawns with low thinking + cheap + model. E2E — a plan→exec→review goal completes; BENCH — cost & wall-clock down vs + frontier-everything **with no increase in escaped defects** (the gate that matters). + +### CE-G8.11 — Team-lead diet (M) + +- **Diagnosis:** §1.5 — 28 leads cost $570; a lead accumulates the whole goal's coordination + history and re-reads it every nudge. +- **Contract:** **summary handoffs** (workers return structured summaries, not raw transcripts, + into the lead's context); **lead-context compaction** at phase boundaries; optionally + **phase-scoped / ephemeral leads** (a fresh lead per phase seeded with the prior phase's + summary) so the resident context never grows unbounded. +- **Owned files:** `team-manager.ts` (worker-result summarization, nudge payload), the lead + prompt assembly, compaction hooks. +- **Tests:** unit — worker result is summarized before injection; lead resident size stays under + a pinned budget across N phases. E2E — a multi-phase goal's lead context does not grow + monotonically with worker count. + +### Expected effect (estimates — the experiment-runner replaces these) + +| Goal | Δ cost | Δ wall-clock | Attacks | +|---|---|---|---| +| CE-G8.7 severity floor + budget | −− | −− | the 70 reviewer-only failures / 123 retries | +| CE-G8.8 affected-only re-verify | −− | −− | the 27-min × 89 re-run cascade | +| CE-G8.9 composed / right-sized gates | −− | −− | tiny goals paying the full suite | +| CE-G8.10 model+thinking tiering | −− | − | maker spend (44%) + first-time correctness | +| CE-G8.11 team-lead diet | −− | − | the $570 orchestrator sink | + +--- + +## §5 Sequencing & dependencies + +``` +WAVE 0 instrument WAVE 1 kill the loop (no new infra) WAVE 2 right-size WAVE 3 (CE-G8.4 + SW) +───────────────── ────────────────────────────────── ───────────────── ───────────────────── +loop+latency metrics → CE-G8.7 severity floor + budget → CE-G8.9 composed gates → in-process agents +(CE-G8.5) → CE-G8.8 affected-only re-verify → CE-G8.10 model/think tier → swarm + cheap subgoals + CE-G8.3 slim read-only ctx CE-G8.11 team-lead diet + council (SW-G1..3) + + Experiment-runner spans every wave → A/B each lever on a real task; measure + retries · wall-clock · cost · escaped-defect rate (never dollars alone). +``` + +- **CE-G8.7** depends only on the structured-findings contract; do first. +- **CE-G8.8** is independent of G8.7 (different seam) — parallelizable. +- **CE-G8.9** depends on the divergence-policy/`mutation_pending` machinery (exists) and the + `WorkflowStore` snapshot path; CE-G8.1's threshold is its default. +- **CE-G8.10** depends on the model-selector (EP G9.2/G9.3) and is **gated** on CE-G8.7's bug + floor (the Hermes guardrail). +- **Wave 3** is gated on the single-container sandbox change + CE-G8.4 (warm-cache reuse), per + the swarm doc; "cheap sub-goals as the unit of work" is the same dependency. + +--- + +## §6 Experiment plan (the point of the experiment-runner) + +These levers are **hypotheses with a number attached**; the experiment-runner extension (in +flight) decides them on real goals instead of by guess. Each experiment fixes one task and varies +one knob, measuring `{ retries, wallClockMs, costUsd, escapedDefects }` (escaped-defects via a +post-merge re-review or a planted-bug control). + +- **CE-G8.7-EXP — severity rule.** Same task; arm A `[high]`-blocks (today), arm B + `[critical] + reviewer-confirmed bug` blocks, arm C `[high]` blocks but retries capped at 2 → + human. Pick the arm that minimizes retries+wall-clock without raising escaped-defects. +- **CE-G8.8-EXP — re-verify scope.** re-run-all vs affected-only on re-signal; expect large + wall-clock/cost drop at equal quality. +- **CE-G8.10-EXP — tiering.** frontier-everything vs plan(frontier)→exec(cheap,no-think)→review(frontier); + the **escaped-defect** column is decisive (Hermes failure mode). +- **CE-G8.9-EXP — composed gates.** template-frozen vs author-composed gate set on a mixed batch + (docs-only, small fix, large feature); measure suite-size vs defect-escape. + +Until the runner lands these are **defaults, not decisions**: ship CE-G8.7/G8.8 behind a flag, +default strict-but-bug-only + affected-only, and flip per experiment evidence. + +--- + +## Appendix A — Reproduction recipes + +All against `.bobbit/state` on the primary dev machine (2026-06-20 snapshot). + +```python +# Retry rate + cause + new-vs-same-commit (the §1.1/§1.2 numbers) +import json, collections +g = json.load(open(".bobbit/state/gates.json")) +total=resig=after_fail=after_pass=samesha=newsha=0; gates=0 +for e in g: + s=e.get("signals",[]) + if not s: continue + gates+=1; total+=len(s) + for i in range(1,len(s)): + resig+=1; pv=(s[i-1].get("verification") or {}).get("status") + after_fail += pv=="failed"; after_pass += pv=="passed" + if s[i-1].get("commitSha")==s[i].get("commitSha"): samesha+=1 + else: newsha+=1 +print(f"gates={gates} signals={total} retries={resig} ({100*resig/total:.0f}%)") +print(f" after-fail={after_fail} after-pass(reset)={after_pass} same-sha={samesha} new-sha={newsha}") + +# Loop driver: which step failed the gate, and finding severities (§1.2/§1.3) +import re +failstep=collections.Counter(); sev=collections.Counter() +for e in g: + for s in e.get("signals",[]): + v=s.get("verification") or {} + if v.get("status")!="failed": continue + for st in v.get("steps",[]): + if st.get("passed") is False: failstep[f"{st.get('type')}:{st.get('name')}"]+=1 + if st.get("type")=="llm-review" and st.get("passed") is False: + out=st.get("output") or "" + for k in ("critical","high","medium","low"): sev[k]+=len(re.findall(r"\[%s\]"%k,out,re.I)) +print(failstep.most_common(8)); print(dict(sev)) + +# Cost by role + the team-lead sink (§1.5): join session-costs.json to sessions.json by id/role +``` + +The per-step duration averages (§1.4) come from iterating `signals[].verification.steps[]` +(`duration_ms`, `name`, `passed`, `type`) on `implementation`-gate records. Spend-by-role joins +`session-costs.json` (`totalCost`, `cacheReadTokens`) to `sessions.json` `sessions[]` by id, +bucketing on `role`; verifier-step records are keyed `llm-review-*` / `agent-qa-*`.