diff --git a/docs/design/agent-memory.md b/docs/design/agent-memory.md new file mode 100644 index 000000000..87fa9cf9e --- /dev/null +++ b/docs/design/agent-memory.md @@ -0,0 +1,95 @@ +# Agent Memory — architecture decision record + +Status: accepted 2026-06-11 (owner interview + research). Governs the memory layers across +workstreams: EP G2/G3.3 (Hindsight pack), EP G1.6 (session-memory provider), GA-R5 +(profile memory), MC Caretaker (retention). Research base: +[code-intelligence-alternatives.md §7](code-intelligence-alternatives.md) + Hindsight +docs/best-practices (verified 2026-06-11). + +## §1 Three layers, three jobs — sessions stay local + +| Layer | What it is | Job | Lifetime | +|---|---|---|---| +| **Session transcripts** (local JSONL + FlexSearch) | the *system of record* | replay, debugging, flight recorder (MC), audit, exact-quote search | prunable by policy, never silently | +| **Markdown staff memory** (per-role/project files) | *curated, authoritative* procedural knowledge — conventions, gotchas, preferences | always-loaded context; human-auditable and editable | permanent, small | +| **Hindsight pack** (optional, EP G2) | *episodic recall* — distilled facts/experiences/entities/beliefs over high-volume history | "what did we decide about X in March", cross-session/cross-agent recall, temporal queries | grows; managed by the daemon | + +**Do we still need local sessions if Hindsight is enabled? Yes — they are different +artifacts.** Hindsight stores *LLM-extracted distillations* (an LLM call per retain), not +transcripts; it is lossy by design and cannot replay a session, settle a "what exactly did +the agent run" dispute, or feed the MC flight recorder. The industry pattern matches: +Claude Code keeps transcripts AND auto-memory; Devin keeps sessions AND Knowledge. Memory +systems answer questions; transcripts prove them. + +**Pruning (owner's instinct, adopted):** transcript retention becomes config +(`sessions.retention`: age/size/per-project caps), and the **Caretaker (MC) runs a +consolidation pass before pruning**: for transcripts past the threshold — ensure the +Hindsight retain happened (if pack enabled), write/refresh the markdown summary, then +prune. Order is the invariant: **distill → then delete; never silently drop unretained +history.** This lands as a Caretaker card under MC once EP G2 exists. + +## §2 Is Hindsight the best daemon? Yes — comparison and reasons + +(Condensed from the research annex §7; URLs there.) + +| System | Verdict | Decisive reasons | +|---|---|---| +| **Hindsight** (vectorize-io) — **CHOSEN** | First-party memory pack | MIT; very active (v0.8.1 June 2026, 60+ releases); **official Node/TS SDK + REST + MCP**; hybrid recall (semantic + BM25 + graph + temporal) with reranking; self-hosted on Postgres+pgvector (no external vector-DB dependency); peer-reviewed design (arXiv:2512.12818); the Nous Hermes coding agent ships it as a native memory provider with daemon autostart + 5-min idle stop — the exact pack pattern EP G3.3 plans | +| mem0 | Rejected | Cloud-leaning OSS; benchmark claims disputed by third parties (LoCoMo rebuttals); weaker self-host story | +| Letta (MemGPT) | Rejected | It's a competing *agent harness* with memory inside, not a memory service for ours | +| Zep / Graphiti | Rejected | Zep discontinued its self-hosted CE; only the Graphiti library remains (bring-your-own graph DB + service layer = we'd build the daemon ourselves) | +| cognee | Rejected | Lightest self-host but Python-only SDK — wrong language for the gateway | +| LangMem | Rejected | Little value outside LangChain/LangGraph | +| File-based only | **Kept — as the default** | Zero ops, auditable, in-context; the 2026 production trend (Claude Code, Cursor, Devin are all text-first). It is layer 2, not a rejected option — Hindsight must *earn* its ops cost (Docker/Postgres, LLM call per retain), which is why the pack is optional and file memory stays authoritative for procedural knowledge | + +Caveat recorded: many "Hindsight is #1" comparisons are published by vectorize.io itself; +the independent signals are the license, activity, SDK fit, and the Hermes adoption. + +## §3 Bank topology: one shared bank, mandatory tags (owner instinct confirmed) + +**Verified Hindsight facts (best-practices docs, 2026-06-11):** banks are fully isolated; +**all operations (retain/recall/reflect) target a single bank — cross-bank search is not +supported**; **tags are the filtering mechanism** (metadata is explicitly NOT filterable); +strict modes (`any_strict`/`all_strict`) exist to prevent cross-scope leakage; their docs +endorse "a shared bank with tags" as a standard pattern; banks carry mission/directives/ +disposition (bank-level reasoning identity). + +**Decision: one shared `bobbit` bank for the whole installation, tag-scoped.** + +- Every retain auto-tagged by the pack from session context — agents never hand-tag: + `project:`, `agent:`, `goal:`, `kind:`. + Worktrees get **no** tag dimension (ephemeral; their learnings belong to the project). +- Default recall = strict-scoped to `project:` (+ untagged/org-wide). Explicit + wider recall = the tool's `scope: project | global | all` arg mapped to tag filters — + so "have we solved this anywhere before?" is **one native query**, which is the decisive + argument: with per-project banks that query is impossible (no cross-bank search) short of + pack-side fan-out across N banks with hand-rolled merging, and reflect could never span + projects at all. +- One bank also means **one entity graph** (the same staff member, library, or convention + resolves to one entity instead of N copies) and one mission/disposition to configure. +- **Costs, with mitigations:** (a) reflect/mental-models consolidate bank-wide, so beliefs + can blend projects — mitigate with `observation_scopes` and project-scoped recall + defaults; (b) project offboarding needs **delete-by-tag** — verify against the targeted + Hindsight version at G2 implementation (checkpoint added to the EP plan's G2 notes; if + absent, fall back to per-project banks for deletion-sensitive installs); (c) multi-tenant + leakage — not our shape (single-operator installations), but strict tag matching is on + by default anyway. +- **Escape hatch kept:** isolation-critical projects (client work under NDA) may opt into + a dedicated bank via project config, accepting invisibility to `scope: all`. Default + remains shared. + +This **changes EP G2's earlier sketch** ("per-project banks", tool arg +`bank: current | global | all`): the manifest description and provider flow now say +*shared tag-scoped bank*, and the tool arg becomes `scope:` backed by tag filters, not +bank switching. Updated in [extension-platform.md](extension-platform.md) (manifest example, +Hindsight pack section). + +## §4 Interaction with the other memory-adjacent plans + +- **EP G1.6 session-memory provider** (FlexSearch over transcripts) is unaffected — it + searches layer 1 and works with zero infrastructure; it is also the graceful degradation + when the Hindsight pack is disabled. +- **GA-R5 profile memory** writes to layer 2 (markdown), optionally mirrored into Hindsight + as `kind:preference` retains when the pack is on. +- **CI repo map / code intelligence** is spatial, not temporal — no overlap (layer table in + [code-intelligence.md §1](code-intelligence.md)). diff --git a/docs/design/agent-swarm-and-reconciliation.md b/docs/design/agent-swarm-and-reconciliation.md new file mode 100644 index 000000000..93c86e1f9 --- /dev/null +++ b/docs/design/agent-swarm-and-reconciliation.md @@ -0,0 +1,209 @@ +# Agent Swarms & Reconciliation — parallelism as the wall-clock lever + +Status: **exploratory / vision** (no implementation) · Drafted 2026-06-19. Workstream **SW** in +[fable-program-execution-plan.md](fable-program-execution-plan.md). +Companion / dependency docs: [time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) +§9 (the latency axis + the process-per-agent root cause this builds on), +[extension-platform.md](extension-platform.md) (the capability/hook/lifecycle-hub substrate this +ships on), [extension-host.md](extension-host.md) (the worker tier), +[agent-memory.md](agent-memory.md) (shared context for swarm members). + +**The question this answers:** Claude Code now fans work out to *many* small specialised agents +(especially for research, but general work too) and runs them in parallel. If spawning an agent +becomes cheap — particularly once agents run **in-process** — can a fleet/swarm plus a +**reconciliation** step (à la [karpathy/llm-council](https://github.com/karpathy/llm-council), +or hermes' Mixture-of-Agents) cut Bobbit's wall-clock and/or raise quality? Where does it fit, +and what must land first? + +**One-paragraph answer.** Yes, and it is the single biggest *wall-clock* lever available, +because every other lever in §9 makes *serial* work cheaper while a swarm converts serial work +into **parallel** work: N independent sub-tasks at ~3 min each stop costing 3N min and start +costing ~3 min. Bobbit already has the parallel substrate (teams run up to `maxConcurrent` 12 +workers concurrently) — what's missing is (a) **cheap spawning** so fan-out of *many small* +agents is economical, (b) a **reconciliation/council** primitive to merge or arbitrate parallel +outputs, and (c) **risk-proportional, model-tiered** swarm members so you don't fan a frontier +model out 12×. All three compose out of machinery already on the roadmap — the capability +registry, the Lifecycle Hub, the model-selector pack, and workflow templates — so this is a +**pack + capability** play, not core surgery. The hard dependency is cost: a swarm of 12 *cold +OS processes* (today's model, §9.3) **amplifies** the process-per-agent tax 12× and would be a +net loss. The enabler is the move to running **full Bobbit in one container**, which makes +per-agent sandboxing unnecessary and lets swarm members run as cheap **in-process / worker_thread +forks that inherit the parent's warm cache** (§9.5 CE-G8.4). Swarm is therefore *gated on* the +sandbox change + in-process spawning; proposed here so the substrate work is shaped to allow it. + +--- + +## §1 Why parallelism is the lever the cost doc doesn't have + +[time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) §9 established that +Bobbit's slowness is **structural serialization**, not slow requests: CC runs *larger* contexts +per request (377k median) and still turns in ~4.4s; Bobbit's wall-clock goes into verifiers on +the critical path (1.34 per work session × ~3 min), multi-agent handoffs, and nudge dead-time. + +Every CE lever (L1 shrink context, L2 cut requests, L3 cheaper model, L4 stop double-paying) +makes a *serial* step cheaper. **None of them removes the serialization.** A swarm does exactly +that: + +| | Serial (today) | Swarm (proposed) | +|---|---|---| +| N independent research probes / file investigations | Σ tᵢ (one agent, N turns) | ~max(tᵢ) (N agents, 1 wave) | +| N independent sub-edits across modules | Σ tᵢ (or N team workers, but each a cold OS spawn) | ~max(tᵢ) on cheap in-process forks | +| Verification | a *downstream* gate (serial tax, §9 F13) | folded into the parallel wave as **peer cross-review** (council) | + +The third row matters most: reconciliation lets a fraction of the verification tax (§9 F5/F13) +move *into* the parallel phase as cross-checking among swarm members, instead of a separate +serial verifier gate after the fact. + +## §2 The two shapes (both already proven elsewhere) + +1. **Fan-out swarm (work/research).** A coordinator decomposes a goal into independent + sub-tasks, spawns one *small, specialised* agent per sub-task (explorer, grep-er, + doc-reader, single-file editor), runs them concurrently, and collects structured summaries — + not raw transcripts — back. This is Claude Code's `utils/swarm/spawnInProcess.js` / + `inProcessRunner.js` model: **in-process** teammates, parent context **inherited** via + `forkContextMessages`, read-only explorers shipped with a **slim context** (`omitClaudeMd`). + Bobbit's team machinery is the serial-process ancestor of this; the upgrade is cheap in-process + members + summary-only returns. + +2. **Council / reconciliation (decisions/quality).** K diverse agents (different + models/personas/prompts) answer the *same* question in parallel; a **reconciler** then + cross-reviews, ranks, and synthesises a single answer. This is + [karpathy/llm-council](https://github.com/karpathy/llm-council) (models answer → peer-rank → + chairman synthesises) and hermes' `tools/mixture_of_agents_tool.py` (reference models in + parallel → aggregator synthesis). It is a **quality** play (ensemble reduces variance on hard + reasoning) *and* a latency play (replaces a serial author→verify loop with parallel + propose+cross-check). Use it for design decisions, ambiguous specs, risky diffs, research + synthesis — not for every edit. + +The two compose: a fan-out swarm for breadth, a council for the decisions where being right the +first time avoids an expensive serial retry. + +## §3 How it composes on the existing platform (no core surgery) + +This is deliberately a **capability + workflow-template** feature on the +[extension-platform.md](extension-platform.md) substrate, honouring its "core grows only +*platform* code" principle. + +- **`swarm` capability** (`provides: [swarm]`). `ctx.capabilities.call("swarm", { tasks, + memberRole, model, reconcile, budget })` → host fans out members on the worker tier, enforces + per-member + aggregate budgets (the §5.2 budget/provenance machinery already specified), + returns structured summaries. Host-mediated, traced, non-fatal on member failure (principle 4). +- **`reconcile` capability** (`provides: [reconcile]`). The council/aggregator step as its own + pack method, so the merge policy (rank+synthesise, majority, chairman-model) is swappable. + hermes' MoA aggregator and llm-council's chairman are the reference designs. +- **`model-selector` capability (EP G9.2)** picks the member model tier and the reconciler tier — + this is exactly the **multi-model-delivery** pattern (EP G9.3: planner frontier+xhigh, + implementer cheap). Swarm members default to a *small/cheap* model; the reconciler gets the + frontier one. Fanning a frontier model out 12× is the anti-pattern the selector prevents. +- **Lifecycle Hub** (extension-platform §5) is the dispatch/budget/provenance host for members, + and `beforeSessionSpawn`/`beforeGoalCreate` proposal hooks (P8) are where a swarm member's + slim context + model tier are assigned without core changes. +- **Workflow templates as packs** (`workflows/*.yaml`): ship `research-swarm.yaml` and + `council-decision.yaml` so a goal can *opt into* the shape, same `WorkflowStore` gate schema + used today. +- **Shared context** via [agent-memory.md](agent-memory.md): members read the same recall bank + so the swarm doesn't re-derive shared facts N times. + +## §4 The hard dependency: cheap spawning (and why the sandbox change is the unlock) + +A swarm is only a win if a member is cheap to start and cheap in tokens. Today (§9.3) every agent +is a **separate OS process** (`rpc-bridge.ts` `spawn`), with its own MCP stdio children, and — +for sandboxed goals — its own `docker exec` into a pool container. A 12-member swarm in that model +pays the per-spawn startup floor **12×** and cold-writes its resident stack **12×** (§9 F14) — a +net wall-clock and cost *loss*. + +Two roadmap items remove the blocker, in order: + +1. **Full Bobbit in a single container (imminent).** Once the *whole* gateway runs inside one + Docker container, per-agent sandboxing is redundant — the trust boundary is the container, not + the agent. That removes the per-member `docker exec`/container tax and, more importantly, + makes it **safe** to run members as **in-process / `worker_threads`** units (the Extension + Host worker tier already runs pack modules this way: terminate-able workers with resource + caps — see [extension-host.md](extension-host.md)). *Per-agent sandboxing is the only reason + members must be separate processes today; collapse that and the cheap path opens.* +2. **Warm-cache reuse for short-lived spawns (§9.5 CE-G8.4).** In-process members can inherit the + coordinator's already-cached prefix (Claude Code's `forkContextMessages`) instead of cold- + writing — directly killing the §9 F14 cold-write tax that would otherwise scale with swarm + width. + +**Sequencing:** SW is gated on (1); it gets dramatically cheaper with (2). Both are independently +motivated in other workstreams — this doc's ask is only that they be **shaped to admit cheap +fan-out** (a pooled/forked member runner, summary-only returns, slim member context). + +## §5 Federation — swarms across multiple Bobbits + +Once multiple Bobbit instances connect through the **Bobbit gateway** (their projects, agents, +runtimes shared), the gateway becomes a natural **swarm scheduler across instances**: a fan-out +wave can place members on whichever Bobbit has spare capacity / the right project worktree / +the right local model, and the reconciler collects across the federation. This is out of scope +for v1 (single-instance, in-process swarm first) but the `swarm` capability's task/return +contract should be **transport-agnostic** (structured task in, structured summary out) so the +same call site works whether a member runs in-process or on a peer Bobbit. Record this seam now; +don't build it yet. + +## §6 Risks & open questions + +1. **Fan-out cost blow-up.** Width × model price can dwarf the wall-clock saving. Mitigation: + cheap-model members by default (model-selector), hard per-wave token/cost budget, and a + max-width cap (start ~4–6, not 12). BENCH-gated like all behaviour-affecting CE work. +2. **Reconciliation is not free.** The council/aggregator turn is itself a frontier call over K + answers; for trivial tasks it costs more than it saves. Reserve council for genuinely + ambiguous/risky decisions; default work to plain fan-out + the existing (risk-proportional, + §9 CE-G8.1) verification. +3. **Decomposition quality is the ceiling.** A swarm only helps when sub-tasks are genuinely + independent; bad decomposition serialises anyway (members blocking on each other) or + duplicates work. The coordinator's decomposition is the hard part and should be evaluated in + the bench suite (CE-G0.3) with a real multi-file task. +4. **Context coherence / merge conflicts.** Parallel editors on a shared worktree race; need + per-member worktrees or file-range leasing, then a reconcile/merge step. Read-only research + swarms avoid this and are the safe v1. +5. **Provenance & debuggability** (platform principle 3): every member's contribution and the + reconciler's rationale must be inspectable, or a swarm becomes an untrustable black box. +6. **Quality evidence.** MoA/llm-council quality gains are task-dependent; validate on the bench + suite before claiming them — don't ship council on faith. + +## §7 Phased plan (SW) + +All phases BENCH-gated (CE-G0.3) on **both** cost and wall-clock; gated on the single-container +sandbox change for the in-process tiers. + +- **SW-G0 — Bench shapes (S).** Add to the CE bench suite: (a) a decomposable multi-file research + task, (b) an ambiguous design-decision task. These are the measuring sticks for everything + below; without them SW is "vibes". Reuse CE-G0.3's runner + the wall-clock report fields + (CE-G8.5). +- **SW-G1 — `swarm` capability + read-only research fan-out (M).** Coordinator decomposes → + N read-only members (slim context, cheap model) → summary-only returns → coordinator + synthesises. Process-based members acceptable here (read-only, no worktree races) to ship value + before the container change; cap width ~4. Tests: capability contract, budget enforcement, + member-failure non-fatal, summary-shape pin. +- **SW-G2 — `reconcile` capability + `council-decision` workflow (M) [BENCH-GATED].** K-agent + parallel answer → reconciler synthesis (pluggable policy: rank+synthesise default). Wire + model-selector for member/reconciler tiers. BENCH: decision-quality on SW-G0(b) vs single-agent + baseline, at acceptable cost. +- **SW-G3 — In-process member runner (L, gated on single-container sandbox).** Run members as + worker_thread/in-process forks inheriting the coordinator's warm cache (CE-G8.4). This is what + makes fan-out *cheap* and unlocks wider swarms + parallel *editing* (with per-member worktrees + or file leasing + a merge reconcile). BENCH: wall-clock + cost vs SW-G1 process model. +- **SW-G4 — Federation-aware scheduling (later, recorded).** Transport-agnostic member placement + across connected Bobbits via the gateway. Design only until single-instance swarm is proven. + +### Expected impact (estimates — bench replaces these) + +| Phase | Axis | Est. effect on affected goal class | +|---|---|---| +| SW-G1 | wall-clock | research/discovery phase Σtᵢ → ~max(tᵢ); biggest on broad investigations | +| SW-G2 | quality (+ avoided serial retries) | fewer wrong-first-time decisions → fewer expensive serial redo loops | +| SW-G3 | wall-clock + cost | makes wide fan-out economical; enables parallel editing; kills cold-write width tax | +| SW-G4 | throughput | horizontal scale across instances | + +--- + +## §8 Relationship to the rest of the program + +- **Depends on:** EP G1.x (pack platform), EP G9.2 model-selector, EP §5 Lifecycle Hub + + Extension Host worker tier; the single-container sandbox change (removes per-agent sandbox → + in-process members); CE-G8.4 warm-cache reuse; CE-G0.3 bench + CE-G8.5 wall-clock metrics. +- **Reframes:** §9's verification tax — council folds cross-review into the parallel wave, + complementing CE-G8.1's risk-proportional gates rather than replacing them. +- **Does not change:** core orchestration; everything ships as packs + capabilities + + workflow templates per the extension-platform principles. diff --git a/docs/design/autoimprovement-implementation-plan.md b/docs/design/autoimprovement-implementation-plan.md new file mode 100644 index 000000000..2e6437290 --- /dev/null +++ b/docs/design/autoimprovement-implementation-plan.md @@ -0,0 +1,213 @@ +# Autoimprovement — Implementation Plan (hand-off) + +Status: ready for execution, not started. Workstream **AI** in +[fable-program-execution-plan.md](fable-program-execution-plan.md). + +Companion to [autoimprovement.md](autoimprovement.md) (the WHAT/WHY — read §1–§6 and +**Appendix A** before implementing; the §2 change-class table and §5 ladder guard rails are +LAW — enforce in code, not prompts). + +> **Anchor baseline:** fable-docs @ 2026-06-11 (master parent `6ec8c8f9`). Locate by symbol +> name; missing symbol ⇒ STOP and re-derive from the cited pattern, never improvise. +> +> **Precision policy:** AI-P1/P2 file/function level; AI-P3…P6 contract level (substrate +> created by P1/P2) — re-verify anchors first. +> +> **Universal rules:** [extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) +> + [fable-program-execution-plan.md §1](fable-program-execution-plan.md). Depends on: +> MC-P0 (global scope) for the Improver staff home; GA-R2 (push triggers — shared seam, +> land there); MC-P3 (flight recorder — stub until it lands, like MC-P2a does). + +--- + +## Goal map + +``` +AI-P1 substrate ─→ AI-P2a judge+pack ─→ AI-P2b Improver ─→ AI-P3a curator ─→ AI-P3b dreaming + AI-P2b ─→ AI-P4 shadow ─→ AI-P5 autonomy ─→ AI-P6 measurement +``` + +--- + +## AI-P1 — Proposal substrate + +**Outcome:** an `ImprovementProposal` can be created by a tool, rendered in the proposal +panel with diff + evidence, decided by a human, and queried over REST. Nothing automatic. + +**Owned files:** NEW `src/server/agent/improvement-store.ts`; NEW +`defaults/tools/proposals/propose_improvement.yaml` (+ wiring in that group's +`extension.ts`); `src/app/proposal-registry.ts` + the proposal panel render path; +`server.ts` (routes); NEW `tests/improvement-store.test.ts`, +`tests/e2e/improvements.spec.ts`, `tests/e2e/ui/improvement-proposal.spec.ts`. + +**Steps** + +1. `improvement-store.ts`: implement the types from autoimprovement.md Appendix A.1 + **verbatim** (copy the interfaces; do not rename fields). JSONL at + `.bobbit/state/improvement/proposals.jsonl`: append-only, status change = append full + record same `id`, last-writer-wins on load; crash-safe write per + [session-store-crash-safety.md](session-store-crash-safety.md). Export + `createProposal`, `decideProposal`, `listProposals`, `getProposal`. +2. Enforce the §2 ceilings **in `decideProposal`**: `decision.by === "policy"` is rejected + outright in this phase (autonomy is AI-P5); class validation rejects unknown + `ChangeClass`. +3. REST per Appendix A.2: `GET /api/improvements`, `POST /api/improvements`, + `POST /api/improvements/:id/decision`. (`/revert` and `/calibration` come later — do + not stub routes.) +4. Tool: `propose_improvement.yaml` mirroring `propose_staff`'s YAML shape in + `defaults/tools/proposals/` (params per Appendix A.3: + `change_class, title, diff?, files?, evidence`); provider `bobbit-extension` calling + `POST /api/improvements`. +5. Client: extend `ProposalType` union (`src/app/proposal-registry.ts:21`), add + `"improvement"` to `PROPOSAL_TYPES` (`:23`), and register a `ProposalTypePlugin` + (`:60`, registry at `PROPOSAL_TYPE_REGISTRY:277`) rendering: title, change-class badge, + evidence bundle (signal, session links, excerpts), unified diff (reuse the existing + proposal diff component — see `src/app/project-proposal-diff.ts`), Approve / Reject + (+required reason on reject). Inline editing rides the editable-proposals machinery — + reuse, don't fork ([editable-proposals.md](editable-proposals.md)). + +**Tests (author first; RED unless noted)** + +- Unit: store round-trip; status append/last-writer-wins; crash-mid-append load; unknown + class rejected; `by: "policy"` rejected. +- API E2E: create → list → decide(approve) → status `approved-human`; + decide(reject, no reason) → 400. +- Browser E2E: hand-POST a proposal; panel renders class badge + diff + evidence; reject + persists reason across reload. + +**Acceptance:** all green; proposal-panel suites for existing types green unmodified; +`tests/test-phase-invariant.test.ts` green (new tests land in the right phases). + +--- + +## AI-P2a — Judge + learned-skills pack (inert plumbing) + +**Outcome:** proposals get judged confidence scores; an approved `skill-new` lands as a +staged skill in a server-scoped `learned-skills` pack; skill usage is tracked. No staff yet. + +**Owned files:** NEW `src/server/agent/improvement-judge.ts`; NEW +`src/server/agent/learned-skills-pack.ts`; skill-usage hook in +`src/server/agent/slash-skills.ts` (one call site); NEW `tests/improvement-judge.test.ts`, +`tests/learned-skills-pack.test.ts`. + +**Steps** + +1. `improvement-judge.ts`: `judgeProposal(p) → {confidence, rubricVersion, model}` — one + non-streaming aux-model call. Model resolution: the `auxiliary.improver-judge` slot + (Appendix A.4); **first lander defines the `auxiliary.*` config shape** per execution + plan §1.4 — follow [per-role-model-overrides.md](per-role-model-overrides.md) + conventions for resolution + fallback-to-main-model on `auto`. Rubric text loaded from + the pack path (Appendix A.3), keyed by `ChangeClass`; missing rubric ⇒ judge returns + no score (never throws). Judge result appended onto the proposal record. +2. `learned-skills-pack.ts`: create-on-first-use server-scoped pack (resolver scope order + already supports it — see `pack-types.ts` scope order in EP plan §0.1); approved + `skill-new.files[]` written under it with a `staged: true` marker the skills catalog + renders as `[learned — staged]`; `pack_activation` per-entity disable must work on + these like any pack skill (pin it). +3. Usage hook: where `slash-skills.ts` resolves/loads a skill, append + `{skillName, sessionId, ts}` to `.bobbit/state/improvement/skill-usage.jsonl` + (cheap fire-and-forget; this feeds AI-P3 lifecycle and AI-P6). + +**Tests:** judge — rubric routing per class, `auto` fallback, missing-rubric no-throw +(mock the model client; no live LLM in unit/e2e). Pack — approve `skill-new` ⇒ skill +resolvable with staged marker; `pack_activation` disable works; usage hook appends once +per load. *(RED)* + +**Acceptance:** green; pack-resolver and skills suites green unmodified. + +## AI-P2b — The Improver (human-in-the-loop learning live) + +**Outcome:** archiving a goal with correction signals produces a judged +`propose_improvement` in the panel, end-to-end, with a human deciding. + +**Owned files:** Improver role + skills + rubrics inside +`market-packs/mission-control/` (coordinate with MC-P4a — if it hasn't landed, this PR +creates the pack skeleton per mission-control.md Appendix A.5 and MC-P4a extends it); +trigger wiring only via existing staff machinery; NEW `tests/e2e/improver-loop.spec.ts`. + +**Steps** + +1. Improver role prompt = autoimprovement.md §1a heuristics **verbatim** (active stance, + corrections-first, umbrella shape, patch-first order 1→4) + standing-orders charter + (GA-R3 format) + §3 evidence-bundle requirements. Lives in the pack, never in server + source. +2. Staff template: `global: true`, role above, triggers `goal_archived` + `gate_failed` + (from GA-R2) + weekly `schedule`, all **disabled by default** (mission-control.md §5). +3. Tool whitelist via role `toolPolicies`: read/search tools + `propose_improvement` + ONLY (the Hermes fork-whitelist discipline, §1a). Pin: Improver role cannot call + write/bash tools. +4. Wire judge: `POST /api/improvements` triggers `judgeProposal` asynchronously; panel + shows confidence when present. + +**Tests:** API E2E with a mock agent (existing mock-agent harness): seed a session +transcript containing a planted correction ("stop doing X"); archive the goal; assert an +inbox entry fires, the Improver session's proposal appears with evidence excerpts +referencing the planted text, judge score attached; approving creates the staged skill +(AI-P2a path). Tool-whitelist pin: Improver session denied a bash call. *(RED)* + +**Acceptance:** the full observe→propose→judge→human-approve→staged-skill path green with +zero live-LLM dependence in CI (mock agent + mocked judge). + +--- + +## AI-P3a — Curator · AI-P3b — Dreaming *(contract level — re-verify anchors)* + +Contracts: autoimprovement.md §1b (Hermes semantics: deterministic transitions, bounded +LLM pass, never-delete, pin, snapshot+rollback, dry-run, first-run grace), §4 lifecycle +states, Appendix A.1 (lock/snapshot paths), A.4 (config defaults), A.5 (owned files). + +- **P3a:** `improvement-curator.ts` — `runCurator({dryRun})`: phase 1 deterministic + (usage-file derived `active→stale→archived`, timings from config; `pinned` exempt; + archive = move under `skill-archive/`, never delete); phase 2 bounded aux-model pass + emitting `skill-lifecycle`/`skill-patch` **proposals** (Bobbit divergence from Hermes: + no direct writes — §1 adopt/adapt table). tar.gz snapshot before any apply; REST: + pin/unpin/restore/dry-run + snapshot list/rollback; staff-page controls. First-run + seeds `lastRunAt = now`. Tests: fake-clock transitions; snapshot/rollback round-trip + (incl. rollback-of-rollback); dry-run mutates nothing (tree hash compare); pinned + untouchable. +- **P3b:** dreaming = Archivist staff scheduled wake whose gate check + (`time → new-transcripts-count → dream.lock`, autoDream ordering from §1) runs + server-side **before** the staff is woken (cheap gates outside the LLM); the wake mines + recent transcripts (read_session/search whitelist) and emits `memory`/`skill-new` + proposals. Tests: gates closed ⇒ no wake; lock prevents concurrent dream; dream run + writes a flight-recorder entry. + +**Acceptance:** staged skill ages stale→archived→restored across fake-clock runs; dreams +appear in the activity panel; zero direct skill mutations anywhere in P3 paths. + +## AI-P4 — Shadow calibration *(contract level)* + +Contract: §5 L0.5 + Appendix A.2 calibration endpoint. Judge runs on every proposal +(AI-P2b already does); `decideProposal` stores +`(confidence, wouldAutoApproveAt: thresholds[], humanDecision)`; `GET +/api/improvements/calibration?class` computes the per-threshold agreement matrix; Improver +staff page renders it ("at θ=0.85: 94% agreement, 1 false-approve / 31"). Tests: matrix +math unit tests with seeded decisions; endpoint e2e. **Zero auto-applies — pin it** (a +test that a high-confidence proposal still requires a human decision at level 0.5). + +## AI-P5 — Graduated autonomy *(contract level)* + +Contracts: §5 ladder + guard rails, Appendix A.4 keys, A.2 `/revert`. Server-side ceiling +clamp (config requesting L1 for `prompt-context` ⇒ rejected with 400 — pin); policy path +in `decideProposal` (`by: "policy"` now legal when class level + threshold + ceiling all +pass); apply with snapshot; flight-recorder entry `improvement.auto-approved` with +`revert` ref; inbox digest of auto-approvals; `POST /api/improvements/:id/revert` +restores snapshot + appends `reverted` + **demotes the class one level** (persisted in +config store); global kill switch (`autoimprovement.enabled: false` ⇒ everything behaves +as L0). Tests: e2e auto-apply happy path; revert demotes; ceiling clamp; kill switch; +every policy decision visible in activity feed. + +## AI-P6 — Measurement loop *(contract level)* + +Contract: §6. Scheduled evaluator pass (inside the curator run) computes +`helped/neutral/regressed` per applied change from: skill-usage + post-use goal/gate +outcomes, recurrence of the originating correction signal, cost deltas when CE-G0.1's +ledger exists (omit cleanly when absent). `regressed` ⇒ flight-recorder entry + automatic +demotion + an auto-generated revert proposal (NOT auto-revert). Test: planted regression +fixture (staged skill whose sessions then fail a gate) detected and demoted within one +evaluation window (fake clock). + +**Program acceptance (AI complete):** with levels at defaults (0) the system only ever +proposes; raising `skill-new` to 1 after shadow calibration auto-applies high-confidence +drafts, every one visible and revertible in the activity panel; killing the switch stops +all autonomy instantly. All suites green; no flaky tests. diff --git a/docs/design/autoimprovement.md b/docs/design/autoimprovement.md new file mode 100644 index 000000000..6c25d36de --- /dev/null +++ b/docs/design/autoimprovement.md @@ -0,0 +1,413 @@ +# Autoimprovement — recursive self-improvement with earned autonomy + +Status: design accepted, not started. This is the WHAT/WHY + contracts (Appendix A). + +> **Execution authority:** implement from +> [autoimprovement-implementation-plan.md](autoimprovement-implementation-plan.md) +> (per-goal owned files, symbol anchors, RED→GREEN tests, acceptance). Sequencing: +> [fable-program-execution-plan.md](fable-program-execution-plan.md). + +**What this is:** Bobbit learning from its own usage — drafting new skills after notable +sessions, patching skills that misfired, proposing AGENTS.md/prompt/config improvements, +consolidating memory — under a trust model where **autonomy is earned, never assumed, and +nothing happens off the record**. + +**Stance (binding):** + +1. **No self-modification without a human-visible artifact.** Every change — even an + auto-approved one — exists as a reviewed-or-reviewable proposal with a diff, an evidence + bundle, and a revert affordance. +2. **Human oversight is the *starting* posture, not the permanent one.** A confidence-scored + ladder (§5) moves low-risk change classes to auto-approval once the system's judgment has + been calibrated against the human's — and demotes them automatically on regression. +3. **Visibility is king.** Approved, auto-approved, rejected, and reverted changes all land + in the same audit surface (the Mission Control flight recorder, + [mission-control.md](mission-control.md) §6). +4. **Apply only through existing channels.** Skills/roles/tools land via the pack system and + config cascade; prompt context via the prompt-section assembly. Never raw file writes + outside those paths. + +Owner at runtime: the **Improver** system staff defined in +[mission-control.md](mission-control.md) §5 (this doc specifies *what* it runs; that doc +specifies *where it lives*). + +--- + +## §1 What the peers do (source study, 2026-06-10; checkouts per [harness-gap-analysis.md](harness-gap-analysis.md) §1) + +### Hermes — the reference self-improvement loop + +Two cooperating mechanisms: + +**(a) Per-turn background review** (`agent/background_review.py`). After every turn the agent +*forks itself* on a daemon thread — same provider/model/credentials/prompt-cache prefix — and +replays the conversation snapshot with two prompts: a **memory review** ("did the user reveal +persona/preferences/expectations worth remembering?") and a **skill review**. The fork runs +with a **tool whitelist limited to memory and skill management tools**; the main conversation +and its cache are never touched. The skill-review prompt is the crown jewel — its heuristics: + +- *Be active*: "most sessions produce at least one skill update… a pass that does nothing is + a missed learning opportunity". +- *User corrections are first-class skill signals* — "stop doing X", "too verbose", "you + always do Y and I hate it" get encoded into the governing skill, not just memory. +- *Target shape*: a small library of **class-level umbrella skills**, each with a rich + SKILL.md plus `references/` for session-specific detail — not "a long flat list of narrow + one-session-one-skill entries". +- *Patch-first preference order*: (1) update a skill that was actually loaded this session → + (2) update an existing umbrella → (3) add a support file under an umbrella → (4) only then + create a new skill. + +**(b) The curator** (`agent/curator.py`, +`website/docs/user-guide/features/curator.md`) — background maintenance so self-created +skills "don't pile up forever". Inactivity-triggered (interval ≥7 days **and** agent idle ≥2 h +— no cron daemon); runs on a configurable **auxiliary model slot** (`auxiliary.curator`). A +run = deterministic lifecycle transitions (`active → stale` after 30 d unused, `→ archived` +after 90 d) then one bounded LLM pass (max 8 iterations) that keeps / patches / consolidates / +archives. Safety furniture worth copying wholesale: **never auto-deletes** (archive is +recoverable); **pinned skills are untouchable**; **tar.gz snapshot before every pass** with +one-command rollback (and rollback-of-rollback); **`--dry-run`** produces the report with no +mutations; **first-run grace** — a fresh install defers the first real pass by one full +interval so the user can pin or opt out first. + +### Claude Code — gate discipline and away-summaries + +`services/autoDream/autoDream.ts` (background memory consolidation, "dreaming") fires a +forked subagent only after three gates pass **cheapest-first**: time gate (hours since +`lastConsolidatedAt`, one stat) → session-count gate (transcripts touched since) → a +consolidation **lock** (no concurrent dreamer), plus a 10-minute scan throttle so a passing +time-gate doesn't re-scan every turn. The dream runs as a first-class background task type +(`tasks/DreamTask/`) with a restricted `canUseTool`, visible in the task UI — dreaming is +*observable*, not hidden. Sibling services: `extractMemories` (per-session), `SessionMemory`, +and `awaySummary.ts` (small-fast-model catch-up — adopted via gap-analysis R4). +*(Mirror is ~2 months stale — verify details against current behavior before leaning on +specifics.)* + +### OpenClaw — authority, not learning + +No skill-learning loop; its contribution here is the **standing-orders convention** +(`docs/automation/standing-orders.md`): Scope / Trigger / Approval gates / Escalation rules as +the written form of delegated authority. Adopted as the authoring format for the Improver's +own charter (gap-analysis G4) — the trust ladder in §5 is, in effect, a machine-enforced +standing order. + +### Adopt / adapt / reject + +| Mechanism | Verdict | +|---|---| +| Per-turn review fork | **Adapt** — per-*session/goal* review instead of per-turn (Bobbit turns are expensive team workflows, not chat turns); tool-whitelisted; cheap aux model | +| Skill-review heuristics (active, corrections-first, umbrella shape, patch-first) | **Adopt verbatim** into the Improver's review prompt | +| Direct writes from the review fork (Hermes writes stores immediately) | **Reject** — violates stance #1; Bobbit's equivalent emits *proposals* | +| Curator lifecycle + snapshots + pin + dry-run + first-run grace + never-delete | **Adopt** | +| autoDream gate ordering + lock + visible task | **Adopt** for the dreaming schedule | +| Aux-model slots for review passes | **Adopt** (gap-analysis G11) | + +--- + +## §2 What can be improved (change classes) + +Every proposal carries exactly one **change class**; class determines blast radius, evidence +requirements, and the autonomy ceiling (§5). + +| Class | Examples | Blast radius | Autonomy ceiling | +|---|---|---|---| +| `memory` | user-profile entries, project facts | additive, bounded | L1 | +| `skill-new` | draft a new staged skill | additive (staged ≠ active) | L1 | +| `skill-patch` | patch/extend an existing **agent-created** skill | mutating, snapshot-backed | L2 | +| `skill-lifecycle` | stale/archive/consolidate (curator) | mutating, recoverable | L2 | +| `prompt-context` | AGENTS.md edits, role prompt tweaks, tool-description fixes | mutating, behavior-wide | L0 only (human always) | +| `tool-new` / `config` | new tool group, config cascade changes, pack updates | mutating–destructive | L0 only | + +Human-authored skills, pinned skills, and anything under `defaults/` or budget-pinned by tests +(e.g. `tests/tool-description-budget.test.ts`, `tests/agents-md-budget.test.ts`) are +**off-limits to auto-apply at every level** — proposals against them always require a human. + +## §3 The improvement loop + +```mermaid +graph LR + O[Observe
outcomes, corrections,
gate failures, cost] --> P[Propose
diff + evidence bundle
+ confidence score] + P --> R{Review} + R -->|"L0 / over-ceiling"| H[Human: proposal panel
approve / edit / reject] + R -->|"auto: conf ≥ threshold
(class at L1/L2)"| A + H --> A[Apply
via packs / config cascade
+ snapshot] + A --> F[Flight recorder
every outcome recorded] + A --> M[Measure
did it help?] + M -->|regression| D[Demote class
+ auto-revert option] + M -->|calibration data| R +``` + +- **Observe.** Signals, cheapest first: goal/gate outcomes (`gate_failed`, retries), + human-signoff rejections, user-correction heuristics in transcripts ("stop doing X", + "too verbose" — the Hermes signal list verbatim), skill load-but-failed events, cost ledger + outliers (token-cost CE-G0.1 when it lands). Transcripts are already retained + (`.bobbit/state/sessions/.jsonl`) and indexed (`src/server/search/flex-store.ts`). +- **Propose.** The Improver emits a structured proposal: change class, unified diff (or new + file tree), **evidence bundle** (transcript excerpts + signal that triggered it + expected + impact + rollback plan), and **confidence score**. Scoring is a *separate* cheap-model judge + pass against a per-class rubric — the proposer never grades its own homework. +- **Review.** Routed by class + ladder level (§5). Human review uses the existing proposal + panel with inline editing ([editable-proposals.md](editable-proposals.md), + [proposal-inline-comments.md](proposal-inline-comments.md)) — approve / edit-then-approve / + reject(+reason). Rejections with reasons are calibration gold (§5). +- **Apply.** Staged-skill writes go to the improvement pack (§4); config/prompt changes go + through the config cascade. Mutating classes snapshot first (curator-style tar.gz of the + affected tree under `.bobbit/state/improvement/snapshots/`). +- **Record.** Every transition (proposed, approved-by-human, approved-by-policy, rejected, + applied, reverted, demoted) appends to the flight recorder with actor + evidence link. +- **Measure.** §6. + +## §4 Skill lifecycle (where learned skills live) + +Learned skills are **pack-delivered, agent-created, and visibly second-class until promoted**: + +- A server-scoped **`learned-skills` pack** (created on first proposal; resolved via the + normal `PackResolver` scope order) holds them — *not* `.claude/skills/` (human-authored + space) and *not* `defaults/` (shipped space). Provenance is therefore structural: you can + always tell what the machine wrote, disable any of it per-entity via `pack_activation`, or + nuke the whole experiment by removing one pack. +- States: `draft` (proposal pending) → `staged` (approved; catalog-visible to agents with a + `[learned — staged]` marker; usage tracked) → `active` (promoted after N successful uses + with no negative signal) → `stale` → `archived` (curator transitions, Hermes timings as + defaults: 30 d / 90 d). `pinned` exempts from all auto-transitions. Archive = + `.bobbit/state/improvement/skill-archive/`, never deletion. +- Shape discipline: the Improver's prompt enforces Hermes' umbrella rule — patch-first, few + class-level skills with `references/`, no one-session confetti. +- Dedup: before drafting, the Improver must search existing skills/packs + (`slash-skills.ts` catalog + marketplace) and prefer `skill-patch`. + +## §5 Trust ladder — confidence-scored, calibrated, demotable + +Autonomy is granted **per change class**, not globally: + +| Level | Meaning | +|---|---| +| **L0** | Every proposal human-reviewed. The starting posture for all classes. | +| **L0.5 shadow** | Still human-reviewed, but the system *records* "would have auto-approved at threshold θ" alongside each human decision. After ≥20 decisions in a class, the calibration report shows: at θ, X% of auto-approvals would have matched the human, Y false-approves. The user promotes a class only when shown that evidence. | +| **L1** | Auto-approve `memory` / `skill-new` (additive classes) when `confidence ≥ θ_class`. Applied immediately; flight-recorder entry reads "approved by policy (0.93 ≥ 0.85)" with one-click revert; a digest of auto-approvals lands in the user's inbox (visibility without interruption). | +| **L2** | Adds mutating classes (`skill-patch`, `skill-lifecycle`) with mandatory pre-apply snapshot + auto-generated revert proposal attached. | + +Guard rails (all enforced in code, not prompt): + +- `prompt-context`, `tool-new`, `config` classes **cannot** be promoted past L0 (table §2). +- **Kill switch**: one settings toggle drops everything to L0 instantly; the Improver staff + pausing (existing staff machinery) halts the loop entirely. +- **Automatic demotion**: a human revert of an auto-approved change, or a measured regression + (§6), drops that class one level and requires re-calibration to climb back. +- Thresholds `θ_class` are user-tunable in settings; defaults conservative (0.85 additive, + 0.9 mutating). + +## §6 Measurement (closing the loop) + +Each applied change records its *expected impact* (from the evidence bundle) and is evaluated +against observed signals over a trailing window: skill usage count + post-use outcome (goal +gate pass/fail in sessions that loaded it), correction-signal recurrence (did "too verbose" +complaints stop?), and cost deltas once CE-G0.1 lands. Outcomes: `helped` / `neutral` / +`regressed` → flight recorder; `regressed` → demotion + revert proposal. Honest scope: v1 +measurement is heuristic, not causal — the point is a feedback signal and demotion trigger, +not a science experiment. + +## §7 Phased implementation plan + +Run the review/judge/dream passes on named aux-model slots from day one +(gap-analysis G11; config shape mirrors [per-role-model-overrides.md](per-role-model-overrides.md)). +Phases assume mission-control.md P0 (global scope) + P4 (flight recorder) exist; P1–P2 below +can land before Mission Control's UI by writing flight-recorder entries only. + +### AI-P1 — Proposal substrate + signal plumbing + +1. Define `ImprovementProposal` (class, diff/files, evidence bundle, confidence, status) in a + new `src/server/agent/improvement-store.ts` (JSONL under `.bobbit/state/improvement/`, + crash-safe write pattern per `session-store-crash-safety.md`). +2. New proposal tool `propose_improvement` under `defaults/tools/proposals/` following the + `propose_staff` pattern; render in the proposal panel via the existing proposal registry + (`src/app/proposal-registry.ts`, parsers in `proposal-parsers.ts`) with diff view + inline + edit + approve/reject(+reason). +3. Push triggers `gate_failed` / `session_errored` (shared with gap-analysis R2) so the + Improver can subscribe. +4. REST: `GET/POST /api/improvements`, `POST /api/improvements/:id/decision`. +5. Tests: unit (store round-trip, class validation), API e2e (decision flow), browser e2e + (panel renders evidence + diff; reject persists reason). + +Acceptance: a hand-fed proposal flows draft → human decision → recorded; nothing auto-applies. + +### AI-P2 — The Improver staff + post-goal review (Hermes loop, proposal-shaped) + +1. Improver ships in the mission-control pack roster (mission-control.md §5): role prompt = + Hermes review heuristics (§1a verbatim: active, corrections-first, umbrella shape, + patch-first) + standing-orders charter; triggers: `goal_archived`, `gate_failed`, weekly + `schedule`. +2. Review run = tool-whitelisted session (read_session/search + `propose_improvement` only — + enforce via role `toolPolicies`). +3. `learned-skills` pack bootstrap (§4): server-scoped pack created on first approved + `skill-new`; staged-skill marker in the skills catalog; usage events recorded (hook the + skill-load path in `slash-skills.ts`). +4. Judge pass: separate cheap-model scoring call with per-class rubric prompts (stored beside + the role in the pack, not hardcoded). +5. Tests: e2e — archive a goal with a planted correction signal ⇒ a `skill-new` proposal with + evidence appears; approving it makes the skill resolvable with staged marker. + +Acceptance: end-to-end learn-from-session path works with a human in the loop at every step. + +### AI-P3 — Curator + dreaming + +1. Curator run (Hermes semantics): deterministic lifecycle transitions + bounded LLM pass + emitting `skill-lifecycle`/`skill-patch` proposals; tar.gz snapshot before any apply; + `pin`/`restore`/`dry-run` via REST + staff-page UI; first-run grace (seed `lastRunAt` on + creation). +2. Dreaming = scheduled Archivist/Improver job with autoDream's gate ordering (time → new + transcript count since last dream → lock file under `.bobbit/state/improvement/`), mining + transcripts for `memory` + `skill-new` proposals; runs as a visible session (Bobbit's + equivalent of DreamTask — it's just a staff wake, already visible in the sidebar). +3. Tests: lifecycle transition unit tests with fake clocks; snapshot/rollback round-trip; + dream gates (no run when gates closed; lock prevents concurrency). + +Acceptance: an unused staged skill ages to stale/archived and back via restore; dreams run +only when gates pass and appear in the flight recorder. + +### AI-P4 — Shadow mode + calibration (L0.5) + +1. Judge runs on every proposal; store `(confidence, wouldAutoApprove@θ, humanDecision)`. +2. Calibration report: `GET /api/improvements/calibration?class=` → agreement matrix per + threshold; surfaced on the Improver staff page ("at θ=0.85: 94% agreement, 1 + false-approve / 31"). +3. Tests: calibration math unit tests; e2e for report endpoint. + +Acceptance: after seeded decisions, the report shows per-class agreement; still zero +auto-applies. + +### AI-P5 — Graduated autonomy (L1/L2) + +1. Settings: per-class level + threshold (enforced server-side against §2 ceilings); global + kill switch. +2. Auto-approval path: policy check → apply → flight-recorder "approved by policy" entry → + inbox digest entry → revert affordance (`POST /api/improvements/:id/revert`, snapshot + restore). +3. Automatic demotion on revert/regression. +4. Tests: e2e — class at L1 + high confidence ⇒ auto-applied + recorded + revertible; revert + ⇒ class demoted; ceiling classes can never be set to L1 (server rejects). + +Acceptance: autonomy on, fully audited, reversible, demotable; kill switch verified. + +### AI-P6 — Measurement loop + +Outcome evaluation per §6 wired into a scheduled Improver pass; `helped/neutral/regressed` +recorded; regression triggers demotion + revert proposal. Acceptance: planted regression +(skill that makes a gate fail) is detected and demoted within one evaluation window. + +--- + +## §8 Open questions (for review, not blockers) + +- Should staged skills be visible to *all* sessions or only sessions in the project where the + signal originated? (Lean: global, since the learned-skills pack is server-scoped; per-entity + `pack_activation` already gives opt-out.) +- Per-turn review (Hermes) vs per-goal review (this doc): per-goal is the right starting + cadence for cost reasons, but chat-style staff sessions may deserve a per-session review on + sleep. Revisit with AI-P2 data. +- Judge model quality: if cheap-model confidence proves noisy in shadow mode, calibration + will show it before any autonomy is granted — that's the point of L0.5. + +Cross-references: [mission-control.md](mission-control.md) (Improver/Archivist staff, flight +recorder, inbox digests) · [harness-gap-analysis.md](harness-gap-analysis.md) G1/G2/G11 · +[time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) (cost ledger CE-G0.1, aux summarizer) · +[skill-ux-and-autonomous-activation.md](skill-ux-and-autonomous-activation.md) · +[editable-proposals.md](editable-proposals.md) · [human-signoff-gates.md](human-signoff-gates.md). +Execution tracking: [fable-program-execution-plan.md](fable-program-execution-plan.md). + +--- + +## Appendix A — Implementation contracts (definite lists; do not re-derive) + +Universal definition-of-done: +[extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) +binds every AI phase. + +### A.1 Types (AI-P1, `src/server/agent/improvement-store.ts`) + +```ts +export type ChangeClass = + | "memory" | "skill-new" | "skill-patch" | "skill-lifecycle" + | "prompt-context" | "tool-new" | "config"; // §2 table is the law for ceilings + +export type ProposalStatus = + | "draft" | "approved-human" | "approved-policy" | "rejected" + | "applied" | "reverted" | "superseded"; + +export interface ImprovementProposal { + id: string; // nanoid, same generator as session ids + ts: string; // ISO-8601 UTC + changeClass: ChangeClass; + title: string; // one line, imperative + diff?: string; // unified diff for patches + files?: { path: string; content: string }[]; // for skill-new (paths relative to the learned-skills pack root) + evidence: { + signal: string; // which observer signal fired (e.g. "user-correction", "gate_failed") + sessionIds: string[]; + excerpts: string[]; // verbatim transcript quotes, ≤ 500 chars each, ≤ 5 + expectedImpact: string; + rollbackPlan: string; + }; + confidence?: number; // 0..1, judge-assigned; absent until judged + judge?: { model: string; rubricVersion: string; wouldAutoApproveAt: number[] }; // thresholds it clears + status: ProposalStatus; + decision?: { by: "human" | "policy"; ts: string; reason?: string; editedBeforeApprove?: boolean }; + applied?: { ts: string; snapshotRef?: string }; + outcome?: "helped" | "neutral" | "regressed"; // AI-P6 +} +``` + +Storage: `.bobbit/state/improvement/proposals.jsonl` (append-only; status changes append a +new full record with same `id` — last-writer-wins on load, crash-safe per +[session-store-crash-safety.md](session-store-crash-safety.md)). Snapshots: +`.bobbit/state/improvement/snapshots//…tar.gz`. Skill archive: +`.bobbit/state/improvement/skill-archive/`. Dream lock: +`.bobbit/state/improvement/dream.lock` (mtime = lastDreamAt; same gate semantics as +autoDream §1). + +### A.2 REST (AI-P1/P4/P5) + +| Route | Notes | +|---|---| +| `GET /api/improvements?status&class&limit` | newest-first | +| `POST /api/improvements` | from the `propose_improvement` tool path only | +| `POST /api/improvements/:id/decision` | body `{approve: boolean, reason?, editedFiles?}` | +| `POST /api/improvements/:id/revert` | restores snapshot, appends `reverted`, demotes class | +| `GET /api/improvements/calibration?class` | `{class, decisions: n, perThreshold: [{theta, agreement, falseApproves}]}` | + +### A.3 Tool + prompts + +`defaults/tools/proposals/propose_improvement.yaml` — params: +`change_class, title, diff?, files?, evidence` (mirror the `propose_staff` YAML shape; +provider `bobbit-extension`). The Improver's review prompt, judge rubrics (one per +`ChangeClass`, versioned `rubricVersion: "1"`), and the dream prompt live in the +mission-control pack: `market-packs/mission-control/skills/improver/…` — **never hardcoded +in server source**, so prompt iteration is a pack update, not a release. + +### A.4 Settings keys (AI-P5; config cascade) + +```yaml +autoimprovement: + enabled: true # global kill switch — false drops everything to L0 behavior + levels: # server clamps to §2 ceilings regardless of config + memory: 0 # 0=L0, 0.5=shadow, 1=L1, 2=L2 + skill-new: 0 + skill-patch: 0 + skill-lifecycle: 0 + thresholds: { memory: 0.85, skill-new: 0.85, skill-patch: 0.9, skill-lifecycle: 0.9 } + curator: { intervalHours: 168, minIdleHours: 2, staleAfterDays: 30, archiveAfterDays: 90 } + dream: { minHours: 24, minSessions: 5 } +auxiliary: # gap-analysis G11; resolution mirrors per-role-model-overrides.md + improver-judge: { model: auto } + dreamer: { model: auto } +``` + +### A.5 Owned files per phase (PR boundaries) + +| Phase | New | Modified | +|---|---|---| +| AI-P1 | `improvement-store.ts`, `defaults/tools/proposals/propose_improvement.yaml` (+extension wiring), proposal-panel parser/view for kind `improvement` | `server.ts` (routes), `proposal-registry.ts`, `proposal-parsers.ts`, `goal-trigger-dispatcher.ts` (push triggers — coordinate with MC-P4/GA-R2, land once) | +| AI-P2 | improver role/skills in mission-control pack; judge module `src/server/agent/improvement-judge.ts`; learned-skills pack bootstrap in `src/server/agent/learned-skills-pack.ts` | skill-load usage hook in `slash-skills.ts` | +| AI-P3 | curator module `src/server/agent/improvement-curator.ts` (+snapshot/rollback helpers) | archivist role (pack), staff page curator controls | +| AI-P4 | calibration math in `improvement-store.ts` + report endpoint | Improver staff page section | +| AI-P5 | — | settings page, `improvement-store.ts` (policy path), `activity-store.ts` writers, inbox digest | +| AI-P6 | outcome evaluator in `improvement-curator.ts` | — | diff --git a/docs/design/client-performance-battery-implementation-plan.md b/docs/design/client-performance-battery-implementation-plan.md new file mode 100644 index 000000000..9dd9f5c5f --- /dev/null +++ b/docs/design/client-performance-battery-implementation-plan.md @@ -0,0 +1,190 @@ +# Client Performance & Battery — Implementation Plan (hand-off) + +Status: ready for execution, not started. Workstream **PB** in +[fable-program-execution-plan.md](fable-program-execution-plan.md). + +Companion to [client-performance-battery.md](client-performance-battery.md) (the WHAT/WHY — +read §1–§6 and **Appendix A**; its A.1 animation table and A.4 poll table are the definite +work lists). + +> **Anchor baseline:** fable-docs @ 2026-06-11 (master parent `6ec8c8f9`). Locate by symbol +> name; missing symbol ⇒ STOP, re-derive from the cited pattern. +> +> **Precision policy:** all goals file/function level (this workstream touches few files). +> +> **Universal rules:** [extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) +> + [fable-program-execution-plan.md §1](fable-program-execution-plan.md). **Seam guard:** +> PB-P2a must land AFTER SN-T2/T3, and PB-P2c after (or rebased onto) SN-T5 — see the +> execution plan §1.4 table; check the §4 checklist before starting those goals. +> +> **Measurement discipline:** no behavior-affecting goal merges without before/after +> numbers from the PB-P0 harness in the PR description, and the doc's §7 table updated in +> the same PR. + +--- + +## Goal map + +``` +PB-P0 harness ─→ PB-P1a anim gating ─→ PB-P1b paint-prop rewrites + └─→ PB-P2a poll demotion (after SN-T2/T3) · PB-P2b scoped tick · + PB-P2c render throttle (after SN-T5) · PB-P2d timer audit +PB-P1 + PB-P2 ─→ PB-P3 battery saver + flag soak + re-baseline +``` + +--- + +## PB-P0 — Measurement harness + +**Outcome:** `window.__bobbitPerf.report()` returns the Appendix A.3 shape in all four §4 +protocol states; baseline table committed. Zero default-path behavior change. + +**Owned files:** NEW `src/app/perf-monitor.ts`; `src/app/state.ts` (ONE guarded line in +`renderApp()`, symbol at `state.ts:687`); `src/app/perf-flags.ts` (flag registration only); +doc §7 table; NEW `tests/perf-monitor.spec.ts` (browser fixture). + +**Steps** + +1. `perf-monitor.ts` per Appendix A.3: render counter (incremented from the guarded hook + inside `renderApp`'s rAF callback — read how `_renderScheduled` works first), timer + wrap (monitor mode only: wrap `window.setInterval`/`setTimeout`, bucket by + `new Error().stack` top app frame), `PerformanceObserver longtask`, + `document.getAnimations().length` sampled 1/s, WS frames/s by type (hook the client WS + dispatch — find the single switch over frame types in `src/app/`). +2. Activation: only when `bobbitPerfFlags` contains `+perfMonitor` + (`isPerfFlagEnabled` — read `perf-flags.ts` and `docs/perf/defer-offscreen-render.md` + for the conventions). Module must not be imported on the default boot path — lazy + `import()` behind the flag check in `main.ts`. +3. Run the §4 protocol (4 states × 60 s × 3 runs × dev+prod) **manually**, fill the doc §7 + baseline columns, include the OS-level energy reading (§4 step 4). + +**Tests:** browser fixture — flag off ⇒ `window.__bobbitPerf` undefined *(GREEN pins the +default path)*; flag on ⇒ report returns every A.3 field with sane types *(RED)*. + +**Acceptance:** baseline table filled in the same PR; `npm run test:unit` green. + +--- + +## PB-P1a — Ambient animation gating + +**Outcome:** idle-or-hidden Bobbit runs **zero** animations +(`document.getAnimations().length === 0`); working agents still animate. + +**Owned files:** NEW `src/app/animation-power.ts`; `src/app/main.ts` (one init call); +`src/app/app.css`, `src/app/goal-dashboard.css`, `src/ui/app.css` (gates per the A.1 +table); NEW `tests/e2e/ui/animation-power.spec.ts`, `tests/animation-gate.test.ts`. + +**Steps** + +1. Re-run the A.1 grep; reconcile drift (new rules ⇒ classify + extend the table in this + PR — execution plan §1.2). +2. `animation-power.ts` per Appendix A.2 **exactly**: inputs are `visibilitychange`, the + session-status update path (find where `state.gatewaySessions` statuses are written — + `api.ts` `updateLocalSessionStatus` and the `refreshSessions` apply step — and call a + re-evaluate from there), `prefers-reduced-motion` media query listener, and (later) + battery-saver. Output: toggle `anims-paused` on `document.documentElement`. No timers. +3. CSS: for every **ambient** row in A.1, add the pause guard + (`html.anims-paused { animation-play-state: paused; }` grouped in one block + per file); for every **status** row, move/confirm the animation under its + working-state selector so it doesn't run at idle (the `blob-*-idle-sleep-breathe` + loops are the flagship: idle/sleep sprites become static frames). Remove `!important` + from `blob-compact-squash*` while adding gates. NEVER a bare `html.anims-paused *` + rule (A.2 freeze hazard). +4. Flag: `-pauseAmbientAnims` restores current behavior (gate the class-toggle, not the + CSS). + +**Tests (author first):** + +- `tests/animation-gate.test.ts` (node): parse the three CSS files; every + `animation:.*infinite` rule must match an A.1 bucket: ambient ⇒ a pause guard exists + for its selector; status ⇒ selector includes a state class; else FAIL ("the missing + test IS the bug" — this is the never-reintroduce pin). *(RED)* +- Browser E2E: all-idle fixture ⇒ class present + `getAnimations()` = 0; mock agent + working ⇒ its sprite animates and count > 0; tab hidden ⇒ 0; flag opt-out restores. + *(RED)* + +**Acceptance:** E2E + pin green; harness shows idle-foreground animations 0 and paint +events <2/s; visual check: working sprites unchanged (record before/after GIF in PR). + +## PB-P1b — Paint-property rewrites + reduced-motion sweep + +**Owned files:** `src/app/app.css` only. **Steps:** rewrite `status-pulse`, +`unseen-dot-pulse`, `pr-conflict-pulse` to opacity/transform on a pseudo-element halo +(FX3 — visually equivalent: pre-rendered shadow layer scaling/fading); add the blanket +reduced-motion rule (FX4) and delete the now-redundant piecemeal opt-outs (grep +`prefers-reduced-motion`, currently 9 sites). **Tests:** extend `animation-gate.test.ts`: +no `@keyframes` in the three files may animate `box-shadow` *(RED until rewritten)*. +**Acceptance:** pin green; screenshots before/after in PR. + +--- + +## PB-P2a — Session-poll demotion *(after SN-T2/T3 — verify §4 checklist first)* + +**Owned files:** `src/app/api.ts` (`startSessionPolling`, symbol near `api.ts:273`, plus +the WS-health signal); NEW `tests/session-poll-policy.test.ts`. + +**Steps:** extract interval selection into a pure +`sessionPollIntervalMs({wsConnected, visible})` → 60 000 when WS healthy, 5 000 otherwise +(read how the client tracks WS connection state first — reuse, don't add a tracker); +immediate `refreshSessions()` on WS reconnect and on `visibilitychange`→visible (the +git-status catch-up pattern, `session-manager.ts` near `startGitStatusPoll`). Rebase onto +SN-T2's coalescing/generation guard — do not reimplement any part of it. + +**Tests:** pure-function unit table *(RED)*; existing +`tests/spurious-idle-unread.spec.ts` green unmodified. **Acceptance:** harness: +idle-foreground network ≤1 sessions fetch/min with WS up; status updates still arrive +(WS push path) — E2E asserts a status change renders without a poll tick (fake timers). + +## PB-P2b — Scoped verification tick + +**Owned files:** `src/app/goal-dashboard.ts` (`startLiveVerifTimer`, symbol near `:1090`). +**Steps:** the 1 s `setInterval(() => renderApp(), 1000)` becomes a scoped updater writing +`textContent` on the elapsed-time node(s) only (give them a stable +`data-testid="verif-elapsed"`); status *transitions* still arrive via the existing WS/event +path and call `renderApp()` as today; keep `stopLiveVerifTimerIfDone` semantics. +**Tests:** browser E2E — while a mock verification runs, elapsed label advances ≥2 s while +the perf-monitor render counter advances 0 in the same window; timer stops when the run +ends *(RED)*. **Acceptance:** dashboard suites green unmodified. + +## PB-P2c — Streaming render throttle *(after/rebased on SN-T5)* + +**Owned files:** `src/app/state.ts` (add `renderAppThrottled(ms = 100)` beside `renderApp`, +reusing `_renderScheduled`/suppression machinery — trailing edge, max one render per +window); the streaming dispatch call sites (`message_update`-class frames in the client WS +switch; terminal frames keep plain `renderApp()`). Flag `-throttleStreamRender`. +**Tests:** unit — burst of N calls in 100 ms ⇒ 1 render, trailing call lands; terminal +frame flushes immediately *(RED)*. Browser E2E — long mock stream: perf-counter ≤15 +renders/s; `follow-tail` suites green unmodified. **Acceptance:** harness streaming state +~10 Hz; no visible degradation (manual check noted in PR). + +## PB-P2d — Timer audit + +**Owned files:** convicted loops only (expect `goal-dashboard.ts`, `dialogs.ts`). +**Steps:** run PB-P0's wakeup-by-callsite report in the idle + dashboard states; commit the +table into the doc (§F2 update, same PR); every surviving loop ≥1 tick/min gets the +git-status pattern (visibility gate + event coalescing + self-cancel). One PR may fix +several loops if each has its own test. **Acceptance:** idle-foreground wakeups <8/min on +the harness; doc table updated. + +--- + +## PB-P3 — Battery saver + flag soak + re-baseline + +**Owned files:** settings page (pattern `tests/e2e/ui/settings.spec.ts`), +`animation-power.ts` (battery-saver input), `api.ts`/`state.ts` (cadence overrides), doc +§7 after-table; flag removals per `docs/perf/defer-offscreen-render.md` §"when to remove". + +**Steps:** (1) `batterySaver` setting (persisted with other UI settings): forces +`anims-paused`, poll 120 s, throttle 250 ms, shimmer off. (2) `navigator.getBattery()` +suggestion banner when discharging — suggest only, never auto-enable; dismissal persists. +(3) Remove soaked temporary flags (keep `anims-paused` as mechanism). (4) Re-run the full +§4 protocol; fill §7 after-columns + OS energy reading; only then evaluate whether any §3 +target is still missed (heavier options stay out of scope unless data convicts — doc §8). + +**Tests:** browser E2E — toggle persists across reload; saver state pauses animations +while a mock agent works (overrides status animations); banner dismissal persists. *(RED)* + +**Acceptance:** §3 targets table met on the harness (idle-foreground: 0 animations, +<2 paint/s, <8 wakeups/min, ≈0% CPU; hidden: 0 renders, ≤1 poll/min; streaming ~10 Hz, +no >50 ms render long-tasks; dashboard: no full-tree render/s) — or a doc deviation note +explaining the miss and the follow-up goal. diff --git a/docs/design/client-performance-battery.md b/docs/design/client-performance-battery.md new file mode 100644 index 000000000..5b126d8fe --- /dev/null +++ b/docs/design/client-performance-battery.md @@ -0,0 +1,422 @@ +# Client performance & battery — analysis and implementation plan + +Status: design accepted, not started. This is the WHAT/WHY + contracts (Appendix A). + +> **Execution authority:** implement from +> [client-performance-battery-implementation-plan.md](client-performance-battery-implementation-plan.md) +> (per-goal owned files, symbol anchors, RED→GREEN tests, measurement discipline). +> Sequencing: [fable-program-execution-plan.md](fable-program-execution-plan.md). + +**The symptom this answers:** Bobbit visibly drains laptop battery — noticeably worse than +leaving a typical web app open. The goal: an *idle* Bobbit tab should cost ≈0% CPU with zero +continuous repaints, and an *active* one should stay smooth even on machines without dedicated +GPU acceleration (soft constraint, not a hard gate). + +**The one-paragraph answer:** The server side was already fixed (see the +[reduce-server-cpu series](reduce-server-cpu-performance-report.md) — idle gateway ≈ 0% CPU), +so the drain is client-side, and it is structural, not one bug: **the UI is never idle.** +Dozens of `infinite` CSS animations (sprites, accessories, pulses, shimmers) keep the +compositor — and on non-GPU machines, the CPU rasterizer — awake 24/7 even when nothing is +happening; two of the pulse animations animate `box-shadow`, which repaints on the main thread +every frame; a 5-second REST poll re-fetches the session list the WebSocket already pushes; +a 1-second timer re-renders the *entire app* while any verification runs; and every WebSocket +streaming delta schedules a full root re-render (rAF-coalesced, so up to ~60 full lit-html +tree evaluations per second during streaming). None of these is individually fatal; together +they mean the browser never reaches its power-saving render-idle state. The plan below is +**instrumentation-first** (same discipline as +[time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md)): we build a small repeatable measurement +harness, capture a baseline, then land tiered fixes — each behind a +[`perf-flags.ts`](../../src/app/perf-flags.ts)-style kill switch and re-measured against the +baseline. + +Related: [docs/perf/defer-offscreen-render.md](../perf/defer-offscreen-render.md) (prior art: +measured, flag-gated, default-on), [docs/perf/bundle-profile.md](../perf/bundle-profile.md), +[docs/design/shrink-initial-bundle.md](shrink-initial-bundle.md). + +--- + +## §1 Why "battery" ≠ "CPU benchmark" + +Battery drain on modern laptops is dominated by how often the machine is *prevented from +sleeping its silicon*, not by average CPU%. Three distinct mechanisms matter here: + +1. **Render-loop liveness.** Any running CSS animation — even a compositor-only + `transform`/`opacity` one — forces the browser to produce frames continuously. The GPU (or + CPU rasterizer without GPU acceleration) never idles, and the display pipeline never drops + to its low-power state. Cheap per frame × 60 fps × every hour the tab is open = the single + biggest battery line item for "decorative" animation. +2. **Main-thread paint.** Animating paint-affecting properties (`box-shadow`, `background`, + layout properties) repaints on the main thread every frame. Strictly worse than (1). +3. **Timer wakeups.** Every `setInterval` tick wakes the process, runs JS, possibly fetches + over the network (radio/Wi-Fi power-up), and possibly re-renders. Browsers throttle timers + in *hidden* tabs but not in visible-but-idle ones. + +The fix plan maps 1:1 onto these: stop animating when nothing is happening (§6 P1), never +animate paint properties (§6 P1), and make timers event-driven or visibility/idle-gated +(§6 P2). + +--- + +## §2 Findings (verified in source, 2026-06-10) + +### F1 — Ambient `infinite` animations never stop (HIGH, the headline) + +Inventory (grep `animation:.*infinite` across `src/**/*.css`): + +- **`src/ui/app.css` (sprite/chat layer)** — ~25 infinite rules: `blob-busy-shadow` (10s), + `blob-crown-idle-sleep-breathe` / `blob-bandana-idle-sleep-breathe` (3.6s), + `blob-compact-squash` / `blob-compact-squash-rigid` (3s, several marked `!important`), + `magnifier-depth-idle` (10s `steps(1)`), `wand-sparkle-a/b`, `flask-bubbles`, + `ask-heartbeat` (1.8s), two `shimmer` variants. +- **`src/app/app.css` (shell layer)** — `bobbit-bob`, `bobbit-breathe`, `bobbit-eyes` (+ `-s` + sidebar variants), `status-pulse` (2s), `unseen-dot-pulse` (1.8s), `gentle-float` (4s), + `bobbit-unread-tap` (1.6s), `bobbit-sidebar-unread-blink` (3.4s `steps(1)`). + +Two important nuances (so the fix targets the right thing): + +- The sprite keyframes themselves are **well-built**: `bobbit-eyes`, `blob-busy-shadow` etc. + animate `transform`/`opacity` only (verified at `src/app/app.css:439`, + `src/ui/app.css:2693`) — compositor-friendly, no per-frame paint. The cost is **liveness** + (§1.1): they run `infinite`, on *every* visible sprite (sidebar shows one per session), at + all times — including when every agent is idle and the user is just reading. +- `status-pulse` and `unseen-dot-pulse` **do animate `box-shadow`** (§1.2): main-thread + repaint every frame, forever, for each status dot / unseen indicator on screen. + +`prefers-reduced-motion` is honored in only **9** CSS sites — e.g. `.unseen-dot` and +`.bobbit-unread-pulse` opt out, but the sprite system, shimmers, and pulses largely don't. + +### F2 — Polling alongside a push channel (MEDIUM-HIGH) + +- **Session list poll** — `src/app/api.ts::startSessionPolling()` (`api.ts:273`) runs + `refreshSessions()` every **5 s** whenever the app is authenticated and visible. The + gateway *already pushes* `session_status` frames over the WebSocket (the same file handles + them ~20 lines above). So in steady state the poll is pure redundancy: a REST round-trip + + state diff + `renderApp()` every 5 s, all day. It exists as a safety net for missed frames — + the right shape is "slow fallback, fast on reconnect" (§6 P2.1). +- **Live verification timer** — `src/app/goal-dashboard.ts:1090` `startLiveVerifTimer()`: + `setInterval(() => renderApp(), 1000)` while any verification step is `running`. A full app + re-render every second, to advance an elapsed-time label. (It does stop when no step is + running — `stopLiveVerifTimerIfDone()` — but verifications can run for many minutes.) +- **Relative-clock timer** — `src/app/render-helpers.ts:338`: `renderApp()` every 60 s. Fine + in isolation; included in the wakeup budget. +- **Good pattern already in-tree** — the git-status poll + (`src/app/session-manager.ts:2821`) is the model citizen: 30 s tick, visibility-gated, + coalesced against event-driven refreshes (`gitStatusLastRefreshAt`, 10 s min spacing), + self-cancelling when the session changes or the repo is known-absent, with a + `visibilitychange` catch-up refresh. P2 below converges the other pollers on this shape. +- ~92 `setInterval`/`setTimeout`-loop sites exist across `src/app` (top files: + `goal-dashboard.ts` ×24, `dialogs.ts` ×16, `session-manager.ts` ×11, `api.ts` ×10). Most + are benign one-shots; the audit in P0 classifies them so we fix the loops, not the noise. + +### F3 — Every event is a full-tree render (MEDIUM) + +`renderApp()` (`src/app/state.ts:687`) is rAF-coalesced (good — multiple calls per frame +collapse to one) but each invocation re-evaluates the **entire root lit-html template** +(sidebar + chat + panels). There are **646** `renderApp()` call sites; during agent streaming, +every `message_update` WS frame schedules one, so a streaming session runs a full-tree +evaluation at up to the frame rate, continuously, for minutes. lit-html diffing makes each +pass cheap-ish, but "cheap × 60/s × whole tree" is exactly the §1.1 problem in JS form. The +server-side coalescing experiment was rejected +([reduce-server-cpu-experiment-stream-coalescing.md](reduce-server-cpu-experiment-stream-coalescing.md)) +— the client is the right layer: batch *visual* updates without delaying state updates. + +### F4 — Dev mode amplifies everything (LOW, but affects the owner's perception) + +`npm run dev:harness` runs vite dev (esbuild transforms, HMR socket, unminified lit) plus the +restart harness. The baseline (§4) must be captured on a **production build** (`npm run +build` + gateway) *and* dev mode, separately, so we know how much of the perceived drain is +dev-only and don't chase ghosts. + +### F5 — What was ruled out + +- Gateway CPU: already measured ≈0% idle (`reduce-server-cpu-baseline.md`); not the drain. +- `deferOffscreenRender` already prevents offscreen transcript paint on activation; long + transcripts are not the idle-drain cause. + +--- + +## §3 Targets (acceptance bar for the whole effort) + +Measured on a production build, mid-range laptop, all four states from §4: + +| State | Target | +|---|---| +| Idle-foreground (agents idle, user reading) | 0 continuous animations running; <2 paint events/s; <8 timer wakeups/min; CPU ≈0% | +| Hidden tab | 0 animations, 0 renders, ≤1 network poll/min | +| Streaming (one active agent) | visual update cadence ~10 Hz (not 60); no long tasks >50 ms from render | +| Dashboard open, verification running | scoped second-tick (no full-tree render/s) | + +"Battery-saver mode" (§6 P3.3) tightens these further on demand. + +--- + +## §4 Phase P0 — Measurement harness (build BEFORE any fix) + +Goal: a repeatable, scriptable protocol that outputs one comparison table per run. Pattern: +the server-side benchmark harness (`reduce-server-cpu-benchmark.md`) and `boot-timing.ts`. + +Implementation steps: + +1. **Add `src/app/perf-monitor.ts`** (debug-only, loaded when + `localStorage.bobbitPerfFlags` contains `+perfMonitor` — reuse the + `isPerfFlagEnabled`/`perf-flags.ts` mechanism). It records, over a sampling window: + - renders/s: increment a counter inside `renderApp()`'s rAF callback (one-line hook, + guarded by the flag); + - timer wakeups/s: in monitor mode only, wrap `window.setInterval`/`setTimeout` to count + scheduled-callback executions by call-site stack bucket; + - long tasks: `PerformanceObserver({ type: "longtask" })`; + - running animations: `document.getAnimations().length` sampled 1/s (this is the direct + "is the page render-idle" probe); + - WS frames/s by frame type (hook the existing WS message dispatch). + Expose `window.__bobbitPerf.report()` returning JSON; the + [`client-debug` skill](../../.claude/skills/client-debug/SKILL.md) is the retrieval path. +2. **Define the four states** as a manual-but-scripted protocol (a new + `docs/perf/battery-protocol.md` section or appendix in this doc): idle-foreground, + hidden, streaming (one mock-agent session — reuse the E2E mock agent), dashboard+verification. + 60 s sample each, three runs, production build and dev build. +3. **Record the baseline table** into this doc (§7 placeholder) — same convention as the + shared baseline table in `reduce-server-cpu-performance-report.md`. +4. OS-level cross-check (manual, documented): macOS `powermetrics`/Activity Monitor "Energy + Impact" or Windows Task Manager power column for the browser process, idle-foreground vs + after P1. This is the number the owner actually feels. + +Acceptance: `window.__bobbitPerf.report()` returns all five metric families in all four +states; baseline table committed. No behavior change when the flag is off (pinned by a unit +test asserting `perf-monitor` is not imported in the default boot path — follow the +`tests/`-side pattern used for `boot-timing`). + +--- + +## §5 Fix inventory (what we change and why it's safe) + +Numbered for reference from the phases: + +- **FX1 Ambient-animation gating.** Introduce one CSS class contract: every *ambient* + (infinite, decorative/status) animation rule gains the guard + `html.anims-paused & { animation-play-state: paused; }` (or is collected under a shared + selector). A new tiny module `src/app/animation-power.ts` toggles `anims-paused` on + `` when: tab hidden (`visibilitychange`), OR no session is in a working state + (derive from existing session status state — the sidebar already knows), OR battery-saver + is on (P3.3). One-shot transition animations (card-slide-in, banner-slide-down) are NOT + ambient and are excluded — pausing those would freeze mid-transition. +- **FX2 Sprites as status, not decoration** (UX improvement, not just a fix): a bobbit + animates **only while its agent is actually working**; idle/sleeping agents show a static + frame (or the existing sleep pose, frozen). Animation regains meaning as an at-a-glance + status signal, and N-sessions × ambient-loops disappears. Implemented as CSS: the + bob/breathe/eyes loops apply only under the existing busy/working status classes + (verify exact class names in `src/ui/bobbit-render.ts` / `app.css` during impl). +- **FX3 Kill paint-property animation.** Rewrite `status-pulse` and `unseen-dot-pulse` to + animate `opacity`/`transform` on a pseudo-element halo instead of `box-shadow` (visually + identical: a pre-rendered shadow layer fading/scaling). `pr-conflict-pulse` same treatment. +- **FX4 Global `prefers-reduced-motion` sweep.** One blanket rule disabling all ambient + animations under reduced-motion (keep opacity cross-fades), replacing the 9 piecemeal + opt-outs. +- **FX5 Session-poll demotion.** `startSessionPolling()` becomes the fallback it was meant to + be: interval 5 s → 60 s while the WS is connected and healthy; immediate `refreshSessions()` + on WS reconnect and on `visibilitychange`→visible (catch-up), mirroring the git-status + pattern. Keep 5 s only while the WS is down. +- **FX6 Scoped second-tick.** Replace `liveVerifTimer`'s `renderApp()` with a scoped update: + the elapsed-time labels become self-updating (a small component/`setInterval` that writes + `textContent` on the specific nodes, or a CSS counter). Full-tree render is reserved for + actual status transitions (which already arrive via WS events). +- **FX7 Streaming render cadence.** In the WS dispatch path for high-frequency frame types + (`message_update` deltas): apply state immediately (never delay data), but schedule the + visual render through a 100 ms trailing-edge throttle instead of per-frame rAF. Terminal + frames (`message_complete`, status changes) flush immediately. Build on the existing + `_renderScheduled` machinery in `state.ts` — add a `renderAppThrottled(ms)` next to + `renderApp()`; only the streaming call sites switch to it. +- **FX8 Timer audit & visibility gates.** From the P0 wakeup-by-callsite report, every + surviving interval ≥1/min gets the git-status-pattern treatment (visibility gate + + event-coalescing + self-cancel). `dialogs.ts`/`goal-dashboard.ts` loops are the known + suspects; fix what the data convicts. +- **FX9 Battery-saver mode.** A settings toggle (and auto-*suggestion* via + `navigator.getBattery()` when discharging — suggest, never silently switch): forces + `anims-paused`, stretches FX5/FX7 cadences (poll 120 s, render throttle 250 ms), disables + shimmer placeholders. State lives with the other UI settings; surfaced in the settings page + and the `bobbitPerfFlags` escape hatch. + +--- + +## §6 Phased implementation plan + +Each phase = one mission goal; land in order; every behavior change ships behind a perf-flag +kill switch (default-on once verified, like `deferOffscreenRender`) and is re-measured with +the P0 harness before the flag's removal cycle. + +### P1 — Stop animating at idle *(FX1–FX4; the battery headline)* + +Steps, in order: + +1. Build `src/app/animation-power.ts`: exports `initAnimationPower()` (called from + `main.ts`) — registers `visibilitychange` + subscribes to session-status state changes; + computes `shouldPause = hidden || noSessionWorking || batterySaver || reducedMotion`; + toggles `anims-paused` class on `document.documentElement`. Perf-flag: + `-pauseAmbientAnims` restores current behavior. +2. CSS sweep #1: tag every `infinite` animation rule listed in F1 with the + `html.anims-paused` pause guard. Mechanical change; keep one shared comment block in each + CSS file pointing back to this doc ("never reintroduce un-gated infinite animations"). +3. CSS sweep #2 (FX2): move the sprite work/idle loops under the busy/working status + selectors so idle sprites are static. Confirm the sleep pose stays as a static frame. +4. Rewrite `status-pulse` / `unseen-dot-pulse` / `pr-conflict-pulse` per FX3. +5. Add the blanket reduced-motion rule (FX4); delete the now-redundant piecemeal opt-outs. +6. Tests: browser E2E (`tests/e2e/ui/animation-power.spec.ts`) asserting (a) idle state ⇒ + `html.anims-paused` present and `document.getAnimations().length === 0`; (b) mock agent + working ⇒ its sprite animates; (c) `visibilitychange` to hidden pauses; (d) perf-flag + opt-out restores animations. Unit test pinning that no `infinite` animation in the two CSS + files lacks the pause guard (regex over the CSS — the "missing test IS the bug" rule). + +Acceptance: idle-foreground `getAnimations()` = 0; paint events/s <2; the E2E suite above +green; visual QA confirms sprites still bob while agents work. + +### P2 — Timer & render hygiene *(FX5–FX8)* + +1. FX5 session-poll demotion in `api.ts` (+ unit test on the interval-selection logic; the + spurious-unread pinning test `tests/spurious-idle-unread.spec.ts` must stay green). +2. FX6 scoped verification tick in `goal-dashboard.ts` (+ E2E: elapsed label advances while + `renderApp` counter — exposed via perf monitor — does not tick per second). +3. FX7 `renderAppThrottled()` in `state.ts` + switch streaming dispatch call sites; E2E: + streaming a long mock response produces ≤15 renders/s (perf-monitor counter) with no + visible degradation (follow-tail still pins to bottom — `follow-tail` tests stay green). +4. FX8 audit: commit the wakeup-by-callsite table to this doc, fix convicted loops on the + git-status pattern. + +Acceptance: idle-foreground wakeups <8/min; hidden-tab network ≤1 req/min; streaming render +cadence ~10 Hz; all existing unit + e2e phases green. + +### P3 — Product polish + +1. **P3.1 Remove temporary flags** that have soaked (per the `defer-offscreen-render.md` + removal convention), keep `anims-paused` as the permanent mechanism. +2. **P3.2 Re-baseline & publish**: re-run the §4 protocol, fill the after-table in §7, include + the OS-level energy reading. If streaming long-tasks remain, only then evaluate the heavier + options (scoped/component-level rendering for chat; sprite-sheet/canvas sprite engine) — + they are deliberately out of scope until data convicts them. +3. **P3.3 Battery-saver mode** (FX9): settings toggle + `getBattery()` suggestion banner + + docs. Browser E2E: toggle persists across reload (the settings E2E pattern, + `tests/e2e/ui/settings.spec.ts`). + +--- + +## §7 Baseline / after (filled by P0 / P3.2) + +| State | Metric | Baseline (dev) | Baseline (prod) | After P1 | After P2 | +|---|---|---|---|---|---| +| *(populated by the harness — renders/s, wakeups/min, animations running, long tasks/min, paint events/s, OS energy)* | | | | | | + +--- + +## §8 Non-goals / explicitly rejected + +- **Server-side stream coalescing** — already tried and rejected for UX latency reasons + (`reduce-server-cpu-experiment-stream-coalescing.md`); FX7 does it client-side at the + visual layer only. +- **Removing the bobbit sprite system** — it is the product's personality + ([bobbit-sprites.md](../bobbit-sprites.md)). FX2 makes it *more* meaningful, not less + present. +- **A WebGL/canvas renderer rewrite** — not justified by any current evidence; revisit only + if P3.2 data demands it. + +Cross-references: [time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) (the sibling +"instrumentation-first" effort, server/cost side), [mission-control.md](mission-control.md) +(the Observer staff can watch for perf regressions once the harness exists), +[harness-gap-analysis.md](harness-gap-analysis.md) §battery for how peer harnesses avoid this +class of cost (mostly by being TUIs — Bobbit's bar is higher because it ships a real UI). +Execution tracking: [fable-program-execution-plan.md](fable-program-execution-plan.md). + +--- + +## Appendix A — Implementation contracts (definite lists; do not re-derive) + +The universal definition-of-done in +[extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) +applies to every phase here. This appendix removes the remaining judgment calls. + +### A.1 Complete `infinite`-animation inventory and classification + +Source of truth: `grep -rn "animation:.*infinite" src --include=*.css` (re-run before +starting; if new rules appeared, classify them with the same three buckets and extend this +table in the same PR). + +Buckets: **ambient** = pause under `html.anims-paused` (P1 step 2) · **status** = must run +*only* under a working/busy state selector (P1 step 3) · **keep** = semantically must keep +running (rare; justify inline). + +| File:line | Keyframes | Bucket | Note | +|---|---|---|---| +| `src/app/app.css:108` | `status-pulse` | status | also FX3: rewrite off `box-shadow` | +| `src/app/app.css:478` | `gentle-float` | ambient | empty-state decoration | +| `src/app/app.css:943` | `unseen-dot-pulse` | ambient | FX3 rewrite; already has reduced-motion opt-out — keep | +| `src/app/app.css:967` | `bobbit-unread-tap` | ambient | | +| `src/app/app.css:985` | `bobbit-sidebar-unread-blink` | ambient | `steps()` — cheap but still wakes compositor | +| `src/app/app.css:1007` | `sidebar-heartbeat` | status | | +| `src/app/app.css:1026` | `sidebar-conic-spin` | status | spinner — runs only while loading; verify it unmounts | +| `src/app/app.css:1064,1075` | `gate-wave`, `gate-blink` | status | gate in-progress only | +| `src/app/app.css:1102` | `title-wave` | ambient | | +| `src/app/app.css:1111` | `pr-conflict-pulse` | ambient | FX3 rewrite | +| `src/app/goal-dashboard.css:781,1039` | `phase-glow`, `phase-glow-rejected` | status | dashboard visible + phase active only | +| `src/app/goal-dashboard.css:793,1070` | `phase-dot-pulse` | status | | +| `src/app/goal-dashboard.css:796` | `status-indicator-pulse` | status | | +| `src/app/goal-dashboard.css:944` | `gate-dot-pulse` | status | | +| `src/ui/app.css:1117,1140` | `shimmer` ×2 | ambient | loading placeholder; disabled entirely in battery-saver | +| `src/ui/app.css:1166` | `ask-heartbeat` | status | pending-ask only | +| `src/ui/app.css:1239,1411,1412,1521,1586+` | `blob-busy-*` family (move/eyes/shadow) | status | already gated on the busy blob — verify the element unmounts when idle | +| `src/ui/app.css:1526,1669` | `blob-*-idle-sleep-breathe` | **ambient** | the idle/sleep "breathing" loops — these are the FX2 headline: idle sprites must be static | +| `src/ui/app.css:1555,1700,1775,…` | `blob-compact-squash(-rigid)` (all ~10 sites) | ambient | several use `!important` — remove the `!important` when adding the gate | +| `src/ui/app.css:1755–2089` | `magnifier-depth-idle` (×7), `wand-depth-idle` | ambient | accessory idle loops | +| `src/ui/app.css:2041,2050` | `flask-bubbles` ×2 | ambient | | +| `src/ui/app.css:2154,2165` | `wand-sparkle-a/b` | ambient | | +| `src/app/app.css:359–474` | `bobbit-bob`, `bobbit-breathe`, `bobbit-eyes(-s)`, squish family | status | sidebar/chat sprites: working state only (FX2) | + +### A.2 `animation-power.ts` contract + +```ts +// src/app/animation-power.ts +export function initAnimationPower(): void; // called once from main.ts after state init +// Internally: +// shouldPause = document.hidden +// || !anySessionWorking() // derive from state.gatewaySessions statuses +// || isBatterySaverOn() // P3.3; returns false until that phase lands +// || matchMedia("(prefers-reduced-motion: reduce)").matches +// → document.documentElement.classList.toggle("anims-paused", shouldPause) +// Inputs: "visibilitychange" listener + a state subscription invoked from the same +// place session status updates already call renderApp(). No polling. No timers. +// Perf-flag: if isPerfFlagEnabled("-pauseAmbientAnims") → never add the class. +``` + +CSS gate (one block per file, appended at end): +`html.anims-paused :is() { animation-play-state: paused; }` +Never use a bare `html.anims-paused *` rule — it would freeze one-shot transitions mid-flight. + +### A.3 `perf-monitor` report shape + +```ts +// window.__bobbitPerf.report() → +{ windowMs: number, + rendersPerSec: number, // renderApp rAF executions + timerWakeups: { perSec: number, byCallsite: Record }, + longTasks: { count: number, totalMs: number }, + runningAnimations: number, // document.getAnimations().length, last sample + wsFramesPerSec: Record } // keyed by frame type +``` + +### A.4 Poll-site checklist (the loops; one-shots are out of scope) + +| Site | Today | Required end state | +|---|---|---| +| `src/app/api.ts:273` `startSessionPolling` | 5 s, visibility-gated | FX5: 60 s while WS healthy; 5 s only while WS down; refresh on reconnect + visible | +| `src/app/goal-dashboard.ts:1090` `startLiveVerifTimer` | 1 s full `renderApp()` | FX6: scoped elapsed-label update only | +| `src/app/render-helpers.ts:338` | 60 s `renderApp()` | keep (within budget) | +| `src/app/session-manager.ts:2821` git-status poll | 30 s, gated, coalesced | keep — this is the reference pattern | +| remaining `setInterval` loops in `goal-dashboard.ts` / `dialogs.ts` | unaudited | P2.4: classify from the P0 wakeup report; loops get the git-status pattern | + +### A.5 Owned files per phase (PR boundaries) + +- **PB-P0**: new `src/app/perf-monitor.ts`; one guarded hook line in `state.ts::renderApp`; + baseline table in this doc. No behavior change. +- **PB-P1**: new `src/app/animation-power.ts`; `main.ts` (one init call); the four CSS files + in A.1; new `tests/e2e/ui/animation-power.spec.ts`; CSS-pinning unit test + `tests/animation-gate.test.ts` (regex: every `infinite` rule in the four files is either + bucket-listed here or fails). +- **PB-P2**: `api.ts`, `goal-dashboard.ts`, `state.ts` (`renderAppThrottled`), streaming + dispatch call sites in `session-manager.ts`/`message-reducer.ts`; tests beside each. +- **PB-P3**: settings page + `animation-power.ts` (battery-saver input); docs. diff --git a/docs/design/code-intelligence-alternatives.md b/docs/design/code-intelligence-alternatives.md new file mode 100644 index 000000000..9c90c0734 --- /dev/null +++ b/docs/design/code-intelligence-alternatives.md @@ -0,0 +1,213 @@ +# Code Intelligence — alternatives considered & evidence (research annex) + +Status: research record, 2026-06-11. Companion to [code-intelligence.md](code-intelligence.md) +(design) and [code-intelligence-implementation-plan.md](code-intelligence-implementation-plan.md) +(execution). This doc exists so future maintainers know **what else was on the table, why the +proposed option won, and which facts that judgment rests on** — re-verify the dated facts +before overturning a decision. + +--- + +## §1 The market moved our way (validation of the core stance) + +The design's bet — *first-party LSP-backed tools in the harness, not MCP add-ons* — is now +the shipped pattern across terminal agents: + +- **Claude Code** shipped a native LSP tool in v2.0.74 (2025-12-19): go-to-definition, + references, hover, workspace symbols, call hierarchies, **automatic post-edit diagnostics**; + servers configured via plugins (11 official language plugins), binaries installed separately + ([changelog](https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md), + [tools reference](https://code.claude.com/docs/en/tools-reference)). +- **GitHub Copilot CLI** shipped LSP support (8 operations incl. rename + call hierarchy) with + repo/plugin/user JSON config and an "LSP Setup skill" for ~14 languages + ([blog, 2026-06-10](https://github.blog/ai-and-ml/github-copilot/give-github-copilot-cli-real-code-intelligence-with-language-servers/)). +- **OpenCode** preconfigures LSP for 18+ languages. **Codex CLI and Gemini CLI** still lack it + — their top-voted open feature requests ([codex#8745](https://github.com/openai/codex/issues/8745)). +- IDE agents (Cursor, Windsurf, Zed) expose at most a *diagnostics* tool to the model and lean + on retrieval indexes — Cursor 2.0 even "simplified the agent to use the LSP less." + +Implications adopted: (a) config-not-binaries plugin model = exactly our `lsp-` +descriptor packs; (b) Claude Code's early bugs (zombie servers, stale pre-edit diagnostics, +gitignored files in references, missing `didOpen`, command injection in binary detection) are +a free checklist — each becomes a CI-3 invariant/test; (c) **post-edit auto-diagnostics** is +table stakes — added to CI-3 scope; (d) differentiation is real: nobody ships a budgeted +ranked repo map, typed multi-checker `code_check`, worktree-keyed supervision, or +capability-swappable engines. + +## §2 LSP engine: build on `vscode-jsonrpc` — vs Serena, bridges, Python libs + +| Option | Verdict | Decisive facts | +|---|---|---| +| **vscode-jsonrpc 9 + vscode-languageserver-protocol 3.18** (npm) — **CHOSEN** | Build the supervisor on these | Microsoft-maintained, republished 2026-06-03, ~10M+ downloads/wk each; protocol README: "tool independent… usable in any node application". coc.nvim (~25k★) is the proven reference for the lifecycle layer. Protocol plumbing ≈ 500–1000 lines / 1–2 wks; the hard part (lifecycle) we own under any option | +| `vscode-languageclient` | Rejected | `engines.vscode` — runs only inside the VS Code extension host (verified package.json) | +| **Serena** (oraios/serena, MIT, ~25k★) | Design reference + fallback pack, not the engine | Python 3.13 + uv runtime per workspace; MCP indirection; live reliability battle in its tracker: 30 GB RAM ([#944](https://github.com/oraios/serena/issues/944)), init hangs ([#1390](https://github.com/oraios/serena/issues/1390), [#937](https://github.com/oraios/serena/issues/937)), stdio init races ([#900](https://github.com/oraios/serena/issues/900)), LS termination loops ([#634](https://github.com/oraios/serena/issues/634)). **Port its designs**: solidlsp's synchronous wrapper + two-tier symbol cache (100–500 ms → <10 ms), auto-download of server binaries | +| multilspy (Microsoft) | N/A directly | Python-only research library (13 langs); its 2026 fix history (server hangs, orphaned processes) confirms lifecycle is where the bodies are buried | +| MCP bridges (mcp-language-server 1.5k★, lsmcp, cclsp, lsproxy) | Rejected | All single-maintainer; most dormant ≥10 months (mcp-language-server since 2025-06, lsmcp since 2025-08); single-root designs; none do worktree-aware pooling. lsproxy (REST, Rust, multi-server pool) is the closest design reference | + +### §2.1 The Serena question, in full ("why not one LSP to rule them all, wrapped and fixed?") + +Owner asked this directly (2026-06-11); recording the complete argument since it's the most +tempting future re-litigation. Compare the two stacks: + +``` +Wrap Serena: agents → MCP protocol → Serena (Python 3.13 + uv sidecar) → solidlsp → language servers +Proposed: agents → first-party code_* tools → LSP supervisor (in the gateway, TS) → language servers +``` + +1. **The valuable part of Serena is the part we'd rewrite anyway.** Its worth is the bottom + layer — server lifecycle + symbol caching — and its tracker shows that battle ongoing + (§2 table: #944 30 GB RAM, #1390/#937 init hangs, #900 stdio races, #634 re-init loops). + The fixes Bobbit needs most — **per-(worktree, language) pooling, idle eviction, disposal + wired into goal-worktree cleanup** — are absent from Serena's one-project-one-session + architecture. That's not a wrapper-sized change; it's the design. +2. **The protocol layer — the part that looks scary — is the cheap part.** Microsoft's + `vscode-jsonrpc` + `vscode-languageserver-protocol` give the full typed client off the + shelf; the lifecycle layer is ~500–1000 lines with coc.nvim as reference. So "buy Serena" + saves the *easy* part while charging a Python-3.13/uv runtime, a sidecar process per + workspace, and a second supervisor-in-a-different-language to debug when it wedges. +3. **The MCP surface is the already-observed failure mode.** Generic MCP tools = no token + budgets, no `file:line` renderers, no tool-guard tiers, no prompt guidance — and Serena's + rename writes files directly, bypassing the preview-diff → review flow. The model-facing + UX, where tool adoption is won or lost, is precisely what a wrapper can't fix. +4. **Serena still pays us twice.** We port its two best designs (synchronous two-tier symbol + cache; auto-download of server binaries, both also in multilspy), and because everything + sits behind the `code.symbol-nav` capability, a Serena-backed pack remains a legitimate + community alternative — and the **escape hatch**: if the CI-3 supervisor stalls in its + wave, wrapping Serena behind the same capability is roughly a week's work, not a rewrite. + +One-sentence verdict: *Serena is the right idea delivered as the wrong dependency — keep the +idea (supervised language servers behind symbol tools), delete the Python/MCP middle, build +the thin lifecycle layer natively where Bobbit's budgets, renderers, review flow, and +worktree model already live.* + +## §3 Per-language servers (research-corrected picks) + +Two picks **changed** from the original draft: TS/JS `typescript-language-server` → **vtsls**; +Python `pyright` → **basedpyright**. + +| Language | Pick | Why / acquisition | Caveats to engineer for | +|---|---|---|---| +| TS/JS | **vtsls** | Wraps VS Code's TS extension (feature parity); Zed's default ([zed#13140](https://github.com/zed-industries/zed/pull/13140)); npm install | tsserver under the hood: plan multi-GB RSS, set max old-space (Zed uses 8 GiB) | +| Python | **basedpyright** | Pyright fork with Pylance-only features (find-implementations, semantic tokens) reimplemented in the open server ([docs](https://docs.basedpyright.com/dev/benefits-over-pyright/pylance-features/)); self-contained pip wheel, no Node needed | `openFilesOnly` default limits analysis — configure workspace-wide; references can be incomplete for unopened files | +| Go | gopls | Canonical; `go install` | Needs Go toolchain + `go.mod`; note gopls ≥0.20 has a built-in experimental MCP server — possible shortcut, evaluate at CI-4 | +| Rust | rust-analyzer | Prebuilt single binaries on GH releases (rustup shim caveat: [rustup#3846](https://github.com/rust-lang/rustup/issues/3846)) | **Worktrees**: set `CARGO_TARGET_DIR` per worktree + `rust-analyzer.cargo.targetDir` or instances serialize on cargo locks ([#10684](https://github.com/rust-lang/rust-analyzer/issues/10684)); weakest multi-root — one instance per cargo workspace; watcher walks all worktrees ([#16534](https://github.com/rust-lang/rust-analyzer/issues/16534)) | +| C# | **razzmatazz/csharp-language-server first**; Microsoft Roslyn LS later | csharp-ls: `dotnet tool install -g csharp-ls`, standard stdio LSP, active. Roslyn LS is what VS Code uses but: Azure-DevOps-feed acquisition, nonstandard named-pipe handshake, custom `workspace/projectInitializationComplete` readiness signal | OmniSharp effectively dead — do not use | +| F# | FsAutoComplete | `dotnet tool install fsautocomplete`; Ionide-maintained | Needs restorable solution (MSBuild project load) | +| Java | JDT-LS | Standard; tarball + **JDK 21+** | Heaviest warm-up (Maven/Gradle import; progress sticks); per-project `-data` workspace dirs; 1–2 GB+ heap | +| Kotlin | JetBrains kotlin-lsp — **defer (alpha)** | Official, but alpha, partially closed-source, IntelliJ-class memory, init timeouts on large projects; community fwcd server deprecated | Ship the card last; document degraded support | +| C/C++ | clangd | Prebuilt zips on GH releases | Practically requires `compile_commands.json` — descriptor should auto-suggest `cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON` / `bear` | + +**Operational invariants the supervisor must encode** (each cited in research; all promoted to +CI-3 pinning tests): strict `initialize`→`initialized` ordering; advertise `workspaceFolders` +capability *and* still pass rootUri (claude-code [#27220](https://github.com/anthropics/claude-code/issues/27220)); +**didOpen-before-query document tracker** (typescript-ls returns empty without it, +[#89](https://github.com/typescript-language-server/typescript-language-server/issues/89)) and +didChange/didClose on disk changes (server must not re-read opened docs); **position encoding** +— negotiate `utf-8` via `general.positionEncodings`, else convert UTF-16 code units (the +"bottom emoji breaks rust-analyzer" class); **readiness gating** — no standard signal exists; +consume `$/progress` + per-server signals (rust-analyzer `experimental/serverStatus.quiescent`, +Roslyn `projectInitializationComplete`) with per-descriptor `startupTimeout`; **client-side +file watching** — honor `workspace/didChangeWatchedFiles` registrations or servers go stale on +git checkouts (gopls depends on it); pass `processId` AND reap on our side (servers commonly +ignore parent-death); `shutdown` → `exit` → wait → SIGKILL; **diagnostics settle** before +`code_diagnostics` returns (push model is async); memory caps + idle eviction + orphan sweep +(the oh-my-opencode jdtls-leak system-freeze failure mode). + +## §4 Structural engine: ast-grep — vs Semgrep, comby, GritQL, srgn, raw tree-sitter + +| Engine | Verdict | Decisive facts | +|---|---|---| +| **ast-grep** — **CHOSEN** | Ship `@ast-grep/cli` as npm dep | MIT, ~14.5k★, very active (sole-maintainer risk noted; mitigated by capability seam). 25–31 langs + custom grammars. `--json=stream` NDJSON with metavariable captures; rewrite with diff preview; YAML rule system. **npm-native**: `@ast-grep/cli` (~569k dl/wk) resolves prebuilt platform binaries via optionalDependencies — no custom shipping; `@ast-grep/napi` (~2M dl/wk) for in-process use (repo map). De-facto agent choice (official MCP + Claude skill; Codemod jssg; oh-my-opencode) | +| Semgrep CE / Opengrep | Rejected for embedding | Python front-end + OCaml core subprocess — needs a Python runtime; seconds of per-invocation startup; rules relicensed restrictively Dec 2024 (engine still LGPL); security-lint-shaped, not edit-shaped. Opengrep fork (Jan 2025, 10+ vendors) is the safer governance if ever needed | +| comby | Rejected | Right interface (single binary, `-json-lines`) but last release June 2022; "v2 prep" commits June 2026 are a flicker, not a maintenance signal | +| GritQL | Rejected | Survived Honeycomb acquisition only by donation to Biome (Dec 2025); volunteer-paced; lives on as Biome's plugin language | +| srgn | Rejected | MIT single binary but 7 languages, no JSON output, beta CLI churn. (Claims that Codex bundles it: **unverified/likely false** — codex-universal ships ripgrep/fd/universal-ctags) | +| raw web-tree-sitter | Used for the repo map, not the tool | It's a parsing primitive — we'd rebuild matching/rewrite/CLI ergonomics ast-grep already has | + +## §5 Repo map: Aider-style ranked map — vs ctags, SCIP, stack-graphs, packers, embeddings + +**Upgrade adopted from research:** the map is not just "files → symbols"; it is a **ranked** +map — defs+refs graph with personalized PageRank, which is what makes 2–8 KB budgets effective. + +- **Aider's mechanism** (the model): tree-sitter `tags.scm` queries → def/ref tags per file → + `MultiDiGraph` (referencer file → definer file, weight `√refs` × multipliers: ×10 chat-mentioned + idents, ×10 well-named long idents, ×0.1 private `_`/defined-in->5-files, ×50 in-chat + referencing file) → personalized PageRank seeded by chat files → rank split across edges to + definitions → binary-search the tag count to the token budget (default 1k, up to 4k on + large-context models; ~2–8× expansion when no files in chat) → render signature lines only; + mtime-keyed disk cache ([repomap blog](https://aider.chat/2023/10/22/repomap.html), + [repomap.py](https://github.com/Aider-AI/aider/blob/main/aider/repomap.py)). +- **Evidence it pays**: RepoGraph (ICLR'25, [arXiv:2410.14684](https://arxiv.org/abs/2410.14684)) + — def/ref graph plug-in lifts SWE-bench-Lite resolve rates across four harnesses (≈+2–2.7 pts + absolute, ≈+33% relative avg); Agentless ([arXiv:2407.01489](https://arxiv.org/abs/2407.01489)) + shows cheap structural repo views rival full agentic exploration; CodePlan (FSE'24) likewise. + Aider itself publishes no ablation — the academic record carries the claim. +- **Node implementation path**: tag extraction via **web-tree-sitter + `tree-sitter-wasms` + prebuilt grammars** reusing Aider's vendored `tags.scm` queries (MIT; battle-tested across + 100+ languages), or `@ast-grep/napi` kind-queries where grammars overlap — decide at CI-5 + with a spike; ranking via `graphology` PageRank; cache by content hash per worktree. +- Rejected: **universal-ctags** (defs only — no refs ⇒ no ranking graph; the reason Aider left + it), **SCIP** (compiler-accurate but per-language indexers + build required; scip-typescript + last released Oct 2024), **stack-graphs** (archived 2025-09-09 — dead; GitHub itself fell + back to tree-sitter tag search), **CodeQL** (license bars closed-source analysis; minutes of + DB build), **repomix/packers** (one-shot dumps, unranked, token cost scales with repo). +- **Embeddings**: genuinely contested. Cursor's eval claims +12.5% avg agent accuracy from + semantic search; Anthropic found agentic search "outperformed RAG by a lot" and ships none; + continue.dev deprecated its embeddings `@codebase`. Both headline numbers are + vendor-interested. Decision: **not in v1** (infra + staleness + privacy cost; weaker fit for + our worktree-keyed model); the `code.map`/search capability seam leaves room for an optional + embeddings pack later if BENCH shows orientation gaps on very large repos. + +## §6 Diagnostics: custom rdjson-shaped schema + errorformat fallback — vs SARIF-internal + +- **Native structured output exists for most of the fleet**: ESLint `--format json`; Biome + `--reporter=rdjson|sarif`; ruff `--output-format json|rdjson|sarif` (best in class); mypy + `--output=json` (≥1.11); `go vet -json` / `go test -json` (compile errors arrive as text in + Output events — caveat); cargo `--message-format=json` (gold standard, includes fixes); + dotnet via `-p:ErrorLog=out.sarif`. **tsc has no JSON** ([TS#46340](https://github.com/microsoft/TypeScript/issues/46340)) + — but `--pretty false` output has been regex-stable for a decade. **pytest**: use built-in + `--junitxml`; the json-report plugin is third-party/lightly maintained. **javac/gradle/maven**: + weakest — text parsing territory. +- **Canonical internal schema: our own minimal record** `{tool,file,line,endLine?,col?,code, + severity,message,fix?}` — essentially a flattened **rdjsonl** (reviewdog diagnostic format). + ruff and Biome already emit rdjson natively, validating the shape. **SARIF is an input, not + the model**: deeply nested, 90% of its surface irrelevant to an agent loop; one SARIF→internal + adapter unlocks dotnet, semgrep, clang, CodeQL at once. +- **Bundle reviewdog's `errorformat`** (single static Go binary, ~50+ preset parsers + user + `-efm` patterns, emits rdjsonl): the universal fallback that covers javac/gradle and the + long tail without writing parsers. This one *does* ship via the `binaries/` mechanism. +- Peer check: Claude Code feeds raw hook stderr text; Cursor pipes LSP diagnostics (uneven by + language); Codex/Devin read raw logs. A typed multi-checker `code_check` is ahead of all of + them; LSP pull-diagnostics is a later fast path (CLI checkers stay source-of-truth because + they reflect project task orchestration — nx/turbo configs, generated code). + +## §7 Memory layer (branch-level check, settles the EP G2 bet) + +Researched because the PR branch commits to Hindsight (EP G2). **Bet validated, framing +sharpened**: vectorize-io/hindsight is MIT, very active (v0.8.1 June 2026), Node SDK, hybrid +retrieval (semantic+BM25+graph+temporal), Postgres+pgvector, and the Nous Hermes agent +integration proves the exact daemon-autostart-pack pattern. Alternatives are worse fits: mem0 +(cloud-leaning, disputed benchmarks), Zep (self-hosted CE discontinued), cognee (no TS SDK), +Letta (a competing harness). **But** the 2026 production trend is text-first (Claude Code +CLAUDE.md/auto-memory, Devin Knowledge, Cursor memories) — so: **markdown staff memory stays +authoritative** for procedural knowledge; Hindsight is the *optional* episodic-memory pack, +gated on Docker/Postgres availability, an LLM key per `retain` priced in. This nuance is +recorded against EP G2 here rather than rewriting the EP doc. **Superseded for depth:** the +full memory decision record — sessions-vs-Hindsight layering, the comparison table with +reasons, and the bank-topology decision (one shared tag-scoped bank) — now lives in +[agent-memory.md](agent-memory.md). + +## §8 Net changes applied to the CI docs after this research + +1. CI-3 engine named: `vscode-jsonrpc` + `vscode-languageserver-protocol`; coc.nvim as + lifecycle reference; solidlsp's sync+cache and multilspy's binary auto-download as ported + designs; §3's operational invariants promoted into CI-3 tests; **post-edit auto-diagnostics** + added to CI-3 scope (match Claude Code). +2. Server picks: **vtsls**, **basedpyright**, **csharp-ls-first**, kotlin deferred (alpha); + per-worktree `CARGO_TARGET_DIR`; clangd compile-commands hint in descriptor. +3. CI-1 shipping: `@ast-grep/cli` npm dependency (platform optionalDependencies) instead of + the custom `binaries/` route; `--json=stream`. +4. CI-2: rdjsonl-shaped internal schema; `errorformat` bundled as fallback; SARIF adapter; + pytest via junitxml; tsc regex parser. +5. CI-5: upgraded from flat symbol listing to **ranked** def/ref-graph map (personalized + PageRank, session-aware seeds), with the RepoGraph/Agentless evidence as rationale. diff --git a/docs/design/code-intelligence-implementation-plan.md b/docs/design/code-intelligence-implementation-plan.md new file mode 100644 index 000000000..2f8e224fd --- /dev/null +++ b/docs/design/code-intelligence-implementation-plan.md @@ -0,0 +1,213 @@ +# Code Intelligence — Implementation Plan (hand-off) + +Status: ready for execution, not started. Workstream **CI** in +[fable-program-execution-plan.md](fable-program-execution-plan.md). + +Companion to [code-intelligence.md](code-intelligence.md) (WHAT/WHY + the layer model and +design stance — its §7 worktree contract and §1 capability names are LAW) and +[code-intelligence-alternatives.md](code-intelligence-alternatives.md) (the research annex: +why each engine/library below beat its alternatives, with citations and the operational +invariants their issue trackers taught us — **read its §3 before implementing CI-3**). + +> **Anchor baseline:** fable-docs @ 2026-06-11 (master parent `6ec8c8f9`). Locate by symbol +> name; missing symbol ⇒ STOP, re-derive from the cited pattern — never improvise. +> +> **Precision policy:** CI-1/CI-2 file/function level; CI-3…CI-7 contract level (re-verify +> anchors; CI-3's supervisor is the largest new component in the program — its goal text +> includes extra invariants). +> +> **Universal rules:** [extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) +> + [fable-program-execution-plan.md §1](fable-program-execution-plan.md). **Seams:** +> tool-activation in `session-setup.ts` (shared — confine to named functions); `binaries/` +> shipping (`binaries.versions.json`, checksums — copy the existing binary's wiring); +> EP G8 capabilities (CI consumes when it lands; until then a thin local registry shim, +> see CI-7). + +--- + +## Goal map + +``` +CI-1 ast-grep tools ──→ CI-5 repo map (reuses CI-1 engine) +CI-2 structured diagnostics (independent) +CI-3 LSP supervisor + code_* tools (TS+Py) ──→ CI-4 language packs + autodiscovery UX +CI-1..CI-5 ──→ CI-6 services chip + graphify viz pack ──→ CI-7 capability swap + BENCH +``` + +## CI-1 — `ast_search` / `ast_rewrite` tool group + +**Outcome:** structural search and preview-diff rewrites across the worktree; regex/sed +edits become obsolete. + +**Owned files:** NEW `defaults/tools/ast/{ast_search.yaml,ast_rewrite.yaml,extension.ts}`; +`@ast-grep/cli` added as a pinned npm dependency (prebuilt platform binaries via +optionalDependencies — no `binaries/` wiring needed; resolve the binary path via +`require.resolve`); NEW `src/server/agent/ast-grep-runner.ts`; NEW +`tests/ast-grep-runner.test.ts`, `tests/e2e/ast-tools.spec.ts`; budget rows in +`tests/tool-description-budget.test.ts`. + +**Steps** + +1. Runner: `runAstGrep({pattern, rewrite?, lang?, paths?, cwd})` → execFile the resolved + binary with `--json=stream` (NDJSON); map results to `{file, line, column, text}[]` + plus metavariable captures (relative paths + ONLY — pin per design §7); output cap + spill via `truncateLargeToolContent` + (`src/server/agent/truncate-large-content.ts`). +2. `ast_search`: tool YAML per the `defaults/tools/team/` anatomy; renderer lists matches + as clickable `file:line` rows (copy a search-style renderer from + `src/ui/tools/renderers/`). +3. `ast_rewrite`: runner in diff mode → unified diff returned as the tool result rendered + by the existing diff renderer; `apply: true` arg performs the write through the normal + file-edit path (so git-status/review flows observe it). Default is preview. +4. Prompt guidance: one paragraph in the CE-G7.1 discovery section (if landed; else the + tool's `docs:` field carries it): "prefer ast_search over grep for code shapes; prefer + ast_rewrite over sed always". + +**Tests (author first; RED):** runner unit (json mapping, relative-path pin, cap+spill, +unknown lang error); e2e — search finds a planted pattern; rewrite previews a diff without +mutating; `apply: true` mutates and git-status reflects it; budget rows. + +**Acceptance:** suites green; manual: an agent in a scratch project completes a +rename-shaped change via ast_rewrite preview→apply with zero bash sed. + +## CI-2 — `code_check` structured diagnostics + +**Outcome:** one tool returns typed diagnostics from the project's checkers; gates can +consume the same parsers. + +**Owned files:** NEW `defaults/tools/code/code_check.yaml` (group seeds the `code` group +early; CI-3 adds siblings) + extension wiring; NEW `src/server/agent/diagnostics/` +(`runner.ts` + `parsers/{tsc,eslint,pytest,go}.ts`); NEW unit tests per parser with +captured-output fixtures; e2e. + +**Steps:** checker set resolution (project config override → autodiscovery by manifest: +`tsconfig.json`⇒tsc+eslint, `pyproject.toml`⇒pytest/ruff if present, `go.mod`⇒go vet); +schema = flattened rdjsonl: `{tool, file, line, endLine?, col?, code?, severity, message, +fix?}[]`; per-tool strategy (annex §6): native JSON for eslint / ruff‑rdjson / mypy +`--output=json` / cargo `--message-format=json` / `go vet -json` (NB compile errors arrive +as text inside `go test -json` Output events); stable regex for `tsc --pretty false` +(`^path(line,col): error TSnnnn: msg`); `--junitxml` for pytest (not the json plugin); +one SARIF→internal adapter (covers dotnet `-p:ErrorLog=…sarif`, semgrep, clang); bundled +reviewdog **`errorformat`** static binary via the `binaries/` mechanism as universal +fallback (preset parsers + user `-efm` from project config). Result = summary counts + +first N (default 50) diagnostics + spill ref; unknown checker ⇒ capped raw fallback, never +an error. Renderer: grouped-by-file list. Verification-harness +consumption is a recorded follow-up (do NOT touch `verification-harness.ts` here — CE-G3.3 +/ CE-G5.2 own it per §1.4). + +**Tests:** parser fixtures (each: real captured output → exact typed rows); resolution +matrix unit test; e2e on a fixture project with a planted type error → structured row. + +**Acceptance:** a failing `tsc` run produces ≤2 KB of tool result instead of the raw log; +ledger note recorded in the doc. + +## CI-3 — LSP supervisor + `code_*` symbol tools *(contract level; the big one)* + +Contracts: design §2; **annex §§2–3 are required reading** (engine rationale + the +operational invariants harvested from rust-analyzer/gopls/tsserver/Serena/Claude Code +issue trackers). **Owned (new):** `src/server/agent/lsp/` (`lsp-supervisor.ts`; +`lsp-client.ts` on **`vscode-jsonrpc` + `vscode-languageserver-protocol`** — coc.nvim's +client layer is the reference implementation; `language-registry.ts` detection rules; +descriptor schema); `defaults/tools/code/{code_definition,code_references,code_symbols, +code_hover,code_rename,code_diagnostics}.yaml`; descriptor data for TypeScript (**vtsls**) ++ Python (**basedpyright**, workspace-wide analysis configured, not `openFilesOnly`). + +Invariants (each gets a pinning test): + +1. Supervisor = idempotent ensure with in-flight dedupe (`sandbox-manager.ts` shape); + key `(worktreeRoot, languageId)`; idle-shutdown default 10 min; max concurrent servers + (default 4, config) + per-server memory ceilings; crash ⇒ restart with backoff, + surfaced non-fatally (tool returns "language server unavailable — falling back" + guidance, never hangs the turn). +2. Protocol correctness: strict `initialize`→`initialized` ordering; advertise + `workspaceFolders` *and* pass `rootUri`; pass `processId` AND reap our side + (`shutdown`→`exit`→wait→SIGKILL); negotiate `general.positionEncodings: utf-8`, else + convert UTF-16 code units at the client boundary. +3. Document lifecycle tracker: `didOpen` before any query against a file (tsserver returns + empty otherwise); `didChange`/`didClose` when our edit tools or git change disk; honor + server `didChangeWatchedFiles` registrations (forward our watcher events) or servers go + stale on checkouts. +4. Readiness gating: per-descriptor warm-up signal (`$/progress` settle, rust-analyzer + `experimental/serverStatus.quiescent`, Roslyn `projectInitializationComplete`) + + `startupTimeout`; `code_diagnostics` waits for the push-diagnostics settle window. +5. Tools degrade gracefully: no descriptor for the file's language ⇒ structured "not + covered; detected language X; install lsp-X pack" result (feeds CI-4's UX), never an + exception. +6. `code_rename` is preview-diff → apply through the file-edit path (CI-1 step 3 pattern); + references results exclude gitignored paths (Claude Code regression #26051 class). +7. All positions/paths relative; budget + spill on every result. +8. Lifecycle: worktree removal (goal cleanup) disposes servers + caches — hook the + existing worktree-cleanup path; Caretaker sweep (MC) catches leaks/orphans (the + jdtls-leak failure mode). Per-worktree env isolation lives in the descriptor (e.g. + rust: `CARGO_TARGET_DIR=/.bobbit-cache/target`). +9. **Post-edit diagnostics push**: after our edit tools touch a covered file, fresh (not + stale pre-edit) diagnostics are injected into the turn, capped; per-descriptor + `diagnostics: false` opt-out. Port Serena's solidlsp symbol cache (two-tier, content- + keyed) for hot `code_symbols`/`code_definition` paths. + +Tests: supervisor unit with a fake LS binary (spawn/dedupe/idle/crash/handshake-order/ +encoding negotiation); e2e against vtsls on a fixture project: definition/references/ +rename round-trip + post-edit diagnostics; uncovered-language fallback; worktree-dispose +kills the server (process poll); stale-diagnostics pin (edit → old diagnostics never +delivered after new edit). + +**Acceptance:** on the Bobbit repo itself, `code_references` on a core symbol returns in +<2 s warm, and the supervisor shows 0 servers after idle timeout. + +## CI-4 — Language packs + autodiscovery UX *(contract level)* + +Descriptor-as-pack: `market-packs/lsp-/` containing one `descriptor.yaml` +(server binary acquisition: npm/pip/download+checksum; args; init options; readiness +signal; health probe; file-detection globs; per-worktree env) — data only, no code. +Bundle `lsp-typescript` (vtsls), `lsp-python` (basedpyright) as built-ins (band per +`builtin-packs.ts`); **cards** (checklist rows, not yet specced), server picks per annex +§3: `lsp-go` (gopls — evaluate its built-in MCP server as a shortcut), `lsp-rust` +(rust-analyzer + CARGO_TARGET_DIR isolation), `lsp-csharp` (**csharp-ls first**, Microsoft +Roslyn LS as a later upgrade — nonstandard handshake), `lsp-fsharp` (FsAutoComplete), +`lsp-jvm` (JDT-LS, JDK 21+, per-project `-data` dirs), `lsp-clangd` (descriptor suggests +generating `compile_commands.json`), `lsp-kotlin` (**last** — kotlin-lsp is alpha). +Autodiscovery surfaces uncovered detected languages through the CI-6 chip + a marketplace +deep-link. Acceptance: installing `lsp-go` on a Go fixture makes `code_definition` work +with zero config; uninstalling degrades gracefully. + +## CI-5 — `code_map` budgeted **ranked** repo map *(contract level; after CI-1)* + +Aider's algorithm, in Node (annex §5 has the full mechanism + evidence): tag extraction +(defs **and** refs) via web-tree-sitter + `tree-sitter-wasms` prebuilt grammars reusing +Aider's MIT `tags.scm` queries — spike vs `@ast-grep/napi` kind-queries first, pick one — +→ def/ref file graph (edge weight `√refs` × multipliers: boost session-mentioned idents +and in-context files, damp `_`-private and defined-everywhere idents) → personalized +PageRank (`graphology`) seeded by session state → signature-only rendering, binary-search +tag count to budget. `code_map(path?, depth?, budget?)`: hard result budget (default 2 KB, +max 8 KB); content-hash cache per worktree under the design-§7 cache dir; relative paths +pin. When EP G1.3 (prompt sections via providers) is merged, add the provider that +contributes a "Repo Map" section under the same budget — the tool stays for on-demand +deeper maps. Acceptance: on the Bobbit repo, default call ≤2 KB, ranking test (a hub +module outranks a leaf util), and cold→warm rebuild ≥10× faster (cache test). + +## CI-6 — Context-services chip + graphify visualization pack *(contract level)* + +- **Chip** (core UI): chat-header widget next to `` showing session + scope (global/project/worktree) + active code-intel services from a new generic + `GET /api/sessions/:id/services` (supervisor + pack-runtime sourced — generic shape so + EP runtimes plug in later). Popover: status/restart/open-panel/install-suggestion. + Browser E2E: chip renders per scope; restart works; suggestion deep-links marketplace. +- **Graphify pack** (marketplace, NOT built-in): wraps graphify per worktree into + `graphify-out/` (gitignored), panel at `#/ext/graphify` rendering its `graph.html`; + install-time doc notes the absolute-path caveat and pins output as derived cache + (never committed; design §4). Pack-litmus tests only — graphify itself is not under test. + +## CI-7 — Capability seam + BENCH validation *(contract level)* + +Until EP G8 lands: a 30-line local registry +(`src/server/agent/code-intel-capabilities.ts`) mapping the four capability names → +implementing module; tools resolve through it (pin: no direct cross-module imports of +engines from tool extensions). When EP G8 merges, swap the shim for pack capabilities +(one PR, the shim's API is G8's `ctx.capabilities.call` shape). Then: run the CE BENCH +suite (CE-G0.3) with CI tools on vs off; record token/turn deltas in +code-intelligence.md §8; this is the workstream's exit gate. + +--- + +Checklist rows (mirrored in the execution plan §4): CI-1 · CI-2 · CI-3 · CI-4 (+7 language +cards) · CI-5 · CI-6 · CI-7. diff --git a/docs/design/code-intelligence.md b/docs/design/code-intelligence.md new file mode 100644 index 000000000..1c79b3876 --- /dev/null +++ b/docs/design/code-intelligence.md @@ -0,0 +1,187 @@ +# Code Intelligence — giving agents IDE superpowers (find, edit, verify) + +Status: design accepted, not started. Workstream **CI** in +[fable-program-execution-plan.md](fable-program-execution-plan.md). + +> **Execution authority:** implement from +> [code-intelligence-implementation-plan.md](code-intelligence-implementation-plan.md). +> **Alternatives & evidence:** every engine/library pick below is justified against its +> competitors, with citations, in +> [code-intelligence-alternatives.md](code-intelligence-alternatives.md) — read it before +> swapping any component. + +**The problem:** Bobbit agents are "an LLM with grep". 56% of tool calls are bash +(time-and-token-cost-efficiency.md §3), discovery is multi-round grep cascades, edits are text +patches whose correctness is only discovered at gate time, and nothing understands symbols, +types, or structure. Peers are converging on the same answer (Serena, Octocode, Aider's +repo-map, graphify): pair fast lexical search with **structural (AST), semantic (LSP), +spatial (repo map), and truth (diagnostics) layers**. + +**Design stance (from owner interview, 2026-06-11):** + +1. **Native first-party tool groups, not MCP-first.** Observed reality: agents underuse + generic MCP tools. First-party groups get budgeted outputs, custom renderers, tool-guard + policies, and prompt guidance — that's why `team_*` gets used and random MCP servers + don't. External engines may sit *behind* a tool group; they are not the model-facing + surface. +2. **Best solution first, swappable forever.** Every layer is addressed through an + extension-platform **capability name** (extension-platform.md §3.1): + `code.structural-search`, `code.symbol-nav`, `code.map`, `code.diagnostics`. Reference + implementations ship as built-in packs that *provide* those capabilities; tools and + prompt sections *consume* capabilities, never concrete packs. Swapping engines later = + installing a different provider pack — zero tool-surface change. +3. **Per-worktree everything.** Indexes, caches, and language-server instances are keyed by + worktree root, stored as derived gitignored state, rebuilt per checkout. Nothing derived + is ever committed; nothing stores absolute paths (the exact failure class graphify hit + across worktrees). +4. **Cost is a feature.** Every tool here exists to replace a more expensive pattern + (grep cascade → one `code_references`; 50 KB test log → 12 structured diagnostics; + read-five-files orientation → one budgeted map section). Each lands with a CE-ledger + measurement expectation (CE-G7.2's rationale, now executed). +5. **Verifiability is a feature.** Structural rewrites preview as diffs through the + existing review flow before applying; every edit path ends in `code_check`; nothing + mutates silently. + +--- + +## §1 The layer model (what exists, what's new) + +| Layer | Question it answers | Bobbit today | This workstream adds | Capability | +|---|---|---|---|---| +| Lexical | "where is this string?" | bash grep/rg (uncapped, 56% of calls) | nothing new — CE-G2 caps it, CE-G7.1 steers it | — | +| **Structural** | "find/replace this syntax shape" | ❌ regex/sed via bash | `ast_search` / `ast_rewrite` tools (ast-grep engine) | `code.structural-search` | +| **Symbol/semantic** | "definition? references? rename? type?" | ❌ | `code_*` LSP tool group + gateway LS supervisor + per-language packs | `code.symbol-nav` | +| **Spatial** | "what's here and how does it connect?" | ❌ (session search ≠ code) | budgeted repo-map (tool now, prompt section when EP providers land); graphify pack for *visualization* | `code.map` | +| **Truth** | "did my change actually work?" | gates run commands, agents read raw logs | `code_check` structured diagnostics (typed results, gate-consumable) | `code.diagnostics` | +| Temporal | "what did we learn before?" | staff memory; FlexSearch sessions | **already planned elsewhere**: Hindsight pack = EP G2; session-memory = EP G1.6; profile = GA-R5 | `memory` (EP) | + +Graphify-vs-Hindsight, settled: **different layers, complementary, both already placed.** +Graphify-shaped tools are spatial (structure of *this checkout, now*); Hindsight is temporal +(what the agent/team *learned over time*) and is literally EP G2. CI adds the spatial layer; +it does not duplicate the temporal one. + +## §2 Symbol layer — native LSP tool group + +- **Surface:** `defaults/tools/code/` — `code_definition`, `code_references`, + `code_symbols` (document/workspace), `code_hover`, `code_rename` (preview-diff → + review-flow apply), `code_diagnostics`. Budgeted outputs (cap + spill per CE-G2), + renderers that show locations as clickable `file:line`. +- **Engine:** a gateway-side **LSP supervisor** (`lsp-supervisor.ts`) built on + `vscode-jsonrpc` + `vscode-languageserver-protocol` (Microsoft-maintained, standalone- + capable; coc.nvim is the reference lifecycle implementation): spawns language servers per + `(worktree, language)`, idle-shutdown (default 10 min), capped concurrent servers, + health-restart — the `sandbox-manager.ts` idempotent-ensure shape. Servers are *processes + the gateway owns*; agents never talk LSP directly. The supervisor also pushes **post-edit + diagnostics** into the turn (the Claude Code v2.0.74 pattern — fix without a separate + build step). The operational invariants (didOpen tracking, position-encoding negotiation, + readiness gating, client-side file watching, orphan reaping…) are enumerated in + [code-intelligence-alternatives.md §3](code-intelligence-alternatives.md) and are CI-3 + pinning tests. +- **Language autodiscovery:** a detection registry maps manifest/file signals → + language-server descriptor (`tsconfig.json`/`package.json` → **vtsls**; + `pyproject.toml` → **basedpyright**; `go.mod` → gopls; `Cargo.toml` → rust-analyzer + (per-worktree `CARGO_TARGET_DIR`); `.csproj` → csharp-ls; `.fsproj` → FsAutoComplete; + `pom.xml`/`build.gradle*` → JDT-LS; `CMakeLists.txt`/compile-commands → clangd; Kotlin + deferred — kotlin-lsp is alpha). Descriptors ship as **per-language packs** + (`lsp-typescript`, `lsp-python`, …): the descriptor is data (binary acquisition, args, + init options, health probe), so adding a fleet language = a pack, not core code. v1 + bundles TypeScript + Python; the rest are cards (CI-4) — autodiscovery *detects* an + uncovered language and surfaces "install the lsp-go pack" in the UI instead of failing. + This config-not-binaries pack model is the same shape Claude Code (LSP plugins) and + Copilot CLI (`.github/lsp.json`) converged on independently. +- **Why not Serena/MCP as the surface:** same capability, but token-opaque outputs, no + renderer, no tool-guard tiers, Python runtime baggage per worktree, and the observed + MCP-underuse problem. Serena remains the design reference and the escape hatch — a + Serena-backed pack behind the same `code.symbol-nav` capability is ~a week's work if the + supervisor stalls. Full argument: + [code-intelligence-alternatives.md §2.1](code-intelligence-alternatives.md). + +## §3 Structural layer — ast-grep tools + +`ast_search(pattern, lang?, paths?)` and `ast_rewrite(pattern, rewrite, …)` over +**ast-grep** (MIT, the de-facto structural engine for agents), shipped as the +`@ast-grep/cli` npm dependency — prebuilt per-platform binaries via optionalDependencies, +no custom shipping; per-call execution with `--json=stream`, nothing to supervise, ~25–31 +languages + custom tree-sitter grammars. `ast_rewrite` never writes directly: it returns a unified diff rendered through +the existing diff renderer; applying goes through the normal edit path so review-pane and +git-status flows see it. This is the single biggest verifiability upgrade over regex/sed — +and the first goal to land (CI-1). + +## §4 Spatial layer — repo map + visualization + +Two deliberately separate concerns: + +- **Prompt/orientation path (in-house, small):** `code_map` — an Aider-style budgeted + **ranked** map: tree-sitter tag extraction (defs **and** refs) → file/symbol graph → + personalized PageRank seeded by session state (files in context, identifiers mentioned in + the goal) → signature-only rendering, binary-searched down to a **hard token budget** + (default 2 KB tool result, arg-raisable to 8 KB; later an EP provider contributes a + prompt section under the same budget machinery as the skills catalog). Content-hash + cached per worktree. The ranking is what makes a 2 KB map useful — flat symbol dumps + aren't (evidence: RepoGraph ICLR'25 ≈+33% relative on SWE-bench-Lite; details and the + Aider algorithm in [code-intelligence-alternatives.md §5](code-intelligence-alternatives.md)). + Owner asked "isn't in-house overkill?" — no: extraction reuses tree-sitter/ast-grep + machinery and Aider's MIT `tags.scm` queries; the *graph database* part of graphify is + precisely what we don't need for prompts. Deterministic, relative-path-only, + worktree-safe by construction. Embeddings indexing: deliberately not in v1 (contested + evidence, real infra cost — annex §5); the capability seam leaves room for a pack. +- **Visualization path (reuse, optional):** a `graphify` **marketplace pack** (not built-in) + wrapping graphify per worktree: output to a gitignored `graphify-out/`, rendered in a pack + panel (`#/ext/graphify`). Known risk to verify at install: absolute-path leakage in + `graph.json` across worktrees (reported upstream; verify against current release — + treat the graph as per-worktree derived cache, never committed, which sidesteps the + team-workflow/merge-driver features entirely). + +## §5 Truth layer — structured diagnostics + +`code_check(scope)` runs the project's known checkers (from project config; autodiscovered +defaults: `tsc --noEmit`, eslint, pytest, `go vet`/`go test`, `cargo check`) and returns +**typed results** — a flattened-rdjsonl record `{tool, file, line, endLine?, col?, code, +severity, message, fix?}[]` + summary counts — never raw logs (cap + spill for the rare +overflow). Parser strategy: native JSON where it exists (eslint, ruff/Biome rdjson, mypy, +cargo, go), stable-regex for `tsc --pretty false`, JUnit XML for pytest, a SARIF→internal +adapter (unlocks dotnet/semgrep/clang in one converter), and the bundled reviewdog +**`errorformat`** static binary as the universal fallback (~50 preset parsers + user `-efm` +patterns) — that one ships via the existing `binaries/` mechanism. Verification gates can +consume the same parser output, and the team-lead's "did it work" loop becomes data-driven. +Unknown checkers fall back to capped raw output (never block). No shipping agent harness +has this layer typed today — Claude Code feeds raw hook text, Codex/Devin read raw logs. + +## §6 UX — "what's running, where?" (owner requirement) + +Code-intel services are invisible infrastructure; the user must be able to see and steer +them from where they live — the chat: + +- **Context services chip** in the chat header (next to the git-status widget): shows the + active session's scope (global / project / worktree) and the live code-intel services for + it (e.g. "TS ✓ · Py ✓ · map 4.2k"). Click → popover: per-service status, restart, open + pack panel (graphify viz, map inspector), install-suggestion for detected-but-uncovered + languages. +- Pack panels remain the deep-dive surface (`#/ext/…`); the chip is the discoverability + bridge the platform currently lacks — designed here, generalized for all packs with + runtimes/services as a follow-up card under EP (the chip reads a generic + "active pack services for this session" endpoint, not CI-specific state). + +## §7 Worktree semantics (the contract) + +Everything in CI obeys: keyed by worktree root · derived state in +`/.bobbit-cache/` or the server cache dir (both gitignored) · relative paths only +in any artifact · LS instances per worktree with pooling/idle-shutdown · goal worktree +cleanup also disposes CI state (Caretaker sweep covers leaks). A pinning test greps CI +artifacts for absolute paths — the graphify lesson, enforced. + +## §8 Cost & verifiability acceptance (program-level) + +- CE ledger (CE-G0.1) dimensions for CI tools; success = bash share of discovery calls + falls materially (CE-G7.1 targets <40%; CI should push further), grep-cascade turns + replaced by single-call lookups on the BENCH suite (CE-G0.3) at equal task success. +- Every mutating surface (`ast_rewrite`, `code_rename`) previews before applying; every + CI tool result is budgeted; `code_check` closes every edit loop. + +Cross-references: [code-intelligence-alternatives.md](code-intelligence-alternatives.md) +(alternatives, citations, research record), +[time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) (CE-G2/G7), +[extension-platform.md](extension-platform.md) (capabilities §3.1, providers, runtimes), +[harness-gap-analysis.md](harness-gap-analysis.md) (peer evidence), +[mission-control.md](mission-control.md) (Caretaker sweep, flight recorder for service +lifecycle events). diff --git a/docs/design/comms-stack/04-current-state-and-backlog.md b/docs/design/comms-stack/04-current-state-and-backlog.md new file mode 100644 index 000000000..900882f0a --- /dev/null +++ b/docs/design/comms-stack/04-current-state-and-backlog.md @@ -0,0 +1,922 @@ +# Bobbit Real-Time Comms Stack — Step-4: Current-State Verdict + Hand-off Task Backlog + +> **Date:** 2026-06-10 · **Baseline:** master @ `6ec8c8f9` · Follows +> [01-understanding.md](01-understanding.md) (S1–S46 seam table), +> [02-analysis.md](02-analysis.md) (verdicts, RC1–RC8, test-fidelity §4), +> [03-remediation-plan.md](03-remediation-plan.md) (WP0–WP11 / PR-0..D). +> +> **What this document is:** (§2) a seam-by-seam verdict on what actually shipped vs. what the +> plan promised; (§3) settlement of the open NEEDS-TRACE residues; (§4) a register of NEW +> defects found beyond the 46 catalogued seams; (§5) the updated invariant ledger with every +> unpinned invariant flagged; (§6) **the deliverable — an ordered backlog of self-contained +> hand-off tasks** (each executable by an agent who sees only that task's text), grouped into +> waves with acceptance gates. +> +> **Status:** backlog not started. Workstream **CS** in +> [../fable-program-execution-plan.md](../fable-program-execution-plan.md) (program +> sequencing + master checklist). Land CS tasks in the §7 merge-map order; `session-manager.ts` +> is also touched by CE-G3 and EP — serialize per the execution plan's shared-seam table. +> +> **Method:** every claim was re-verified against the current tree by parallel adversarial +> audits (file:line anchors below are current as of `6ec8c8f9`, not the drifted anchors in +> docs 01–03). SDK ground truth was read from the installed `pi-coding-agent`/`pi-agent-core` +> and cross-checked against the published 0.77.0 tarball. +> +> **Environment caveat (re-verify before implementing):** `package.json` declares +> `@earendil-works/pi-* 0.77.0` but this checkout's `node_modules` contains only +> `@mariozechner/pi-*` (pi-coding-agent at **0.67.5**). The SDK-behaviour claims below +> (steer expansion-before-queue, steering queue retained across abort, ack-after-compaction, +> `switch_session` no-replay, SDK-internal auto-retry default-on) were verified against the +> published 0.77.0 sources and agree across three independent reads — but implementers must +> `npm install` and re-confirm SDK anchors in their own tree. + +--- + +## 1. Executive summary + +**What shipped.** One PR (#674, `d520620f`, 2026-06-01) landed: WP0 (partial), WP1 (RC2 +images), WP2 (RC1 dedup, minus the prefixed-S17 leg), WP3 (S2 outbox + S31 caps), WP4 (RC3 +seq-less broadcasts, minus the snapshot-reconcile step), the S9 half of WP5, the S8 +grace-race + S40 halves of WP9/WP10, S42, S14, S3, and (earlier, separately) the WP11 +search-flush teardown fix. **What did not ship:** WP5's S25 leg, WP6 (S16/S38), WP7 +(S19/S20), WP8 except S14 (S32/S34/S35/S37), most WP10 standalones (S26/S33/S36/S43), S4, +S13, S29, two of the three WP11 flake fixes, and several promised WP0 harness legs (faithful +mock abort shape, `message_start`, steer-image threading, the behavioural auto-retry/status +harnesses). + +**Symptom-family status.** +- **(B) images** — substantially fixed (render-from-content + live enrichment, faithful + pins). Residuals: S26 steered images (data-loss, untouched), a **new deterministic + duplicate** on blank-text attachment-only sends (F10), document attachments still never + reach the model (S20). +- **(A) duplicates** — reducer-level holes closed and pinned. Residuals: the prefixed + error-recovery echo (S17 leg, deliberately deferred, still open), a raw-role keying edge + (F11a), S4 double-Enter (never shipped), and several **new server-side duplicate-turn + mechanisms** that dwarf the reducer class: SDK-vs-Bobbit double auto-retry (F5), + graceful-abort steer double-delivery (F3a), recovery-after-kill at-least-once re-dispatch + (F2b, F9). +- **(C) freeze / dead Stop / lost prompts** — Stop is now reliable (3s force-kill, seq'd + synthetic `agent_end`); the outbox closes the headline silent-loss. Residuals: S25 resume + stall (open), outbox loss on navigation-away and mid-flush (F14), steers parked dormant in + the SDK (F3b — deterministic, including across restart), prompts that never auto-drain + after `preparing`/restore (F7), **a gateway-crash bug** (F1), and a respawn-concurrency + cluster that can split-brain tabs and corrupt the status channel (F2). +- **(D) jank / teardown** — almost nothing shipped (only S14/S31). S19/S20/S32/S34/S35/S37 + all still open as catalogued. + +**The biggest single discovery of this pass:** the prior audit's queue/steer/respawn +*interaction surfaces* hide a cluster of HIGH-severity, mostly deterministic server-side +bugs (F1–F7) that none of the 46 catalogued seams covered. They share three root causes: +**(RC-A)** no lifecycle fencing of the old `SessionInfo` across in-place respawn; **(RC-B)** +Bobbit's `idle` status conflates "dispatchable" with two busy states the SDK actually has +(compacting, internal-retry backoff); **(RC-C)** steer delivery is neither at-most-once nor +at-least-once — the shadow ledger, the SDK steering queue, and the persisted prompt queue +each own the text at different moments with no reconciliation contract. + +--- + +## 2. Seam-by-seam current status (all 46 + plan packages) + +Verdicts: ✅ fixed+faithfully pinned · 🟡 partially fixed / fixed-with-open-edge · +❌ still open (unshipped) · ⚪ refuted/obsolete (no action). Anchors are current. + +| Seam | Verdict | Current state (anchors) | +|---|---|---| +| S1 slot | 🟡 | Image leg closed: slot demoted to one-shot fallback `remote-agent.ts:2562-2580`, live enrich `message-reducer.ts:241`. Open: document/PDF tiles still slot-dependent; fallback branch itself unpinned. | +| S2 send-drop | 🟡 | Outbox shipped (`remote-agent.ts:243-245,1435-1477`, flush on auth_ok `:749`; pinned by `tests/remote-agent-outbox.spec.ts`, real bundle). Open edges → **F14**: navigation-away discards a non-empty outbox (`src/app/session-manager.ts:930-941` caches only when `connected`); `_flushOutbox` doesn't re-push on failure (`:1472-1477`); `unsent:true` rendered by nothing; no browser E2E. | +| S3 IME | ✅ | `MessageEditor.ts:507-513` (isComposing + keyCode 229, before slash-menu). Pinned: `message-editor-ime.spec.ts` (real component). | +| S4 double-Enter | ❌ | Never shipped (assigned PR-B). Editor still clears after awaits (`AgentInterface.ts:1467-1491`), `onSend` un-awaited `:2113`, no re-entrancy flag; fixture clears synchronously (inverse of prod). → **CS-D9**. | +| S5 seq-less bypass | ✅ | All three frames via `emitSessionEvent` (`session-manager.ts:2642,2677,6166`). Pinned: `seqless-broadcast-exhaustive.test.ts` (source-walk, scoped to session-manager.ts only — caveat noted in ledger). Exhaustive re-scan found **no remaining S5-class hole**; the on-attach `compaction_start` unicast (`handler.ts:356`) and the `BOBBIT_E2E` replay shim are intentional/benign. | +| S6 image render gap | ✅ | `Messages.ts:246-267` rich-wins-else-content + `imageAttachmentsFromContent:789-809`. Pinned: `user-message-image-render.spec.ts` (real Lit bundle). | +| S7/S10 id-less empty-text dup | ✅ | `synth:seq:` stamp `message-reducer.ts:249-251`, `plainTextEquivKey:90-94`, empty-aware multisets `:393-399,441-451`. Pinned: `message-reducer-dedup.test.ts` (real reducer). Full-stack E2E leg still missing (mock abort shape unfaithful) → **CS-H1**. | +| S8 wedged Stop | 🟡 | Grace race fixed: listener-before-abort, un-awaited IIFE+catch, force-kill at ~3s (`session-manager.ts:6101-6136`), seq'd synthetic agent_end `:6166`. Watchdog deliberately dropped (plan §2.5.2). Residual: force-kill branch never clears `streamingStartedAt`/persisted `wasStreaming` → stale heartbeat payload + phantom continuation after a later hard crash → **CS-R9**. Pinned: `session-manager-force-abort-grace.test.ts` (real SessionManager). | +| S9 overflow stall | ✅ | Overflow sets `_seqInitialized=false` (`remote-agent.ts:1672-1683`); three recovery paths consistent. Pinned: `remote-agent-seq-overflow.spec.ts` (real bundled `handleServerMessage`). | +| S11 hand-off flicker | ⚪ | Settled by analysis: reducer mutates state before emit; container clear + list inclusion commit in the same task's microtask drain — no paintable blank frame. No action; pin only if the hand-off ever moves off the synchronous path. | +| S12 | ⚪ | Stays refuted (container self-heal pinned). | +| S13 30s ack double-dispatch | ❌ | Unchanged: `rpc-bridge.ts:445` 30s default, pending deleted on timeout `:454-457`, `compact()` gets 120s `:521`; recovery re-dispatches via `recoverPromptDispatch` (`session-manager.ts:2166-2209`). → **CS-R8**. | +| S14 UTF-8 split | ✅ | Persistent `StringDecoder` (`rpc-bridge.ts:213-214,374,380`). Pinned: `rpc-bridge-utf8-split.test.ts` — real decoder + real `handleData`, but the test mirrors the stdout wiring itself (a revert of `:374` would pass); decoders never `end()`-flushed / not re-created in `_attachProcessHandlers` (low — respawns build new bridges). Hardening pin → **CS-H2**. | +| S15 switch_session seq inflation | ⚪ | Closed from source (0.77.0 AND 0.67.5): `switchSession` rebuilds state with a single `session_start`, no per-message replay. The `restoring` guard comment at `session-manager.ts:3694-3704` is **stale and false** → cleanup in **CS-P3**. | +| S16 continuation prompt | ❌ | WP6 never shipped: `restoreSession:3766-3776` unconditional on `wasStreaming`; all grant modes + restartAgent + role-switch reach it via `_respawnAgentInPlace:3015`. → **CS-D6** (after CS-R2). | +| S17 skill-expand dup | 🟡 | Unprefixed client fallback shipped (`reconstructModelText`, `message-reducer.ts:102-117,282-288`) but is **inert in production** (expansion is server-side; optimistic rows never carry `skillExpansions`). The real defect — prefixed error-recovery echo missing the exact-match splice (`session-manager.ts:553-584` vs prefix at `:1941`) — is still open and now joined by the same-seam envelope leak (huntB C10). → **CS-D3**. Characterization pin at `message-reducer-dedup.test.ts:75`. | +| S18 optimistic survivor | 🟡 | Multiset consume shipped + over-dedup non-regressions pinned (`message-reducer.ts:485-499,301-312`). Open edge: `plainTextEquivKey` uses **raw role**, so a `user-with-attachments` optimistic vs `role:user` snapshot copy (documents / blank-text) never matches → duplicate (F11a) → **CS-D2**. | +| S19 queue base64 broadcast | ❌ | `broadcastQueue` still broadcasts+persists `toArray()` (`session-manager.ts:1782-1789`); a single ~1.5MB image can trip the 4MiB overflow-terminate on a slow client. → **CS-P1**. | +| S20 document bytes | ❌ | `remote-agent.ts:950-955` ships full attachments; `dispatchDirectPrompt` (`:2225`) forwards only `(text, images)`. → **CS-P1**. | +| S21 orphaned banner | 🟡 | Frames now seq'd+replayable (S5 fix). Open: snapshot handler never clears `_state.autoRetryPending` (WP4 step 4 unimplemented; only writers `remote-agent.ts:2374,2386,2401`); the `clients.size===0` cancel-skip (`session-manager.ts:2663-2679`) + forceAbort-idle path leaves a stuck banner for a later resume. → **CS-D7**. | +| S22 status mirror tests | ❌ | `session-manager-status.test.ts` inline mirrors and `fixtures/remote-agent-status.html` (omits `_maybeReplayGrant` + archived branch) still the only pins. → **CS-H2**. | +| S23 seq fixture mirrors | 🟡 | Overflow branch now real-code-pinned; plain dedup/reorder/baseline + `_advanceTopLevelSeq` + resume_gap still mirror-pinned (`remote-agent-seq-dedup.html`, `remote-agent-sequence-hole.html`). → **CS-H2**. | +| S24 | ⚪ | Stays refuted. | +| S25 perm resume hole | ❌ | `EventBuffer.pushFrame` still non-retaining (`event-buffer.ts:41-43`); resume `since()` has a permanent hole across a DENIED/TIMED-OUT perm; `event-buffer.test.ts:209-233` still pins the **pre-fix** contract. Severity now medium-low (S9 overflow self-heals after 500 events). → **CS-D8**. | +| S26 steered images | ❌ | Four drop points: `rpc-bridge.ts:479-481` (no images param), batch `session-manager.ts:2073,2089`, rollback `:2098-2100`, reconcile `:2136-2138`, drain batch keeps only `steered[0].images` `:2257-2259`. SDK supports steer images (rpc-mode `session.steer(message, images)`). → **CS-D4**. | +| S27 steer-echo expansion mismatch | **confirmed-latent** | Settled from source, no unobservable left: SDK `steer()` expands `/skill:`-prefixed and template-prefixed text **before** queueing (agent-session `_expandSkillCommand`/`expandPromptTemplate`); Bobbit's ledger holds raw text (`session-manager.ts:2073,2087`); `_consumeSteerEcho` exact-`indexOf` misses (`:2114-2123`, its own comment admits it) → stale entry re-dispatches on a later abort (skill runs twice). → folded into **CS-R3/CS-R4**. | +| S28 | ⚪ | Stays refuted. | +| S29 reconnect refetch | ❌ | `onReconnect` (`remote-agent.ts:755-771`) → 3 REST calls per reconnect (`src/app/session-manager.ts:1384-1390`); git-status deduped, bg-processes + annotations not. Mitigated by backoff. → **CS-P4**. | +| S30/S41/S43/S44/S45 | ⚪ | Stay refuted (S43's residual listener leak → **CS-P3**). | +| S31 payload caps | ✅ | `WS_MAX_PAYLOAD_BYTES=256MiB` (`server.ts:264,1346`); composer measures the **real serialized frame** (`MessageEditor.ts:56,62-73`) as the FIRST statement of `handleSend` (`:654-668`, draft preserved). Pinned: `ws-max-payload.test.ts` + `message-editor-size-guard.spec.ts` (real component). | +| S32 proposal scans | ❌ | `remote-agent.ts:2125-2170`, fresh RegExps per delta. → **CS-P2**. | +| S33 truncated flush | ❌ | `remote-agent.ts:2462-2468` plain `break`. → **CS-P3**. | +| S34 rect loop | ❌ | `AgentInterface.ts:672-684` via `_updateAndPin:723-733`. → **CS-P2**. | +| S35 sync base64 | ❌ | `attachment-utils.ts:88-95`. → **CS-P2**. | +| S36 unguarded handler broadcast | ❌ | `handler.ts:154-196`. → **CS-P3**. | +| S37 aigw header fork | ❌ | `aigw-manager.ts:381-389`; the two aigw tests **pin the defect**. → **CS-P2**. | +| S38 grant replay | 🟡 | Delivery leg fixed (replay rides the outbox). Open: fixed 200ms timer, still fired from the idempotent/heartbeat branch (`remote-agent.ts:1232-1241,1719-1721`); zero tests of any kind. → **CS-D6**. | +| S39 deferred-block a11y | ❌ | Unchanged; correctly re-scoped out of WP10 (own staged package; not in this backlog — UI-a11y, not comms). | +| S40 forceAbort cancels retry | ✅ | `session-manager.ts:6089-6093` before the `:6096` early-return. Pinned in the force-abort-grace test (real SessionManager). | +| S42 splice multiset | ✅ | `splice-inflight-message.ts:85-130` count-map. Pinned: `session-manager-getmessages-splice.test.ts:170-186`. | +| S46 mime mislabel | ✅/🟡 | MIME preserved end-to-end. Cosmetic residual: `image-N.png` filename hardcoded in both helpers — and post-WP1 the generic name now shows **live** too. Fold into any future image touch (noted, not tasked). | +| WP11 | 🟡 | search-flush ENOENT ✅ (awaited `closeAll`, `remove()` deliberately fire-and-forget per the docs deviation note); project-assistant-saved-state ✅; pre-compaction-history ❌ (no `__pinAgentSessionFile`); dynamic-chat-tabs ❌ (still samples the GET-mount hash, not `currentFilenameTab.state.contentHash`). → **CS-T1**. | +| repro-h3 case (A) | fixme | Still `test.fixme` (`repro-h3-snapshot-live-interleave.spec.ts:144`), correctly — owned by the separate snapshot-live-race goal; the deterministic gate (WP0 step 10) was never authored. → **CS-T2**. | + +--- + +## 3. NEEDS-TRACE residues — settled + +| Residue | Outcome | +|---|---| +| **S11** | **Refuted as a defect.** The state mutation precedes the emit; the container clear and the message-list inclusion are issued synchronously in one task and commit in its microtask drain — the browser cannot paint between them. The two-scheduler structure remains (re-pin only if the hand-off is ever made async). | +| **S13** | **Confirmed latent, fix unshipped.** Code path fully proven (ack emitted only after in-prompt auto-compaction; 30s vs the maintainers' own 120s compact budget). The wall-clock measurement is not decision-relevant. → **CS-R8**. | +| **S15** | **Closed.** No per-message replay in `switchSession` (verified in both 0.77.0 and the installed 0.67.5). The `restoring` guard's comment is stale/false → cleanup in **CS-P3**. | +| **S27** | **Settled — confirmed latent with an exact trigger** (no unobservable left): a live steer whose text starts with `/skill:` or `/` is expanded by the SDK before echo; the raw-text ledger entry never clears; a later abort in the same process lifetime re-dispatches it (skill runs twice). Severity low-medium; compounds with F4 (ledger never reconciled on respawn). → **CS-R3/CS-R4**. | +| **S29** | **Still open by design** (PR-D deferred); partially mitigated (reconnect backoff, git-status in-flight dedup, seq-resume instead of full refetch). → **CS-P4**. | + +--- + +## 4. New findings register (beyond the 46) + +Detailed diagnosis/repro/fix lives in the task cards (§6); this table maps finding → task. +Severity = (data-loss vs transient) × reachability. + +| ID | Finding (one line) | Severity | Task | +|---|---|---|---| +| **F1** | `drainQueue`'s bare synchronous `rpcClient.prompt()` throw inside the recovery `setTimeout(0)` is an `uncaughtException` → `process.exit(1)` — **the whole gateway dies** (window: in-flight dispatch ∩ in-place respawn) | **HIGH (total outage)** | CS-R1 | +| **F2** | No lifecycle fencing across `_respawnAgentInPlace`: stale-`SessionInfo` writers corrupt `statusVersion` past the carried frame-of-reference (permanently wrong status, heartbeat can't heal), clobber the persisted store (duplicate turn), leave the auto-retry timer alive (phantom turn into the new process); concurrent respawns / `addClient` dormant-revives spawn N children on one `.jsonl` (split-brain tabs, leaked processes, lost seq seed); grant `newTools` computed from role not `session.allowedTools` (double grant-respawn) | **HIGH** | CS-R2 | +| **F3** | Steer delivery semantics: (a) graceful abort + in-flight steer → text delivered **twice** (SDK retains it, Bobbit re-enqueues from the ledger); (b) the `agent_end` "safety net" (`session-manager.ts:2413-2418`) and the live-steer/turn-end race dispatch `steer()` to an **idle** SDK → text parks dormant (no echo, pill gone, phantom splice row, injected into an arbitrary future turn — or lost on restart) | **HIGH (duplicate model instruction / silent input loss)** | CS-R3 | +| **F4** | Steer ledger durability: in-memory only — in-place respawn and gateway crash silently destroy accepted-but-undelivered steers (bubble persists, model never sees it); `_dispatchSteer`'s failure rollback double-enqueues when a reconcile already drained the ledger; S27 expansion mismatch leaves permanent entries | **MEDIUM-HIGH (input loss)** | CS-R4 | +| **F5** | The SDK's **internal** auto-retry is enabled by default (2s base) and Bobbit's `maybeAutoRetryTransient` (~1s) races it: Bobbit's retry fires into the SDK's idle backoff window → duplicate user message / unsolicited turn / double spend; SDK-internal attempts also inflate `consecutiveErrorTurns` toward the human-gate park | **HIGH (duplicate turns, cost)** | CS-R5 | +| **F6** | No compaction gate on any dispatch path: `enqueuePrompt`/`drainQueue`/`retryLastPrompt`/implicit-unstick all dispatch into an in-flight compaction (status is `idle` during compaction); SDK `compact()` clobbers in-flight turn state, and **manual compact disconnects all agent listeners** → a turn started then streams into the void and skips `.jsonl` persistence (permanent message loss) | **HIGH (message loss / context corruption)** | CS-R6 | +| **F7** | Prompts accepted during `preparing`/`starting` and queues restored at boot **never auto-drain** (no idle-transition drains the queue); during a respawn window `enqueuePrompt` returns success-shaped `{status:"queued"}` without enqueuing | **MEDIUM (delivery stall / silent loss)** | CS-R7 | +| **F8** | = S13 (30s ack double-dispatch), confirmed unshipped | MEDIUM | CS-R8 | +| **F9** | `messageQueue` + `wasStreaming` ride the 1s store debounce (not in `RECOVERY_CRITICAL_FIELDS`): hard-kill windows yield a phantom continuation turn, a duplicate prompt (at-least-once re-dispatch of an accepted prompt), or a lost queued prompt; plus the WP9 residual (force-kill never clears `streamingStartedAt`/`wasStreaming`) | MEDIUM | CS-R9 | +| **F10** | Blank-text attachment-only send → **deterministic duplicate user bubble**: server synthesizes `ATTACHMENT_ONLY_TEXT` ("Attachments:"), optimistic text is `""`, no reconciliation path matches | MEDIUM (deterministic, single-tab) | CS-D1 | +| **F11** | Reducer hardening: (a) raw-role `plainTextEquivKey` defeats S18 dedup for document/blank optimistic rows; (b) live-echo artifact consume picks the EARLIEST same-text historical row in a no-optimistic tab (multi-tab row vanishes); (c) `synth:seq:` ids reuse after a gateway restart (resume_gap doesn't reset reducer state) → sub-second row morph | LOW-MEDIUM | CS-D2 | +| **F12** | = prefixed-S17 + huntB C10: error-recovery prefix breaks the skill-expansion splice AND leaks the orphaned envelope in `pendingSkillExpansions` (can mis-splice a future byte-identical message) | LOW-MEDIUM | CS-D3 | +| **F13** | = S26 steered images (4 drop points) | MEDIUM (data loss) | CS-D4 | +| **F14** | Outbox edges: navigation-away discards a non-empty outbox (silent loss with "queued" pill shown); `_flushOutbox` drops frames on mid-flush close/throw (no re-push); `unsent:true` has zero UI readers; outboxed `retry` frames invisible; no spawned-gateway E2E (the auth_ok→flush wiring is untested) | MEDIUM | CS-D5 | +| **F15** | = S16 + S38 (grant/restart continuation double-prompt; heartbeat-branch replay) | MEDIUM | CS-D6 | +| **F16** | = S21 residual (snapshot never clears `autoRetryPending`; zero-clients cancel orphan) | LOW | CS-D7 | +| **F17** | = S25 perm resume hole | MEDIUM-LOW | CS-D8 | +| **F18** | = S4 double-Enter | MEDIUM | CS-D9 | +| **F19** | Late RPC `response` frames after a `sendCommand` timeout are re-broadcast as seq'd session events (`rpc-bridge.ts:688-698`); a straggling compact response can synthesize a spurious `compaction_end{success:false}` client-side (`remote-agent.ts:2706-2718`) | LOW | CS-P3 | +| **F20** | `_advanceTopLevelSeq` gap path + skipped `compaction_end` → snapshot handler re-adds the compacting placeholder (stale "Compacting…" card). Narrow post-S5 | LOW | (documented; fold into any CS-D8 follow-up) | + +**NEEDS-TRACE (named unobservable + settling experiment):** +1. *Extension-command steer infinite redispatch* — a steer starting with a registered + extension command makes SDK `steer()` throw → rollback re-enqueues → re-fails at every + tool boundary, silently. Settle: steer `/login` mid-turn against the real agent; watch + `queue_update` churn. (If confirmed → fold into CS-R3 acceptance.) +2. *Stop during prompt preflight ghost bubble* — abort before the run starts → no + `agent_end` for the pending prompt → `recoverPromptDispatch` deliberately drops it; the + optimistic bubble may persist with no echo until the next snapshot. Settle: client-side + trace of the rendering outcome. (Arguably intended — user pressed Stop.) +3. *Compaction spinner stuck after `restart_agent` mid-compaction* — child killed before + `compaction_end`; handler flips `isCompacting` on the old object only. Settle: mock + harness — manual compact, restart mid-flight, assert the card reaches a terminal state. + (CS-R6's gate should make this window rarer; verify there.) +4. *`exit`-vs-buffered-stdout race* triggering F1 without a respawn (child prints + `agent_end` then exits; Node may deliver `exit` first). Settle: stub child. CS-R1's fix + covers it regardless. +5. *`deliverLiveSteer` cap-park doesn't cancel a pending auto-retry timer* + (`session-manager.ts:1994-2003` vs `enqueuePrompt:1880`) — product decision more than a + trace; note in CS-R5. + +--- + +## 5. Invariant ledger (current) — abridged to deltas + unpinned set + +The full 50-invariant ledger with per-invariant enforcement points and fidelity judgments +was compiled during this audit; the load-bearing summary: + +**Faithfully pinned (real production code under test):** IME guard; composer size guard +(first-statement ordering + draft retention); outbox enqueue/flush/pill (file:// real +bundle); overflow re-baseline (real `handleServerMessage`); seq-less-bypass-empty +(structural, scoped); EventBuffer push/seq/eviction/canResumeFrom; synth:seq stamping + +empty-text multisets + over-dedup non-regressions (real reducer); live==snapshot image +enrichment (real reducer) + tile render (real Lit bundle); streaming-row single-surface; +StreamingMessageContainer self-heal; forceAbort grace race + retry-cancel (real +SessionManager); splice multiset; UTF-8 decoder (object-level); queue persistence +exactly-once across restart (`steer-gateway-restart.spec.ts`); store durability +(tmp+fsync+rename + epoch latch); search-flush teardown. + +**UNPINNED or mirror-pinned (each is a gap a regression walks through silently):** +1. S4 double-Enter — unenforced and unpinned. (CS-D9) +2. Prompt slow-ack budget (S13) — unpinned; mock acks instantly. (CS-R8) +3. Steer shadow-ledger transitions (push/rollback/consume/reconcile incl. the S27 + miss-path) — zero unit coverage; steer E2Es are happy-path. (CS-R3/R4) +4. Steered-image forwarding (S26) — unpinned. (CS-D4) +5. `broadcastQueue` payload projection (S19) — no projection exists. (CS-P1) +6. `snapshot-clears-streaming-message.test.ts` is a **source-text regex scan** — behaviourally unpinned. (CS-H2) +7. Snapshot-authoritative `autoRetryPending` clear — missing invariant + code. (CS-D7) +8. Auto-retry timer fire/cancel behaviour — `auto-retry-policy.test.ts` is a pure-decision + mirror; `queue-dispatch.spec.ts` uses a fake timer object; the `status!=='idle'` fire + guard is untested. (CS-H2, then CS-R5) +9. Handler-local broadcast back-pressure (S36) — unguarded and unpinned. (CS-P3) +10. Grant replay (S38) — zero tests. (CS-D6) +11. Respawn continuation suppression (S16) — unenforced. (CS-D6) +12. Perm-hole resume (S25) — unpinned; `event-buffer.test.ts:209-233` pins the *defective* + contract. (CS-D8) +13. Client seq dedup/reorder baseline, `_advanceTopLevelSeq`, resume_gap, and the status + state machine — still pinned only by hand-copied HTML mirrors + (`remote-agent-seq-dedup.html`, `remote-agent-sequence-hole.html`, + `remote-agent-status.html` — the latter omits `_maybeReplayGrant` and the archived + branch); heartbeat/resync server halves are inline mirrors in + `session-manager-status.test.ts`. (CS-H2) +14. The `eventSeq` plumb feeding the synth:seq stamp (`remote-agent.ts:2365`) — no + real-RemoteAgent test; a frozen/zeroed seq would collapse all id-less rows to one id. (CS-H2) +15. `broadcastStatus` single-writer routing — convention + comment only, no structural pin. (CS-H2) +16. Respawn `_snapshotStreamingFrameOfReference` threading — the carry-over tests mirror + the threading inline rather than driving `_respawnAgentInPlace`. (CS-R2 acceptance) +17. statusVersion monotonicity **across concurrent stale writers** — violated today (F2a); + no pin. (CS-R2) +18. Steer-ledger / queue / `.jsonl` exactly-once delivery contract — does not exist as an + invariant today (F3/F4/F9). (CS-R3/R4/R9 define and pin it) + +--- + +## 6. THE BACKLOG — ordered waves of self-contained hand-off tasks + +Conventions for every task: **master stays green** (`npm run check`, `npm run test:unit`, +`npm run test:e2e` at/above current baseline); **no flaky tests** (a failure is a real +bug); **test-first** — author the red test before the fix (where master-red would violate +the green gate, use a characterization-green + `test.fixme` tagged with this task's ID); +minimal root-cause change; never start background servers from bash (use the harness); +all anchors verified at `6ec8c8f9` — re-verify before editing. Wave order is +(symptom impact × confidence × low risk) under dependencies; tasks within a wave are +parallel unless a dependency says otherwise. + +--- + +### WAVE 0 — Harness fidelity (unblocks faithful pins for waves 1–2) + +#### CS-H1 — Complete the mock-agent fidelity work (WP0 leftovers + steer semantics) +- **Serves:** symptom families A and C; prerequisite for CS-R3, CS-D1, CS-D4 E2E legs. +- **Targeted code:** `tests/e2e/mock-agent-core.mjs` (abort case ~`:2112-2165`, steer case + ~`:2057-2110`, prompt echo ~`:679-690`), `tests/e2e/in-process-mock-bridge.mjs`, + `src/ui/components/Messages.ts:500/748` (add `data-testid="aborted-banner"` only), + NEW `tests/mock-abort-shape.test.ts`. +- **Diagnosis:** the mock still cannot express three real wire behaviours, so whole defect + classes are untestable: (1) the DEFAULT abort emits only `agent_end`+idle — the real SDK + emits `message_start → message_end(role:'assistant', stopReason:'aborted', + content:[{type:'text',text:''}], errorMessage) → turn_end → agent_end → idle`; the + empty-text aborted row is exactly the S7/S10 dedup class. (2) `message_start` is never + emitted anywhere. (3) The mock converts a steer into an immediate prompt and models + abort-with-pending-steer as a *drop* — the real SDK **retains** the steering queue across + a graceful abort and replays it at the next run's initial steering poll; all five steer + E2Es currently pass against wrong semantics. Steer `images` are also discarded. +- **Steps:** (a) default abort → the full faithful sequence above (keep `MOCK_ABORT_AS_ERROR=1` + as the explicit error-shape override); audit every abort-consuming spec for the new DOM + banner row / message counts / first-message_end ordering. (b) emit + `{type:'message_start', message}` before every `message_end`. (c) steer: retain + steered texts across a graceful abort and replay them (own `message_start/end(user)`) + at the start of the next run; add a `MOCK_STEER_DROP_ON_ABORT=1` override for the old + behaviour; thread `msg.images` through the steer case. (d) `data-testid="aborted-banner"`. +- **Repro/red tests:** `mock-abort-shape.test.ts` (node:test over `MockAgentCore`) asserting + the exact sequence incl. `turn_end`; a steer-retention unit case (steer → abort → next + prompt → echo includes the retained steer exactly once). +- **Non-goals:** no production `src/` change beyond the `data-testid` attribute; do not + change default echo timing. +- **Acceptance:** `git diff src/` contains only the testid; all existing unit+e2e pass with + updated assertions; the two new node:test files green; every `message_end` in mock output + preceded by `message_start`. +- **Risk:** the abort-shape change perturbs abort-consuming specs (known: `abort-status-e2e`, + `steer-during-bash-tool`, error-gated-drain tests) — widen assertions deliberately, do not + silence. Rollback: revert the mock file. + +#### CS-H2 — Retire the mirror/inline pins; add the missing real-code harnesses +- **Serves:** invariant-ledger gaps #6, #8, #13–15; S22/S23. +- **Targeted code:** tests only. Replace/supplement: `tests/fixtures/remote-agent-status.html` + + `remote-agent-status.spec.ts` (mirror omits `_maybeReplayGrant` + archived branch); + `tests/fixtures/remote-agent-seq-dedup.html` / `remote-agent-sequence-hole.html` (dedup, + reorder, baseline, `_advanceTopLevelSeq`, resume_gap branches); inline mirrors in + `tests/session-manager-status.test.ts:107-159`; `tests/snapshot-clears-streaming-message.test.ts` + (source-regex → behavioural); NEW `tests/session-manager-auto-retry-fire-cancel.test.ts` + (real `SessionManager` + `t.mock.timers`: fire-while-idle, cancel-on-new-prompt, + forceAbort-cancels — last one is green already, keep as regression pin); extend the + bundled real-RemoteAgent seq spec (`tests/remote-agent-seq-overflow.spec.ts` pattern / + `tests/fixtures/remote-agent-seq-entry.ts`) with: (i) dedup/reorder/baseline/resume_gap + cases (retiring the HTML mirrors), (ii) **the eventSeq plumb pin** — two id-less + `message_end`s at seqs N, N+1 → two distinct `synth:seq:` ids in `_state.messages`, + (iii) a `session_status` case exercising `_maybeReplayGrant` coupling + archived branch. + Add a stderr-tail + wiring variant to `tests/rpc-bridge-utf8-split.test.ts` (spawned stub + CLI writing a split multibyte char, so a revert of `rpc-bridge.ts:374` goes red). + Optional structural pin: every `session.status` write routes through `broadcastStatus` + (source-walk, mirroring `seqless-broadcast-exhaustive.test.ts`). +- **Acceptance:** each retired mirror either deleted or reduced to a comment pointing at the + real-code pin; new tests green; no production change. +- **Dependencies:** none; parallel with CS-H1. Low risk (test-only). + +**Gate 0:** `git diff src/` empty bar the testid; suites green; the mock can express the +real abort/steer/image shapes; the seq/status/auto-retry real-code harnesses exist. + +--- + +### WAVE 1 — Server reliability: crash, duplicate turns, lost input (the new F1–F7 cluster) + +#### CS-R1 — Gateway crash: make `drainQueue` un-crashable (F1) +- **Serves:** symptom C (and total-outage prevention). Root cause RC-A edge. +- **Targeted code:** `src/server/agent/rpc-bridge.ts:445-448` (`sendCommand` throws + synchronously when `!this.process?.stdin`); `src/server/agent/session-manager.ts:2289` + (bare `session.rpcClient.prompt(...)` in `drainQueue`), `:2208` (recovery + `setTimeout(0)` redrain), `src/server/cli.ts:280-296` (uncaughtException handler exits 1 + for non-EPIPE). +- **Diagnosis:** trigger — (1) a prompt dispatch is in flight (long window: >30s + auto-compaction before the ack); (2) any in-place respawn starts: `_respawnAgentInPlace` + calls `unsubscribe()` **before** `stop()` (`:3022-3024`), so `process_exit` never reaches + `handleAgentLifecycle` and status never becomes `terminated`; (3) the pending prompt + rejects "Agent process exited…" → `recoverPromptDispatch` passes its status gate + (`:2172-2176`, status still `streaming`) → re-enqueue + `setTimeout(0) drainQueue(oldSession)`; + (4) the timer fires with `oldSession.rpcClient.process === null` → **synchronous throw + inside a timer → uncaughtException → `process.exit(1)`** — every session dies. +- **Repro:** mock bridge whose `prompt()` never acks; `enqueuePrompt` (idle) then + `restartAgent()` while the dispatch is pending; assert the tick-0 redrain does not throw + out of the timer (today it does — assert via an injected uncaughtException listener in a + child-process harness, or unit-assert `sendCommand` returns a rejected promise). +- **Fix direction:** (a) make `sendCommand` never throw synchronously — return + `Promise.reject` for the no-process case; (b) belt-and-braces: `drainQueue` checks + `session.rpcClient.running` and wraps the dispatch in try/catch routing to + `recoverPromptDispatch`. **Non-goals:** the broader respawn fencing (CS-R2) — this task + only removes the crash. +- **Acceptance:** new red→green unit test (real `RpcBridge`): `sendCommand` with no child + rejects (never throws); a `drainQueue` against a stopped bridge recovers instead of + throwing. Existing `rpc-bridge-lifecycle.test.ts` green. +- **Risk:** callers relying on the synchronous throw (grep `sendCommand` call sites) — + verify each handles rejection. Rollback: two small reverts. + +#### CS-R2 — Respawn lifecycle fencing: mutex + old-`SessionInfo` neutralization (F2) +- **Serves:** symptom C (frozen status, split-brain tabs, duplicate turns, leaked + processes). Root cause **RC-A** — fixes F2a–f once. +- **Targeted code:** `src/server/agent/session-manager.ts` — `_respawnAgentInPlace` + (`:3015-3045`), `restoreSession` (`:3600-3780`), `restartAgent` (`:3052`), + `_restartSessionWithUpdatedRole` (`:2957`), `grantToolPermission` (`:2804-2870`), + `addClient` dormant revive (`:6028-6046`), `recoverSandboxSessions`, `ensureSessionAlive` + (`:5221`), forceAbort respawn (`:6226-6273`), `recoverPromptDispatch` (`:2166-2209`), + `maybeAutoRetryTransient`/timer fire (`:2594-2649`), `cancelPendingAutoRetry` call-site + audit, `drainQueue` (`:2240-2310`). +- **Diagnosis (five legs, one root cause — no fencing of the old object):** + (a) **statusVersion corruption:** the frame-of-reference is snapshotted at `:3023`, but + in-flight `recoverPromptDispatch`/redrain then broadcast idle/streaming **on the old + object**, bumping its statusVersion past the snapshot and delivering those versions to + still-attached clients; the new session re-issues the consumed numbers with different + statuses → the client's `v<=last` gate drops them; the heartbeat never bumps versions → + **permanently wrong status** (e.g. stuck "streaming"/Stop on an idle session). + (b) **store clobber → duplicate turn:** the stale redrain persists `wasStreaming:true` + and a `messageQueue` containing a row the old child may already have accepted into the + `.jsonl` (the kill ate the ack, not the prompt) while `restoreSession` consumes the same + `ps` → continuation prompt + re-dispatch = the same user turn runs twice. + (c) **auto-retry timer survives every respawn:** `cancelPendingAutoRetry` is called at + `:1880,:2734,:5329,:6093,:6320` — never on any respawn path; the timer's guards read the + OLD object (`sessions.has(id)` true via the new session; old `status` frozen `idle`) → + fires `retryLastPrompt` into the fresh process → phantom "[SYSTEM: The model API returned + an error…]" turn. + (d) **concurrent respawns:** no per-session mutex; `sessions.delete` happens after the + first await (child shutdown up to 3s), so a second trigger (double-click Restart, grant + click, `recoverSandboxSessions`, `ensureSessionAlive`, `addClient` revive) starts a second + full `restoreSession` → **two pi children appending to one `.jsonl`**, loser child leaks + (cost double-count); the first run's `finally` deletes `ps._restartFrameOfReference` so + the second restore can seed `EventBuffer` at seq 1 → every post-respawn frame silently + dropped by attached clients. + (e) **`addClient` dormant revive:** the dormant entry stays `terminated` in the map for + the whole restore; every attach in that window fires another `restoreSession`; each + revive's `.then()` registers its ws on whatever the map holds at that moment → tabs split + across two `SessionInfo`s. Also (f): `grantToolPermission` computes `newTools` from the + *role* rather than `session.allowedTools` (`:2804-2831`), so a second grant during the + first's respawn also respawns. +- **Repro:** (d/e) delaying `RpcBridgeFactory` (2s start); dormant session; `addClient` + twice 100ms apart → factory must be invoked once (today twice). (c) errored turn arms the + timer; `restartAgent`; advance mock timers → `retryLastPrompt` must not dispatch. (a/b) + never-acking bridge + `restartAgent` mid-dispatch → assert no statusVersion regression on + the new session and no duplicate row in the restored queue. +- **Fix direction:** a per-session **respawn generation + in-flight restore promise**: + every entry point (`addClient` revive, `restartAgent`, `_restartSessionWithUpdatedRole`, + forceAbort respawn, `recoverSandboxSessions`, `ensureSessionAlive`) joins the in-flight + promise instead of starting a second restore. A **neutralization epilogue** on the old + object at the top of `_respawnAgentInPlace`: mark it terminated/fenced, clear + `clients`, `cancelPendingAutoRetry(session)`, and stamp a generation that + `recoverPromptDispatch`, `drainQueue`, and the retry-timer closure check before acting + (stale generation → no-op). Fix (f) by computing `newTools` from `session.allowedTools` + for session-scoped grants. **Non-goals:** WP6's continuation-prompt suppression (CS-D6 — + but note both edit `restoreSession`/`_respawnAgentInPlace`: land CS-R2 first, rebase + CS-D6); prompt idempotency keys (CS-R9). +- **Acceptance:** red→green real-`SessionManager` tests for (c), (d/e), plus a + generation-fencing unit (stale recover/redrain no-ops); the in-place-respawn carry-over + pins (`restart-preserves-streaming-frame.test.ts`, `sandbox-recovery-*`) stay green and + are **extended to drive the real `_respawnAgentInPlace`** (closing ledger gap #16); no + statusVersion regression observable by a fake client across a respawn-with-inflight-dispatch. +- **Risk:** the highest-touch task in the backlog (one file, many functions). Blast radius: + every respawn path incl. sandbox recovery. Mitigate by keeping the fence read-only for + all paths except the five entry points; rollback = revert the epilogue + generation + checks. + +#### CS-R3 — Steer delivery semantics: exactly-once between Bobbit and the SDK (F3 + S27 trigger) +- **Serves:** symptom A (duplicate steer) + C (silently lost steer). Root cause **RC-C**. +- **Targeted code:** `src/server/agent/session-manager.ts` — the `agent_end` "safety net" + (`:2413-2418`), `_dispatchSteer` (`:2069-2105`), `_consumeSteerEcho` (`:2114-2123`), + `_reconcileAfterAbort` (`:2133-2141`) + its call sites (agent_end-while-aborting ~`:2427`, + forceAbort `:6160`), `deliverLiveSteer` (`:1994-2050`), drainQueue steered batch + (`:2254-2259`); `src/server/agent/rpc-bridge.ts` (steer `:479-481`; optionally a + `clear_queue` passthrough — SDK rpc supports it). +- **Diagnosis:** + (a) **Graceful abort double-delivery:** `_dispatchSteer` pushes the ledger and the SDK + enqueues the steer; user clicks Stop; the **graceful** settle path leaves the child alive + — the SDK retains the steer in its steering queue, but `_reconcileAfterAbort` also + re-enqueues the ledger text and the post-abort drain dispatches it via `prompt` → the + next run echoes the prompt AND the initial steering poll injects the retained original → + the steer appears twice in the transcript and twice in model context. The reconcile is + only correct for force-kill (process dead). + (b) **Dormant steer:** `rpcClient.steer()` on a non-running agent only enqueues — it + never starts a run. Two paths do this: the agent_end safety net (deterministic — steers + still queued at a non-tool-final `agent_end` are dispatched via `steer()` *after* the run + ended, strictly before `drainQueue` would have dispatched them correctly via `prompt`), + and the live-steer-vs-turn-end race (incl. team-manager nudges, `team-manager.ts:1907,1952`). + Result: pill removed, no echo, ledger entry never clears (phantom `inflight-steer:` row in + every snapshot), text invisible to the model until it pops into an arbitrary future turn — + or is permanently lost on gateway restart (ledger is memory-only, row already removed from + the persisted queue). + (c) **S27:** the SDK expands `/skill:`/template steers before queueing; the echo carries + expanded text; the raw-text ledger entry never matches → permanent entry → duplicate + dispatch on a later abort. +- **Repro:** (a) real-`SessionManager` + fake bridge that retains steer texts across + `abort()` and replays them on the next `prompt()` (CS-H1's mock semantics): streaming → + steer("X") → graceful abort → assert exactly one user-message "X" (today two). (b) fake + bridge that parks steers when no run is active: streaming text-only turn → + `steerQueued(id)` → `agent_end` → assert the text was dispatched via `prompt` or still + queued (today: `steer` RPC after agent_end, queue empty, ledger stuck). (c) steer text + `/skill:x`, echo the expanded body, `forceAbort` → assert `inFlightSteerTexts` empty and + no re-dispatch. +- **Fix direction:** (1) delete the agent_end safety net — leftover steered rows flow into + `drainQueue`'s existing steered-batch `prompt` path; (2) make `_reconcileAfterAbort` + bridge-aware: on graceful abort (process alive) do NOT re-enqueue — the SDK still owns + the text; keep the ledger entry so the splice keeps the bubble, and let the next run's + steering poll deliver it (or expose `clear_queue` over RPC and use it to make abort + at-most-once, then re-enqueue uniformly — pick one model and pin it); re-enqueue only on + force-kill; (3) `_dispatchSteer`'s catch: skip the row re-enqueue when the ledger splice + missed (another path owns the rows — fixes the delayed-failure double-enqueue); (4) make + `_consumeSteerEcho` expansion-tolerant (consume when the echo *starts with* / maps to a + ledger entry after `/skill:`-expansion, or track a dispatch id). **Non-goals:** ledger + persistence (CS-R4); steer images (CS-D4). +- **Acceptance:** the three repro tests red→green; all five steer E2Es green **against the + CS-H1 retain-and-replay mock** (they currently pass against drop semantics — they must be + re-validated, not just re-run); a new invariant test: after any abort path, for every + ledger entry there is exactly one of {SDK-queued copy, Bobbit-queued row} (the + exactly-once contract). +- **Dependencies:** CS-H1 (mock steer semantics). Coordinate with CS-R2 (same file; disjoint + functions except forceAbort — rebase order CS-R2 → CS-R3). +- **Risk:** steer UX semantics change subtly (a steer landing at turn-end becomes a queued + prompt — that is the *correct* documented semantics). Rollback per sub-fix. + +#### CS-R4 — Steer ledger durability: persist + reconcile on respawn (F4) +- **Serves:** symptom C (user input silently lost). Root cause RC-C persistence leg. +- **Targeted code:** `src/server/agent/session-manager.ts` `_dispatchSteer` (`:2074-2087` — + row removal persisted before the RPC; ledger push in-memory only), `_respawnAgentInPlace` + (no reconcile call), `restoreSession`; `src/server/agent/session-store.ts:33-118` + (`PersistedSession` — `inFlightSteerTexts` is not a field), `RECOVERY_CRITICAL_FIELDS` + (`:513-522`); `src/server/agent/splice-inflight-message.ts` (snapshot splice reads the + in-memory ledger). +- **Diagnosis:** a steer the SDK accepted mid-turn is destroyed by any in-place respawn + (grant click, role switch, Restart, sandbox recovery): queue row already removed (and the + removal persisted), ledger discarded with the old object, the `wasStreaming` continuation + prompt resumes the turn **without the user's redirection** — and on reload even the + bubble vanishes (splice reads an empty ledger). Across a gateway crash the same holds. +- **Repro:** real-`SessionManager`: dispatch a live steer (fake bridge holds it), then + `_respawnAgentInPlace` → assert the steer text is re-enqueued as a steered row (today: + gone). Crash leg: persist-and-reload cycle via the store, assert the ledger text survives + as a queued row. +- **Fix direction:** persist the ledger (a `pendingSteerTexts` field next to + `messageQueue`, synchronous-flush class); `restoreSession` re-enqueues persisted ledger + entries as steered rows at front; `_respawnAgentInPlace` calls the (CS-R3 + bridge-aware) reconcile before tearing down. **Non-goals:** changing live-steer dispatch. +- **Acceptance:** repro tests red→green; `steer-gateway-restart.spec.ts` extended with a + mid-turn live steer surviving the restart exactly once; no double-delivery when combined + with CS-R3 (the exactly-once invariant test covers the join). +- **Dependencies:** CS-R3 (defines the reconcile semantics). Risk: store schema addition — + forward-compatible (absent field = empty). + +#### CS-R5 — Disable the double auto-retry layer (F5) +- **Serves:** symptom A/C (duplicate turns, unsolicited turns, premature error-park, cost). + Root cause **RC-B**. +- **Targeted code:** `src/server/agent/rpc-bridge.ts` (bridge start — send + `{type:"set_auto_retry", enabled:false}`; the rpc command exists, rpc-mode `:410-413`) OR + `src/server/agent/session-manager.ts` `maybeAutoRetryTransient` (`:2594-2649`) + + `handleAgentLifecycle` (`consecutiveErrorTurns` increment `:2377`); the SDK's internal + retry is **enabled by default** (settings-manager `enabled ?? true`, maxRetries 3, base + 2000ms) and Bobbit never calls `set_auto_retry` (zero hits in `src/`). +- **Diagnosis:** on a retryable provider error the SDK emits `agent_end` first, then sleeps + 2s·2ⁿ⁻¹ and resumes via `agent.continue()` (no new user message). Bobbit sees the errored + `agent_end` → arms its own timer (~1s first attempt — earlier than the SDK's 2s) → fires + while status is idle (the SDK's backoff window is by construction an idle-status window) + → `retryLastPrompt` re-sends the prompt as a **new user message**; the SDK's own continue + then hits a busy agent and is swallowed. Duplicate user message / unsolicited "[SYSTEM: + continue]" turn / double spend; SDK-internal attempts also increment + `consecutiveErrorTurns` toward the human-gate park during an episode the SDK was + handling. +- **Repro:** mock agent emits `message_end{stopReason:'error', errorMessage:'overloaded_error'}` + + `agent_end`, then accepts the next prompt; assert exactly one driving dispatch. +- **Fix direction (pick one layer, recommend Bobbit-owned):** send `set_auto_retry + enabled:false` at every bridge start so Bobbit's policy (with its UI banner, cancel paths, + and S40 integration) is the only retry layer. Alternative: observe the SDK's + `auto_retry_start`/`auto_retry_end` events (currently unhandled) and suppress/cancel + Bobbit's timer — more moving parts. Also decide NEEDS-TRACE #5 (cap-park in + `deliverLiveSteer` doesn't cancel the timer) here. **Non-goals:** changing backoff + schedules. +- **Acceptance:** red→green unit (real SessionManager + mock bridge capturing + `set_auto_retry`); the existing auto-retry banner E2E green; a regression pin that an + SDK-style internal-retry sequence does not double-dispatch. +- **Risk:** behaviour change for users relying on SDK silent retry — Bobbit's banner+timer + replaces it 1:1. Rollback: drop the command. + +#### CS-R6 — Compaction dispatch gate (F6) +- **Serves:** symptom C (vanished messages, corrupted turns). Root cause **RC-B**. +- **Targeted code:** `src/server/agent/session-manager.ts` — `enqueuePrompt` direct branch + (`:1949-1968`), `drainQueue` (`:2240+`), `retryLastPrompt` (`:2734+`), the + implicit-unstick branch, the `compaction_start`/`compaction_end` handling + (`session.isCompacting` set `:2476/:2490`, read today only by `handler.ts:737`). +- **Diagnosis:** during compaction the SDK is not streaming and Bobbit's status is `idle`; + `enqueuePrompt`'s only gate is `status==='idle'` → prompts/steers/retries dispatch + straight into an in-flight compaction. SDK `compact()` then clobbers + `agent.state.messages`; **manual compact disconnects all agent event listeners for its + whole duration** — a turn started in that window streams into the void and skips `.jsonl` + persistence (permanent message loss). Reachable without exotic timing: queued prompt + + `/compact`; threshold auto-compaction at `agent_end` racing the same `drainQueue`; a + prompt typed during the visible compaction card. +- **Repro:** mock harness: start a (mock) compaction, `enqueuePrompt` → assert the prompt + is parked and dispatches only after `compaction_end` (today: direct dispatch). +- **Fix direction:** treat `session.isCompacting` as busy at every dispatch site (park in + `promptQueue`); drain on `compaction_end`. This single server-side gate also removes the + SDK's concurrent-compaction and manual-disconnect hazards from production reach. + **Non-goals:** touching SDK compaction internals; the composer disable is optional polish. +- **Acceptance:** red→green real-`SessionManager` test; `/compact`-then-prompt E2E ordering + test; NEEDS-TRACE #3 (restart mid-compaction spinner) re-checked under the gate. +- **Dependencies:** none, but **must land before or with CS-R7** (drain-on-idle widens this + hole if the gate is absent — the agent_end→drain path would dispatch into the + threshold auto-compaction window). + +#### CS-R7 — Drain the queue on every transition into idle (F7) +- **Serves:** symptom C (queued prompt stuck forever). +- **Targeted code:** `src/server/agent/session-manager.ts` — `drainQueue` call sites + (today: agent_end `:2456`, enqueue-while-idle `:1968`, recovery timer `:2208`, forceAbort + respawn `:6273`); the idle broadcasts that drain nothing: `session-setup.ts:1036`, + `restoreSession:3746`, `_respawnAgentInPlace:3041`, `assignRole:5083`; the + respawn-window enqueue bug `:1833-1834` (returns `{status:"queued"}` without enqueuing); + `ws/handler.ts:489-510` (the comment promising "they'll drain when the session becomes + idle" — currently false). +- **Diagnosis:** prompts accepted during `preparing`/`starting`, and queues restored at + boot with `wasStreaming:false`, sit as pills forever — no `starting→idle` transition + drains. Internal callers (team prompts, triggers) hitting the respawn window get a + success-shaped return with the message dropped. +- **Repro:** enqueue during `preparing` (mock setup delay); assert dispatch on the idle + transition (today: parked). Boot-restore with a persisted queue; assert drain. +- **Fix direction:** a single drain hook on every transition into `idle` (either inside + `broadcastStatus` when `newStatus==='idle'` or an epilogue at the four call sites), + gated on CS-R6's compaction check; make the respawn-window `enqueuePrompt` either await + the in-flight restore (CS-R2's promise) or return an explicit error. +- **Acceptance:** red→green tests for both repro legs; the handler comment becomes true; + abort-status-e2e (post-abort drain) stays green. +- **Dependencies:** CS-R6 (ordering constraint above); CS-R2 (restore promise, soft). + +#### CS-R8 — S13: prompt ack must survive auto-compaction (F8) +- **Serves:** symptom A (duplicate turn after a long pause). +- **Targeted code:** `src/server/agent/rpc-bridge.ts:445` (30s default), `:454-457` + (pending deleted on timeout — late ack orphaned at `:688-691`), `:466-476` (`prompt()`), + `:521-522` (compact's 120s budget); `session-manager.ts:2166-2209` + (`recoverPromptDispatch` re-dispatch). +- **Diagnosis:** the SDK acks a prompt only after in-prompt auto-compaction; >30s + compaction → timeout → recovery re-enqueues → the next `agent_end` (i.e. the successful + completion of the very prompt that "timed out") re-dispatches it — the same user message + runs twice. +- **Repro:** stub CLI whose prompt handler delays its success line past an injectable + lowered ack timeout; assert the prompt resolves once and exactly one `prompt` command is + ever written (today: two). +- **Fix direction:** keep the pending entry alive past the timeout so a late ack resolves + it and cancels recovery (preferred — preserves a liveness signal), or raise the + prompt-specific budget ≥120s; rely on the `process_exit` path (`:407-432`) for genuine + death. **Non-goals:** changing the busy-guard recovery machinery (correct for genuine + rejections). +- **Acceptance:** the stub-CLI test red→green; `rpc-bridge-lifecycle.test.ts` green; + enqueuePrompt/REST error mapping unchanged for genuine failures. + +#### CS-R9 — Crash-window durability for `messageQueue`/`wasStreaming` + force-kill bookkeeping (F9 + WP9 residual) +- **Serves:** symptom A/C after hard crashes. +- **Targeted code:** `src/server/agent/session-store.ts:513-539` + (`RECOVERY_CRITICAL_FIELDS` + 1s debounce); `session-manager.ts` force-kill branch + (~`:6136-6167` — never clears `streamingStartedAt` or persists `wasStreaming:false`), + agent_end clears (`:2445-2447`), heartbeat payload (`:879-885`), boot continuation + (`:3766-3776`). +- **Diagnosis:** hard-kill inside the debounce window → disk keeps `wasStreaming:true` + after a completed turn (phantom continuation on boot), or keeps a dequeued row + (duplicate turn: continuation + re-dispatch of a prompt already in the `.jsonl`), or + loses a freshly queued prompt entirely. Separately, the force-kill branch leaves + in-memory `streamingStartedAt` set (heartbeat ships a stale value) and persisted + `wasStreaming:true` (a later hard crash then fires a continuation for a turn the user + deliberately killed). +- **Repro:** store-level: mutate queue/wasStreaming, simulate kill before the debounce + flush, reload, assert the documented post-state. Force-kill: extend + `session-manager-force-abort-grace.test.ts` to assert the store update. +- **Fix direction:** promote `messageQueue` + `wasStreaming` to the synchronous-flush set + (both mutate at low frequency — turn boundaries and queue mutations); clear + `streamingStartedAt` + persist `wasStreaming:false` in the force-kill branch. The full + idempotency-key design (dedupe a re-dispatched prompt against the `.jsonl` tail) is a + **non-goal** here — document it as the residual at-least-once edge. +- **Acceptance:** red→green store tests; force-abort test extended; restart E2Es green. + +**Gate 1:** all Wave-1 reds green; a soak of the steer/abort/respawn E2E suite (incl. the +re-validated steer specs) 20/20 under `--repeat-each=20`; no statusVersion regression +observable across a respawn-with-inflight-dispatch; the exactly-once steer invariant test +green. + +--- + +### WAVE 2 — Client-side duplicates, losses, and the deferred RC1/RC5 edges + +#### CS-D1 — Blank-text attachment-only duplicate bubble (F10) +- **Targeted code:** `src/server/agent/rpc-bridge.ts:152-162` (`ATTACHMENT_ONLY_TEXT` + synthesis, applied at `session-manager.ts:1847`); `src/app/remote-agent.ts:908-955` + (prompt build) or `src/app/message-reducer.ts:264-300` (fallback candidate set). +- **Diagnosis:** attach an image, send with an empty composer (allowed: + `MessageEditor.ts:564`, `AgentInterface.ts:1423`). Optimistic text `""`; server + synthesizes "Attachments:"; echo text ≠ `""` → id-fallback, text-fallback and + `reconstructModelText` all miss; snapshot keys (`…|EMPTY|` vs `…|Attachments:`) never + match → **two user bubbles every time**, healing only on reload. Pre-existing, unlisted + in the seam table; it is RC1's no-correlation-id root cause via a third text-divergence + source. +- **Repro (real reducer, no harness change):** `optimistic-prompt {role:'user-with-attachments', + content:[{text:''}], attachments:[img]}` → live `message_end {role:'user', + content:[{text:'Attachments:'},{type:'image',…}]}` → assert 1 row (today 2). +- **Fix direction:** export the synthesis rule (pure constant/function) and mirror it + client-side when building the optimistic row's candidate text set — or carry the + optimistic id through the prompt frame (the principled RC1 fix; bigger). **Non-goal:** + changing the server-side synthesis. +- **Acceptance:** reducer pin red→green; an E2E once CS-H1's auto-image-echo lands + (blank-text + image → one bubble live and after reload). + +#### CS-D2 — Reducer hardening bundle (F11 a/b/c) +- **Targeted code:** `src/app/message-reducer.ts:90-94` (`plainTextEquivKey` — raw role), + `:301-312` (live-echo artifact consume — earliest-match), `:249-251` (synth:seq stamp); + `src/app/remote-agent.ts:1697-1706` (resume_gap — no reducer epoch reset). +- **Diagnosis:** (a) `user-with-attachments` optimistic vs `role:'user'` snapshot copy + (documents / blank-text; images are enriched back and safe) never key-match → duplicate + until the next snapshot. (b) In a tab with no optimistic row, the artifact consume + `findIndex` eats the EARLIEST id-less same-text user row — a *historical* row vanishes + transiently when the same text is sent again from another tab. (c) After a gateway + restart, `EventBuffer` restarts at seq 1; `resume_gap` re-baselines `_highestSeq` without + clearing reducer rows → a new `synth:seq:N` replaces an unrelated old `synth:seq:N` row + for the sub-second window before the snapshot. +- **Repro:** three reducer-unit cases (the agent reports give exact shapes); all + expressible today. +- **Fix direction:** (a) normalise `user-with-attachments`→`user` inside + `plainTextEquivKey`; (b) prefer the LAST matching artifact (`findLastIndex`) or scope to + the snapshot tail; (c) include a connection-epoch nonce in the synth id + (`synth:seq::`) or clear positive-`_order` synth ids on `resume_gap`. +- **Acceptance:** three pins red→green; the existing dedup + over-dedup suite stays green. + +#### CS-D3 — Prefixed error-recovery splice (S17 leg + F12 envelope leak) +- **Targeted code:** `src/server/agent/session-manager.ts:553-584` + (`spliceSkillExpansionsIntoEvent`, exact `p.modelText === body` match `:575`), `:1861-1867` + (envelope stashed keyed on the UNprefixed modelText), `:1941-1942` (the + `buildErrorRecoveryPrefix(...)` dispatch; structure `[SYSTEM:…]\n\n${userText}` — + commit 52f11884 only reworded the snippet); characterization pin + `tests/message-reducer-dedup.test.ts:75`. +- **Diagnosis:** errored prior turn + slash-skill prompt → dispatch is prefixed → the echo + body never exact-matches the envelope → the user sees `/foo` AND a "[SYSTEM: previous + turn failed…]\n\n" wall of text, persistent until reload; the orphaned + envelope stays in `pendingSkillExpansions` and can mis-splice a future byte-identical + message. +- **Repro:** real-`SessionManager` unit — stash an envelope, dispatch prefixed, feed a + synthetic prefixed echo through `emitSessionEvent`, assert the splice rewrites it (today: + miss). Expressible now, no WP0 dependency. +- **Fix direction:** stash the envelope keyed on the **prefixed** text at the `:1941` call + site (smallest), or make the splice match a prefix-stripped body. Then flip the + characterization test at `message-reducer-dedup.test.ts:75` to the fixed expectation. + **Non-goal:** the inert client-side `reconstructModelText` path (leave as harmless + defence; optionally annotate it as production-inert). +- **Acceptance:** unit pin red→green; characterization test flipped; an errored-turn + + slash-skill E2E asserting exactly one user row. + +#### CS-D4 — S26: steered images end-to-end (F13) +- **Targeted code:** `src/server/agent/rpc-bridge.ts:479-481` (+ interface `:109`) — add + `images?`; `session-manager.ts:2073/2089` (`_dispatchSteer` union + `rows.flatMap(r=>r.images??[])`), `:2098-2100` (rollback re-enqueue with images), + `:2136-2138` (reconcile with images), `:2257-2259` (drain batch union); + `tests/e2e/mock-agent-core.mjs` steer-images threading (CS-H1). +- **Diagnosis:** four drop points lose steered images silently; the SDK supports steer + images (`session.steer(message, images)` in rpc-mode). User symptom: "look at this + screenshot" steers the model demonstrably never saw. +- **Repro:** unit — steer RPC payload contains `images`; real-`SessionManager` — a steered + row with images reaches `rpcClient.steer(text, images)` and survives rollback/reconcile; + batch — two steered rows with distinct images dispatch the union. +- **Fix direction:** as targeted above. **Verify first** that the installed SDK's steer RPC + accepts images (env caveat in the header); if not, reject image steers with a visible + error instead of silently threading. **Non-goals:** document attachments on steers. +- **Acceptance:** unit pins red→green; E2E with CS-H1's image echo (steered image renders + AND reaches the mock's prompt/steer command args). +- **Dependencies:** CS-H1; coordinate with CS-R3/CS-R4 (same functions — land after, rebase). + +#### CS-D5 — Outbox hardening (F14) +- **Targeted code:** `src/app/session-manager.ts:930-941` (navigation-away caches only when + `connected`; else `remote.disconnect()` discards the instance + outbox); + `src/app/remote-agent.ts:1464-1477` (`_flushOutbox` — empties up front, skips non-OPEN, + swallows throws, no re-push), `:217` (`unsent:true` — zero UI readers), `:749` (auth_ok + flush wiring — untested); NEW `tests/e2e/ui/send-outbox.spec.ts` (spawned gateway). +- **Diagnosis:** (a) VPN flap → prompt queued (pill shown) → user switches session → + switches back → new `RemoteAgent`, outbox and pill gone — silent loss of typed intent + exactly when the outbox was supposed to protect it. (b) A socket closing mid-flush drops + the remaining frames (the guard's own failure recreates S2). (c) The promised + "waiting to send" affordance doesn't exist (pills look ordinary; `retry` frames have no + pill at all). (d) The auth_ok→flush wiring has no test — deleting the `_flushOutbox()` + call passes the suite. +- **Repro:** (a) bundled-agent spec: non-OPEN ws, `prompt('x')`, simulate the + session-switch path, assert the outbox survives (today: discarded). (b) flush with a + socket that closes after frame 1 of 3 → assert frames 2–3 re-queued. (d) spawned-gateway + E2E: kill the socket, prompt, reconnect, assert exactly one server copy. +- **Fix direction:** cache the outgoing agent even when reconnecting (it self-reconnects); + re-push unsent remainder in `_flushOutbox`; render `unsent:true` distinctly in the pill + strip (plan §2.5.2's chosen UX); decide whether `retry` belongs in the pill channel. + **Non-goals:** cross-reload outbox persistence (accepted loss, no false "sent" state); + ack-based delivery (S13's territory). +- **Acceptance:** the three reds green; the E2E pins auth_ok→flush; exactly-once on flush + (no duplicate when the server also queued via another tab — document the multi-tab + duplicate-intent caveat). + +#### CS-D6 — WP6: respawn continuation suppression + grant-replay gating (F15 = S16+S38) +- **Targeted code:** `src/server/agent/session-manager.ts:3766-3776` (continuation gate), + `:3032-3035` (`_respawnAgentInPlace` finally — delete the transient flag), `restartAgent` + (`:3052+` mutatePs) and the **role-switch** caller of `_restartSessionWithUpdatedRole` + (`:2957` — the three grant modes `:2850/:2856/:2870` must NOT set the flag); + `src/app/remote-agent.ts:1719-1721` (remove the heartbeat-branch `_maybeReplayGrant`), + `:1232-1241` (replay timer — keep). +- **Diagnosis:** `restoreSession` fires "[SYSTEM: …continue where you left off…]" whenever + persisted `wasStreaming` — true for every deliberate respawn (grant pauses the turn + before agent_end). Grant path: server continuation + client 200ms replay = two driving + prompts. Plan review-option (i) is adopted: **keep** the continuation for the GRANT path + (it is the robust driver; `switch_session` does not auto-continue), suppress only for + `restartAgent` and role-switch; drop only the redundant heartbeat-branch client replay. +- **Repro:** real-`SessionManager`: persisted `wasStreaming:true` + `restartAgent` → assert + zero continuation prompts (today 1); genuine boot restore → still 1. Client: heartbeat + idle with a pending grant replay → no send; transition idle → one send (bundled-agent + spec — also closes ledger gap #10). +- **Fix direction:** `_suppressContinuationPrompt` transient on `ps` per the WP6 spec + (suppressed branch still clears `wasStreaming`); negative pin that + `recoverSandboxSessions` does NOT set it. **Non-goals:** the 200ms timer redesign; + suppressing for sandbox recovery. +- **Acceptance:** WP6's acceptance list verbatim (03-remediation §WP6): no continuation + bubble on restart/role-switch; grant still driven; boot/Docker restores keep the prompt; + `restart-preserves-streaming-frame`, `sandbox-recovery-*`, grant E2Es green. +- **Dependencies:** **after CS-R2** (same functions; CS-R2's fencing changes + `_respawnAgentInPlace`/`restoreSession` — rebase this on top). + +#### CS-D7 — Snapshot-authoritative auto-retry banner (F16, WP4 step 4) +- **Targeted code:** `src/app/remote-agent.ts` `case "messages"` snapshot branch + (~`:1546-1638`, near the streaming clear `:1574/:1588`) — add + `this._state.autoRetryPending = null;` (NOT in `case "state"`); + `tests/e2e/ui/auto-retry-banner.spec.ts` (extend). +- **Diagnosis:** the only banner clears are live events (`:2374,2386,2401`); a cancel + emitted with zero clients (`session-manager.ts:2663-2679` skip) or evicted from the ring + leaves a reconnecting tab showing "Retrying in Xs…" forever (reachable via + forceAbort-idle on a detached session, e.g. the team-abort route). +- **Repro:** inject `auto_retry_pending` via the window hook; drive a `messages` snapshot + WITHOUT a cancel; assert the banner is gone (today: stays). +- **Fix direction & acceptance:** exactly WP4 step 4 + its spec extension; the now-seq'd + pending replays after the snapshot if a retry is genuinely pending, so no flicker. + +#### CS-D8 — S25: perm-frame resume hole (F17) +- **Targeted code:** `src/server/agent/event-buffer.ts:41-43` (`pushFrame` retain a benign + `{type:'noop'}` seq-holder); `tests/event-buffer.test.ts:209-233` (BOTH tests pin the + pre-fix contract — update to size 3 / `[1,2,3]`); NEW `tests/perm-frame-resume-hole.test.ts`; + `perm-frame-late-joiner-seq-gap.test.ts` must stay green (still one `pushFrame` callsite + `session-manager.ts:2909`; on-attach still via `getPendingToolPermission`, + `handler.ts:436-444`). +- **Diagnosis:** a perm consumes seq N, never retained; resolved-while-disconnected + (DENY `:2943-2949` / 5-min timeout `:2913-2916` / implicit deny `:2896-2900`) → resume + `since(fromSeq)` replays around the hole → the client gap-buffers everything behind N. + Self-heals only after 500 buffered events (S9 overflow). Do NOT retain the full payload: + the resume loop routes ring entries into `handleAgentEvent`, which has no + `tool_permission_needed` case — the card renders only from the top-level frame; a full + payload would spuriously emit and never render. +- **Repro:** `buf.push(1..3); buf.pushFrame(); buf.push(5,6)`; feed `since(3)` into the + FakeClient sequencer from the late-joiner test → today seqs 5–6 strand. +- **Fix direction & acceptance:** WP5 steps 2–4 + both test edits; eviction-window test + re-verified (pushFrame now counts toward the 1000 cap); the retained holder never paints + a card or a `_state.messages` row. + +#### CS-D9 — S4: double-Enter sends twice (F18) +- **Targeted code:** `src/ui/components/MessageEditor.ts` `handleSend` (~`:654-679` — no + clear, no re-entrancy flag); `src/ui/components/AgentInterface.ts:1467-1491` (clear after + awaits), `:2113` (un-awaited onSend); fixture `tests/message-editor-send.html` clears + synchronously (the inverse of production — replace per the CS-H2 real-bundle pattern). +- **Diagnosis:** Enter #1 fires `onSend` without clearing; `sendMessage` suspends on the + IndexedDB `providerKeys.get` await; Enter #2 (auto-repeat / fast double) re-reads the + same value → two identical prompts, two turns (each optimistic id unique, so no + collapse). +- **Repro:** real-component bundle: macrotask-resolving storage stub, two `keydown{Enter}` + in a tick → assert one send (today two). +- **Fix direction:** clear `this.value`/`this.attachments` synchronously in `handleSend` + BEFORE invoking `onSend`, restoring on the abort/early-return paths (note: the size guard + from S31 is already the first statement — keep it first), OR a synchronous `_sending` + flag. **Non-goal:** touching the await ordering in `sendMessage`. +- **Acceptance:** red→green on the real bundle; size-guard + IME + draft specs stay green. + +**Gate 2:** all Wave-2 reds green; the duplicate-bubble class has zero known reproducible +triggers (blank-text, role-keying, prefixed-skill, double-Enter all pinned); outbox +survives navigation and mid-flush failure; banner reconciles on snapshot; perm resume +hole-free. + +--- + +### WAVE 3 — Payload, hot-path, perf (the unshipped PR-D) + +These four tasks are specified in detail in [03-remediation-plan.md](03-remediation-plan.md) +(WP7, WP8, WP10) — the specs there remain accurate (anchors drifted; current ones below). + +#### CS-P1 — WP7 payload slimming (S19+S20) +- `session-manager.ts:1782-1789` (`broadcastQueue` → `toBroadcastArray()`/`toPersistArray()` + via a new pure `queue-projection.ts`; persist keeps `images[].data` for the restart + re-dispatch invariant); `remote-agent.ts:950-955` (+ pure `stripAttachmentsForWire` in + `attachment-utils.ts` — documents lose `content`/`preview` on the wire, images' bytes ride + `images[].data` once); defensive ingestion strip sparing `msg.images`. Acceptance: WP7's + list; key risk — UI pill strip must render metadata-only rows; verify document chips + render from metadata after reload. Independent. + +#### CS-P2 — WP8 hot-path bundle (S32, S34, S35, S37) +- S32: `remote-agent.ts:2125-2170` — `textHasProposalTag` indexOf precheck + cached + regexes; byte-identical matched behaviour. S34: `AgentInterface.ts:672-684/723-733` — + rAF-coalesce ONLY `_updateAndPin`'s jump-button call (the shared wrapper has ~11 callers + incl. the scroll handler `:1198/:1207` — keep those synchronous); cancel in + `disconnectedCallback`; the scroll-spec suite is the blast radius. S35: + `attachment-utils.ts:88-95` — async `arrayBufferToBase64` yielding every ~1MB. S37: + `aigw-manager.ts:381-389` — `"${BOBBIT_SESSION_ID}"` template; **update the two + literal-pinning tests** (`aigw-headers.test.ts`, `aigw-header-resolver.test.ts`) to the + throw-equivalence invariant — they currently pin the defect. Commit each seam separately. + +#### CS-P3 — Standalone hygiene bundle (S33, S36, S43, F19, S15-comment) +- S33: `remote-agent.ts:2462-2468` trailing-edge flush (`_truncatedFlushTimer`; clear on + `message_end`/reset; the no-stale-post-finalize assertion is the load-bearing red test). +- S36: extract `guardedBroadcast` (owns the WeakSets, diag label as param), behaviour + pinned unchanged for `session-manager.broadcast`, then point `handler.ts:154-196` at it. +- S43: `rpc-bridge.ts:312-316` — `removeAllListeners()` before kill (+ drop the dead + `.toString()`); spy-pin in `rpc-bridge-lifecycle.test.ts`. +- F19: filter `parsed.type === "response"` from the event fan-out at + `rpc-bridge.ts:688-698` (late post-timeout responses must not become seq'd session + events); client guard at `remote-agent.ts:2706-2718` optional. +- S15: rewrite the stale `restoring` comment at `session-manager.ts:3694-3704` (or simplify + the guard) — the "replays every message" claim is false on pi ≥0.67. + +#### CS-P4 — S29 reconnect hydration coalescing +- `src/app/session-manager.ts:1384-1390` + `:2730-2740`: add in-flight dedup for + bg-processes and annotation-store refreshes (mirror the git-status guard `:2855-2857`); + optionally debounce `onReconnect` bursts. Low risk; perf-only. + +**Gate 3:** PR-D acceptance (03-plan Gate 4) holds: `queue_update` frames carry no base64; +no document bytes on the wire; hot-path pins green; all scroll/jump specs green within the +existing `settleFrames` budget. + +--- + +### WAVE 4 — Test-debt mop-up + +#### CS-T1 — WP11 leftovers (two flaky specs) +- pre-compaction-history: implement the test-only `__pinAgentSessionFile` hook + (`session-manager.ts:4531-4541` is the production overwrite site) + the forced-`getState` + prerequisite assertion, per WP11 steps 4–5 (UX decision #29 already chose the hook). +- dynamic-chat-tabs: sample the collapse hash from the field production reads — + `currentFilenameTab.state.contentHash` via `page.evaluate` poll (decision at + `src/app/preview-panel.ts:216-218`) — not the GET-mount response. Acceptance: 20/20 under + `--repeat-each=20`; no quarantine/retries/timeout bumps. + +#### CS-T2 — repro-h3 case (A) deterministic gate +- Author the deterministic snapshot↔in-flight gate (reuse the shipped `USER_ECHO_DELAY` + knob; escalate to a `SNAPSHOT_GATE` trigger only if flaky) but **leave case A + `test.fixme`** — the quarantine comment binds it to the separate snapshot-live-race goal + (docs/design/snapshot-live-race-fix.md), which flips it in its own PR. This task only + removes the harness excuse. + +**Gate 4 (done):** symptoms A–D have no known reproducible trigger; every invariant in §5's +unpinned list is either pinned or explicitly accepted with a documented owner; suites green +with zero flakes. + +--- + +## 7. Dependency / merge map + +``` +CS-H1 ──┬─→ CS-R3 ──→ CS-R4 + ├─→ CS-D4 (also after CS-R3/R4 — same functions) + └─→ CS-D1 (E2E leg only) +CS-H2 (independent) +CS-R1 (independent, land first — trivial + total-outage fix) +CS-R2 ──→ CS-D6 (same functions; rebase) +CS-R6 ──→ CS-R7 (ordering constraint: gate before drain-on-idle) +CS-R5, CS-R8, CS-R9, CS-D1..D3, CS-D5, CS-D7..D9, CS-P1..P4, CS-T1..T2: parallel +File-conflict hotspots: session-manager.ts (CS-R1..R9, CS-D3, CS-D6, CS-P3 — + serialize merges in task order), remote-agent.ts (CS-D2/D5/D7, CS-P2/P3 — distinct + functions, merge cleanly), rpc-bridge.ts (CS-R1, CS-R8, CS-D4, CS-P3). +``` + +Recommended landing order: **CS-R1 → CS-H1‖CS-H2 → CS-R2 → CS-R5‖CS-R6 → CS-R3 → CS-R7 → +CS-R4 → CS-R8‖CS-R9 → Wave 2 (D1–D9, D6 after R2) → Wave 3 → Wave 4.** diff --git a/docs/design/extension-platform-implementation-plan.md b/docs/design/extension-platform-implementation-plan.md new file mode 100644 index 000000000..e8fd28071 --- /dev/null +++ b/docs/design/extension-platform-implementation-plan.md @@ -0,0 +1,771 @@ +# Extension Platform — Implementation Plan (hand-off) + +Status: ready for execution, not started. Workstream **EP** in +[fable-program-execution-plan.md](fable-program-execution-plan.md) (program sequencing + +master checklist; its §0-equivalent rules ARE this document's §0). + +Companion to [extension-platform.md](extension-platform.md) (the WHAT/WHY — read its §3–§12 +before implementing anything). This document is the HOW: a tree of goals, each independently +mergeable and verifiable, written so an implementer agent **that has not seen any other goal** +can execute its goal from this text alone. + +> **Anchor baseline:** master @ `6ec8c8f9` (2026-06-10). Line numbers WILL drift — always +> locate by **symbol name** (function/class/field), and treat line numbers as hints. If a +> named symbol does not exist where described, STOP and re-derive from the pattern file cited +> next to it; do not improvise a parallel mechanism. +> +> **Precision policy:** Goals G1–G3 are specified to file/function level. G4–G9 are specified +> to contract level (their substrate is created by G1–G3 and will have evolved); their +> implementers must re-verify every anchor first and follow the stated contracts exactly. + +--- + +## 0. Universal rules — every goal's definition of done + +These apply to EVERY sub-goal below. A sub-goal is not done until all of them hold. + +1. **Read before editing.** Run `rg "" docs/ tests/ src/` for each seam you touch and + read the hits (AGENTS.md rule). Pinning tests are the invariants — if your change breaks + one, fix your change, not the test. +2. **Test-first.** Author the goal's listed tests BEFORE the implementation; confirm each is + RED on the unmodified tree where the spec says so, then flip GREEN. New invariants you + introduce must each get a pinning test. +3. **Gates:** `npm run check` clean · `npm run test:unit` green · `npm run test:e2e` green + (run the suites relevant to your change per AGENTS.md: UI-only → unit; server → unit+e2e). + Real Docker tests go ONLY in `tests/manual-integration/`. +4. **Every user-facing feature needs a browser E2E** (navigation, happy path, persistence + across reload) — pattern: `tests/e2e/ui/settings.spec.ts`. +5. **No flaky tests.** No `test.skip`/quarantine/retries/timeout bumps. Deterministic fixtures + only. +6. **Master stays green; minimal change.** Do not refactor neighbouring code, do not touch + seams owned by other goals (each goal lists its owned files). +7. **Conventions:** LF line endings; lowercase-kebab file names; new server code under + `src/server/…` mirrors sibling style; packs built like `market-packs/pr-walkthrough` + (TS in `src/`, built bundles in `lib/`, wired into `scripts/build-market-packs.mjs` and — + for built-in band packs — `scripts/copy-builtin-packs.mjs` `FIRST_PARTY_PACKS`). +8. **Coordination note:** `src/server/agent/session-manager.ts` and `session-setup.ts` are + also touched by the comms-stack reliability backlog + ([comms-stack/04-current-state-and-backlog.md](comms-stack/04-current-state-and-backlog.md)). + Keep edits confined to the named functions; rebase rather than resolve broad conflicts. + +### 0.1 Patterns library (copy these, don't invent) + +| Need | Copy from | +|---|---| +| Pack-scoped contribution loader (yaml per file, validation, containment) | `src/server/agent/pack-contributions.ts` (`loadPackContributions`, `PanelContribution`) | +| Executing pack module code with timeout/isolation | `src/server/extension-host/module-host-worker.ts` (`ModuleHost.invoke(InvokeRequest, timeoutMs)`) + `route-dispatcher.ts` (`RouteRegistry.resolve`) | +| Generated pi extension calling the gateway | `src/server/agent/tool-guard-extension.ts` (`generateToolGuardExtension`, invoked from `session-setup.ts::resolveToolActivation`, appended as `--extension` to `plan.bridgeOptions.args`) | +| Prompt section with byte budget | `src/server/agent/system-prompt.ts` (`buildSkillsCatalogSection`, `skillsCatalogBudget`, `getPromptSections`, `persistPromptSections`) | +| Docker execFile discipline | `src/server/agent/project-sandbox.ts` (DOCKER_BIN, MSYS env handling) | +| Idempotent ensure-with-inflight-dedupe supervisor | `src/server/agent/sandbox-manager.ts` | +| Marketplace REST + atomic install | `src/server/agent/marketplace-install.ts` (`installPack`/`uninstallPack`/`updatePack`) | +| Per-entity activation | `src/server/agent/project-config-store.ts` (`DisabledRefs`, `getPackActivation`/`setPackActivation`) + `pack-resolver.ts` `ActivationFilter` | +| API E2E boot | `tests/e2e/e2e-setup.ts` (`base()`, `bobbitDir()`, `gitCwd()`) | +| Marketplace browser E2E | `tests/marketplace-active-project.spec.ts` | +| Exemplar packs | `market-packs/artifacts` (tool+renderer+panel), `market-packs/pr-walkthrough` (routes/panels/entrypoints + built-in band) | + +### 0.2 One architectural refinement vs the design doc + +`extension-platform.md` §4 sketches the new entity types as `EntityLoader` entities. **The +implementation uses the pack-contributions path instead** (the path panels/entrypoints/routes +already use), because the new types are **pack-scoped** — two packs may each ship a provider +with id `memory` and BOTH must be active; `EntityLoader` name-merging (shadowing by name +across packs) is the wrong semantics. Concretely: providers, hooks, mcp, pi-extensions, +runtimes, and workflow templates are loaded by `pack-contributions.ts`-style loaders into the +`PackContributionRegistry`, keyed `(packId, contributionId)`, with `contents.` lists as +the activation catalogue and `DisabledRefs.` as the per-entity toggles — exactly the +existing entrypoints model. `EntityType` in `pack-types.ts` is NOT extended. Record this in +any code comment that would otherwise cite the design doc's sketch. + +--- + +## 1. Goal map + +``` +G1 Providers + Lifecycle Hub (P1) + ├─ G1.1 Manifest schema 2 + activation plumbing (no deps) + ├─ G1.2 Lifecycle Hub core + worker dispatch + trace (after G1.1) + ├─ G1.3 sessionSetup hook → Dynamic Context prompt section (after G1.2) + ├─ G1.4 Per-turn hooks: gateway events + provider-bridge (after G1.3) + ├─ G1.5 Market UI: provider listing + toggles (after G1.1; parallel with G1.2-4) + └─ G1.6 session-memory pack (after G1.4) +G2 Hindsight pack, external mode (P2) + ├─ G2.1 REST client + stub-server harness (no deps; parallel with G1) + ├─ G2.2 Pack core: provider + routes + config + banks (after G1.4, G2.1) + └─ G2.3 Tools + panel + entrypoints (after G2.2) +G3 Managed runtimes (P3) + ├─ G3.1 Runtime manifest + secrets/env/ports (pure) (after G1.1) + ├─ G3.2 PackRuntimeSupervisor + REST (docker mocked) (after G3.1) + └─ G3.3 Wire-up: lifecycle hooks, hindsight managed mode, (after G3.2, G2.3) + consent line, manual-integration test +G4 Memory depth (P4): G4.1 beforeCompact ordering · G4.2 memory browser v2 · + G4.3 dual-provider dedupe/priority + cost (after G2.x; G4.* parallel) +G5 MCP as pack content (P5): G5.1 entity+discovery band+UI (after G1.1) +G6 Hooks + pi-extensions (P6): + ├─ G6.1 hooks entity + command dispatcher + Hub mapping (after G1.4) + ├─ G6.2 PreToolUse/PostToolUse via tool-guard generalization (after G6.1) + └─ G6.3 pi-extensions entity + trust acknowledgment (after G1.1) +G7 Claude plugin adapter (P7): G7.1 detect+browse · G7.2 convert+skipped report · + G7.3 UI + real-marketplace E2E (after G5, G6) +G8 Capabilities + selectors (P8): + ├─ G8.1 Capability registry (provides/requires/call) (after G1.2) + ├─ G8.2 Selector decision points + proposals + policy (after G8.1) + └─ G8.3 Proposal UI + fixture selector pack (after G8.2) +G9 Workflow templates + flagship packs (P9): + ├─ G9.1 workflows entity + instantiation (after G1.1) + ├─ G9.2 model-selector capability pack (after G8.1) + └─ G9.3 multi-model-delivery pack (after G9.1, G9.2) +``` + +**Merge lanes (parallelizable):** Lane A: G1.1→G1.2→G1.3→G1.4→G1.6 · Lane B: G1.5 · +Lane C: G2.1 (immediately) then G2.2→G2.3 · Lane D: G3.1→G3.2 (after G1.1) · +Lane E: G5, G6.3, G9.1 (after G1.1). **File-conflict hotspots:** `pack-contributions.ts` + +`pack-contribution-registry.ts` (G1.1, G3.1, G5, G6, G9.1 — serialize merges in goal order); +`session-setup.ts` (G1.3, G1.4, G6.3); `server.ts` route block (most goals — keep each goal's +routes in one contiguous block); `marketplace-install.ts` (G3.3, G7.2). + +--- + +## G1 — Providers + Lifecycle Hub + +### G1.1 Manifest schema 2 + provider contribution + activation plumbing + +**Outcome.** `pack.yaml` accepts `schema: 2` with six new `contents` keys and +`provides`/`requires`; `providers/.yaml` files load into the contribution registry; +per-provider activation toggles work end-to-end through the existing REST. No runtime behavior +yet (nothing dispatches providers) — independently mergeable because everything added is inert +until G1.2. + +**Owned files.** `src/server/agent/pack-manifest.ts`, `pack-types.ts` (types only), +`pack-contributions.ts`, `src/server/extension-host/pack-contribution-registry.ts`, +`project-config-store.ts` (DisabledRefs), the pack-activation REST handlers. + +**Context.** `validateManifest` (`pack-manifest.ts:71-138`) currently: requires +`contents.{roles,tools,skills}` arrays, optional `contents.entrypoints` (safe basenames), +**rejects `contents.mcp`** at `:95-97`, ignores unknown top-level keys. +`loadPackContributions(packRoot, manifest)` (`pack-contributions.ts:111`) loads +panels (auto-discovered), entrypoints (filtered by `contents.entrypoints`), routes. +`DisabledRefs` (`project-config-store.ts`) = `{roles?, tools?, skills?, entrypoints?}`; +accessors `getPackActivation`/`setPackActivation` (`:855/:868`). + +**Steps.** +1. `pack-types.ts`: add to `PackManifest`: + `schema?: number` (default 1), `provides?: string[]`, `requires?: string[]`, and + `contents` gains `providers?/hooks?/mcp?/piExtensions?/runtimes?/workflows?: string[]` + (YAML keys: `pi-extensions` maps to `piExtensions`). Do NOT touch `EntityType` (§0.2). +2. `pack-manifest.ts::validateManifest`: + - parse `schema` (must be a positive integer if present; default 1); + - for each new contents key: if present, must be `string[]` of safe basenames (reuse the + entrypoints validation helper); default `[]`; + - `contents.mcp` present **and** `schema >= 2` → accepted; `schema < 2` → keep the existing + rejection message verbatim; + - `provides`/`requires`: optional `string[]`, entries match `/^[a-z0-9][a-z0-9-]*$/`; + - if `schema > 2`: log one warning (`pack.yaml: schema 3 is newer than supported (2)`), + still load the v2 subset. +3. `pack-contributions.ts`: add `ProviderContribution` + `{ id, kind: "memory"|"selector"|"generic", module, hooks: string[], runtime?, budget: + {maxTokens:number, timeoutMs:number}, defaultEnabled: boolean, config?, listName, + sourceFile, packRoot }` + `loadProviders(packRoot, manifest)` mirroring + `loadEntrypoints` exactly: only files named in `contents.providers` load + (`providers/.yaml`); `module` resolved relative to the yaml and containment-checked + against packRoot (same guard the routes module uses); `hooks` entries must be from + `["sessionSetup","beforePrompt","afterTurn","beforeCompact","sessionShutdown"]` + (unknown → load error for THAT provider, pack still loads); budget defaults + `{maxTokens: 1600, timeoutMs: 1500}` clamped to `maxTokens∈[64,8192]`, + `timeoutMs∈[100,10000]`. Duplicate provider id within a pack → `PackContributionError` + (same as panels). +4. `pack-contribution-registry.ts`: index providers like entrypoints (winning-pack collapse, + activation filtering via `DisabledRefs.providers` ↔ `listName`); add + `listProviders(projectId): ProviderContribution[]` returning only installed+active+enabled. +5. `project-config-store.ts`: extend `DisabledRefs` with + `providers?/hooks?/mcp?/piExtensions?/runtimes?/workflows?: string[]`; the pack-activation + REST catalogue response must include the new types (find the handler via + `rg "pack-activation" src/server` and extend the catalogue builder symmetrically). +6. Cache invalidation: confirm `invalidateResolverCaches()` already drops + `packContributionRegistry` (it does — verify, don't re-add). + +**Tests (author first).** +- Extend the existing pack-manifest unit test file (find via `rg "validateManifest" tests/`): + schema-2 manifest with all six keys parses; `contents.mcp` rejected at schema 1 (existing + test stays green) and accepted at schema 2 (RED before step 2); bad `provides` entry + rejected; `schema: 3` loads with warning. +- New `tests/pack-providers-loader.test.ts`: fixture pack dir (tmp) with two providers — + valid one loads with clamped budget; unknown hook name → that provider errored, other loads; + module outside pack root → error; duplicate id → `PackContributionError`; provider NOT in + `contents.providers` → not loaded. +- Extend the registry test (find via `rg "PackContributionRegistry" tests/`): disabled ref in + `DisabledRefs.providers` hides the provider; re-enabling restores it. +- API E2E: `PUT /api/marketplace/pack-activation` round-trips a `providers` disabled ref. + +**Acceptance.** All above green; `git grep "EntityType" | grep -i provider` → no hits (the +refinement honored); zero behavior change for v1 packs (run the full existing marketplace +test set). + +**Non-goals.** Dispatching providers (G1.2); loaders for hooks/mcp/pi-extensions/runtimes/ +workflows (G5/G6/G3/G9 own those — but the *manifest keys* are accepted now). + +--- + +### G1.2 Lifecycle Hub core + worker dispatch + budgets + trace + +**Outcome.** A `LifecycleHub` that resolves enabled providers and dispatches a named hook to +each on the Extension Host worker tier with per-provider timeout, collects `ContextBlock[]`, +applies budgets, fences content, and records a trace. Pure server core + fixture-driven tests; +nothing calls it yet from session paths (mergeable independently). + +**Owned files (new).** `src/server/agent/context-blocks.ts`, +`src/server/agent/lifecycle-hub.ts`, `src/server/agent/context-trace-store.ts`. **Edited:** +`src/server/extension-host/module-host-worker.ts` (one union member). + +**Steps.** +1. `context-blocks.ts`: + ```ts + export interface ContextBlock { + id: string; title: string; providerId: string; + authority: "memory"|"skill"|"tool"|"workflow"|"role"|"generic"; + content: string; reason: string; priority: number; tokenEstimate: number; + } + export function estimateTokens(s: string): number; // ceil(s.length / 4) — same heuristic as PromptSection.tokens + export function fenceBlock(b: ContextBlock): string; + // `\n{content}\n` + // reason/title attribute values: strip newlines + escape double quotes. + export function applyBudgets(blocks: ContextBlock[], perProviderMax: Map, + globalMax: number): { kept: ContextBlock[]; omitted: {block: ContextBlock; why: string}[] }; + // sort by priority desc then provider order; truncate the first over-budget block's + // content to fit (append "…[truncated]"), drop the rest; never partially emit a block + // smaller than 32 tokens — drop instead. + ``` +2. `module-host-worker.ts`: extend `InvokeRequest.exportKind` union with `"providers"`; the + worker resolves the member on the module's **default export object** for that kind (routes + use named exports — mirror whichever branch handles member lookup and add the + default-export branch). No other worker changes. +3. `lifecycle-hub.ts`: + ```ts + export type LifecycleHook = "sessionSetup"|"beforePrompt"|"afterTurn"|"beforeCompact"|"sessionShutdown"; + export interface HookCtx { sessionId: string; projectId?: string; scope: "project"|"global"; + cwd: string; goalId?: string; roleName?: string; prompt?: string; + turn?: { index: number }; budget: { maxTokens: number }; + config: Record; // provider config values + runtime?: { baseUrl: string; headers: Record; status: string }; + gateway: { baseUrl: string; token: string }; // loopback REST for trusted provider code + } + export class LifecycleHub { + constructor(deps: { registry: PackContributionRegistry; moduleHost: ModuleHost; + trace: ContextTraceStore; gatewayInfo: () => {baseUrl:string;token:string}; + globalMaxTokens?: number /* default 4000 */ }); + async dispatch(hook: LifecycleHook, base: Omit): + Promise<{ blocks: ContextBlock[]; diagnostics: HubDiagnostic[] }>; + } + ``` + `dispatch`: resolve providers for `projectId` whose `hooks` include `hook`; for each, + `ModuleHost.invoke({ exportKind: "providers", member: hook, url: , + packRoot, ctx-as-arg })` with `timeoutMs = provider.budget.timeoutMs`. Per-provider + failure/timeout ⇒ diagnostic `{providerId, hook, error|"timeout", ms}`, continue. Validate + each returned block (shape + string fields + `providerId` forced to the provider's id + + `tokenEstimate` recomputed host-side); then `applyBudgets`. Record one trace entry per + dispatch: `{ts, hook, sessionId, providers: [{id, ms, blocks, omitted, error?}]}`. +4. `context-trace-store.ts`: `appendTrace(sessionId, entry)` / + `readTrace(sessionId, limit?)` over JSONL at + `/session-context-trace/.jsonl` (atomic append; create dir lazily; + cap file at 2MB by dropping oldest — simple rewrite). + +**Tests (author first).** +- `tests/context-blocks.test.ts`: fencing escapes quotes/newlines in attributes; budget: 3 + blocks/2 fit → third dropped with reason; oversize first block truncated; <32-token + remainder dropped not truncated; global cap binds before per-provider headroom. +- `tests/lifecycle-hub.test.ts` (node:test, real `ModuleHost`, fixture modules in tmp dir): + (a) two fixture providers return blocks → merged, budgeted, provenance forced; + (b) provider that `await new Promise(r=>setTimeout(r, 5000))` with `timeoutMs: 200` → + diagnostic `timeout`, other provider unaffected, dispatch total time < 1s; + (c) provider throws → diagnostic, no crash; (d) provider returning malformed blocks → + blocks dropped with diagnostic; (e) trace file contains the dispatch record. +- `tests/context-trace-store.test.ts`: append/read round-trip; 2MB cap behavior. + +**Acceptance.** All green; `ModuleHost` route/action tests untouched and green; no session +code calls the hub yet (grep). + +**Non-goals.** Prompt integration (G1.3); endpoints (G1.4); selector hooks (G8). + +--- + +### G1.3 sessionSetup hook → "Dynamic Context" prompt section + +**Outcome.** New sessions dispatch `sessionSetup` through the Hub; returned blocks render as a +**Dynamic Context** section in the system prompt and appear in the existing prompt-sections +inspector with provenance. Proven by a deterministic fixture pack. + +**Owned files.** `src/server/agent/system-prompt.ts`, `session-setup.ts`, +`session-manager.ts` (construction/injection of the Hub only), `tests/fixtures/packs/provider-demo/`. + +**Context.** Pipeline steps run in order `resolveBridgeOptions(:331) → resolveGoalExtensions(:389) +→ resolveTools(:414) → resolvePrompt(:451) → resolveToolActivation(:606)`, driven by +`executePlan` (insertion point ~line 699, before `resolvePrompt`) and `executeWorktreeAsync` +(~line 887). `PromptParts` (`system-prompt.ts:294`), `getPromptSections(:543)`, +`persistPromptSections(:654)` called from `session-manager.ts:1645`. + +**Steps.** +1. `system-prompt.ts`: add `dynamicContext?: ContextBlock[]` to `PromptParts`. In + `getPromptSections` (and the parallel branch in `assembleSystemPrompt` — note both + call sites of `buildSkillsCatalogSection` at `:471` and `:600`; mirror that duality), when + `dynamicContext?.length`, append a final section + `{label: "Dynamic Context", source: "providers", content: blocks.map(fenceBlock).join("\n\n")}` + AFTER all existing sections (it is the freshest, lowest-authority content; placement + pinned by test). +2. `session-setup.ts`: new async step `resolveDynamicContext(plan, ctx)` inserted before + `resolvePrompt` in BOTH `executePlan` and `executeWorktreeAsync`: if `ctx.lifecycleHub` is + set, `const {blocks} = await ctx.lifecycleHub.dispatch("sessionSetup", {...ids, scope: + plan.projectId ? "project" : "global", prompt: plan.initialPromptText ?? undefined})`; + stash on the plan so `resolvePrompt` copies them into `parts.dynamicContext`. Entire step + wrapped in try/catch → log + proceed (provider failures never block a spawn). + Add `lifecycleHub?: LifecycleHub` to `PipelineContext`. +3. `session-manager.ts`: construct one `LifecycleHub` at manager init (deps: the existing + contribution registry + a `ModuleHost` instance + trace store + gateway info from the + same source the tool guard uses) and pass it through the pipeline context. Keep this the + ONLY new session-manager edit. +4. Fixture pack `tests/fixtures/packs/provider-demo/`: + `pack.yaml` (schema 2, `contents.providers: [demo]`), `providers/demo.yaml` + (hooks: all five; budget 512/1000), `lib/provider.mjs` — deterministic: `sessionSetup` + returns one block `{id:"demo:setup", title:"Demo", content:"DEMO_SETUP_BLOCK ", + reason:"fixture", priority:10}`; every hook also POSTs `{hook, sessionId}` to + `ctx.gateway.baseUrl + "/api/ext/route/record"`? — NO: simpler and dependency-free, append + to a file under `ctx.cwd + "/.provider-demo-log"` (the worker has fs; the test reads the + file). Loading: verify `buildPackList` supports a `BOBBIT_BUILTIN_PACKS_DIR` override; if + it does not exist on the current tree, add it (env var consulted where the builtin packs + dir is resolved; production default unchanged) — that override is this fixture's loader + and G-wide test seam. + +**Tests (author first).** +- Unit `tests/dynamic-context-section.test.ts`: `getPromptSections` with two blocks → last + section is "Dynamic Context", fenced content, token count populated; absent when no blocks. +- API E2E `tests/e2e/provider-session-setup.spec.ts` (in-process gateway, + `BOBBIT_BUILTIN_PACKS_DIR` → fixtures): create session → `GET + /api/sessions/:id/prompt-sections` contains the Dynamic Context section with + `DEMO_SETUP_BLOCK`; disable the provider via pack-activation → new session has no section; + fixture log shows exactly one `sessionSetup`. +- Codegen-free; no browser E2E here (inspector UI already renders sections — verify by the + existing prompt-sections UI spec if one exists; otherwise G1.5 covers UI). + +**Acceptance.** Sections appear with provenance; spawn latency without providers unchanged +(no Hub configured ⇒ zero-cost path); failure injection (fixture module throws) still spawns. + +**Non-goals.** Per-turn dispatch (G1.4); UI (G1.5). + +--- + +### G1.4 Per-turn hooks: gateway-event hooks + provider-bridge extension + endpoints + +**Outcome.** `afterTurn`/`sessionShutdown` fire from the gateway's existing event stream; +`beforePrompt` (per turn) and `beforeCompact` fire via a Bobbit-generated **provider-bridge pi +extension** with blocking semantics; per-turn blocks are traceable via +`GET /api/sessions/:id/context-trace`. + +**Owned files (new).** `src/server/agent/provider-bridge-extension.ts`. **Edited:** +`session-setup.ts::resolveToolActivation` (append one more `--extension`), `server.ts` +(4 endpoints), `session-manager.ts` (two event-handler call sites). + +**Design decisions an implementer must follow exactly:** +- **Never mutate the user's message text.** Injecting recall into the outgoing prompt text + would corrupt the transcript echo and re-open the optimistic-reconciliation duplicate class + (see comms-stack docs). Per-turn injection goes through the **system prompt tail**: + the bridge handles pi's `before_agent_start`; inspect the event payload type in + `node_modules/@…/pi-coding-agent/dist/core/extensions/types.d.ts` + (`BeforeAgentStartEvent` / `BeforeAgentStartEventResult`): if the event exposes the current + system prompt, return `{ systemPrompt: stripPreviousTail(current) + tail }`; the tail is + delimited `\n…` so + replacement is idempotent turn-over-turn. If the event does NOT expose the current prompt, + maintain the full prompt client-side: the gateway endpoint returns + `{ tail, fullSystemPrompt }` (gateway knows the assembled prompt — it built it) and the + extension returns the recomposed prompt. Pick whichever the types support; pin with the + codegen snapshot test. +- **Bridge transport** = the tool-guard pattern verbatim: generated TS reads + `BOBBIT_GATEWAY_URL` + `BOBBIT_TOKEN`, `fetch` with an AbortController timeout (2500ms + beforePrompt / 5000ms beforeCompact); on ANY failure return undefined (turn proceeds + without dynamic context — non-fatal is the platform invariant). + +**Steps.** +1. Endpoints (one contiguous block in `server.ts`, bearer-authed like the tool-grant + endpoints): + - `POST /api/sessions/:id/provider-hooks/before-prompt` body `{prompt}` → + `hub.dispatch("beforePrompt", …)` → `{ tail: string, blocks: }` (tail = fenced + blocks joined inside the delimiters; empty string when none). + - `POST …/before-compact` → dispatch, returns `{}` when all flushes settle (bounded by per- + provider timeouts). + - `GET /api/sessions/:id/context-trace?limit=` → trace store read. + - (afterTurn/shutdown have NO endpoint — gateway-internal, next step.) +2. `session-manager.ts`: in the existing agent-lifecycle event handling, on `turn_end` (or + `agent_end` when turns aren't granular) fire-and-forget + `hub.dispatch("afterTurn", {…, turn:{index}})`; on session archive/stop path dispatch + `sessionShutdown`. Locate by `rg "turn_end|handleAgentLifecycle" src/server/agent/session-manager.ts` + — add ONLY the dispatch calls. +3. `provider-bridge-extension.ts`: `generateProviderBridgeExtension(sessionId): string | + undefined` — returns a written tmp file path (mirror `writeToolGuardExtension`'s file + handling); generated source subscribes `before_agent_start` and `session_before_compact` + per the design decisions above. Generate ONLY when at least one enabled provider declares + `beforePrompt` or `beforeCompact` (registry query) — zero overhead otherwise. +4. `session-setup.ts::resolveToolActivation`: after the tool-guard block (`:622-629` pattern), + call the generator and push `--extension ` when defined. + +**Tests (author first).** +- `tests/provider-bridge-extension.test.ts`: codegen snapshot (delimiters present; gateway URL + read; abort timeout present); not generated when no provider wants the hooks. +- API E2E `tests/e2e/provider-turn-hooks.spec.ts` (provider-demo fixture): drive a turn via + the mock agent → fixture log contains `beforePrompt` then `afterTurn`; `GET + …/context-trace` lists both dispatches with timing; kill switch — disable provider, next + turn logs nothing; before-prompt endpoint with a hanging provider responds within budget + with empty tail (RED until timeout handling correct). +- Idempotent-tail unit: applying the tail twice yields one delimited region. + +**Acceptance.** Per-turn injection visible in the mock agent's received system prompt +(in-process mock exposes it — verify via the bridge spy or transcript); a turn with the +provider down completes normally; trace endpoint paginates. + +**Non-goals.** Tool-call hooks (G6.2); compaction-content mutation; UI. + +--- + +### G1.5 Market UI: provider listing + activation toggle + +**Outcome.** The Marketplace pack-detail UI lists a pack's providers (id, kind, hooks, budget, +origin) with working per-provider enable/disable that takes effect synchronously. + +**Owned files.** The marketplace UI module(s) — locate with +`rg -l "pack-activation|packActivation" src/app src/ui` — plus the catalogue REST already +extended in G1.1. + +**Steps.** Mirror exactly how entrypoint toggles render and persist today (same component, +new section "Providers"); show `kind` and the hook list as chips; disabled state persists via +the existing `PUT /api/marketplace/pack-activation`. + +**Tests.** Browser E2E `tests/e2e/ui/marketplace-providers.spec.ts` (model: +`tests/marketplace-active-project.spec.ts`): provider row visible for the fixture pack; toggle +off → persists across reload → (combined with G1.3's API assertion) new sessions skip it. + +**Non-goals.** Provider settings forms (each pack's panel owns its config UI); trace UI. + +--- + +### G1.6 `session-memory` pack + +**Outcome.** Built-in, default-on memory: before each turn the agent receives bounded +"Relevant past work" blocks recalled from the project's existing search index; new sessions +recall against the goal/task spec. Zero external dependencies. This is the platform's flagship +litmus. + +**Owned files (new).** `market-packs/session-memory/` (pack.yaml schema 2; +`providers/recall.yaml` hooks `[sessionSetup, beforePrompt]`, budget `{1200, 800}`; +`src/provider.ts` → built `lib/provider.mjs`); entries in `scripts/build-market-packs.mjs` +(`PACKS` const, line ~91) and `scripts/copy-builtin-packs.mjs` (`FIRST_PARTY_PACKS`, line 23). + +**Provider behavior (spec — implement exactly):** +1. `beforePrompt(ctx)`: skip when `!ctx.prompt` or prompt < 8 chars. Query + `GET {ctx.gateway.baseUrl}/api/search?q=&projectId= + &type=all&limit=8` with bearer `ctx.gateway.token`, 600ms fetch timeout. +2. Filter results: drop rows where `sessionId === ctx.sessionId` (never recall the ongoing + conversation); drop score below a floor (tune with the fixture; start 0.05); keep top K=3. +3. One block per result: + `{ id: "session-memory:"+result.id, title: "Relevant past work", authority: "memory", + priority: 5, reason: "BM25 recall for current prompt", + content: "[] (session <sessionId>, <ISO date>)\n<snippet with <b> tags stripped>" }`. +4. `sessionSetup(ctx)`: same flow keyed on the goal/task spec text when present (the Hub + passes it as `ctx.prompt` for sessionSetup — G1.3 wired `plan.initialPromptText`; extend + to goal spec if available on the plan). +5. Config (in `providers/recall.yaml` `config`): `k` (default 3), `minScore`, + `includeArchived` (default true), `sources` (default all). Read from `ctx.config`. + +**Tests (author first).** +- Unit `tests/session-memory-provider.test.ts` (drive the BUILT `lib/provider.mjs` directly + with a stubbed `fetch` injected via ctx or global): current-session rows filtered; K cap; + score floor; snippet `<b>` stripping; fetch timeout ⇒ returns `{blocks: []}` (never throws). +- API E2E `tests/e2e/session-memory-recall.spec.ts`: seed project with one session containing + a distinctive token (drive the mock agent to emit it, let the indexer flush); create a + second session and send a prompt containing that token; assert the Dynamic Context tail + delivered to the agent (or the trace) contains a `session-memory:` block referencing the + first session; toggle the pack off ⇒ no block. +- Browser E2E: rely on G1.5's toggle spec + extend it to assert this pack's row exists + (it ships in the built-in band, so it's present by default). + +**Acceptance.** Recall works on a fresh dev install with no configuration; total added +latency per turn ≤ the 800ms provider budget (pinned by the timeout test); disabling the pack +returns the system to byte-identical prompts (assert via prompt-sections diff in the E2E). + +**Non-goals.** Embeddings/semantic ranking; summarization artifacts; cross-project recall; +panel UI. + +--- + +## G2 — Hindsight pack (external-URL mode) + +> Implementer note: verify the real Hindsight REST paths against the targeted Hindsight +> release BEFORE coding the client; the contract below is Bobbit's internal client interface — +> the stub mirrors THIS contract, and `hindsight-client` adapts it to upstream paths. +> +> Bank topology is decided: **one shared tag-scoped bank**, tools take `scope:` mapped to +> tag filters (NOT `bank:` switching) — rationale + auto-tag taxonomy in +> [agent-memory.md §3](agent-memory.md). Two upstream checks at this goal: (1) confirm +> tag-filtered recall with strict matching via REST/SDK; (2) confirm **delete-by-tag** +> exists for project offboarding — if absent, record the per-project-bank fallback for +> deletion-sensitive installs in the pack README. + +### G2.1 REST client + stub-server harness *(parallel with all of G1)* + +**Outcome.** `market-packs/hindsight/src/hindsight-client.ts` (built to +`lib/hindsight-client.mjs`) + an in-process stub server usable by every later test. + +**Client contract.** +```ts +export interface HindsightClient { + health(): Promise<{ok: boolean}>; + recall(bank: string, query: string, opts?: {maxTokens?: number}): Promise<{memories: {text: string; score?: number; id?: string}[]}>; + retain(bank: string, content: string, opts?: {tags?: Record<string,string>; sync?: boolean}): Promise<void>; + reflect(bank: string, prompt: string): Promise<{text: string}>; + listBanks(): Promise<{banks: string[]}>; +} +export function createClient(cfg: {baseUrl: string; apiKey?: string; timeoutMs?: number}): HindsightClient; +``` +Every method: AbortController timeout (default 1500ms), throws typed `HindsightError` +(`kind: "timeout"|"http"|"network"`, status?). + +**Stub** `tests/e2e/hindsight-stub.mjs`: `startHindsightStub({port?: 0})` → +`{url, calls: RecordedCall[], setHealthy(bool), seedMemories(bank, mem[]), close()}` — +an `http.createServer` with canned JSON for the five operations, recording every call +(method, path, bank, body) for assertions. + +**Tests.** `tests/hindsight-client.test.ts` against the stub: round-trips; timeout ⇒ +`HindsightError{kind:"timeout"}` within budget; 500 ⇒ http error; auth header sent when +apiKey set. + +### G2.2 Pack core: provider + routes + config + bank scoping + +**Outcome.** `market-packs/hindsight/` with provider (`providers/memory.yaml`, hooks: all +five), pack routes (`status, recall, retain, reflect, banks, config`), config persistence in +the pack store, and scope-aware banks — fully working against an external URL. + +**Spec highlights (implement exactly; full shapes in extension-platform.md §11):** +- Bank ids: `bobbit-proj-<projectId>` / `bobbit-global`; `beforePrompt` recalls BOTH in + parallel (`Promise.allSettled`, one shared deadline = provider `timeoutMs`), merges by + score, caps to budget. `sessionSetup` recalls vs goal/task spec. +- `afterTurn`: build a compact turn summary (user text + final assistant text, capped 2000 + chars) → `retain(projectBank, …, {tags:{sessionId, goalId, roleName}})`, **async**; on error + push `{content, tags, ts}` onto a retry queue in the pack store (`host.store` from routes / + `ctx.store` from provider — same packId namespace), retry queue head on every later + afterTurn; cap queue at 100 (drop oldest, count surfaced via `status` route). +- `beforeCompact`: `retain(…, {sync: true})` of the about-to-be-lost span summary. +- `sessionShutdown`: drain queue best-effort (one pass). +- Config route persists `{mode: "external"|"managed", externalUrl, apiKeyRef, autoRecall, + autoRetain, recallBudget}` in the pack store keyed per scope; provider reads it via + `ctx.config` (G1.1 loader merges store-config over yaml defaults — add that merge here if + G1.1 shipped static config only; keep the merge in the loader, not the provider). +- Dormancy: if neither a healthy runtime (G3) nor `externalUrl` is configured, every hook + returns immediately (`{blocks: []}`) — the pack is installed-but-dormant. + +**Tests.** Unit (bank derivation incl. global fan-out; retain-queue retry incl. cap; +dormancy). API E2E `tests/e2e/hindsight-external.spec.ts` with the stub: configure +externalUrl via the config route → sessionSetup+beforePrompt blocks appear (trace/prompt +sections), turn_end produces a retain with correct bank+tags; `setHealthy(false)` ⇒ session +unaffected + diagnostic recorded + status route reports unhealthy; stub recovers ⇒ queued +retain flushes. + +### G2.3 Tools + panel + entrypoints + +**Outcome.** `hindsight_recall` / `hindsight_reflect` / `hindsight_retain` agent tools +(tool group `tools/hindsight/`, `provider: {type: bobbit-extension, extension: extension.ts}` +— copy the `defaults/tools/shell/` + tool-guard auth pattern; tools POST the pack's routes), +each accepting `bank: "current"|"global"|"all"`; native panel +(`panels/hindsight-memory.yaml` + built `lib/HindsightPanel.js`) with status card, mode/URL +settings, memory search, recent retains + queue counter; two entrypoints (command palette + +deep link) — copy `market-packs/pr-walkthrough` shapes. + +**Tests.** API E2E: `hindsight_recall` invoked by the mock agent round-trips through the +route to the stub with the resolved bank; per-project pack disable ⇒ tools absent from the +session's tool list. Browser E2E `tests/e2e/ui/hindsight-panel.spec.ts`: open panel via +command palette; configure external URL; status flips to connected (stub); search renders +seeded memories; settings persist across reload. + +--- + +## G3 — Managed runtimes + +### G3.1 Runtime manifest + secrets/env/ports (pure, no Docker) + +**Outcome.** `runtimes/<id>.yaml` parsing/validation (`RuntimeContribution`: compose path +containment-checked, services, healthcheck, ports `host: auto`, volumes, secrets +`generate|prompt`, `startPolicy: on-enable|on-demand`), loaded via the contributions path +(G1.1 pattern, `contents.runtimes`); secret generation (`crypto.randomBytes(24).toString +("base64url")`, stored via `SecretsStore` under `runtime.<pack>.<id>.<NAME>`, idempotent); +`.env` rendering (0600) under `~/.bobbit/state/pack-runtimes/<pack>/`; host-port allocation +(bind `net.Server` to 0, record, persist in a state JSON, re-validate on load). All pure + +unit-tested; nothing executes Docker. + +**Tests.** `tests/runtime-manifest.test.ts` (validation incl. compose-path escape rejection); +`tests/runtime-secrets-env.test.ts` (idempotent generation; env file mode 0600; `${secret:X}` +interpolation); `tests/runtime-ports.test.ts` (allocation, persistence, revalidation when the +recorded port is taken). + +### G3.2 PackRuntimeSupervisor + REST (Docker mocked) + +**Outcome.** `src/server/runtimes/pack-runtime-supervisor.ts`: +`ensureRuntime(packName, id)` (idempotent, in-flight dedupe — copy `sandbox-manager.ts`), +`start/stop/restart/status/logs`; compose invocation +`docker compose -p bobbit-pack-<pack> -f <pack>/<compose> --env-file <env> up -d` via the +`project-sandbox.ts` execFile discipline (DOCKER_BIN, MSYS env); health = poll declared HTTP +path until `startupTimeoutMs`; status machine +`docker-unavailable|stopped|starting|running|unhealthy`. REST: `GET /api/pack-runtimes`, +`POST /api/pack-runtimes/:id/{start|stop|restart}`, `GET /api/pack-runtimes/:id/logs?tail=`. +Docker binary injectable (constructor `execFileImpl`) so all tests mock it. + +**Tests.** `tests/pack-runtime-supervisor.test.ts`: mocked exec walks +stopped→starting→running on health 200; health timeout ⇒ unhealthy; missing docker ⇒ +docker-unavailable (exec ENOENT); concurrent `ensureRuntime` ⇒ one compose invocation; stop +issues `compose stop` (not down). API E2E for the three REST routes against the mocked +supervisor. + +### G3.3 Wire-up: lifecycle, Hindsight managed mode, consent, manual-integration + +**Outcome.** Enable-with-runtime = zero-step Hindsight: pack enable (activation) with +`startPolicy: on-enable` starts the runtime **only from an explicit user action** (the enable +click handler — pinned: no auto-start on boot/install); provider `ctx.runtime` injected +(`{baseUrl: http://127.0.0.1:<port>, headers, status}`) and `managed` mode preferred when +healthy; disable ⇒ `compose stop`; `uninstallPack` (pre-delete seam at the function head, +`marketplace-install.ts::uninstallPack`) ⇒ `compose down`, volumes preserved; explicit purge +flag ⇒ `down -v` + state-dir removal. Hindsight pack gains `runtimes/hindsight.yaml` + +`runtime/compose.yaml` (digest-pinned images; default memory-formation model preconfigured +per design §11). Enable card shows the capability summary line for runtimes (images, ports, +volumes) + the memory disclosure (design §8.4). Manual-integration test +(`tests/manual-integration/hindsight-runtime.test.ts`): real compose up → health → +recall/retain round-trip → disable stops → volume survives `updatePack`. + +**Tests.** Unit: keep-vs-purge logic; no-auto-start pin (boot with enabled pack + stopped +runtime ⇒ supervisor not invoked). API E2E (mocked docker): enable → start invoked; disable → +stop; uninstall default keeps volumes (no `-v` in argv), purge adds it. Browser E2E: runtime +status card states + logs view (mocked), enable-card disclosure text present. + +--- + +## G4 — Memory depth *(contract level — re-verify anchors; substrate = G1/G2/G3)* + +- **G4.1 beforeCompact ordering.** Pin that the bridge's `session_before_compact` POST + completes (or times out) BEFORE compaction proceeds, and Hindsight's sync retain lands + before the compacted span is dropped. Test: stub asserts retain arrives before the mock + agent emits `compaction_end` (ordering assertion via recorded timestamps). +- **G4.2 Memory browser v2.** Panel: list/filter (session/goal tags) /delete memories via new + pack routes wrapping `listMemories`/`delete`; retain-queue and operations views. Browser + E2E: browse/filter/delete survives reload. +- **G4.3 Dual-provider compose.** When both memory providers contribute: stable order + (hindsight priority > session-memory), near-duplicate suppression (normalized-text overlap + ≥0.8 ⇒ keep higher-priority block), cost line (memory-formation token estimate) in the + panel. Unit tests on the dedupe; E2E with both enabled asserting one merged Dynamic Context + section under the global budget. + +## G5 — MCP as pack content *(single goal)* + +Lift is already manifest-side (G1.1 accepts `contents.mcp` at schema 2). Add: `mcp/<name>.yaml` +loader (`{name, command|url, args?, env?}` → `McpServerConfig`), a **pack band** in +`mcp-manager.ts::discoverServers` (`:125-170`) inserted at the LOWEST priority position +(packs must never shadow user/project MCP configs — priority below "custom directories"); +provenance (`originPackName`) carried into the tool-info summaries; uninstall ⇒ configs gone +on next discovery (registry-driven, no persistence). Consent: capability summary line lists +each server's command/URL (G3.3 card machinery; if G3.3 hasn't merged, add the line to the +install response only). Tests: unit (loader + precedence: same server name in user config and +pack ⇒ user wins); API E2E (pack-shipped stdio MCP fixture server reachable via existing +`mcp_<server>` meta-tool; uninstall removes); browser E2E (provenance badge in tools UI). + +## G6 — Hooks + pi-extensions + +- **G6.1 hooks entity + dispatcher.** `hooks/<id>.yaml` + (`{event: SessionStart|UserPromptSubmit|Stop|PreCompact|SessionEnd|PreToolUse|PostToolUse, + command, timeoutMs?, matcher?}`) + Claude-layout `hooks/hooks.json` accepted as an + alternative source (one listName per file). Command dispatcher: `execFile` with JSON event + on stdin, capture stdout JSON; map onto the Hub per design §5.5 (SessionStart/ + UserPromptSubmit stdout `additionalContext` → ContextBlock). Non-tool events only in this + goal. Tests: mapping-table unit; E2E fixture pack hook injects context visible in the tail. +- **G6.2 PreToolUse/PostToolUse.** Generalize the tool-guard long-poll endpoint so hook + verdicts (`{block, reason}` / input mutation) ride the SAME generated guard extension + (extend `generateToolGuardExtension` inputs rather than adding a second tool_call + subscriber). Tests: fixture hook blocks a named tool (E2E); allow-path latency unchanged + (no hook ⇒ no extra HTTP round-trip — pin by codegen snapshot). +- **G6.3 pi-extensions entity + trust gate.** `pi-extensions/<name>.yaml` → module path; + appended as `--extension` in `resolveToolActivation` ONLY when the pack has a recorded + acknowledgment. Trust store: `~/.bobbit/state/pack-trust.json` + (`{packName, scope, acknowledgedAt, capabilities}`); default-deny; Market UI acknowledgment + flow + revoke. Hooks (G6.1) consume the same gate. Tests: default-deny pin (un-acked pack + contributes nothing); ack → extension in spawn args (inspect `plan.bridgeOptions.args` via + a unit on the pipeline step); browser E2E ack + revoke across reload. + +## G7 — Claude plugin adapter + +- **G7.1 Detect + browse.** Source sync detects `.claude-plugin/marketplace.json`; source row + gains `format`; browse lists plugins (name/description/version from each `plugin.json`). +- **G7.2 Convert + skipped report.** `claude-plugin-adapter.ts` invoked inside `installPack` + between staging copy and `writeMeta`/rename (seam at `marketplace-install.ts:514-529`): + conversion table per design §9 (`commands/`→skills commands-flat, `skills/`→skills, + `agents/*.md`→roles, `hooks/hooks.json`→hooks, `.mcp.json`→mcp, `${CLAUDE_PLUGIN_ROOT}` + rewrite, synthesized `pack.yaml schema: 2`); returns `skipped: [{feature, reason}]` + embedded in the install response and `.pack-meta.yaml` (`sourceFormat: claude-plugin`). + Tests: fixture plugin repo → converted pack resolves skills/roles through the normal + cascade; every unsupported feature in the fixture appears in `skipped`; nothing silently + dropped (fixture includes a statusline + an unsupported MCP transport). +- **G7.3 UI + real-marketplace E2E.** Source-format badge; skipped report rendered + post-install; browser E2E installs a pinned-commit public Claude marketplace plugin and its + slash command appears in the composer. + +## G8 — Capabilities + selectors + +- **G8.1 Registry.** `provides`/`requires` (already parsed in G1.1) → capability index in the + contribution registry (a capability implementation = a named pack route or provider member + declared `capability: <name>` in its yaml); `ctx.capabilities.call(name, input)` available + to providers and pack routes — host resolves by pack precedence, dispatches on the worker + tier with a 5s default timeout, records to the trace. Install-time check: missing required + capability ⇒ warning in install response + Market UI hint. Tests: precedence; missing-dep + warning; call round-trip provider→capability across two fixture packs; timeout isolation. +- **G8.2 Decision points + proposals + policy.** Hub gains `beforeGoalCreate` / + `beforeSessionSpawn` dispatch points wired into goal-creation and session-spawn paths + (locate via `rg "createGoal|createSession" src/server` and insert PRE-validation, never + post). Selector providers (`kind: selector`) receive typed summaries (roles via + `RoleManager.listRoles`, workflows via `WorkflowManager`, skills catalog, MCP servers, + models via `model-registry`) and return the `SelectorProposal` shape (design §12) — + schema-validated; invalid ⇒ dropped + diagnostic. Approval policy module + (`selector-policy.ts`, pure, fully unit-tested): auto-apply iff pre-session ∧ no tool-grant + expansion vs the default role ∧ model available ∧ persona patch additive-and-bounded + (≤1200 chars, append-only) ∧ confidence ≥ 0.75; else `requiresApproval`. Application goes + through existing stores ONLY (role assignment, workflow selection APIs). Tests: policy + truth-table unit (every clause has a RED case); E2E fixture selector auto-applies a safe + role and flags an unsafe one. +- **G8.3 Proposal UI + fixture pack.** Goal-creation UI surfaces the proposal + (accept/decline, reason shown); decline ⇒ deterministic defaults. Browser E2E both paths. + +## G9 — Workflow templates + flagship packs + +- **G9.1 workflows entity.** `workflows/<id>.yaml` (same gate schema `workflow-validator` + validates today) loaded as TEMPLATES into the registry; instantiation API + (`POST /api/projects/:id/workflows/instantiate {packName, templateId}`) copies through + `WorkflowManager.createWorkflow` (validation reused; collision ⇒ suffix `-2`); goal-creation + UI offers templates alongside project workflows. Project.yaml remains the only source of + truth; goal snapshotting untouched (pin: instantiate → mutate template pack → existing goal + unchanged). Tests: loader/validation reuse; instantiation E2E; snapshot-immunity pin. +- **G9.2 model-selector pack.** `provides: [model-selector]`; route implementing + `{task, spec?, candidates, constraints?} → {model, thinkingLevel?, reason}` with a + deterministic rule table first (cost/capability tiers from `model-registry` data) and an + optional LLM refinement behind config (strict JSON, timeout, fallback to the table). Tests: + rule-table unit; capability call E2E. +- **G9.3 multi-model-delivery pack.** Roles (`planner` frontier+xhigh, `implementer` cheap + precise, `qa`, `reviewer` — each promptTemplate carries the initial-spec handoff stanza), + workflow template plan→implement→qa→review with a ralph-loop verify step (re-run until + green, bounded iterations), `requires: [model-selector]`. Tests: instantiation E2E; goal + snapshot carries per-gate roles with pinned models; selector swap path (capability mocked). + +--- + +## 2. Verification matrix (run per goal; full sweep before each parent-goal close) + +| Check | Command | +|---|---| +| Types | `npm run check` | +| Unit phase | `npm run test:unit` | +| E2E phase | `npm run test:e2e` | +| Real Docker (G3.3, G4 managed paths only) | `npm run test:manual` | +| Pack builds | `node scripts/build-market-packs.mjs && node scripts/copy-builtin-packs.mjs` (then re-run e2e) | +| No quarantines added | `rg "test.skip|fixme" tests/ --new-only` vs base | + +Parent-goal close-out additionally requires: the phase's acceptance criteria in +[extension-platform.md §13](extension-platform.md) all demonstrably true, and a short +goal-report noting any anchor drift corrected (so later goals inherit fresh anchors). diff --git a/docs/design/extension-platform.md b/docs/design/extension-platform.md new file mode 100644 index 000000000..5a6f30847 --- /dev/null +++ b/docs/design/extension-platform.md @@ -0,0 +1,746 @@ +# Bobbit Extension Platform — pack-first providers, hooks, runtimes, and ecosystem compat + +Status: design / implementation blueprint. Each phase in §13 is written to convert 1:1 into a +mission goal with acceptance criteria. + +> **Execution authority:** implement from +> [extension-platform-implementation-plan.md](extension-platform-implementation-plan.md) +> (goal map, owned files, RED→GREEN tests) — including its **§0.2 refinement** (new entity +> types load via the pack-contributions path, NOT new `EntityLoader` entities; that +> supersedes the §4 sketch below). Program-wide sequencing + master checklist: +> [fable-program-execution-plan.md](fable-program-execution-plan.md). + +--- + +## 1. Mission & principles + +Bobbit should stop growing core features and become an **extension platform**: memory, context +providers, skills, roles/personas, workflow templates, MCP servers, lifecycle hooks, managed +services, and UI panels all arrive as **marketplace packs** — installable, updatable, +per-entity toggleable, and uninstallable — instead of as core code. + +Principles (binding on every phase): + +1. **Pack-first from day one.** New machinery (providers, hooks, runtimes, capabilities) ships + as pack entity types immediately. The reference implementations — `session-memory` and + `hindsight` — are real packs in `market-packs/`, with zero feature code in core. Core grows + only *platform* code: loaders, dispatchers, supervisors, policy. +2. **Host-controlled compilation.** Packs and selector LLMs *propose*; Bobbit validates, + budgets, fences, records provenance, and applies at safe lifecycle boundaries. No pack gets + raw system-prompt rewrites or unbounded injection. +3. **Everything inspectable.** Every injected context block carries source, provider, reason, + and token estimate, and is visible in the existing prompt-sections inspector. Invisible + context selection cannot be trusted or debugged. +4. **Failures are non-fatal.** A dead memory service, a hung provider, a crashed worker — the + session always proceeds; the degradation is surfaced, never silent. +5. **Great defaults, changeable settings.** A pack that needs a service (Hindsight + Postgres) + must work from a single Enable click with generated secrets, a preconfigured cheap model, + and sane budgets — while exposing every knob for power users. +6. **One UX for memory.** The agent "just remembers". Which engine produced a memory (built-in + session search vs Hindsight) is provenance detail, not user burden. +7. **Trusted code, honest framing.** Packs are trusted code with resource/crash isolation only + (the deliberate stance of the Extension Host — see `docs/design/extension-host.md` and the + isolation simplification in #732). We disclose capabilities at install/enable; we do not + pretend to sandbox. + +Compatibility targets in scope: **Claude Code plugin format** (plugin marketplaces install as +packs), **MCP servers as pack content**, **pi extensions as pack content**, and +**Claude-style lifecycle hooks**. + +--- + +## 2. Substrate inventory (what we build on — verified anchors) + +| Substrate | Where | What we reuse | +|---|---|---| +| Pack resolver | `src/server/agent/pack-types.ts:26` (`EntityType`, `EntityLoader<T>` at `:138`), `pack-resolver.ts`, `pack-list.ts` | One ordered pack list, scope order `builtin < server < global-user < project`, `pack_order`, `pack_activation` per-entity disable, conflict reporting, synchronous cache invalidation. New entity type = new loader. | +| Manifest validation | `src/server/agent/pack-manifest.ts:71-138` | Schema V1; unknown top-level keys ignored (forward compat); `contents.mcp` hard-rejected at `:95-97` (lifted for `schema: 2`). | +| Marketplace install | `src/server/agent/marketplace-install.ts`, `marketplace-source-store.ts`, REST in `server.ts` (§ marketplace routes) | Git/local/builtin sources, atomic staging install, `.pack-meta.yaml` provenance, update/uninstall. The staging step is the seam for the Claude-plugin format adapter. | +| Built-in first-party band | `src/server/agent/builtin-packs.ts`, `scripts/copy-builtin-packs.mjs`, `BOBBIT_BUILTIN_PACKS_DIR` | Pre-installed, resolve-in-place packs (e.g. `pr-walkthrough`): the delivery vehicle for `session-memory` and `hindsight`, and the test seam for fixture packs. | +| Extension Host execution tier | `src/server/extension-host/module-host-worker.ts`, `route-dispatcher.ts`, `action-dispatcher.ts`; contract in `src/shared/extension-host/host-api.ts` | Confined `worker_threads` execution of trusted pack server modules: per-call timeout (terminate-on-timeout), concurrency cap, per-session rate limit, import containment, pack-scoped stores, surface-binding tokens. Providers run on this tier. | +| Generated pi extensions | `src/server/agent/tool-guard-extension.ts:31` (codegen, `pi.on("tool_call")` + HTTP long-poll via `BOBBIT_GATEWAY_URL`/`BOBBIT_TOKEN`), `tool-activation.ts` (`writeMcpProxyExtensions`, `resolveExtensionPath`) | The proven gateway⇄agent bridge pattern. The provider-bridge extension (§5.4) and hook bridge (§13 P6) are new instances of it. | +| pi extension surface | `node_modules/@…/pi-coding-agent/dist/core/extensions/types.d.ts`, `system-prompt.d.ts` | Events: `before_agent_start` (systemPrompt replace), `context`, `tool_call`/`tool_result`, `turn_start/end`, `agent_start/end`, `session_before_compact`, `session_shutdown`, `input`; `pi.registerTool`; `BuildSystemPromptOptions`. | +| Prompt assembly + inspector | `src/server/agent/system-prompt.ts:294-448` (`PromptParts` → `assembleSystemPrompt`), `persistPromptSections` `:654`, `GET /api/sessions/:id/prompt-sections` (`server.ts:11354`) | Ordered prompt sections with byte budgets (`skillsCatalogBudget` pattern) and a shipping inspector. Provider blocks land here. | +| Session search | `src/server/search/flex-store.ts:155` (`FlexSearchStore`, BM25 + recency boost, chunking), `search-service.ts`, `indexer.ts`, `GET /api/search`; `transcript-reader.ts` + the `read_session` tool | Goals/sessions/messages/staff indexed per project; transcripts kept indefinitely in `.bobbit/state/sessions/<id>.jsonl`. This is the `session-memory` provider's substrate. Missing today: automatic prompt-time recall — exactly the gap providers fill. | +| Docker supervision patterns | `src/server/agent/project-sandbox.ts` (DOCKER_BIN/execFile discipline, MSYS env handling), `sandbox-manager.ts` (idempotent ensure + in-flight dedupe), `aigw-manager.ts` (supervise external HTTP service) | The shapes the pack-runtime supervisor mirrors (not reuses — sandboxes are per-project repo isolation; runtimes are server-scoped services). | +| Secrets | `SecretsStore` (state-dir, gitignored) | At-rest storage for runtime secrets. | +| Stores | `RoleStore`/`RoleManager` (`Role{promptTemplate, accessory, toolPolicies, model, thinkingLevel}`), `WorkflowStore`/`WorkflowManager` (project.yaml-scoped, goal snapshots), `McpManager` (9 discovery locations incl. Claude's), `slash-skills.ts` (incl. `.claude/skills`/`.claude/commands`), `model-registry.ts` | Selectors and workflow templates operate **through** these stores, never around them. | +| Exemplar packs | `market-packs/artifacts` (tool+renderer+panel litmus), `market-packs/pr-walkthrough` (routes/panels/entrypoints, built-in band) | The authoring patterns and the test-litmus convention every new entity type follows. | + +--- + +## 3. Entity model + +Six new pack entity types, plus a capability registry. They stay distinct because they have +three genuinely different execution substrates and two authoring models: + +| Entity | Declared at | Code runs | Purpose | +|---|---|---|---| +| `providers` | `providers/<id>.yaml` + module | **Gateway**, Extension Host worker tier | Typed lifecycle participants: contribute context blocks, react to turns/compaction/shutdown, make proposals (selector kind). | +| `hooks` | `hooks/<id>.yaml` (or Claude `hooks/hooks.json`) | Gateway-spawned child processes | Claude-style declarative command hooks (JSON stdin/stdout); adapt onto the same dispatcher as providers. | +| `mcp` | `mcp/<name>.yaml` | n/a (config) | MCP server configs merged into `McpManager` discovery with pack provenance. | +| `pi-extensions` | `pi-extensions/<name>.yaml` → module | **Agent subprocess** (`--extension`) | Escape hatch + pi-ecosystem compat. Hooks pi events without registering tools. | +| `runtimes` | `runtimes/<id>.yaml` + compose file | **Docker**, gateway-supervised | Managed services (health, ports, volumes, secrets) a pack depends on. | +| `workflows` | `workflows/<id>.yaml` | n/a (templates) | Workflow **templates** instantiated into the project store at project/goal setup. `project.yaml` remains the source of truth; goal snapshotting is unchanged. | + +Notes on the boundaries: + +- **A provider is host-side, not a pi extension.** Prompt assembly happens in the gateway, so + before-prompt contribution with provenance is only possible host-side; recall/retain then + costs no per-call subprocess hop (the gateway already sees every prompt and every + `turn_end`); secrets (e.g. Hindsight keys) never enter the agent process; and the worker + tier's timeout/terminate machinery gives failure isolation for free. The agent-side reach a + provider occasionally needs is delivered by one Bobbit-generated bridge extension (§5.4), + not by pack code in the agent. +- **Provider tools are just tools.** `hindsight_recall` ships as a normal pack tool group with + the existing `bobbit-extension` provider type (the `defaults/tools/shell/` pattern), whose + pi extension calls back to the pack's own routes. No new "provider tool" concept; role + `toolPolicies`, the tool guard, and `pack_activation` apply unchanged. +- **`pi-extensions` is not a new trust tier.** Pack tools already run arbitrary code in the + agent process; this entity just admits it for event-only extensions. Both are gated by the + same acknowledgment (§8). + +### 3.1 Capability registry (inter-pack composition) + +Packs can publish and consume named capabilities: + +```yaml +# pack.yaml fragments +provides: [model-selector] # this pack implements the capability +requires: [model-selector] # this pack calls it (install-time check: warn + suggest) +``` + +A capability is implemented by a pack route or provider method registered under the +capability name. Consumers call it through the host: + +```ts +const res = await ctx.capabilities.call("model-selector", { + task: "implementation", spec, candidates, constraints: { maxCostTier: 2 }, +}); +``` + +The call is host-mediated: routed to the providing pack's module on the worker tier, budgeted +(timeout, payload caps), and recorded in the trace. This is how a persona selector consults a +"best model selector", how a workflow template asks for stage-model assignment, and how future +packs compose without importing each other's code. Conflicts (two packs providing the same +capability) resolve by pack precedence, same as entities. + +--- + +## 4. pack.yaml schema v2 + +Additive; `schema: 2` marks packs using the new keys. v1 validators ignore unknown keys, so a +v2 pack on an old server degrades to its v1 subset (roles/tools/skills/entrypoints load; the +rest is ignored). v2 servers warn when `schema` is newer than supported. + +```yaml +schema: 2 +name: hindsight +description: Persistent agent memory backed by Hindsight (recall/retain/reflect, shared tag-scoped bank — see docs/design/agent-memory.md). +version: 1.0.0 +contents: + roles: [] + tools: [hindsight] # existing: tool GROUP dirs under tools/ + skills: [] + entrypoints: [hindsight-open, hindsight-route] + providers: [memory] # NEW → providers/memory.yaml + hooks: [] # NEW → hooks/<id>.yaml (or hooks/hooks.json, Claude layout) + mcp: [] # NEW → mcp/<name>.yaml (v1 rejection lifted when schema >= 2) + pi-extensions: [] # NEW → pi-extensions/<name>.yaml + runtimes: [hindsight] # NEW → runtimes/hindsight.yaml + workflows: [] # NEW → workflows/<id>.yaml (templates) +provides: [] # NEW: capability names this pack implements +requires: [] # NEW: capability names this pack consumes +routes: # existing (pack-level routes) + module: lib/routes.mjs + names: [status, recall, retain, reflect, banks, config] +``` + +Per-entity declaration files: + +```yaml +# providers/memory.yaml +id: hindsight-memory +kind: memory # memory | selector | generic +module: ../lib/provider.mjs # contained in pack root (existing path-guard discipline) +hooks: [sessionSetup, beforePrompt, afterTurn, beforeCompact, sessionShutdown] +runtime: hindsight # optional: binds to runtimes/<id>; host injects {baseUrl, headers} +budget: { maxTokens: 1200, timeoutMs: 1500 } +defaultEnabled: true +config: # typed settings surface, rendered in the pack's settings UI + mode: { type: enum, values: [managed, external], default: managed } + externalUrl: { type: string, optional: true } + apiKey: { type: secret, optional: true } + autoRecall: { type: boolean, default: true } + autoRetain: { type: boolean, default: true } + recallBudget:{ type: number, default: 1200 } +``` + +```yaml +# runtimes/hindsight.yaml +id: hindsight +kind: docker-compose +compose: ../runtime/compose.yaml # images pinned by digest in the compose file +services: [hindsight, postgres] +healthcheck: { service: hindsight, path: /health, intervalMs: 5000, startupTimeoutMs: 120000 } +ports: + hindsight: { container: 8888, host: auto } # host port allocated + persisted by supervisor +volumes: [pg-data] # named volumes; survive pack update +secrets: + POSTGRES_PASSWORD: { generate: true } # crypto-random at first start +startPolicy: on-enable # never auto-start without a user action +``` + +```yaml +# workflows/multi-model-delivery.yaml (template) +id: multi-model-delivery +name: Multi-model delivery +description: Plan with a frontier model, implement cheap+precise, QA and review with independent models. +gates: [...] # same shape WorkflowStore validates today +``` + +Validator deltas (`pack-manifest.ts`): accept the six new `contents` keys (string arrays, +`isSafeBasename`-guarded), accept `provides`/`requires` (string arrays), lift the +`contents.mcp` rejection when `schema >= 2` (keep rejecting for v1 manifests). New +`EntityType` members + one `EntityLoader<T>` per type (`pack-types.ts:26/:138` — the seam was +designed for this). `pack_activation` extends to all six types so per-entity disable works on +day one. + +--- + +## 5. The Lifecycle Hub + +One new gateway component — `src/server/agent/lifecycle-hub.ts` — the minimal host-controlled +orchestrator. It resolves enabled providers/hooks via the PackResolver, dispatches lifecycle +events to them on the Extension Host worker tier, validates and budgets their output, and +records provenance. + +### 5.1 Hook points + +| Hook | Fires from | May contribute | On failure/timeout | +|---|---|---|---| +| `sessionSetup` | New pipeline step in session setup, between tool resolution and `resolvePrompt` (`session-setup.ts`) | `ContextBlock[]` → new `PromptParts.dynamicContext` → rendered as a **"Dynamic Context"** prompt section, persisted by `persistPromptSections` (visible in the existing inspector) | Provider skipped; diagnostic recorded; session proceeds | +| `beforePrompt` | Gateway prompt-submit path, before `RpcBridge.prompt()` (and per-turn via the bridge, §5.4) | `ContextBlock[]` → fenced `<context-block …>` blocks prepended to the outgoing prompt | Hard timeout (default 1500ms) ⇒ prompt sent without blocks; diagnostic recorded | +| `afterTurn` | Existing event subscription (`turn_end`/`agent_end` in the session manager's lifecycle handler) | Nothing (fire-and-forget; receives the turn transcript slice) | Logged | +| `beforeCompact` | Compaction detection (`session_before_compact` via the bridge) | Nothing in P1 (notification: flush/retain before context loss) | Logged | +| `sessionShutdown` | Session archive/stop path | Nothing (flush) | Logged | +| `beforeGoalCreate` / `beforeSessionSpawn` | Goal-creation and session-spawn paths (P8) | Typed **proposals** (§12) — never direct mutation | Proposal dropped; deterministic defaults | + +Deferred deliberately: mid-turn `context` message mutation, compact-output mutation, +`tool_call`/`tool_result` interception for packs (P6 generalizes the existing tool-guard +long-poll for hooks), `before_provider_request` telemetry. + +### 5.2 Context blocks, budgets, provenance + +```ts +export interface ContextBlock { + id: string; // "<providerId>:<local-id>" + title: string; // "Relevant past work" + providerId: string; + authority: "memory" | "skill" | "tool" | "workflow" | "role" | "generic"; + content: string; // fenced by the HOST, never raw-injected + reason: string; // why this was included — shown in the inspector + priority: number; + tokenEstimate: number; // host-computed (~4 chars/token heuristic, as PromptSection.tokens) +} +``` + +Budgets: per-provider `budget.maxTokens` (default 1200–1600) and a global dynamic-context cap +(default 4000), enforced by priority-ordered truncation — the same approach as +`SKILLS_CATALOG_BUDGET` (`system-prompt.ts`). All blocks are wrapped by the host: + +```xml +<context-block id="hindsight-memory:abc" source="hindsight" authority="memory" + reason="Recall for: refactor the queue dispatcher"> +... +</context-block> +``` + +Trace: `sessionSetup` blocks appear in prompt-sections (free inspector win). Per-turn +`beforePrompt` blocks and all diagnostics (timeouts, skips, budget omissions) append to +`.bobbit/state/session-context-trace/<sessionId>.jsonl`, exposed at +`GET /api/sessions/:id/context-trace`. + +### 5.3 Provider module contract + +```ts +// lib/provider.mjs — runs on the Extension Host worker tier (trusted, confined) +export default { + async sessionSetup(ctx) { return { blocks: [...] }; }, + async beforePrompt(ctx) { return { blocks: [...] }; }, // ctx.prompt, ctx.budget + async afterTurn(ctx) { /* fire-and-forget */ }, + async beforeCompact(ctx) { /* flush */ }, + async sessionShutdown(ctx){ /* flush */ }, +}; +// ctx: { sessionId, projectId?, scope: "project" | "global", cwd, goalId?, roleName?, +// prompt?, turn?, budget, config, // values from providers/<id>.yaml config +// runtime?: { baseUrl, headers, status }, // when bound to a runtime +// store, // pack-scoped KV (existing pack-store) +// capabilities: { call(name, input) }, // §3.1 +// log } +``` + +Providers are trusted ambient code (fetch/fs available — consistent with routes.mjs). The host +validates *output* (schema, budget, fencing), not the code. + +### 5.4 The provider-bridge pi extension + +One Bobbit-generated extension per session (codegen modeled on +`generateToolGuardExtension`, `tool-guard-extension.ts:31`), wired in the same session-setup +step that writes the tool guard: + +- `before_agent_start` → `POST /api/sessions/:id/provider-hooks/before-prompt` with the + pending user input; the gateway runs `beforePrompt` across enabled providers (bounded), and + the extension appends the returned fenced text via pi's systemPrompt/append mechanism. The + gateway also refreshes the persisted prompt-sections snapshot so the inspector stays + truthful per turn. +- `turn_end` → fire-and-forget `POST …/after-turn`. +- `session_before_compact` → blocking-with-timeout `POST …/before-compact`. +- `session_shutdown` → `POST …/shutdown`. + +Auth and transport identical to the tool guard (`BOBBIT_GATEWAY_URL` + `BOBBIT_TOKEN`). + +### 5.5 Claude-hook adapter mapping + +The `hooks` entity registers command hooks onto the same Hub: + +| Claude hook | Hub point | Semantics | +|---|---|---| +| `SessionStart` | `sessionSetup` | stdout `additionalContext` → ContextBlock | +| `UserPromptSubmit` | `beforePrompt` | stdout-injected context (matches Claude semantics) | +| `Stop` | `afterTurn` | notification | +| `PreCompact` | `beforeCompact` | notification | +| `SessionEnd` | `sessionShutdown` | notification | +| `PreToolUse` / `PostToolUse` | tool-guard generalization (P6) | block/mutate via the existing long-poll endpoint | + +Spawned via `execFile`, JSON on stdin, timeout-bound, non-fatal. + +--- + +## 6. Memory architecture — two tiers, one UX + +The promise: **Bobbit gives the impression of infinite memory.** Two engines deliver it behind +one surface; neither is special-cased in core. + +- **Tier 1 — `session-memory` (built-in pack, default-on, zero dependencies).** Uses the + existing FlexSearch index and transcripts: `beforePrompt` queries BM25+recency over past + sessions/goals/messages in the project and contributes top-K bounded "Relevant past work" + blocks (each linking its source sessionId); `sessionSetup` recalls against the goal/task + spec. Every Bobbit user gets automatic recall with no Docker, no service, no setup — and + this pack is the litmus implementation that proves the provider API (P1). +- **Tier 2 — `hindsight` (pack, pre-installed dormant).** Semantic memory: consolidation, + reflect, mental models, cross-session synthesis — the upgrade. Detailed in §11. +- **One UX.** Both emit the same ContextBlocks under one budget; provenance disambiguates in + the inspector. Default priority: `hindsight > session-memory`; cross-engine dedupe is a P4 + refinement. The agent-facing story is simply: the agent remembers. + +**Scope-awareness (designed in from day one).** Provider contexts carry +`scope: "project" | "global"`. Memory bank routing: + +- Project sessions → bank `bobbit-proj-<projectId>` **and** a shared `bobbit-global` bank, + recalled in parallel under one budget. +- The `hindsight_recall` tool and the panel accept `bank: current | global | all` + (`all` = fan-out across banks). +- A future gateway-level "Mission Control" surface (global sessions/staff above projects — + see §14) slots into `scope: "global"` without rework. + +Retains are tagged `sessionId` / `goalId` / `roleName` so the panel can filter per +session/goal. + +--- + +## 7. Managed runtimes + +New core: `src/server/runtimes/pack-runtime-supervisor.ts`, mirroring the discipline of +`project-sandbox.ts` (DOCKER_BIN, execFile wrapper, MSYS env handling) and `sandbox-manager.ts` +(lazy idempotent `ensureRuntime(id)` with in-flight dedupe), plus the +supervise-an-HTTP-service flavor of `aigw-manager.ts`. + +- **Compose project** `bobbit-pack-<name>` — stable across pack updates, so named volumes (and + therefore data) survive updates; image digests pinned in the pack's compose file mean an + update recreates the app container against the same volume. +- **Env/secrets**: generated `.env` (mode 0600) under `~/.bobbit/state/pack-runtimes/<pack>/`; + `generate: true` secrets created with `crypto.randomBytes` on first start; values stored via + `SecretsStore`; secrets never sent to panels (status only). +- **Ports**: `host: auto` allocates an ephemeral host port, persists it, re-validates on boot; + the bound provider reads the resolved `baseUrl` from its ctx — never hardcodes. +- **Status machine**: `docker-unavailable | stopped | starting | running | unhealthy`, driven + by the declared HTTP healthcheck. +- **REST**: `GET /api/pack-runtimes`, `POST /api/pack-runtimes/:id/{start|stop|restart}`, + `GET /api/pack-runtimes/:id/logs?tail=`. +- **Lifecycle**: `startPolicy: on-enable` — and a pinned platform rule: **a pre-installed pack + never starts containers without a deliberate user action.** Disable ⇒ `compose stop` (data + kept). Uninstall ⇒ `compose down` with **volumes preserved by default**; an explicit + "Delete data" confirmation runs `down -v` and removes the state dir. Reinstall reattaches + existing volumes. +- **Fallback**: when Docker is unavailable, the runtime reports `docker-unavailable` and the + consuming provider's `external` mode (user-supplied URL/key) is the first-class path, not an + error state. + +--- + +## 8. Trust & consent + +Packs are **trusted code**. The platform inherits the Extension Host stance: worker confinement +is resource/crash isolation and import hygiene, not a security sandbox; pack tools and +pi-extensions run with ambient access in the agent process, as tools always have. What v2 adds +is **disclosure granularity and deliberate activation**: + +1. **Capability summary at install/enable**, computed from the manifest: "runs code in the + gateway" (providers/hooks), "runs code in the agent" (tools/pi-extensions — already true + today, now stated), "connects MCP servers" (command/URL listed per server), "runs Docker + services" (images, ports, volumes), "requests secrets" (names + prompts), "provides/requires + capabilities". Shown inline on the install/enable card; no separate consent ceremony. +2. **Per-entity activation** — `pack_activation` covers all six new entity types. Disabling a + provider unregisters its hooks synchronously (`invalidateResolverCaches()` + Hub + re-resolve); disabling a runtime stops its containers. +3. **Hooks and pi-extensions need a one-time per-pack acknowledgment** (arbitrary commands on + the gateway host / arbitrary code in the agent process), default-deny until acknowledged. + Claude-converted packs additionally show the exact hook command lines. +4. **Memory consent posture** (owner decision): enabling a memory pack is the consent — + auto-recall and auto-retain start at Enable, with the enable card disclosing inline what is + stored (per-project bank), which model forms memories, and a cost note. Switching to + `external` mode (content leaves the machine) shows a one-line notice. Per-project opt-out + rides `pack_activation`. + +--- + +## 9. Claude Code plugin compatibility + +A **format adapter at install time**, not a parallel install system. The atomic-staging step in +`marketplace-install.ts` is the seam: + +- `MarketplaceSourceStore` sources gain `format: "bobbit" | "claude-plugin"`, auto-detected at + sync (presence of `.claude-plugin/marketplace.json`); browse lists plugins as packs. +- Install conversion into the staging dir: + +| Claude plugin piece | Bobbit pack entity | +|---|---| +| `commands/*.md` | skills (the `commands-flat` layout already exists, `pack-list.ts`) | +| `skills/` | skills | +| `agents/*.md` | roles (frontmatter name/description/model → `Role{name, promptTemplate, model}`; tool restrictions best-effort → `toolPolicies`) | +| `hooks/hooks.json` | `hooks` entities (§5.5 mapping) | +| `.mcp.json` | `mcp` entities | +| `scripts/`, assets | copied verbatim; `${CLAUDE_PLUGIN_ROOT}` rewritten to the pack root | + +- A `pack.yaml` (`schema: 2`) is synthesized; `.pack-meta.yaml` records + `sourceFormat: claude-plugin`. +- **Unsupported features are reported, never silently dropped**: statusline, output styles, + permission/sandbox settings, MCP transports `McpManager` can't handle → structured + `skipped: [{feature, reason}]` in the install response, rendered in the UI. +- Full fidelity depends on the `mcp` (P5) and `hooks` (P6) phases; the adapter phase (P7) is + sequenced after them to avoid shipping a degraded story. + +--- + +## 10. Reference pack: `session-memory` + +``` +market-packs/session-memory/ + pack.yaml # schema 2; contents.providers: [recall] + providers/recall.yaml # kind: memory; hooks: [sessionSetup, beforePrompt]; budget 1200/800ms + lib/provider.mjs # queries GET /api/search (or the search service via a host route) + panels/ + entrypoints/ # optional: small "what was recalled" panel (can defer to inspector) +``` + +Behavior: `beforePrompt` → FlexSearch query from the user prompt (BM25 + recency, project +scope, archived included) → top-K results formatted as compact blocks with session links; +`sessionSetup` → recall keyed on goal/task spec. Deterministic, no LLM, no network beyond the +gateway. Ships in the built-in band, **default-on** (it only reads data Bobbit already has). +Config: K, budget, source types (sessions/goals/messages), include-archived. + +This pack is also the platform's litmus: its tests pin the provider loader, Hub dispatch, +budget clamp, timeout-skip, provenance, and activation toggles against a real pack. + +## 11. Reference pack: `hindsight` + +``` +market-packs/hindsight/ + pack.yaml # §4 example + providers/memory.yaml # hooks: sessionSetup, beforePrompt, afterTurn, beforeCompact, sessionShutdown + runtimes/hindsight.yaml # §4 example + runtime/compose.yaml # hindsight + pgvector postgres, digest-pinned + tools/hindsight/ + hindsight_recall.yaml # provider: { type: bobbit-extension, extension: extension.ts } + hindsight_reflect.yaml + hindsight_retain.yaml + extension.ts # pi.registerTool ×3 → POST pack routes (tool-guard auth pattern) + panels/hindsight-memory.yaml # native panel + entrypoints/hindsight-open.yaml, hindsight-route.yaml + lib/provider.mjs, routes.mjs, HindsightPanel.js, hindsight-client.mjs # one shared REST client + src/ # TS sources, built like the other market packs +``` + +- **Provider flow**: `sessionSetup` → recall vs goal/task spec → "Relevant memory" section; + `beforePrompt` → recall vs user prompt (project-scoped + org-wide tag filters in one + query, 1500ms timeout ⇒ skip); `afterTurn` → **async** retain of the turn, auto-tagged + `project:/agent:/goal:/kind:` from session context (failures queue in the pack store, + retried next turn); `beforeCompact` → synchronous retain of salient facts before context + loss; `sessionShutdown` → flush. +- **Tools**: explicit `hindsight_recall` / `hindsight_reflect` / `hindsight_retain` for the + model, with `scope: project | global | all` mapped to tag filters on the shared bank — + NOT bank switching; Hindsight has no cross-bank search, which is why the topology is one + shared tag-scoped bank (decision + verified facts: + [agent-memory.md §3](agent-memory.md)). (Hindsight's own MCP server remains usable via + normal MCP discovery, orthogonally.) +- **Panel** (native, same-origin): status + mode setup (managed vs external), memory search, + recent retains, retain-queue/operations view, runtime health/logs links, settings (model for + memory formation, budgets, toggles). +- **Modes**: `managed` (default when Docker present — Enable → supervisor up → healthy → + works) | `external` (URL + API key). **Great defaults**: cheap memory-formation model + preconfigured (reusing the user's existing configured key by default), generated Postgres + password, sane recall budget — power users can change all of it, but nobody has to. +- **Distribution**: built-in first-party band, **dormant** until enabled (no containers, no + hooks active). +- **Failure modes (all non-fatal)**: Docker absent → setup card offers external mode; service + unhealthy/timeout → recall skipped + diagnostic in trace + status badge; retain failure → + queued with counter in panel; provider crash → worker isolation, provider marked errored for + the session. + +--- + +## 12. Selectors & composition + +The "something clever": after each prompt — or before creating a goal / spawning a session — a +selector decides which personas, workflows, skills, MCP servers, and models fit best. In this +platform, **selectors are themselves provider packs** (`kind: selector`), and the things they +select over are the existing stores. + +- **Decision points**: `beforeGoalCreate`, `beforeSessionSpawn`, `beforePrompt` (selector + flavor). The Hub hands selectors typed summaries — available roles (`RoleManager`), workflow + templates + project workflows (`WorkflowManager`), skills catalog (`slash-skills`), MCP + servers (`McpManager`), models (`model-registry`) — within an input budget. +- **Typed proposals, never mutation**: + +```ts +interface SelectorProposal { + role?: { roleName: string; model?: string; thinkingLevel?: string; + personaPatch?: string /* bounded, additive */; confidence: number; reason: string }; + workflow?: { action: "reuse" | "instantiate-template" | "ask-project-assistant"; + workflowId?: string; templateId?: string; confidence: number; reason: string }; + skills?: { include: string[]; reason: string }; + mcp?: { servers: string[]; reason: string }; + model?: { model: string; thinkingLevel?: string; reason: string }; +} +``` + +- **Approval policy (host-enforced)** — auto-apply when: the decision is pre-session/pre-goal, + the selected role does not expand tool grants beyond the default, the model is + available/not-more-expensive than policy allows, persona/AGENTS.md changes are additive and + bounded, and confidence clears threshold. Otherwise ask the user (mid-session role/model + change, tool expansion, new/cloned workflows, expensive optional steps, low confidence). + Workflow *creation* always goes through the existing WorkflowManager APIs and user approval. +- **LLM-backed selectors**: strict-JSON schema output, hard timeout, deterministic fallback + (invalid output ignored). The selector may call models via its own config or via the + `model-selector` capability. +- **Capability composition (the model-selector example)**: a `model-selector` pack + `provides: [model-selector]` — input `{task, spec, candidates, constraints}` → output + `{model, thinkingLevel, reason}`. The persona selector, workflow templates, and verification + steps all `requires: [model-selector]` and call it via `ctx.capabilities.call`. +- **Flagship composition — the `multi-model-delivery` pack**: roles + (`planner` pinned to a frontier model at high thinking, `implementer` pinned to a cheap + precise model, `qa` and `reviewer` pinned to independent strong models — each gate carrying + the initial spec) + a workflow template wiring plan → implement → QA → review with a + ralph-loop (re-run until green) step + `requires: [model-selector]` so stage models can be + chosen dynamically per goal. Installing one pack gives a project a complete multi-model + delivery pipeline; the selector offers it when a goal's prompt fits. + +--- + +## 13. Phases + +Each phase is one mission goal: independently shippable, master-green +(`npm run check`, `test:unit`, `test:e2e` at/above baseline), test-first per AGENTS.md (every +user-facing feature gets unit + browser E2E; real Docker only in `tests/manual-integration/`). + +### P1 — Schema v2 + providers entity + Lifecycle Hub + `session-memory` pack +- **Outcome**: packs can ship lifecycle providers; Bobbit dispatches + `sessionSetup/beforePrompt/afterTurn/beforeCompact/sessionShutdown` with budgets, fencing, + and provenance; the built-in `session-memory` pack gives every user automatic recall of past + work (the infinite-memory impression) with zero dependencies. +- **Scope**: `pack-types.ts` (+`"providers"`), `pack-manifest.ts` (schema 2 keys), new + provider loader (pattern: `pack-contributions.ts`), `src/server/agent/lifecycle-hub.ts`, + provider dispatch on the worker tier (pattern: `route-dispatcher.ts`), session-setup + pipeline step + provider-bridge codegen (pattern: `tool-guard-extension.ts`), + `system-prompt.ts` (`PromptParts.dynamicContext`), `server.ts` + (`/api/sessions/:id/provider-hooks/*`, `/api/sessions/:id/context-trace`), Market UI + provider listing + activation toggle, `market-packs/session-memory/`, plus a deterministic + `provider-demo` fixture pack (loaded via `BOBBIT_BUILTIN_PACKS_DIR`) recording hook + invocations to its pack store. +- **Acceptance**: manifest validation accepts/round-trips schema-2 `contents.providers`; + budget clamp + timeout-skip + non-fatal failure pinned by unit tests; bridge codegen + snapshot test; API E2E — session with provider-demo shows its block in prompt-sections with + provenance, afterTurn recorded, disabled-via-activation ⇒ no block; session-memory E2E — a + second session in a project recalls content from the first (block visible in inspector and + in the outgoing prompt), toggle off ⇒ none; browser E2E — inspector renders the Dynamic + Context section; Market UI toggle persists across reload. +- **Non-goals**: runtimes/Docker; Hindsight; selectors; mid-turn `context` mutation; + hooks/mcp/pi-extensions/workflows entities. +- **Dependencies**: none. + +### P2 — `hindsight` pack v1 (external-URL mode) +- **Outcome**: the Hindsight pack works end-to-end against a user-supplied Hindsight URL: + scope-aware banks (project + global), auto-recall, async auto-retain with retry queue, + the three tools, routes, and the native panel. +- **Scope**: `market-packs/hindsight/` (everything in §11 except the runtime), + build-script entry, provider config surface. +- **Acceptance**: unit — bank-id derivation incl. global fan-out, block formatting/fencing, + retain-queue retry (REST client mocked); API E2E against an **in-process stub Hindsight + server** — sessionSetup + beforePrompt blocks injected, turn_end retains with correct bank + + tags, stub down ⇒ session unaffected + diagnostic, `hindsight_recall` tool round-trips with + `bank: all`; browser E2E — panel configures external URL, shows status, search returns stub + results; per-project disable ⇒ no injection; persists across reload. +- **Non-goals**: managed Docker; mental-models UI; cross-engine dedupe. +- **Dependencies**: P1. + +### P3 — Managed runtimes + Hindsight zero-step enable +- **Outcome**: `contents.runtimes` supported; the supervisor brings up Hindsight + Postgres + with generated secrets, allocated port, and healthcheck; **Enable → working memory with no + manual steps**; uninstall preserves data unless explicitly purged. +- **Scope**: runtimes entity (manifest/loader), `src/server/runtimes/pack-runtime-supervisor.ts` + + runtime-manifest parser, REST endpoints, `.env`/secrets handling, uninstall hook in + `marketplace-install.ts`, panel status/logs/start-stop wiring, `runtime/compose.yaml` in the + pack. +- **Acceptance**: unit — manifest validation, port allocation/persistence, secret-generation + idempotence, compose arg construction (execFile mocked), keep-vs-purge; API E2E — mocked + docker binary walks stopped→starting→running on health 200; docker-unavailable surfaces the + external-mode setup card path; browser E2E — runtime status card states + logs view + (mocked); **manual-integration** (real Docker) — enable → compose up → healthy → + recall/retain round-trip → disable stops; volume survives a pack update. Pinned platform + rule: pre-installed band never auto-starts containers. +- **Non-goals**: arbitrary third-party runtime marketplace policy; Kubernetes; remote Docker. +- **Dependencies**: P1, P2. + +### P4 — Memory depth & polish +- **Outcome**: compaction never loses memory-worthy content; users can browse/manage memories; + the two memory tiers compose cleanly. +- **Scope**: beforeCompact sync-retain ordering (bridge + pack), memory-browser panel v2 + (browse/filter by session/goal, delete, retain-queue + operations view, reflect surface), + hindsight↔session-memory priority/dedupe under the shared budget, cost surfacing for memory + formation. +- **Acceptance**: API E2E — forced compaction triggers retain *before* compaction completes + (stub asserts ordering); browser E2E — memory browser lists/filters/deletes and survives + reload; unit — dedupe/priority under budget. +- **Non-goals**: mental-model auto-refresh scheduling; multi-bank admin UI. +- **Dependencies**: P2 (P3 for managed-mode paths). + +### P5 — MCP as pack content +- **Outcome**: packs ship MCP server configs that flow into existing discovery, meta-tools, + proxy extensions, and policy — with pack provenance and per-entity activation. +- **Scope**: lift `pack-manifest.ts:95-97` rejection for schema 2; `mcp` loader; a pack-sourced + discovery band in `mcp-manager.ts` (below user configs — a pack must never shadow a user's + own MCP config); conflict reporting; Market UI provenance badge; capability summary entry. +- **Acceptance**: unit — manifest + loader + precedence-vs-user-config; API E2E — pack-shipped + MCP server produces meta-tools via the existing `writeMcpProxyExtensions` path and is + removed on uninstall; browser E2E — pack-provenance badge in the tools UI. +- **Non-goals**: packs shipping MCP server *binaries*; new secret mechanisms for MCP. +- **Dependencies**: P1 (manifest plumbing precedent). + +### P6 — Hooks + pi-extensions entities +- **Outcome**: packs ship Claude-style command hooks (mapped per §5.5, including + PreToolUse/PostToolUse via a generalized tool-guard long-poll) and raw pi extension modules + (added to the session `--extension` list) — both behind a one-time per-pack trust + acknowledgment. +- **Scope**: `hooks` + `pi-extensions` loaders; hook dispatcher (execFile, JSON protocol, + timeouts); tool-guard endpoint generalization; extension-list assembly in session setup; + trust-grant store + consent UI. +- **Acceptance**: unit — hook mapping table, default-deny gate; API E2E — fixture pack's + PreToolUse hook blocks a tool call; un-acknowledged pack contributes no hooks/extensions; + browser E2E — acknowledgment flow, revoke works after reload. +- **Non-goals**: sandboxing claims; hook-driven prompt rewrites beyond the fenced path. +- **Dependencies**: P1. + +### P7 — Claude Code plugin marketplace adapter +- **Outcome**: a Claude plugin marketplace adds as a source; its plugins browse and install as + packs with full mapping (skills/commands/agents/hooks/mcp) and an explicit skipped-features + report. +- **Scope**: format detection in source sync; `claude-plugin-adapter.ts` conversion at the + staging seam; `${CLAUDE_PLUGIN_ROOT}` rewrite; UI source badge + skipped report rendering. +- **Acceptance**: unit — conversion mapping table incl. skip cases; API E2E — fixture plugin + repo installs, skills resolve with correct precedence, hooks/mcp entities materialize; + browser E2E — install a real public Claude marketplace plugin from the Market UI, slash + command appears in the composer, skipped report shown. +- **Non-goals**: bidirectional export (Bobbit packs → Claude plugins). +- **Dependencies**: P5, P6 (full fidelity). + +### P8 — Capability registry + selector framework +- **Outcome**: packs compose via `provides`/`requires` + `ctx.capabilities.call`; selector + providers make validated, policy-gated proposals at `beforeGoalCreate` / + `beforeSessionSpawn` / `beforePrompt`. +- **Scope**: manifest `provides`/`requires` + install-time dependency check; capability + dispatch on the worker tier; Hub decision points wired into goal-creation and session-spawn + paths; proposal types + approval policy (§12); proposal UI (accept/decline affordance); + fixture selector pack. +- **Acceptance**: unit — registry resolution incl. precedence + missing-dependency warning; + proposal validation (over-grant ⇒ requiresApproval; invalid JSON ⇒ deterministic fallback); + API E2E — fixture selector proposes role+workflow at goal creation, auto-applies under safe + policy, asks otherwise; browser E2E — proposal surfaced at goal creation, accept/decline + round-trips. +- **Non-goals**: codegraph provider; mid-session automatic role changes. +- **Dependencies**: P1. + +### P9 — Workflow-template entity + flagship selector packs +- **Outcome**: packs ship workflow templates; first-party `model-selector` capability pack and + `multi-model-delivery` pack deliver the multi-model pipeline (frontier-model planning → + cheap precise implementation → independent QA → independent review, ralph-loop) as a single + install. +- **Scope**: `workflows` loader (templates → instantiation through existing + `WorkflowManager`/`WorkflowStore` APIs at project setup or goal creation; project.yaml stays + the source of truth; goal snapshotting unchanged); `market-packs/model-selector/`; + `market-packs/multi-model-delivery/` (roles + template + `requires: [model-selector]`). +- **Acceptance**: unit — template instantiation + validation reuse (`workflow-validator`); + API E2E — installing the pack offers the template, instantiation creates a valid project + workflow, goal snapshot carries stage roles with pinned models, selector swaps a stage model + via the capability; browser E2E — template offered in goal-creation UI; per-gate role/model + visible. +- **Non-goals**: workflow auto-creation by LLM without approval; cross-project workflow sync. +- **Dependencies**: P8. + +--- + +## 14. Later (recorded, not goals) + +- **Mission Control as an extension** — global sessions/staff above projects ("talk to the + gateway before/instead of a project"). Needs one core enabler: a gateway-level pseudo-scope + for sessions/staff plus cross-project service APIs (search-all-projects, spawn-goal-in- + project). The memory architecture is already global-ready (`scope: "global"`, the + `bobbit-global` bank, `bank: all` fan-out); the surface itself (panel + routes + tools) fits + the Extension Host. Sketch only; not a phase. +- Codegraph/LSP context provider; `POST /api/context/compile-preview`; mid-turn pi `context` + message mutation; dynamic `setActiveTools` selection; `before_provider_request` telemetry; + per-conflict capability pinning. The ContextBlock schema keeps + `reason/priority/authority/tokenEstimate` so these slot in without breaking changes. + +## 15. Risks & open questions + +1. **Docker availability/Windows**: Docker Desktop/WSL2; `docker compose` v2 vs legacy + binary detection; MSYS path handling (reuse `project-sandbox.ts` discipline). External mode + must remain a first-class path, not an error state. +2. **Hindsight OCI images**: confirm a pullable, digest-pinned image (amd64+arm64). If none is + published, the pack needs a build step — decide before P3. +3. **Memory-formation LLM key**: default = reuse the user's configured key with a cheap model + (zero-step), documented on the enable card with a cost note. A separate-key option exists + in settings. Trade-off: memory ops bill to the main key. +4. **beforePrompt latency**: up to `timeoutMs` (1500ms default) added per turn when memory is + enabled. Mitigations: timeout-skip (pinned), prompt-hash recall cache, prefetch-at-turn-end + for the next turn. Measure in P2 and record the budget. +5. **Ports & multi-server**: persisted auto-allocated ports must re-validate on boot; two + Bobbit servers on one machine sharing a compose project name need a server-identity suffix — + decide in P3. +6. **Postgres migrations on pack update**: minor updates ride volume persistence + app-run + migrations; a schema-breaking Hindsight update needs a documented path (backup-before- + update?). Open. +7. **Secrets at rest**: 0600 `.env` + `SecretsStore`; no OS keychain in v1. Threat model: + localhost-bound ports + bearer keys; document it. +8. **Bank lifecycle**: project deletion → prompt to delete the bank (default keep); projectId + reuse across servers sharing one Hindsight could collide — consider a server-salt in the + bank id. +9. **CI has no Docker**: the stub-Hindsight E2E is the real gate; managed-mode regressions + surface only in `tests/manual-integration/` — consider a scheduled weekly manual run. +10. **Schema-v2 on v1 servers**: silent degradation (new keys ignored). v2 servers warn on + newer-than-supported `schema`; packs requiring v2 should say so in `description`. +11. **Selector trust creep**: selectors must stay proposal-only; the approval policy is the + single enforcement point — pin it with tests (over-grant ⇒ approval required) before any + selector pack ships. +12. **Token estimation**: the ~4 chars/token heuristic (already used by `PromptSection.tokens`) + is approximate; fine for budgeting, recorded as a known approximation. diff --git a/docs/design/fable-program-execution-plan.md b/docs/design/fable-program-execution-plan.md new file mode 100644 index 000000000..b23a1b2f3 --- /dev/null +++ b/docs/design/fable-program-execution-plan.md @@ -0,0 +1,315 @@ +# Fable program — execution plan (hand-off for the main Bobbit session) + +Status: living tracking document. This is the **single sequencing source of truth** for all +work proposed on the fable-docs branch. Hand this file to the orchestrating Bobbit session; +it creates goals from §3, in lane order, and ticks §4 as PRs merge. + +The workstreams (each doc owns its *content* — tasks, contracts, acceptance; this doc owns +*ordering, slicing, and the checklist*): + +| Workstream | Design (WHAT/WHY) | Execution authority (HOW) | +|---|---|---| +| **EP** extension platform | [extension-platform.md](extension-platform.md) | [extension-platform-implementation-plan.md](extension-platform-implementation-plan.md) (G1.1–G9) | +| **CE** time & token-cost efficiency | [time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) §1–§5, **§9** (latency + arch root cause); [verification-loop-economics.md](verification-loop-economics.md) (verification-loop tax + right-sizing, trust-first) | same doc, §6 (CE-G0.1–G7.2) + **§9.5 (CE-G8 latency)** + **CE-G8.7–G8.11** (verification-loop-economics §4), BENCH gates | +| **CS** comms-stack reliability | [comms-stack/01–03](comms-stack/01-understanding.md) | [comms-stack/04-current-state-and-backlog.md](comms-stack/04-current-state-and-backlog.md) §6–§7 (waves + merge map) | +| **SN** sidebar & goal-nesting | [sidebar-goal-nesting-audit.md](sidebar-goal-nesting-audit.md) §1–§3 | same doc, §4 (T1–T9) | +| **PB** client perf/battery | [client-performance-battery.md](client-performance-battery.md) | [client-performance-battery-implementation-plan.md](client-performance-battery-implementation-plan.md) | +| **MC** Mission Control | [mission-control.md](mission-control.md) | [mission-control-implementation-plan.md](mission-control-implementation-plan.md) | +| **AI** autoimprovement | [autoimprovement.md](autoimprovement.md) | [autoimprovement-implementation-plan.md](autoimprovement-implementation-plan.md) | +| **GA** gap-analysis easy wins | [harness-gap-analysis.md](harness-gap-analysis.md) | [gap-easy-wins-implementation-plan.md](gap-easy-wins-implementation-plan.md) | +| **CI** code intelligence | [code-intelligence.md](code-intelligence.md) | [code-intelligence-implementation-plan.md](code-intelligence-implementation-plan.md) | +| **SW** agent swarms & reconciliation | [agent-swarm-and-reconciliation.md](agent-swarm-and-reconciliation.md) (exploratory) | same doc, §7 (SW-G0–G4; gated on single-container sandbox + CE-G8.4) | + +**An implementer agent receives exactly one goal ID and reads: the execution-authority doc's +goal section → the contracts it cites → §1 below. Nothing else is required.** + +--- + +## §1 Binding rules (no exceptions, no re-litigating) + +1. **Universal definition-of-done**: + [extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) + applies verbatim to every PR in every workstream — read-before-edit (`rg` the symbol in + docs/tests/src first), tests authored first and shown RED where expressible, `npm run + check` + `test:unit` + relevant `test:e2e` green, browser E2E for every user-facing + feature, no flaky tests, minimal change, master stays green, anchors located by **symbol + name** (line numbers are hints; if a named symbol is missing, STOP and re-derive from the + cited pattern file — never improvise a parallel mechanism). Its §0.1 patterns library is + the copy-from list for all workstreams. +2. **The docs are the spec.** Implement exactly what the owning doc's task/phase says, using + its appendix contracts (types, catalogs, file layouts) as written. If reality forces a + deviation, amend the owning doc **in the same PR** with a `> Deviation:` note at the + affected section — doc and code must never disagree after a merge. Architectural + deviation ⇒ stop and escalate to a human. +3. **One task/phase = one goal; one goal = 1–3 PRs.** Never batch. Every PR leaves the + product working and mergeable on its own (flag-gated or additive when incomplete). +4. **Shared-seam serialization.** These seams are touched by multiple workstreams. The first + goal to land owns the change; later goals rebase onto it — never parallel-edit a seam: + + | Seam | Touched by | Order | + |---|---|---| + | `api.ts` `refreshSessions`/`startSessionPolling` | SN-T2/T3 · PB-P2a (FX5) | **SN-T2 → SN-T3 → PB-P2a** | + | sidebar render paths (`sidebar.ts`, `render-helpers.ts`, nesting) | SN-T5/T6/T7 · PB-P2c (FX7) | SN first; PB-P2c after or rebased | + | `session-manager.ts` / `session-setup.ts` | CS-R\*/D\*/P3 · CE-G3 · EP G1.3/G1.4 | CS merge-map order governs; CE/EP confine edits to named functions and rebase | + | `goal-trigger-dispatcher.ts` push triggers (`gate_failed`/`session_errored`) | GA-R2 · MC-P4 · AI-P1 | land once in GA-R2 | + | `activity-store.ts` / `recordActivity()` | MC-P3 · MC-P2 · AI-P5 | land in MC-P3; earlier callers stub | + | `auxiliary.*` model-slot config | AI-P2 · GA-R4 · CE-G6.1 | one shape ([per-role-model-overrides.md](per-role-model-overrides.md)); first lander defines it | + | `verification-harness.ts` | CE-G3.3 · CE-G5.2 (CI-2's gate-consumption is a recorded follow-up, NOT in CI-2) | sequence per CE doc | + | tool-activation in `session-setup.ts` | CI-1/2/3 · EP G1.4 · CS | confine to named functions; rebase order per wave | + | EP G8 capability registry | CI-7 consumes | CI ships a local shim until G8 merges; swap PR after | +5. **Tracking discipline.** Tick the §4 row **in the merging PR**; update the owning doc's + `Status:` line when a workstream's parent goal completes. The orchestrating session + treats merged-but-unticked as a bug. + +## §2 Dependency graph + +```mermaid +graph TD + subgraph "independent lanes — start day one" + CS[CS waves R1→…→T2<br/>per its §7 merge map] + SN1[SN-T1/T2/T4/T8/T9] --> SN2[SN-T3, T5] --> SN3[SN-T6, T7] + CE[CE-G0 → CE lanes per its §6] + EP[EP G1→G9 per its goal map] + PB0[PB-P0 harness] --> PB1[PB-P1a/b animations] + GA2[GA-R2 triggers]; GA3[GA-R3 standing orders] + end + SN2 --> PB2[PB-P2a-d timers/renders] --> PB3[PB-P3 battery saver] + PB1 --> PB2 + MC0[MC-P0 global scope] --> MC1[MC-P1 sessions+sidebar] --> MC2[MC-P2a/b meta-tool] + MC1 --> MC3[MC-P3 flight recorder] + MC2 --> MC4[MC-P4a/b pack+crew]; MC3 --> MC4; GA2 --> MC4 + MC4 --> MC5[MC-P5 briefing/onboarding/budgets] + MC0 --> AI1[AI-P1 substrate]; GA2 --> AI1 + AI1 --> AI2[AI-P2a/b Improver] --> AI3[AI-P3a/b curator+dreaming] + MC4 --> AI2; MC3 --> AI5 + AI2 --> AI4[AI-P4 shadow] --> AI5[AI-P5 autonomy] --> AI6[AI-P6 measurement] + CI12[CI-1 ast tools, CI-2 diagnostics] --> CI3[CI-3 LSP supervisor] --> CI456[CI-4 lang packs, CI-5 repo map, CI-6 services chip] + CI456 --> CI7[CI-7 capability swap + BENCH] + EP -.G8 capabilities.- CI7 + GA6[GA-R6 importers] --> EP + GA5[GA-R5 user profile] -.coordinate.- EP +``` + +**The same graph in plain English.** Read "·" as "can run at the same time". + +- **Six lanes need nothing from anyone and can all start on day one, in parallel:** + the comms-stack reliability backlog (starting with making the gateway's message-queue + drain un-crashable, then respawn fencing, steer delivery, and the rest of its own + ordered waves); the sidebar-nesting cleanup (pin the pure tree builders with tests + first — that's the safety net for everything after); the cost-efficiency lane (per-turn + usage ledger first, because every later claim is measured against it); the extension + platform (manifest schema 2 → lifecycle hub → prompt-section providers, in order); + the client-performance lane (measurement harness first, then animation gating); and + the two small gap wins (one-shot/`gate_failed` triggers, standing-orders docs). +- **Sidebar before performance-on-the-sidebar.** The performance lane's session-poll + demotion and render work wait for the sidebar's concurrency hardening + (refresh-dedup, archived freshness), and the streaming render throttle explicitly + rebases on the big nesting render unification. Battery saver comes last, after all of + it soaks behind a flag. +- **Mission Control is a strict chain:** legitimize the global scope → Mission Control + sessions + sidebar entry → then two parallel branches (the `bobbit` meta-tool; the + flight recorder) → which rejoin for the mission-control pack + system-staff crew (this + also wants the triggers win, so staff can be woken by events) → briefing, onboarding, + and budgets close it out. +- **Autoimprovement rides on Mission Control:** the proposal substrate needs the global + scope and triggers; the Improver needs the meta-tool + crew; then curator/dreaming, + shadow calibration, graduated autonomy (which also wants the flight recorder for its + evidence trail), and the measurement loop — strictly in that order. +- **Code intelligence is its own lane:** ast-grep tools and structured diagnostics first + (independent of everything); then the LSP supervisor; then language packs, the repo + map, and the services chip in parallel; the final capability-swap goal waits on the + extension platform's capability registry (a local shim covers the gap). +- **Two soft couplings:** the Hermes/OpenClaw skill importers feed the extension + platform's pack format, and the user-profile memory win should coordinate with (not + block on) the platform's session-memory pack. + +**Day-1 starter set** (parallel, no collisions): AGENTS.md trim (§5) · CS-R1 · SN-T1 · +PB-P0 · CE-G0.1 · EP G1.1 · GA-R2 · GA-R3. + +## §2.1 THE ORDER — waves (the operational answer to "what do we work on now?") + +The §2 graph is the truth about *dependencies*; this section is the truth about *sequence*. +A wave starts only when its blocking entries from the previous wave are merged (non-blocking +stragglers from an earlier wave may continue in parallel). Within a wave, everything is +parallel-safe **provided the §1.4 seam table is respected**. + +```mermaid +flowchart TD + W0["WAVE 0 — prereq (hours) + AGENTS.md trim → suite green"] + W1["WAVE 1 — foundations, all parallel + CS-R1 (outage fix) · SN-T1 (pin tree builders) + PB-P0 (perf baseline) · CE-G0.1 (cost ledger) + EP G1.1 (manifest v2) · GA-R2 (triggers) · GA-R3 (standing orders)"] + W2["WAVE 2 — hygiene + scope + CS: H1‖H2 → R2 → R5‖R6 (its §7 order) + SN: T2 → T3, T4 · PB: P1a → P1b + CE: G0.2, G0.3, G1.0 · EP: G1.2 → G1.3 + MC: P0 → P1 · GA: R9, R6 (anytime)"] + W3["WAVE 3 — spines + CS: R3 → R7 → R4 → R8‖R9 · SN: T5 (the big one), T8, T9 + PB: P2a (after SN-T2/T3) → P2b → P2d + CE: post-G1.0 lanes (G2, G7; BENCH-gated G4/G5 wait for G0.3) + EP: G1.4 → G1.5 → G1.6 · MC: P2a → P2b, P3 · AI: P1 + CI: CI-1 (ast tools) · CI-2 (diagnostics)"] + W4["WAVE 4 — features on the spines + SN: T6, T7 · PB: P2c (after SN-T5) → P3 + CS: waves 2–4 (D*, P*, T*) · EP: G2 → G3 + MC: P4a → P4b · AI: P2a → P2b · GA: R4, R5 (R5 prefers EP G1.6) + CI: CI-3 (LSP supervisor) → CI-5 (repo map)"] + W5["WAVE 5 — the loop closes + AI: P3a → P3b → P4 → P5 → P6 · MC: P5 + EP: G4…G9 · CE: remaining BENCH-gated goals + CI: CI-4 (language packs) · CI-6 (services chip + graphify viz) · CI-7 (capability swap + BENCH)"] + W0 --> W1 --> W2 --> W3 --> W4 --> W5 +``` + +**The same waves in plain English** — what each wave actually does, and what runs in +parallel inside it. Everything listed within one wave can be in flight simultaneously +(different people/sessions) as long as no two touch the same §1.4 seam. + +- **Wave 0 — hours, not days.** Trim AGENTS.md and get the test suite green. Nothing else + starts until the suite is trustworthy, because every later goal proves itself with tests. +- **Wave 1 — foundations, seven things at once.** Fix the gateway crash (comms); pin the + sidebar's pure tree builders with real-source tests; stand up the performance + measurement harness; stand up the per-turn cost ledger; land manifest schema 2 for + packs; ship the one-shot/`gate_failed` triggers; write the standing-orders convention. + None of these share files; all seven can merge in any order. +- **Wave 2 — hygiene and scope.** Comms does its harness-fidelity work and then respawn + fencing + retry/compaction gating; sidebar hardens `refreshSessions` concurrency, then + archived-freshness and the goal-graph module; performance gates ambient animations and + rewrites paint properties; cost-efficiency adds the cost lens UI, the benchmark harness + (the BENCH gate everything risky waits for), and the SDK upgrade; the extension + platform builds the lifecycle hub and the sessionSetup → "Dynamic Context" section; + Mission Control legitimizes the global scope and gets its sidebar entry; `bobbit + doctor` and the skill importers can land any time here. +- **Wave 3 — the spines (the heaviest wave).** Comms lands steer exactly-once delivery, + idle-drain, and ledger durability; sidebar does the big one — unifying nesting into a + single render path — plus the role-picker and cleanup bundles; performance demotes + session polling and scopes the verification tick (now safe, because the sidebar + concurrency work merged in wave 2); cost-efficiency starts its post-upgrade lanes + (tool-output budgets, discovery guidance); the extension platform wires per-turn hooks, + market UI, and the session-memory pack; Mission Control ships the `bobbit` meta-tool + and the flight recorder; autoimprovement lays its proposal substrate; **code + intelligence starts here** with the ast-grep tools and structured diagnostics — both + independent, both immediately useful to every coding agent. +- **Wave 4 — features on the spines.** Sidebar finishes mobile reuse and keyed lists; + performance lands the streaming render throttle (rebased on the wave-3 render + unification) and then battery saver; comms clears its client-side duplicate/loss + backlog; the extension platform ships the Hindsight memory pack, then managed runtimes; + Mission Control assembles the pack + system-staff crew; autoimprovement turns on the + human-in-the-loop Improver; the "while you were away" summary and user-profile memory + land; code intelligence builds the LSP supervisor (TypeScript + Python first), then + the ranked repo map on top of the wave-3 ast tooling. +- **Wave 5 — the loop closes.** Autoimprovement runs curator → dreaming → shadow + calibration → graduated autonomy → the measurement loop, strictly in that order; + Mission Control finishes briefing/onboarding/budgets; the extension platform completes + its ecosystem (memory depth, MCP-as-pack, hooks, the Claude-plugin adapter, + capabilities + selectors, workflow templates); cost-efficiency executes its remaining + BENCH-gated diets (the team-lead role rewrite chief among them); code intelligence + fans out language packs (Go, Rust, C#, F#, JVM, clangd), ships the services chip + + graphify visualization pack, and swaps its local capability shim for the platform + registry, validating the whole workstream on BENCH. + +Milestones the waves deliver (what the user can feel): + +| After | The product visibly gains | +|---|---| +| Wave 1 | green suite, cost + perf both measurable, staff can fire one-shot/`gate_failed` triggers | +| Wave 2 | battery: idle tab stops animating; sidebar races hardened; Mission Control entry exists | +| Wave 3 | `bobbit` meta-tool live (chat can run everything); flight recorder visible; one nesting render path | +| Wave 4 | system-staff crew + dashboard; human-in-the-loop self-improvement proposing skills | +| Wave 5 | dreaming, calibrated autonomy with revert/demotion, briefing + onboarding; EP ecosystem complete | + +**Rule of thumb for the orchestrator:** at any moment, prefer (1) the lowest-numbered +unfinished wave, (2) within it, the entry that unblocks the most §2 edges, (3) never two +in-flight goals on the same §1.4 seam. + +## §3 Lanes and PR slicing + +Effort: S ≤ 1 day · M ≤ 3 days · L ≈ a week. EP, CE, and CS are **not re-sliced here** — +their own docs already slice to mergeable units; follow them verbatim (EP goal map order; +CE lanes with BENCH gates; CS §7 merge-map order). The lanes below slice the remaining +workstreams: + +### Lane SN — sidebar/nesting ([sidebar-goal-nesting-audit.md](sidebar-goal-nesting-audit.md) §4; one PR per task) + +T1 (M, no deps) → T2 (M) → T3 (S) → T4 (M) → T5 (L, after T1+T4) → T6 (M) → T7 (S) → +T8 (S, independent) → T9 (S, independent). T2/T3/T5/T7 gate PB-P2 (§1.4). + +### Lane PB — perf/battery (execute from [client-performance-battery-implementation-plan.md](client-performance-battery-implementation-plan.md)) + +| PR | Scope | Effort | Mergeable because | +|---|---|---|---| +| PB-P0 | `perf-monitor.ts` + baseline table | S | flag-gated, zero behavior change | +| PB-P1a | `animation-power.ts` + CSS gates (A.1 table) + pinning test | M | default-on flag, kill switch | +| PB-P1b | FX3 box-shadow→opacity rewrites + blanket reduced-motion | S | pure CSS, visually identical | +| PB-P2a | FX5 poll demotion *(after SN-T2/T3)* | S | behavior identical while WS down | +| PB-P2b | FX6 scoped verification tick | S | dashboard-local | +| PB-P2c | FX7 `renderAppThrottled` + streaming sites *(after/rebased on SN-T5)* | M | flag-gated | +| PB-P2d | FX8 timer audit + convicted-loop fixes | M | per-loop independent | +| PB-P3 | battery-saver mode + flag-soak removal + after-table | M | additive setting | + +### Lane MC→AI — Mission Control then autoimprovement (execute from [mission-control-implementation-plan.md](mission-control-implementation-plan.md), [autoimprovement-implementation-plan.md](autoimprovement-implementation-plan.md), [gap-easy-wins-implementation-plan.md](gap-easy-wins-implementation-plan.md)) + +| PR | Scope | Effort | Mergeable because | +|---|---|---|---| +| GA-R2 | `at` one-shot + `gate_failed`/`session_errored` triggers (schema in gap doc §5) | M | additive trigger types | +| GA-R3 | standing-orders template + staff-assistant guidance | S | docs+prompt only | +| MC-P0 | global scope + `PersistedStaff.global` (A.1–A.2) | M | invisible until MC-P1 | +| MC-P1 | global sessions + sidebar top entry + E2E | M | complete visible feature | +| MC-P2a | meta-tool registry + catalog pinning test (A.3) | M | server-only | +| MC-P2b | `defaults/tools/bobbit/` + tiers + policy wiring + e2e | M | ships complete with guards | +| MC-P3 | activity store + REST/WS + panel (A.4) | M | immediate audit value | +| MC-P4a | mission-control pack: roles/skills/templates + panel (A.5) | M | installs; crew inert | +| MC-P4b | "create crew" bootstrap + crew E2E + Caretaker dry-run | M | completes crew | +| AI-P1 | improvement store + `propose_improvement` + panel kind (A.1–A.3) | L | manual proposals useful alone | +| AI-P2a | judge + learned-skills pack bootstrap + usage hook | M | inert until Improver | +| AI-P2b | Improver role + post-goal review + e2e | M | human-in-loop learning complete | +| AI-P3a | curator (lifecycle/snapshots/pin/dry-run) | M | maintenance standalone | +| AI-P3b | dreaming job + Archivist wiring | M | config-gated | +| AI-P4 | shadow mode + calibration report | M | zero autonomy granted | +| AI-P5 | levels/thresholds + policy path + revert + demotion + kill switch | L | ships OFF (levels 0) | +| AI-P6 | outcome evaluator + regression demotion | M | completes loop | +| MC-P5 | briefing + onboarding + budgets (3 sub-PRs allowed) | L | each sub-PR standalone | +| GA-R4 | away-summary slice | M | independent; shared `auxiliary.*` | +| GA-R5 | bounded user-profile memory | M | coordinate w/ EP `session-memory` | +| GA-R6 | hermes/openclaw skill import adapters | M | marketplace seam | +| GA-R9 | `bobbit doctor` | M | standalone CLI | + +## §4 Master checklist (tick in the merging PR) + +- **Prereq**: [ ] AGENTS.md trim (§5) +- **CS** (its §7 order): [ ] R1 · [ ] H1 · [ ] H2 · [ ] R2 · [ ] R5 · [ ] R6 · [ ] R3 · [ ] R7 · [ ] R4 · [ ] R8 · [ ] R9 · [ ] D1 · [ ] D2 · [ ] D3 · [ ] D4 · [ ] D5 · [ ] D6 · [ ] D7 · [ ] D8 · [ ] D9 · [ ] P1 · [ ] P2 · [ ] P3 · [ ] P4 · [ ] T1 · [ ] T2 +- **SN**: [ ] T1 · [ ] T2 · [ ] T3 · [ ] T4 · [ ] T5 · [ ] T6 · [ ] T7 · [ ] T8 · [ ] T9 +- **PB**: [ ] P0 · [ ] P1a · [ ] P1b · [ ] P2a · [ ] P2b · [ ] P2c · [ ] P2d · [ ] P3 +- **CE**: [ ] G0.1 · [ ] G0.2 · [ ] G0.3 · [ ] G1.0 · [ ] G1.1 · [ ] G1.2 · [ ] G2.1 · [ ] G2.2 · [ ] G3.1 · [ ] G3.2 · [ ] G3.3 · [ ] G4.1 · [ ] G4.2 · [ ] G4.3 · [ ] G5.1 · [ ] G5.2 · [ ] G6.1 · [ ] G7.1 +- **EP**: [ ] G1.1 · [ ] G1.2 · [ ] G1.3 · [ ] G1.4 · [ ] G1.5 · [ ] G1.6 · [ ] G2.1 · [ ] G2.2 · [ ] G2.3 · [ ] G3.1 · [ ] G3.2 · [ ] G3.3 · [ ] G4 · [ ] G5 · [ ] G6 · [ ] G7 · [ ] G8 · [ ] G9 +- **GA**: [ ] R2 · [ ] R3 · [ ] R4 · [ ] R5 · [ ] R6 · [ ] R9 +- **MC**: [ ] P0 · [ ] P1 · [ ] P2a · [ ] P2b · [ ] P3 · [ ] P4a · [ ] P4b · [ ] P5 +- **AI**: [ ] P1 · [ ] P2a · [ ] P2b · [ ] P3a · [ ] P3b · [ ] P4 · [ ] P5 · [ ] P6 +- **CI**: [ ] CI-1 · [ ] CI-2 · [ ] CI-3 · [ ] CI-4 · [ ] CI-5 · [ ] CI-6 · [ ] CI-7 · + language-pack cards: [ ] lsp-go · [ ] lsp-rust · [ ] lsp-csharp · [ ] lsp-fsharp · [ ] lsp-jvm · [ ] lsp-clangd · [ ] lsp-kotlin +- **Cards** (post-dependency, owner-accepted): [ ] Caretaker transcript-retention consolidation pass (distill→prune; after EP G2 + MC P3; [agent-memory.md §1](agent-memory.md)) + +## §5 Prerequisite fix (before any lane) + +`tests/agents-md-budget.test.ts` fails on the branch base: AGENTS.md is 6,233 bytes vs the +6,144 budget. One S-size PR: move detail into `docs/`, get the suite green. Every lane +starts from green. + +## §6 Doc-format contract (parity rule for any future design doc in this program) + +Every workstream doc must carry, and these eight do: + +1. A `Status:` line (state + baseline commit where relevant) and a **workstream pointer to + this file**. +2. A hand-off backlog where each task names: owned files (NEW vs modified), diagnosis or + contract reference, the fix/steps, tests to author first, acceptance criteria. +3. Exact contracts for anything an implementer would otherwise invent (types, catalogs, + schemas, directory layouts) — in the doc body or an `Appendix A`. +4. Anchors by **symbol name** with a verified-at baseline, and the §1.1 stop-rule when an + anchor is missing. +5. Seam-overlap warnings pointing at §1.4 of this file. + +A new design doc that lacks any of these is not ready to enter §3/§4. diff --git a/docs/design/gap-easy-wins-implementation-plan.md b/docs/design/gap-easy-wins-implementation-plan.md new file mode 100644 index 000000000..1738ccc2d --- /dev/null +++ b/docs/design/gap-easy-wins-implementation-plan.md @@ -0,0 +1,179 @@ +# Gap-Analysis Easy Wins — Implementation Plan (hand-off) + +Status: ready for execution, not started. Workstream **GA** in +[fable-program-execution-plan.md](fable-program-execution-plan.md). + +Companion to [harness-gap-analysis.md](harness-gap-analysis.md) (the WHAT/WHY — read the +owning G-section for each goal; §5 holds the R2 trigger schema, which is LAW). + +> **Anchor baseline:** fable-docs @ 2026-06-11 (master parent `6ec8c8f9`). Locate by symbol +> name; missing symbol ⇒ STOP, re-derive from the cited pattern. +> +> **Universal rules:** [extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) +> + [fable-program-execution-plan.md §1](fable-program-execution-plan.md). GA-R2 is a +> **shared seam** (MC-P4 and AI-P1 consume its triggers) — it lands here, exactly once. + +Goals are independent unless stated; R2 and R3 are Day-1 starters. + +--- + +## GA-R2 — One-shot `at` trigger + `gate_failed`/`session_errored` push triggers + +**Outcome:** any staff can fire once at an ISO time, and react to gate failures / session +errors, per the schema in harness-gap-analysis.md §5 (verbatim — field names included). + +**Owned files:** `src/server/agent/staff-store.ts` (`StaffTrigger` config union), +`src/server/agent/staff-trigger-engine.ts` (`TriggerEngine`, schedule eval near `:220`), +`src/server/agent/goal-trigger-dispatcher.ts` (`GoalTriggerDispatcher`, symbol at `:30`), +the gate-failure + session-error mutation points (located in step 3), staff edit page +trigger editor (`src/app/staff-page.ts`); NEW `tests/staff-trigger-at.test.ts`, +`tests/staff-trigger-push.test.ts`; extend the staff-triggers e2e suite; +`docs/staff-triggers.md` (table rows). + +**Steps** + +1. **`at` one-shot.** Extend the `schedule` trigger config union with `{ at: string }` + (ISO-8601). In `TriggerEngine`'s schedule evaluation (the branch that currently + requires `trigger.config.cron` — `if (!trigger.config.cron) return false;` near + `:220`): when `at` is present instead, fire iff `now >= at && !lastFired`, then set + `enabled: false` on the trigger record (kept for audit, per the §5 contract). Invalid + `at` (unparseable / cron+at both set) ⇒ reject at PUT-validation time, not silently at + eval time. +2. **Trigger editor UI:** the staff edit page's schedule row gains a cron-vs-datetime + choice (native `datetime-local`, serialized to ISO UTC). Disabled-after-fire state + renders with a "fired <ts>" hint. +3. **Push triggers.** Read how `GoalTriggerDispatcher` wires `onGoalCreated`/ + `onGoalArchived` from `GoalStore.put`/`archive` (`docs/staff-triggers.md` §"Why push, + not poll"), then mirror it exactly for: (a) `gate_failed` — from the gate + verification-result mutation point (find the single place a gate step transitions to + failed in the verification harness; the human-signoff doc §1 names the step types); + (b) `session_errored` — from the session-status write where status becomes `errored` + (**`session-manager.ts` is a §1.4 shared seam** — confine to the status-transition + function, coordinate with CS merge order). Both types require a `prompt` + (the `goal_created` required-prompt rule — same validation path). +4. Docs: add both rows + the `at` variant to the `docs/staff-triggers.md` type table. + +**Tests (author first; RED):** `at` fires once with a fake clock, then `enabled === false` +and never re-fires; `at` in the past at creation fires on next tick exactly once; +cron+at rejected; push: gate-fail fixture enqueues one inbox entry with the trigger's +prompt; re-failing the same gate re-fires (each transition is an event); prompt-less push +trigger rejected at PUT. E2E: editor round-trip (create `at` trigger → reload → fired +state visible). + +**Acceptance:** new + existing trigger/inbox suites green; `docs/staff-triggers.md` +updated in the same PR. + +--- + +## GA-R3 — Standing-orders convention (docs + assistant guidance) + +**Outcome:** a documented Scope/Trigger/Approval-gates/Escalation template, and the staff +creation assistant proposes prompts in that shape. + +**Owned files:** `docs/staff-agents.md` (new §"Standing orders"); the staff assistant's +guidance text in `src/server/agent/staff-assistant.ts`; NEW assertion in the staff +assistant's existing test coverage. + +**Steps:** (1) add the template + one filled example (the gap doc §G4 anatomy, adapted to +a Bobbit staff prompt) to `docs/staff-agents.md`; (2) extend the assistant's +system-prompt guidance: proposed `systemPrompt`s for recurring/triggered staff SHOULD +follow the four-section template (do not hard-fail free-form prompts); (3) the +mission-control pack roster (MC-P4a) consumes this format — note it there, build nothing +for it here. + +**Tests:** assistant prompt-text pin (the guidance mentions the four section names); +docs link-check. **Acceptance:** S-size PR, no behavior change beyond assistant guidance. + +--- + +## GA-R4 — "While you were away" summary slice + +**Outcome:** opening a session with a large unread backlog shows a collapsible +cheap-model summary banner above the transcript tail. Off by default (setting). + +**Owned files:** server summarizer endpoint (reuse the compaction-summary generation path +— locate via `docs/compaction.md` / `compaction-history.md` before writing anything new); +`auxiliary.away-summary` model slot (**shared `auxiliary.*` seam — §1.4: reuse the shape +the first lander defined; if none landed yet, this PR defines it per +[per-role-model-overrides.md](per-role-model-overrides.md)**); client banner component in +the chat header area; settings toggle; NEW `tests/e2e/ui/away-summary.spec.ts`. + +**Steps:** (1) `GET /api/sessions/:id/away-summary?since=<lastSeen>` — summarizes messages +since `since` with the aux model, returns `{summary, messageCount}`; hard cap input via +the existing truncation helper (`truncate-large-content.ts`); (2) client: when the +setting is on and unread count ≥ threshold (default 10), fetch + render collapsible +banner; dismissal stores per-session; (3) never block transcript render on the fetch. + +**Tests:** API e2e with mock model (deterministic stub summary); browser E2E — banner +renders over threshold, absent under, dismissal survives reload, setting-off ⇒ no fetch +(network spy). **Acceptance:** zero added latency to session open (banner is async); +suites green. + +--- + +## GA-R5 — Bounded global user-profile memory *(coordinate with EP `session-memory`)* + +**Outcome:** a Hermes-style bounded user profile (~500 tokens) injected as a stable prompt +section into every session, agent-managed via one tool. + +**Owned files:** NEW `src/server/agent/user-profile-store.ts` +(`.bobbit/state/user-profile.md`, char-capped 1,375 like Hermes USER.md); prompt section +via `system-prompt.ts` `PromptParts` (the byte-budget pattern — +`buildSkillsCatalogSection`/`skillsCatalogBudget`, §0.1 patterns library); NEW +`defaults/tools/memory/user_profile.yaml` (+ group extension) with +`add/replace/remove` actions (substring matching, Hermes semantics per gap doc §G1); +NEW tests (store cap/round-trip; prompt-section budget pin; tool e2e). + +**Steps:** (1) store with frozen-snapshot semantics — section text captured at session +start, mid-session writes persist but don't mutate the live prompt (cache discipline, +gap doc §G1); (2) section renders with usage % header exactly like the Hermes format so +the agent knows capacity; (3) **coordination rule:** if EP G1.6 (`session-memory` pack) +has landed, this section must register through the provider path instead of a hardcoded +`PromptParts` entry — check the EP checklist first; the store + tool are identical either +way. **Acceptance:** profile survives restart; over-cap add returns a "consolidate first" +error to the tool, never truncates silently; prompt-sections inspector shows the block +with provenance. + +--- + +## GA-R6 — Hermes/OpenClaw skill import adapters + +**Outcome:** `bobbit` can import a Hermes skill dir (`~/.hermes/skills/...`) or an +OpenClaw workspace skill as a pack, via the marketplace's existing staging seam. + +**Owned files:** the staging-adapter seam in +`src/server/agent/marketplace-install.ts` (the same insertion point the Claude-plugin +adapter uses per [pack-based-marketplace.md](pack-based-marketplace.md) — read that first); +NEW `src/server/agent/skill-import-adapters.ts` (two pure functions: +`adaptHermesSkill(dir) → staged pack tree`, `adaptOpenclawSkill(dir) → staged pack tree`); +fixture skills under `tests/fixtures/`; NEW `tests/skill-import-adapters.test.ts` + one +marketplace e2e install case per format. + +**Steps:** both formats are SKILL.md-shaped (gap doc §G9); map: SKILL.md → pack skill +entity; `references/`/`scripts/`/`templates/` → carried verbatim; name collisions → +standard pack-precedence rules (no special-casing). Reject (with a clear error) skills +whose SKILL.md frontmatter is missing required fields rather than guessing. +**Acceptance:** fixture imports install/uninstall cleanly; imported skill resolvable and +`pack_activation`-toggleable; malformed fixture rejected with actionable message. + +--- + +## GA-R9 — `bobbit doctor` + +**Outcome:** a CLI subcommand that detects-and-explains (default) and repairs (`--fix`) +known state/config drift, OpenClaw-doctor style (gap doc §G8). + +**Owned files:** NEW `src/server/cli/doctor.ts` wired into the existing CLI entry (find +how `run`/`run.cmd` dispatch subcommands first); NEW `tests/doctor.test.ts`. + +**Steps:** v1 checks (each = `{id, severity, explain, canFix}`): (1) orphaned staff +records (reuse the orphan-detector logic — import, don't duplicate); (2) stale worktrees +(worktree-pool list vs git state, reuse `orphan-remote-branch-cleanup` machinery's +read side); (3) pack integrity (`.pack-meta.yaml` vs source manifest); (4) config keys +unknown to the current schema (report-only, never delete); (5) state-dir lock files older +than 24 h. `--fix` applies only `canFix` items, **always backing up the touched file to +`.bobbit/state/doctor-backups/<ts>/` first**; `--dry-run` = default behavior. Output: +human table + `--json`. **Tests:** one seeded-breakage fixture per check, assert +detect → fix → re-run clean → backup exists. **Acceptance:** `bobbit doctor` on a healthy +state dir exits 0 with "no findings"; Caretaker integration is explicitly OUT of scope +(future MC follow-up). diff --git a/docs/design/harness-gap-analysis.md b/docs/design/harness-gap-analysis.md new file mode 100644 index 000000000..7bfb1435c --- /dev/null +++ b/docs/design/harness-gap-analysis.md @@ -0,0 +1,293 @@ +# Harness gap analysis — OpenClaw, Hermes, Claude Code vs Bobbit + +Status: research complete; roadmap not started. Feature/UX gaps only — **cost/token +efficiency is explicitly out of scope** here because +[time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) §4 already covers the Claude Code / +Hermes comparison on that axis. + +> **Execution authority** for the §5 quick wins (R2–R6, R9): +> [gap-easy-wins-implementation-plan.md](gap-easy-wins-implementation-plan.md). +> Sequencing: [fable-program-execution-plan.md](fable-program-execution-plan.md). + +**The question this answers:** what do the strongest peer harnesses have that Bobbit lacks, +which of those Bobbit actually wants given its direction (the best harness out there — +extensible and fun for coders, researchers, and small-business owners alike), and which are +easy wins on substrate that already exists? + +**The one-paragraph answer:** Bobbit's core — goals/workflows with verification gates, team +orchestration, worktree/sandbox isolation, packs/marketplace, and a real web UI — is ahead of +all three peers; nothing in their orchestration stories needs copying. The real gaps cluster +in four places: **(1) memory** — all three peers remember things across sessions +automatically; Bobbit only has per-staff memory and a search index nobody consults at prompt +time; **(2) the self-improvement loop** — Hermes reviews every turn for skill/memory updates +and runs a background curator; Claude Code ships background "dreaming" memory consolidation; +Bobbit has neither (→ [autoimprovement.md](autoimprovement.md)); **(3) reach** — OpenClaw +meets users in 30+ messaging channels with companion apps while Bobbit is a localhost web +app; **(4) ambient automation UX** — peers have one-shot reminders, away summaries, and +"standing orders" conventions on top of cron, where Bobbit has (good) staff triggers but +nothing above them. Several gaps are genuinely cheap because the substrate (staff, triggers, +inbox, packs, session search) already exists — the roadmap in §5 sequences the easy wins +first. + +--- + +## §1 Method + +Source study of three checkouts (2026-06-10/11): + +| Harness | Repo / commit | Freshness | +|---|---|---| +| OpenClaw | `github.com/openclaw/openclaw` @ `2d404f1b` (2026-06-11) | current master | +| Hermes | `github.com/NousResearch/hermes-agent` @ `a8f404b` (2026-06-10) | current master | +| Claude Code | leaked-source mirror `github.com/yasasbanukaofficial/claude-code` @ `a371abb` (2026-04-05) | **~2 months stale** — findings cross-checked against public docs/changelog; anything load-bearing is flagged *(verify against current behavior)* | + +File citations below are repo-relative within those checkouts. "Easy win" means: high user +value, ≤ a few days of work, and lands on an existing Bobbit substrate without new +architecture. + +Bobbit baseline assumed known: goals/workflows/gates (`docs/goals-workflows-tasks.md`), +staff + triggers + inbox (`docs/staff-agents.md`, `docs/staff-triggers.md`), packs/marketplace +(`docs/marketplace.md`), session search (`src/server/search/`), sandbox/worktrees, the +planned extension platform ([extension-platform.md](extension-platform.md)). + +--- + +## §2 Comparison matrix + +✅ strong · 🟡 partial · ❌ absent. The right column says where the gap is handled. + +| Axis | OpenClaw | Hermes | Claude Code | Bobbit | Disposition | +|---|---|---|---|---|---| +| Goal/workflow orchestration, verification gates | 🟡 Task Flow | 🟡 kanban/delegate | 🟡 coordinator mode, subagents | ✅ | Bobbit ahead — keep | +| Sandbox & worktree isolation | 🟡 | 🟡 Docker | 🟡 | ✅ | Bobbit ahead — keep | +| Plugin/extension ecosystem | ✅ 143 extensions + ClawHub | 🟡 skills + agentskills.io hub | 🟡 plugins | 🟡 packs (platform designed, landing) | Covered by extension-platform.md | +| Cross-session memory (automatic) | ✅ memory extensions + workspace files | ✅ MEMORY/USER.md + per-turn review | ✅ memdir + autoDream | ❌ (staff memory only; search unused at prompt time) | **G1** | +| Self-improving skills | 🟡 (static skills, hub) | ✅ per-turn review + curator | 🟡 autoDream (memory only) | ❌ | **G2** → autoimprovement.md | +| Scheduled automation | ✅ cron + heartbeat | ✅ cron jobs w/ skills | 🟡 | ✅ staff `schedule` triggers | parity | +| One-shot "remind me at" | ✅ `--at` | ✅ one-shot schedules | 🟡 | ❌ | **G3** (easy) | +| Ambient/idle automation (heartbeat, commitments) | ✅ heartbeat + inferred commitments | 🟡 idle-triggered curator | 🟡 autoDream gates | ❌ | **G3** | +| Standing orders / persistent authority | ✅ first-class docs/convention | 🟡 SOUL.md | 🟡 CLAUDE.md | 🟡 role/staff prompts (no convention) | **G4** (easy, docs+templates) | +| Away summary / catch-up | 🟡 tasks audit | ❌ | ✅ `awaySummary` (small-fast model) | ❌ (unread dots only) | **G5** (easy) → mission-control.md briefing | +| Messaging channels (Telegram/WhatsApp/…) | ✅ 30+ channels, pairing, access groups | 🟡 Discord/telephony tools | ❌ (terminal/IDE/web) | ❌ | **G6** (strategic) | +| Companion / fun layer | 🟡 | ❌ | ✅ `buddy` (species/rarities/stats) | ✅ bobbit sprites | parity; see §G7 | +| Config doctor / migrations | ✅ `openclaw doctor --fix` | 🟡 | 🟡 | ❌ | **G8** (medium) | +| Cross-harness import | ✅ `migrate-claude`, `migrate-hermes` extensions | ✅ openclaw skill imports | ❌ | 🟡 Claude-plugin compat planned | **G9** (easy, extends planned work) | +| Onboarding for non-coders | ✅ companion apps, first-run UX priority | 🟡 | ✅ `projectOnboardingState`, MagicDocs, PromptSuggestion | 🟡 add-project flow | **G10** → mission-control.md | +| Multi-model / aux-model routing | ✅ per-extension providers | ✅ auxiliary slots (curator, vision, compression…) | ✅ small-fast-model pattern | 🟡 per-role models | **G11** (covered partly by token-cost CE-G6) | +| Trajectory datagen for training | ❌ | ✅ batch runner + `trajectory_compressor.py` | ❌ | ❌ | out of scope (note only) | +| Voice / TTS | ✅ | 🟡 | ❌ | ❌ | future; rides G6 channels | + +--- + +## §3 Per-gap analysis + +### G1 — Automatic cross-session memory (HIGH value, M effort) + +What peers do: + +- **Hermes**: two bounded files — `MEMORY.md` (2,200 chars) + `USER.md` (1,375 chars) — + injected as a frozen snapshot at session start (cache-safe), managed by the agent via a + `memory` tool with add/replace/remove, and **fed by a per-turn background review fork** + (`agent/background_review.py`) that asks "did the user reveal preferences/expectations + worth saving?". Bounded size forces consolidation; usage % is shown in the header + (`website/docs/user-guide/features/memory.md`). +- **Claude Code**: `memdir` auto-memory, `services/extractMemories` (per-session extraction), + `services/SessionMemory`, and `services/autoDream` consolidation (G2). +- **OpenClaw**: workspace bootstrap files auto-injected every session (`AGENTS.md`, `SOUL.md`, + `USER.md`, `MEMORY.md`, `IDENTITY.md`…) plus pluggable memory backends + (`extensions/memory-core`, `memory-lancedb`, `memory-wiki`, `active-memory`). + +Bobbit today: staff agents have pinned memory; sessions have nothing; the FlexSearch session +index exists but **nothing consults it at prompt time** (already identified as the provider +gap in extension-platform.md §2). + +Disposition: the *retrieval* half is the extension platform's `session-memory` reference pack +— don't duplicate. The cheap, immediately-valuable half is **the Hermes-style bounded user +profile**: a global `USER.md`-equivalent (server-scoped, ~500 tokens, agent-managed via a +small tool, injected as a stable prompt section through the existing `PromptParts` assembly +in `src/server/agent/system-prompt.ts`). Frozen-snapshot semantics preserve the cache prefix +(matches Bobbit's existing cache discipline per time-and-token-cost-efficiency §1). The *writing* half +(when to save) belongs to the autoimprovement loop (→ autoimprovement.md §loop, "observe"). + +### G2 — Self-improving skills (HIGH value; the subject of its own doc) + +Hermes is the reference implementation: per-turn background review with a tool whitelist +(`agent/background_review.py` — "most sessions produce at least one skill update"; patch-first +preference order; class-level umbrella skills) plus the idle-triggered **curator** +(`agent/curator.py`, `website/docs/user-guide/features/curator.md`): lifecycle +`active → stale → archived`, never-delete, pin/rollback/dry-run, tar.gz snapshots before every +pass, aux-model review. Claude Code's `autoDream` shows the gate discipline (time gate → +session-count gate → lock, forked subagent, first-class `DreamTask`). Full adopt/adapt/reject +analysis and Bobbit's design: **[autoimprovement.md](autoimprovement.md)**. + +### G3 — One-shot triggers + ambient automation (easy win) + +OpenClaw's automation taxonomy (`docs/automation/index.md`) distinguishes exact-time cron, +one-shot `--at`, **heartbeat** (flexible periodic batched checks with main-session context), +background-task ledger (`openclaw tasks list/audit`), and **inferred commitments** ("user +mentioned an interview → check in after"). Hermes cron jobs attach skills and persist per-job +output dirs (`cron/jobs.py`). + +Bobbit has the staff `schedule` (cron) trigger and the inbox. Missing, in increasing effort: + +1. **One-shot trigger** — `{ type: "schedule", config: { at: ISO } }` self-disabling after + fire. Small change to `staff-trigger-engine.ts` + trigger editor UI. Unlocks "remind me / + check on X at 5pm" via any staff. *(easy win)* +2. **Goal-event coverage** — triggers already cover `goal_created`/`goal_archived`; add + `gate_failed` and `session_errored` push triggers (same dispatcher pattern, + `goal-trigger-dispatcher.ts`) so system staff can react to failures. *(easy win)* +3. **Heartbeat-style digest** — not a new mechanism: a cron-triggered system staff whose + prompt says "scan what changed; act only if needed" (→ mission-control.md Observer). + Inferred commitments are deliberately **not** adopted as machinery — they're a prompt + pattern for Global Staff, documented in the standing-orders template (G4). + +### G4 — Standing orders convention (easy win, mostly docs) + +OpenClaw's `docs/automation/standing-orders.md` formalizes *permanent operating authority*: +each program defines **Scope / Trigger / Approval gates / Escalation rules**, lives in +auto-injected workspace files, and pairs with cron for enforcement ("cron says when, standing +orders say what you may do"). This is precisely the right authoring convention for Bobbit +staff system prompts — and it's what makes the autoimprovement trust ladder legible. + +Disposition: adopt as **convention + template, zero machinery**: a documented standing-order +template in `docs/staff-agents.md` (or a new `docs/standing-orders.md`), the staff-creation +assistant (`src/server/agent/staff-assistant.ts`) nudged to structure proposed prompts that +way, and the mission-control system-staff roster (mission-control.md §5) written in that +format from day one. + +### G5 — Away summary / catch-up (easy win) + +Claude Code's `services/awaySummary.ts` + `hooks/useAwaySummary.ts`: when the user returns +after a gap, a **small/fast model** (`getSmallFastModel()`) summarizes what happened while +they were away, seeded with session memory. Bobbit's equivalent surface already exists — the +staff inbox and unread dots — but nothing *synthesizes*. Disposition: the **Morning briefing / +Observer staff** in mission-control.md §"Added ideas" is the strategic home; a tactical +slice (per-session "while you were away" header when unread count is high, generated by a +cheap model) can ship independently and reuses the compaction-summary plumbing. *(easy win +for the tactical slice)* + +### G6 — Channels (strategic; the "anyone, not just coders" play) + +OpenClaw's moat: 30+ channel docs (`docs/channels/` — WhatsApp, Telegram, Signal, Discord, +Slack, iMessage, Matrix, SMS, Teams…), device pairing, per-group access controls, broadcast +groups, bot-loop protection. For Bobbit's small-business-owner audience this is the +difference between "a dev tool I open" and "an assistant I text". + +Disposition: **channels as packs**, riding the extension platform — a channel is exactly a +pack `runtime` (long-lived supervised service) + `provider` (session bridge) per +extension-platform.md §3. Do not build bespoke channel code in core. Sequencing: after +extension-platform P5 (runtimes). First channel: **Telegram** (simplest bot API, no phone +pairing). The flight-recorder/notification-policy work in mission-control.md defines what +gets *pushed* to a channel. Not an easy win — listed for the roadmap because of strategic +weight. + +### G7 — Fun/companion layer (parity, one borrowed idea) + +Claude Code ships a full companion pet (`src/buddy/` — species, hats, eyes, **rarities with +weights, named stats**) — gamified identity for what Bobbit already does better with live +status sprites. Borrowed idea worth a note in mission-control.md: Bobbit accessories already +exist per staff/role; a light "collection" surface (which accessories your staff have earned) +is cheap delight. No goal proposed; tracked as an idea only. + +### G8 — `bobbit doctor` (medium) + +OpenClaw treats config drift as a first-class product problem: breaking config changes ship +with a doctor migration; `openclaw doctor --fix` detects old shapes, explains, backs up, +rewrites (VISION.md "Configuration compatibility"). Bobbit's config cascade +(`docs/internals.md`) has no equivalent; stale `.bobbit/config` shapes surface as runtime +errors. Disposition: a `bobbit doctor` CLI subcommand (checks: config schema, orphaned staff, +stale worktrees, pack integrity, state-dir locks) — natural pairing with the Caretaker staff +(mission-control.md), which can *run* doctor checks on schedule and inbox the findings. + +### G9 — Cross-harness import (easy win, extends planned work) + +OpenClaw ships `extensions/migrate-claude` and `extensions/migrate-hermes`; Hermes auto-uses +skills imported from OpenClaw (`~/.hermes/skills/openclaw-imports/`, +`tests/skills/test_openclaw_migration.py` shows hardening). Bobbit already plans Claude-plugin +compat in the marketplace ([pack-based-marketplace.md](pack-based-marketplace.md)); extend the +same staging-adapter seam (`marketplace-install.ts`) with **import adapters for Hermes skills +and OpenClaw workspace skills** (both are SKILL.md-shaped already). One adapter module per +format + fixture-pack tests. Grows the pack catalog for free. + +### G10 — Onboarding (handled in mission-control.md) + +Claude Code invests heavily here (`projectOnboardingState.ts`, `services/MagicDocs`, +`services/PromptSuggestion`, `outputStyles`); OpenClaw makes first-run UX a stated top +priority. Bobbit's answer is the Mission Control first-run flow (mission-control.md §1): +land in a chat that can set everything up. Output styles (tone/verbosity presets per +audience) are noted as a cheap follow-on — a role-prompt preset, no machinery. + +### G11 — Auxiliary-model routing (note) + +Hermes routes every side task (curator, vision, compression, session search) through named +`auxiliary.<task>` model slots with one picker UI. Bobbit has per-role models and the +token-cost plan adds a cheap summarizer (CE-G6.1). When autoimprovement lands its review/judge +passes, give them named aux slots from day one rather than hardcoding models — one config +surface, matching `per-role-model-overrides.md` conventions. + +--- + +## §4 Things examined and deliberately not adopted + +- **Inferred commitments as machinery** (OpenClaw) — prompt-pattern value without the + false-positive machinery; folded into standing-orders templates (G4). +- **Trajectory datagen / model training** (Hermes `batch_runner.py`, + `trajectory_compressor.py`) — out of charter for a harness; transcripts are already + retained if this ever changes. +- **TUI parity** — peers are terminal-first; Bobbit's web UI is the differentiator. The cost + is the battery discipline documented in + [client-performance-battery.md](client-performance-battery.md). +- **Bespoke channel integrations in core** — channels arrive as packs or not at all (G6). + +--- + +## §5 Prioritized roadmap + +Easy wins first; each row names the owning doc/goal so this table stays a router, not a +second backlog. + +| # | Item | Gap | Effort | Owner | +|---|---|---|---|---| +| R1 | Animation/timer battery fixes | — | S–M | [client-performance-battery.md](client-performance-battery.md) P1–P2 | +| R2 | One-shot (`at`) staff trigger + `gate_failed`/`session_errored` push triggers | G3 | S | new goal off this doc | +| R3 | Standing-orders template + staff-assistant nudge | G4 | S (docs) | new goal off this doc | +| R4 | "While you were away" summary slice (cheap model) | G5 | S–M | tactical slice; strategic home is mission-control.md briefing | +| R5 | Bounded global user-profile memory (Hermes USER.md pattern) | G1 | M | new goal; prompt-section + tool; coordinate with extension-platform `session-memory` | +| R6 | Hermes/OpenClaw skill import adapters | G9 | M | extends marketplace adapter seam | +| R7 | Mission Control core (global scope, meta-tool, system staff) | G3/G5/G10 | L | [mission-control.md](mission-control.md) | +| R8 | Autoimprovement loop (review, curator, trust ladder) | G2 | L | [autoimprovement.md](autoimprovement.md) | +| R9 | `bobbit doctor` | G8 | M | new goal; pairs with Caretaker staff | +| R10 | Telegram channel pack (first channel) | G6 | L | after extension-platform runtimes phase | +| R11 | Aux-model slots for review/judge passes | G11 | S (when R8 lands) | folded into autoimprovement phases | + +Implementation notes for R2–R4 (the goals this doc owns directly, written to be executable +without re-deriving context): + +- **R2**: extend `StaffTrigger` config union in `src/server/agent/staff-store.ts` with + `{ at: string }` alongside `{ cron }`; `staff-trigger-engine.ts` fires once when + `now >= at && !lastFired`, then sets `enabled: false`; trigger-editor UI on the staff edit + page gains a datetime field; e2e test beside the existing trigger specs. For the push + triggers, mirror the `goal_created` wiring in `goal-trigger-dispatcher.ts` from the gate / + session-status mutation points, required-prompt rule included (`docs/staff-triggers.md`). +- **R3**: add the Scope/Trigger/Approval/Escalation template to `docs/staff-agents.md`; update + the staff assistant's proposal guidance in `src/server/agent/staff-assistant.ts`; seed the + mission-control pack roster (mission-control.md §5) in that format. +- **R4**: server: on session activation with unread count above threshold, run a + cheap-model summarization over messages since `lastSeen` (reuse the compaction-summary + path; aux-model slot per G11); client: collapsible "while you were away" banner above the + transcript tail. Behind a setting; browser E2E for banner render + dismissal persistence. + +R2 trigger contract (so implementation cannot drift): extend the `StaffTrigger` union with + +```ts +{ type: "schedule"; config: { cron: string; timezone?: string } | { at: string /* ISO-8601 */ } } +{ type: "gate_failed"; config: {}; prompt: string } // required prompt, like goal_created +{ type: "session_errored"; config: {}; prompt: string } +``` + +`at` semantics: fire once when `now >= at && !lastFired`, then set `enabled: false` (record +kept for audit). Push triggers dispatch from the gate-failure and session-error mutation +points via `goal-trigger-dispatcher.ts`, mirroring `goal_created` exactly (including the +required-prompt validation in `docs/staff-triggers.md`). + +Execution tracking for every row above: [fable-program-execution-plan.md](fable-program-execution-plan.md). diff --git a/docs/design/mission-control-implementation-plan.md b/docs/design/mission-control-implementation-plan.md new file mode 100644 index 000000000..a0e78118e --- /dev/null +++ b/docs/design/mission-control-implementation-plan.md @@ -0,0 +1,268 @@ +# Mission Control — Implementation Plan (hand-off) + +Status: ready for execution, not started. Workstream **MC** in +[fable-program-execution-plan.md](fable-program-execution-plan.md). + +Companion to [mission-control.md](mission-control.md) (the WHAT/WHY — read its §1–§8 and +**Appendix A contracts** before implementing anything). This document is the HOW: goals +written so an implementer agent that has not seen any other goal can execute its goal from +this text plus the cited contracts alone. + +> **Anchor baseline:** fable-docs @ 2026-06-11 (master parent `6ec8c8f9`). Line numbers WILL +> drift — locate by **symbol name**; if a named symbol does not exist where described, STOP +> and re-derive from the cited pattern file; do not improvise a parallel mechanism. +> +> **Precision policy:** MC-P0…P3 are specified to file/function level. MC-P4/P5 are +> contract level (their substrate is created by P0–P3); re-verify every anchor first. +> +> **Universal rules:** [extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) +> (definition of done, patterns library) + [fable-program-execution-plan.md §1](fable-program-execution-plan.md) +> (doc-is-the-spec deviation protocol, shared-seam serialization) bind every goal below. + +--- + +## Goal map + +``` +MC-P0 global scope ──→ MC-P1 sessions+sidebar ──┬─→ MC-P2a registry ─→ MC-P2b tool group + └─→ MC-P3 flight recorder +MC-P2b + MC-P3 + GA-R2 ──→ MC-P4a pack ─→ MC-P4b crew ──→ MC-P5 briefing/onboarding/budgets +``` + +--- + +## MC-P0 — Global scope legitimization + +**Outcome:** `projectId: "system"` + `global: true` is a valid, first-class home for staff; +zero orphan-banner noise; nothing user-visible yet. + +**Owned files:** `src/server/agent/project-registry.ts`, `staff-store.ts`, +`staff-manager.ts`, `server.ts` (orphaned-staff route only), `docs/staff-agents.md`; +NEW `tests/global-staff-scope.test.ts`. + +**Steps** + +1. `project-registry.ts`: next to `SYSTEM_PROJECT_ID` (top of file), add + `export function isSystemScope(id: string): boolean` and + `export function missionControlDir(): string` per mission-control.md Appendix A.1 + (`path.join(stateDir, "mission-control")`, `mkdirSync` recursive, lazily). Do NOT touch + `participatesInVisibleOrder()` — the system project stays out of the visible order; + add a one-line pinning test for that instead. +2. `staff-store.ts`: extend `interface PersistedStaff` (symbol at `:60`) with + `global?: boolean`; in the store's load path (inside `class StaffStore`, the same + normalisation pass that runs `normalizeStaffAccessory`), normalise missing/non-boolean + `global` → `false`. Persist round-trip unchanged otherwise. +3. `staff-manager.ts`: in the create/edit validation that currently resolves a real project + (the path documented in `docs/staff-agents.md` §"Project and cwd anchoring"), add the + explicit branch from Appendix A.2: `global === true` ⇒ require + `projectId === SYSTEM_PROJECT_ID`, cwd inside `missionControlDir()` (default to it when + blank), force worktree off (ignore `worktree` flag; never set `branch`/`worktreePath`), + leave `sandboxed` semantics untouched. `global === false` ⇒ existing rules byte-for-byte. +4. Orphan route: in `server.ts`, locate the `GET /api/staff/orphaned` handler; exclude + records with `global === true`. Legacy records (no `projectId`, or system-project + without `global`) remain orphans — **no migration**. +5. Update `docs/staff-agents.md` §"Legacy staff records" with one paragraph distinguishing + intentionally-global staff (and a pointer to mission-control.md). + +**Tests (author first; RED on unmodified tree where marked)** + +- `tests/global-staff-scope.test.ts` (node:test, store + manager level — follow + `tests/staff-orphan-reassign.test.ts` for fixtures): + (a) load normalisation `global` missing → `false`; + (b) create global staff with blank cwd → cwd = missionControlDir() *(RED)*; + (c) global staff with cwd outside missionControlDir → rejected *(RED)*; + (d) global + real projectId → rejected *(RED)*; + (e) worktree flag ignored for global staff *(RED)*; + (f) orphan listing excludes `global: true`, still includes legacy orphan *(RED)*; + (g) pinning: system project never appears in visible order (GREEN today — pins step 1). + +**Acceptance:** all new tests green; full staff suites +(`tests/staff-*.test.ts`, `tests/e2e/staff*.spec.ts`) green unmodified. + +--- + +## MC-P1 — Mission Control sessions + sidebar entry + +**Outcome:** a pinned Mission Control entry at the very top of the sidebar (desktop + +mobile); the user can create/use/archive a global chat session that survives reload and +gateway restart. + +**Owned files:** session-create path (`server.ts` + `src/server/agent/session-setup.ts` — +**confine to the project-resolution branch**; this file is a §1.4 shared seam), +`src/app/sidebar.ts`, `src/app/render.ts` (mobile shell), `src/app/state.ts` (nav id only); +NEW `tests/e2e/ui/mission-control.spec.ts`. + +**Steps** + +1. Server: allow session creation with `projectId: "system"`. Find where session creation + resolves the project (the same resolution the projectless path in + `tests/e2e/sessions-projectless.spec.ts` pins — read that spec first); when + `isSystemScope(projectId)`, set cwd = `missionControlDir()`, skip worktree/git setup + entirely (same code path as non-git projects), and tag the session record so the client + can group it (reuse the existing `projectId` field — no new field). +2. Sidebar: in `renderSidebar()` (`src/app/sidebar.ts`, symbol `renderSidebar`), render a + Mission Control block **before** the project loop: header row (nav id + `mission-control-header`), expandable; lists sessions with + `projectId === SYSTEM_PROJECT_ID` and staff with `global === true` (staff sub-grouped + System/Global by a `roleId`-independent flag — for now: pack-roster names, see MC-P4; + until then one flat Staff group). Add "+ New chat" button posting a system-scope + session. Reuse `renderStaffSidebarSection(filteredList, projectId)` with the system + scope rather than forking it. +3. Mobile: mirror in `render.ts::renderSidebarShellMobile` (same data, existing mobile + row components). +4. Exclusions: Mission Control does not participate in project reorder + (`renderProjectReorderHandle` must not render for it) and is not draggable; keyboard + nav follows `docs/sidebar-keyboard-navigation.md` (its nav id participates in the + ordered nav list). + +**Tests (author first)** + +- `tests/e2e/ui/mission-control.spec.ts` (browser E2E, pattern + `tests/e2e/ui/settings.spec.ts`): entry visible at top (strictly above first project); + create chat → session opens; reload → entry + session persist; archive session → + disappears; keyboard navigation reaches the header. *(all RED)* +- Extend `tests/e2e/sessions-projectless.spec.ts` with one system-scope creation case + asserting cwd = missionControlDir and no worktree. *(RED)* + +**Acceptance:** E2E green incl. restart-persistence (sessions.json round-trip — assert via +reload case); existing sidebar suites (`tests/e2e/ui/side-panel-tabs.spec.ts`, +project-reorder specs) green unmodified. + +--- + +## MC-P2a — `bobbit` meta-tool registry (server only) + +**Outcome:** the operation registry exists, is pinned by tests, and dispatches in-process — +no tool exposed to any model yet. + +**Owned files:** NEW `src/server/agent/bobbit-meta-tool.ts`; NEW +`tests/bobbit-meta-tool.test.ts`. + +**Steps** + +1. Implement `BobbitOp`/`OpTier` and the **complete catalog v1 table** from + mission-control.md Appendix A.3 — every row, no additions, no omissions. For each row, + resolve the cited REST route to its `server.ts` handler **by reading + `docs/rest-api.md` + the handler source now**; record the resolved path template in the + registry entry. +2. `export async function dispatchBobbitOp(op, args, ctx)`: validates `args` against the + row's `argsSchema`, then invokes the same internal handler function REST uses (extract a + shared function where the route body is inline — smallest possible extraction, no + behavior change). `sensitive` rows: return a structured `requiresConfirmation` marker — + the ask-flow wiring is MC-P2b's job. `mutate`/`sensitive`: call `recordActivity()` — + until MC-P3 lands, a no-op stub exported from the same module (one TODO referencing + MC-P3). +3. `export function describeBobbitOp(op?)` returning `{op, tier, doc, argsSchema}` rows. + +**Tests (author first)** + +- `tests/bobbit-meta-tool.test.ts`: (a) **catalog pinning** — every registry row resolves + to a callable handler (no 404-shaped dispatch), and the exported op enum equals the + registry keys exactly *(RED)*; (b) schema validation rejects malformed args per op + *(RED)*; (c) `sensitive` ops return the confirmation marker, never execute directly + *(RED)*; (d) read ops do not invoke `recordActivity`, mutate ops do (spy on the stub) + *(RED)*. + +**Acceptance:** new tests green; `npm run check` clean; zero diff outside owned files +except minimal handler extractions. + +## MC-P2b — `bobbit` tool group + tiers + policy + +**Outcome:** Mission Control sessions/staff can call `bobbit`/`bobbit_describe` +end-to-end with ask-gating on sensitive ops. + +**Owned files:** NEW `defaults/tools/bobbit/{bobbit.yaml,bobbit_describe.yaml,extension.ts}`; +tool-policy default for system scope (locate where role/tool policies resolve per scope — +`docs/staff-agents.md` §roleId notes `session-setup.ts` applies them); extend +`tests/tool-description-budget.test.ts`; NEW `tests/e2e/bobbit-meta-tool.spec.ts`. + +**Steps** + +1. Copy `defaults/tools/team/` anatomy exactly (YAML fields: `name`, `description`, + `summary`, `params`, `provider: {type: bobbit-extension, extension: extension.ts}`, + `group: Bobbit`, `docs`, `detail_docs`). `extension.ts` bridges to the gateway via the + `tool-guard-extension.ts` HTTP long-poll pattern (§0.1 patterns library) and calls + `dispatchBobbitOp`. +2. Sensitive ops: wire the `requiresConfirmation` marker into the existing blocking-ask + flow (`docs/blocking-tools.md` / `docs/non-blocking-ask.md` — find the tool that + currently blocks on user confirmation and copy its server round-trip). +3. Default availability: system-scope sessions/staff get the group enabled; project + sessions get it only via explicit role tool-policy opt-in. Pin both directions. + +**Tests:** budget rows for both YAML descriptions; API E2E (`tests/e2e/`, in-process +gateway): a system-scope session calls `projects.list`, creates a temp project +(`projects.add` behind a scripted confirm), creates a nested goal in it, sends a message to +a second session, enqueues a staff inbox entry; asserts a project session WITHOUT opt-in +gets a tool-not-available error. *(all RED)* + +**Acceptance:** E2E green; `tool-description-budget` green; tool-guard/policy suites green +unmodified. + +--- + +## MC-P3 — Flight recorder + +**Outcome:** `recordActivity()` is real; `GET /api/activity` + WS live tail + Activity +panel under the Mission Control sidebar entry. + +**Owned files:** NEW `src/server/agent/activity-store.ts`; `server.ts` (one GET route + WS +subscribe handling in `src/server/ws/`); NEW `src/app/activity-panel.ts`; `sidebar.ts` +(Activity nav row); NEW `tests/activity-store.test.ts`, `tests/e2e/ui/activity-panel.spec.ts`. + +**Steps** + +1. Implement `ActivityEntry` + `recordActivity()` exactly per mission-control.md Appendix + A.4 (JSONL append, crash-safe pattern from + [session-store-crash-safety.md](session-store-crash-safety.md); rotate at 8 MB to + `activity-YYYY-MM-DD.jsonl`). Replace the MC-P2a stub (delete the TODO). +2. REST: `GET /api/activity` with `since/actor/tier/projectId/limit` filters (default 100, + max 500, newest-first). +3. WS: `subscribe_activity` opt-in frame; fan out `activity_entry` ONLY to subscribed + sockets (goal-fanout precedent — + [reduce-server-cpu-experiment-goal-fanout.md](reduce-server-cpu-experiment-goal-fanout.md)). +4. UI: Activity nav row in the Mission Control block → panel listing entries (actor chip, + action, target link, evidence link, revert button only when `revert` present — + revert wiring itself lands with AI-P5; render disabled with tooltip until then), filter + controls, live-tail via the subscription. + +**Tests:** unit — rotation boundary, filter math, crash-safe append (kill mid-write +fixture); browser E2E — meta-tool mutate call appears in panel without reload (live tail), +filters narrow, read ops absent. *(RED)* + +**Acceptance:** every `mutate`/`sensitive` call from MC-P2b's E2E visible with correct +actor attribution; WS fanout test proves non-subscribed sockets receive nothing. + +--- + +## MC-P4a/P4b — mission-control pack + crew *(contract level — re-verify anchors)* + +Contracts: mission-control.md §5 (roster + trigger table + disabled-by-default rule), +Appendix A.5 (pack layout), A.6 (owned files). Depends on GA-R2's push triggers (shared +seam — if GA-R2 hasn't landed, land the triggers there first, never here). + +- **P4a:** pack skeleton copying `market-packs/pr-walkthrough/` (build wiring: + `scripts/build-market-packs.mjs` + `copy-builtin-packs.mjs` `FIRST_PARTY_PACKS`); roles + written in the standing-orders format (GA-R3); panel route `#/ext/mission-control`. + Litmus tests per the `market-packs/artifacts` convention. +- **P4b:** idempotent `POST /api/mission-control/crew` instantiating staff records from + `staff-templates/crew.yaml` (idempotency key: staff name + global flag); Observer + triggers enabled, others disabled (§5); panel "Create crew" button. E2E: create crew ⇒ + 4 global staff with correct triggers; re-POST ⇒ no duplicates; Caretaker manual wake + dry-run sweep ⇒ flight-recorder entry; panel deep-link survives reload. + +**Acceptance:** pack installs/uninstalls cleanly; crew E2E green; activity panel shows the +dry-run entry. + +## MC-P5 — Briefing, onboarding, budgets *(contract level; 3 separable PRs)* + +Contracts: mission-control.md §7. (1) Observer briefing = inbox digest assembled from: +flight recorder window, cost routes (`docs/session-cost.md`), notification-policy stuck +predicate, AI calibration endpoint when present (omit section when absent). (2) First-run: +zero registered projects ⇒ client routes to a Mission Control chat seeded with the +onboarding prompt (drives the existing add-project proposal flow — do not build a new +wizard). (3) Budgets: `budgets.{project,staff}.monthlyUsd` config keys; Observer breach +escalation entry + briefing line. Browser E2E per feature (first-run on a fresh state dir; +briefing digest renders; budget breach escalates). + +**Acceptance:** mission-control.md §1's small-business-owner sentence works on a clean +install: project created + Friday `schedule` trigger attached via chat alone. diff --git a/docs/design/mission-control.md b/docs/design/mission-control.md new file mode 100644 index 000000000..09b5556a5 --- /dev/null +++ b/docs/design/mission-control.md @@ -0,0 +1,463 @@ +# Mission Control — the global scope, global staff, and the flight recorder + +Status: design accepted, not started. This is the WHAT/WHY + contracts (Appendix A). + +> **Execution authority:** implement from +> [mission-control-implementation-plan.md](mission-control-implementation-plan.md) +> (per-goal owned files, symbol anchors, RED→GREEN tests, acceptance). Sequencing: +> [fable-program-execution-plan.md](fable-program-execution-plan.md). + +**What this is:** a first-class **global scope** at the top of the sidebar — the main entry +point for using Bobbit. Mission Control sessions can set up new projects, talk to any agent +in any project, create goals (and goals-of-goals) anywhere, and manage everything Bobbit can +do — because everything is an API call. It is home to **Global Staff** (the user's personal +assistants) and **Global System Staff** (Bobbit's own housekeeping crew: cleanup, +memory/housekeeping scheduling, dreaming, observability, and the autoimprovement loop from +[autoimprovement.md](autoimprovement.md)). And it is where the **flight recorder** lives — +the audit surface that makes growing agent autonomy trustworthy. + +**Why now:** Bobbit's destination is "the best harness out there, usable by anyone — coders, +researchers, small-business owners". Today the entry point is a *project*, which presumes the +user already thinks in repos. Mission Control inverts that: you land in one conversation that +can do everything, and projects become things it sets up *for* you +([harness-gap-analysis.md](harness-gap-analysis.md) G10). Peers point the same direction — +OpenClaw's assistant-first framing and standing-orders authority model, Claude Code's +away-summaries and background dream tasks — but none of them has Bobbit's goals/staff/packs +substrate to anchor it. + +--- + +## §1 UX — one place that runs everything + +- **Placement:** a pinned **Mission Control** entry at the very top of the left sidebar, + above all projects — rendered by `renderSidebar()` (`src/app/sidebar.ts:1458`) and the + mobile shell (`render.ts::renderSidebarShellMobile`), expandable like a project section: + its sessions, its staff (Global Staff and System Staff sub-grouped), and an **Activity** + entry (the flight recorder, §6). It does not participate in project reordering + (`docs/sidebar-project-reorder.md` — hidden/system projects already don't). +- **First run:** instead of today's empty-state dead end, a new user lands in a Mission + Control chat that interviews them and sets up their first project(s) via the `bobbit` + meta-tool (§3) — reusing the existing add-project preflight and project-onboarding flows + (`docs/add-project-preflight.md`, `docs/design/project-onboarding-ux.md`) rather than + replacing them: Mission Control *drives* the same proposals the UI drives. +- **Daily rhythm:** the **morning briefing** (§7.1) — overnight activity, costs, pending + proposals, stuck goals — arrives as an inbox digest; the user reads it in the Mission + Control chat and acts by replying. +- **Tone:** Mission Control speaks plain language. A small-business owner should be able to + say "set up a folder for my invoices and remind me every Friday to send them" and get a + project + a scheduled Global Staff trigger, never seeing the words "worktree" or "cron". + +## §2 The global scope (core) + +The synthetic hidden project already exists: `SYSTEM_PROJECT_ID = "system"` +(`src/server/agent/project-registry.ts:41`). Today it is a quarantine zone — staff stored +under it are "orphans" to re-home (`docs/staff-agents.md` §Legacy staff records). Mission +Control promotes it to a first-class, still-synthetic, un-deletable scope: + +- **Visibility:** keep it out of `visibleProjects()` (`project-registry.ts:117`) — it is not + a project row; the sidebar renders it as the dedicated top entry instead. No drag-reorder, + no archive, no delete. +- **Sessions:** global sessions attach to the system project. Projectless session handling + already exists (`tests/e2e/sessions-projectless.spec.ts` pins current behavior — extend, + don't fork). **Cwd:** a dedicated `mission-control/` workspace dir under the gateway's + state dir — `missionControlDir()` per A.1, i.e. `<bobbit-dir>/state/mission-control/`, + where `bobbitDir()` resolves `BOBBIT_DIR` env → else `<gateway root>/.bobbit` + (`src/server/bobbit-dir.ts`); it is `~/.bobbit/…` only when `BOBBIT_DIR` points at home. + A real, writable directory so file tools work; never a project root and never + `config.defaultCwd`. +- **Staff:** `staff-store.ts` / `staff-manager.ts` accept `projectId: "system"` as a *valid* + owner: project-anchoring validation (staff-agents.md §Project and cwd anchoring) gains an + explicit system-scope branch (cwd must be inside the mission-control workspace; worktree + mode is always off — there is no repo). The orphan detector + (`GET /api/staff/orphaned`) distinguishes **legacy orphans** (no `projectId`) from + **intentionally global** staff (`projectId === "system"` *and* a new `global: true` field + on `PersistedStaff`) so the orphan banner stops flagging them. Legacy records without the + flag remain orphans — no silent migration. +- **Goals:** global-scope goals are **not** introduced. Goals stay project-owned (worktrees, + branches, and gates are repo concepts). Mission Control creates goals *in* target projects + via the meta-tool — which already covers "goals with goals" since the nested-goals API is + project-scoped (`docs/nested-goals.md`). + +## §3 The `bobbit` meta-tool group (core) + +"Manage anything Bobbit-related — everything is an API call." Rather than one tool per REST +endpoint (the context-bloat failure mode), reuse the shipped MCP meta-tool aggregation +pattern (`docs/mcp-meta-tools.md`, `docs/design/mcp-meta-tool-aggregation.md`) against +Bobbit's own gateway: + +- New tool group `defaults/tools/bobbit/` exposing two tools: + - `bobbit(operation, args)` — `operation` constrained by a typo-proof enum; + - `bobbit_describe(operation?)` — returns the JSON schema + doc string for an operation + (lazy discovery, same shape as `mcp_describe`). +- The **operation catalog is a curated allowlist**, not a blind mirror of + `server.ts::handleApiRoute()`: each entry maps `operation → {method, path template, schema, + tier}` in one registry module (`src/server/mcp/bobbit-meta-tool.ts` or sibling — follow + where the MCP meta-tool implementation lives). Initial catalog: projects + (list/add/preflight), sessions (list/create/send-message/archive), goals + (list/create/nest/archive), staff (list/create-proposal/wake/inbox-enqueue), worktrees + (list/cleanup), packs (list/install from registered sources), config (read), costs (read), + activity (read, §6). +- **Safety tiers** enforced in the registry, not the prompt: `read` (free) · `mutate` + (allowed but every call lands in the flight recorder) · `sensitive` (delete project, + bulk-archive, pack install — requires the existing ask/blocking-tool confirmation flow, + `docs/blocking-tools.md` / `docs/non-blocking-ask.md`). Role `toolPolicies` and the tool + guard apply unchanged — this is just a tool group. +- **Auth:** calls execute in-process against the same handlers REST uses (no self-HTTP + loop), attributed to the calling session in the flight recorder. +- **Availability:** in the default tool policy for Mission Control sessions/staff; available + to project sessions only by explicit role/tool-policy opt-in. + +Cross-project communication falls out of the catalog: `sessions.send-message` (delivers into +any session's prompt queue, `docs/prompt-queue.md`) and `staff.inbox-enqueue` (the existing +inbox contract, `docs/staff-inbox.md`) — no new channel machinery. + +## §4 Global Staff (user-facing) + +Plain staff agents owned by the global scope: personal assistant, researcher, email/invoice +chaser — anything not tied to a repo. They get the existing machinery unchanged (roles, +accessory, memory, cron/manual triggers, inbox) plus the `bobbit` meta-tool so they can act +across projects. Authoring convention: **standing orders** (Scope / Trigger / Approval gates / +Escalation — gap-analysis G4) baked into the staff-creation assistant's guidance. + +## §5 Global System Staff (Bobbit's own crew) + +Bobbit-discretionary housekeeping staff, **defined as data** (roles + skills + trigger +templates) and shipped as the **mission-control built-in pack** (§7 split). They are real +staff records the user can inspect, pause, retune, or delete — discretion with a leash. Every +run writes to the flight recorder. Roster v1: + +| Staff | Charter (standing-order form) | Triggers | +|---|---|---| +| **Caretaker** | Sweep stale worktrees/branches (builds on `docs/design/orphan-remote-branch-cleanup.md` and the worktree pool), stale archived sessions, dangling sandbox containers. *Approval gate:* deletion lists over N items → inbox proposal first. | weekly `schedule`, `goal_archived` | +| **Archivist** | Transcript/index hygiene; memory consolidation scheduling; **dreaming** — gate-disciplined idle reflection over recent transcripts producing memory/skill proposals (mechanism in autoimprovement.md AI-P3). | nightly `schedule` (gates inside) | +| **Improver** | Runs the autoimprovement loop (autoimprovement.md §7). | `goal_archived`, `gate_failed`, weekly | +| **Observer** | Observability digests: cost/burn-rate, failure patterns, flaky behaviors, perf-flag regressions; assembles the **morning briefing** (§7.1). *Escalation:* anomalies (cost spike, repeated gate failures) → immediate inbox entry, not the next briefing. | daily `schedule`, `session_errored` | + +All four use existing trigger machinery (`docs/staff-triggers.md`) plus the two new push +triggers (`gate_failed`, `session_errored`) shared with gap-analysis R2 and autoimprovement +AI-P1. The pack ships triggers **disabled-by-default except Observer**; enabling the crew is +a one-click, per-staff choice during Mission Control onboarding — discretion is granted, not +presumed. + +## §6 The flight recorder (core) + +Visibility is king: every autonomous action is on the record, queryable, and (where +applicable) revertible. + +- **Store:** append-only JSONL `.bobbit/state/activity.jsonl` (crash-safe append; daily + rotation with size cap), entries: + `{ts, actor: {kind: session|staff|policy|user, id}, action, target, tier, evidence?, + revert?: {kind, ref}, projectId?}`. +- **Writers:** the `bobbit` meta-tool (every `mutate`/`sensitive` call), system-staff wake + results, autoimprovement decisions (autoimprovement.md §3 "Record"), Caretaker sweeps. + One `recordActivity()` helper in the server; writers call it explicitly — no magic + middleware pass that would record noise. +- **Read surface:** `GET /api/activity?since&actor&tier&projectId` (paginated) + a WS event + for live tail; rendered as the **Activity** panel under the Mission Control sidebar entry — + filterable timeline, each row: actor chip, action, target link, evidence link, revert + button when `revert` present. +- **Retention:** keep the JSONL indefinitely (it's small text); the panel reads windowed. + +## §7 Added product ideas riding the same substrate + +1. **Morning briefing** (Observer): daily digest — what ran overnight, cost/burn vs budget, + pending proposals, stuck goals (notification-policy "idle stuck" rule feeds this), + calibration status from autoimprovement shadow mode. Delivered as a staff inbox digest; + later deliverable to external channels when channel packs exist (gap-analysis G6). +2. **Global cost guardrails**: per-project and per-staff soft budgets (config), surfaced in + the briefing and as an Observer escalation on breach; builds on per-session cost tracking + (`docs/session-cost.md`) and the planned CE-G0.1 ledger. Soft-stop only in v1 (warn + + require confirmation for new expensive sessions), never hard-kill mid-goal. +3. **First-run onboarding** (§1): Mission Control chat as the guided setup path. + +## §8 Architectural split — what is core, what is pack + +Decision (user-confirmed): **core with pack-shaped seams**, leaning on the extension platform +([extension-platform.md](extension-platform.md)) where it genuinely fits. Pack panels, +renderers, launchers, and deep-link routes already work today (`src/app/pack-panels.ts`, +`pack-entrypoints.ts`, `#/ext/<routeId>`, the pr-walkthrough built-in pack) — so the split +is honest, not aspirational: + +| Piece | Lives | Why | +|---|---|---| +| Global scope legitimization, staff `global` flag, cwd rules | **core** | project-registry/staff-store invariants; nothing else can own them | +| Sidebar top entry + Activity navigation | **core** | the sidebar shell is core UI; packs cannot (and should not) claim the top slot | +| `bobbit` meta-tool group + operation registry + tiers | **core** | security-sensitive chokepoint; must be budget-pinned and test-pinned like other `defaults/tools/` | +| Flight recorder store + REST/WS | **core** | platform-level audit primitive; packs *write into* it (a natural future Extension-Host API) | +| **mission-control pack** (built-in, first-party — `market-packs/mission-control/`): system-staff roles, skills, standing-order prompts, trigger templates; the Mission Control **dashboard panel** (briefing view, crew status, activity timeline UI) at `#/ext/mission-control` | **pack** | pure content + panel — exactly what `builtin-packs.ts` + `copy-builtin-packs.mjs` + pack panels deliver today (pr-walkthrough precedent) | +| Staff-roster *instantiation* from pack data | core loader seam | v1: a small core bootstrap reads the pack's staff templates ("create crew" action); migrates to the extension platform's `workflows`/provider entities when those phases land — the seam is the template format, which stays pack-owned either way | + +Dependency on extension-platform phases: **none for MC-P0…P4** (panels/routes exist); +the staff-template entity and any provider-based enrichment ride later platform phases +without rework because the data already lives in the pack. + +## §9 Phased implementation plan + +Order matters: P0→P2 are the spine; P3/P4 can proceed in parallel after P1. + +```mermaid +graph LR + P0[MC-P0 global scope] --> P1[MC-P1 sessions + sidebar] + P1 --> P2[MC-P2 bobbit meta-tool] + P1 --> P3[MC-P3 flight recorder] + P2 --> P4[MC-P4 mission-control pack + system staff] + P3 --> P4 + P4 --> P5[MC-P5 briefing + onboarding + budgets] + P4 -.-> AI[autoimprovement.md AI-P2+] +``` + +### MC-P0 — Global scope legitimization + +1. `project-registry.ts`: keep `SYSTEM_PROJECT_ID` hidden from `visibleProjects()`; add + `isSystemScope(id)` helper; ensure the system project's anchor dir is the + `missionControlDir()` workspace under the state dir (A.1; created lazily). +2. `staff-store.ts`: add `global?: boolean` to `PersistedStaff` (loader normalises missing → + `false`); `staff-manager.ts` accepts `projectId: "system"` + `global: true` with + system-scope cwd validation (inside the mission-control workspace) and worktree forced + off. +3. Orphan logic: `GET /api/staff/orphaned` excludes `global: true` records; orphan banner + unchanged for legacy records. Update `docs/staff-agents.md` orphan section. +4. Tests: unit (store normalisation, scope validation); API e2e (create global staff; orphan + endpoint excludes it; legacy orphan still flagged — extend + `tests/staff-orphan-reassign.test.ts` / `tests/e2e/staff-patch-reassign.spec.ts` + patterns). + +Acceptance: a global staff can be created, woken, and edited; zero orphan-banner noise; all +existing staff/project suites green. + +### MC-P1 — Mission Control sessions + sidebar entry + +1. Session creation accepts the system scope (extend the projectless path pinned by + `tests/e2e/sessions-projectless.spec.ts`); cwd = mission-control workspace. +2. Sidebar: pinned top entry in `renderSidebar()` + mobile shell; expandable section listing + global sessions and staff (System Staff sub-grouped); keyboard-nav per + `docs/sidebar-keyboard-navigation.md`; excluded from project reorder. +3. "New Mission Control chat" affordance in the entry header. +4. Tests: browser E2E `tests/e2e/ui/mission-control.spec.ts` — entry visible at top, create + chat, persists across reload, archived session disappears (navigation/happy + path/persistence/cleanup per the AGENTS.md E2E rule). + +Acceptance: a user can converse in a Mission Control session that survives gateway restart +(sessions.json) and reload. + +### MC-P2 — `bobbit` meta-tool + +1. Operation registry module (catalog table in §3): typed entries, tier, schema; unit-test + pinning that every entry resolves to a real route handler and that the enum matches the + registry (the "missing test IS the bug" rule). +2. Tool group `defaults/tools/bobbit/` (two tools; description budget — extend + `tests/tool-description-budget.test.ts` pins). +3. In-process dispatch into the same handler path as REST; tier enforcement: `sensitive` → + blocking-ask flow; `mutate`/`sensitive` → `recordActivity()` (stub until MC-P3, then + real). +4. Default tool policy wiring for system-scope sessions/staff. +5. Tests: API e2e — meta-tool session creates a project, creates a nested goal in it, sends + a message to another session, enqueues a staff inbox entry; `sensitive` op blocks on ask; + describe returns schemas. + +Acceptance: a Mission Control session performs the §3 catalog end-to-end against a temp +project; no operation outside the allowlist is callable. + +### MC-P3 — Flight recorder + +1. `recordActivity()` + JSONL store (+ rotation) in `src/server/agent/activity-store.ts`; + REST `GET /api/activity` with filters; WS live-tail event (scoped fanout — follow the + goal-fanout lesson from `reduce-server-cpu-experiment-goal-fanout.md`: only explicitly + subscribed sockets). +2. Activity panel UI under the Mission Control sidebar entry (core view; the pack panel in + MC-P4 may embed the same component). +3. Writers: meta-tool (MC-P2 stub flips on), staff wake completion summaries. +4. Tests: unit (rotation, filter math); browser E2E (action appears live in panel; filters; + revert button only when `revert` present). + +Acceptance: every `mutate` meta-tool call from MC-P2's e2e is visible in the panel with +actor attribution. + +### MC-P4 — mission-control pack: system staff + dashboard panel + +1. `market-packs/mission-control/`: roles + skills + standing-order prompts + trigger + templates for the §5 roster; panel + route (`#/ext/mission-control`) following the + pr-walkthrough pack layout; ships via `scripts/copy-builtin-packs.mjs` + + `builtin-packs.ts`. +2. Core "create crew" bootstrap: instantiate staff records from pack templates (one REST + action, idempotent, user-invoked from the panel; Observer trigger enabled, others + disabled per §5). +3. New push triggers `gate_failed` / `session_errored` in `goal-trigger-dispatcher.ts` + (shared dependency with autoimprovement AI-P1 and gap-analysis R2 — land once, here or + there, whichever goal runs first). +4. Tests: pack-litmus tests per the `market-packs/artifacts` convention; e2e — create crew ⇒ + four staff exist under the global scope with correct triggers; Caretaker dry-run sweep + writes a flight-recorder entry; panel deep-link survives reload. + +Acceptance: one click creates the crew; a scheduled Observer tick produces an inbox digest; +everything visible in the activity panel. + +### MC-P5 — Briefing, onboarding, budgets + +1. Observer briefing prompt + digest formatting (inbox entry kind reused); "stuck goals" + sourced from the notification-policy predicate. +2. First-run flow: when no projects are registered, land on Mission Control with the + onboarding prompt (drives the existing add-project proposal flow). +3. Budgets: config keys (per-project/per-staff monthly soft budget), Observer breach + escalation, briefing line item. +4. Tests: browser E2E for first-run path (fresh state dir ⇒ Mission Control onboarding ⇒ + project created via chat proposal); unit for budget math; e2e for breach escalation + entry. + +Acceptance: the §1 "small-business owner" sentence works end-to-end on a clean install. + +--- + +## §10 Open questions (for review, not blockers) + +- Sidebar naming: "Mission Control" vs something shorter ("HQ", "Home"). UX copy decision; + the plan uses Mission Control throughout. +- Should project sessions be able to *see* the activity feed (read-only) via the meta-tool + `read` tier? Lean yes — visibility is king cuts both ways. +- Crew model defaults: system staff should likely default to a cheaper model via role + `model` overrides (`per-role-model-overrides.md`) — decide per-staff during MC-P4 with + token-cost guidance. + +Cross-references: [autoimprovement.md](autoimprovement.md) · +[harness-gap-analysis.md](harness-gap-analysis.md) (G3/G4/G5/G6/G10) · +[client-performance-battery.md](client-performance-battery.md) (Observer watches the perf +baseline once the harness exists) · [extension-platform.md](extension-platform.md) · +[time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) (budgets, aux models) · +`docs/staff-agents.md` · `docs/staff-triggers.md` · `docs/staff-inbox.md` · +`docs/mcp-meta-tools.md` · `docs/rest-api.md`. +Execution tracking: [fable-program-execution-plan.md](fable-program-execution-plan.md). + +--- + +## Appendix A — Implementation contracts (definite lists; do not re-derive) + +The universal definition-of-done in +[extension-platform-implementation-plan.md §0](extension-platform-implementation-plan.md) +(read-before-edit, test-first RED→GREEN, gates, browser E2E, no flake, minimal change) is +binding for every MC phase. This appendix fixes the contracts a weaker implementer would +otherwise have to invent. + +### A.1 Constants and storage layout + +```ts +// project-registry.ts (exists): SYSTEM_PROJECT_ID = "system" +// NEW in project-registry.ts: +export function isSystemScope(projectId: string): boolean; // === SYSTEM_PROJECT_ID +export function missionControlDir(): string; // path.join(stateDir, "mission-control"); mkdir lazily +``` + +State additions under `.bobbit/state/`: +`mission-control/` (cwd workspace for global sessions/staff) · `activity.jsonl` (+ rotated +`activity-YYYY-MM-DD.jsonl`, rotate when current file > 8 MB). + +### A.2 `PersistedStaff` change (MC-P0) + +```ts +// staff-store.ts — add ONE field; loader normalises missing → false: +global?: boolean; // true ⇒ projectId === "system" is legitimate, not an orphan +// Validation matrix (staff-manager.ts): +// global:true ⇒ projectId must be "system"; cwd inside missionControlDir(); worktree +// forced off; sandboxed allowed (plain boolean, unchanged semantics) +// global:false ⇒ existing rules unchanged (real project required) +// GET /api/staff/orphaned: exclude records with global === true. Everything else unchanged. +``` + +### A.3 `bobbit` meta-tool — file layout and operation catalog v1 + +Tool-group anatomy copies `defaults/tools/team/` exactly: one YAML per tool + +`extension.ts`, `provider: { type: bobbit-extension, extension: extension.ts }`, +`group: Bobbit`. Files: `defaults/tools/bobbit/bobbit.yaml`, `bobbit_describe.yaml`, +`extension.ts`. The operation registry lives server-side in a new +`src/server/agent/bobbit-meta-tool.ts`; the extension calls the gateway over the existing +tool-extension HTTP bridge (`tool-guard-extension.ts` pattern) — never spawning a second +HTTP client against localhost REST. + +```ts +export type OpTier = "read" | "mutate" | "sensitive"; +export interface BobbitOp { + op: string; // enum value exposed to the model + tier: OpTier; + method: "GET" | "POST" | "PATCH" | "PUT" | "DELETE"; + path: string; // REST path template, e.g. "/api/projects/:id/goals" + argsSchema: object; // JSON schema; bobbit_describe returns it verbatim + doc: string; // one-liner; bobbit_describe returns it +} +``` + +Catalog v1 (complete; adding an op later = registry row + pinning-test row, nothing else): + +| op | tier | REST | +|---|---|---| +| `projects.list` | read | `GET /api/projects` | +| `projects.preflight` | read | the add-project preflight route (`docs/add-project-preflight.md`) | +| `projects.add` | sensitive | the add-project route used by the UI proposal flow | +| `sessions.list` | read | `GET /api/sessions` | +| `sessions.create` | mutate | `POST /api/sessions` | +| `sessions.send_message` | mutate | the prompt-queue enqueue route (`docs/prompt-queue.md`) | +| `sessions.archive` | sensitive | existing archive route | +| `goals.list` | read | `GET /api/goals` (per project) | +| `goals.tree` | read | the nested-goals tree route (`docs/nested-goals.md`) | +| `goals.create` | mutate | existing goal-create route (supports `parentId` for nesting) | +| `goals.archive` | sensitive | existing archive route | +| `staff.list` | read | `GET /api/staff` | +| `staff.wake` | mutate | existing wake route | +| `staff.inbox_enqueue` | mutate | `POST /api/staff/:id/inbox` | +| `staff.propose` | mutate | drives the staff-proposal flow (returns proposal for human accept — never creates directly) | +| `worktrees.list` | read | worktree-pool route | +| `worktrees.cleanup` | sensitive | existing cleanup route | +| `packs.list` | read | marketplace list route | +| `packs.install` | sensitive | `marketplace-install` route, registered sources only | +| `config.read` | read | config-cascade read route | +| `costs.read` | read | session-cost routes (`docs/session-cost.md`) | +| `activity.read` | read | `GET /api/activity` (MC-P3) | + +Resolve each "existing route" by reading `docs/rest-api.md` + the matching `server.ts` +handler **at implementation time**; the pinning test (`tests/bobbit-meta-tool.test.ts`) +asserts every catalog row dispatches to a registered handler, so a renamed route fails the +test rather than drifting silently. Tier enforcement: `sensitive` ⇒ route through the +blocking-ask flow (`docs/blocking-tools.md`); `mutate`+`sensitive` ⇒ `recordActivity()`. + +### A.4 Flight recorder (MC-P3) + +```ts +// src/server/agent/activity-store.ts +export interface ActivityEntry { + ts: string; // ISO-8601 UTC + actor: { kind: "session" | "staff" | "policy" | "user"; id: string; label?: string }; + action: string; // e.g. "bobbit.goals.create", "improvement.auto-approved" + target?: { kind: string; id: string; projectId?: string }; + tier: "read" | "mutate" | "sensitive"; // reads are recorded only for `sensitive`-adjacent audits — default: don't record reads + evidence?: string; // link/path, e.g. session id or proposal id + revert?: { kind: "improvement" | "api"; ref: string }; +} +export function recordActivity(e: ActivityEntry): void; // append + size-based rotation +// REST: GET /api/activity?since=<iso>&actor=<id>&tier=<t>&projectId=<id>&limit=<n (default 100, max 500)> +// WS: event type "activity_entry", delivered ONLY to sockets that sent +// {type:"subscribe_activity"} (goal-fanout lesson — no broadcast). +``` + +### A.5 mission-control pack layout (MC-P4) + +Copy `market-packs/pr-walkthrough/` structure; built-in band via +`scripts/copy-builtin-packs.mjs` `FIRST_PARTY_PACKS`. + +``` +market-packs/mission-control/ + pack.yaml # name, schema, contents: panels, entrypoints, roles, skills + roles/{caretaker,archivist,improver,observer}.yaml # standing-order-format prompts (§5) + skills/… # crew skills (briefing format, sweep checklists) + staff-templates/crew.yaml # name/role/accessory/triggers per §5 table; consumed by the + # core "create crew" REST action (idempotent by staff name) + src/panel.ts → lib/ # dashboard panel at #/ext/mission-control +``` + +### A.6 Owned files per phase (PR boundaries) + +| Phase | New files | Modified files | +|---|---|---| +| MC-P0 | — | `project-registry.ts`, `staff-store.ts`, `staff-manager.ts`, `server.ts` (orphan route), `docs/staff-agents.md` | +| MC-P1 | `tests/e2e/ui/mission-control.spec.ts` | session-create path (`server.ts`/`session-setup.ts`), `src/app/sidebar.ts`, `src/app/render.ts` (mobile shell) | +| MC-P2 | `defaults/tools/bobbit/*`, `src/server/agent/bobbit-meta-tool.ts`, `tests/bobbit-meta-tool.test.ts` | tool-activation default policy for system scope, `tests/tool-description-budget.test.ts` | +| MC-P3 | `src/server/agent/activity-store.ts`, activity panel module in `src/app/` | `server.ts` (REST+WS), `sidebar.ts` (Activity nav) | +| MC-P4 | `market-packs/mission-control/*` | `copy-builtin-packs.mjs`, `goal-trigger-dispatcher.ts` (new push triggers, if not landed via AI-P1/GA-R2 first — coordinate, land once) | +| MC-P5 | — | Observer role prompt (pack), first-run routing in `src/app/`, config keys + budget check, briefing tests | diff --git a/docs/design/sidebar-goal-nesting-audit.md b/docs/design/sidebar-goal-nesting-audit.md new file mode 100644 index 000000000..0d1f6c2b1 --- /dev/null +++ b/docs/design/sidebar-goal-nesting-audit.md @@ -0,0 +1,440 @@ +# Sidebar & Goal-Nesting Audit — Findings + Ordered Task Backlog + +> **Date:** 2026-06-10 · **Baseline:** master @ `6ec8c8f9` · Companion to the comms-stack +> audit ([comms-stack/04-current-state-and-backlog.md](comms-stack/04-current-state-and-backlog.md)) +> but a distinct surface: the left navigation pane (projects/goals/sessions) and the +> goal/sub-goal nesting model. All file:line anchors verified against the current tree. +> +> **Scope:** race conditions and redraw correctness in the sidebar's data flow; whether +> goal/subgoal nesting is modelled and rendered optimally; reuse/composability defects. +> Each backlog task (§4) is self-contained for hand-off to an implementer agent. +> +> **Status:** audit complete, backlog (T1–T9) not started. Workstream **SN** in +> [fable-program-execution-plan.md](fable-program-execution-plan.md). ⚠️ Seam overlap with +> the battery plan ([client-performance-battery.md](client-performance-battery.md)): +> **T2/T3 and PB-FX5 both rewrite `api.ts` `refreshSessions`/`startSessionPolling`** — land +> T2 (concurrency hardening) first, then FX5 (poll demotion) on top; **T5/T7 reshape the +> sidebar render paths PB-FX7 throttles** — sequence per the execution plan, not ad hoc. + +--- + +## 1. Executive summary + +The sidebar is fed by a **5s REST poll (`refreshSessions`) + ~25 scattered ad-hoc +`refreshSessions()` call sites + WS-pushed point patches**, with **no in-flight coalescing +and no monotonic-generation acceptance guard** — every "X briefly reverts/reappears" +symptom traces to this one hole (D1–D5). The nesting UI maintains **two parallel render +paths** for one parent/child relation: Path A (spawned children rendered inside a +team-lead's expanded block, `render-helpers.ts::renderTeamGroup`) and Path B (the pure +forest, `sidebar-nesting.ts::buildNestedGoalForest`), kept disjoint by a third function +(`computeSpawnedClaim`) whose doc-comment literally promises to "mirror render-helpers +exactly". That mirror discipline has already failed once in the wild (commit `62755c3d`, +"don't render spawned sub-goals at top level") and still contains live disagreement windows +— invisible goals (D10), duplicate rows (D9), three different badge semantics (D6). Mobile +is a third, drifted near-copy with **no forest path at all** (D7). The same goal tree is +independently re-derived in at least **six places** (S2). Test coverage is dominated by +**hand-copied mirror fixtures that have already drifted from source**, and the core forest +builder has **zero direct tests** (S5). + +The structural moves that remove these bug classes by construction: (1) a single sync +engine with epoch discipline, (2) a single memoized goal-graph module, (3) one recursive +tree renderer shared by desktop/mobile/archived — deleting the claim/mirror machinery. + +What is already sound (verified, no action): the 5cfc0306 "Loading… blanking" fix is a real +root-cause fix; rAF render coalescing is respected by the sidebar; expansion/scroll state +survives re-renders; gate-status/PR caches already implement the correct in-flight/token +patterns (the model for T2); goal-dashboard descendant fetches are guarded; unread/read +mark race is defended; cross-tab project reorder is sound. + +--- + +## 2. Confirmed defects + +### D1 — `refreshSessions` has no in-flight dedup and no monotonic-generation guard (root-cause class) +- **Anchors:** `src/app/api.ts:299-498` (no reentry guard; unconditional + `state.gatewaySessions = newSessions` at `:367`; `state.sessionsGeneration = + sessionsData.generation` at `:388-390`; goals merge `:397-443`). Contrast with the guards + the codebase already built elsewhere: `_prRefreshInFlight` (`api.ts:1119-1122`), + gate-status tokens (`api.ts:1084-1115`). +- **Trigger:** any two overlapping invocations — trivially reachable: the 5s poll + (`api.ts:273-280`), `visibilitychange` (`main.ts:986-992`), hashchange route handlers + (`main.ts:293-543`), and the WS `goal_state_changed`/`goal_child_spawned`/`cost_changed`/ + `goal_spec_changed` handlers (`remote-agent.ts:1778-1800`) all call it un-coordinated. A + `goal_child_spawned` burst during plan execution fires N concurrent fetches. +- **Symptom:** last-**arriving** (not last-issued) response wins. A stale snapshot can + (a) regress `sessionsGeneration`, (b) revert a WS-applied status/title for ≤5s + (flickering sprite/unread pulse, spurious notification beep via `_prevSessionStatus`, + `api.ts:351-365`), (c) resurrect removed rows. Self-heals on the next poll tick — unless + the tab is hidden (polling is visibility-gated), in which case stale state persists until + refocus. +- **Severity:** medium (transient wrong UI, spurious beeps; high reachability). +- **Why tests miss it:** `tests/sidebar-loading-flash.test.ts` pins only the initial-load + flag; nothing pins ordering across overlapping fetches. +- **Repro:** stub `gatewayFetch` with two controllable promises; call `refreshSessions()` + twice; resolve the second (newer generation) first, then the first; assert state keeps + the newer generation. Fails today. +- **Fix:** coalesce (return the in-flight promise) + accept a response only if + `generation >= state.sessionsGeneration` (sessions and goals independently), mirroring + the gate-status token pattern. **Non-goal:** moving to WS-pushed lists. + +### D2 — WS point-updates race in-flight poll snapshots (ghost session after `session_removed`) +- **Anchors:** `src/app/session-manager.ts:1396-1412` (`onSessionRemoved` filters the list + immediately), `remote-agent.ts:1951-1960`; resurrect path = D1's `api.ts:367`. Same class + for `updateLocalSessionStatus`/`updateLocalSessionTitle` (`api.ts:251-271`). +- **Trigger:** poll issued at T0; session archived at T1 (WS removes the row); poll + response (snapshot from T0) applies at T2 → the row reappears for ≤5s; clicking it + 404s/reconnects to a dead session. +- **Severity:** medium. **Fix:** subsumed by D1's epoch guard (removal bumps the server + generation, so the stale snapshot is dropped). + +### D3 — Goal archived by an agent/another tab vanishes entirely when "See Archived" is on +- **Anchors:** goals merge rule `api.ts:403-421` — the live payload is authoritative; a + goal newly archived **server-side** is absent from the live payload and not in + `preservedArchived` (the local copy isn't flagged archived) → dropped from `state.goals` + outright. The archived list refreshes only on toggle/search/load-more/initial + (`sidebar.ts:1090-1101`, `api.ts:493-497`); the WS `goal_state_changed` handler only + calls `refreshSessions()` (`remote-agent.ts:1778-1789`), never the archived page. +- **Trigger:** showArchived ON; a team-lead calls `goal_archive_child` (or a second browser + archives the goal). Sidebar: the goal disappears from the live section and does **not** + appear in Archived until the user toggles See-Archived off/on. +- **Severity:** medium-high perceived ("my goal is gone"); no data loss. +- **Fix:** when a previously-live goal id disappears from the live payload while + `state.showArchived && archivedGoalsLoaded()`, trigger a debounced first-page archived + re-fetch; or have the server include `archivedSince` deltas. **Non-goal:** keeping + archived pages fully live. + +### D4 — Duplicate concurrent archived-goals fetches; inconsistent loaded-flag discipline +- **Anchors:** `api.ts:871-879` — `_archivedGoalsLoaded = true` is set **after** the await + (`fetchArchivedSessions` sets its flag **before**, `api.ts:529-537`). + `_ensureArchivedForSearch` (`sidebar.ts:1090-1101`) checks the flag per keystroke → every + keystroke during the in-flight window issues another fetch; each first-page response + **replaces** the archived slice (`api.ts:887-891`), racing a concurrent "Load more" + append (`api.ts:882-886`). +- **Severity:** low-medium (wasted fetches; pagination cursor/content mismatch under fast + typing + load-more). +- **Fix:** set the flag pre-await + in-flight promise dedup, same shape as sessions. + +### D5 — Optimistic archive can flash back to live +- **Anchors:** `eagerMarkArchived` (`api.ts:1425-1434`) then `await refreshSessions()` + (`api.ts:1364, :1391`). An unrelated poll issued pre-DELETE that lands after the awaited + refresh reverts the goal to live for one tick (D1 class). +- **Severity:** low (cosmetic flash). Fixed by D1's epoch guard. + +### D6 — Descendant-count badge has three different meanings (and is often missing) +- **Anchors:** Path B badge = transitive descendant count over the **claim-filtered** + forest input (`sidebar.ts:1377-1384` excludes claimed children before + `buildNestedGoalForest`; count from `sidebar-nesting.ts:133-159`). Path A badge = + **direct non-archived children only** (`render-helpers.ts:862`). Mobile = no badge at all + (`render.ts:549` passes no opts). +- **Symptom:** a parent whose children are all claimed by its team-lead shows **no badge** + (its forest node has zero children) even though docs/nested-goals.md promises one; a + spawned child's badge undercounts grandchildren. +- **Severity:** cosmetic-medium, always visible to subgoal users. +- **Fix:** compute the badge from the **full unfiltered goal graph** (one shared + `descendants(goalId)`), independent of which render path draws the row. (→ T4/T5.) + +### D7 — Mobile has no forest path; divergent near-copy of desktop bucketing +- **Anchors:** `render.ts:461-549`: top-level goals render flat via `renderGoalGroup(goal)` + — no `buildNestedGoalForest`, no indentation, no truncation row, no + archived-into-live-forest folding, no title-suffix pass. Additionally mobile drops goals + without `projectId` (`render.ts:485`) while desktop resolves via the parent-chain walk + (`sidebar.ts:1599-1611`) — a spawned child that inherits its project from its parent + renders on desktop but is **silently missing on mobile**. +- **Trigger:** a child goal not claimed by Path A (parent's lead terminated/purged, or + unstamped) renders as an un-indented top-level row on mobile; a child with inherited + projectId is missing on mobile entirely. +- **Severity:** medium. Commit `62755c3d` fixed only the duplicate-row symptom of this + divergence, not the divergence. (→ T6.) + +### D8 — Role-picker popover renders once per project section +- **Anchors:** `${renderRolePickerDropdown()}` inside the per-project Sessions header loop — + `sidebar.ts:1433` (desktop) and `render.ts:577` (mobile). With N projects and + `state.rolePickerOpen`, N identical `position:fixed` popovers stack. The capture-phase + keyboard handler and `_focusCwdIfNeeded` target + `document.querySelector(".cwd-combobox input")` — the **first** copy in DOM order + (`sidebar.ts:637-639, :786-789`), which can sit *under* the painted copy: focus/caret + land on an occluded element. +- **Severity:** low (state-driven copies stay in sync; caret/focus artifacts + N× DOM and + listeners). **Fix:** render the popover exactly once at the sidebar root (it is + `position:fixed`-anchored anyway). (→ T8.) + +### D9 — Unstamped children render under multiple leads (duplicate rows) +- **Anchors:** `selectSpawnedChildren`'s unstamped branch matches whenever + `leadId === parentLeadId` (`sidebar-spawned-children.ts:63-68`), and both call sites pass + `leadId` as its own `parentLeadId`: the live lead (`render-helpers.ts:1489-1495`) **and + each archived lead** (`render-helpers.ts:1620-1621`, rendered via `renderLeadWithMembers` + `:1627-1650` alongside the live branch). +- **Trigger:** showArchived ON; the goal has a live team-lead **and** ≥1 archived team-lead + (team restarted); a child with `parentGoalId` set but no `spawnedBySessionId` (created + via `createGoal(..., parentGoalId)` from the proposal modal's subgoal prefill, or legacy + data). The child renders under the live lead AND under each archived lead. +- **Severity:** medium — the exact bug class the dedupe E2E was written for, but + `tests/e2e/ui/sidebar-spawned-children-dedupe.spec.ts` only covers stamped same-title + siblings, and `tests/sidebar-spawned-children.test.ts` tests one call at a time (no + global at-most-once assertion). The browser fixture + (`tests/sidebar-spawned-children.html`) is a drifted mirror without the `parentLeadId` + parameter. +- **Fix (tactical):** unstamped children attach to exactly one lead (live if present, else + most-recent archived). **Strategic:** T5 removes the per-lead enumeration entirely. + +### D10 — Claim/Path-A lead-choice disagreement can hide a goal completely +- **Anchors:** `computeSpawnedClaim` picks `liveLeadCandidates[0]` in **input order**; its + own comment concedes the order may disagree with Path A's createdAt sort and asserts the + consequence is harmless (`sidebar-spawned-children.ts:176-188`). It is not: if a stamped + child's `spawnedBySessionId` is lead₂ but Path A picks lead₁ (`render-helpers.ts:1347` + sort + `:1446` find), the claim excludes the child from the forest **and** Path A never + renders it → the goal is invisible in the sidebar. +- **Trigger:** two live `team-lead` sessions for one goal (respawn/recovery) with server + list order ≠ createdAt order. +- **Severity:** medium impact, low-medium reachability. Fixed by construction under T5. + +### D11 — Archived-forest exclusion filter is global, not tree-scoped +- **Anchors:** `render-helpers.ts:887-894` — an archived goal is excluded from the bottom + Archived forest if its `spawnedBySessionId` matches **any** archived team-lead in + `state.archivedSessions`. If that lead's parent goal is purged/not rendered in this + section, the goal renders **nowhere**. Placement is also pagination-dependent: before the + lead's archived page loads the goal shows in the forest; after "Load more" it silently + relocates under the lead. +- **Severity:** low-medium (invisible/jumping archived goals). +- **Fix:** exclude only when the spawning lead will actually be rendered in this project's + archived section. (→ T9, or by construction under T5.) + +### D12 — Detached parent-cycles vanish silently +- **Anchors:** `sidebar-nesting.ts:186-195` — root detection is "no parent or parent + absent"; the `visited` cycle guard (`:135-150`) fires only for cycles reachable from a + root. A detached A↔B cycle yields no root → neither renders, no warning. +- **Severity:** very-low reachability; silent-data-hiding class. **Fix:** after root + collection, emit unvisited goals as degraded roots + `console.warn`. (→ T9.) + +### D13 — Minor confirmed items +- **Auto-expand misses team goals:** `api.ts:429` checks only `s.goalId === g.id`, not + `teamGoalId` — newly discovered team goals don't auto-expand; the dedupe E2E's helper + comment ("expansion timing-sensitive") works around this symptom + (`sidebar-spawned-children-dedupe.spec.ts:26-31`). +- **Render-time fetch staleness:** the on-demand team-agents fetch inside `renderGoalGroup` + (`render-helpers.ts:1375-1403`) can push into `state.archivedSessions` *after* + `clearArchivedSessionsState()` emptied it (toggle-off race) — no staleness token. +- **Unbounded maps:** `_prevSessionStatus` (api.ts), `gateStatusCache`/`prStatusCache` + never pruned for deleted goals; `expandedGoals` localStorage accretes dead ids (only + archived ones are pruned, `state.ts:650-655`). +- **Stale comment claiming a nonexistent optimization:** `computeSpawnedClaim` says "Avoid + an O(parents × sessions) scan: bucket sessions by parent goal id once" + (`sidebar-spawned-children.ts:160-163`) — the code below does exactly the + O(parents × sessions) filter-per-parent it claims to avoid, per project per render frame + (again on mobile). + +--- + +## 3. Structural findings (the defect factories) + +### S1 — Two render paths + a hand-maintained mirror "claim" function for one relation +Path A: `renderGoalGroup → renderTeamGroup → selectSpawnedChildren` +(`render-helpers.ts:1450-1545` + archived-leads branch `:1594-1652`). Path B: +`buildNestedGoalForest → renderNestedNode` (`sidebar.ts:1310-1400`). Disjointness is +enforced by `computeSpawnedClaim`, documented as a mirror of render-helpers. This produced +`62755c3d`, D6, D9, D10, D11 — and two different expansion semantics for the same child +(Path A children hide under `isTeamLeadExpanded(lead)`, Path B children under +`expandedGoals.has(parent)`; the same goal switches regimes when its lead's session is +purged). +**Refactor:** one pure, memoized per-project tree build that assigns every goal exactly one +display anchor: `{ parent: goalId | null, slot: "forest" | { teamLeadId } }`. One recursive +renderer consumes it for live, archived, desktop, mobile, collapsed. `computeSpawnedClaim` +is deleted; the double-render and invisible-goal classes become unrepresentable. +**Migration risk:** medium — placement edge cases (unstamped, archived-lead, pagination); +must land after the pinning tests (T1) and behind the existing E2E suite. + +### S2 — Six independent goal-tree derivations +`buildNestedGoalForest` (sidebar-nesting.ts); `selectSpawnedChildren`/`computeSpawnedClaim` +(sidebar-spawned-children.ts); `countDescendants` (goal-descendants-count.ts) + +`collectGoalIdsFor` (`api.ts:1399-1417`); the `projectIdForGoal` chain-walk +(`sidebar.ts:1599-1611`); `buildChildSummaries` (`goal-dashboard-children-tab.ts:35-57`); +the plan-tab direct-children filter (`goal-dashboard-plan-tab.ts:98-113`). Each +re-implements parent indexing, archived rules, and cycle guards with subtle differences. +**Refactor:** `src/app/goal-graph.ts` — one memoized index (`byId`, `childrenByParent`, +`descendants()`, `directChildren()`, `depthOf()`, `projectOf()` with chain fallback, +cycle-safe), invalidated on `goalsGeneration` change. Removes D7's mobile projectId drop +and the badge inconsistency by construction. + +### S3 — Data-flow: ad-hoc pull, no epoch discipline +`refreshSessions` is the de-facto store update but is called from ~25 sites with no +coordination (D1–D5). Gate-status and PR caches already demonstrate the correct patterns in +this codebase — sessions/goals, the most important lists, are the only ones without them. +**Refactor:** a `requestSync()` façade: coalesce concurrent callers onto one in-flight +promise, debounce WS-event bursts (~100ms), apply responses only when the generation is +monotonic. All call sites switch mechanically. + +### S4 — Desktop/mobile/collapsed are three drifting copies +Project bucketing + section rendering exists 3×: `renderSidebar`/`renderProjectContent` +(`sidebar.ts:1458-1724`), `renderMobileLanding` (`render.ts:461-622`), +`renderCollapsedSidebar` (`sidebar.ts:1730-1887`). Mobile has drifted twice already +(`62755c3d`; D7). Shared pieces (`renderGoalGroup`, `renderProjectArchivedSection`, +`renderStaffSidebarSection`) prove the parameterized pattern works. +**Refactor:** extract `bucketSidebarDataByProject()` (pure, using S2's `projectOf`) + a +variant-parameterized project-section renderer (`"desktop" | "mobile"`). Collapsed stays +bespoke but consumes the same buckets. + +### S5 — Test fidelity: mirrors with confirmed drift; the core builder untested +`tests/sidebar-spawned-children.html` reimplements `selectSpawnedChildren` **without** the +`parentLeadId` fallback and the archived-first sort — it pins a stale version. +`tests/sidebar-hierarchy.html` and `tests/sidebar-goal-rendering.html` self-describe as +mirrors. `buildNestedGoalForest` has **no direct tests at all**. The real renderer is +exercised only by spawned-gateway E2Es. +**Refactor:** the pure modules are already DOM-free with test overrides +(`_setSubgoalsEnabledForTesting`) — pin them with node:test against the **real source**; +retire mirror fixtures or rebuild them on bundled real modules. + +### S6 — Unkeyed list rendering +No `repeat()` anywhere in the sidebar (only MessageList/tab code uses it). All rows are +`.map()` → positional DOM reuse: when a row is inserted/re-sorted (new session, archived +bucket boundary moves), DOM nodes rebind to different entities — keyboard focus and running +CSS animations (`bobbit-unread-pulse`, spinners) silently transfer to a different +session/goal. Expansion state is module-state (survives correctly), so this is the +remaining redraw-correctness gap. +**Refactor:** `repeat(items, x => x.id, render)` for project sections, goal nodes, session +rows. Low risk, mechanical. + +--- + +## 4. Ordered task backlog (hand-off ready) + +Conventions: master stays green (`npm run check`, `npm run test:unit`, `npm run test:e2e`); +test-first (red on master where expressible); no flaky tests; anchors verified at +`6ec8c8f9` — re-verify before editing. + +### T1 — Pin the pure tree builders with real-source tests *(no deps; prerequisite for T5/T6)* +- **Files:** NEW `tests/sidebar-nesting.test.ts`; extend + `tests/sidebar-spawned-children.test.ts`; delete-or-rebuild + `tests/sidebar-spawned-children.html` (drifted mirror: missing `parentLeadId` + the + archived-first sort). +- **Diagnosis:** `buildNestedGoalForest` (the core tree builder) has zero direct tests; the + existing browser fixture pins a stale mirror (S5); nothing asserts global at-most-once + emission of a child across the live-lead + archived-lead branches (D9's gap). +- **Fix:** node:test suites importing the real TS (`_setSubgoalsEnabledForTesting(true)`), + covering: orphan promotion, depth cap + `truncatedChildrenCount`, archived-first sort, + title suffixes (root + nested), cycle stubs, flag-off flattening; plus a cross-branch + test asserting **global at-most-once emission** of an unstamped child across live-lead + + archived-lead `selectSpawnedChildren` calls — this test documents D9 red. +- **Acceptance:** tests run in the unit phase; the D9 case is either fixed inline (tactical + single-anchor rule) or committed as a tracked known-failure pin for T5. + +### T2 — `refreshSessions` concurrency hardening *(no deps)* +- **Files:** `src/app/api.ts:299-498`; NEW `tests/refresh-sessions-race.test.ts` (extract + the apply/merge step into a pure helper, the `session-load-state.ts` pattern). +- **Diagnosis:** D1/D2/D5 — no in-flight coalescing, no monotonic-generation acceptance; + last-arriving stale snapshot clobbers WS-fresh state (ghost rows, status flicker, + spurious beeps). +- **Fix:** module-level in-flight promise returned to concurrent callers; drop session/goal + payloads whose `generation` is below current (independently for sessions and goals); keep + the archived-preserve merge rule. +- **Acceptance:** interleaved-resolution unit test (old response after new → state keeps + new) red→green; `tests/sidebar-loading-flash.test.ts` stays green; no E2E regressions. + +### T3 — Archived freshness + fetch dedup *(after T2)* +- **Files:** `src/app/api.ts:403-421, :871-912`; `src/app/sidebar.ts:1090-1101`. +- **Diagnosis:** D3 (goal archived elsewhere vanishes while showArchived is on), D4 + (loaded-flag set post-await; replace-vs-append race under typing + load-more). +- **Fix:** set `_archivedGoalsLoaded` pre-await + in-flight promise; in the goals merge, + detect ids that disappeared from the live payload while + `showArchived && archivedGoalsLoaded()` and debounce a first-page archived re-fetch. +- **Acceptance:** unit test for the disappearance→refetch trigger; rapid-typing search + issues exactly one archived-goals fetch; NEW E2E: archive a goal via API while + showArchived is on → it appears in the Archived section without toggling. + +### T4 — Single `goal-graph.ts` module *(no hard deps; before T5)* +- **Files:** NEW `src/app/goal-graph.ts`; migrate `goal-descendants-count.ts`, + `api.ts::collectGoalIdsFor`, `sidebar.ts::projectIdForGoal`, + `goal-dashboard-children-tab.ts::buildChildSummaries` candidates, + `goal-dashboard-plan-tab.ts` candidate filter. +- **Diagnosis:** S2 — six divergent derivations of one tree. +- **Fix:** memoized id/parent indices + `descendants()`, `directChildren()`, `projectOf()` + (chain walk), `depthOf()`, all cycle-guarded; invalidate on `state.goalsGeneration`. +- **Acceptance:** behaviour-preserving (existing tests green); new unit tests for the + `projectOf` chain fallback and walk-through-archived descendant counting. + +### T5 — Unify nesting into one render path *(after T1, T4; the big one)* +- **Files:** `src/app/sidebar-nesting.ts` (extend the node model with team-lead display + slots); `src/app/render-helpers.ts` (delete `renderTeamGroup`'s spawned-children + + archived-lead `spawnedSubGoalsOf` blocks; `renderGoalGroup` becomes pure row+sessions); + `src/app/sidebar.ts:1346-1456`; delete `computeSpawnedClaim` from + `src/app/sidebar-spawned-children.ts`; update `src/app/render.ts` mobile exclusion. +- **Diagnosis:** S1/D6/D9/D10/D11 — dual paths + the mirror claim function. +- **Fix:** the forest assigns each goal exactly one display anchor (under the parent's + team-lead slot when a stamped/attributable lead exists, else a plain forest child); the + badge always = transitive descendants from the full graph (T4); one expansion semantic + (child visibility = parent goal expanded AND, when slotted, lead expanded — pick one rule + and pin it). +- **Acceptance:** T1's placement table-tests green (incl. unstamped-child at-most-once); + `tests/e2e/ui/sidebar-spawned-children-dedupe.spec.ts`, `sidebar-child-loading.spec.ts`, + archived-layout specs green; NEW E2E asserting the descendant badge on a parent whose + children are claimed. +- **Dependencies:** T1, T4. Risk: medium (placement edges); rollback = revert to the dual + path (keep the new tests as documentation of intent). + +### T6 — Mobile/desktop reuse *(after T5)* +- **Files:** `src/app/render.ts:461-622`, `src/app/sidebar.ts`, NEW shared + `bucketSidebarDataByProject` helper. +- **Diagnosis:** D7 — mobile flat-renders unclaimed children, silently drops + inherited-project goals, no badges/truncation. +- **Fix:** mobile consumes the same forest renderer with `variant: "mobile"` (the + `renderProjectArchivedSection` pattern). +- **Acceptance:** mobile E2E (pattern: `sidebar-mobile-archived-per-project.spec.ts`) + showing a nested child indented under its parent on mobile AND a child with inherited + projectId visible. + +### T7 — Keyed sidebar lists *(after T5, to avoid churn)* +- **Files:** `src/app/sidebar.ts`, `src/app/render-helpers.ts`, `src/app/render.ts`. +- **Diagnosis:** S6 — positional DOM reuse transfers focus/animations between entities on + insertions/re-sorts. +- **Fix:** `repeat()` keyed by entity id for project sections, goal nodes, session rows. +- **Acceptance:** type-check + existing browser E2Es; a focused row's action button still + targets the same session after a new session is inserted above it (new E2E or fixture). + +### T8 — Single-instance role-picker popover *(independent)* +- **Files:** `src/app/sidebar.ts:1433, :485-615`; `src/app/render.ts:577`. +- **Diagnosis:** D8 — N popovers for N projects; keyboard/focus targets the first DOM copy, + which can be occluded. +- **Fix:** render `renderRolePickerDropdown()` once at the sidebar root; keep per-project + anchor-rect state. +- **Acceptance:** with ≥2 projects and the picker open, exactly one `.cwd-combobox` exists; + keyboard nav focuses the visible input. + +### T9 — Cleanups bundle *(independent, low risk)* +- **Files/items:** `src/app/api.ts:429` (add `teamGoalId` to auto-expand — also lets the + dedupe E2E drop its timing workaround); prune `_prevSessionStatus` + gate/PR caches on + goal/session removal; `src/app/sidebar-nesting.ts` (emit detached-cycle nodes as degraded + roots + warn, D12); `src/app/render-helpers.ts:887-894` (tree-scope the archived-forest + exclusion, D11 — skip if T5 already subsumed it); staleness token for the on-demand + team-agents fetch (`render-helpers.ts:1375-1403`); delete the false "bucket once" comment + in `sidebar-spawned-children.ts:160-163` (moot after T5). +- **Acceptance:** unit tests per item; an archived goal with a purged parent + archived + lead renders in the Archived section. + +**Suggested order:** T1 ‖ T2 → T3 → T4 → T5 → T6 ‖ T7 → T8 ‖ T9. T2/T3 (data-flow) and +T1/T4/T5 (tree model) are independent tracks until T5; everything visual lands after T5 to +avoid double churn. + +--- + +## 5. Checked and sound (no action) + +- **The 5cfc0306 "Loading… blanking" fix is a real root-cause fix**: initial-load keyed off + `sessionsGeneration` in a pure, real-source-tested helper (`session-load-state.ts`, + `tests/sidebar-loading-flash.test.ts`). +- **rAF coalescing** (`state.ts:687-712`) is respected by the sidebar; drag suppression + buffers exactly one render; the streaming bypass (PATH B) is confined to `AgentInterface` + as documented. +- **Expansion/scroll state survives re-renders** (module/state-backed with localStorage) — + poll ticks do not collapse accordions. +- **Gate-status and PR caches** have correct last-writer-wins tokens / in-flight guards + (`api.ts:1084-1161`) — the model T2 copies. +- **Goal-dashboard descendant fetches** are guarded (in-flight flags + + `currentGoalId === goalId` staleness checks, `goal-dashboard.ts:306-327`); no N+1 per + nesting level (single `/descendants` call). +- **Unread/read-state:** `markSessionVisited`'s `_readMirror` max() defends against the + poll racing the mark-read POST (`render-helpers.ts:226-275`); `updateLocalSessionStatus` + deliberately never touches `lastActivity` (pinned by `tests/spurious-idle-unread.spec.ts`). +- **`buildNestedGoalForest` internals** are well-built (Map dedupe, deterministic sort, + per-root cycle guard, orphan promotion, depth-cap truncation) — it lacks tests and a + second-consumer discipline, not quality. +- **Cross-tab project reorder** is sound (optimistic apply + echo + `projects_changed` + broadcast + equality gate). diff --git a/docs/design/time-and-token-cost-efficiency.md b/docs/design/time-and-token-cost-efficiency.md new file mode 100644 index 000000000..124ca44ce --- /dev/null +++ b/docs/design/time-and-token-cost-efficiency.md @@ -0,0 +1,879 @@ +# Time & Token-Cost Efficiency — audit, findings, and remediation plan + +Status: **investigation complete, remediation not started** · Audited: 2026-06-10 against +master @ `4e7fce1f`; **latency axis + architecture root-cause added 2026-06-19** (§9, from a +second 326-session snapshot — shapes corroborate §1). Workstream **CE** in +[fable-program-execution-plan.md](fable-program-execution-plan.md) (program sequencing + +master checklist). +Companion docs: [extension-platform-implementation-plan.md](extension-platform-implementation-plan.md) +(several goals here ride its pack surfaces), [comms-stack/04-current-state-and-backlog.md](comms-stack/04-current-state-and-backlog.md) +(overlapping retry findings). + +**The question this answers:** Bobbit *feels* more expensive **and slower** per unit of work +than Claude Code or hermes-agent. Is it, why, and what do we change? (Cost: §1–§7. Wall-clock +latency and the inherent architectural root cause: **§9**.) + +**The one-paragraph answer:** Prompt caching is working (only 0.008% of input tokens were +uncached across 246 sessions), so the spend is not a caching bug. The spend is +**resident-context size × API-request count × model price**: every tool-call round re-reads the +session's full context at cache-read price, Bobbit's resident context is large (18k–55k tokens +of system prompt before any conversation), tool results accumulate unbounded (no spill/truncation +tier), orchestration multiplies turns (idle nudges, verifier gates, repeated commands), and +every agent — lead, worker, reviewer — runs the same frontier model. Claude Code and Hermes are +cheaper because they cap tool output, spill large results to disk, keep resident prompts lean, +and (Hermes) summarize with a cheap auxiliary model. All of those are adoptable. The plan below +is instrumentation-first: we build the measurement/benchmark harness *before* the +behavior-affecting cuts, so every change is validated on data, not vibes. + +--- + +## §0 Universal rules for executing this plan + +Same rules as the extension-platform plan; restated because goals here will be handed off +independently: + +1. **Test-first.** Every sub-goal lists pinning tests; write them RED before the fix where the + spec says so. No flaky tests — a flaky test is a real bug. +2. **Locate code by symbol name, not line number.** Line anchors below were verified on + master @ `4e7fce1f` and will drift. If a symbol is missing, STOP and re-derive from the + pattern files, don't guess. +3. **Gates.** `npm run check` + `npm run test:unit` for everything; `npm run test:e2e` for + server changes; `npm run test:manual` for session-lifecycle changes. Browser E2E for every + user-facing surface (pattern: `tests/e2e/ui/settings.spec.ts`). +4. **Measurement gate (specific to this plan).** Goals marked **[BENCH-GATED]** change agent + behavior (prompts, models, truncation). They may not merge on code review alone: run the + benchmark suite (CE-G0.3) before and after on the same task set and attach the comparison to + the PR. A regression in task success rate blocks the merge regardless of cost savings. +5. **Environment caveat.** Several SDK-layer findings below were verified against the *stale* + `node_modules` (`@mariozechner/pi-* 0.67.5`) while `package.json` declares + `@earendil-works/pi-* 0.77.0` and the fork's latest is **0.79.1**. CE-G1.0 re-verifies every + SDK claim against the version actually adopted before any SDK-dependent goal starts. + +--- + +## §1 Where the money actually goes (empirical) + +All numbers measured 2026-06-10 from this machine. Reproduction recipes in §7. The owner notes +the codebase moves fast — re-measure before relying on any specific number; the *shapes* are the +durable findings. + +### 1.1 Aggregate (`.bobbit/state/session-costs.json`, 246 sessions) + +| Metric | Value | +|---|---| +| Total recorded spend | **$2,768.90** | +| Uncached input tokens | 284,670 (**0.008%** of input) | +| Cache-read tokens | **3,570,141,959** (3.57B) | +| Cache-write tokens | 125,505,474 | +| Output tokens | 8,097,011 | +| Median session cost | $4.37 | +| Top-10 sessions' share | **$1,420.50 (51%)** | + +Verdict: **caching works**. The bill is dominated by cache *reads* — i.e., how big the resident +context is and how many API requests re-read it — plus output. Cache-read is ~10× cheaper than +fresh input, and we still spent ~$2.7k. Without working caching this history would have cost +five figures; with it, the levers left are context size, request count, and model price. + +### 1.2 The outliers + +| Session | Cost | cacheRead | cacheWrite | Output | What it is | +|---|---|---|---|---|---| +| `0dbda4f4` "Team Lead: Meg Abyte" | **$901.99** | 1,021M | 57.6M | 1.23M | team-lead orchestrator | +| `acce11b7` "Team Lead: Banksy" | $222.26 | 261M | 13.7M | 229k | team-lead orchestrator | +| `llm-review-69d1e3e3` | $46.73 | 77M | 1.0M | 77k | workflow gate verifier | +| `llm-review-0cd4b16b` | $41.14 | 63M | 1.1M | 90k | workflow gate verifier | +| (next 6, all llm-review or team) | $29–40 each | 37–64M | ~1M | 80–103k | — | + +Two shapes account for the tail: + +- **Long-lived team-leads.** The 35KB team-lead role + goal + AGENTS.md + tools is resident for + the session's whole life; every coordination turn (nudge, status steer, gate event) re-reads + it. 1.02B cache-read tokens ≈ thousands of API requests × a six-figure-token context. +- **`llm-review` verifiers** (`verification-harness.ts::runLlmReviewStep`, spawn at + `verification-harness.ts:2542`). One-shot reviewers that re-ingest role + AGENTS.md + goal + spec + the change under review, do tens of tool-call rounds each (each round = full + context re-read), with up to 3 bounded retries — at $29–47 a pop in the tail, these run on + every gated workflow step. + +### 1.3 Resident prompt anatomy (`.bobbit/state/session-prompts/*-prompt.json`) + +Recent sessions in this repo (tokens, from the persisted inspector snapshots): + +| Section | Recent typical | Worst observed | Source | +|---|---|---|---| +| Role | 6,254 (code-reviewer) | **20,180** (older review session) | `defaults/roles/*.yaml` | +| Project AGENTS.md | **9,021** | 17,309 | `readAllAgentFiles()` cascade — *project-dependent*: the 9k example is another project's hand-written playbook; this repo's AGENTS.md is ~1.3k tokens | +| Tools | 5,600 | 6,565 | `toolManager.getToolDocsForPrompt()` | +| System Prompt | 3,855 | 3,855 | `.bobbit/config/system-prompt.md` (15.5KB) overriding `defaults/system-prompt.md` (18.8KB) | +| Goal | 650–2,100 | 7,856 | user-authored goal spec | +| Available Skills | 150 | ~2,000 | budget-capped (`SKILLS_CATALOG_BUDGET` 16KB) | +| Working Directory | 206 | 206 | template | +| **Total resident** | **~18k–26k** | **~55k** | before any conversation history | + +Role files on disk: `team-lead.yaml` **35,491 chars** (~8.9k tokens — 40% of the whole +`defaults/roles/` corpus), `qa-tester` 9.9KB, `docs-writer` 6.7KB, `code-reviewer` 5.6KB. + +Key ratio: **Role + AGENTS.md = 60–68% of resident prompt tokens** in the measured sessions. +The base system prompt everyone assumes is the problem is only ~7%. + +### 1.4 Transcript anatomy (`~/.bobbit/agent/sessions/`, 971MB total; `.bobbit/state` 258MB) + +Three representative sessions: + +| Session | Size | Assistant msgs | Shape | +|---|---|---|---| +| `audit-subg-…` (goal work) | 18.0MB | 1,771 | bash = **56%** of 1,727 tool calls; `read` 233 calls / 747KB returned (max single read 41KB); grep up to 48KB per result | +| `nested-goa-…` (team) | 14.5MB | 2,644 | **30+ idle-agent nudge turns**; `git push … \| tail -3` executed **17× identically**; `npm run check` 9×; team_spawn/task churn | +| `session-_pool-…` (images) | 7.2MB | 19 | 3 `generate_image` results = **5.1MB of base64 (73% of the transcript)** | + +These three patterns — uncapped tool results accumulating in history, coordination turns that +each re-read full context, and multi-MB base64 images living in context — are the per-turn +multiplier on top of the resident-prompt floor. + +### 1.5 The SDK version situation + +| Layer | Version | Note | +|---|---|---| +| `package.json` / lockfile | `@earendil-works/pi-* `**0.77.0** | the fork is already adopted on paper | +| `node_modules` (this machine) | `@mariozechner/pi-*` **0.67.5** | stale install — `src/` still imports `@mariozechner/*` paths in places (`rg "@mariozechner/pi" src/`) | +| Fork latest | **0.79.1** (2026-06-09) | github.com/earendil-works/pi | +| Upstream latest | 0.73.1 (2026-05-07) | effectively superseded by the fork | + +Fork releases since 0.67 ship several fixes we currently pay for not having: + +- **0.76.0** — `excludeFromContext` on the bash RPC command (keep command output out of model + context); `retry.provider.maxRetries` honored with **SDK retries defaulting to 0** (this is + the fix for the double-retry stack — Bobbit's ~1s retry racing the SDK's hidden 2s retry — + catalogued as F5 in the comms-stack audit); better context-overflow detection. +- **0.77.0** — `--exclude-tools` (drop unused built-ins per session); session disposal aborts + in-flight compaction/retry/bash; `streamingBehavior` on input events. +- **0.78.1** — `ctx.getSystemPromptOptions()` for extensions. +- **0.79.0** — **prompt cache-hit rate surfaced** (`CH` in pi's footer — the measurement we + want, already computed SDK-side); compaction summarization prompt fix. +- **0.79.1** — Claude Fable 5 + adaptive thinking/`xhigh`. + +--- + +## §2 The cost model (how to think about every change) + +Per API request: `cost ≈ resident_tokens × p_cacheRead + new_tokens × p_cacheWrite + output_tokens × p_output` + +Per session: `cost ≈ Σ_turns Σ_toolRounds(turn) [above]` — **each tool-call round inside a turn +is its own API request that re-reads everything**. + +So there are exactly four levers, and every goal in §6 maps to one: + +| Lever | Goals | +|---|---| +| **L1 — shrink resident context** (prompt diet, tool-result budgets, image-by-reference, compaction) | CE-G2, CE-G4, CE-G6 | +| **L2 — cut request count** (turn economy: nudges, repeated commands, verifier scoping, discovery efficiency) | CE-G3, CE-G7 | +| **L3 — cheaper price per token** (model/thinking tiering per role/stage) | CE-G5 | +| **L4 — stop paying twice** (retry dedup, SDK upgrade hygiene) | CE-G1 | + +Worked example: an llm-review session with 63M cache-read tokens at ~150k resident context ≈ +~420 API requests. Halving the resident context (L1) *and* running it on a model with ⅓ the +price (L3) is a ~6× cost reduction on that session class with zero reduction in rounds. + +--- + +## §3 Findings (ranked, with evidence and difficulty) + +Severity = estimated share of current spend addressable. Difficulty = engineering risk +including behavior change. Anchors verified on master @ `4e7fce1f`. + +### F1 — No tool-result budget tier (HIGH, difficulty M) +Tool results enter history verbatim and stay there for the session's life; every later request +re-reads them. Largest observed: 48KB grep results, 41KB single reads, 13–51KB bash logs. +`truncateLargeToolContent()` exists (`src/server/agent/truncate-large-content.ts`, referenced at +`session-manager.ts:45`) but is applied opportunistically, not as a systematic policy. Neither +per-result spill-to-disk nor a per-turn aggregate budget exists. Compare Hermes' three-layer +system (§5). *Caveat: verify against pi 0.79.1 — fork may have added caps (CE-G1.0).* + +### F2 — Resident prompt floor is 18k–55k tokens, dominated by Role + AGENTS.md (HIGH, difficulty M–L, behavior-affecting) +§1.3. `team-lead.yaml` alone is 8.9k tokens of always-resident prose, much of it recipe-style +content that could live in skills/docs loaded on demand. The AGENTS.md cascade +(`system-prompt.ts::readAllAgentFiles`, recursive `@ref` resolution, depth 5) has **no token +budget** — a verbose project playbook silently costs its tokens × every request of every +session in that project. + +### F3 — Orchestration turn multiplication (HIGH for team sessions, difficulty M) +The two most expensive sessions ever are team-leads (§1.2). Contributors: idle-agent nudges +(30+ observed in one session; templates in `team-manager.ts` ~1165–1254, debounce exists — +`WORKER_IDLE_NUDGE_DEBOUNCE_MS = 5s`, exponential backoff for lead idle — but each nudge is +still a full turn at full resident context); worker status steers per task transition; gate +inspections (5 × ~13KB observed). The nudge *text* is small (~85 tokens); the **turn it forces +is not** — at a 100k-token resident context, every nudge turn costs ~$0.03–0.15 in cache reads +alone before the model does anything. + +### F4 — Uniform frontier model everywhere (HIGH, difficulty S–M) +Workers, verifiers, and assistants inherit the lead's/session default model and thinking level. +The override *mechanism* exists (`review-model-override.ts::applyReviewModelOverrides`, +`clampReviewThinking` in `verification-harness.ts`; `initialModel` pin in +`session-setup.ts::resolveBridgeOptions`) but there are no tiered **defaults**: nothing says +"llm-review verifiers default to a cheap model" or "coder workers run one tier below the lead". +The owner already wants multi-model pipelines (extension-platform P8/P9 model-selector); this +finding is the static-default version of that. + +### F5 — llm-review economics (MED-HIGH, difficulty M) +$29–47 per review at the tail (§1.2), and reviews run per gated workflow step with up to 3 +bounded retry attempts (`verification-harness.ts:~2550`). Each reviewer re-ingests the full +resident stack even when the reviewed artifact is a small diff. No diff-scoped context, no +cheap-model default, no cap on tool-call rounds. + +**Refinement (2026-06-19): verifier cache economics are *inverted* — write-bound, not +read-bound.** §1.1's headline ("the bill is cache reads") is a global average that does **not** +hold for the `llm-review` class. Measured on the 2026-06-19 snapshot (189 verifier sessions), +cache *writes* are 40–48% of each verifier's (write+read) tokens, vs a 3% global write/read +ratio. Converting to base-input-equivalent units (write ×1.25, read ×0.1, fresh ×1.0): + +| Class | write share | read share | fresh share | +|---|---|---|---| +| Verifiers (`llm-review`, n=189) | **34%** | 51% | 15% | +| Work sessions (n=144) | 23% | 68% | 9% | + +The cause is structural (see §9): each verifier is a **separate, short-lived OS process** that +writes the full resident stack *cold* and dies before the cache write amortises — it can never +inherit the lead's warm cache. **Remediation implication:** for verifiers the dominant lever is +cold-**write** bytes (shrink the resident context per spawn — CE-G8.2) and/or warm-cache reuse +(CE-G8.4), **not** request-count reduction as CE-G3.3 frames it. Re-scope CE-G3.3 accordingly. + +### F6 — Images as base64 in context (MED overall, EXTREME per-incident, difficulty S) +One `generate_image` result = 2.4–2.7MB of base64 in the transcript and in context; 73% of a +$10.75 session was image bytes. Results should be written to disk and returned by reference +(path + dimensions + thumbnail for the UI), never inlined into model context. + +### F7 — Repeated identical commands (MED, difficulty S) +`git push … | tail -3` ×17, `npm run check` ×9 in single sessions. Each repeat is a full +request round. No guard, hint, or memoization exists. + +### F8 — Stale SDK + double retry billing (MED, difficulty S) +§1.5. Running 0.67.5 forfeits `excludeFromContext`, `--exclude-tools`, retry control, and +cache-hit telemetry. The SDK's hidden default-on retry (~2s) underneath Bobbit's own auto-retry +(`session-manager.ts:2594–2656`) can double-bill failed turns — fixed by fork 0.76.0's +`retry.provider.maxRetries` honoring, **if we upgrade and set it**. + +### F9 — No per-turn cost instrumentation (BLOCKER for everything else, difficulty S–M) +`session-costs.json` (`cost-tracker.ts`, recorded from `message_end` usage at +`session-manager.ts:~3103`) is **cumulative per session only**. No per-turn deltas, no cache-hit +ratio surfaced in Bobbit's UI, no per-goal/per-project rollup, no compaction-cost attribution. +We cannot currently answer "what did this nudge/turn/verifier cost" without jq archaeology — +which is why this audit needed one. + +### F10 — Discovery inefficiency: bash is 54–56% of all tool calls (MED, difficulty S–M) +Agents shell out for grep/find/cat instead of using capped tools with offsets; whole-file reads +of 41KB; no "grep -l first, then targeted read with offset/limit" discipline in prompts or +skills. Each inefficient discovery step is both a bigger result (L1) and often an extra round +(L2). pi's built-ins do include `grep`/`find`/`read` — the gap is output caps + guidance, not +tool existence. *(pi's `edit` is also reported as whole-file-rewrite at 0.67.5 — re-verify on +0.79.1; if still true it inflates output tokens on large files.)* + +### F11 — Compaction is a cost black box (LOW-MED, difficulty S) +Compaction summarization calls aren't attributed in cost tracking, and Bobbit doesn't configure +a proactive policy or a cheap summarizer model (pi's `compaction-sidecar` defaults apply). + +### F12 — Storage bloat (cosmetic, difficulty S) +112MB of `session-prompts/` snapshots and 971MB of transcripts — not an API-cost issue, but +prompt snapshots duplicate identical AGENTS.md/role content per session; worth content-hashing +during G0 work since the same code is being touched. + +**Verified non-problems** (don't spend effort here): prompt caching (works, §1.1); per-turn +system-prompt mutation (none — written once at spawn, `--system-prompt` path, cache survives); +skills catalog (150 tokens, budget-capped); tool description budgets (pinned by +`tests/tool-description-budget.test.ts`); goal-tree serialization (only nesting stanzas ≤1.5k); +MCP tool docs (off-wire pointers already). + +--- + +## §4 What Claude Code and Hermes do that we don't + +From source study of `/Users/aj/Documents/dev/claude-code/src` and +`/Users/aj/Documents/dev/hermes-agent` (local checkouts, 2026-06-10): + +| Technique | Claude Code | Hermes | Bobbit today | Adopting goal | +|---|---|---|---|---| +| Stable cache prefix w/ explicit static/dynamic boundary | `SYSTEM_PROMPT_DYNAMIC_BOUNDARY`, `cacheBreak` flags, system-reminders injected user-side | system prompt memoized; ephemeral prompt injected at API-time, outside cache | ✅ already good (prefix stable) | — | +| Read caps + offsets | 25k-token read cap, 256KB pre-read stat check, offset/limit params | `read_file` unbounded but spills | partial (no enforced cap) | CE-G2.1 | +| **Tool-result spill-to-disk** | success/error only for write/edit | **3-layer budget: per-tool cap → >100KB results persisted to disk w/ 1.5KB preview + path → 200KB per-turn aggregate spill** | ❌ | CE-G2.1 | +| Deferred tool loading | `defer_loading` + ToolSearch when tool docs >10% of context | ❌ (67 tools inline) | partial (MCP docs off-wire) | CE-G1.2 (`--exclude-tools`) | +| **Cheap auxiliary model for summarization** | ❌ (manual /compact) | **auto-compress at 50% context via cheap model, 20% summary ratio, last-20-messages protected, tool-result pruning pre-pass** | ❌ | CE-G6.1 | +| Lazy skills | partial | `skill_list`/`skill_view` split — full skill body loaded only on use | ✅ catalog is names+descriptions | — | +| Sub-agent economics | minimal task prompt, summary-only return | parent context compressed before sub-agent call | ❌ full resident stack per sub-agent | CE-G3.3, CE-G5 | +| File-read dedup | `fileStateCache` (path+mtime) | — | ❌ | CE-G7.1 (guidance) / later | +| Image budgeting | downsampling + flat 1,600-token estimate | flat per-image budget | ❌ base64 inline | CE-G2.2 | +| Cost attribution | per-request workload headers, cost tracker | per-call usage accumulation | per-session cumulative only | CE-G0.1 | + +The headline gap is **Hermes' three-layer tool-result budget** and **auxiliary-model +compression** on the L1 lever, and Claude Code's **read caps/deferral** on L1/L2. Nothing +either tool does on caching is missing in Bobbit — confirming §1.1. + +--- + +## §5 Design positions (settled — do not re-litigate during execution) + +1. **Instrumentation before surgery.** CE-G0 lands first. Behavior-affecting goals are + BENCH-GATED (§0 rule 4). The owner explicitly wants data-driven before/after validation, + "not guessing". +2. **The benchmark harness is a product feature, not a dev script.** It ships as a first-class + Bobbit capability ("run this task suite, compare cost/outcome between two configurations"), + designed so the run-suite/report surfaces can later be packaged as an extension-platform + pack (panel + provider) — but the core runner lives in the server like the verification + harness does. This is the owner's "test/validation harness as an extension would be a + massive win". +3. **Adopt the pi fork at latest (0.79.x), don't patch blind.** `@earendil-works/pi` is our + patchable fork already declared in package.json; first move is *upgrade + re-verify*, and + only then patch the fork where gaps remain (tool-result caps, edit semantics) — upstreaming + into the fork rather than working around it in Bobbit when the fix is SDK-shaped. +4. **Spill, don't destroy.** Truncation policies always preserve the full artifact on disk with + a preview + path in context, so the agent can re-read on demand. Never silently drop + information (mirrors the "no silent caps" culture). +5. **Tiering is defaults, not hard-coding.** Model/thinking tiers live in role/project config + with sane defaults; the extension-platform model-selector (P8/P9) later makes them dynamic. + This plan must not conflict with that — it defines the *static* layer the selector overrides. +6. **Real measured numbers stay in this doc** with their date and machine context; re-measure + rather than trust them after substantial changes. + +--- + +## §6 Remediation plan — goals + +Hand-off format matches `extension-platform-implementation-plan.md`: each sub-goal is +independently mergeable, master-green, with tests and acceptance criteria. Lanes: G0 and G1 are +parallel day-one lanes; G2/G3 depend on G1.0's re-verification; G4/G5 are BENCH-GATED on G0.3. + +**Latency workstream (CE-G8)** lives in §9.5 — it targets *wall-clock*, not just dollars +(risk-proportional gates, slim sub-agent context, warm-cache reuse, latency instrumentation, +"solo fast" mode). CE-G8.1/G8.3 are the quick wins and can start day-one alongside G0/G1. + +**Conflict hotspots:** `session-manager.ts` and `session-setup.ts` are also touched by the +comms-stack backlog (CS-R*) — coordinate merge order; `verification-harness.ts` is touched by +CE-G3.3 and CE-G5.2 (sequence them). + +--- + +### CE-G0 — Instrumentation & benchmark harness *(lane A — start immediately)* + +**Outcome:** every token and dollar is attributable (per turn, per session, per goal, per +project), visible in the UI, and comparable across runs of the same task. + +#### CE-G0.1 Per-turn usage ledger (S) +- On every `message_end` with usage (existing hook: `trackCostFromEvent`, + `session-manager.ts:~3103`), append a record to + `.bobbit/state/session-usage/<sessionId>.jsonl`: + `{ts, turnIndex, model, inputTokens, outputTokens, cacheReadTokens, cacheWriteTokens, costUsd, contextEstimate, trigger}` — + `trigger` distinguishes `user-prompt | steer | nudge | auto-retry | verification | compaction` + (plumb from the enqueue site; nudges/steers already flow through known call sites in + `team-manager.ts` / `session-manager.ts::enqueuePrompt`). +- Attribute compaction summarization usage to the session (closes F11's blind spot). +- REST: `GET /api/sessions/:id/usage` (paginated), `GET /api/projects/:id/usage/rollup` + (per-goal, per-role, per-trigger aggregates). +- **Tests:** unit — ledger append/rotation, trigger attribution table-driven; API E2E — rollup + math over a fixture ledger. Pin: cumulative ledger total equals `session-costs.json` entry + (±1 token) for a replayed fixture. + +#### CE-G0.2 Cost lens UI (M) +- Session view: per-turn cost strip (cost, cache-hit %, trigger badge), session total with + cacheR/cacheW/output breakdown. Goal/project view: rollup table (cost by role, by trigger + class — "coordination" vs "work" vs "verification"), top-N expensive sessions. +- Surface the SDK's cache-hit rate once on ≥0.79 (CE-G1.1) instead of recomputing. +- **Tests:** browser E2E (navigation, rendering from fixture ledger, persistence across + reload). UI-only → `test:unit` + the E2E. + +#### CE-G0.3 Benchmark harness (L) — the BENCH gate +- New server module (pattern: `verification-harness.ts`): a **bench suite** is a YAML file of + repeatable tasks — `{id, fixtureRepo (tests/fixtures or a pinned git ref), goalSpec, role, + successCheck (command or llm-review rubric), maxCost, maxTurns}`. +- `POST /api/bench/run {suiteId, label, overrides}` runs each task as a normal (sandboxed, + worktree-isolated) session, records outcome + the CE-G0.1 ledger, and produces a run report: + `{task, success, costUsd, turns, toolCalls, wallClock, residentPromptTokens}`. +- `GET /api/bench/compare?a=<runId>&b=<runId>` → per-task and aggregate deltas; this comparison + JSON (or its rendered table) is what BENCH-GATED PRs attach. +- CI never runs real-LLM benches (`tests/manual-integration/` is the only gate-exempt path); + unit/E2E cover the runner with a stubbed agent. +- Seed suite `bench/suites/core-dev.yaml`: (1) scoped bugfix in a fixture repo, (2) small + feature + tests, (3) code-review of a prepared diff, (4) a team goal with 2 workers — + deliberately covering worker, reviewer, and orchestration cost shapes. +- **Extension hook (deliberate):** the report store and compare endpoint are plain REST + files + so a future pack (extension-platform P1 surfaces: panel + entrypoint) can ship "Cost Lab" UI + without core changes. Record this seam in the code. +- **Tests:** unit — suite parsing, report math, compare; API E2E — stubbed-agent run end to + end; manual-integration — one real run of the smallest task. + +--- + +### CE-G1 — SDK adoption & config hygiene *(lane B — start immediately)* + +#### CE-G1.0 Upgrade to `@earendil-works/pi-*` latest 0.79.x and re-verify claims (M) +- `npm install` against latest fork; migrate any remaining `@mariozechner/pi*` imports in `src/` + (`rg "@mariozechner/pi" src/` — UI and server hits listed in the audit); fix breakages; + full gate run incl. `test:manual`. +- **Re-verify and document** (append a dated note to this doc's §1.5): does 0.79.x truncate + tool results? what are `read`/`grep`/`bash` output caps? is `edit` still whole-file-rewrite? + what does compaction default to? This report re-scopes CE-G2/G6/G7 before they start. +- Set `retry.provider.maxRetries` explicitly and align with Bobbit's auto-retry so exactly one + layer retries (cross-ref comms-stack F5/CS backlog — same fix, coordinate). +- **Tests:** existing suites are the regression net; add a pinning test that Bobbit's spawn + config sets the retry setting (no silent SDK-default regression on future upgrades). + +#### CE-G1.1 Cache-hit telemetry adoption (S, after G1.0) +- Ingest the SDK's cache-hit rate / usage detail into the CE-G0.1 ledger instead of deriving it. + +#### CE-G1.2 Context-exclusion adoption (S–M, after G1.0) +- Use `excludeFromContext` for every Bobbit-initiated bash whose output the model doesn't need + (health probes, bookkeeping, the verification harness's own setup commands). +- Use `--exclude-tools` from role tool policy so sessions don't carry built-ins their role + forbids (today the guard extension denies at call-time but the definitions still ship). +- **Tests:** unit — spawn args include exclusions per role fixture; E2E — excluded tool absent + from the session's advertised tools. + +--- + +### CE-G2 — Tool-output budget tier *(L1; after CE-G1.0 re-verification)* + +#### CE-G2.1 Three-layer result budget with spill-to-disk (M–L) +Adopt Hermes' model, scoped to whatever 0.79.x doesn't already do: +- **Layer 1** per-tool caps (defaults, config-overridable): bash 20KB (head+tail split), grep + 20KB, read 50KB (with offset/limit guidance — see CE-G7.1), default 32KB. +- **Layer 2** any result over its cap → write full output to + `.bobbit/state/tool-spill/<sessionId>/<n>.txt`, return preview (first ~1.5KB to a line + boundary) + `[full output: <path>, <N> bytes — read with offset/limit if needed]`. +- **Layer 3** per-turn aggregate budget (~200KB): if a turn's results exceed it, spill the + largest non-spilled results. +- Implementation seam: prefer the SDK layer (fork patch — it owns tool execution) with a + Bobbit-side fallback in the tool wrappers; decide per CE-G1.0's report. Spill files are + session-scoped and cleaned with session deletion. +- **Hermes' exact constants** (source: `hermes-agent/tools/budget_config.py`, verified + 2026-06-19) — adopt as starting defaults: `DEFAULT_RESULT_SIZE_CHARS = 100_000` (per-result), + `DEFAULT_TURN_BUDGET_CHARS = 200_000` (per-turn aggregate), `DEFAULT_PREVIEW_SIZE_CHARS = 1_500` + (inline snippet after spill). Resolution order: `pinned > tool_overrides > registry per-tool > + default`. **Non-obvious gotcha to copy:** Hermes *pins* `read_file = inf` in `PINNED_THRESHOLDS` + — a read tool that itself spills creates an infinite persist→read→persist loop, so the read path + must be exempt from the budget (caps belong on `read` via offset/limit guidance — CE-G7.1 — not + via spill). +- **Tests (write RED first):** unit — cap/preview/spill matrix per layer, path cleanup; E2E — + oversized fixture command produces preview + readable spill file; pin that previews always + carry the path (no silent truncation). + +#### CE-G2.2 Images by reference (S) +- Any tool producing images (incl. `generate_image`) writes to disk and returns + `{path, width, height, bytes}` + a one-line description in model context; the UI renderer + loads from the path. Transcript stores the reference, not the base64. +- **Tests:** unit — renderer + reducer handle reference form; E2E — image tool round-trip + shows the image in UI with context payload under 1KB. Expected effect: the §1.4 image + session's shape ($10.75, 73% image bytes) becomes impossible. + +--- + +### CE-G3 — Turn economy *(L2)* + +#### CE-G3.1 Nudge & status batching (M) +- Coalesce per-agent status changes into one steer per quiet window (extend the existing 5s + debounce to batch *content*, not just delay it); suppress idle nudges that carry no new + actionable delta since the last one (track last-nudge state hash); keep the existing + exponential backoff. +- Add `trigger` attribution (CE-G0.1) so the coordination share of team-session cost is a + dashboard number; target: coordination turns <15% of team-session spend. +- **Tests:** unit — batching/suppression state machine (table-driven); E2E — fixture team + session emits N status events → ≤1 steer per window. + +#### CE-G3.2 Repeated-command guard (S) +- In the tool-guard path (`tool-guard-extension.ts` pattern — it already sees every + `tool_call`), detect an identical read-only bash command (allowlisted prefixes: `git status`, + `npm run check`, build/test commands) repeated within the same turn-window with no + intervening file changes, and prepend a short note to the result: + `[identical command run <N> ago — output unchanged? consider reusing]` — informational, never + blocking (the 17× `git push` case had legitimate retries in it; we hint, not forbid). +- **Tests:** unit — detection window logic; fixture E2E — note appears on 3rd identical call. + +#### CE-G3.3 llm-review scoping (M) **[BENCH-GATED]** +- Verifier sessions get a *review-scoped* context: the diff/artifact under review + acceptance + criteria + a slim reviewer role — not the full AGENTS.md cascade by default (config flag to + restore it for projects whose review genuinely needs it); cap tool-call rounds per attempt + (default ~20, configurable per gate) with an explicit "out of budget → fail with reason" + result instead of unbounded exploration. +- **Tests:** unit — context assembly for review sessions; E2E — round cap produces the + structured failure; BENCH — review-task success rate unchanged on the bench suite's review + task while cost drops. + +--- + +### CE-G4 — Prompt diet *(L1; BENCH-GATED; the behavior-affecting tier)* + +#### CE-G4.1 team-lead role rewrite: 35.5KB → ≤12KB (M–L, **highest-risk/highest-value**) **[BENCH-GATED]** +- Restructure `defaults/roles/team-lead.yaml`: keep identity, decision rules, and invariants + resident; move recipes/walkthroughs (the bulk) into skills or `docs/` loaded on demand via + the existing skills mechanism. Mechanical rule: anything the lead needs *every* turn stays; + anything needed in specific situations becomes a discoverable skill. +- Run the bench team-task before/after (N=3 runs each, same model); ship only if success and + coordination-quality metrics hold. +- **Tests:** pin a role-size budget test (like the tool-description budget) so it can't silently + regrow: `defaults/roles/*.yaml` each ≤ a per-role budget, sum ≤ corpus budget. + +#### CE-G4.2 AGENTS.md cascade budget (M) **[BENCH-GATED]** +- Token-budget `readAllAgentFiles()` output (default ~6k tokens, project-overridable): + priority order (project AGENTS.md > global > referenced docs), truncate at section boundaries + with an explicit `[truncated — full file at <path>; read on demand]` tail (spill rule, §5.4). +- Surface the measured size in prompt-sections (already visible) and warn in the UI when a + project's cascade exceeds budget — make the cost of a 17k-token playbook *visible* to its + author instead of silent. +- **Tests:** unit — budget/truncation boundary cases; pin the truncation-marker invariant. + +#### CE-G4.3 Role corpus diet + lazy role docs (S–M) **[BENCH-GATED]** +- Apply the G4.1 treatment to `qa-tester` (9.9KB) and `docs-writer` (6.7KB); add the role-size + budget test for all roles (delivered in G4.1). + +--- + +### CE-G5 — Model & thinking tiering *(L3)* + +#### CE-G5.1 Per-role model/thinking defaults (M) **[BENCH-GATED]** +- Extend role schema (role files + `role-store.ts`) with optional `model:` and `thinking:`; + precedence: explicit session pin > role default > project default > server default. + Resolution lives in `session-setup.ts::resolveBridgeOptions` (the pin mechanism exists; + this adds the role/project layers). +- Ship conservative defaults: leads = project default (frontier); workers = one tier below + unless role says otherwise; assistants = mid-tier. Show the resolved model per session in the + UI (it's a cost-relevant fact users should see). +- Forward-compat: this is the static layer the extension-platform P8 model-selector proposes + *overrides* to — keep the resolution function pure and injectable. +- **Tests:** unit — precedence matrix (table-driven); E2E — role fixture resolves expected + model in spawn args; BENCH — worker-task success holds on the cheaper tier. + +#### CE-G5.2 llm-review default model (S) **[BENCH-GATED]** +- Default verifier model = configurable `verification.reviewModel`, shipped as a mid-tier model; + wire through the existing `applyReviewModelOverrides` path. With CE-G3.3, targets the + $29–47 review tail directly. +- **Tests:** unit — override precedence incl. the new default; BENCH — review task quality holds. + +--- + +### CE-G6 — Compaction & history hygiene *(L1)* + +#### CE-G6.1 Proactive compaction policy + cost attribution (M) +- Configure pi's compaction proactively (threshold ~60–70% of context window — confirm exact + knobs in CE-G1.0's report) with a **cheap summarizer model** if the SDK supports a separate + compaction model (if not: fork patch, Hermes-style auxiliary model). +- Attribute compaction usage in the CE-G0.1 ledger (`trigger: compaction`). +- Coordinate with comms-stack F6 (compaction dispatch gate) — same code region, land the safety + fix first. +- **Tests:** manual-integration — long fixture session compacts proactively, session continues + correctly, ledger shows attributed compaction cost. + +--- + +### CE-G7 — Discovery efficiency *(L2; cheap, do early alongside G2)* + +#### CE-G7.1 Discovery guidance + skills (S) +- Add a concise (≤600 token) "efficient code discovery" section to the base system prompt / + relevant roles: grep with `-l` first → targeted read with offset/limit → never `cat` whole + files via bash; prefer built-in capped tools over bash for search; don't re-read unchanged + files. Ship a `code-discovery` skill with the longer playbook (lazy-loaded). +- Measure: bash share of tool calls (CE-G0.1 ledger dimension) — target <40% from 56%. +- **Tests:** prompt-section snapshot test; BENCH comparison is the real validation (bundle with + a G2 bench run to share runs). + +#### CE-G7.2 (Later, recorded) Code-graph/LSP pack +- AST/LSP-backed "find usages / go to definition" as an extension-platform pack — already in + that plan's Later section; cost rationale recorded here: replaces multi-round grep cascades + with single tool calls. + +--- + +### Expected impact (estimates — the bench harness replaces these with measurements) + +| Goal | Lever | Est. effect on affected session class | +|---|---|---| +| CE-G2 | L1 | 10–30% (work sessions; eliminates image-session blowups entirely) | +| CE-G3 | L2 | 15–30% of team/verifier session cost | +| CE-G4 | L1 | 20–40% of resident-context cost (≈ proportional cut in cache-reads) | +| CE-G5 | L3 | 30–60% on worker/verifier spend (price-sheet arithmetic) | +| CE-G1 | L4 | small steady-state; prevents double-billed retries; unlocks G2/G6 | +| CE-G6/G7 | L1/L2 | 5–15% each, mostly on long sessions | + +These do not multiply cleanly (overlapping bases), but a realistic compounded outcome on the +expensive session classes (team-leads, llm-reviews — 51%+ of spend) is **2–4×** cheaper at +equal task success, which is the gap the owner perceives vs Claude Code/Hermes. + +--- + +## §7 Reproduction recipes + +```bash +# Aggregate + cache ratio +jq -r '[.[] | {i:(.inputTokens//0), cr:(.cacheReadTokens//0), cw:(.cacheWriteTokens//0), o:(.outputTokens//0), c:(.totalCost//0)}] | + "sessions=\(length) uncachedIn=\([.[].i]|add) cacheRead=\([.[].cr]|add) cacheWrite=\([.[].cw]|add) out=\([.[].o]|add) totalCost=$\([.[].c]|add)"' \ + .bobbit/state/session-costs.json + +# Top sessions by cost +jq -r 'to_entries | sort_by(-(.value.totalCost//0)) | .[0:10][] | + "\(.key)\t$\(.value.totalCost)\tcacheR=\(.value.cacheReadTokens)\tout=\(.value.outputTokens)"' \ + .bobbit/state/session-costs.json + +# Prompt-section sizes for recent sessions +for f in $(ls -t .bobbit/state/session-prompts/*-prompt.json | head -5); do + jq -r '[.sections[] | "\(.label): \(.tokens)"] | join("\n")' "$f"; echo ---; done + +# Tool-call distribution in a transcript +jq -s '[.[] | select(.type=="message" and .message.role=="assistant") | .message.content[]? | + select(.type=="toolCall") | .name] | group_by(.) | map({tool:.[0],n:length}) | sort_by(-.n)' \ + ~/.bobbit/agent/sessions/<file>.jsonl + +# Repeated identical bash commands +jq -s '[.[] | select(.type=="message" and .message.role=="assistant") | .message.content[]? | + select(.type=="toolCall" and .name=="bash") | .arguments.command] | group_by(.) | + map({cmd:.[0],reps:length}) | map(select(.reps>1)) | sort_by(-.reps)' \ + ~/.bobbit/agent/sessions/<file>.jsonl + +# Largest tool results +jq -s '[.[] | select(.type=="message" and .message.role=="toolResult") | + {tool:.message.toolName, bytes:(.message.content|tostring|length)}] | sort_by(-.bytes) | .[0:20]' \ + ~/.bobbit/agent/sessions/<file>.jsonl +``` + +## §8 Risks & open questions + +1. **Prompt-diet regressions** (G4) — the team-lead role encodes hard-won behavior; the bench + gate + role-budget pin are the mitigations, but the bench suite must include a real team + task or G4.1 is being validated on the wrong distribution. +2. **Truncation hiding needed info** (G2) — mitigated by spill+preview+path and the read-on- + demand loop; watch for agents that never learn to follow the path (add guidance in G7.1). +3. **Fork divergence** — patching `earendil-works/pi` (G2 SDK-side, G6 auxiliary model) carries + maintenance cost; prefer upstreamable patches with tests in the fork repo. +4. **Bench cost & flakiness** — real-LLM benches cost real money and have run-to-run variance; + N=3 minimum, compare medians, and keep suites small. Bench runs are manual-only, never CI. +5. **Cheaper-model failure modes** (G5) — a worker that fails and retries on a cheap model can + cost more than succeeding once on a frontier model; the ledger's per-trigger attribution + (retry share by model) is the early-warning metric. +6. **Data drift** — every number in §1 is a 2026-06-10 snapshot of one machine spanning + multiple projects and weeks of model-price changes; CE-G0 exists so this document never + needs to be hand-reproduced again. + +--- + +## §9 Latency axis & the process-per-agent root cause *(added 2026-06-19)* + +§1–§8 optimise **dollars**. The owner's lived complaint is also **wall-clock**: "goals take a +*really* long time; Claude Code felt a lot faster." Cost and time are correlated but not the +same — a $1 verifier still makes you wait three minutes — so latency needs its own axis, and the +investigation surfaced an **inherent architectural root cause** that several §3 symptoms +(F2 resident floor, F5 verifier economics) trace back to. All numbers here are a 2026-06-19 +snapshot of `.bobbit/state` (326 sessions, $1,069, 1.16B cache-read) on the primary dev machine; +the *shapes* corroborate §1's 2026-06-10 snapshot. Repro recipes in §9.7. + +### 9.1 The surprise: per-request latency is *not* where Bobbit loses + +| Metric | Bobbit | Claude Code | Source | +|---|---|---|---| +| Context tokens per API request (median) | ~18k–55k resident + history | **377,590** | CC is *larger* and still fast | +| Per-API-round latency (median / mean / p90) | comparable per round | **4.4s / 15.1s / 26.5s** | CC `~/.claude/projects/*.jsonl` timestamps | + +Claude Code runs **bigger** contexts per request and still turns each round in ~4.4s median. So +Bobbit is **not** slow because individual requests are heavy. The wall-clock goes somewhere +structural — the orchestration model inserts machine work *between you and "done"* that, in +Claude Code, either doesn't happen or is you doing it instantly in real time. + +### 9.2 Where the wall-clock actually goes (measured) + +| Signal | Value | Why it's wall-clock you wait through | +|---|---|---| +| Verifier (`llm-review`) sessions vs work sessions | **189 vs 141 — 1.34 per unit of work** | Each gated step spawns a verifier that runs *after* the work, on the critical path | +| Verifier session duration | median **2.8 min**, max **9.9 min** (10h total, 175 traced) | `review-findings` blocks `team_complete`; serialized, not overlapped with work | +| Work session duration | median **20.5 min** | multi-turn, multi-tool | +| Per-spawn startup floor | process boot + N MCP stdio spawns + optional `docker exec` + role/AGENTS/tool-doc load | paid *before the first token*, every spawn | +| Nudge/backoff dead time | 5s worker-idle debounce, 5-min stuck-quiet, 30-min long-streaming suppression, exp backoff (cap 12h) | literal inserted sleeps in the coordination loop (`team-manager.ts`) | + +Claude Code has none of these: you are the orchestrator and reviewer, in real time, so there is +zero machine wall-clock between turns. **The slowness is the price of unattended autonomy + +automated quality gates** — the fix is making them *proportional to risk*, not removing them. + +### 9.3 Root cause: process-per-agent (inherent to the gateway architecture) + +Bobbit's deepest inherent tax is that **every agent — lead, worker, and each of the 189 +verifiers — is a separate OS process.** `rpc-bridge.ts` does +`spawn(process.execPath, [pi-cli, ...])` per session; each session connects its own **MCP stdio +child processes** (`mcp-client.ts::_connectStdio`); sandboxed goals add a `docker exec` into a +pool container on top (`rpc-bridge.ts:~658`). + +Claude Code does the opposite — sub-agents are **in-process forks** (`tools/AgentTool/runAgent.ts` +is an async generator; `utils/swarm/spawnInProcess.js`) that **inherit the parent's already-warm +context** via `forkContextMessages`, and it *deliberately* trims per-spawn context: the +read-only Explore/Plan agents drop CLAUDE.md — source comment: *"Dropping claudeMd here saves +~5–15 Gtok/week across 34M+ Explore spawns"* (`runAgent.ts:~387`). Hermes likewise runs tools +in-process against one resident context. + +This single choice explains the symptoms: +- **Verifier write-thrash (F5 refinement).** A separate process has its own prompt-cache + lifecycle, so it *cannot* inherit the lead's warm cache — it writes the full resident stack + **cold** and dies before the write amortises. Hence 34% of verifier cost is cache *writes* + (vs 3% global). +- **The per-agent latency floor (9.2).** Process boot + MCP stdio handshakes + optional + container exec + role/AGENTS/tool-doc load happen before the model emits a token, paid 189× + for verifiers alone. +- **Why the resident-prompt floor (F2) is so expensive.** It isn't just resident once — it's + re-written cold on every short-lived spawn. + +This is not a quick fix (it fights the gateway/session model), but naming it reframes the +priority order: **shrink what each cold spawn must write, and reuse warm cache where possible**, +before chasing request-count. + +### 9.4 New findings (continuing the §3 F-numbering) + +#### F13 — Verification on the critical path with no risk-proportionality (HIGH for latency, difficulty M) +`review-findings` is server-enforced before `team_complete`; a 2-line tooltip tweak triggers the +same full-context ~3-min verifier as a 500-line refactor. No diff-size threshold, no "skip gate +under N lines", no fast-path. This is the single biggest *wall-clock* tax. (Cost cousin: F5.) + +#### F14 — Cold-spawn write tax from process-per-agent (HIGH for cost+latency, difficulty L, **inherent**) +§9.3. Short-lived spawns (verifiers, small worker tasks) never amortise their cold cache write +and pay a process+MCP+container startup floor. Inherent to the architecture; mitigable by +context-diet (F15) now and warm-cache reuse (CE-G8.4) later. + +#### F15 — No slim sub-agent context profile (MED-HIGH, difficulty S–M) +Every spawned agent inherits the full AGENTS.md cascade + full role, including read-only +verifiers that never commit, lint, or open PRs and so cannot act on most of it. Claude Code's +`omitClaudeMd` for read-only sub-agents is the proven pattern. **This is the highest-leverage, +lowest-risk quick win** — attacks F5's 34% write share with zero behaviour risk. + +#### F16 — Coordination dead-time is invisible (MED for latency, difficulty S–M) +Nudge debounce/backoff and the per-spawn startup floor are pure wall-clock that **no token or +cost metric captures**, so CE-G0's ledger (dollars/tokens) would still not surface them. Latency +needs its own instrumentation dimension (wall-clock per turn, queue/startup/verify waits). + +#### F17 — No "solo fast" execution mode (MED, difficulty S) +Every goal pays the full team+gate orchestration even when the user would have just used a +single interactive agent. There is no lightweight single-agent, no-gate path for low-risk work — +so the cases most comparable to Claude Code pay the most overhead relative to their value. + +### 9.5 Remediation — latency workstream (CE-G8) + +Hand-off format matches §6. Lane independent of G0–G7 except where noted. **CE-G8.1 and CE-G8.3 +are the quick wins; do them first.** + +> **Deepened by measured loop data + new levers:** see +> [verification-loop-economics.md](verification-loop-economics.md) — a 2026-06-20 snapshot showing +> 51% of gate signals are retries (70% reviewer-driven, ~40 h of repeated impl verification) and +> adding goals **CE-G8.7** (severity floor + round budget), **CE-G8.8** (affected-only +> re-verification), **CE-G8.9** (goal-author-composed / agent-right-sized workflows), **CE-G8.10** +> (plan→cheap-exec→frontier-review + per-role thinking tiering), **CE-G8.11** (team-lead diet). +> It leads with the **trust-first north star**: no lever ships if it raises the escaped-defect rate. Behaviour-affecting items are BENCH-GATED on CE-G0.3 plus a +**wall-clock** comparison (extend the bench report with `wallClock`, `startupMs`, `verifyMs`). + +#### CE-G8.1 Risk-proportional verification gates (M) **[BENCH-GATED]** — *biggest latency win* +- Add a per-gate **fast-path**: skip or downgrade `review-findings` when the change under review + is below a configurable threshold (default ~30 changed lines, no non-doc/test files, + green `npm run check`) — config-overridable per project/workflow; never silently skip a gate a + workflow marks `required: strict`. +- Where not skipped, run a **lighter review** (cheap model + diff-scoped context, ties to + CE-G3.3/CE-G5.2) instead of the full verifier. +- **Tests:** unit — threshold/skip decision table (lines, file classes, strictness); E2E — a + sub-threshold gated step completes without spawning a full verifier; BENCH — review task + quality holds on the suite while median goal wall-clock drops. + +#### CE-G8.2 Verifier cold-write reduction (S–M) — *attacks F5's 34% write share* +- Pair with CE-G3.3/F15: verifiers spawn with a **diff-scoped, slim** context (artifact + + acceptance criteria + slim reviewer role), measured by cache-**write** tokens per verifier, + not just rounds. Target: halve verifier write share. +- **Tests:** unit — verifier context assembly excludes the AGENTS.md cascade by default; E2E — + spawned verifier's resident snapshot under a byte budget; **measure cacheWrite/verifier** in + the CE-G0.1 ledger before/after. + +#### CE-G8.3 Slim sub-agent / read-only context profile (S–M) — *quick win, zero behaviour risk* +- Adopt Claude Code's `omitClaudeMd` pattern: read-only roles (verifiers, explorers, reviewers) + get a context profile that **drops the project AGENTS.md cascade and commit/PR/lint rules** + they cannot act on. Resolution lives where role context is assembled (`session-setup.ts`). +- **Tests:** unit — profile resolution per role (read-only ⇒ no cascade); E2E — a read-only + spawn's prompt sections omit AGENTS.md; pin a per-profile resident-size budget (like the + tool-description budget) so it can't silently regrow. + +#### CE-G8.4 Warm-cache reuse for short-lived spawns (L, **inherent-architecture**, research-first) +- Investigate letting a verifier/sub-agent **reuse the parent's warm cache** instead of + cold-spawning a fresh process: options are (a) an in-process review path for non-sandboxed + goals, (b) a pooled warm agent process keyed by resident-prefix hash, (c) SDK/fork support for + prefix sharing across processes. Decide feasibility against the gateway model in a spike; + this is the real ceiling on F14 but must not regress sandbox isolation. +- **Tests:** spike + design note appended here before any code; if adopted, E2E that a reused + spawn records ~0 cold cache-write for the shared prefix. + +#### CE-G8.5 Latency instrumentation (S–M) — *extends CE-G0.1, closes F16* +- Add wall-clock dimensions to the CE-G0.1 ledger: per-turn `wallClockMs`, and per-spawn + `startupMs` (spawn→first-token), `queueMs` (enqueue→start), `verifyMs` (gate signal→verdict). + Surface a "time breakdown" alongside the cost lens (CE-G0.2): work vs coordination vs + verification vs startup. +- Extend the bench report (CE-G0.3) with `wallClock`, `startupMs`, `verifyMs` so CE-G8.1’s + BENCH gate can assert *time* not just dollars. +- **Tests:** unit — timing capture + attribution; API E2E — rollup math over a fixture ledger. + +#### CE-G8.6 "Solo fast" goal mode (S) — *closes F17* +- A goal option that runs a **single agent, no team, no gates** (or design-doc only) for + low-risk work — the Claude-Code-equivalent path — selectable at goal creation, with the + trade-off (no automated verification) made explicit in the UI. +- **Tests:** unit — mode disables team/gate machinery; E2E — a solo-fast goal completes without + spawning verifiers; browser E2E for the creation toggle + persisted choice. + +### Expected impact (latency — estimates; the bench harness replaces these) + +| Goal | Axis | Est. effect | +|---|---|---| +| CE-G8.1 | wall-clock | removes ~1.34× × ~3 min serialized verify from small goals — the dominant tax | +| CE-G8.2/G8.3 | cost+wall-clock | halves verifier cache-**write** share + cuts per-spawn load; zero behaviour risk (G8.3) | +| CE-G8.4 | cost+wall-clock | eliminates cold-write for reused prefixes (ceiling fix; highest risk) | +| CE-G8.5 | visibility | makes coordination/startup dead-time a dashboard number (blocker for tuning) | +| CE-G8.6 | wall-clock | low-risk goals skip orchestration entirely | + +### 9.6 Parallelism & swarms — the serial→parallel lever (forward-looking) + +Every lever in §6/§9.5 makes a *serial* step cheaper; none removes the **serialization** that +§9.2 identified as the real wall-clock tax. The one lever that does is **parallelism**: fan a +goal out to many small specialised agents (Claude Code's in-process swarm) and, for +decisions/quality, add a **reconciliation/council** step +([karpathy/llm-council](https://github.com/karpathy/llm-council); hermes' +`mixture_of_agents_tool.py`). N independent ~3-min sub-tasks then cost ~3 min, not 3N; and +peer cross-review folds part of the verification tax (F13/F5) *into* the parallel wave instead +of a serial downstream gate. + +The blocker is cost: a swarm of *cold OS processes* (§9.3) amplifies the process-per-agent tax +(F14) by the swarm width — a net loss. The unlock is the imminent **single-container Bobbit** +change (per-agent sandboxing becomes redundant → members run as cheap **in-process / +worker_thread forks** that inherit the parent's warm cache, CE-G8.4) and, later, **federation** +across connected Bobbits via the gateway. Because it composes from machinery already on the +roadmap (capability registry, Lifecycle Hub, model-selector pack, workflow templates), it's a +**pack + capability** feature, not core surgery. + +This is large enough to warrant its own doc — see +**[agent-swarm-and-reconciliation.md](agent-swarm-and-reconciliation.md)** (workstream **SW**: +fan-out swarm, council/reconciliation, the cheap-spawning dependency chain, federation, and a +BENCH-gated phased plan SW-G0–G4). It is *gated on* this doc's CE-G8.4 (warm-cache reuse) + +the sandbox change, and reuses CE-G0.3/CE-G8.5 (bench + wall-clock metrics) as its measuring +stick. + +### 9.7 Reproduction recipes (latency) + +```bash +# Verifier vs work session counts + cache-write share (the inverted economics) +python3 - <<'PY' +import json,statistics +d=json.load(open(".bobbit/state/session-costs.json")) +for lbl,pred in [("VERIFIER",lambda k:k.startswith("llm-review")),("WORK",lambda k:not k.startswith("llm-review"))]: + rows=[v for k,v in d.items() if pred(k)] + cw=sum(v.get("cacheWriteTokens",0) for v in rows); cr=sum(v.get("cacheReadTokens",0) for v in rows) + print(f"{lbl}: n={len(rows)} write/read={cw/max(1,cr):.3f}") +PY + +# Bobbit verifier wall-clock (sessionSetup→Shutdown) from context traces +for f in .bobbit/state/session-context-trace/llm-review-*.jsonl; do + python3 -c 'import json,sys; t=[json.loads(l)["ts"] for l in open(sys.argv[1])]; print((max(t)-min(t))/1000)' "$f" +done | sort -n | awk '{a[NR]=$1} END{print "median(s)=" a[int(NR/2)], "max(s)=" a[NR]}' + +# Claude Code per-API-round latency baseline +python3 - <<'PY' +import json,glob,datetime,statistics +def ms(t): + try: return datetime.datetime.fromisoformat(t.replace("Z","+00:00")).timestamp()*1000 + except: return None +gaps=[] +for f in glob.glob("~/.claude/projects/*/*.jsonl".replace("~",__import__("os").path.expanduser("~"))): + a=[ms(json.loads(l).get("timestamp")) for l in open(f) if '"assistant"' in l] + a=[x for x in a if x] + gaps+=[ (y-x)/1000 for x,y in zip(a,a[1:]) if 0<(y-x)/1000<600 ] +if gaps: print(f"CC per-round gap: median {statistics.median(gaps):.1f}s p90 {sorted(gaps)[int(len(gaps)*0.9)]:.1f}s") +PY +``` diff --git a/docs/design/verification-loop-economics.md b/docs/design/verification-loop-economics.md new file mode 100644 index 000000000..33de7bc24 --- /dev/null +++ b/docs/design/verification-loop-economics.md @@ -0,0 +1,422 @@ +# Verification-Loop Economics — the retry tax, and right-sizing the gate suite + +Status: **investigation complete, remediation not started** · Measured 2026-06-20 against a live +`.bobbit/state` snapshot on the primary dev machine (28 goals, 633 sessions, $2,184 tracked +spend, 240 gate signals, 379 verifier runs), cross-checked against 1,461 Claude Code sessions and +the `~/.hermes` council harness. Workstream **CE** in +[fable-program-execution-plan.md](fable-program-execution-plan.md) — this doc **deepens +CE-G8.1** (risk-proportional verification) with measured loop data and adds the goals +**CE-G8.7–G8.11**. +Companion docs: [time-and-token-cost-efficiency.md](time-and-token-cost-efficiency.md) (§9 latency +axis + the process-per-agent root cause this builds on; the F5/F13/F14/F15 findings), +[agent-swarm-and-reconciliation.md](agent-swarm-and-reconciliation.md) (the parallel end-state +these levers converge toward), [extension-platform-implementation-plan.md](extension-platform-implementation-plan.md) +(model-selector G9.2/G9.3 — the substrate for model/thinking tiering), [agent-memory.md](agent-memory.md). + +**The question this answers:** goals run for hours and re-spawn fix/review agents long after the +work looks done. Where does that time actually go, is it the *agents* or the *machinery*, and +what is the cheapest change that bends the curve? + +**The one-paragraph answer.** The dominant tax is the **verification loop**, and it is driven by +**reviewer opinion, not broken code**. Half of all gate-signal events (123 of 240, **51%**) are +*retries* of a gate that was already signaled; 112 of those 123 retries follow a **failed** +verification, and **70% of failed signals were a reviewer returning FAIL while the build/tests +passed** — only ~9% were a real build/test failure. Each implementation re-signal re-runs the +*entire* phase suite (Build+Check+Unit+E2E + gap-analysis + code-review + security ≈ **27 min of +machine compute**, ~9–10 min wall) even when one nit changed, because there is **no +severity→verdict mapping** (any single subjective `[high]` blocks the gate) and **no +affected-only re-verification** (a re-signal cascade-resets and re-runs everything). Across the +dataset that is **≈40 h of repeated implementation-gate verification**. The fix is four +data-driven, mostly-infra-free levers — a **severity floor + round budget**, **affected-only +re-verification**, **goal-author-composed / agent-right-sized workflows**, and **model+thinking +tiering (plan→cheap→frontier-review)** — each validated by the experiment-runner before it ships, +with one hard guardrail learned from Hermes: a cheap tier is only safe behind an **independent +reviewer that checks ground truth and never self-certifies**. + +--- + +## North star — the invariant every lever bows to + +**Bobbit's defining value is that autonomous agent work is *trustworthy*: verified to a high bar +with minimal human babysitting. That is non-negotiable and ranks above every optimization in this +doc.** Speed, token cost, swarm width, thinking budget, and model tier are all **subordinate** — +no lever may ship if it raises the **escaped-defect rate**, which is why every behaviour-affecting +goal here is BENCH-gated on defect-escape, not dollars (§0.4, §6). + +What we are cutting is **waste, not assurance**: re-reviewing code that did not change, looping on +subjective nits, running heavyweight gates on trivial diffs, and re-reading bloated context. Every +lever either (a) holds quality constant while removing that waste, or (b) *strengthens* +verification — the **strengthen-free / weaken-gated asymmetry** (CE-G8.9), the +**reviewer-confirmed-bug floor** (CE-G8.7), the **non-removable build/security gate floor**, and +the **ground-truth reviewer guardrail** on any cheap tier (CE-G8.10, from Hermes). Gates stay +re-openable; nothing self-certifies. + +The end-state we are building toward — Fable-style **fan-out → synthesize → concrete plan → +execute → loop** across many agents (see [agent-swarm-and-reconciliation.md](agent-swarm-and-reconciliation.md)) +— is valuable *precisely because* it sits on this trust foundation. Parallelism multiplies output; +verification is what makes that output safe to trust at scale. The levers in this doc make +verification **cheaper and better-targeted**, never weaker, so that scaling up the agent count +later compounds quality instead of diluting it. + +--- + +## §0 Universal rules + +Same as the extension-platform / CE plans (restated because goals here are handed off +independently): + +1. **Test-first.** Every CE-G8.x goal below lists pinning tests; write them RED first. No flaky + tests. +2. **Locate code by symbol name, not line number.** Anchors here were verified on the + 2026-06-20 snapshot and will drift; if a symbol is missing, STOP and re-derive — don't guess. +3. **Gates.** `npm run check` + `npm run test:unit` for everything; `+ test:e2e` for + server/UI changes. +4. **BENCH-gated.** Every behaviour-affecting lever ships behind an experiment-runner A/B + (see §6) that measures retries, wall-clock, cost, **and** escaped-defect rate — never just + dollars. A latency/quality win on vibes is not a win. +5. **Shared-seam serialization.** `verification-logic.ts`, `gate-store.ts`, `team-manager.ts`, + `session-setup.ts` are hot seams (also touched by CE-G8.1–G8.6, CS, EP). `rg` the symbol and + serialize per execution-plan §1.4. + +--- + +## §1 The measured loop (the data) + +All figures from the 2026-06-20 snapshot; reproduction recipes in **Appendix A**. + +### 1.1 Half of all gate signals are do-overs + +| Quantity | Value | +|---|---| +| Distinct gates signaled ≥ once | 117 | +| Total signal events | 240 | +| **Re-signals (retries)** | **123 — 51% of all signals** | +| Gates that looped (≥ 1 retry) | 48 | +| Retries following a **FAILED** verification | 112 | +| Retries following an already-**PASSED** gate (cascade-reset re-run of green work) | 11 | +| Retries at a **new** commit (a fix landed) | 108 | +| Retries at the **same** commit (pure re-verification — flaky / metadata / reset) | 15 | + +Worst offenders are all the `implementation` gate: **11 retries** on *Hierarchical goal metadata* +and on *Hindsight setup & deploy modes*; 7 on *P3 modes consent* and *Experiment runner*; median +≈ 2.6 retries per implementation gate. + +### 1.2 Failures are reviewer verdicts, not broken builds + +Cause attribution of the 110 failed signals that carry step detail: + +| Cause | Count | +|---|---| +| **Reviewer (llm-review) said FAIL, build/tests green** | **70** | +| Mixed (reviewer + a command both failed) | 30 | +| Real build/test failure only | 10 | + +Which step actually failed the gate (count across all failed signals): + +| Failing step (role) | Times it failed the gate | +|---|---| +| **Code quality review** (`code-reviewer`) | **83** | +| Gap analysis (`spec-auditor`) | 40 | +| Security review (`security-reviewer`) | 31 | +| Unit tests *(real)* | 20 | +| E2E tests *(real)* | 19 | +| Regression / design / docs reviews | ~20 | + +`code-reviewer` alone failed more gates than **all build+test failures combined**. + +### 1.3 The findings are mostly subjective, yet any one blocks the gate + +Severity tags across **failed** review steps: `[critical] 18 · [high] 112 · [medium] 88 · +[low] 33`. There are only **18 critical findings in the entire dataset**. But the verdict is a +**binary pass/fail an LLM picks** (`verification_result(verdict)` in +`defaults/tools/tasks/verification_result.yaml`) with **no severity→verdict contract**: a single +`[high]` — and most "high" are subjective quality (AGENTS.md growth, UI inconsistency, "missing +test coverage"), not broken functionality — fails the gate. + +### 1.4 The cascade: one nit re-runs everything + +`gate-store.ts` cascade-resets all transitive downstream gates on a re-signal, and +`verification-logic.ts` re-runs the **whole** phase suite from scratch — there is no +"only re-run steps whose inputs changed". Measured per-step averages on the `implementation` +gate: Build 8 s · Check 14–22 s · Unit 45–76 s · **E2E 227–260 s** · Post-impl gap 145–208 s · +**Code quality 266 s** · Security 94–198 s. **One full re-run ≈ 27 min of machine compute** +(~9–10 min wall, since same-phase steps overlap). With **89 implementation re-signals** in the +dataset that is **≈ 40 h of repeated implementation-gate verification alone**, the bulk of it +re-checking code that did not change. + +### 1.5 Cost shape (corroborates CE §9) + +Spend by bucket (this snapshot): MAKER (coder/test/docs) **$968** · **TEAM-LEAD $570 across just +28 sessions** · VERIFIER steps **$549** (368) · other ~$98. The team-lead bucket is the +under-weighted sink — 28 long-lived leads cost more than all 368 verifier runs and ~60% of all +230 maker sessions, because a lead lives 30–66 h and re-reads its bloated coordination context on +every nudge (615M cache-read tokens across 28 leads). Cost falls out of wall-clock: every retry +is *both* minutes you wait *and* tokens you pay. + +--- + +## §2 Root causes (fishbone) + +``` + VERIFICATION DESIGN ORCHESTRATION MODEL STRATEGY + • no severity→verdict map • team-lead = context sink • one frontier tier for all + • binary pass/fail on opinion • 1 lead per (sub)goal • no plan→cheap→review split + • full suite re-runs on re-sig • verify serial, after work • reviewer not risk-tiered + \ | / + \ | / + ────────────────────────────────────────────────────────────────────────────────► GOALS TAKE HOURS & COST $$ + / \ + / \ + PROCESS MODEL CONTEXT + • process-per-agent (cold spawn) • full AGENTS.md on every agent + • no warm-cache reuse • no slim read-only profile + • MCP stdio per agent • re-read whole context each turn +``` + +The CE §9 doc owns the **process-model** and **context** bones (CE-G8.2/G8.3 + the inherent +F14). This doc owns the **verification-design**, **orchestration** (team-lead diet), and +**model-strategy** bones. + +--- + +## §3 What the other harnesses teach + +| Dimension | Claude Code | hermes-agent (council) | Bobbit today | +|---|---|---|---| +| Reviewer | human, inline / real-time | 1 independent `reviewer-opus`, re-verifies vs **ground truth** | 5–7 automated verifiers per gate, **re-run every re-signal** | +| Per-turn latency | ~4 s (warm in-process) | — | ~291 s (cold process-per-agent) | +| Sub-agents | opt-in; **97.6% of work uses none** | kanban tasks + MoA fan-out | always team + full gate suite | +| Gate loops | none (no gates) | FAIL → scoped fix (cards can't reopen) | 123 re-signals, full-suite re-run | +| Quality risk | human must catch it | **cheap models hallucinated PASS / "done" on RED CI; 48 FAIL vs 34 APPROVE** | nit-driven loops; non-deterministic reviewers | + +Two opposite lessons, both binding on the levers below: + +1. **Claude Code (the speed lesson).** Fast because the maker stays *warm and in-process* across + turns and fan-out is *opt-in* (97.6% of work uses no sub-agent). Portable: keep makers warm + (CE-G8.4), make sub-agents opt-in, and **don't re-run the full review suite on every + re-signal** (CE-G8.8). It has *no* automated assurance — that is the part Bobbit should keep. +2. **Hermes (the quality lesson).** Cheap workers **self-certified garbage**: claimed a clean + PASS while master CI was RED 8/8, fabricated commit SHAs, and "done" cards could not reopen so + wrong results persisted. It only stayed sane because **one independent `reviewer-opus` + re-verified against ground truth (CI, real SHAs)**. This is the **hard guardrail** on the + cheap-executor lever (CE-G8.10): a cheap tier is only safe behind a strong reviewer that + checks reality and never trusts the worker's self-report; and gates must remain + **re-openable** (Bobbit's already are — preserve that). + +--- + +## §4 The levers (hand-off backlog) + +Ordering rationale: Wave-1 (G8.7/G8.8) needs no new infra and removes roughly half of all gate +work (the retries). Wave-2 (G8.9/G8.10/G8.11) right-sizes *what* runs and on *which model/think +budget*. The parallel end-state lives in [agent-swarm-and-reconciliation.md](agent-swarm-and-reconciliation.md). + +### CE-G8.7 — Severity floor + review-round budget (S–M) **[BENCH-GATED]** — *biggest bang/effort* + +- **Diagnosis:** §1.2/§1.3. 70 of 110 failures are reviewer-only; the verdict has no severity + contract, so subjective `[high]`/`[medium]`/`[low]` block the gate and force a full re-run. +- **Contract:** make `verification_result` carry **structured findings** `{ severity, kind: + "bug"|"quality", file, line, note }`, and add a gate-level **block policy**: a gate FAILs only + on a finding that is `severity ∈ {critical, high}` **and** `kind == "bug"` (reviewer-confirmed, + reproducible). `quality`/`medium`/`low` findings are **recorded as advisory** on the gate and + surfaced to the team-lead, not auto-looped. Add a **round budget**: after N (default 2) auto + re-review rounds the gate escalates to a human decision instead of spawning another fix→re-sig + cycle. The exact rule is **a tunable**, decided by CE-G8.7-EXP (§6), defaulting to + strict-but-bug-only. +- **Owned files:** `verification_result.yaml` (+ its `extension.ts`); the reviewer role prompts + (`defaults/roles/code-reviewer.yaml`, `spec-auditor.yaml`, `security-reviewer.yaml`, + `reviewer.yaml`) to emit structured severities and the bug-vs-quality distinction; + `verification-logic.ts` (verdict aggregation → block decision); `gate-store.ts` (advisory + findings persisted, round counter). +- **Tests:** unit — block-decision table (severity × kind × strictness); a `[high] quality` + finding does NOT fail the gate, a `[high] bug` does; round-budget escalation after N. E2E — a + gate signaled with only advisory findings passes and records them; the (N+1)th retry escalates + rather than re-spawns. + +### CE-G8.8 — Affected-only re-verification (M) **[BENCH-GATED]** + +- **Diagnosis:** §1.4. A re-signal re-runs the entire phase suite even when one file changed; 89 + impl re-signals ≈ 40 h of recompute. +- **Contract:** on re-signal, compute the diff between the new commit and the previously-verified + commit; **re-run only the steps whose inputs intersect the changed files** (a step declares its + input scope: command steps by component/path globs, llm-review by the diff it reviews). Steps + whose inputs are unchanged **reuse the prior passed result** (extends the existing same-SHA + verification cache in `gate-store.ts` to a *changed-paths* cache). Cascade-reset still + invalidates downstream *content* gates, but their *unchanged* steps reuse cache. +- **Owned files:** `verification-logic.ts` (input-scope resolution + skip decision), `gate-store.ts` + (changed-paths cache key), workflow step schema (`component`/`run`/`prompt` gain an optional + `inputs:` glob list; absent ⇒ "always run", preserving today's behaviour). +- **Tests:** unit — changed-paths → step-skip table; a docs-only re-signal skips Build/Unit/E2E + and re-runs only Documentation. E2E — second signal at a new commit touching one file re-runs + only the affected steps; cache markers (`[reused — inputs unchanged]`) appear. +- **Caveat:** never skip a step a workflow marks `required: strict`; never reuse `human-signoff` + (already non-cacheable). + +### CE-G8.9 — Goal-author-composed / agent-right-sized workflows (M) **[BENCH-GATED]** + +- **Diagnosis:** today the workflow is a **frozen template picked at creation** and snapshotted + into the goal; a 2-line tooltip goal pays the same 5–7-verifier suite as a 500-line refactor. + The machinery to do better already exists (project assistant generates workflows; `propose_goal` + validates a workflow id; the workflow-editor schema; `WorkflowStore`). +- **Contract — two parts:** + 1. **Compose at creation.** A standardized step lets the goal-creating session **assemble the + gate set for *this* goal from a catalog** and record an **explicit rationale** per + included/excluded gate ("docs-only → no security/e2e", "concurrency fix → add soak gate", + "research → design+review only, no build"). Persist the rationale on the goal for audit. + 2. **Right-size in flight.** The team-lead may **propose** enabling/disabling gates after + seeing the real diff — reusing the **divergence-policy gradient + `mutation_pending` + approval card** (same machinery as plan/sub-goal mutation). **Asymmetry is the safety + model:** *strengthening* (ADD/upgrade a gate) is autonomous under `balanced`/`autonomous`; + *weakening* (disable/downgrade) is **gated**, with a **non-removable floor** — build, + typecheck, and security-on-code-changes are never agent-disableable (mirrors the bypass + human-only rule). Every toggle is justified, audit-logged, and reversible. +- **Owned files:** a `compose_workflow` proposal tool (or extend `propose_goal`); the + proposal-panel workflow surface (`src/app/proposal-panels.ts`); `WorkflowStore` snapshot path; + the gate-mutation route + `mutation_pending` plumbing (`team-manager.ts`, the policy/divergence + code); a `gateFloor` allow-list (non-disableable gate ids/kinds). +- **Tests:** unit — floor enforcement (cannot disable build/typecheck/security on a code goal); + asymmetry table (ADD autonomous, DISABLE gated by policy). E2E — a docs-only goal composed + without build/e2e completes without spawning those verifiers; an agent disable-security + proposal is held for human approval; browser E2E for the creation-time composer + persisted + rationale. +- **Why it beats a pure line-count threshold (CE-G8.1):** the author/lead reasons about the + *actual* change and records *why* — auditable and smarter than a heuristic. CE-G8.1's threshold + becomes the *default* the composer starts from. + +### CE-G8.10 — Model + thinking tiering: plan → cheap executor → frontier review (M) **[BENCH-GATED]** + +- **Diagnosis:** §1.5 — every role runs the same frontier model at the same thinking budget; + makers are 44% of spend. Front-loading reasoning into a precise plan lets a cheap/fast model + execute and a frontier model review — cutting cost *and* wall-clock. +- **Contract:** a **plan gate** whose deliverable is a *precise change-list* (exactly what to + change, where, with acceptance per item); a **cheap, low/no-thinking executor** role that + implements the change-list mechanically; a **frontier reviewer** (high thinking) that checks + plan + diff **against ground truth**. Tiers resolved by the **model-selector** (EP G9.2/G9.3); + thinking budget set **per role × task risk** (Bobbit already supports per-role `thinkingLevel` + in `session-setup.ts`) — executors default to no/low extended thinking, planners/reviewers to + high. +- **Hard guardrail (Hermes, §3.2):** the cheap tier is only enabled behind CE-G8.7's + reviewer-confirmed-bug floor and a reviewer that verifies reality (CI/tests/SHAs), never the + worker's self-report. If plan quality is the ceiling, a wrong plan faithfully produces wrong + code — so the plan gate is itself reviewed (frontier) before execution. +- **Owned files:** model-selector capability (EP); role definitions for `planner` / cheap + `executor`; `session-setup.ts` (per-role model + thinking resolution); workflow templates that + wire the plan→exec→review sequence. +- **Tests:** unit — tier resolution per role/risk; executor spawns with low thinking + cheap + model. E2E — a plan→exec→review goal completes; BENCH — cost & wall-clock down vs + frontier-everything **with no increase in escaped defects** (the gate that matters). + +### CE-G8.11 — Team-lead diet (M) + +- **Diagnosis:** §1.5 — 28 leads cost $570; a lead accumulates the whole goal's coordination + history and re-reads it every nudge. +- **Contract:** **summary handoffs** (workers return structured summaries, not raw transcripts, + into the lead's context); **lead-context compaction** at phase boundaries; optionally + **phase-scoped / ephemeral leads** (a fresh lead per phase seeded with the prior phase's + summary) so the resident context never grows unbounded. +- **Owned files:** `team-manager.ts` (worker-result summarization, nudge payload), the lead + prompt assembly, compaction hooks. +- **Tests:** unit — worker result is summarized before injection; lead resident size stays under + a pinned budget across N phases. E2E — a multi-phase goal's lead context does not grow + monotonically with worker count. + +### Expected effect (estimates — the experiment-runner replaces these) + +| Goal | Δ cost | Δ wall-clock | Attacks | +|---|---|---|---| +| CE-G8.7 severity floor + budget | −− | −− | the 70 reviewer-only failures / 123 retries | +| CE-G8.8 affected-only re-verify | −− | −− | the 27-min × 89 re-run cascade | +| CE-G8.9 composed / right-sized gates | −− | −− | tiny goals paying the full suite | +| CE-G8.10 model+thinking tiering | −− | − | maker spend (44%) + first-time correctness | +| CE-G8.11 team-lead diet | −− | − | the $570 orchestrator sink | + +--- + +## §5 Sequencing & dependencies + +``` +WAVE 0 instrument WAVE 1 kill the loop (no new infra) WAVE 2 right-size WAVE 3 (CE-G8.4 + SW) +───────────────── ────────────────────────────────── ───────────────── ───────────────────── +loop+latency metrics → CE-G8.7 severity floor + budget → CE-G8.9 composed gates → in-process agents +(CE-G8.5) → CE-G8.8 affected-only re-verify → CE-G8.10 model/think tier → swarm + cheap subgoals + CE-G8.3 slim read-only ctx CE-G8.11 team-lead diet + council (SW-G1..3) + + Experiment-runner spans every wave → A/B each lever on a real task; measure + retries · wall-clock · cost · escaped-defect rate (never dollars alone). +``` + +- **CE-G8.7** depends only on the structured-findings contract; do first. +- **CE-G8.8** is independent of G8.7 (different seam) — parallelizable. +- **CE-G8.9** depends on the divergence-policy/`mutation_pending` machinery (exists) and the + `WorkflowStore` snapshot path; CE-G8.1's threshold is its default. +- **CE-G8.10** depends on the model-selector (EP G9.2/G9.3) and is **gated** on CE-G8.7's bug + floor (the Hermes guardrail). +- **Wave 3** is gated on the single-container sandbox change + CE-G8.4 (warm-cache reuse), per + the swarm doc; "cheap sub-goals as the unit of work" is the same dependency. + +--- + +## §6 Experiment plan (the point of the experiment-runner) + +These levers are **hypotheses with a number attached**; the experiment-runner extension (in +flight) decides them on real goals instead of by guess. Each experiment fixes one task and varies +one knob, measuring `{ retries, wallClockMs, costUsd, escapedDefects }` (escaped-defects via a +post-merge re-review or a planted-bug control). + +- **CE-G8.7-EXP — severity rule.** Same task; arm A `[high]`-blocks (today), arm B + `[critical] + reviewer-confirmed bug` blocks, arm C `[high]` blocks but retries capped at 2 → + human. Pick the arm that minimizes retries+wall-clock without raising escaped-defects. +- **CE-G8.8-EXP — re-verify scope.** re-run-all vs affected-only on re-signal; expect large + wall-clock/cost drop at equal quality. +- **CE-G8.10-EXP — tiering.** frontier-everything vs plan(frontier)→exec(cheap,no-think)→review(frontier); + the **escaped-defect** column is decisive (Hermes failure mode). +- **CE-G8.9-EXP — composed gates.** template-frozen vs author-composed gate set on a mixed batch + (docs-only, small fix, large feature); measure suite-size vs defect-escape. + +Until the runner lands these are **defaults, not decisions**: ship CE-G8.7/G8.8 behind a flag, +default strict-but-bug-only + affected-only, and flip per experiment evidence. + +--- + +## Appendix A — Reproduction recipes + +All against `.bobbit/state` on the primary dev machine (2026-06-20 snapshot). + +```python +# Retry rate + cause + new-vs-same-commit (the §1.1/§1.2 numbers) +import json, collections +g = json.load(open(".bobbit/state/gates.json")) +total=resig=after_fail=after_pass=samesha=newsha=0; gates=0 +for e in g: + s=e.get("signals",[]) + if not s: continue + gates+=1; total+=len(s) + for i in range(1,len(s)): + resig+=1; pv=(s[i-1].get("verification") or {}).get("status") + after_fail += pv=="failed"; after_pass += pv=="passed" + if s[i-1].get("commitSha")==s[i].get("commitSha"): samesha+=1 + else: newsha+=1 +print(f"gates={gates} signals={total} retries={resig} ({100*resig/total:.0f}%)") +print(f" after-fail={after_fail} after-pass(reset)={after_pass} same-sha={samesha} new-sha={newsha}") + +# Loop driver: which step failed the gate, and finding severities (§1.2/§1.3) +import re +failstep=collections.Counter(); sev=collections.Counter() +for e in g: + for s in e.get("signals",[]): + v=s.get("verification") or {} + if v.get("status")!="failed": continue + for st in v.get("steps",[]): + if st.get("passed") is False: failstep[f"{st.get('type')}:{st.get('name')}"]+=1 + if st.get("type")=="llm-review" and st.get("passed") is False: + out=st.get("output") or "" + for k in ("critical","high","medium","low"): sev[k]+=len(re.findall(r"\[%s\]"%k,out,re.I)) +print(failstep.most_common(8)); print(dict(sev)) + +# Cost by role + the team-lead sink (§1.5): join session-costs.json to sessions.json by id/role +``` + +The per-step duration averages (§1.4) come from iterating `signals[].verification.steps[]` +(`duration_ms`, `name`, `passed`, `type`) on `implementation`-gate records. Spend-by-role joins +`session-costs.json` (`totalCost`, `cacheReadTokens`) to `sessions.json` `sessions[]` by id, +bucketing on `role`; verifier-step records are keyed `llm-review-*` / `agent-qa-*`.