diff --git a/CLAUDE.md b/CLAUDE.md index 69bfa5428..68f8f277c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -222,6 +222,7 @@ Only updated when a feature introduces a technology not already listed here. - TypeScript 5.x (strict; Article XV). No Python change. + `@debrief/stac-writer` (interface + core), `node:fs`/`node:crypto` (fs adaptor — existing), `idb` (web-shell adaptor — existing), `@debrief/schemas` (`FeatureCollection`, `StacItem`), VS Code Extension API (`window.showWarningMessage`). **No new runtime dependencies.** (268-save-atomicity) - TypeScript 5.x (strict mode, Article XV); React 18.x. No + `golden-layout` v2 (analysis + catalog layouts), (281-ui-review-p1-p2-fixes) - TypeScript 5.x (strict — Article XV) in `apps/vscode`; Python 3.11 unchanged (debrief-calc/debrief-stac consumed as-is) + VS Code Extension API — **Language Model Tools API** (`vscode.lm.registerTool` + `contributes.languageModelTools`); existing `calcService` (debrief-calc stdio), `stacService` (catalog), `resultsPanelService`, `openPlotsService`, `toolMatchAdapter`, `sessionManager`, `@debrief/schemas`, `@debrief/session-state` (`LogService`) (284-copilot-plot-editing) +- None new. Reads existing STAC catalog; workspace-level config files (`.github/`, `.vscode/`, `*.code-workspace`) are committed artefacts (285-copilot-best-practices) ## Before Pushing @@ -305,6 +306,6 @@ Each wrapper: Note: `vitest` does not catch TypeScript type errors — only `tsc` (run during typecheck) does. The `pnpm build` step also runs `tsc`, but typecheck is the explicit CI gate. ## Recent Changes +- 285-copilot-best-practices: Added TypeScript 5.x (strict — Article XV) in `apps/vscode`; Python 3.11 untouched (existing `debrief_stac.mcp_server` consumed as-is) + VS Code Extension API (existing ^1.99 engine; new API used behind capability checks: `McpServerDefinitionProvider`, custom agents/prompt files are host-side file conventions, not API); `@vscode/prompt-tsx` (**new dev/runtime dependency** — justified under Constitution Check Article IX); existing `src/copilot/` modules (`summarize.ts`, `resultHelpers.ts`, `runToolTool.ts`, `registerLmTools.ts`), `calcService`, `stacService` - 284-copilot-plot-editing: Added TypeScript 5.x (strict — Article XV) in `apps/vscode`; Python 3.11 unchanged (debrief-calc/debrief-stac consumed as-is) + VS Code Extension API — **Language Model Tools API** (`vscode.lm.registerTool` + `contributes.languageModelTools`); existing `calcService` (debrief-calc stdio), `stacService` (catalog), `resultsPanelService`, `openPlotsService`, `toolMatchAdapter`, `sessionManager`, `@debrief/schemas`, `@debrief/session-state` (`LogService`) - 281-ui-review-p1-p2-fixes: Added TypeScript 5.x (strict mode, Article XV); React 18.x. No + `golden-layout` v2 (analysis + catalog layouts), -- 256-prefix-aware-stac-typing: One schema-driven step in the LinkML `gen-typescript` post-processor (`shared/schemas/scripts/generate.py`) rewrites modelled slot keys to their on-disk `debrief:` `slot_uri` across `StacExtensionProperties`, `StacSummaries`, and `StacAsset`, so the STAC writers' `props['debrief:*']` / `asset['debrief:toolId']` access sites gain typed slots on read **and** write (both hosts' write-path `props` re-typed to `StacItemProperties`). Models `debrief:toolId` + `debrief:snapshotTimestamp` as `StacAsset` slots (additive Pydantic regen); `debrief:label` excluded (feature/annotation key, not STAC). On-disk JSON unchanged; reuses the existing `src/generated` drift gate. diff --git a/specs/285-copilot-best-practices/checklists/requirements.md b/specs/285-copilot-best-practices/checklists/requirements.md new file mode 100644 index 000000000..a528ffa86 --- /dev/null +++ b/specs/285-copilot-best-practices/checklists/requirements.md @@ -0,0 +1,48 @@ +# Specification Quality Checklist: Copilot Chat Integration Best-Practices Upgrade + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-07-11 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- The spec names existing artefacts (the four `debrief_*` tool ids, the #284 + priming file path, "MCP", "Code Server") where they ARE the subject of the + requirement rather than an implementation choice — consistent with project + precedent for successor-to-spike specs (#284, #235). Specific host API + identifiers (`chat.tools.eligibleForAutoApproval`, `.agent.md`, prompt-tsx, + `McpServerDefinitionProvider`) are deliberately kept OUT of FRs and left to + plan.md; they appear only in the user description quoted in the header. +- SC-005's ≥15% compact-serialization target is measurable against the + committed #284 token-probe fixtures. +- No [NEEDS CLARIFICATION] markers were required: scope boundaries came from + the #284 findings report and the user's explicit deferral of local-model + work; the pinned-model choice and built-in tool set are documented as + implementation-time decisions in Assumptions. +- Not a UI feature (configuration, tool-result contract, and research + deliverables); User Interface Flow section omitted per template guidance. diff --git a/specs/285-copilot-best-practices/contracts/customization-pack.md b/specs/285-copilot-best-practices/contracts/customization-pack.md new file mode 100644 index 000000000..6048b2fad --- /dev/null +++ b/specs/285-copilot-best-practices/contracts/customization-pack.md @@ -0,0 +1,70 @@ +# Contract: Copilot Customization Pack (US1–US3) + +Binding placement, format, and sync rules for the workspace artefacts. The +drift check and unit tests enforce this contract in CI. + +## Placement matrix + +| Artefact | Canonical | Synced copy (preview workspace root) | +|---|---|---| +| Priming | `.github/copilot-instructions.md` | `preview/workspace/samples/.github/copilot-instructions.md` | +| Agent | `.github/agents/debrief-analyst.agent.md` | `preview/workspace/samples/.github/agents/debrief-analyst.agent.md` | +| Prompts ×8 | `.github/prompts/demo-*.prompt.md` | `preview/workspace/samples/.github/prompts/demo-*.prompt.md` | + +Rules: +- Canonical is the only hand-edited location. Copies are byte-identical. +- `scripts/check-copilot-customization-drift.mjs` exits non-zero on any hash + mismatch or missing copy; wired into `task lint` / CI lint step. +- `apps/vscode/.github/copilot-instructions.md` is deleted (content relocated). + +## Agent file format (`debrief-analyst.agent.md`) + +```markdown +--- +name: Debrief Analyst +description: +model: +tools: ['debrief_searchPlots', 'debrief_summarizeCurrentPlot', + 'debrief_listTools', 'debrief_runTool'<, minimal built-ins>] +--- + +``` + +## Prompt file format (`demo--.prompt.md`) + +```markdown +--- +agent: Debrief Analyst +description: +--- + +``` + +Parity rule: a unit test asserts each prompt body matches the corresponding +utterance in `apps/vscode/tests/unit/copilot/scenarios.transcript.test.ts`'s +corpus (single source of scenario truth). + +## Approval policy declaration + +`apps/vscode/package.json`: + +```jsonc +"contributes": { + "configurationDefaults": { + "chat.tools.eligibleForAutoApproval": { "debrief_runTool": false } + } +} +``` + +Echoed in `preview/workspace/debrief-preview.code-workspace` `settings`. +The extension MUST NOT set any read-only tool to auto-approved; pre-approval +is a documented user action only (quickstart §Approvals). + +## Degradation guarantees + +- Hosts < 1.106: agent/prompt files inert; approval-eligibility key ignored; + `prepareInvocation` confirmation + invoke-time mutation guard still gate + every mutating run (unchanged #284 behaviour). +- Pinned model unavailable: host falls back to the user's default model with + visible indication; no extension code involved. diff --git a/specs/285-copilot-best-practices/contracts/plot-summary-v2.md b/specs/285-copilot-best-practices/contracts/plot-summary-v2.md new file mode 100644 index 000000000..03ee5beec --- /dev/null +++ b/specs/285-copilot-best-practices/contracts/plot-summary-v2.md @@ -0,0 +1,57 @@ +# Contract: Plot Summary v2 — budget-aware tool result (US4) + +The result contract for `debrief_summarizeCurrentPlot` (and the compact- +serialization rule for all four tools). This is the tool-result contract the +E13 offline panel inherits. + +## Serialization (all tools) + +- All JSON text-part results are compact (no indentation/pretty-printing). +- `approxTokens` is computed over the emitted payload. +- Gate: fixture test asserts ≥15% payload reduction vs #284 pretty-printed + baseline at each probe size (8 / 32 / 100 / 250 features). + +## Summary rendering modes + +| Host capability | Result part | Behaviour | +|---|---|---| +| `tokenizationOptions` supplied | prompt-tsx part | priority-pruned to the caller's budget | +| absent (older host / direct invokeTool) | compact-JSON text part | full content; `INVENTORY_CAP` backstop only | + +Parity rule: at unconstrained budget both modes carry identical field content. + +## Priority order (highest survives longest) + +1. Plot identity (`plotId`, `title`, `timeSpan`), feature counts, `openPlots`, + and the `shedding` notice itself +2. Per-feature `id`, `name`, `type`, `platform`, `pointCount` +3. Per-feature `timeSpan` +4. Per-feature `spatialDigest` + +## Shedding notice + +Whenever any content is shed (budget pruning or cap): + +- `truncated: true` +- `shedding.omitted`: content classes dropped, in shed order +- `shedding.guidance`: instructs the model to narrow scope via selection, + `debrief_searchPlots`, or `featureNames`/`featureIds` + +Minimum-budget floor: plot identity + counts + guidance are always emitted; +never an empty or malformed result. + +## Spatial digest + +- Format: `"/"`, sector ∈ {N, NE, E, SE, S, SW, W, NW, C} + (feature bbox centroid vs 3×3 grid over plot bbox), extent ∈ {pt, local, + wide} (feature/plot bbox diagonal ratio). +- Omitted (field absent) for features without geometry — never fabricated. +- Deterministic for identical inputs. +- Acceptance: "the northern track" resolves to the correct feature id from the + summary alone in the scenario replay (SC-006). + +## Evidence regeneration + +`evidence/token-budget-v2.md` re-tabulates the four probe sizes (now with +digests, compact form) against 4k / 8k / 32k local-model context windows, +superseding the #284 table for the E13 record (FR-018). diff --git a/specs/285-copilot-best-practices/data-model.md b/specs/285-copilot-best-practices/data-model.md new file mode 100644 index 000000000..79330df79 --- /dev/null +++ b/specs/285-copilot-best-practices/data-model.md @@ -0,0 +1,109 @@ +# Data Model: Copilot Chat Integration Best-Practices Upgrade (285) + +No LinkML schema changes. All entities are either workspace configuration +artefacts (validated by drift check + unit tests) or TypeScript view types +local to `apps/vscode/src/copilot/` (extension-internal, not cross-boundary — +Article II untouched; the summary view is extended per the derived-not-rewritten +rule where it mirrors existing types). + +## 1. PrimingInstructions (workspace artefact) + +| Field | Notes | +|---|---| +| canonical path | `.github/copilot-instructions.md` (repo root) | +| synced copies | `preview/workspace/samples/.github/copilot-instructions.md` | +| content | Debrief vocabulary, tool conventions (summarise-before-edit, list-before-run, never fabricate, edits-are-unsaved) — carried over from #284 file | +| validation | drift check: SHA-256 of each copy == canonical (CI-fatal) | + +## 2. DebriefAnalystAgent (workspace artefact — `.agent.md`) + +| Front-matter field | Value / constraint | +|---|---| +| `name` | `Debrief Analyst` | +| `description` | analyst-facing, one sentence | +| `model` | pinned Sonnet-class picker id (implementation-time choice) | +| `tools` | exactly: 4 `debrief_*` tools + documented minimal built-ins | +| body | embedded conventions (subset of PrimingInstructions) | + +Placement mirrors PrimingInstructions (canonical + synced copy, same drift check). + +## 3. ScenarioPrompt (workspace artefact — `.prompt.md` × 8) + +| Field | Notes | +|---|---| +| filename | `demo--.prompt.md`, n = 1..8 (5 happy, 3 fail-safe) | +| front matter | `agent: Debrief Analyst` (inherits model + tools) | +| body | scenario utterance, verbatim-equivalent to replay-corpus utterance | +| validation | unit test: body ↔ `scenarios.transcript.test.ts` corpus parity | + +## 4. ApprovalPolicyDeclaration (configuration) + +| Field | Notes | +|---|---| +| mechanism | `configurationDefaults` → `chat.tools.eligibleForAutoApproval` | +| `debrief_runTool` | `false` (never auto-approvable) | +| read-only tools | absent (host/user discretion; pre-approval documented, never granted by extension) | +| echo | same keys in `debrief-preview.code-workspace` settings | +| backstop | existing `prepareInvocation` confirmation + invoke-time mutation guard (unchanged) | + +## 5. PlotSummaryView v2 (TypeScript view type — extends #284 shape) + +``` +PlotSummaryView { + plotId, title, + timeSpan: { start, end } | null, + features: FeatureInventoryEntry[], + truncated: boolean, // now: true when ANY content was shed (budget or cap) + shedding?: SheddingNotice, // NEW — present when truncated + approxTokens: number, // computed over emitted payload (compact) + openPlots: OpenPlotView[], + selectionOnly?: true +} + +FeatureInventoryEntry { + id, name, type, platform, + timeSpan, // priority: shed 2nd + pointCount, + spatialDigest?: string // NEW — "sector/extent" e.g. "NW/wide"; omitted + // when feature has no geometry; priority: shed 1st +} + +SheddingNotice { + omitted: string[] // which content classes were shed, in order + guidance: string // how to narrow scope (selection / search / featureNames) +} +``` + +**Priority order (highest survives longest)**: plot identity + counts + +SheddingNotice → feature id/name/type → per-feature timeSpan → spatialDigest. +`INVENTORY_CAP = 200` retained as final backstop only. + +**State/rendering**: rendered via prompt-tsx element (`summaryPrompt.tsx`) when +the host supplies `tokenizationOptions`; compact-JSON text part otherwise +(identical field content at full budget — parity asserted in tests). + +## 6. SpatialDigest (derived value) + +| Component | Domain | Derivation | +|---|---|---| +| sector | `N NE E SE S SW W NW C` | feature bbox centroid vs 3×3 grid over plot bbox | +| extent | `pt | local | wide` | feature bbox diagonal / plot bbox diagonal buckets | + +Deterministic (Article I.4). Undefined (omitted) for zero-geometry features. + +## 7. McpServerRegistration (extension config + provider) + +| Field | Notes | +|---|---| +| provider id | `debrief.stacMcp` | +| definition | stdio: resolved Python interpreter → `python -m debrief_stac.mcp_server` | +| env/args | catalog path from `debrief.stacCatalogPath` | +| gate | `debrief.mcp.autoStart` setting (existing, default false) | +| failure mode | log diagnostic; LM tools unaffected (never an error dialog) | + +## 8. ComparisonFindingsNote (evidence artefact) + +`evidence/mcp-vs-lmtool-comparison.md`: query-matrix parity table (FR-020), +per-scenario routing outcome + latency under config A/B (R9), setup-friction +notes, recommendation for the E13 tool-surface contract. Follows the #284 +findings-report structure. diff --git a/specs/285-copilot-best-practices/evidence/opening-context.md b/specs/285-copilot-best-practices/evidence/opening-context.md new file mode 100644 index 000000000..fadca1791 --- /dev/null +++ b/specs/285-copilot-best-practices/evidence/opening-context.md @@ -0,0 +1,31 @@ +# Opening Context: Copilot Chat Integration Best-Practices Upgrade (285) + +## Hook + +| Spike #284 (before) | This feature (after) | +|---|---| +| Domain priming file sat where the host never loaded it — the with/without-priming A/B was comparing nothing against nothing | Priming placed at the roots the host actually scans, with a CI check that fails if the copies drift; the applied-context indicator proves it loaded | +| Demos were free-typed prompts against whatever model and tool set the operator's profile happened to have | A "Debrief Analyst" agent with a pinned model and just the four Debrief tools, plus all eight scenarios as one-keystroke `/demo-…` slash commands | +| The mutation confirmation could be silenced by the editor's global "auto-approve everything" setting | `debrief_runTool` is declared ineligible for auto-approval — the plain-language gate survives even "/yolo" mode | +| Plot summaries were pretty-printed JSON, capped at a fixed 200 features, with no way to answer "the northern track" | Compact serialization, a summary that sheds detail in priority order against the host's real token budget, and a coarse spatial digest per feature | +| Catalog search only available as an extension-contributed tool | The existing Python STAC MCP server registered with agent mode, with a written MCP-vs-extension routing comparison for E13 | + +## What We're Building + +Spike #284 proved a chat agent can drive Debrief through a four-tool surface — search plots, summarise the open plot, list the analysis tools, run one — with a plain-language confirmation gating every mutation. It also left a findings report full of named gaps, and in the meantime VS Code shipped the customization stack the spike predates: workspace instruction files, prompt files, custom agents, per-tool approval policy, and native MCP support. + +This feature brings the spike surface up to that best practice, for two reasons that matter more than polish. First, demos need to be reproducible: an operator on a fresh machine should get from "workspace open" to "first scenario running" with an agent picker and a slash command, not a checklist of model and tool settings. Second, the research measurements need to be trustworthy. The most embarrassing find here was that the #284 domain-priming file lived at a path the host never scans — so the priming A/B results were comparing nothing against nothing. Fixing placement, pinning the model, restricting the tool list, and making the summary prune against the host's actual token budget (via `@vscode/prompt-tsx`) all serve the same end: when we measure how well a model routes across the Debrief tools, we're measuring the model, not our setup noise. Each feature entry in the summary also gains a coarse spatial digest — a compass sector plus a relative-extent bucket, a few characters per feature — so "the northern track" resolves from the summary alone, closing a gap #284 named explicitly. + +## How It Fits + +Like #284, this is a demo and research surface, not a production commitment — Debrief's offline-by-default principle still governs shipped capability, and the local-model work stays deliberately deferred (#235). What this feature feeds is epic E13, the future in-Debrief NL panel: the curated agent (one model, only the Debrief tools) approximates exactly the loop that panel will run, so routing findings transfer; the budget-aware summary is the tool-result contract the panel inherits; and the MCP experiment — registering the existing Python STAC server with the editor's agent mode, no new service code — tests the standing bet that building Debrief's services MCP-first pays off when AI integrations arrive. The bypass-proof mutation gate strengthens the other standing story: the safety lives in the extension, not in prose the model can ignore. + +## Key Decisions + +- **Canonical priming at the repo root, synced copy at the preview workspace root, CI drift check.** The host only loads `.github/copilot-instructions.md` at the opened workspace's root, and the preview Code Server opens a workspace whose root is `samples/` — so one file can't cover both. Symlinks were rejected as fragile across Windows git and Docker; a hash-comparison script fails CI when copies drift (R1). +- **Restrict the tool list, don't instruct the model.** The #284 routing probe showed model choice dominates routing quality, and models routinely ignore instructions about which tools to use — but they can't escape a restricted tool list. The Debrief Analyst agent pins a Sonnet-class model and offers only the four Debrief tools plus a minimal built-in set (R2). A chat participant (`@debrief`) was rejected: agent-mode tools are the platform direction and the shape E13 will actually run. +- **Scenario prompts are asserted equal to the replay corpus.** The eight `/demo-…` prompt files carry the same utterances as the transcript-replay test fixtures, with a unit test enforcing it — live demos and automated verification cannot drift apart (R3). +- **Two-layer approvals.** `debrief_runTool` is declared ineligible for auto-approval so blanket convenience settings can't bypass the confirmation; on older hosts the existing prepare-invocation gate still holds. Read-only tools are pre-approved by a documented one-time user action — the extension never grants approval itself (R4). +- **Budget pruning via prompt-tsx, not a hand-rolled heuristic.** The host's mechanism prunes against the caller's actual tokenizer and budget; our char/4 estimate can't. Content sheds in a documented priority order — plot identity and feature ids last to go, spatial digests first — with an explicit partial-inventory notice, replacing the blunt 200-feature cap (R5, R6). Dropping pretty-printed indentation alone targets a ≥15% payload reduction at every #284 probe size. +- **Spatial digest as labels, not coordinates.** A 3×3 compass sector plus an extent bucket (`"NW/wide"`) costs 3–8 characters per feature; centroid lat/lon was rejected because raw numbers cost more tokens and models reason worse over them (R7). +- **MCP experiment is registration glue only.** The Python STAC server already exists; the extension registers it via `McpServerDefinitionProvider` behind a capability check, failure is non-fatal, and the deliverable is a written comparison — routing, latency, friction — with a recommendation for the E13 contract (R8, R9). The editor-coupled tools stay extension-contributed regardless: MCP servers can't see the open editor. diff --git a/specs/285-copilot-best-practices/plan.md b/specs/285-copilot-best-practices/plan.md new file mode 100644 index 000000000..c2f13b525 --- /dev/null +++ b/specs/285-copilot-best-practices/plan.md @@ -0,0 +1,124 @@ +# Implementation Plan: Copilot Chat Integration Best-Practices Upgrade + +**Branch**: `285-copilot-best-practices` | **Date**: 2026-07-11 | **Spec**: [spec.md](spec.md) +**Input**: Feature specification from `/specs/285-copilot-best-practices/spec.md` + +## Summary + +Upgrade the #284 Copilot spike surface to current VS Code/Copilot best practice +across five slices: (1) place the domain-priming file where the host actually +loads it (repo root + preview workspace root, canonical source + CI drift +check); (2) ship a "Debrief Analyst" custom agent (`.agent.md`, pinned model, +restricted tool list) and the eight #284 scenarios as `.prompt.md` slash +commands; (3) declare `debrief_runTool` ineligible for auto-approval via +contributed configuration defaults and document read-only pre-approval; +(4) make tool results compact and the plot summary budget-aware via +`@vscode/prompt-tsx` priority-ranked rendering, adding a per-feature spatial +digest; (5) register the existing `debrief_stac` Python MCP server through +`McpServerDefinitionProvider` and record an MCP-vs-LM-tool routing comparison +for E13. No changes to the four-tool contract's behavioural invariants +(dirty-only edits, confirmation gate, corrective fail-safes, telemetry, +provenance). + +## Technical Context + +**Language/Version**: TypeScript 5.x (strict — Article XV) in `apps/vscode`; Python 3.11 untouched (existing `debrief_stac.mcp_server` consumed as-is) +**Primary Dependencies**: VS Code Extension API (existing ^1.99 engine; new API used behind capability checks: `McpServerDefinitionProvider`, custom agents/prompt files are host-side file conventions, not API); `@vscode/prompt-tsx` (**new dev/runtime dependency** — justified under Constitution Check Article IX); existing `src/copilot/` modules (`summarize.ts`, `resultHelpers.ts`, `runToolTool.ts`, `registerLmTools.ts`), `calcService`, `stacService` +**Storage**: None new. Reads existing STAC catalog; workspace-level config files (`.github/`, `.vscode/`, `*.code-workspace`) are committed artefacts +**Testing**: vitest unit tests in `apps/vscode/tests/unit/copilot/` (extends the 48-test + 8-scenario transcript-replay suite); CI drift-check for priming copies; token-probe regeneration script +**Target Platform**: VS Code desktop 1.99+ (graceful degradation) with full behaviour on 1.106+ (custom agents, per-tool approval policy); browser-preview Code Server workspace +**Project Type**: VS Code extension + workspace configuration pack (no web/mobile split) +**Performance Goals**: Typical-plot summary (incl. spatial digest) within documented typical budget; compact serialization ≥15% payload reduction at every #284 probe size; over-budget summaries degrade by priority, never error +**Constraints**: #284 invariants preserved (FR-023); Copilot cloud dependency acceptable for this demo/research surface only (Article I governs shipped capability — same posture as #284); MCP server failure non-fatal +**Scale/Scope**: 4 LM tools, 8 scenario prompts, 1 custom agent, 2 workspace roots, ~10 existing test files extended + +## Constitution Check + +*GATE: evaluated pre-Phase-0 and re-checked post-Phase-1 — PASS (one justified note).* + +- **Article I (offline by default)**: This feature extends the #284 demo/research surface, which is explicitly not a production commitment; the offline successor is E13/#235. The MCP experiment and budget-aware summary *reduce* eventual cloud coupling (the summary contract and MCP serving both transfer to a local-model panel). No shipped core capability gains a network dependency. Same justification as #284, recorded there and re-affirmed here. +- **Article III (provenance)**: unchanged — `utterance` provenance capture and telemetry are preserved invariants (FR-023). +- **Article IV (boundaries)**: services untouched; the MCP experiment reuses the existing thin FastMCP wrapper over `debrief_stac` (IV.3 — domain logic stays in the pure library). No frontend persistence added. +- **Article VI/VII (testing, tests define done)**: every slice has an automated gate — drift check (US1), scenario replay extension (US2), approval-declaration assertion (US3), summary-format tests + regenerated probe (US4), result-parity matrix (US5). +- **Article VIII (specs before code)**: spec.md merged first; this plan + ADR note for the prompt-tsx dependency. +- **Article IX (dependencies)**: **`@vscode/prompt-tsx` is a new dependency** — Microsoft-maintained, purpose-built for exactly this (token-budget-aware LM tool results), no transitive baggage of note, pinned. Alternative (hand-rolled priority pruning) rejected: re-implements the host's sanctioned budget mechanism without access to the caller's tokenizer. Justification recorded here per Article IX.1. +- **Article X (security)**: no secrets; no new network calls from the extension (Copilot itself is the transport, as in #284). +- **Article XV (strict types)**: all new TS strict; prompt-tsx elements fully typed; the summary view type is extended, not weakened. + +## Project Structure + +### Documentation (this feature) + +```text +specs/285-copilot-best-practices/ +├── plan.md # This file +├── research.md # Phase 0 output +├── data-model.md # Phase 1 output +├── quickstart.md # Phase 1 output +├── contracts/ +│ ├── customization-pack.md # File formats + placement contract (US1–US3) +│ └── plot-summary-v2.md # Budget-aware summary result contract (US4) +├── checklists/requirements.md +└── tasks.md # /speckit.tasks output (not created here) +``` + +### Source Code (repository root) + +```text +.github/ +├── copilot-instructions.md # NEW — canonical priming (moved from apps/vscode/.github/) +├── agents/debrief-analyst.agent.md # NEW — custom agent (US2) +└── prompts/demo-*.prompt.md # NEW — 8 scenario prompts (US2) + +preview/workspace/samples/ +├── .github/copilot-instructions.md # NEW — synced copy for the preview workspace root (US1) +├── .github/agents/ # NEW — synced copy (US2) +└── .github/prompts/ # NEW — synced copy (US2) + +preview/workspace/debrief-preview.code-workspace # MODIFIED — approval settings (US3) + +apps/vscode/ +├── package.json # MODIFIED — configurationDefaults (approval policy), +│ # contributes.mcpServerDefinitionProviders (US5), +│ # @vscode/prompt-tsx dependency +├── .github/copilot-instructions.md # REMOVED (relocated to canonical root) +├── src/copilot/ +│ ├── resultHelpers.ts # MODIFIED — compact serialization (US4) +│ ├── summarize.ts # MODIFIED — spatial digest + priority ranks (US4) +│ ├── summaryPrompt.tsx # NEW — prompt-tsx element for budget-aware rendering (US4) +│ ├── summarizeCurrentPlotTool.ts # MODIFIED — renderElementJSON + tokenizationOptions (US4) +│ └── mcpProvider.ts # NEW — McpServerDefinitionProvider for debrief_stac (US5) +├── src/extension.ts # MODIFIED — register MCP provider (guarded, non-fatal) +└── tests/unit/copilot/ # EXTENDED — summary v2, digest, compact, approval decl, + # drift check, scenario prompts ↔ replay parity + +scripts/check-copilot-customization-drift.mjs # NEW — CI drift check (US1/FR-003) + +services/stac/src/debrief_stac/mcp_server.py # UNCHANGED (consumed as-is) +``` + +**Structure Decision**: All extension work stays inside the existing +`apps/vscode/src/copilot/` quarantine established by #284; customization +artefacts are workspace content at the two demo workspace roots with the repo +root as canonical source and a script-enforced sync. + +## Media Components + +None — VS Code extension configuration/infrastructure feature; no shared React +components change. Blog media will be chat-session screenshots captured per +quickstart into `evidence/screenshots/` (live Copilot session in the preview, +as done for #284). + +## Storybook E2E Testing + +None - no interactive UI components. + +## Web-Shell E2E Testing + +None - no extension workflow changes (the web-shell does not host Copilot chat; +evidence screenshots come from the preview Code Server session per quickstart). + +## Complexity Tracking + +No constitution violations requiring justification beyond the Article IX +dependency note recorded in the Constitution Check above. diff --git a/specs/285-copilot-best-practices/quickstart.md b/specs/285-copilot-best-practices/quickstart.md new file mode 100644 index 000000000..01b05805c --- /dev/null +++ b/specs/285-copilot-best-practices/quickstart.md @@ -0,0 +1,73 @@ +# Quickstart: Copilot Chat Best-Practices Upgrade (285) + +Drive the upgraded demo surface and produce the evidence. Builds on the #284 +quickstart; differences are marked **NEW**. + +## Prerequisites + +- VS Code **1.106+** for the full experience (custom agents, per-tool approval + policy). 1.99+ works with #284-level behaviour. +- GitHub Copilot Chat signed in, agent mode enabled. +- Debrief extension built from this branch; debrief-calc Python env resolvable. +- Open one of the two supported workspaces: the preview + `debrief-preview.code-workspace` (Code Server) or the repo root. + +## 1. Verify priming loads (US1 — SC-001) + +1. Send any chat request in agent mode. +2. Expand the request's applied-context/instructions indicator. +3. Confirm "copilot-instructions.md" (Debrief priming) is listed. Screenshot → + `evidence/screenshots/priming-applied-{preview,root}.png`. + +## 2. Run demos the new way (US2 — SC-002/003) + +1. Open the chat agent picker → select **Debrief Analyst** (**NEW**). Verify + the pinned model shows; if your subscription lacks it, the host falls back + to your default model (expected, notice shown). +2. Type `/` → the eight `demo-*` commands appear (**NEW**). +3. Run `/demo-1-open-plot` … `/demo-8-…` in order. Each must reproduce its + #284 scripted outcome (terminal tool, gating, refusals). Record a live + session for SC-003. + +## 3. Approvals (US3 — SC-004) + +One-time demo setup (**NEW**): Command Palette → **Chat: Manage Tool +Approval** → pre-approve `debrief_searchPlots`, `debrief_summarizeCurrentPlot`, +`debrief_listTools` at workspace scope. Do NOT pre-approve `debrief_runTool`. + +Bypass-proof check: enable `chat.tools.global.autoApprove` (or `/yolo`), run a +mutating scenario — the plain-language confirmation MUST still appear +(`debrief_runTool` is declared ineligible for auto-approval). Decline → plot +byte-identical. Screenshot both. + +## 4. Budget-aware summary (US4 — SC-005/006) + +- `/demo-2-summarise` on a typical plot: feature entries now include + `spatialDigest` (e.g. `"NW/wide"`). Ask "colour the northern track red" — + the model must target the correct feature id without a follow-up call. +- Load the 250-feature probe fixture, summarise: output fits the granted + budget, `shedding` notice lists omitted content and narrowing guidance. +- Regenerate the probe table: `evidence/token-budget-v2.md`. + +## 5. MCP hybrid experiment (US5 — SC-007) + +1. Set `debrief.mcp.autoStart: true` (**NEW** — registers the existing Python + STAC MCP server via the extension; no manual mcp.json). +2. Config A: disable `debrief_searchPlots` in the tool picker → search via the + MCP-served tool. Config B: inverse. +3. Run the parity script (query matrix × fixtures) — result sets identical. +4. Run the search-terminating scenarios under A and B; record routing + + latency → `evidence/mcp-vs-lmtool-comparison.md`. +5. Kill the Python env and restart: LM tools must keep working; MCP failure is + a log line, not a dialog (FR-021). + +## Automated gates (run before pushing) + +```sh +task verify # lint (incl. drift check), typecheck, tests +node scripts/check-copilot-customization-drift.mjs # standalone drift check +``` + +Unit suites extended in `apps/vscode/tests/unit/copilot/`: summary v2 + +digest + shedding, compact-serialization reduction, prompt↔replay-corpus +parity, approval-declaration presence, MCP provider guard. diff --git a/specs/285-copilot-best-practices/research.md b/specs/285-copilot-best-practices/research.md new file mode 100644 index 000000000..a0ed6cacf --- /dev/null +++ b/specs/285-copilot-best-practices/research.md @@ -0,0 +1,221 @@ +# Research: Copilot Chat Integration Best-Practices Upgrade (285) + +All Technical Context unknowns resolved. Sources: VS Code official docs +(`code.visualstudio.com/api/extension-guides/ai/*`, `/docs/agent-customization/`, +release notes 1.102/1.103/1.106), the #284 findings report +(`specs/284-copilot-plot-editing/evidence/findings-report.md`), and repo +inspection (2026-07-11). + +## R1 — Priming file placement + canonical source + +**Decision**: Canonical priming lives at repo-root `.github/copilot-instructions.md` +(relocated from `apps/vscode/.github/`). A synced copy is committed at +`preview/workspace/samples/.github/copilot-instructions.md` because the preview +Code Server opens `preview/workspace/debrief-preview.code-workspace`, whose only +folder is `samples/` — that folder is the workspace root the host scans. +A CI script (`scripts/check-copilot-customization-drift.mjs`) fails when any +synced copy's content hash differs from the canonical file (covers agents/ and +prompts/ copies too). + +**Rationale**: VS Code applies `.github/copilot-instructions.md` only at the +workspace root of the opened folder/workspace. The #284 file at +`apps/vscode/.github/` is loaded in neither demo workspace — confirmed by +inspecting `preview/entrypoint.sh` (opens the `.code-workspace`) and the +workspace file (single folder `samples`). + +**Alternatives considered**: Symlinks (rejected — fragile across git on +Windows and Docker COPY); single root-only file (rejected — the preview +workspace root is `samples/`, not the repo root, so it would still miss the +primary demo surface); build-time copy in Dockerfile.preview only (rejected — +invisible in the repo, drift risk inverted, and desktop users opening +`samples/` directly would miss it). + +## R2 — Custom agent definition + +**Decision**: `.github/agents/debrief-analyst.agent.md` (VS Code 1.106+ +convention; older `.chatmode.md` name auto-migrates, older hosts ignore the +file entirely). Front matter: `name: Debrief Analyst`, `description` in analyst +terms, `model` pinned to a Sonnet-class Copilot picker entry (exact id chosen +at implementation time from the current picker; #284 routing probe justifies +the class), `tools:` restricted to the four `debrief_*` tools plus the minimal +built-ins the scenarios require (determined empirically per spec Assumption; +expected: none or search-only). Body embeds the tool-usage conventions +(summarise-before-edit, list-before-run, never fabricate ids) so the persona +carries them independent of workspace instructions (FR-005). + +**Rationale**: Restricting the tool list is the reliable routing lever — +community experience (and the #284 probe's model-sensitivity finding) shows +models ignore *instructions* about tool choice far more often than they escape +a restricted tool *list*. A pinned model makes demos reproducible; VS Code +falls back to the user's default model with a visible indication when the +pinned model is unavailable (FR-006 satisfied by host behaviour — documented +in quickstart, no code needed). + +**Alternatives considered**: Chat participant (`@debrief`) — rejected: platform +direction is agent-mode tools; a participant forks the interaction model away +from the E13 loop we're researching. Toolsets (`toolsets.jsonc`) alone — +rejected as primary: user-profile-scoped, not committable workspace content; +may be mentioned in docs as optional convenience. + +## R3 — Scenario prompt files + +**Decision**: Eight `.github/prompts/demo-*.prompt.md` files, one per #284 +scenario (5 happy-path, 3 fail-safe), named to sort in demo order +(`demo-1-open-plot.prompt.md` …). Front matter pins `agent: Debrief Analyst` +(and thereby model + tools); body is the scenario utterance plus minimal +context. Prompts appear as `/demo-…` slash commands. The scenario text is kept +verbatim-equivalent to the transcript-replay fixtures so the replay suite and +the live prompts cannot drift apart (a unit test asserts the prompt bodies +match the replay corpus utterances). + +**Rationale**: Prompt files are the platform's sanctioned reproducible-demo +surface (Burke Holland's instructions/prompts/agents decision matrix; official +docs). Tying them to the replay corpus keeps FR-008 testable and prepares the +ground for the future sequence-aware routing harness (out of scope here) to +reuse the same corpus. + +**Alternatives considered**: Documentation-only scripted prompts (status quo — +rejected: free-typing degrades live demos and permits drift); VS Code tasks or +snippets (rejected: not chat-native, no agent/model pinning). + +## R4 — Approval policy: bypass-proof mutation gate, frictionless read-only + +**Decision**: Two-layer approach. +1. `apps/vscode/package.json` `configurationDefaults` contributes + `chat.tools.eligibleForAutoApproval` marking `debrief_runTool` **not** + eligible — host-level blanket auto-approve (`chat.tools.global.autoApprove`, + "/yolo") then cannot bypass the confirmation. The same defaults are echoed + in `preview/workspace/debrief-preview.code-workspace` settings so the + preview enforces it even if extension-contributed defaults are overridden. +2. Read-only tools are NOT auto-approved by us. Quickstart documents the + one-time "Chat: Manage Tool Approval" workspace pre-approval of + `debrief_searchPlots`, `debrief_summarizeCurrentPlot`, `debrief_listTools` + (host 1.106+ supports approving all tools from an extension in one action). + +**Rationale**: FR-010 requires the gate to survive user convenience settings; +`eligibleForAutoApproval` is the host's mechanism for exactly this. FR-012 +forbids the extension silently granting approval — pre-approval must remain a +user/host action, hence documentation not code. Defence in depth (FR-013): the +existing `prepareInvocation` `confirmationMessages` gate and the invoke-time +`mutation/*` guard (from #284, T025) remain untouched, so hosts predating the +setting still confirm. + +**Alternatives considered**: Marking read-only tools eligible-and-approved +programmatically — no such API exists, and it would violate FR-012's +user-controlled principle. Doing nothing (status quo) — rejected: a demo under +"/yolo" would silently apply mutations, undermining the defence-posture story. + +## R5 — Compact serialization (quick win) + +**Decision**: `resultHelpers.jsonResult()` drops the `null, 2` pretty-print +arguments — single-line JSON. `approxTokens` continues to be computed over the +emitted payload (it already stringifies the same value, so it becomes honest +automatically). A fixture test asserts ≥15% payload reduction at each #284 +probe size (SC-005). + +**Rationale**: Indentation is pure token cost with zero model value; the #284 +probe numbers were measured over pretty-printed output, so this alone buys +budget headroom and corrects the standing E13 dataset. + +**Alternatives considered**: None serious — this is strictly better. (CSV/ +tabular encodings rejected: loses the schema-shaped structure the model uses.) + +## R6 — Budget-aware summary via prompt-tsx + +**Decision**: Add `@vscode/prompt-tsx` (pinned). New `summaryPrompt.tsx` +renders the plot summary as a priority-ranked element tree; `invoke()` in +`summarizeCurrentPlotTool.ts` calls `renderElementJSON(element, props, +options.tokenizationOptions)` and returns a `LanguageModelPromptTsxPart`, +falling back to the compact-text path when the host passes no +`tokenizationOptions` (older hosts / direct `invokeTool` callers — and the +existing text path is kept for the transcript-replay tests). Priority order +(highest survives longest): plot identity + counts + truncation notice → +feature id/name/type → per-feature time spans → spatial digest. The +`INVENTORY_CAP = 200` remains only as a final backstop, no longer the primary +size control (FR-015). When pruning occurs the rendered output includes the +partial-inventory notice with narrowing guidance (FR-016). + +**Rationale**: prompt-tsx is the host's sanctioned budget mechanism — it prunes +against the *caller's actual tokenizer and budget* rather than our char/4 +heuristic. Priority ranking implements FR-015's documented shed order. + +**Alternatives considered**: Hand-rolled pruning against `approxTokens` +(rejected — duplicates host machinery, wrong tokenizer, more code to test); +raising/lowering INVENTORY_CAP (rejected — blunt, feature-count is a poor +proxy for tokens once digests vary per feature). + +## R7 — Spatial digest + +**Decision**: Per-feature coarse digest computed from the feature's bbox +centroid relative to the plot-wide bbox: a 3×3 compass-sector label +(`N`, `NE`, `E`, `SE`, `S`, `SW`, `W`, `NW`, `C`) plus a relative-extent bucket +(`pt` | `local` | `wide` — fraction of plot bbox diagonal). Emitted as a short +string field (e.g. `"NW/wide"`), omitted for features without geometry +(never fabricated — edge case in spec). Cost ≈ 3–8 chars/feature, measured in +the regenerated token probe (FR-018). + +**Rationale**: Closes the #284 Q2 gap ("the northern track") with the smallest +token footprint that still resolves positional references; sector+extent is +deterministic (Article I.4 reproducibility) and needs no geometry in the +summary. + +**Alternatives considered**: Centroid lat/lon (rejected — numeric coordinates +cost more tokens and models reason worse over raw numbers than labels); +per-feature bbox (rejected — 4 numbers/feature blows the budget on large +plots); leaving to a follow-up drill-down tool (rejected — spec requires +summary-only resolution, SC-006). + +## R8 — MCP registration for the existing Python STAC server + +**Decision**: `contributes.mcpServerDefinitionProviders` in package.json + +`vscode.lm.registerMcpServerDefinitionProvider('debrief.stacMcp', provider)` in +a new `src/copilot/mcpProvider.ts`. The provider returns one +`McpStdioServerDefinition` launching the existing server +(`debrief_stac.mcp_server:main`, FastMCP over stdio) using the same Python +interpreter resolution `calcService` already performs, with the catalog path +from `debrief.stacCatalogPath`. Registration is wrapped in a capability check +(`'registerMcpServerDefinitionProvider' in vscode.lm`) and try/catch — startup +failure logs a diagnostic and leaves the LM tools untouched (FR-021). The +existing `debrief.mcp.autoStart` setting (already present in the preview +workspace, currently `false`) gates the experiment on/off, satisfying the +comparison protocol's need to disable one side per run. + +**Rationale**: FR-019's "no manual per-user server configuration" is exactly +what `McpServerDefinitionProvider` exists for; the Python server already +exposes catalog search, so the extension work is registration glue only (spec +Assumption). Reusing `calcService`'s interpreter resolution avoids a second +Python-discovery code path. + +**Alternatives considered**: Asking users to add the server to their user-level +`mcp.json` (rejected — manual, violates FR-019); HTTP transport +(`McpHttpServerDefinition`) (rejected — stdio is what the server implements and +avoids a port); porting search to a Node MCP server (rejected — duplicates the +Python service, violates the reuse assumption and Article IV thin-wrapper +intent). + +## R9 — Comparison protocol (US5 evidence) + +**Decision**: Fixed query matrix (text / time / platform / bbox × 3 catalog +fixtures) executed under two configurations: (A) MCP server enabled + +`debrief_searchPlots` disabled via the tool picker, (B) inverse. Result-set +parity asserted by an automated script against the catalog fixtures (FR-020, +SC-007); routing outcome + latency recorded per scenario run in the live +preview session; findings note written to +`evidence/mcp-vs-lmtool-comparison.md` following the #284 findings-report +discipline (FR-022). + +**Rationale**: Both-enabled runs are non-deterministic about which tool the +model picks (spec edge case) — disable-one-side-per-run is the only clean +control. + +## R10 — Host-version posture + +**Decision**: Keep `engines.vscode` at the current ^1.99. Custom agents, +prompt files, approval-eligibility settings, and the MCP provider are all +either inert files or capability-checked API on older hosts; the four LM tools +behave exactly as in #284 there (graceful-degradation edge case in spec). +Quickstart states 1.106+ as the "full experience" version. + +**Rationale**: No hard dependency on newest host features (spec edge case); +bumping engines would gate the whole extension on 1.106 for what is +workspace-content sugar. diff --git a/specs/285-copilot-best-practices/spec.md b/specs/285-copilot-best-practices/spec.md new file mode 100644 index 000000000..62896acba --- /dev/null +++ b/specs/285-copilot-best-practices/spec.md @@ -0,0 +1,279 @@ +# Feature Specification: Copilot Chat Integration Best-Practices Upgrade + +**Feature Branch**: `285-copilot-best-practices` +**Created**: 2026-07-11 +**Status**: Draft +**Input**: User description: "Copilot Chat integration best-practices upgrade — optimise the #284 spike surface for demoing and research: priming file location fix, Debrief Analyst custom agent, prompt-file demo scenarios, budget-aware token-efficient tool results with spatial digest, approvals hardening, MCP hybrid catalog-search experiment. Successor to spike #284." + +## Positioning + +Spike #284 proved that a chat agent can drive Debrief through a four-tool surface +(`debrief_searchPlots`, `debrief_summarizeCurrentPlot`, `debrief_listTools`, +`debrief_runTool`). Its findings report recommends adopting that surface as the +contract for the future offline NL panel (epic E13) and names concrete gaps. +Meanwhile the host platform has moved: VS Code shipped native MCP support (GA), +automatic tool grouping above a hard per-request tool limit, and a customization +stack (workspace instructions, prompt files, custom agents, per-tool approval +policy) that the spike predates. + +This feature upgrades the spike surface to current best practice so that +**demos are reproducible and friction-free** and **the research measurements +(model routing, priming effect, token budgets) are trustworthy**. Like #284 it +is demo/research-facing, not a production commitment: Copilot's cloud dependency +remains acceptable here because the deliverable is knowledge and demo capability +feeding E13 (Constitution Article I still governs shipped capability). The +local-model validation remains deliberately deferred (see #235). + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Domain priming actually loads (Priority: P1) + +The #284 domain-priming file lives at `apps/vscode/.github/copilot-instructions.md`, +but the host only auto-applies a priming file found at the **workspace root** of +the folder open in the editor. In the browser preview (Code Server) and in any +session opened at the repo root, the priming file is silently ignored — which +both weakens live demos and invalidates the with/without-priming comparisons the +research relies on. An analyst or demo operator opens the demo workspace, asks a +domain question in chat, and the model demonstrably has the Debrief vocabulary +and conventions loaded. + +**Why this priority**: Every other measurement and demo behaviour in this +feature (and in #284's standing findings) assumes priming is active. If the file +never loads, the priming A/B results are comparing nothing against nothing. +Cheapest fix, largest correctness impact. + +**Independent Test**: Open each supported demo workspace (preview Code Server +workspace; repo root), inspect the chat request's applied-instructions +indicator, and confirm the Debrief priming content is listed as applied. + +**Acceptance Scenarios**: + +1. **Given** the browser-preview Code Server workspace is open, **When** a user sends any chat request in agent mode, **Then** the chat UI's applied-context indicator shows the Debrief priming instructions were included in the request. +2. **Given** the repository is opened at its root in a desktop editor, **When** a user sends a chat request, **Then** the same priming instructions are applied. +3. **Given** the priming content is maintained in one canonical location, **When** it is edited, **Then** every workspace that loads it reflects the edit without manual copying (or a documented, automated sync keeps copies identical). + +--- + +### User Story 2 - Reproducible demos: curated agent + scripted scenarios (Priority: P1) + +A demo operator (or a stakeholder following the quickstart) selects a +**"Debrief Analyst"** agent from the chat agent picker. That agent runs with a +pinned model and a restricted tool list — the four Debrief tools plus the +minimum built-ins — so the demo behaves the same regardless of the user's +defaults, and the model is never distracted by dozens of unrelated tools. The +operator then invokes any of the #284 demo scenarios as a one-keystroke slash +command (e.g. `/demo-find-plot`), each scenario pre-wired to the right agent, +model, and tools. + +**Why this priority**: This is the core "optimise what we're demoing / +researching" ask. The #284 routing probe showed model choice dominates +tool-call quality, and community experience shows models routinely ignore +*instructions* to use specific tools — restricting the available tool list is +the reliable lever. A curated agent also approximates the future offline panel +(a model in a loop with only the Debrief surface), making research findings +transfer better to E13. + +**Independent Test**: On a fresh machine/profile with the workspace open, +select the Debrief Analyst agent, run each scenario slash command, and confirm +each completes its scripted flow without manual tool-picker or model-picker +setup. + +**Acceptance Scenarios**: + +1. **Given** a fresh editor profile with the extension installed and the demo workspace open, **When** the user opens the chat agent picker, **Then** a "Debrief Analyst" agent is offered, described in analyst terms. +2. **Given** the Debrief Analyst agent is selected, **When** the user inspects its available tools, **Then** only the four Debrief tools (plus a documented minimal set of built-ins) are enabled, and the model is the pinned one. +3. **Given** the demo workspace, **When** the user types `/` in chat, **Then** each of the eight #284 scenarios (5 happy-path + 3 fail-safe) is offered as a named prompt command, and invoking it reproduces that scenario's scripted flow. +4. **Given** a scenario prompt is invoked under the Debrief Analyst agent, **When** it completes, **Then** the outcome matches the corresponding #284 transcript expectation (same terminal tool, same gating behaviour). + +--- + +### User Story 3 - Mutation approvals can never be bypassed; read-only tools are frictionless (Priority: P2) + +A security-conscious stakeholder watches a demo where the operator has enabled +the editor's global "auto-approve everything" convenience mode. Even so, when +the chat runs a mutating Debrief tool, the plain-language confirmation dialog +still appears — the extension declares its mutating tool ineligible for +auto-approval. Conversely, the three read-only tools are pre-approved at +workspace scope by a documented one-time setup step, so a demo shows exactly +one kind of dialog: the meaningful mutation gate. + +**Why this priority**: Defence posture is a first-class product story +("the safety lives in the extension, not the prose" — #284 findings). Making +the mutation gate bypass-proof strengthens that story; removing read-only +confirmation noise makes demos watchable. Depends on nothing else in this +feature. + +**Independent Test**: Enable the editor's global tool auto-approval, run a +mutating tool via chat, and observe the confirmation still appears; run each +read-only tool after the documented pre-approval step and observe no dialogs. + +**Acceptance Scenarios**: + +1. **Given** the editor's global tool auto-approval setting is enabled, **When** chat invokes the mutating run-tool path, **Then** the plain-language confirmation dialog is still presented and the run does not proceed without explicit approval. +2. **Given** the confirmation dialog is declined, **When** the model reports back, **Then** the outcome is "declined", nothing is applied, and the plot remains byte-identical (unchanged from #284 behaviour). +3. **Given** the documented one-time pre-approval steps have been followed, **When** chat invokes search, summarise, or list-tools, **Then** no approval dialog interrupts the flow. +4. **Given** a fresh workspace where pre-approval has NOT been performed, **When** a read-only tool is invoked, **Then** the default host approval flow appears (no silent auto-approval is granted by the extension itself). + +--- + +### User Story 4 - Token-efficient, budget-aware plot summary with spatial digest (Priority: P2) + +An analyst with a very large plot open asks the agent to summarise it. Instead +of a fixed 200-feature cap with a blunt `truncated` flag, the summary adapts to +the token budget the host grants the request: highest-value content (plot +metadata, feature names/ids/types) survives, lower-priority detail (per-feature +time spans, the new spatial digest) is shed first, and the model is told what +was omitted. Every feature entry now carries a coarse **spatial digest** (e.g. +relative position/extent) so requests like "the northern track" become +answerable from the summary alone — closing the Q2 gap named in the #284 +findings. All tool results also stop wasting tokens on decorative formatting. + +**Why this priority**: Token cost per summary is the standing measurement the +E13 local-model decision rests on ("numbers, not vibes"). Making results +budget-aware improves both live demo quality on large plots and the fidelity of +the token-budget dataset. Larger effort than US1–US3, hence P2. + +**Independent Test**: Summarise fixture plots at the four #284 token-probe +sizes; verify small plots gain the spatial digest with modest size growth, and +oversized plots degrade by dropping documented lower-priority content rather +than hard-truncating at a fixed count. + +**Acceptance Scenarios**: + +1. **Given** a typical plot (tens of features), **When** it is summarised, **Then** each feature entry includes a coarse spatial digest sufficient to distinguish "northern" from "southern" features, and the reported approximate token size remains within the documented budget for typical plots. +2. **Given** a plot large enough to exceed the granted budget, **When** it is summarised, **Then** the summary fits the budget by omitting content in the documented priority order, and it tells the model the inventory is partial and how to narrow scope. +3. **Given** any tool result, **When** its serialized form is inspected, **Then** it contains no purely decorative whitespace/indentation (compact serialization), and the reported approximate token count reflects the emitted payload. +4. **Given** the #284 token-probe fixture sizes, **When** the probe is re-run, **Then** an updated fits/doesn't-fit table against representative local-model context windows is produced for the E13 record. + +--- + +### User Story 5 - MCP hybrid experiment: catalog search served by the existing Python MCP server (Priority: P3) + +A researcher configures the extension to expose Debrief's **existing** Python +STAC MCP server to the editor's agent mode, so catalog search is available as +an MCP tool alongside (or instead of) the extension-contributed search tool. +They run the routing scenarios both ways and record a comparison — routing +quality, latency, confirmation behaviour, setup friction — as a findings note +feeding E13 and validating the strategy bet that Debrief's MCP-first services +pay off in AI integrations ("the payoff of the MCP architectural choice"). + +**Why this priority**: Highest research value per line of new code (the Python +MCP server already exists), but it is an experiment with a written comparison +as its deliverable, not a demo-critical capability — the editor-coupled tools +(summarise, run-tool) must stay extension-contributed regardless, because MCP +servers cannot see the open editor or selection. + +**Independent Test**: With the MCP server registered, ask the agent to find a +plot using only the MCP-served search tool (extension search tool disabled); +confirm results match the extension tool for the same queries; comparison note +exists in evidence. + +**Acceptance Scenarios**: + +1. **Given** the extension is installed, **When** agent mode starts in the demo workspace, **Then** the Debrief STAC MCP server is discoverable and its catalog-search capability is listed among available tools without manual server configuration by the user. +2. **Given** identical search queries (text, time, platform, spatial), **When** run via the MCP-served tool and via the extension search tool, **Then** the returned plot sets are identical. +3. **Given** the routing scenarios that terminate in search, **When** each is run under both configurations, **Then** a written comparison records per-configuration routing outcome, latency, and any behavioural differences, with a recommendation for the E13 tool-surface contract. +4. **Given** the MCP server fails to start (e.g. missing Python environment), **When** agent mode is used, **Then** the extension-contributed search tool still works and the failure is reported non-fatally. + +--- + +### Edge Cases + +- Priming file present at two roots (repo root and preview workspace): content drift between copies — mitigated by canonical-source + sync check (US1, FR-003). +- The pinned model in the custom agent is unavailable to the user's Copilot subscription: the agent must degrade to the user's default model with a visible indication, not fail silently — demo docs note the fallback. +- Host version predates custom agents / per-tool approval policy: older editors ignore the new files/settings; the four tools must keep working exactly as in #284 (graceful degradation, no hard dependency on newest host features). +- A scenario prompt is invoked with no plot open: fail-safe scenarios must behave as scripted (corrective refusal), which is itself part of the demo set. +- Budget-aware summary asked for a budget smaller than the minimum useful summary: emit the highest-priority core (plot id, title, counts) plus an explicit "narrow the scope" instruction, never an empty or garbled result. +- Spatial digest on features without geometry (e.g. metadata-only entries): digest omitted for that entry, never fabricated. +- MCP-served search and extension search both enabled simultaneously: the model may pick either; the comparison protocol must control for this by disabling one side per run. +- Auto-approval ineligibility flag on a host that does not support it: the mutation gate must still hold via the existing prepare-invocation confirmation (defence in depth). + +## Requirements *(mandatory)* + +### Functional Requirements + +**Priming placement (US1)** + +- **FR-001**: The Debrief domain-priming instructions MUST be automatically applied to chat requests in every supported demo workspace: the browser-preview (Code Server) workspace and the repository opened at its root. +- **FR-002**: Applied priming MUST be verifiable by a demo operator from the chat UI (the request's applied-context/instructions indicator), and the verification step MUST be documented in the demo quickstart. +- **FR-003**: The priming content MUST have a single canonical source; if placement requires copies, an automated check MUST fail (in CI) when copies drift from the canonical source. + +**Curated agent (US2)** + +- **FR-004**: The workspace MUST provide a "Debrief Analyst" custom agent, selectable in the chat agent picker, with: a pinned model choice, a restricted tool list containing the four Debrief tools plus a documented minimal built-in set, and an analyst-facing description. +- **FR-005**: The custom agent definition MUST embed (or reference) the Debrief tool-usage conventions (summarise-before-edit, list-before-run, never fabricate ids) so the persona carries them even where workspace instructions are not loaded. +- **FR-006**: The agent MUST degrade gracefully when the pinned model is unavailable: chat remains usable with the user's default model, and demo documentation states the expected behaviour. + +**Scenario prompt files (US2)** + +- **FR-007**: Each of the eight #284 demo scenarios (5 happy-path, 3 fail-safe) MUST be invocable as a named slash-command prompt in the demo workspace, pre-wired to the Debrief Analyst agent and its tool set. +- **FR-008**: Each scenario prompt MUST reproduce the corresponding #284 scripted-transcript flow: same terminal tool, same confirmation gating, same fail-safe refusals. +- **FR-009**: The demo quickstart MUST be updated to drive demos through the agent picker + slash commands, replacing free-typed prompts as the primary demo path. + +**Approvals (US3)** + +- **FR-010**: The mutating run-tool capability MUST be declared ineligible for auto-approval, such that host-level blanket auto-approval settings do not bypass its confirmation dialog. +- **FR-011**: The existing plain-language confirmation (tool, target plot, target features by name, parameters) MUST remain the gate content; declining MUST continue to resolve as "declined" with zero changes applied. +- **FR-012**: A documented one-time workspace setup MUST allow the three read-only tools to run without approval dialogs; the extension itself MUST NOT silently grant approval (user/host-controlled only). +- **FR-013**: On hosts lacking per-tool approval-eligibility support, the mutation confirmation MUST still be enforced by the existing invocation-preparation gate (no regression from #284). + +**Token-efficient results (US4)** + +- **FR-014**: All tool results MUST use compact serialization (no pretty-printed indentation); reported approximate token counts MUST be computed over the emitted payload. +- **FR-015**: The plot summary MUST adapt its content to the token budget granted per request, shedding content in a documented priority order (core plot identity and feature ids last to go; per-feature detail and spatial digest first), replacing the fixed feature-count cap as the primary size control. +- **FR-016**: When content is shed, the summary MUST state that the inventory is partial and instruct the model how to narrow scope (selection, search, or feature filters). +- **FR-017**: Each feature entry in the summary MUST include a coarse spatial digest (relative position/extent within the plot) sufficient to resolve positional references such as "the northern track", omitted (not fabricated) for features without geometry. +- **FR-018**: The token-budget evidence table (probe sizes vs representative local-model context windows) MUST be regenerated with the new summary format and recorded for E13. + +**MCP hybrid experiment (US5)** + +- **FR-019**: The extension MUST be able to register Debrief's existing Python STAC MCP server with the editor's agent mode without manual per-user server configuration, exposing catalog search as an MCP-served tool. +- **FR-020**: The MCP-served search MUST return result sets identical to the extension-contributed search tool for the same query inputs (text, time range, platform, spatial extent). +- **FR-021**: MCP server startup failure MUST be non-fatal: the extension-contributed tools remain fully functional and the failure is surfaced as a log/diagnostic, not an error dialog. +- **FR-022**: A written comparison (routing outcome per scenario, latency, confirmation behaviour, setup friction, recommendation for the E13 contract) MUST be captured as evidence, following the #284 findings-report discipline. + +**Cross-cutting** + +- **FR-023**: All #284 behavioural invariants MUST be preserved: chat edits applied dirty-only with zero on-disk writes, structured corrective fail-safe results, per-invocation telemetry, and provenance capture of the analyst utterance. +- **FR-024**: The existing automated verification layers (unit tests, 8-scenario transcript replay) MUST remain green, extended to cover the new summary format and approval-eligibility declaration. + +### Key Entities + +- **Priming instructions**: The canonical Debrief domain-vocabulary and conventions document applied to every chat request; one source, placement per workspace. +- **Debrief Analyst agent**: A named chat persona definition — description, pinned model, restricted tool list, embedded conventions — the curated environment for demos and research runs. +- **Scenario prompt**: A named, parameterless (or minimally parameterised) invocable command reproducing one #284 demo scenario; the unit of reproducible demos and future trajectory-level evaluation. +- **Budget-aware plot summary**: The thinned plot inventory, now priority-ranked and budget-adaptive, each entry optionally carrying a spatial digest; the tool-result contract the E13 offline panel inherits. +- **Approval policy declaration**: The per-tool statement of auto-approval eligibility (mutating: never; read-only: host/user discretion). +- **MCP-served catalog search**: The same catalog-search capability delivered through the existing Python MCP service; comparison subject of the hybrid experiment. +- **Comparison findings note**: The written MCP-vs-extension-tool comparison feeding the E13 tool-surface contract decision. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: In both supported demo workspaces, the applied-context indicator confirms priming is loaded on 100% of chat requests, verified and screenshotted for evidence. +- **SC-002**: A demo operator on a fresh editor profile can go from "workspace open" to "first scenario completed" using only the agent picker and a slash command, with zero manual tool-picker or model-picker configuration, in under 2 minutes. +- **SC-003**: All eight scenario prompts reproduce their #284 scripted outcomes (terminal tool, gating, refusals) — 8/8 in a recorded live session. +- **SC-004**: With host-level blanket auto-approval enabled, 100% of mutating invocations still present the confirmation dialog; with documented pre-approval applied, 0 approval dialogs appear across the three read-only tools in a full demo run. +- **SC-005**: A typical plot summary (tens of features) including spatial digests stays within the documented typical-plot token budget; an over-budget plot degrades in the documented priority order with an explicit partial-inventory notice — both verified against the #284 probe fixture sizes, and compact serialization alone reduces summary payload size by a measurable amount (target ≥15%) at every probe size. +- **SC-006**: "The northern track" class of positional reference resolves to the correct feature id from the summary alone in the scenario replay, without a follow-up tool call. +- **SC-007**: The MCP-served search returns identical result sets to the extension search for 100% of the comparison query matrix, and the written comparison note answers: does routing quality differ, and which serving mode should the E13 contract prefer? +- **SC-008**: Existing #284 verification (48 unit tests + 8-scenario replay) remains green throughout, extended with new assertions for the summary format and approval declaration. + +## Assumptions + +- The browser-preview Code Server workspace root and the repo root are the two workspaces that matter for demos; other ad-hoc folder opens are out of scope. +- The pinned demo model is a Sonnet-class model, per the #284 routing-probe finding that the stronger model dominates tool-routing quality; exact identifier is an implementation-time choice, and probe data should inform it. +- "Minimal built-in set" for the custom agent will be determined during implementation by running the scenarios and enabling only what they require; the spec bounds it to "documented and deliberately chosen", not a specific list. +- The eight #284 scenarios are adopted as-is; authoring new scenarios is out of scope (the sequence-aware routing-probe harness recommended by #284 is a separate follow-up item, though scenario prompts should be written so that harness can reuse them). +- The spatial digest granularity (e.g. compass-sector + relative extent) is an implementation choice; the requirement is only that positional references like "northern" resolve correctly. +- The MCP experiment reuses the existing Python STAC MCP service; no new Python service capability is written for it beyond registration/config glue. +- Local-model work remains deferred (#235); nothing in this feature depends on a local runtime. + +## Out of Scope + +- The sequence-aware, multi-turn routing-probe harness (recommended by #284; separate backlog item). +- Local-model validation of the tool surface (#235, deliberately deferred). +- Serving the editor-coupled tools (summarise, run-tool) over MCP — architecturally excluded because MCP servers cannot access the open editor, selection, or dirty-edit path. +- Any change to the offline in-Debrief NL panel itself (E13 epic; this feature only feeds its contract). +- New demo scenarios beyond the #284 eight. diff --git a/specs/285-copilot-best-practices/tasks.md b/specs/285-copilot-best-practices/tasks.md new file mode 100644 index 000000000..fce292105 --- /dev/null +++ b/specs/285-copilot-best-practices/tasks.md @@ -0,0 +1,234 @@ +# Tasks: Copilot Chat Integration Best-Practices Upgrade + +**Feature**: `285-copilot-best-practices` | **Spec**: [spec.md](spec.md) | **Plan**: [plan.md](plan.md) + +Successor to spike #284. Six slices across five user stories: priming placement +(US1), curated agent + scenario prompts (US2), approvals hardening (US3), +budget-aware token-efficient summary with spatial digest (US4), MCP hybrid +catalog-search experiment (US5). All #284 behavioural invariants preserved. + +## Evidence Requirements + +**Evidence Directory**: `specs/285-copilot-best-practices/evidence/` +**Media Directory**: `specs/285-copilot-best-practices/media/` + +### Planned Artifacts + +| Artifact | Description | Captured When | +|----------|-------------|---------------| +| test-summary.md | vitest results: extended copilot suite (summary v2, digest, compact, approval decl, prompt↔corpus parity, MCP guard) + drift check | After all tests pass | +| usage-example.md | Walkthrough: select Debrief Analyst agent, run a `/demo-*` scenario, mutation gate under /yolo | After US2+US3 complete | +| screenshots/priming-applied-preview.png | Applied-context indicator showing priming loaded (preview workspace) | US1 (live preview session) | +| screenshots/priming-applied-root.png | Same, repo-root workspace | US1 | +| screenshots/agent-picker.png | Debrief Analyst in the chat agent picker with pinned model | US2 | +| screenshots/mutation-gate-yolo.png | Confirmation dialog still shown under global auto-approve | US3 | +| screenshots/summary-northern-track.png | Positional reference resolved from summary alone | US4 | +| token-budget-v2.md | Regenerated probe table (4 sizes × 4k/8k/32k) with digests + compact form | US4 (supersedes #284 table) | +| mcp-vs-lmtool-comparison.md | Query-matrix parity + per-config routing/latency + E13 recommendation | US5 (findings note) | + +### Media Content + +| Artifact | Description | Created When | +|----------|-------------|--------------| +| evidence/opening-context.md | Cached opener (Hook/What/How/Key Decisions) | Done during /speckit.plan | +| media/shipped-post.md | Feature post combining cached opener + ship-time evidence | Polish phase | + +### PR Creation + +| Action | Description | Created When | +|--------|-------------|--------------| +| Feature PR | Already open (#676); evidence + tasks pushed to it | Polish phase | +| Blog PR | PR in debrief.github.io with post | Triggered by /speckit.pr | + +## Phase 1: Setup + +**Goal**: Add the one new dependency and record the ADR so downstream tasks can import it. + +- [ ] T001 Add `@vscode/prompt-tsx` (pinned) to `apps/vscode/package.json` devDependencies/dependencies and install `apps/vscode/package.json` +- [ ] T002 Record ADR for the `@vscode/prompt-tsx` dependency (Article IX justification from plan.md Constitution Check) `docs/project_notes/decisions.md` + +**Checkpoint**: `pnpm --filter debrief-vscode install` resolves; `import { renderElementJSON } from '@vscode/prompt-tsx'` typechecks. + +## Phase 2: Foundation + +**Goal**: Shared plumbing every story leans on — the drift-check script that +enforces the customization-pack sync contract, and confirmation that the +scenario-truth corpus exists to bind prompts and replay tests together. Blocks +US1/US2 (drift check) but not US4/US5. + +- [ ] T003 Create the customization-pack drift-check script (SHA-256 canonical vs synced copies for priming/agent/prompts; exit non-zero on mismatch or missing copy) `scripts/check-copilot-customization-drift.mjs` +- [ ] T004 Wire the drift check into the lint step so CI runs it `Taskfile.yml` +- [ ] T005 [P][test] Add a unit test asserting the drift-check script passes on identical copies and fails on injected drift `apps/vscode/tests/unit/copilot/customizationDrift.test.ts` +- [ ] T006 Confirm/annotate the scenario-truth corpus (the 8 #284 utterances) in the transcript-replay test as the single source the prompt files must match `apps/vscode/tests/unit/copilot/scenarios.transcript.test.ts` + +**Checkpoint**: `node scripts/check-copilot-customization-drift.mjs` runs (passes trivially until copies exist); corpus utterances enumerated for T012 to reference. + +## Phase 3: US1 — Domain priming actually loads (P1) + +**Goal**: The Debrief priming file is applied to chat requests in both demo +workspaces, from one canonical source, drift-guarded. + +**Independent test**: Open each demo workspace, send a chat request, confirm the +applied-context indicator lists the Debrief priming (FR-001/002). + +- [ ] T007 Relocate the priming file to the canonical repo root, preserving #284 content `.github/copilot-instructions.md` +- [ ] T008 Remove the mis-placed #284 priming file `apps/vscode/.github/copilot-instructions.md` +- [ ] T009 Add the byte-identical synced copy at the preview workspace root (folder `samples/` per `debrief-preview.code-workspace`) `preview/workspace/samples/.github/copilot-instructions.md` +- [ ] T010 Verify the drift check now covers the priming copy (extend script paths if needed) `scripts/check-copilot-customization-drift.mjs` + +**Checkpoint**: drift check green; priming present at both roots; SC-001 screenshots capturable in Polish. + +## Phase 4: US2 — Reproducible demos: curated agent + scenario prompts (P1) + +**Goal**: A "Debrief Analyst" custom agent (pinned model, restricted tool list) +and the eight #284 scenarios as `/demo-*` slash commands, all drift-guarded and +corpus-bound. + +**Independent test**: On a fresh profile, select Debrief Analyst, run each +`/demo-*` command, confirm the scripted outcome without manual picker setup +(FR-004/007/008, SC-002/003). + +- [ ] T011 Determine the minimal built-in tool set empirically (run the 8 scenarios, enable only what they require) and record the chosen model id + built-ins in research.md Assumption `specs/285-copilot-best-practices/research.md` +- [ ] T012 Author the Debrief Analyst custom agent (front matter: name, description, pinned model, `tools` = 4 `debrief_*` + minimal built-ins; body embeds conventions per FR-005) `.github/agents/debrief-analyst.agent.md` +- [ ] T013 [P] Author the 5 happy-path scenario prompts, bodies verbatim-equivalent to the corpus utterances, `agent: Debrief Analyst` front matter `.github/prompts/demo-1-open-plot.prompt.md` +- [ ] T014 [P] Author the 3 fail-safe scenario prompts (no-plot, ambiguous-reference, invented-tool) `.github/prompts/demo-6-no-plot.prompt.md` +- [ ] T015 Add byte-identical synced copies of the agent + 8 prompts at the preview workspace root `preview/workspace/samples/.github/agents/debrief-analyst.agent.md` +- [ ] T016 [test] Add the prompt↔corpus parity test (each `demo-*.prompt.md` body matches its `scenarios.transcript.test.ts` utterance) `apps/vscode/tests/unit/copilot/promptCorpusParity.test.ts` +- [ ] T017 Update the drift check to include the agent + prompt copies; run it `scripts/check-copilot-customization-drift.mjs` + +**Checkpoint**: `/demo-1` … `/demo-8` appear under Debrief Analyst; parity + drift tests green; SC-002/003 evidenced in Polish live session. + +## Phase 5: US3 — Bypass-proof mutation approvals; frictionless read-only (P2) + +**Goal**: `debrief_runTool` cannot be auto-approved even under global auto- +approve; read-only pre-approval is documented (never granted by the extension). + +**Independent test**: Enable `chat.tools.global.autoApprove`, run a mutating +tool, confirmation still appears; declining leaves the plot byte-identical +(FR-010/011/013, SC-004). + +- [ ] T018 Contribute `configurationDefaults.chat.tools.eligibleForAutoApproval` = `{ "debrief_runTool": false }` `apps/vscode/package.json` +- [ ] T019 Echo the same setting in the preview workspace so it holds even if extension defaults are overridden `preview/workspace/debrief-preview.code-workspace` +- [ ] T020 [test] Assert the approval-eligibility declaration exists and marks only `debrief_runTool` ineligible; assert read-only tools are NOT programmatically auto-approved (FR-012) `apps/vscode/tests/unit/copilot/approvalPolicy.test.ts` +- [ ] T021 [test] Regression: confirm the existing `prepareInvocation` confirmation + invoke-time mutation guard still fire (defence-in-depth, FR-013) — extend existing prepare/invoke tests `apps/vscode/tests/unit/copilot/runTool.prepare.test.ts` + +**Checkpoint**: mutation gate survives /yolo (verified live in Polish); read-only pre-approval documented in quickstart (already present); no extension-granted approval. + +## Phase 6: US4 — Budget-aware token-efficient plot summary with spatial digest (P2) + +**Goal**: Compact serialization on all tool results; a per-feature spatial +digest; and a budget-aware summary that sheds content in priority order via +prompt-tsx instead of a fixed cap. + +**Independent test**: Summarise the 4 probe fixture sizes — small plots gain +digests with modest growth; over-budget plots degrade by documented shed order +with a partial-inventory notice; "northern track" resolves from summary alone +(FR-014–018, SC-005/006). + +- [ ] T022 Make `jsonResult()` emit compact JSON (drop `null, 2`); keep `approxTokens` computed over the emitted payload `apps/vscode/src/copilot/resultHelpers.ts` +- [ ] T023 [P][test] Assert ≥15% payload reduction vs the #284 pretty-printed baseline at each probe size (8/32/100/250) `apps/vscode/tests/unit/copilot/compactSerialization.test.ts` +- [ ] T024 Add the spatial-digest derivation (3×3 compass sector + pt/local/wide extent from feature-vs-plot bbox; omit for zero-geometry features) `apps/vscode/src/copilot/summarize.ts` +- [ ] T025 Extend `FeatureInventoryEntry`/`PlotSummaryView` to v2 shape (`spatialDigest?`, `shedding?` notice, `truncated` now = any-content-shed) using derived types per the boundary-types rule `apps/vscode/src/copilot/types.ts` +- [ ] T026 [P][test] Assert digest determinism + correctness (northern feature → `N*` sector; geometry-less feature → field omitted, never fabricated) `apps/vscode/tests/unit/copilot/spatialDigest.test.ts` +- [ ] T027 Author the prompt-tsx summary element with priority ranks (identity+counts+notice > id/name/type > timeSpan > digest) `apps/vscode/src/copilot/summaryPrompt.tsx` +- [ ] T028 Wire `summarizeCurrentPlotTool.invoke` to render via `renderElementJSON(..., options.tokenizationOptions)` → `LanguageModelPromptTsxPart`, falling back to the compact-text path when `tokenizationOptions` is absent `apps/vscode/src/copilot/summarizeCurrentPlotTool.ts` +- [ ] T029 Emit the `shedding` notice (omitted classes + narrowing guidance) whenever pruning or the `INVENTORY_CAP` backstop drops content; enforce the minimum-budget floor (identity+counts+guidance always present) `apps/vscode/src/copilot/summarize.ts` +- [ ] T030 [test] Summary-v2 tests: text/prompt-tsx field parity at full budget; priority shed order under shrinking budget; partial-inventory notice present when shed; floor never empty `apps/vscode/tests/unit/copilot/summarize.test.ts` + +**Checkpoint**: `pnpm --filter debrief-vscode typecheck && test` green; digests present; over-budget degrades gracefully; token-budget-v2 regenerable in Polish. + +## Phase 7: US5 — MCP hybrid catalog-search experiment (P3) + +**Goal**: Register the existing Python STAC MCP server with agent mode (no +manual mcp.json), gated by `debrief.mcp.autoStart`, non-fatal on failure; then +produce the MCP-vs-LM-tool routing comparison for E13. + +**Independent test**: With the MCP server enabled and `debrief_searchPlots` +disabled, the agent finds a plot via the MCP-served tool; result sets match the +extension tool across the query matrix; comparison note exists (FR-019–022, +SC-007). + +- [ ] T031 Implement the `McpServerDefinitionProvider` returning one `McpStdioServerDefinition` for `debrief_stac.mcp_server` (reuse `calcService` Python-interpreter resolution; catalog path from `debrief.stacCatalogPath`) `apps/vscode/src/copilot/mcpProvider.ts` +- [ ] T032 Register the provider in activation behind a capability check (`'registerMcpServerDefinitionProvider' in vscode.lm`) + try/catch (log diagnostic, LM tools unaffected) and gate on `debrief.mcp.autoStart` `apps/vscode/src/extension.ts` +- [ ] T033 Contribute `mcpServerDefinitionProviders` `apps/vscode/package.json` +- [ ] T034 [test] Provider-guard tests: absent-API host → no throw, LM tools intact; startup failure → diagnostic not error dialog (FR-021); definition targets the correct module + catalog path `apps/vscode/tests/unit/copilot/mcpProvider.test.ts` +- [ ] T035 Write the query-matrix parity script (text/time/platform/bbox × 3 fixtures) asserting MCP-served results == `searchCatalog` results (FR-020) `apps/vscode/tests/unit/copilot/mcpSearchParity.test.ts` + +**Checkpoint**: MCP server discoverable when enabled; parity green; failure non-fatal. Comparison findings note produced in Polish (needs live routing runs). + +## Phase 8: Polish & Cross-Cutting Concerns + +### Cross-cutting verification + +- [ ] T036 Run `task verify` (lint incl. drift check, typecheck, full test suite) and confirm the #284 invariants still hold: dirty-only edits, corrective fail-safes, telemetry, provenance (FR-023/024) + +### Evidence Collection + +- [ ] T037 Capture test results using template (`.specify/templates/evidence/test-summary-template.md`) `specs/285-copilot-best-practices/evidence/test-summary.md` +- [ ] T038 Create usage demonstration (select Debrief Analyst → run a `/demo-*` scenario → mutation gate under /yolo) `specs/285-copilot-best-practices/evidence/usage-example.md` +- [ ] T039 [P] Capture priming-applied screenshots in both workspaces (live preview Code Server session — see quickstart §1) `specs/285-copilot-best-practices/evidence/screenshots/priming-applied-preview.png` +- [ ] T040 [P] Capture the agent-picker + `/demo-*` slash-command screenshots `specs/285-copilot-best-practices/evidence/screenshots/agent-picker.png` +- [ ] T041 [P] Capture the mutation-gate-under-/yolo screenshot `specs/285-copilot-best-practices/evidence/screenshots/mutation-gate-yolo.png` +- [ ] T042 [P] Capture the "northern track" resolved-from-summary screenshot `specs/285-copilot-best-practices/evidence/screenshots/summary-northern-track.png` +- [ ] T043 [P] Regenerate the token-budget probe table (4 sizes × 4k/8k/32k, with digests + compact form; supersedes the #284 table) `specs/285-copilot-best-practices/evidence/token-budget-v2.md` +- [ ] T044 Produce the MCP-vs-LM-tool comparison findings note (query-matrix parity + per-config routing/latency from a live session + E13 recommendation) `specs/285-copilot-best-practices/evidence/mcp-vs-lmtool-comparison.md` + +### Media Content + +- [ ] T045 Create feature blog post (Content Specialist; first three sections verbatim from `evidence/opening-context.md`, remainder from ship-time evidence) `specs/285-copilot-best-practices/media/shipped-post.md` + +### PR Creation + +- [ ] T046 Create PR and publish blog: run `/speckit.pr` (feature PR #676 already open — updates it with evidence; opens the debrief.github.io blog PR) + +**Task T046 must run last. It depends on all evidence and media tasks (T037–T045) being complete.** + +## Dependencies + +**Story completion order** (by priority; stories are independently testable): + +- **Setup (P1)** → **Foundation (T003–T006)** block everything that uses the + drift check (US1, US2) and the prompt-tsx import (US4). US3 and US5 depend + only on Setup. +- **US1 (P1, T007–T010)** — independent once Foundation's drift script exists. +- **US2 (P1, T011–T017)** — needs Foundation (drift check + corpus). T011 + (empirical model/tool choice) precedes T012 (agent file). T013/T014 (prompts) + are [P]. T016 parity test needs T013/T014 + T006 corpus. +- **US3 (P2, T018–T021)** — independent of US1/US2/US4/US5; only needs the repo. +- **US4 (P2, T022–T030)** — needs Setup (T001 prompt-tsx). T022 compact is + standalone; T024→T025→T027→T028→T029 is the summary chain; T023/T026 are [P] + tests. No dependency on US1–US3. +- **US5 (P3, T031–T035)** — needs the existing `debrief_stac` MCP server (already + present) and `calcService` interpreter resolution. Independent of US1–US4. +- **Polish (T036–T046)** — after all stories. T046 (`/speckit.pr`) is strictly + last; live-session evidence tasks (T038–T042, T044) require a built extension. + +**Cross-file contention**: `apps/vscode/package.json` is touched by T001, T018, +T033 — sequence these (not [P]). `scripts/check-copilot-customization-drift.mjs` +is touched by T003, T010, T017 — sequence. `summarize.ts` by T024, T029 — +sequence. + +## Implementation Strategy + +**Incremental delivery** — each story is a shippable increment, and the two P1 +stories are the highest-value/lowest-risk pair (config files + a placement fix), +so they can land and be demoed first: + +1. **MVP demo-ready (US1 + US2)**: priming loads + Debrief Analyst agent + eight + `/demo-*` commands. This alone delivers the core "optimise what we're + demoing/researching" ask — reproducible demos on a fixed model and tool set. +2. **Harden (US3)**: bypass-proof mutation gate — small, independent, strengthens + the defence-posture story for the same demos. +3. **Deepen the research signal (US4)**: budget-aware summary + spatial digest — + the largest slice; improves both large-plot demos and the E13 token dataset. +4. **Strategic experiment (US5)**: MCP hybrid — highest research value per line + (server already exists), delivered as a comparison note feeding E13. + +**Parallelism**: within US2, the happy-path and fail-safe prompt authoring +(T013/T014) run in parallel; within US4, the compact-serialization and +digest-determinism tests (T023/T026) run in parallel; across stories, US3, US4, +and US5 have no interdependencies and can be worked concurrently once Setup + +Foundation land (mind the shared-file contention notes above). + +**Non-negotiable**: T036 gates on all #284 invariants surviving; T046 +(`/speckit.pr`) runs only after every evidence and media artifact exists.