feat(v1.5): MCP integration — readonly stdio server (phases 16-19)

.planning/PROJECT.md

@@ -8,6 +8,17 @@ A Go CLI tool that connects directly to Hubble Relay via gRPC, observes dropped/
 
 Automatically generate correct CiliumNetworkPolicies from observed Hubble denials so that SREs spend zero time manually writing network policies in default-deny environments.
 
+## Current Milestone: v1.5 MCP Integration
+
+**Goal:** Expose cpg as a readonly MCP server (stdio) so an LLM harness can run a live Hubble capture session, analyze dropped flows, and review generated policies — the LLM brings the intelligence, cpg stays deterministic.
+
+**Target features:**
+- `cpg mcp` subcommand — MCP server over stdio transport (the harness spawns the process)
+- Session tools: start_session / status / stop_session — background Hubble capture
+- Ephemeral session tmpdir (`os.MkdirTemp`, respects `$TMPDIR`): policies YAML + evidence + cluster-health.json written via the existing writers with existing FIFO caps — flat memory profile, no parallel in-memory path
+- Query tools: dropped flows (dropclass-classified), generated policies, explain/evidence, cluster health — all implemented as readers over the session tmpdir artifacts
+- Readonly guarantee: never mutates the cluster, never writes outside the session tmpdir; tmpdir cleaned at stop_session and server shutdown
+
 ## Requirements
 
 ### Validated
@@ -47,17 +58,32 @@ Automatically generate correct CiliumNetworkPolicies from observed Hubble denial
 - ✓ Zero reachable vulnerabilities: cilium v1.19.4, x/net v0.55.0, `toolchain go1.25.12` (govulncheck clean in CI) — v1.4
 - ✓ Genuine stream failures exit non-zero; LostEvents/PoliciesFailed counted; `--timeout` actually applied — v1.4
 - ✓ Policy-ref validation (empty/traversal) on evidence and output writers — v1.4
+- ✓ `cpg mcp` protocol-safe stdio skeleton: pure JSON-RPC stdout (IOTransport pre-swap capture + global os.Stdout→stderr backstop), zero tools registered (SRV-02) — v1.5 Phase 16
+- ✓ Unified stderr logging: zap + go-sdk logs bridged via zapslog (SRV-03) — v1.5 Phase 16
+- ✓ Atomic temp+rename policy writes in `pkg/output/writer.go` (torn-read safe for future concurrent MCP readers, SEC-02) — v1.5 Phase 16
+- ✓ `pkg/session` single-slot Manager: `capturing → stopped → gone` lifecycle with retention, idempotent stop, autonomous transition on both crash and clean pipeline exit (SESS-02, SESS-03) — v1.5 Phase 17
+- ✓ `start_session`/`get_status`/`stop_session` MCP tools wired into `cpg mcp`, opaque session_id, "session not found or expired" contract with D-02 stopped-session queryability (SESS-01, SESS-04, SESS-06) — v1.5 Phase 17
+- ✓ Bounded transport-kill cleanup: `Shutdown()` cancels session + setup contexts, per-step deadlines so one wedged cleanup cannot block process exit, tmpdir removal (SESS-05) — v1.5 Phase 17
+- ✓ `list_dropped_flows`: paginated composed view (`samples[]` live from evidence + `aggregates[]` from cluster-health.json with `available_after_stop` marker mid-capture), explicit "sampled/aggregated view, not a raw flow log" description (QRY-01) — v1.5 Phase 18
+- ✓ `list_policies` (paginated metadata) + `get_policy` (full CNP YAML + absolute tmpdir path, single atomic read) keyed by namespace+workload (QRY-02) — v1.5 Phase 18
+- ✓ `get_evidence`: paginated per-rule attribution byte-identical to `cpg explain --output json` via promoted `pkg/explain` (Filter/Output/Render*/ParsePeerLabel exported, flowsource-style promotion) (QRY-03) — v1.5 Phase 18
+- ✓ `get_cluster_health`: typed passthrough via `pkg/hubble.ReadClusterHealth`, 4-state branch (capturing → `available_after_stop`; stopped+absent+no-error → "zero drops"; crash → isError citing pipeline error), remediation URLs intact (QRY-04) — v1.5 Phase 18
+- ✓ QRY-05 contract on all 8 tools: `structuredContent`+`outputSchema` (explicit `*jsonschema.Schema` dropclass enum — go-sdk has no enum tags), truthful annotations, taxonomy-teaching descriptions, actionable `isError`; opaque fail-closed cursor; `session.DeriveSessionPaths` single source of truth for tmpdir layout — v1.5 Phase 18
+- ✓ SEC-01 structural readonly audit: `TestMCPAuditReadonlyReachability` — SSA/RTA callgraph from `runMCPServer`, BFS-filtered to cpg-owned functions, direct-call scan; K8s write verbs (incl. DeleteCollection/UpdateStatus/ApplyStatus) fail unconditionally, fs writes (incl. Chmod/Truncate/Symlink/…) gated by an exact 5-function allowlist; mutation-tested diagnostic (function symbol + call path) — v1.5 Phase 19
+- ✓ SRV-01 + SRV-04: real-subprocess stdio e2e — `-race`-built `cpg mcp` driven over real pipes against an in-process fake Hubble observer relay; graceful lifecycle (initialize → 8-tool handshake/schema/annotation proof → session + 5 query tools → stop → clean exit, byte-pure stdout) and ungraceful-disconnect variant (stdin kill → bounded self-exit, tmpdir removed, relay stream cancelled; 5/5 stability under `-race`) — v1.5 Phase 19
+- ✓ SEC-03 README `## MCP Server (cpg mcp)` section: harness `env` block (KUBECONFIG/PATH/TMPDIR + why), secrets posture (L7 paths/labels reach the LLM; headers never captured), exec-credential-plugin non-interactive caveat + credential-persistence note, session model — v1.5 Phase 19
 
 ### Active
 
-<!-- Awaiting v1.5 scoping via /gsd-new-milestone. -->
+<!-- v1.5 MCP Integration — requirements being defined via /gsd-new-milestone. -->
 
-- _(defined during `/gsd-new-milestone`)_
+- _(being defined — see REQUIREMENTS.md once written)_
 
 ### Planned
 
 <!-- v1.5 candidates: lint/release debt from the v1.4 audit + feature candidates carried over from earlier deferrals. -->
 
+- [ ] Audit-mode onboarding (default-deny sans casse) + cpg-dedicated skills/agents — v1.6 candidate — **full ideation doc: `.planning/drafts/v1.6-audit-onboarding-and-cpg-agent-tooling.md`** (verified code refs, design, open decision, research questions, candidate REQ IDs AUD-01..04 / SKL-01..05): (a) `--include-audit`/`include_audit` — étendre le filtre de verdict à `Verdict_AUDIT` (client.go:198/209/213, file.go:116, aggregator.go:417 — flows AUDIT portent le drop reason, vérifié parser threefour); (b) bootstrap: génération default-deny CNP (`enableDefaultDeny`, Cilium ≥1.15) + activation PolicyAuditMode per-endpoint namespace-scoped; (c) DÉCISION OUVERTE: mutation pilotée par cpg (flag serveur `cpg mcp --enable-audit-bootstrap`, audit = propriété de session, revert lifecycle-bound via fan-out SESS-05, watcher nouveaux pods, revert-only-ours + TTL, SEC-01 évolue en preuve 2-modes, RBAC pods/exec à documenter) vs CLI-only `cpg audit` (MCP reste readonly pur). Discuté 2026-07-22.
 - [ ] Lint debt zero: 16 errcheck + 10 staticcheck SA1019 + drop CI `only-new-issues` flag — v1.5 candidate (LINT-01..03, archived in milestones/v1.4-REQUIREMENTS.md)
 - [ ] Release hardening: `release.yml` minimal permissions review, govulncheck job pinning follow-through — v1.5 candidate (RELSEC-01..02)
 - [ ] `cpg replay` exit-code parity on truncated input (currently logs Error but exits 0; live stream exits non-zero) — v1.5 candidate (conscious call from PR #16 review)
@@ -141,14 +167,17 @@ Automatically generate correct CiliumNetworkPolicies from observed Hubble denial
 | GitHub Actions pinned to release-tag SHAs (verified via `git ls-remote`) | Mutable tags are a supply-chain risk; one agent-suggested pin pointed at an untagged branch commit — verification against real tags is part of the pin | ✓ Good — shipped v1.4 |
 | Stream failure ⇒ non-zero exit (behavior change) | Debug-logged clean exits hid mid-capture relay crashes; operators/CI must see failure | ✓ Good — shipped v1.4; replay truncation exit parity deferred (v1.5 candidate) |
 | Milestone executed via direct multi-agent workflow (no gsd plans) | Audit remediation with a complete findings inventory doesn't benefit from per-phase planning ceremony; review→verify→fix→PR pipeline replaces it | ✓ Good — v1.4 shipped same-day; keep gsd plans for feature milestones |
+| `mcp.IOTransport` with pre-swap stdout capture, never `mcp.StdioTransport{}` | StdioTransport reads package-level os.Stdout lazily inside Connect() — combined with the D-01 global swap it would bind the JSON-RPC wire to stderr and hang the server | ✓ Good — shipped v1.5 Phase 16; acceptance criteria forbid StdioTransport |
+| `go.uber.org/zap/exp` v0.3.0 as separate direct dependency | zapslog is NOT bundled in zap v1.27.1 (independently versioned module) — corrected a locked research claim via operator-approved legitimacy gate | ✓ Good — shipped v1.5 Phase 16 |
+| Seam-audit identity assertion guarded against test2json aliasing | `go test -json` makes the testing framework alias os.Stderr = os.Stdout in-process; unguarded global-identity assertions flake by mode, not by race | ✓ Good — root-caused and guarded v1.5 Phase 16 |
 
 ## Current State
 
 **Shipped:** v1.0 (2026-03-08), v1.1 (2026-04-24), v1.2 (2026-04-25), v1.3 (2026-04-26), and v1.4 (2026-07-20).
 
-**Codebase:** 10 packages (`pkg/{labels,policy,output,hubble,k8s,dedup,flowsource,evidence,diff,dropclass}` + `cmd/`). **484 tests passing with `-race`** across 10 packages (up from 418 at v1.3 close). CI green and operational for the first time (build + race tests + lint + govulncheck). Deps: cilium v1.19.4, toolchain go1.25.12. Known debt: 26 lint issues (16 errcheck + 10 SA1019) gated by `only-new-issues`, scoped to v1.5. Release-please continues to handle product SemVer tagging.
+**Codebase:** 12 packages (`pkg/{labels,policy,output,hubble,k8s,dedup,flowsource,evidence,diff,dropclass,session,explain}` + `cmd/`). **607 tests passing with `-race`** across 12 packages (up from 539 at Phase 17 close). CI green and operational (build + race tests + lint + govulncheck). Deps: cilium v1.19.4, go-sdk v1.6.1, jsonschema-go v0.4.3 (direct since Phase 18), toolchain go1.25.12. Known debt: 26 lint issues (16 errcheck + 10 SA1019) gated by `only-new-issues`, scoped to v1.5; pre-existing `pkg/session` shared-`/tmp` test flake documented in `phases/18-query-tools/deferred-items.md`. Release-please continues to handle product SemVer tagging.
 
-**Next milestone:** v1.5 — awaiting scoping. Candidates: lint debt zero, release hardening, replay exit parity, plus feature candidates in Planned.
+**Current milestone:** v1.5 MCP Integration — ALL 4 PHASES COMPLETE (16-19, 2026-07-20 → 2026-07-21). Phase 19 (Security Hardening & End-to-End Validation) closed 2026-07-21: SEC-01 audit (mutation-tested), SRV-01/SRV-04 real-stdio e2e both variants green under `-race`, SEC-03 README harness docs; verified 5/5 (one tracking-only gap closed same-day); code review (0 critical, 4 warnings) fixed same-day (audit self-check de-tautologized, verb/fs watchlists extended, subprocess t.Cleanup kill-guard). All 18 v1.5 requirements complete — milestone ready for `/gsd-complete-milestone`. Tests: 610 across 12 packages (607 at Phase 18 close + audit + 2 e2e variants), all `-race`.
 
 ## Evolution
 
@@ -168,4 +197,4 @@ This document evolves at phase transitions and milestone boundaries.
 4. Update Context with current state
 
 ---
-*Last updated: 2026-07-20 after v1.4 Audit Fable5 milestone (29 findings landed via PR #16, first green CI, 484 tests).*
+*Last updated: 2026-07-21 — v1.5 Phase 19 (Security Hardening & End-to-End Validation) completed, verified 5/5, review findings fixed; all v1.5 phases (16-19) done.*

.planning/REQUIREMENTS.md

@@ -0,0 +1,103 @@
+# Requirements: CPG — Cilium Policy Generator
+
+**Defined:** 2026-07-20
+**Milestone:** v1.5 MCP Integration
+**Core Value:** Automatically generate correct CiliumNetworkPolicies from observed Hubble denials so that SREs spend zero time manually writing network policies in default-deny environments.
+
+## v1 Requirements
+
+Requirements for milestone v1.5. Each maps to roadmap phases.
+
+### MCP Server Core
+
+- [x] **SRV-01**: SRE can register cpg in an MCP harness (`cpg mcp`, stdio transport) and the initialize handshake succeeds with all tools listed
+- [x] **SRV-02**: stdout carries only JSON-RPC frames across a full session lifecycle — enforced by explicit wiring of every stdout-defaulting seam (`PipelineConfig.Stdout`, dry-run `diffOut`, cobra `SilenceUsage`/`SilenceErrors`) and verified by an automated stdout-purity test on an in-memory transport
+- [x] **SRV-03**: All server logs go to stderr through the existing zap logger; go-sdk internal logging is bridged into it via `zap/exp/zapslog` (one unified log stream)
+- [x] **SRV-04**: An end-to-end stdio integration test drives `initialize → start_session → get_status → each query tool → stop_session → exit` under `-race`, plus an ungraceful-disconnect variant proving port-forward and tmpdir are cleaned up within a bounded deadline
+
+### Session Lifecycle
+
+- [x] **SESS-01**: LLM can `start_session` (namespace / all-namespaces + existing generate filters) and receives an opaque `session_id`; the capture pipeline runs in a background goroutine on a detached cancellable context, writing all artifacts to an ephemeral session tmpdir (`os.MkdirTemp`)
+- [x] **SESS-02**: Exactly one concurrent session: a second `start_session` while one is active is rejected with an actionable error naming the active `session_id` — never queued, never silently replaced
+- [x] **SESS-03**: LLM can `get_status(session_id)` and gets coarse state — capturing/stopped, elapsed time, artifact file counts on disk (no live pipeline counters in v1.5; documented behavior)
+- [x] **SESS-04**: LLM can `stop_session(session_id)`: pipeline context cancelled, artifacts finalized (cluster-health.json, session stats), final summary returned
+- [x] **SESS-05**: Transport termination for any reason (stdin EOF, harness crash) triggers full cleanup fan-out — cancel session context, close port-forward, remove tmpdir — each step with a bounded deadline so one wedged cleanup cannot block process exit
+- [x] **SESS-06**: Any session-scoped tool called with an unknown, purged, or replaced `session_id` returns a crisp "session not found or expired" error (SEP-2567 handle semantics); a retained STOPPED session stays queryable per D-02 (17-CONTEXT.md), resolving the SESS-06 ↔ QRY-04 (Phase 18) tension
+
+### Query Tools
+
+- [x] **QRY-01**: LLM can `list_dropped_flows(session_id, …filters)` as a composed view over the existing capped evidence samples + aggregate health counts, paginated (`limit`/`cursor`/`total_count`/`has_more`); the tool description explicitly states it is a sampled/aggregated view, not a raw flow log
+- [x] **QRY-02**: LLM can `list_policies(session_id)` (metadata: name, workload, direction, rule counts) and `get_policy(session_id, name)` (full CNP YAML + absolute tmpdir path); reads are torn-read safe against the writing pipeline
+- [x] **QRY-03**: LLM can `get_evidence(session_id, …filters)` for per-rule flow attribution, reusing the promoted `pkg/explain` JSON renderer verbatim, paginated
+- [x] **QRY-04**: LLM can `get_cluster_health(session_id)`: passthrough of cluster-health.json including per-reason Cilium remediation URLs; while the session is still capturing it returns an explicit "available after stop_session" result (not an error)
+- [x] **QRY-05**: Every data-returning tool ships `structuredContent` + `outputSchema` (typed Go structs), truthful annotations (`readOnlyHint` etc.), a description that teaches the dropclass taxonomy (policy-actionable vs infra/transient), and `isError` errors with specific actionable text
+
+### Security & Hardening
+
+- [x] **SEC-01**: Readonly guarantee is structural: the MCP composition root registers only read-path handlers — no K8s write verb reachable from `cpg mcp`, no filesystem write outside the session tmpdir — verified by an audit test, re-runnable for every future tool
+- [x] **SEC-02**: `pkg/output/writer.go` writes policy YAML atomically (temp+rename, same pattern as the evidence and health writers) — landed as an early standalone change before any query tool reads that directory
+- [x] **SEC-03**: README MCP section documents harness configuration (explicit `env` block: `KUBECONFIG`/`PATH`/`TMPDIR` — MCP hosts don't inherit the shell env), the secrets posture (HTTP paths/labels reach the LLM context; headers are never captured), and the exec-credential-plugin non-interactive hang caveat
+
+## v2 Requirements
+
+Deferred to future milestones. Tracked but not in current roadmap.
+
+### MCP Enhancements
+
+- **FLOW-01**: Dedicated flow-sample writer (4th pipeline tee target with its own FIFO cap) for a full-fidelity `list_dropped_flows` — fast-follow if the composed view proves insufficient
+- **LIVE-01**: Live mid-session counters (periodic health flush or `*SessionStats` hook on `PipelineConfig`) for `get_status`/`get_cluster_health`
+- **REDACT-01**: Best-effort redaction of HTTPPath query strings / token-like values before tool results reach the LLM
+
+### Debt (carried from v1.4 audit)
+
+- **LINT-01..03**: Lint debt zero — 16 errcheck + 10 SA1019 + drop CI `only-new-issues` flag
+- **RELSEC-01..02**: Release hardening — `release.yml` minimal permissions, govulncheck job pinning follow-through
+- **REPLAY-01**: `cpg replay` exit-code parity on truncated input
+
+## Out of Scope
+
+Explicitly excluded. Documented to prevent scope creep.
+
+| Feature | Reason |
+|---------|--------|
+| Mutating `apply_policy` tool | Breaks the readonly guarantee outright; no `cpg apply` CLI exists to wrap; structural exclusion is the SEC-01 mechanism |
+| MCP resources for policy YAML | Real client-support gap (Claude Code surfaces resources only via manual @mention); plain absolute tmpdir paths are strictly better for stdio same-filesystem deployment |
+| Elicitation & sampling | Sampling would be a new transport for the AI-plausibility feature explicitly shelved 2026-04-25; elicitation's statefulness is disproportionate for a single-user local process |
+| MCP prompt templates | No validated use case; tool descriptions carry the guidance instead |
+| Progress notifications | Correlate to a blocking request; start/poll/stop design has no blocking call to attach them to |
+| HTTP/SSE transport + OAuth | Spec scopes authorization to HTTP transports; v1.5 is stdio-only local |
+| Multi-session capacity | Single concurrent session by design (flat memory profile); `session_id` schema stays forward-compatible if this ever changes |
+
+## Traceability
+
+Which phases cover which requirements. Updated during roadmap creation.
+
+| Requirement | Phase | Status |
+|-------------|-------|--------|
+| SRV-01 | Phase 19 | Complete |
+| SRV-02 | Phase 16 | Complete |
+| SRV-03 | Phase 16 | Complete |
+| SRV-04 | Phase 19 | Complete |
+| SESS-01 | Phase 17 | Complete |
+| SESS-02 | Phase 17 | Complete |
+| SESS-03 | Phase 17 | Complete |
+| SESS-04 | Phase 17 | Complete |
+| SESS-05 | Phase 17 | Complete |
+| SESS-06 | Phase 17 | Complete |
+| QRY-01 | Phase 18 | Complete |
+| QRY-02 | Phase 18 | Complete |
+| QRY-03 | Phase 18 | Complete |
+| QRY-04 | Phase 18 | Complete |
+| QRY-05 | Phase 18 | Complete |
+| SEC-01 | Phase 19 | Complete |
+| SEC-02 | Phase 16 | Complete |
+| SEC-03 | Phase 19 | Complete |
+
+**Coverage:**
+- v1 requirements: 18 total
+- Mapped to phases: 18
+- Unmapped: 0 ✓
+
+---
+*Requirements defined: 2026-07-20*
+*Last updated: 2026-07-21 after Phase 19 (security-hardening-end-to-end-validation) completion — SRV-01, SRV-04, SEC-01, SEC-03 all complete; all 18 v1.5 requirements closed*