diff --git a/.claude/skills/cpg-policy-review/SKILL.md b/.claude/skills/cpg-policy-review/SKILL.md index ae0b10a..813be25 100644 --- a/.claude/skills/cpg-policy-review/SKILL.md +++ b/.claude/skills/cpg-policy-review/SKILL.md @@ -13,7 +13,7 @@ description: > # cpg-policy-review Router for an offline policy audit. This skill routes to `cpg explain` and -the `get_evidence` tool for evidence, and points at the README rather than +the `get_evidence` tool for evidence, and points at the docs rather than restating flag lists or field shapes. ## Step 1: Locate the target @@ -44,8 +44,8 @@ For each rule `cpg explain` surfaces, check: evidence actually shows traffic from? A rule matching a whole namespace when evidence shows only one workload talking is a candidate to tighten. - **L7 anchoring:** for HTTP/DNS rules, is the L7 detail (method, path, - matchName) actually present and correctly anchored? See the README's - [L7 Prerequisites](../../../README.md#l7-prerequisites) section for the + matchName) actually present and correctly anchored? See the + [L7 guide](../../../docs/l7-guide.md) for the capture-time requirements a rule needs before L7 attribution is even possible — do not re-derive those requirements here, route to it. - **Missing DNS-53 companions:** an egress rule permitting a `toFQDNs` peer @@ -67,7 +67,7 @@ genuinely varied evidence samples justifying the breadth. Summarize findings per rule: keep as-is, tighten (with the specific narrower selector/port/L7 filter suggested), or flag as a dedup candidate. Point the -operator at [Explain policies](../../../README.md#explain-policies) in the -README for the full worked examples of `cpg explain` usage if they want to +operator at [Explain & evidence](../../../docs/explain.md) +for the full worked examples of `cpg explain` usage if they want to reproduce a finding themselves. This skill reports; the operator decides whether and how to edit the policy YAML. diff --git a/README.md b/README.md index cfd4b06..ed225e3 100644 --- a/README.md +++ b/README.md @@ -2,22 +2,8 @@ **Cilium Policy Generator** -- because writing CiliumNetworkPolicies by hand in a default-deny cluster is nobody's idea of a good Friday night. -## Code quality - -![desloppify scorecard](docs/scorecard.png) - -Tracked with [desloppify](https://github.com/peteromallet/desloppify) — strict score, regenerated on each code-health pass. - `cpg` connects to Hubble Relay, watches dropped flows in real time, and generates the CiliumNetworkPolicy YAML files that would allow them. You run it, wait for traffic to get denied, and it writes the fix. Then you review, commit, and apply through your GitOps pipeline like a responsible adult. -## The problem - -You've deployed Cilium with default-deny. Good for you. Now every new service, every port change, every cross-namespace call gets blocked until someone writes the right policy YAML by hand. You stare at `hubble observe --verdict DROPPED`, translate flow fields into Cilium API objects in your head, and pray you got the label selectors right. - -Or you let `cpg` do it. - -## How it works - ``` Hubble Relay (gRPC) | @@ -36,8 +22,6 @@ Or you let `cpg` do it. you review & git push ``` -Flows are aggregated by namespace and workload on a configurable interval (default 5s), so you get one policy per workload -- not one per packet. Existing files are read, merged (new ports and peers appended), and only rewritten if something actually changed. - ## Install ```bash @@ -48,734 +32,48 @@ kubectl krew install cilium-policy-gen go install github.com/SoulKyu/cpg/cmd/cpg@latest ``` -Or build from source: - -```bash -git clone https://github.com/SoulKyu/cpg.git -cd cpg -make build -# binary lands in ./bin/cpg -``` - -When installed via krew, use `kubectl cilium-policy-gen` instead of `cpg`. Same flags, same behavior. - -Requires Go 1.25+ for source builds. - -## Supported Cilium versions - -cpg targets **Cilium >= 1.14**. Clusters below that floor are detected at -connect time and logged as a warning -- never blocked, so reduced-RBAC and -CI service accounts still get a working (if unverified) run. - -Individual capabilities carry their own, higher floors: - -| Feature | Cilium version | Notes | -|---------|-----------------|-------| -| Baseline cpg operation | >= 1.14 | Declared floor -- the lowest version any shipped code path assumes | -| `PolicyVerdictNotify` audit-action bit | >= 1.8 | PR [#11843](https://github.com/cilium/cilium/pull/11843) | -| `Verdict_AUDIT` via the Hubble flow API (`--include-audit`) | >= 1.10 | PR [#14785](https://github.com/cilium/cilium/pull/14785) / [#14923](https://github.com/cilium/cilium/pull/14923) | -| `cilium-dbg` binary naming (was `cilium`) | >= 1.15 | PR [#28085](https://github.com/cilium/cilium/pull/28085) | -| `enableDefaultDeny` CNP field | >= 1.16 | PR [#30572](https://github.com/cilium/cilium/pull/30572) -- used by `cpg bootstrap` / `get_bootstrap_policy`; see the [bootstrap runbook](docs/bootstrap-runbook.md) | -| `policy.cilium.io/proxy-visibility` annotation | <= 1.16 | Removed from the agent runtime at 1.17 -- PR [#35019](https://github.com/cilium/cilium/pull/35019) | -| Observer `GetNodes()` RPC / `version` field | >= 1.10 | PR [#13979](https://github.com/cilium/cilium/pull/13979) | - -`cpg bootstrap -n ` generates a namespaced default-deny `CiliumNetworkPolicy` -- -see the [bootstrap runbook](docs/bootstrap-runbook.md) for the full audit-mode onboarding -workflow (per-endpoint audit mode, `cpg generate --include-audit`, and cleanup). - -## Readonly by default - -cpg is readonly by default. Every command -- `cpg generate`, `cpg replay`, `cpg bootstrap`, -`cpg explain`, and `cpg mcp` (see [MCP Server](#mcp-server-cpg-mcp) below) -- only lists, -watches, and reads Kubernetes and Hubble data. None of them writes to the cluster. - -The one exception is `cpg audit-window`, the single scoped, lifecycle-bound mutating command: -it flips per-endpoint `PolicyAuditMode` on for the endpoints in a target namespace, watches for -newly-created endpoints and flips those too, then reverts every flip it made -- on Ctrl+C, on -`--ttl` expiry, on any exit path -- the moment the foreground command stops. It never touches an -endpoint that was already in audit mode before it started, and it never touches the daemon-wide -audit mode setting. See the [bootstrap runbook](docs/bootstrap-runbook.md) for the full -`cpg audit-window` workflow, including its new-endpoint race window. - -`cpg audit-window` requires an RBAC step-up that no readonly command needs: `pods/exec` (create, -scoped to `kube-system`) to reach each node's cilium-agent pod, and `ciliumendpoints` -(list/watch) to discover and track endpoints. Grant both verbs only to whatever principal -actually runs `cpg audit-window`. The exec transport is WebSocket with SPDY fallback (same as -kubectl since 1.30+). - -**RBAC scoping limitation:** Kubernetes RBAC `resourceNames` requires exact, static pod names, -but Cilium agent pod names are DaemonSet-generated (`cilium-`) and not stable -across restarts -- there is no way to scope `pods/exec` to "only cilium-agent pods" by name. -The grant is effectively "exec into any pod in `kube-system`" for whatever role carries it. If -your cluster runs an admission controller (Kyverno, OPA Gatekeeper), consider policy-based -scoping there as defense-in-depth -- this is not something cpg itself enforces or guarantees. - -## Audit-mode onboarding (default-deny with zero real drops) - -The full runbook lives in [docs/bootstrap-runbook.md](docs/bootstrap-runbook.md) -- this is the -short version for a namespace **with live traffic**. The trick: flipping an endpoint's -`PolicyAuditMode` is a no-op until a policy matches, so open the audit window *first* and the -default-deny apply never drops anything -- every would-be drop surfaces as an `AUDIT` verdict -instead. - -```bash -# 1. Open the audit window FIRST (dedicated terminal — foreground, auto-reverts on exit/TTL) -cpg audit-window -n production --ttl 30m - -# 2. Apply the default-deny bootstrap policy — flows become AUDIT verdicts, not drops -cpg bootstrap -n production | kubectl apply -f - - -# 3. Capture the audited traffic and generate the allow policies -cpg generate -n production --include-audit - -# 4. Review, then apply (always a human act) -cpg explain production/api-server --json -kubectl apply -f ./policies/production/ - -# 5. Close the window (Ctrl+C or let --ttl expire) — audit flips revert automatically, -# default-deny now enforces against fully-covered traffic -``` - -On a **fresh namespace** (no live traffic) you can invert steps 1 and 2 -- bootstrap first -means the namespace is protected from the very first pod, and there is nothing running to -drop. Both orders, the new-endpoint race window, and the RBAC details are covered in the -[runbook](docs/bootstrap-runbook.md). +More options (source builds, supported Cilium versions, k9s plugin) in the [installation guide](docs/installation.md). ## Quick start ```bash -# Point at a namespace. cpg auto port-forwards to hubble-relay. +# Live: point at a namespace, cpg auto port-forwards to hubble-relay cpg generate -n production -# Explicit relay address -cpg generate --server localhost:4245 - -# All namespaces, debug logging -cpg --debug generate --all-namespaces - -# TLS -cpg generate --server relay.example.com:443 --tls -n production - -# Opt-in L7: HTTP method/path + DNS matchName. See "L7 Prerequisites". -cpg generate -n production --l7 -``` - -That's it. Leave it running. Go generate some traffic (or wait for someone else to). Ctrl+C when you're done -- cpg flushes remaining flows and prints a session summary before exiting. - -Policies show up in `./policies//.yaml`. - -## Quick start (offline replay) - -Prefer to iterate on policy generation without reproducing traffic? Capture once, replay many: - -```bash -# Capture dropped flows for N minutes +# Offline: capture once, replay many hubble observe --output jsonpb --follow > drops.jsonl -# Ctrl+C when done capturing - -# Replay through cpg — reuse the file as many times as you want cpg replay drops.jsonl -n production -# Opt-in L7: HTTP method/path + DNS matchName. See "L7 Prerequisites". -cpg replay drops.jsonl --l7 -n production -``` - -`cpg replay` accepts `-` to read from stdin and transparently decompresses `.gz` files. - -## Flags - -``` -cpg generate [flags] - -Connection: - -s, --server string Hubble Relay address (auto port-forward if omitted) - --tls Enable TLS for gRPC connection - --timeout duration Connection timeout (default 10s) - -Filtering: - -n, --namespace strings Namespace filter (repeatable) - -A, --all-namespaces Observe all namespaces - --ignore-protocol strs Drop flows whose L4 protocol matches; repeatable / comma-separated. - Valid: tcp, udp, icmpv4, icmpv6, sctp (case-insensitive) - --ignore-drop-reason strs - Exclude flows by Cilium drop reason name before classification; - repeatable / comma-separated / case-insensitive. - Passing a reason already classified as infra or transient emits - a warning (it is already suppressed by default). - --include-audit Also ingest Verdict_AUDIT flows alongside DROPPED (opt-in). - Default: DROPPED-only (pre-v1.6 behavior unchanged). - -CI integration: - --fail-on-infra-drops Exit with code 1 when ≥1 infra drop is observed (default: - always exit 0). Use in CI/cron pipelines to alert on cluster - health issues. - -Output: - -o, --output-dir string Output directory (default "./policies") - -Aggregation: - --flush-interval dur Aggregation flush interval (default 5s) - -Deduplication: - --cluster-dedup Skip policies matching live cluster state (needs RBAC) - -Global: - --debug Debug logging - --log-level string Log level: debug, info, warn, error (default "info") - --json JSON log format -``` - -## What it generates - -Given a dropped ingress flow to a pod labeled `app.kubernetes.io/name: api-server` on port 8080/TCP from a pod with `app: frontend`: - -```yaml -apiVersion: cilium.io/v2 -kind: CiliumNetworkPolicy -metadata: - name: cpg-api-server - namespace: production -spec: - endpointSelector: - matchLabels: - app.kubernetes.io/name: api-server - ingress: - - fromEndpoints: - - matchLabels: - app: frontend - toPorts: - - ports: - - port: "8080" - protocol: TCP -``` - -External traffic (world identity) gets CIDR-based rules (`fromCIDR` / `toCIDR`) with /32 addresses instead of endpoint selectors, because you can't exactly match a label on the internet. - -### With `--l7` (opt-in HTTP + DNS) - -When `--l7` is set and Hubble is producing L7 flow records (see [L7 Prerequisites](#l7-prerequisites)), cpg attaches HTTP method/path and DNS `toFQDNs` to the relevant rules. Same fixture as above plus an observed `GET /api/v1/users` and a DNS query for `api.example.com`: - -```yaml -apiVersion: cilium.io/v2 -kind: CiliumNetworkPolicy -metadata: - name: cpg-api-server - namespace: production -spec: - endpointSelector: - matchLabels: - app.kubernetes.io/name: api-server - ingress: - - fromEndpoints: - - matchLabels: - app: frontend - toPorts: - - ports: - - {port: "8080", protocol: TCP} - rules: - http: - - {method: GET, path: ^/api/v1/users$} - egress: - - toFQDNs: - - matchName: api.example.com - toPorts: - - ports: - - {port: "53", protocol: UDP} - - {port: "53", protocol: TCP} - rules: - dns: - - matchName: api.example.com - # Companion kube-dns rule auto-injected for every CNP with toFQDNs (DNS-02). - - toEndpoints: - - matchLabels: - k8s-app: kube-dns - io.kubernetes.pod.namespace: kube-system - toPorts: - - ports: - - {port: "53", protocol: UDP} - - {port: "53", protocol: TCP} - rules: - dns: - - matchName: api.example.com -``` - -HTTP paths are emitted as anchored, `regexp.QuoteMeta`'d RE2 regexes (`^/api/v1/users$`). Methods are uppercase-normalized. Header / Host rules are never generated (anti-feature, see Limitations). - -## Offline replay - -`cpg replay ` feeds a Hubble jsonpb capture through the same pipeline as the live stream. It is the right tool when you want: - -- **Deterministic iteration.** Re-run the same input as you tweak label selection, dedup logic, or flush intervals. -- **Offline workflow.** Capture on a jumphost, replay on your laptop. -- **Post-mortem reproduction.** Keep the capture alongside the policy in your GitOps repo so anyone can reproduce what cpg saw. - -Capture: - -```bash -hubble observe --output jsonpb --follow > drops.jsonl -``` - -Replay: - -```bash -cpg replay drops.jsonl -n production -cpg replay drops.jsonl.gz -n production # gzip transparent -cat drops.jsonl | cpg replay - # stdin -``` - -Flags shared with `generate` (`--output-dir`, `--cluster-dedup`, `--flush-interval`, `--ignore-protocol`, `--ignore-drop-reason`, `--fail-on-infra-drops`, `--include-audit`) work identically. Non-DROPPED verdicts (unless `--include-audit` also admits AUDIT) and malformed lines are skipped with counters surfaced in the session summary. - -## L7 Prerequisites - -`cpg --l7` translates what Hubble shows it. Hubble only emits `Flow.L7` -records (HTTP method/path, DNS query) when traffic is proxied — by Envoy -for HTTP, by the DNS proxy for DNS. cpg cannot turn that on for you. -If you see the warning - -``` ---l7 set but no L7 records observed -``` - -this section is the fix. - -### Two-step workflow - -Refining L4 policies with L7 rules is a two-step dance, and the order -matters: - -1. **Deploy L4 first.** Run `cpg generate -n ` (no `--l7`). - Review the generated CiliumNetworkPolicies, commit, apply. You are - now moving toward default-deny with port/peer-level enforcement. - -2. **Enable L7 visibility** on the workloads you want refined. Three - options below — pick whichever fits your operational model. - -3. **Re-run cpg with `--l7`.** With visibility enabled, Hubble starts - emitting `Flow.L7` records and cpg attaches `rules.http` for HTTP - and `toFQDNs` + the kube-dns companion for DNS to the relevant - egress rules. - - ```bash - cpg generate --l7 -n - # or, on a captured stream - cpg replay drops.jsonl --l7 -n - ``` - -### Three ways to enable L7 visibility - -1. **Recommended for ad-hoc bootstrap — proxy-visibility annotation.** - The legacy workload-level annotation that triggers Envoy / DNS proxy - redirection without enforcing rules. Works only through **Cilium 1.16** - -- removed from the agent runtime at **1.17** (a no-op on 1.17+). See - the **Supported Cilium versions** table above for the full matrix: - - ```bash - kubectl annotate pod -n -l app.kubernetes.io/name= \ - policy.cilium.io/proxy-visibility=',' - ``` - - Easy to apply, easy to remove -- on clusters where it still works - (<= 1.16). Marked deprecated upstream before removal. - -2. **Recommended for permanent enforcement — bootstrap L7 CNP.** Ship - a starter CiliumNetworkPolicy with a permissive L7 rule. The mere - *presence* of an L7 rule on a workload triggers Cilium to proxy that - workload's traffic — match-all `{}` in the HTTP rule and `"*"` in - the DNS matchPattern lights up visibility without enforcing - anything. See the snippet below. - -3. **Cluster-wide prerequisite — `enable-l7-proxy: true`.** In the - `kube-system/cilium-config` ConfigMap. Default `true` on most - installs, but required for any of the above to work. cpg's - `--l7` pre-flight check (VIS-04) flags it explicitly when missing - or set to false. - -### Starter L7-visibility CNP - -Copy-pasteable, valid Cilium YAML. Replace the two placeholders -(namespace + workload label) before applying: - -```yaml -apiVersion: cilium.io/v2 -kind: CiliumNetworkPolicy -metadata: - name: cpg-l7-visibility-bootstrap - namespace: production # replace with your namespace -spec: - endpointSelector: - matchLabels: - app.kubernetes.io/name: my-app # replace with your workload's label - egress: - # match-all HTTP rule triggers Envoy without enforcing a path/method. - - toEndpoints: - - {} - toPorts: - - ports: - - {port: "80", protocol: TCP} - rules: - http: - - {} - # match-all DNS rule triggers DNS proxy visibility for kube-dns. - - toEndpoints: - - matchLabels: - k8s-app: kube-dns - io.kubernetes.pod.namespace: kube-system - toPorts: - - ports: - - {port: "53", protocol: UDP} - - {port: "53", protocol: TCP} - rules: - dns: - - matchPattern: "*" -``` - -Apply, observe traffic, then run `cpg --l7`. Once cpg's generated -policy covers everything you need, this bootstrap CNP can be deleted — -its only job was the visibility side-effect of Envoy / DNS proxy -injection. - -### Capture-window guidance - -Run cpg long enough to capture one full traffic cycle for the -workloads in question. A single observation produces a single rule -with `flow_count=1`, which `cpg explain` surfaces as low-confidence -evidence. For periodic batch jobs, capture across at least one period. - -### Known v1.2 limitations - -cpg v1.2 ships with a documented set of known limitations and edge cases — most are intentional trade-offs (e.g., no HTTP header rules to avoid secret leakage), a few are deferred to v1.3+. Read them **before deploying generated policies** to production: - -→ **[docs/KNOWN_LIMITATIONS.md](docs/KNOWN_LIMITATIONS.md)** — full list with workarounds and tracking IDs. - -Highlights worth knowing up front: - -- L7 visibility prerequisite — `--l7` requires Cilium Envoy proxy + per-workload visibility trigger; cpg cannot bootstrap it (limitation #1). -- HTTP path explosion on REST APIs with IDs — one literal rule per observed `(method, path)`; no auto-collapse in v1.2 (limitation #2, `HTTP-FUT-01`). -- Header-based rules never generated — anti-feature to prevent secret leakage (limitation #3). -- DNS REFUSED denials are missed — `Verdict_FORWARDED` not yet supported (limitation #4, `L7-FUT-01`). -- kube-dns companion selector hardcoded `k8s-app=kube-dns` — autodetect across CNI distributions deferred to v1.3 (limitation #6, `DNS-FUT-02`). - -## Dry-run - -Preview what `generate` or `replay` would write without touching any file: - -```bash -cpg replay drops.jsonl --dry-run # with unified diff -cpg replay drops.jsonl --dry-run --no-diff # log-only -cpg generate -n production --dry-run -``` - -In `--dry-run` mode, all stages of the pipeline run normally: you still see unhandled-flow warnings, cluster-dedup hits, and aggregation logs. Only the filesystem write step is suppressed. When an existing file would change, a unified diff is printed to stdout (colored on a tty, plain otherwise). - -## Deduplication - -cpg tries hard not to waste your time: - -- **File dedup**: if the merged result is identical to what's already on disk, it skips the write. -- **Cross-flush dedup**: if the same policy was written in a previous flush cycle, it's not rewritten. -- **Cluster dedup** (`--cluster-dedup`): fetches live CiliumNetworkPolicies from the cluster and skips policies that already match. Needs `list` RBAC on `ciliumnetworkpolicies.cilium.io`. - -## Unhandled flows - -Not every dropped flow can become a policy rule. cpg reports what it skips so you can investigate: - -- **INFO summary** at each flush cycle -- structured counters by skip reason -- **DEBUG detail** per unique flow -- logged once, with source, destination, port, protocol, and destination labels - -Enable debug logging to see individual flows: - -```bash -cpg --debug generate -n production -# or -cpg --log-level debug generate -n production -``` - -### Skip reasons - -| Reason | What it means | -|--------|---------------| -| `no_l4` | Flow has no L4 layer (no port/protocol info) | -| `nil_endpoint` | Source or destination endpoint is nil | -| `empty_namespace` | Target endpoint has no namespace (non-reserved identity) | -| `nil_source` | Ingress flow with nil source endpoint | -| `nil_destination` | Egress flow with nil destination endpoint | -| `unknown_protocol` | L4 layer present but protocol not TCP/UDP/ICMP | -| `world_no_ip` | World (external) traffic without an IP address | - -### Example output - -At INFO level (default): - -``` -INFO Unhandled flows summary {"no_l4": 42, "nil_endpoint": 8, "world_no_ip": 3} -``` - -At DEBUG level: - -``` -DEBUG Unhandled flow {"src": "default/nginx", "dst": "kube-system/coredns", "port": "53", "proto": "UDP", "reason": "no_l4", "dst_labels": ["k8s:app=coredns"]} -``` - -Reserved identity flows (like `reserved:host` or `reserved:kube-apiserver`) are reported separately as WARN logs with guidance to use CiliumClusterwideNetworkPolicy instead. - -## Explain policies - -After a run, every emitted rule has per-flow evidence recorded alongside the YAML. Inspect it with `cpg explain`: - -```bash -cpg explain production/api-server -cpg explain production/api-server --peer app=frontend -cpg explain production/api-server --ingress --port 8080 -cpg explain ./policies/production/api-server.yaml --since 1h --json - -# L7 filters (literal exact match, AND-combined). Require evidence captured with --l7. -cpg explain production/api-server --http-method GET -cpg explain production/api-server --http-method GET --http-path '^/api/v1/users$' -cpg explain production/api-server --dns-pattern api.example.com -``` - -`--http-path` matches the literal regex stored in evidence — that is, the -anchored, `regexp.QuoteMeta`'d form produced by the builder -(`^/api/v1/users$`), not the raw observed path. `--dns-pattern` matches the -literal `matchName` stored in evidence (trailing dot stripped); v1.2 emits -no wildcards, so passing `*.example.com` will simply not match anything. -When any L7 filter is set, L4-only rules (no L7Ref in evidence) are -excluded from the result. - -When the evidence carries L7 attribution, `cpg explain` renders it -inline. Text format prints a single indented line per rule -(`L7: HTTP GET /api/v1/users` or `L7: DNS api.example.com`); JSON and -YAML formats include an `l7` sub-object on each rule with the relevant -fields (`protocol`, `http_method`, `http_path`, `dns_matchname`). - -Example output: - -``` -Policy: cpg-api-server (production) -Latest session: 2026-04-24 14:02 → 14:15 (source: replay) - -Ingress rule - Peer: app=frontend (endpoint) - Port: 8080/TCP - Flow count: 23 - First seen: 2026-04-24 14:02:11 - Last seen: 2026-04-24 14:15:48 - - Sample flows: - 14:02:11 default/frontend → production/api-server TCP/8080 - 14:02:13 default/frontend → production/api-server TCP/8080 - ... -``` - -### Where is evidence stored? - -Evidence lives outside the output directory to keep GitOps clean: - -- **Linux:** `$XDG_CACHE_HOME/cpg/evidence` (defaults to `~/.cache/cpg/evidence`) -- **macOS:** `~/Library/Caches/cpg/evidence` - -The path is keyed by a hash of the absolute output directory, so multiple workspaces coexist without collision. - -To share evidence with a colleague or archive it: - -```bash -cpg replay drops.jsonl -n production --evidence-dir ./evidence -# ... ship ./evidence alongside the policies -cpg explain production/api-server --evidence-dir ./evidence -``` - -Disable capture with `--no-evidence`. Tune retention per rule with `--evidence-samples` (default 10) and per policy with `--evidence-sessions` (default 10). - -## MCP Server (cpg mcp) - -`cpg mcp` runs a readonly [MCP](https://modelcontextprotocol.io) server over stdio: your LLM harness spawns `cpg mcp` as a subprocess, and it exposes a single live Hubble capture session at a time. The LLM reads dropped flows, the CiliumNetworkPolicy YAML cpg generates, per-rule flow evidence, and cluster health through the tools below — cpg never mutates your cluster and never writes outside its own session tmpdir. - -| Tool | Description | -|------|-------------| -| `start_session` | Start a live Hubble capture session in the background; returns an opaque `session_id` immediately. Only one session at a time — stop the current one before starting another. Accepts `include_audit` to also ingest `Verdict_AUDIT` flows alongside DROPPED (opt-in; default preserves pre-v1.6 DROPPED-only behavior). | -| `get_status` | Coarse session state (capturing/stopped), elapsed time, and on-disk artifact counts. Works for a stopped-but-retained session too. | -| `stop_session` | Cancel the capture, finalize `cluster-health.json` and session stats, and return the final summary. Idempotent — a second call returns the same summary, never an error. | -| `list_dropped_flows` | Paginated, two-section view of dropped flows: policy-actionable samples plus infra/transient/noise aggregate counts. Sampled/aggregated, not a raw flow log. | -| `list_policies` | Paginated metadata for every generated CiliumNetworkPolicy in the session — namespace, workload, rule counts, YAML path. | -| `get_policy` | Full CiliumNetworkPolicy YAML plus metadata for one namespace/workload pair (discover pairs via `list_policies`). | -| `get_evidence` | Paginated per-rule flow evidence for one policy, byte-identical in shape to `cpg explain --json`. | -| `get_cluster_health` | The session's finalized cluster-health report: per-drop-reason counts by node/workload, plus Cilium-docs remediation URLs. | -| `get_bootstrap_policy` | A namespaced default-deny CiliumNetworkPolicy as YAML (same artifact as `cpg bootstrap`), returned as read-only tool content with the detected Cilium version and compat verdict — no filesystem writes. | - -### Harness configuration - -```json -{ - "mcpServers": { - "cpg": { - "command": "cpg", - "args": ["mcp"], - "env": { - "KUBECONFIG": "/home/you/.kube/config", - "PATH": "/usr/local/bin:/usr/bin:/bin", - "TMPDIR": "/tmp" - } - } - } -} -``` - -MCP hosts do not inherit your shell environment — the `env` block above is not optional decoration, it is the only environment `cpg mcp` will ever see. Set each key explicitly: - -- **`KUBECONFIG`** — cpg's client-go loader resolves clusters the same way `kubectl` does (`KUBECONFIG` env, then `~/.kube/config`, then in-cluster config). Omit it and the process silently falls back to whatever exists at the default path — which may not exist, or may point at the wrong cluster. -- **`PATH`** — if your kubeconfig authenticates via an `exec` credential plugin (see the caveat below), client-go shells out to a binary on `PATH`. Without it, that lookup fails. -- **`TMPDIR`** — the session's `os.MkdirTemp`-rooted working directory (policies, evidence, `cluster-health.json`) honors `$TMPDIR`. Leave it unset and the process falls back to the platform default, which is usually fine but worth setting explicitly if your host sandboxes `/tmp`. - -### Secrets posture - -With `--l7`-style visibility enabled on a session (see [L7 Prerequisites](#l7-prerequisites)), HTTP paths and methods, FQDNs, and workload labels reach the LLM context through tool results — that's the query tools doing their job. `Authorization`, `Cookie`, and other headers are **never** captured; cpg has never generated or stored header-based rules, an anti-feature carried forward from v1.2 specifically to avoid this kind of leakage. There is no redaction pass in v1.5 — whatever the pipeline observes is what the LLM sees, unfiltered (a dedicated redaction feature is deferred to v2). If you're pointing an MCP session at a cluster carrying sensitive path, FQDN, or label data, know what crosses the boundary before you start the session. - -### Exec-credential-plugin caveat - -Kubeconfigs authenticating via an `exec` plugin — `aws eks get-token`, `gke-gcloud-auth-plugin`, `azure kubelogin`, and similar — expect an interactive terminal for re-auth (an expired SSO session, a first-time device-code flow). Under an MCP host, `cpg mcp`'s real stdin is the JSON-RPC channel, not a keyboard: an interactive prompt has nowhere to go, and `start_session` hangs instead of failing fast. Before wiring cpg into a harness, verify headless auth actually works — running `kubectl get pods` from a non-interactive shell (no TTY) is the fastest check — or point at a static, already-authenticated kubeconfig instead. `start_session`'s bounded setup timeout turns the common exec-plugin re-auth hang (during dial/port-forward) into an actionable error rather than a silent wait — the one known exception is a hang specifically inside kubeconfig load itself (`k8s.LoadKubeConfig()` takes no `ctx`), which neither this timeout nor cancellation can bound. One more thing worth knowing: some `exec`/OIDC plugins refresh and persist credentials back to the kubeconfig file on disk as a side effect of successful auth — expected client-go behavior, not something cpg itself does, but worth knowing if you otherwise treat that file as read-only. - -### Session model - -One capture session at a time — `start_session` returns an error if a session is already running. A stopped session isn't discarded: `get_status` and `stop_session` keep returning its final state, and the query tools keep serving its artifacts, until the next `start_session` call or the server process exits. - -## Agent tooling - -Repo-local Claude Code skills and agent under `.claude/` route an LLM operator through cpg-specific workflows on top of `cpg mcp`'s tool surface. Skills name workflow steps and tool names only — argument schemas and result shapes are always discovered live via `tools/list`, never restated in skill prose. - -| Skill | Purpose | -|-------|---------| -| `cpg-triage` | Drive a live MCP session end-to-end: start capture, classify dropped flows, present each generated policy with its evidence, recommend what to apply. | -| `cpg-audit-onboard` | Guide onboarding a new namespace: bootstrap and audit-window (human-run CLI), drive capture via MCP, end with the human applying policy. | -| `cpg-policy-review` | Audit already-generated policies offline via `cpg explain` and evidence — over-broad rules, L7 anchoring, DNS-53 companions, dedup. | -| `cpg-health-report` | Turn a session's `cluster-health.json` into a self-contained HTML report of infra drops by node/workload with remediation links. | -| `cpg-mcp-smoke` | Post-release smoke test of a tagged binary's MCP server against the fake-relay e2e harness. | - -`cpg-operator` (`.claude/agents/cpg-operator.md`) is the single repo-local agent that drives live MCP session lifecycles on behalf of `cpg-triage` and `cpg-audit-onboard` — it is not invoked directly. - -## Label selection - -Labels are chosen with a priority hierarchy: - -1. `app.kubernetes.io/name` if present (Kubernetes standard) -2. `app` if present (common convention) -3. All labels minus a denylist (pod-template-hash, controller-revision-hash, etc.) - -This means generated policies survive rolling updates and don't accidentally pin to a specific ReplicaSet. - -## Auto port-forward - -When you omit `--server`, cpg finds the `hubble-relay` pod in `kube-system` using your kubeconfig and sets up a port-forward automatically. One less terminal tab to manage. - -## k9s plugin - -You can trigger cpg directly from k9s on a namespace. Drop this into `$XDG_CONFIG_HOME/k9s/plugins.yaml` (usually `~/.config/k9s/plugins.yaml`): - -```yaml -plugins: - cpg: - shortCut: Shift-G - description: Generate Cilium policies from dropped flows - scopes: - - namespace - command: cpg - background: false - args: - - generate - - -n - - $NAME - - --cluster-dedup -``` - -Navigate to a namespace in k9s, press `Shift-G`, and cpg starts streaming dropped flows for that namespace. Ctrl+C to stop -- policies land in `./policies//`. - -If you installed via krew instead of `go install`, replace `command: cpg` with `command: kubectl` and prepend `cilium-policy-gen` to the args: - -```yaml -plugins: - cpg: - shortCut: Shift-G - description: Generate Cilium policies from dropped flows - scopes: - - namespace - command: kubectl - background: false - args: - - cilium-policy-gen - - generate - - -n - - $NAME - - --cluster-dedup -``` - -## Project structure - -``` -cmd/cpg/ CLI entrypoint (cobra): generate, replay, explain -pkg/labels/ Label selection, denylist, endpoint/peer selector builders -pkg/policy/ Flow-to-CiliumNetworkPolicy builder, merge, semantic dedup, attribution -pkg/output/ Directory-organized YAML writer with merge-on-write -pkg/hubble/ Live gRPC client, aggregator, pipeline orchestration -pkg/k8s/ Kubeconfig loading, port-forward, cluster policy fetching, version detection -pkg/flowsource/ Flow stream abstraction: live gRPC or jsonpb file source -pkg/evidence/ Per-rule flow attribution (cpg explain) -pkg/diff/ Unified YAML diff (cpg generate/replay --dry-run) -``` - -## Development - -```bash -make test # run tests with race detector -make lint # golangci-lint -make build # build binary to ./bin/cpg -make all # lint + test + build -``` - -The test suite covers label selection, policy building, merging, output writing, flow aggregation, pipeline orchestration, and dedup logic. No live cluster required -- the Hubble gRPC client is mocked via interfaces. - -## Exit codes - -| Code | Meaning | -|------|-------------------------------------------------------------------------| -| 0 | Success — policies generated (or previewed). Default even with infra drops. | -| 1 | `--fail-on-infra-drops` was set **and** ≥1 infra drop was observed. | - -Any other non-zero exit means cpg encountered a fatal error (connection -failure, bad flag, etc.). - -### CI / cron example - -```bash -# Alert when infra drops appear in a captured window -cpg replay /tmp/last-hour.jsonl --fail-on-infra-drops \ - || alert-team "cpg detected infra drops — check cluster-health.json" -``` - -With `cpg generate` (live stream — run for a fixed window with timeout): - -> Note: `--preserve-status` ensures `timeout` propagates `cpg`'s exit code (0 vs 1) instead of -> returning 124 when the deadline is reached. Without it, a CI job that hits the timeout would -> mask whether infra drops were detected. - -```bash -timeout --preserve-status 300 cpg generate -n production --fail-on-infra-drops \ - || alert-team "infra drops in production — see cluster-health.json" +# Opt-in L7 (HTTP method/path + DNS) +cpg generate -n production --l7 ``` -## Limitations +Leave it running, generate some traffic, Ctrl+C when done. Policies land in `./policies//.yaml` -- reviewing and applying them stays your job (no auto-apply, ever). -Honest ones: +Onboarding a namespace to default-deny with zero real drops? That's the [bootstrap runbook](docs/bootstrap-runbook.md). -- **L4 by default; L7 opt-in via `--l7`.** Without the flag, cpg generates port-level policies (v1.1 byte-stable). With `--l7`, cpg attaches HTTP `method` + anchored regex `path` and DNS `toFQDNs` (literal `matchName`) to the matching rules. Several L7 features are intentionally deferred to v1.3: - - No HTTP `Headers` / `Host` / `HostExact` rules (anti-feature: secret leakage into committed YAML). - - No HTTP path templating / auto-collapse — one rule per observed `(method, path)` pair (`--l7-collapse-paths`, HTTP-FUT-01). - - No DNS `matchPattern` glob inference — only literal `matchName` (`--l7-fqdn-wildcard-depth`, DNS-FUT-01). - - No FQDN inference from L4-to-IP correlation (DNS-FUT-03). - - REFUSED DNS denials surface as `Verdict_FORWARDED` and are missed (`--include-l7-forwarded`, L7-FUT-01). +## Readonly by default - See [L7 Prerequisites](#l7-prerequisites) for the two-step workflow and the starter visibility CNP. -- **No auto-apply.** cpg writes YAML files. Applying them is your job, presumably through whatever GitOps tooling you already have. This is intentional -- auto-applying network policies in production is how you get paged at 3am. -- **Namespace-scoped only.** It generates CiliumNetworkPolicy, not CiliumClusterwideNetworkPolicy. Cluster-wide policies are typically hand-crafted by platform teams who know what they're doing (allegedly). -- **Named ports aren't resolved.** You get port numbers, not service port names. Port 8080 is port 8080. Less ambiguity, more grep-ability. +Every command only lists, watches, and reads Kubernetes and Hubble data -- none writes to the cluster. The single exception is `cpg audit-window`, a scoped, lifecycle-bound command that flips per-endpoint `PolicyAuditMode` and reverts every flip on exit. Full guarantees and RBAC details in the [security model](docs/security.md). + +## Documentation + +Full documentation lives in [`docs/`](docs/README.md): + +| | | +|---|---| +| [Installation](docs/installation.md) | krew, `go install`, source builds, supported Cilium versions, k9s plugin | +| [Getting started](docs/getting-started.md) | First live capture, offline replay, audit-mode onboarding | +| [Bootstrap runbook](docs/bootstrap-runbook.md) | Default-deny onboarding with `cpg bootstrap` + `cpg audit-window` | +| [Audit mode](docs/audit-mode.md) | How the audit-window, bootstrap, and `--include-audit` pieces fit together | +| [Policy generation](docs/policy-generation.md) | Generated YAML examples, label selection, dedup, unhandled flows | +| [L7 guide](docs/l7-guide.md) | HTTP/DNS rules with `--l7`: prerequisites and visibility bootstrap | +| [Explain & evidence](docs/explain.md) | `cpg explain` -- the flow evidence behind every rule | +| [MCP server](docs/mcp-server.md) | `cpg mcp` for LLM harnesses: tools, configuration, secrets posture | +| [Security model](docs/security.md) | Readonly guarantees and RBAC requirements | +| [CLI reference](docs/cli-reference.md) | All flags, dry-run, exit codes, CI/cron integration | +| [Known limitations](docs/KNOWN_LIMITATIONS.md) | Honest list with workarounds and tracking IDs | +| [Development](docs/development.md) | Project structure, build/test targets, code quality scorecard | ## License diff --git a/cmd/cpg/audit_docs_test.go b/cmd/cpg/audit_docs_test.go index 6803f01..0280bb2 100644 --- a/cmd/cpg/audit_docs_test.go +++ b/cmd/cpg/audit_docs_test.go @@ -12,36 +12,41 @@ import ( // criterion 5 (see // .planning/phases/23-managed-audit-window-sec-01-evolution/23-04-PLAN.md). // -// It pins the fact that README.md keeps the MCP server's "readonly, period" -// claim byte-for-byte AND documents `cpg audit-window` as the one scoped, -// lifecycle-bound mutating command with its exclusive RBAC step-up -// (pods/exec create, ciliumendpoints list/watch) -- so a future edit cannot -// silently drop either half of that honest disclosure. +// It pins the fact that docs/mcp-server.md keeps the MCP server's +// "readonly, period" claim byte-for-byte AND docs/security.md documents +// `cpg audit-window` as the one scoped, lifecycle-bound mutating command +// with its exclusive RBAC step-up (pods/exec create, ciliumendpoints +// list/watch) -- so a future edit cannot silently drop either half of that +// honest disclosure. (Originally pinned against README.md; the prose moved +// to docs/ when the README was slimmed down to an overview.) // // Assertions use strings.Contains over the file content directly (not shell // grep), mirroring readme_compat_test.go's style, so the test runs // deterministically with no cluster, no build tags, and no external // process. func TestReadmeAuditWindowSection(t *testing.T) { - data, err := os.ReadFile("../../README.md") - require.NoError(t, err, "README.md must be readable at the repo root") - readme := string(data) + mcpData, err := os.ReadFile("../../docs/mcp-server.md") + require.NoError(t, err, "docs/mcp-server.md must exist and be readable") - assert.Contains(t, readme, + assert.Contains(t, string(mcpData), "cpg never mutates your cluster and never writes outside its own session tmpdir", - "T-23-11: README must keep the MCP server's readonly claim verbatim") + "T-23-11: docs/mcp-server.md must keep the MCP server's readonly claim verbatim") - assert.Contains(t, readme, "cpg audit-window", - "AUD-03 c5: README must document cpg audit-window as the mutating command") + secData, err := os.ReadFile("../../docs/security.md") + require.NoError(t, err, "docs/security.md must exist and be readable") + security := string(secData) - assert.Contains(t, readme, "PolicyAuditMode", - "AUD-03 c5: README must name the per-endpoint PolicyAuditMode option") + assert.Contains(t, security, "cpg audit-window", + "AUD-03 c5: security model must document cpg audit-window as the mutating command") - assert.Contains(t, readme, "pods/exec", - "AUD-03 c5: README must list the pods/exec RBAC step-up") + assert.Contains(t, security, "PolicyAuditMode", + "AUD-03 c5: security model must name the per-endpoint PolicyAuditMode option") - assert.Contains(t, readme, "ciliumendpoints", - "AUD-03 c5: README must list the ciliumendpoints RBAC step-up") + assert.Contains(t, security, "pods/exec", + "AUD-03 c5: security model must list the pods/exec RBAC step-up") + + assert.Contains(t, security, "ciliumendpoints", + "AUD-03 c5: security model must list the ciliumendpoints RBAC step-up") } // TestRunbookAuditWindowStep is the golden pinning test for AUD-03 diff --git a/cmd/cpg/readme_compat_test.go b/cmd/cpg/readme_compat_test.go index 82b096f..f14cf59 100644 --- a/cmd/cpg/readme_compat_test.go +++ b/cmd/cpg/readme_compat_test.go @@ -12,8 +12,9 @@ import ( // TestReadmeCompatSection is the golden consistency test for COMPAT-01 and // COMPAT-03 (see .planning/phases/21-cilium-compatibility-matrix-runtime-detection). // -// It pins two facts about README.md's user-facing Cilium compatibility prose -// so a future edit cannot silently regress them: +// It pins two facts about the user-facing Cilium compatibility prose in +// docs/installation.md (originally in README.md; moved when the README was +// slimmed down to an overview) so a future edit cannot silently regress them: // // 1. COMPAT-01: a "## Supported Cilium versions" section exists, declaring // the documented floor (1.14) plus the PR-verified per-feature floor @@ -29,16 +30,16 @@ import ( // the check, and so the test runs deterministically with no cluster, no // build tags, and no external process. func TestReadmeCompatSection(t *testing.T) { - data, err := os.ReadFile("../../README.md") - require.NoError(t, err, "README.md must be readable at the repo root") + data, err := os.ReadFile("../../docs/installation.md") + require.NoError(t, err, "docs/installation.md must exist and be readable") readme := string(data) assert.Contains(t, readme, "## Supported Cilium versions", - "COMPAT-01: README must declare the 'Supported Cilium versions' section") + "COMPAT-01: installation guide must declare the 'Supported Cilium versions' section") for _, tok := range []string{"1.14", "1.15", "1.16", "1.17"} { assert.Contains(t, readme, tok, - "COMPAT-01: README compat section must carry the PR-verified version token %q", tok) + "COMPAT-01: compat section must carry the PR-verified version token %q", tok) } // T-21-02-01 mitigation: pin the merged-PR citations backing the three @@ -47,17 +48,17 @@ func TestReadmeCompatSection(t *testing.T) { // just the bare version numbers. for _, pr := range []string{"#28085", "#30572", "#35019"} { assert.Contains(t, readme, pr, - "COMPAT-01: README compat table must cite merged PR %q backing its version claim", pr) + "COMPAT-01: compat table must cite merged PR %q backing its version claim", pr) } // AUD-02 c5 (22-03-PLAN.md): the existing enableDefaultDeny/1.16 compat // row must stay cross-referenced to `cpg bootstrap` and the runbook it // links to, rather than regressing to a bare, uncontextualized PR // citation or -- worse -- fragmenting into a duplicate row. - assert.Contains(t, readme, "docs/bootstrap-runbook.md", - "AUD-02 c5: README must link docs/bootstrap-runbook.md") + assert.Contains(t, readme, "bootstrap-runbook.md", + "AUD-02 c5: compat section must link the bootstrap runbook") assert.Contains(t, readme, "cpg bootstrap", - "AUD-02 c5: README must mention cpg bootstrap alongside its enableDefaultDeny compat row") + "AUD-02 c5: compat section must mention cpg bootstrap alongside its enableDefaultDeny compat row") lines := strings.Split(readme, "\n") @@ -76,7 +77,7 @@ func TestReadmeCompatSection(t *testing.T) { // states the real 1.16/1.17 removal boundary within its own prose // block (the lines from that mention down to the next blank line). assert.True(t, proxyVisibilityBoundaryStated(lines), - "COMPAT-03: README must state the proxy-visibility removal boundary "+ + "COMPAT-03: installation guide must state the proxy-visibility removal boundary "+ "(1.16 or 1.17) in the same paragraph as a 'proxy-visibility' mention") } diff --git a/cmd/cpg/replay_test.go b/cmd/cpg/replay_test.go index c2609ad..537e7cc 100644 --- a/cmd/cpg/replay_test.go +++ b/cmd/cpg/replay_test.go @@ -330,7 +330,7 @@ func TestReplay_L7HTTP_DisabledByteStable(t *testing.T) { // TestReplay_L7HTTP_EmptyFixtureFiresWarning asserts VIS-01: when --l7 is set // against an L4-only fixture, the pipeline emits exactly one warning -// referencing #l7-prerequisites and produces no http: block in the output. +// referencing docs/l7-guide.md and produces no http: block in the output. func TestReplay_L7HTTP_EmptyFixtureFiresWarning(t *testing.T) { logs := initObservedLoggerForTesting(t) @@ -357,7 +357,7 @@ func TestReplay_L7HTTP_EmptyFixtureFiresWarning(t *testing.T) { fields := e.ContextMap() hint, ok := fields["hint"].(string) assert.True(t, ok, "hint field must be a string") - assert.Contains(t, hint, "#l7-prerequisites", "hint must reference README anchor") + assert.Contains(t, hint, "docs/l7-guide.md", "hint must reference the L7 guide") assert.Contains(t, e.Message, "--l7", "warning must reference --l7 flag verbatim") if ws, ok := fields["workloads"].([]interface{}); ok { assert.NotEmpty(t, ws, "workloads must be non-empty") diff --git a/cmd/cpg/skills_test.go b/cmd/cpg/skills_test.go index 4b21f34..2fdf899 100644 --- a/cmd/cpg/skills_test.go +++ b/cmd/cpg/skills_test.go @@ -22,14 +22,14 @@ const wantToolCount = 9 // identifiers — every one of the 9 current MCP tool names starts with // start_/get_/stop_/list_ (see 24-RESEARCH.md "MCP Tool Reference"). Skill // authors are required (by this test) to always backtick-quote tool names -// in prose, mirroring the convention already visible in README.md's MCP -// tool table. Deliberately narrow so it excludes arg-name prose like +// in prose, mirroring the convention already visible in docs/mcp-server.md's +// MCP tool table. Deliberately narrow so it excludes arg-name prose like // all_namespaces. var toolNameToken = regexp.MustCompile("`((?:start|get|stop|list)_[a-z_]+)`") // TestSkillsConsistencyTripwire is the compiled mitigation for the exact // drift Phase 24 targets: a future MCP tool added without a corresponding -// skill/agent/README sweep. It enumerates the live tool registry (never a +// skill/agent/docs sweep. It enumerates the live tool registry (never a // hardcoded list), then cross-checks it against the six markdown artifacts // authored in plan 24-01. func TestSkillsConsistencyTripwire(t *testing.T) { @@ -65,12 +65,12 @@ func TestSkillsConsistencyTripwire(t *testing.T) { "cpg-mcp-smoke/SKILL.md must mention every registered tool (coverage floor); missing %q", name) } - readmeData, err := os.ReadFile("../../README.md") - require.NoError(t, err, "README.md must be readable at the repo root") + readmeData, err := os.ReadFile("../../docs/mcp-server.md") + require.NoError(t, err, "docs/mcp-server.md must exist and be readable") readme := string(readmeData) assert.Contains(t, readme, "## Agent tooling", - "README must declare the 'Agent tooling' section") + "docs/mcp-server.md must declare the 'Agent tooling' section") for _, skill := range []string{ "cpg-triage", @@ -80,7 +80,7 @@ func TestSkillsConsistencyTripwire(t *testing.T) { "cpg-mcp-smoke", } { assert.Contains(t, readme, skill, - "README '## Agent tooling' section must list skill %q", skill) + "docs/mcp-server.md 'Agent tooling' section must list skill %q", skill) } } diff --git a/docs/KNOWN_LIMITATIONS.md b/docs/KNOWN_LIMITATIONS.md index 3538f29..ed31d7d 100644 --- a/docs/KNOWN_LIMITATIONS.md +++ b/docs/KNOWN_LIMITATIONS.md @@ -16,7 +16,7 @@ For the full list of upcoming features, see [`.planning/PROJECT.md`](../.plannin cpg cannot bootstrap visibility itself. Without these prerequisites, `--l7` has nothing to read. -**Workaround:** Follow the two-step workflow documented in the README under [L7 Prerequisites](../README.md#l7-prerequisites). Deploy the starter L7-visibility CNP first, capture again, then re-run `cpg generate --l7`. +**Workaround:** Follow the two-step workflow documented in the [L7 guide](l7-guide.md). Deploy the starter L7-visibility CNP first, capture again, then re-run `cpg generate --l7`. **Tracking:** No issue — by-design constraint of Cilium architecture. Future UX improvement (`cpg setup-visibility ` interactive bootstrap) is on the v1.3 wishlist but not committed. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..0db0387 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,32 @@ +# cpg documentation + +**cpg** (Cilium Policy Generator) watches dropped flows through Hubble Relay and generates +the CiliumNetworkPolicy YAML that would allow them. This is the full documentation — +for the elevator pitch, see the [project README](../README.md). + +## Getting started + +| Guide | What's inside | +|-------|---------------| +| [Installation](installation.md) | krew, `go install`, source builds, supported Cilium versions, k9s plugin | +| [Getting started](getting-started.md) | First live capture, offline replay, auto port-forward | +| [Bootstrap runbook](bootstrap-runbook.md) | Onboarding a namespace to default-deny with zero real drops (audit-window workflow) | + +## Guides + +| Guide | What's inside | +|-------|---------------| +| [Policy generation](policy-generation.md) | The pipeline, generated YAML examples, label selection, deduplication, unhandled flows | +| [Audit mode](audit-mode.md) | How `cpg audit-window`, `cpg bootstrap`, and `--include-audit` fit together; lifecycle guarantees | +| [L7 guide](l7-guide.md) | HTTP/DNS rules with `--l7`: prerequisites, the two-step workflow, visibility bootstrap | +| [Explain & evidence](explain.md) | `cpg explain` — per-rule flow evidence behind every generated rule | +| [MCP server](mcp-server.md) | `cpg mcp` — readonly MCP server for LLM harnesses, tool catalog, harness configuration | +| [Security model](security.md) | Readonly-by-default guarantees, the one mutating command, RBAC requirements | + +## Reference + +| Reference | What's inside | +|-----------|---------------| +| [CLI reference](cli-reference.md) | All flags, dry-run mode, exit codes, CI/cron integration | +| [Known limitations](KNOWN_LIMITATIONS.md) | Documented limitations and edge cases, with workarounds and tracking IDs | +| [Development](development.md) | Project structure, build and test targets | diff --git a/docs/audit-mode.md b/docs/audit-mode.md new file mode 100644 index 0000000..63fbb84 --- /dev/null +++ b/docs/audit-mode.md @@ -0,0 +1,110 @@ +# Audit mode + +This page explains how cpg's audit-mode support works and how the pieces fit together. +For the step-by-step onboarding procedure, use the [bootstrap runbook](bootstrap-runbook.md). + +## What audit mode is + +Cilium's `PolicyAuditMode` is a per-endpoint switch: when enabled, policy violations are +**reported instead of enforced**. Traffic that a matching policy would drop keeps flowing, +and Hubble surfaces it as an `AUDIT` policy verdict instead of `DROPPED`. + +That is exactly the signal cpg wants when onboarding a namespace to default-deny: apply the +deny policy, let real traffic run, and read the `AUDIT` verdicts as the list of allow rules +you still need — with **zero real drops** along the way. + +> **Never enable daemon-wide `policy-audit-mode`.** The cluster-wide ConfigMap flag disables +> enforcement for every endpoint on every node until someone flips it back. Everything cpg +> does uses the **per-endpoint** form, scoped to the endpoints being onboarded. The +> [runbook](bootstrap-runbook.md) opens with the full warning. + +## The three building blocks + +Milestone v1.6 shipped audit-mode support as three composable pieces: + +| Piece | Role | +|-------|------| +| `cpg bootstrap -n ` | Emits the namespaced default-deny CiliumNetworkPolicy (stdout-only, you apply it) | +| `cpg audit-window -n --ttl 30m` | Opens a managed, TTL-bounded per-endpoint audit window — the one mutating command | +| `--include-audit` | Makes `cpg generate` / `cpg replay` ingest `Verdict_AUDIT` flows alongside `DROPPED` | + +Combined: open the window, apply default-deny, capture with `--include-audit`, review and +apply the generated policies, close the window. The [runbook](bootstrap-runbook.md) walks +each step in order (and explains why the order differs between a live-traffic namespace and +a fresh one). + +## `cpg audit-window` lifecycle + +`cpg audit-window` is a foreground, supervised command: + +1. **Discover** every `CiliumEndpoint` in the target namespace. +2. **Flip** `PolicyAuditMode` on each one, via `pods/exec` into that endpoint's node's + cilium-agent pod (the same passthrough `kubectl exec` uses; WebSocket transport with + SPDY fallback, matching kubectl 1.30+). +3. **Watch** for newly-created endpoints in the namespace and flip those too as they appear. +4. **Revert** every flip it made — on Ctrl+C, on SIGTERM, on `--ttl` expiry, on any exit + path — before the process exits. There is no separate disable step to forget. + +Guarantees worth knowing: + +- **Always bounded.** The window has a TTL on every run (default 30m); there is no + unbounded mode. When it expires, the reverts run and the command exits. +- **Never touches pre-existing audit state.** An endpoint already in audit mode before the + window opened is left exactly as found. +- **Never touches the daemon-wide setting.** Only per-endpoint flips, only in the target + namespace. +- **Reverts are reported.** If a revert cannot land before the shutdown deadline, the + affected endpoints are reported as possibly stuck rather than silently abandoned — + verify those manually (the runbook's [Verify Enforcement](bootstrap-runbook.md#verify-enforcement) + section shows the `cilium-dbg` check). + +### The new-endpoint race (documented, not solved) + +A pod whose `CiliumEndpoint` the watcher has not yet observed and flipped is still enforcing +for that brief interval — its traffic can be dropped instead of audited. The watch typically +reacts within the same second, but there is no hard upper bound (apiserver load, watch +reconnects). For bursty scale-ups, wait a few seconds after scaling before generating the +traffic you want captured. + +### RBAC + +`cpg audit-window` is the one command needing a step-up beyond readonly: `pods/exec` +(create, in `kube-system`) and `ciliumendpoints` (list/watch). The +[security model](security.md#rbac-requirements) covers the grants and the scoping +limitation that comes with `pods/exec`. + +## Capturing audit verdicts + +`--include-audit` is **opt-in** on both capture surfaces — the default stays `DROPPED`-only, +matching pre-v1.6 behavior: + +```bash +cpg generate -n --include-audit +cpg replay drops.jsonl --include-audit -n +``` + +On the MCP server, `start_session` accepts the equivalent `include_audit` parameter, and the +session's final summary surfaces the audit-verdict count alongside the other session stats — +see the [MCP server guide](mcp-server.md). + +If `--include-audit` is set but zero `AUDIT` flows materialize while flows were observed, +cpg emits a single warning at the end of the run (AUD-01) — the usual cause is capturing +before the default-deny policy was applied (flips are no-ops until a policy matches) or +after the audit window closed. + +## Version requirements + +| Capability | Cilium version | +|------------|----------------| +| `PolicyVerdictNotify` audit-action bit | >= 1.8 | +| `Verdict_AUDIT` via the Hubble flow API (`--include-audit`) | >= 1.10 | +| `enableDefaultDeny` CNP field (`cpg bootstrap`) | >= 1.16 | + +Full matrix with PR citations in the +[Supported Cilium versions](installation.md#supported-cilium-versions) table. + +## Related + +- [Bootstrap runbook](bootstrap-runbook.md) — the step-by-step onboarding procedure +- [Security model](security.md) — why audit-window is the single mutating command +- [Getting started](getting-started.md#audit-mode-onboarding-default-deny-with-zero-real-drops) — the five-command short version diff --git a/docs/bootstrap-runbook.md b/docs/bootstrap-runbook.md index 5d954be..e24bef1 100644 --- a/docs/bootstrap-runbook.md +++ b/docs/bootstrap-runbook.md @@ -19,14 +19,14 @@ from observed traffic instead of hand-writing it. ## Prerequisites -- `cpg` installed and on `PATH` (see the main [README](../README.md#install)). +- `cpg` installed and on `PATH` (see the [installation guide](installation.md)). - A working `kubeconfig` pointed at the target cluster, with the same RBAC `cpg generate` already needs (`pods` list/get, `ciliumnetworkpolicies` read, and -- for this runbook -- `create`/ `apply` once you're ready to land the generated policy). - Cilium **>= 1.16** on the target cluster. `cpg bootstrap` detects the cluster's Cilium version and refuses to emit an artifact below that floor, because the `enableDefaultDeny` field it relies on is silently pruned by the CRD schema on older clusters (see the - [Supported Cilium versions](../README.md#supported-cilium-versions) table). If version + [Supported Cilium versions](installation.md#supported-cilium-versions) table). If version detection fails (no reachable cluster), `cpg bootstrap` warns and proceeds -- useful for offline/CI artifact generation, but confirm the target cluster's version yourself before applying anything it produces. @@ -71,7 +71,7 @@ access: exactly the state the rest of this runbook safely fills in. Prefer to review before applying? Redirect the artifact to a file instead of piping it (`cpg bootstrap -n > default-deny.yaml`), inspect it, then `kubectl apply -f` it yourself. The -[MCP](../README.md#mcp-server-cpg-mcp) `get_bootstrap_policy` tool returns the same YAML as +[MCP](mcp-server.md) `get_bootstrap_policy` tool returns the same YAML as read-only tool-result content, for harnesses that want to inspect it programmatically before an operator applies it. @@ -126,8 +126,8 @@ generating the traffic you want captured. **RBAC:** `cpg audit-window` needs `pods/exec` (create, in `kube-system`) to reach each node's cilium-agent, and `ciliumendpoints` (list/watch) to discover endpoints -- a step-up beyond -anything else this runbook or `cpg generate` needs. See the main README's -[Readonly by default](../README.md#readonly-by-default) section for the full RBAC posture and +anything else this runbook or `cpg generate` needs. See the +[security model](security.md) for the full RBAC posture and its scoping limitation. If you need the underlying `$ENDPOINT`/`$CILIUM_POD` values for the Hubble observation step diff --git a/docs/cli-reference.md b/docs/cli-reference.md new file mode 100644 index 0000000..a48563a --- /dev/null +++ b/docs/cli-reference.md @@ -0,0 +1,92 @@ +# CLI reference + +## Flags + +``` +cpg generate [flags] + +Connection: + -s, --server string Hubble Relay address (auto port-forward if omitted) + --tls Enable TLS for gRPC connection + --timeout duration Connection timeout (default 10s) + +Filtering: + -n, --namespace strings Namespace filter (repeatable) + -A, --all-namespaces Observe all namespaces + --ignore-protocol strs Drop flows whose L4 protocol matches; repeatable / comma-separated. + Valid: tcp, udp, icmpv4, icmpv6, sctp (case-insensitive) + --ignore-drop-reason strs + Exclude flows by Cilium drop reason name before classification; + repeatable / comma-separated / case-insensitive. + Passing a reason already classified as infra or transient emits + a warning (it is already suppressed by default). + --include-audit Also ingest Verdict_AUDIT flows alongside DROPPED (opt-in). + Default: DROPPED-only (pre-v1.6 behavior unchanged). + +CI integration: + --fail-on-infra-drops Exit with code 1 when ≥1 infra drop is observed (default: + always exit 0). Use in CI/cron pipelines to alert on cluster + health issues. + +Output: + -o, --output-dir string Output directory (default "./policies") + +Aggregation: + --flush-interval dur Aggregation flush interval (default 5s) + +Deduplication: + --cluster-dedup Skip policies matching live cluster state (needs RBAC) + +Global: + --debug Debug logging + --log-level string Log level: debug, info, warn, error (default "info") + --json JSON log format +``` + +`cpg replay` shares `--output-dir`, `--cluster-dedup`, `--flush-interval`, `--ignore-protocol`, +`--ignore-drop-reason`, `--fail-on-infra-drops`, and `--include-audit` with `generate` — they +work identically. It also accepts `-` to read from stdin and transparently decompresses `.gz` files. + +Evidence flags (`--evidence-dir`, `--no-evidence`, `--evidence-samples`, `--evidence-sessions`) +and the `cpg explain` filters are covered in [Explain & evidence](explain.md). + +## Dry-run + +Preview what `generate` or `replay` would write without touching any file: + +```bash +cpg replay drops.jsonl --dry-run # with unified diff +cpg replay drops.jsonl --dry-run --no-diff # log-only +cpg generate -n production --dry-run +``` + +In `--dry-run` mode, all stages of the pipeline run normally: you still see unhandled-flow warnings, cluster-dedup hits, and aggregation logs. Only the filesystem write step is suppressed. When an existing file would change, a unified diff is printed to stdout (colored on a tty, plain otherwise). + +## Exit codes + +| Code | Meaning | +|------|-------------------------------------------------------------------------| +| 0 | Success — policies generated (or previewed). Default even with infra drops. | +| 1 | `--fail-on-infra-drops` was set **and** ≥1 infra drop was observed. | + +Any other non-zero exit means cpg encountered a fatal error (connection +failure, bad flag, etc.). + +## CI / cron integration + +```bash +# Alert when infra drops appear in a captured window +cpg replay /tmp/last-hour.jsonl --fail-on-infra-drops \ + || alert-team "cpg detected infra drops — check cluster-health.json" +``` + +With `cpg generate` (live stream — run for a fixed window with timeout): + +> Note: `--preserve-status` ensures `timeout` propagates `cpg`'s exit code (0 vs 1) instead of +> returning 124 when the deadline is reached. Without it, a CI job that hits the timeout would +> mask whether infra drops were detected. + +```bash +timeout --preserve-status 300 cpg generate -n production --fail-on-infra-drops \ + || alert-team "infra drops in production — see cluster-health.json" +``` diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..4867a08 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,32 @@ +# Development + +## Project structure + +``` +cmd/cpg/ CLI entrypoint (cobra): generate, replay, explain +pkg/labels/ Label selection, denylist, endpoint/peer selector builders +pkg/policy/ Flow-to-CiliumNetworkPolicy builder, merge, semantic dedup, attribution +pkg/output/ Directory-organized YAML writer with merge-on-write +pkg/hubble/ Live gRPC client, aggregator, pipeline orchestration +pkg/k8s/ Kubeconfig loading, port-forward, cluster policy fetching, version detection +pkg/flowsource/ Flow stream abstraction: live gRPC or jsonpb file source +pkg/evidence/ Per-rule flow attribution (cpg explain) +pkg/diff/ Unified YAML diff (cpg generate/replay --dry-run) +``` + +## Build and test + +```bash +make test # run tests with race detector +make lint # golangci-lint +make build # build binary to ./bin/cpg +make all # lint + test + build +``` + +The test suite covers label selection, policy building, merging, output writing, flow aggregation, pipeline orchestration, and dedup logic. No live cluster required -- the Hubble gRPC client is mocked via interfaces. + +## Code quality + +![desloppify scorecard](scorecard.png) + +Tracked with [desloppify](https://github.com/peteromallet/desloppify) — strict score, regenerated on each code-health pass. diff --git a/docs/explain.md b/docs/explain.md new file mode 100644 index 0000000..da6c3f7 --- /dev/null +++ b/docs/explain.md @@ -0,0 +1,67 @@ +# Explain & evidence + +After a run, every emitted rule has per-flow evidence recorded alongside the YAML. Inspect it with `cpg explain`: + +```bash +cpg explain production/api-server +cpg explain production/api-server --peer app=frontend +cpg explain production/api-server --ingress --port 8080 +cpg explain ./policies/production/api-server.yaml --since 1h --json + +# L7 filters (literal exact match, AND-combined). Require evidence captured with --l7. +cpg explain production/api-server --http-method GET +cpg explain production/api-server --http-method GET --http-path '^/api/v1/users$' +cpg explain production/api-server --dns-pattern api.example.com +``` + +`--http-path` matches the literal regex stored in evidence — that is, the +anchored, `regexp.QuoteMeta`'d form produced by the builder +(`^/api/v1/users$`), not the raw observed path. `--dns-pattern` matches the +literal `matchName` stored in evidence (trailing dot stripped); cpg emits +no wildcards, so passing `*.example.com` will simply not match anything. +When any L7 filter is set, L4-only rules (no L7Ref in evidence) are +excluded from the result. + +When the evidence carries L7 attribution, `cpg explain` renders it +inline. Text format prints a single indented line per rule +(`L7: HTTP GET /api/v1/users` or `L7: DNS api.example.com`); JSON and +YAML formats include an `l7` sub-object on each rule with the relevant +fields (`protocol`, `http_method`, `http_path`, `dns_matchname`). + +Example output: + +``` +Policy: cpg-api-server (production) +Latest session: 2026-04-24 14:02 → 14:15 (source: replay) + +Ingress rule + Peer: app=frontend (endpoint) + Port: 8080/TCP + Flow count: 23 + First seen: 2026-04-24 14:02:11 + Last seen: 2026-04-24 14:15:48 + + Sample flows: + 14:02:11 default/frontend → production/api-server TCP/8080 + 14:02:13 default/frontend → production/api-server TCP/8080 + ... +``` + +## Where is evidence stored? + +Evidence lives outside the output directory to keep GitOps clean: + +- **Linux:** `$XDG_CACHE_HOME/cpg/evidence` (defaults to `~/.cache/cpg/evidence`) +- **macOS:** `~/Library/Caches/cpg/evidence` + +The path is keyed by a hash of the absolute output directory, so multiple workspaces coexist without collision. + +To share evidence with a colleague or archive it: + +```bash +cpg replay drops.jsonl -n production --evidence-dir ./evidence +# ... ship ./evidence alongside the policies +cpg explain production/api-server --evidence-dir ./evidence +``` + +Disable capture with `--no-evidence`. Tune retention per rule with `--evidence-samples` (default 10) and per policy with `--evidence-sessions` (default 10). diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 0000000..2933622 --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,95 @@ +# Getting started + +Not installed yet? Start with the [installation guide](installation.md). + +## Live capture + +```bash +# Point at a namespace. cpg auto port-forwards to hubble-relay. +cpg generate -n production + +# Explicit relay address +cpg generate --server localhost:4245 + +# All namespaces, debug logging +cpg --debug generate --all-namespaces + +# TLS +cpg generate --server relay.example.com:443 --tls -n production + +# Opt-in L7: HTTP method/path + DNS matchName. See the L7 guide. +cpg generate -n production --l7 +``` + +That's it. Leave it running. Go generate some traffic (or wait for someone else to). Ctrl+C when you're done -- cpg flushes remaining flows and prints a session summary before exiting. + +Policies show up in `./policies//.yaml`. + +### Auto port-forward + +When you omit `--server`, cpg finds the `hubble-relay` pod in `kube-system` using your kubeconfig and sets up a port-forward automatically. One less terminal tab to manage. + +## Offline replay + +Prefer to iterate on policy generation without reproducing traffic? Capture once, replay many: + +```bash +# Capture dropped flows for N minutes +hubble observe --output jsonpb --follow > drops.jsonl +# Ctrl+C when done capturing + +# Replay through cpg — reuse the file as many times as you want +cpg replay drops.jsonl -n production +cpg replay drops.jsonl.gz -n production # gzip transparent +cat drops.jsonl | cpg replay - # stdin + +# Opt-in L7: HTTP method/path + DNS matchName. See the L7 guide. +cpg replay drops.jsonl --l7 -n production +``` + +`cpg replay ` feeds a Hubble jsonpb capture through the same pipeline as the live stream. It is the right tool when you want: + +- **Deterministic iteration.** Re-run the same input as you tweak label selection, dedup logic, or flush intervals. +- **Offline workflow.** Capture on a jumphost, replay on your laptop. +- **Post-mortem reproduction.** Keep the capture alongside the policy in your GitOps repo so anyone can reproduce what cpg saw. + +Flags shared with `generate` (`--output-dir`, `--cluster-dedup`, `--flush-interval`, `--ignore-protocol`, `--ignore-drop-reason`, `--fail-on-infra-drops`, `--include-audit`) work identically. Non-DROPPED verdicts (unless `--include-audit` also admits AUDIT) and malformed lines are skipped with counters surfaced in the session summary. + +## Audit-mode onboarding (default-deny with zero real drops) + +The full runbook lives in [bootstrap-runbook.md](bootstrap-runbook.md) -- this is the +short version for a namespace **with live traffic**. The trick: flipping an endpoint's +`PolicyAuditMode` is a no-op until a policy matches, so open the audit window *first* and the +default-deny apply never drops anything -- every would-be drop surfaces as an `AUDIT` verdict +instead. + +```bash +# 1. Open the audit window FIRST (dedicated terminal — foreground, auto-reverts on exit/TTL) +cpg audit-window -n production --ttl 30m + +# 2. Apply the default-deny bootstrap policy — flows become AUDIT verdicts, not drops +cpg bootstrap -n production | kubectl apply -f - + +# 3. Capture the audited traffic and generate the allow policies +cpg generate -n production --include-audit + +# 4. Review, then apply (always a human act) +cpg explain production/api-server --json +kubectl apply -f ./policies/production/ + +# 5. Close the window (Ctrl+C or let --ttl expire) — audit flips revert automatically, +# default-deny now enforces against fully-covered traffic +``` + +On a **fresh namespace** (no live traffic) you can invert steps 1 and 2 -- bootstrap first +means the namespace is protected from the very first pod, and there is nothing running to +drop. Both orders, the new-endpoint race window, and the RBAC details are covered in the +[runbook](bootstrap-runbook.md); how the pieces work under the hood is the +[audit mode guide](audit-mode.md). + +## Next steps + +- [Policy generation](policy-generation.md) — what the generated YAML looks like and why +- [CLI reference](cli-reference.md) — every flag, dry-run mode, exit codes +- [L7 guide](l7-guide.md) — HTTP/DNS rules with `--l7` +- [Explain & evidence](explain.md) — inspect the flows behind every rule diff --git a/docs/installation.md b/docs/installation.md new file mode 100644 index 0000000..259755a --- /dev/null +++ b/docs/installation.md @@ -0,0 +1,88 @@ +# Installation + +## Package managers + +```bash +# kubectl krew +kubectl krew install cilium-policy-gen + +# go install +go install github.com/SoulKyu/cpg/cmd/cpg@latest +``` + +When installed via krew, use `kubectl cilium-policy-gen` instead of `cpg`. Same flags, same behavior. + +## From source + +```bash +git clone https://github.com/SoulKyu/cpg.git +cd cpg +make build +# binary lands in ./bin/cpg +``` + +Requires Go 1.25+ for source builds. + +## Supported Cilium versions + +cpg targets **Cilium >= 1.14**. Clusters below that floor are detected at +connect time and logged as a warning -- never blocked, so reduced-RBAC and +CI service accounts still get a working (if unverified) run. + +Individual capabilities carry their own, higher floors: + +| Feature | Cilium version | Notes | +|---------|-----------------|-------| +| Baseline cpg operation | >= 1.14 | Declared floor -- the lowest version any shipped code path assumes | +| `PolicyVerdictNotify` audit-action bit | >= 1.8 | PR [#11843](https://github.com/cilium/cilium/pull/11843) | +| `Verdict_AUDIT` via the Hubble flow API (`--include-audit`) | >= 1.10 | PR [#14785](https://github.com/cilium/cilium/pull/14785) / [#14923](https://github.com/cilium/cilium/pull/14923) | +| `cilium-dbg` binary naming (was `cilium`) | >= 1.15 | PR [#28085](https://github.com/cilium/cilium/pull/28085) | +| `enableDefaultDeny` CNP field | >= 1.16 | PR [#30572](https://github.com/cilium/cilium/pull/30572) -- used by `cpg bootstrap` / `get_bootstrap_policy`; see the [bootstrap runbook](bootstrap-runbook.md) | +| `policy.cilium.io/proxy-visibility` annotation | <= 1.16 | Removed from the agent runtime at 1.17 -- PR [#35019](https://github.com/cilium/cilium/pull/35019) | +| Observer `GetNodes()` RPC / `version` field | >= 1.10 | PR [#13979](https://github.com/cilium/cilium/pull/13979) | + +## k9s plugin + +You can trigger cpg directly from k9s on a namespace. Drop this into `$XDG_CONFIG_HOME/k9s/plugins.yaml` (usually `~/.config/k9s/plugins.yaml`): + +```yaml +plugins: + cpg: + shortCut: Shift-G + description: Generate Cilium policies from dropped flows + scopes: + - namespace + command: cpg + background: false + args: + - generate + - -n + - $NAME + - --cluster-dedup +``` + +Navigate to a namespace in k9s, press `Shift-G`, and cpg starts streaming dropped flows for that namespace. Ctrl+C to stop -- policies land in `./policies//`. + +If you installed via krew instead of `go install`, replace `command: cpg` with `command: kubectl` and prepend `cilium-policy-gen` to the args: + +```yaml +plugins: + cpg: + shortCut: Shift-G + description: Generate Cilium policies from dropped flows + scopes: + - namespace + command: kubectl + background: false + args: + - cilium-policy-gen + - generate + - -n + - $NAME + - --cluster-dedup +``` + +## Next steps + +- [Getting started](getting-started.md) — first live capture and offline replay +- [Bootstrap runbook](bootstrap-runbook.md) — onboarding a namespace to default-deny diff --git a/docs/l7-guide.md b/docs/l7-guide.md new file mode 100644 index 0000000..2277e06 --- /dev/null +++ b/docs/l7-guide.md @@ -0,0 +1,130 @@ +# L7 guide (HTTP + DNS rules) + +`cpg --l7` translates what Hubble shows it. Hubble only emits `Flow.L7` +records (HTTP method/path, DNS query) when traffic is proxied — by Envoy +for HTTP, by the DNS proxy for DNS. cpg cannot turn that on for you. +If you see the warning + +``` +--l7 set but no L7 records observed +``` + +this page is the fix. + +## Two-step workflow + +Refining L4 policies with L7 rules is a two-step dance, and the order +matters: + +1. **Deploy L4 first.** Run `cpg generate -n ` (no `--l7`). + Review the generated CiliumNetworkPolicies, commit, apply. You are + now moving toward default-deny with port/peer-level enforcement. + +2. **Enable L7 visibility** on the workloads you want refined. Three + options below — pick whichever fits your operational model. + +3. **Re-run cpg with `--l7`.** With visibility enabled, Hubble starts + emitting `Flow.L7` records and cpg attaches `rules.http` for HTTP + and `toFQDNs` + the kube-dns companion for DNS to the relevant + egress rules. + + ```bash + cpg generate --l7 -n + # or, on a captured stream + cpg replay drops.jsonl --l7 -n + ``` + +## Three ways to enable L7 visibility + +1. **Recommended for ad-hoc bootstrap — proxy-visibility annotation.** + The legacy workload-level annotation that triggers Envoy / DNS proxy + redirection without enforcing rules. Works only through **Cilium 1.16** + -- removed from the agent runtime at **1.17** (a no-op on 1.17+). See + the [Supported Cilium versions](installation.md#supported-cilium-versions) + table for the full matrix: + + ```bash + kubectl annotate pod -n -l app.kubernetes.io/name= \ + policy.cilium.io/proxy-visibility=',' + ``` + + Easy to apply, easy to remove -- on clusters where it still works + (<= 1.16). Marked deprecated upstream before removal. + +2. **Recommended for permanent enforcement — bootstrap L7 CNP.** Ship + a starter CiliumNetworkPolicy with a permissive L7 rule. The mere + *presence* of an L7 rule on a workload triggers Cilium to proxy that + workload's traffic — match-all `{}` in the HTTP rule and `"*"` in + the DNS matchPattern lights up visibility without enforcing + anything. See the snippet below. + +3. **Cluster-wide prerequisite — `enable-l7-proxy: true`.** In the + `kube-system/cilium-config` ConfigMap. Default `true` on most + installs, but required for any of the above to work. cpg's + `--l7` pre-flight check (VIS-04) flags it explicitly when missing + or set to false. + +## Starter L7-visibility CNP + +Copy-pasteable, valid Cilium YAML. Replace the two placeholders +(namespace + workload label) before applying: + +```yaml +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: cpg-l7-visibility-bootstrap + namespace: production # replace with your namespace +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: my-app # replace with your workload's label + egress: + # match-all HTTP rule triggers Envoy without enforcing a path/method. + - toEndpoints: + - {} + toPorts: + - ports: + - {port: "80", protocol: TCP} + rules: + http: + - {} + # match-all DNS rule triggers DNS proxy visibility for kube-dns. + - toEndpoints: + - matchLabels: + k8s-app: kube-dns + io.kubernetes.pod.namespace: kube-system + toPorts: + - ports: + - {port: "53", protocol: UDP} + - {port: "53", protocol: TCP} + rules: + dns: + - matchPattern: "*" +``` + +Apply, observe traffic, then run `cpg --l7`. Once cpg's generated +policy covers everything you need, this bootstrap CNP can be deleted — +its only job was the visibility side-effect of Envoy / DNS proxy +injection. + +## Capture-window guidance + +Run cpg long enough to capture one full traffic cycle for the +workloads in question. A single observation produces a single rule +with `flow_count=1`, which `cpg explain` surfaces as low-confidence +evidence. For periodic batch jobs, capture across at least one period. + +## Known limitations + +cpg ships with a documented set of known limitations and edge cases — most are intentional trade-offs (e.g., no HTTP header rules to avoid secret leakage), a few are deferred to v1.3+. Read them **before deploying generated policies** to production: + +→ **[KNOWN_LIMITATIONS.md](KNOWN_LIMITATIONS.md)** — full list with workarounds and tracking IDs. + +Highlights worth knowing up front: + +- L7 visibility prerequisite — `--l7` requires Cilium Envoy proxy + per-workload visibility trigger; cpg cannot bootstrap it (limitation #1). +- HTTP path explosion on REST APIs with IDs — one literal rule per observed `(method, path)`; no auto-collapse (limitation #2, `HTTP-FUT-01`). +- Header-based rules never generated — anti-feature to prevent secret leakage (limitation #3). +- DNS REFUSED denials are missed — `Verdict_FORWARDED` not yet supported (limitation #4, `L7-FUT-01`). +- kube-dns companion selector hardcoded `k8s-app=kube-dns` — autodetect across CNI distributions deferred to v1.3 (limitation #6, `DNS-FUT-02`). diff --git a/docs/mcp-server.md b/docs/mcp-server.md new file mode 100644 index 0000000..0ec3c15 --- /dev/null +++ b/docs/mcp-server.md @@ -0,0 +1,67 @@ +# MCP server (`cpg mcp`) + +`cpg mcp` runs a readonly [MCP](https://modelcontextprotocol.io) server over stdio: your LLM harness spawns `cpg mcp` as a subprocess, and it exposes a single live Hubble capture session at a time. The LLM reads dropped flows, the CiliumNetworkPolicy YAML cpg generates, per-rule flow evidence, and cluster health through the tools below — cpg never mutates your cluster and never writes outside its own session tmpdir. + +## Tool catalog + +| Tool | Description | +|------|-------------| +| `start_session` | Start a live Hubble capture session in the background; returns an opaque `session_id` immediately. Only one session at a time — stop the current one before starting another. Accepts `include_audit` to also ingest `Verdict_AUDIT` flows alongside DROPPED (opt-in; default preserves pre-v1.6 DROPPED-only behavior). | +| `get_status` | Coarse session state (capturing/stopped), elapsed time, and on-disk artifact counts. Works for a stopped-but-retained session too. | +| `stop_session` | Cancel the capture, finalize `cluster-health.json` and session stats, and return the final summary. Idempotent — a second call returns the same summary, never an error. | +| `list_dropped_flows` | Paginated, two-section view of dropped flows: policy-actionable samples plus infra/transient/noise aggregate counts. Sampled/aggregated, not a raw flow log. | +| `list_policies` | Paginated metadata for every generated CiliumNetworkPolicy in the session — namespace, workload, rule counts, YAML path. | +| `get_policy` | Full CiliumNetworkPolicy YAML plus metadata for one namespace/workload pair (discover pairs via `list_policies`). | +| `get_evidence` | Paginated per-rule flow evidence for one policy, byte-identical in shape to `cpg explain --json`. | +| `get_cluster_health` | The session's finalized cluster-health report: per-drop-reason counts by node/workload, plus Cilium-docs remediation URLs. | +| `get_bootstrap_policy` | A namespaced default-deny CiliumNetworkPolicy as YAML (same artifact as `cpg bootstrap`), returned as read-only tool content with the detected Cilium version and compat verdict — no filesystem writes. | + +## Harness configuration + +```json +{ + "mcpServers": { + "cpg": { + "command": "cpg", + "args": ["mcp"], + "env": { + "KUBECONFIG": "/home/you/.kube/config", + "PATH": "/usr/local/bin:/usr/bin:/bin", + "TMPDIR": "/tmp" + } + } + } +} +``` + +MCP hosts do not inherit your shell environment — the `env` block above is not optional decoration, it is the only environment `cpg mcp` will ever see. Set each key explicitly: + +- **`KUBECONFIG`** — cpg's client-go loader resolves clusters the same way `kubectl` does (`KUBECONFIG` env, then `~/.kube/config`, then in-cluster config). Omit it and the process silently falls back to whatever exists at the default path — which may not exist, or may point at the wrong cluster. +- **`PATH`** — if your kubeconfig authenticates via an `exec` credential plugin (see the caveat below), client-go shells out to a binary on `PATH`. Without it, that lookup fails. +- **`TMPDIR`** — the session's `os.MkdirTemp`-rooted working directory (policies, evidence, `cluster-health.json`) honors `$TMPDIR`. Leave it unset and the process falls back to the platform default, which is usually fine but worth setting explicitly if your host sandboxes `/tmp`. + +## Secrets posture + +With `--l7`-style visibility enabled on a session (see the [L7 guide](l7-guide.md)), HTTP paths and methods, FQDNs, and workload labels reach the LLM context through tool results — that's the query tools doing their job. `Authorization`, `Cookie`, and other headers are **never** captured; cpg has never generated or stored header-based rules, an anti-feature carried forward from v1.2 specifically to avoid this kind of leakage. There is no redaction pass in v1.5 — whatever the pipeline observes is what the LLM sees, unfiltered (a dedicated redaction feature is deferred to v2). If you're pointing an MCP session at a cluster carrying sensitive path, FQDN, or label data, know what crosses the boundary before you start the session. + +## Exec-credential-plugin caveat + +Kubeconfigs authenticating via an `exec` plugin — `aws eks get-token`, `gke-gcloud-auth-plugin`, `azure kubelogin`, and similar — expect an interactive terminal for re-auth (an expired SSO session, a first-time device-code flow). Under an MCP host, `cpg mcp`'s real stdin is the JSON-RPC channel, not a keyboard: an interactive prompt has nowhere to go, and `start_session` hangs instead of failing fast. Before wiring cpg into a harness, verify headless auth actually works — running `kubectl get pods` from a non-interactive shell (no TTY) is the fastest check — or point at a static, already-authenticated kubeconfig instead. `start_session`'s bounded setup timeout turns the common exec-plugin re-auth hang (during dial/port-forward) into an actionable error rather than a silent wait — the one known exception is a hang specifically inside kubeconfig load itself (`k8s.LoadKubeConfig()` takes no `ctx`), which neither this timeout nor cancellation can bound. One more thing worth knowing: some `exec`/OIDC plugins refresh and persist credentials back to the kubeconfig file on disk as a side effect of successful auth — expected client-go behavior, not something cpg itself does, but worth knowing if you otherwise treat that file as read-only. + +## Session model + +One capture session at a time — `start_session` returns an error if a session is already running. A stopped session isn't discarded: `get_status` and `stop_session` keep returning its final state, and the query tools keep serving its artifacts, until the next `start_session` call or the server process exits. + +## Agent tooling + +Repo-local Claude Code skills and agent under `.claude/` route an LLM operator through cpg-specific workflows on top of `cpg mcp`'s tool surface. Skills name workflow steps and tool names only — argument schemas and result shapes are always discovered live via `tools/list`, never restated in skill prose. + +| Skill | Purpose | +|-------|---------| +| `cpg-triage` | Drive a live MCP session end-to-end: start capture, classify dropped flows, present each generated policy with its evidence, recommend what to apply. | +| `cpg-audit-onboard` | Guide onboarding a new namespace: bootstrap and audit-window (human-run CLI), drive capture via MCP, end with the human applying policy. | +| `cpg-policy-review` | Audit already-generated policies offline via `cpg explain` and evidence — over-broad rules, L7 anchoring, DNS-53 companions, dedup. | +| `cpg-health-report` | Turn a session's `cluster-health.json` into a self-contained HTML report of infra drops by node/workload with remediation links. | +| `cpg-mcp-smoke` | Post-release smoke test of a tagged binary's MCP server against the fake-relay e2e harness. | + +`cpg-operator` (`.claude/agents/cpg-operator.md`) is the single repo-local agent that drives live MCP session lifecycles on behalf of `cpg-triage` and `cpg-audit-onboard` — it is not invoked directly. diff --git a/docs/policy-generation.md b/docs/policy-generation.md new file mode 100644 index 0000000..7bed753 --- /dev/null +++ b/docs/policy-generation.md @@ -0,0 +1,178 @@ +# Policy generation + +## How it works + +``` + Hubble Relay (gRPC) + | + [cpg generate] + | + stream dropped flows + | + aggregate by workload + | + build CiliumNetworkPolicy + | + merge with existing files + | + write YAML to disk + | + you review & git push +``` + +Flows are aggregated by namespace and workload on a configurable interval (default 5s), so you get one policy per workload -- not one per packet. Existing files are read, merged (new ports and peers appended), and only rewritten if something actually changed. + +## What it generates + +Given a dropped ingress flow to a pod labeled `app.kubernetes.io/name: api-server` on port 8080/TCP from a pod with `app: frontend`: + +```yaml +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: cpg-api-server + namespace: production +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: api-server + ingress: + - fromEndpoints: + - matchLabels: + app: frontend + toPorts: + - ports: + - port: "8080" + protocol: TCP +``` + +External traffic (world identity) gets CIDR-based rules (`fromCIDR` / `toCIDR`) with /32 addresses instead of endpoint selectors, because you can't exactly match a label on the internet. + +### With `--l7` (opt-in HTTP + DNS) + +When `--l7` is set and Hubble is producing L7 flow records (see the [L7 guide](l7-guide.md)), cpg attaches HTTP method/path and DNS `toFQDNs` to the relevant rules. Same fixture as above plus an observed `GET /api/v1/users` and a DNS query for `api.example.com`: + +```yaml +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: cpg-api-server + namespace: production +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: api-server + ingress: + - fromEndpoints: + - matchLabels: + app: frontend + toPorts: + - ports: + - {port: "8080", protocol: TCP} + rules: + http: + - {method: GET, path: ^/api/v1/users$} + egress: + - toFQDNs: + - matchName: api.example.com + toPorts: + - ports: + - {port: "53", protocol: UDP} + - {port: "53", protocol: TCP} + rules: + dns: + - matchName: api.example.com + # Companion kube-dns rule auto-injected for every CNP with toFQDNs (DNS-02). + - toEndpoints: + - matchLabels: + k8s-app: kube-dns + io.kubernetes.pod.namespace: kube-system + toPorts: + - ports: + - {port: "53", protocol: UDP} + - {port: "53", protocol: TCP} + rules: + dns: + - matchName: api.example.com +``` + +HTTP paths are emitted as anchored, `regexp.QuoteMeta`'d RE2 regexes (`^/api/v1/users$`). Methods are uppercase-normalized. Header / Host rules are never generated (anti-feature, see [Limitations](#limitations)). + +## Label selection + +Labels are chosen with a priority hierarchy: + +1. `app.kubernetes.io/name` if present (Kubernetes standard) +2. `app` if present (common convention) +3. All labels minus a denylist (pod-template-hash, controller-revision-hash, etc.) + +This means generated policies survive rolling updates and don't accidentally pin to a specific ReplicaSet. + +## Deduplication + +cpg tries hard not to waste your time: + +- **File dedup**: if the merged result is identical to what's already on disk, it skips the write. +- **Cross-flush dedup**: if the same policy was written in a previous flush cycle, it's not rewritten. +- **Cluster dedup** (`--cluster-dedup`): fetches live CiliumNetworkPolicies from the cluster and skips policies that already match. Needs `list` RBAC on `ciliumnetworkpolicies.cilium.io`. + +## Unhandled flows + +Not every dropped flow can become a policy rule. cpg reports what it skips so you can investigate: + +- **INFO summary** at each flush cycle -- structured counters by skip reason +- **DEBUG detail** per unique flow -- logged once, with source, destination, port, protocol, and destination labels + +Enable debug logging to see individual flows: + +```bash +cpg --debug generate -n production +# or +cpg --log-level debug generate -n production +``` + +### Skip reasons + +| Reason | What it means | +|--------|---------------| +| `no_l4` | Flow has no L4 layer (no port/protocol info) | +| `nil_endpoint` | Source or destination endpoint is nil | +| `empty_namespace` | Target endpoint has no namespace (non-reserved identity) | +| `nil_source` | Ingress flow with nil source endpoint | +| `nil_destination` | Egress flow with nil destination endpoint | +| `unknown_protocol` | L4 layer present but protocol not TCP/UDP/ICMP | +| `world_no_ip` | World (external) traffic without an IP address | + +### Example output + +At INFO level (default): + +``` +INFO Unhandled flows summary {"no_l4": 42, "nil_endpoint": 8, "world_no_ip": 3} +``` + +At DEBUG level: + +``` +DEBUG Unhandled flow {"src": "default/nginx", "dst": "kube-system/coredns", "port": "53", "proto": "UDP", "reason": "no_l4", "dst_labels": ["k8s:app=coredns"]} +``` + +Reserved identity flows (like `reserved:host` or `reserved:kube-apiserver`) are reported separately as WARN logs with guidance to use CiliumClusterwideNetworkPolicy instead. + +## Limitations + +Honest ones: + +- **L4 by default; L7 opt-in via `--l7`.** Without the flag, cpg generates port-level policies (v1.1 byte-stable). With `--l7`, cpg attaches HTTP `method` + anchored regex `path` and DNS `toFQDNs` (literal `matchName`) to the matching rules. Several L7 features are intentionally deferred to v1.3: + - No HTTP `Headers` / `Host` / `HostExact` rules (anti-feature: secret leakage into committed YAML). + - No HTTP path templating / auto-collapse — one rule per observed `(method, path)` pair (`--l7-collapse-paths`, HTTP-FUT-01). + - No DNS `matchPattern` glob inference — only literal `matchName` (`--l7-fqdn-wildcard-depth`, DNS-FUT-01). + - No FQDN inference from L4-to-IP correlation (DNS-FUT-03). + - REFUSED DNS denials surface as `Verdict_FORWARDED` and are missed (`--include-l7-forwarded`, L7-FUT-01). + + See the [L7 guide](l7-guide.md) for the two-step workflow and the starter visibility CNP. +- **No auto-apply.** cpg writes YAML files. Applying them is your job, presumably through whatever GitOps tooling you already have. This is intentional -- auto-applying network policies in production is how you get paged at 3am. +- **Namespace-scoped only.** It generates CiliumNetworkPolicy, not CiliumClusterwideNetworkPolicy. Cluster-wide policies are typically hand-crafted by platform teams who know what they're doing (allegedly). +- **Named ports aren't resolved.** You get port numbers, not service port names. Port 8080 is port 8080. Less ambiguity, more grep-ability. + +The full list with workarounds and tracking IDs lives in [KNOWN_LIMITATIONS.md](KNOWN_LIMITATIONS.md). diff --git a/docs/security.md b/docs/security.md new file mode 100644 index 0000000..e8b8ea4 --- /dev/null +++ b/docs/security.md @@ -0,0 +1,46 @@ +# Security model + +## Readonly by default + +cpg is readonly by default. Every command -- `cpg generate`, `cpg replay`, `cpg bootstrap`, +`cpg explain`, and `cpg mcp` (see the [MCP server guide](mcp-server.md)) -- only lists, +watches, and reads Kubernetes and Hubble data. None of them writes to the cluster. + +The one exception is `cpg audit-window`, the single scoped, lifecycle-bound mutating command: +it flips per-endpoint `PolicyAuditMode` on for the endpoints in a target namespace, watches for +newly-created endpoints and flips those too, then reverts every flip it made -- on Ctrl+C, on +`--ttl` expiry, on any exit path -- the moment the foreground command stops. It never touches an +endpoint that was already in audit mode before it started, and it never touches the daemon-wide +audit mode setting. See the [bootstrap runbook](bootstrap-runbook.md) for the full +`cpg audit-window` workflow, including its new-endpoint race window. + +## RBAC requirements + +`cpg audit-window` requires an RBAC step-up that no readonly command needs: `pods/exec` (create, +scoped to `kube-system`) to reach each node's cilium-agent pod, and `ciliumendpoints` +(list/watch) to discover and track endpoints. Grant both verbs only to whatever principal +actually runs `cpg audit-window`. The exec transport is WebSocket with SPDY fallback (same as +kubectl since 1.30+). + +**RBAC scoping limitation:** Kubernetes RBAC `resourceNames` requires exact, static pod names, +but Cilium agent pod names are DaemonSet-generated (`cilium-`) and not stable +across restarts -- there is no way to scope `pods/exec` to "only cilium-agent pods" by name. +The grant is effectively "exec into any pod in `kube-system`" for whatever role carries it. If +your cluster runs an admission controller (Kyverno, OPA Gatekeeper), consider policy-based +scoping there as defense-in-depth -- this is not something cpg itself enforces or guarantees. + +Readonly commands need much less: the only documented extra grant is `--cluster-dedup`, +which needs `list` on `ciliumnetworkpolicies.cilium.io`. + +## What never happens + +- **No auto-apply.** cpg writes YAML files to disk; applying them is always a human act, + presumably through your GitOps pipeline. +- **No HTTP header rules.** `Authorization`, `Cookie`, and other headers are never captured + or turned into rules -- an anti-feature to prevent secret leakage into committed YAML. + +## Related + +- [Audit mode](audit-mode.md) — how the audit-window lifecycle and its guarantees work +- [MCP server guide](mcp-server.md) — secrets posture when exposing sessions to an LLM +- [Bootstrap runbook](bootstrap-runbook.md) — the full audit-window workflow and its RBAC details diff --git a/pkg/hubble/pipeline.go b/pkg/hubble/pipeline.go index 2adbbe1..9ca841b 100644 --- a/pkg/hubble/pipeline.go +++ b/pkg/hubble/pipeline.go @@ -358,7 +358,7 @@ func RunPipelineWithSource(ctx context.Context, cfg PipelineConfig, source flows cfg.Logger.Warn("--l7 set but no L7 records observed in window", zap.Strings("workloads", agg.ObservedWorkloads()), zap.Uint64("flows", stats.FlowsSeen), - zap.String("hint", "see README L7 prerequisites: #l7-prerequisites"), + zap.String("hint", "see L7 visibility prerequisites: docs/l7-guide.md"), ) } diff --git a/pkg/hubble/pipeline_l7_test.go b/pkg/hubble/pipeline_l7_test.go index 5db163e..4e018dd 100644 --- a/pkg/hubble/pipeline_l7_test.go +++ b/pkg/hubble/pipeline_l7_test.go @@ -119,11 +119,11 @@ func TestPipeline_L7Empty_FiresWarning(t *testing.T) { for _, e := range logs.All() { if strings.Contains(e.Message, "no L7 records observed") { matches++ - // hint must include the README anchor + flag name. + // hint must include the L7 guide path + flag name. fields := e.ContextMap() hint, ok := fields["hint"].(string) assert.True(t, ok, "hint field must be a string") - assert.Contains(t, hint, "#l7-prerequisites") + assert.Contains(t, hint, "docs/l7-guide.md") assert.Contains(t, e.Message, "--l7") // workloads slice must be non-empty and sorted. ws, ok := fields["workloads"].([]interface{})