diff --git a/README.md b/README.md index 438760b..96e34c1 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,12 @@ -[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/paramify/paramify-fetchers) - - # Paramify Fetchers +[![CI](https://github.com/paramify/paramify-fetchers/actions/workflows/ci.yml/badge.svg)](https://github.com/paramify/paramify-fetchers/actions/workflows/ci.yml) +[![License: GPLv3](https://img.shields.io/badge/License-GPLv3-1467ff.svg)](LICENSE) +[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-1467ff.svg)](pyproject.toml) +[![Version](https://img.shields.io/badge/version-0.2.0-1467ff.svg)](CHANGELOG.md) +[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/paramify/paramify-fetchers) + Fetchers are small scripts that collect compliance evidence from your infrastructure and write it to disk as JSON. A separate uploader stage pushes that evidence to Paramify. This repo contains the fetchers, the runner that executes them, and the uploader — the fetchers themselves never talk to Paramify directly. ``` @@ -11,49 +14,6 @@ Fetchers are small scripts that collect compliance evidence from your infrastruc (on disk, per run) (separate stage) ``` -The `paramify` CLI is the way in — list the catalog, then inspect any one -fetcher's contract: - -![Browsing the fetcher catalog with the paramify CLI](docs/demo/catalog.gif) - ---- - -## Quick start - -**Prerequisites:** Python 3.10+. The CLIs your fetchers need (`aws`, `jq`, `curl`, `kubectl`, etc.) must be on your `PATH` — install only what applies to the categories you'll run. Each service's credential setup guide is in `fetchers//README.md`. - -```bash -# 1. Clone and install -git clone https://github.com/paramify/paramify-fetchers.git -cd paramify-fetchers -python -m venv .venv && source .venv/bin/activate # recommended -pip install -e . -# To add the TUI: pip install -e '.[all]' - -# 2. Browse available fetchers by category -paramify catalog - -# 3. Start a manifest and wire in your fetchers -paramify manifest init -paramify manifest add okta_phishing_resistant_mfa -paramify manifest set-secret okta_phishing_resistant_mfa api_token OKTA_API_TOKEN -paramify manifest set-secret okta_phishing_resistant_mfa org_url OKTA_ORG_URL -# The manifest builder reports missing secrets after each step — -# keep going until it says the manifest is runnable. - -# 4. Set your credentials and run -export OKTA_API_TOKEN= -export OKTA_ORG_URL=https://your-org.okta.com -paramify validate manifest.yaml -paramify run manifest.yaml # evidence → ./evidence/run-/ - -# 5. Upload to Paramify -export PARAMIFY_UPLOAD_API_TOKEN= # see uploaders/paramify_evidence/README.md for setup -paramify upload # push the latest run -``` - -Use `paramify describe ` to see exactly what secrets and config any fetcher needs. Each service has a credential setup guide in its fetcher directory — for example, [`fetchers/okta/README.md`](fetchers/okta/README.md) covers creating an Okta API token and the required admin role. See [`examples/`](examples/) for complete worked manifests (multi-region AWS, GitLab fanout, etc.) and [`deploy/README.md`](deploy/README.md) for running on a schedule in Docker or Kubernetes. - --- ## Supported services @@ -100,64 +60,101 @@ Azure · and more --- -## How it runs +## Install -Four pieces, kept deliberately separate: +**Prerequisites:** Python 3.10+. The CLIs your fetchers need (`aws`, `jq`, `curl`, `kubectl`, etc.) must be on your `PATH` — install only what applies to the categories you'll run. Each service's credential setup guide is in `fetchers//README.md`. -- **Fetcher** — a small script (`fetcher.py` or `fetcher.sh`) that collects from - *one* source and writes a JSON file. It reads everything it needs from - environment variables and writes only to `EVIDENCE_DIR`. -- **`fetcher.yaml`** — the fetcher's self-description: its name, what secrets and - config it needs, what it outputs, and its `evidence_set` identity. Ships with - the code, validated against a schema. Customers never edit this. -- **Run manifest** — the customer's intent: which fetchers to run, with what - config, against what targets. Lives in the customer's environment, not here. -- **Runner** — reads `fetcher.yaml` files and a manifest, resolves secrets and - config into environment variables, and executes each fetcher. +```bash +git clone https://github.com/paramify/paramify-fetchers.git +cd paramify-fetchers +python -m venv .venv && source .venv/bin/activate +pip install -e '.[all]' # '[all]' bundles the TUI; use `pip install -e .` for the headless CLI only +``` -```mermaid -flowchart LR - subgraph infra["runs on customer infrastructure"] - direction LR - Y["fetcher.yaml
self-description"] --> R["runner"] - M["run manifest
which fetchers + config"] --> R - R -->|"secrets + config
as env vars"| F["fetcher
one source each"] - F -->|"raw JSON"| R - R -->|"wrap in envelope"| E[("evidence files
one run dir")] - E --> U["uploader"] - end - U -->|"Paramify REST v0 · HTTPS only"| P[("Paramify")] +There are three ways to drive it — an interactive **TUI**, an **AI agent**, or the **CLI** directly. All three go through one facade (`framework.api`), so they behave identically; pick whichever fits how you work. + +--- + +## The TUI + +The fastest way in. `paramify tui` browses the catalog, builds and validates a +manifest, runs it, and reviews evidence — all without leaving the keyboard: + +![The paramify terminal UI](docs/demo/tui.gif) + +> **Zero-credential first run:** the bundled `demo_hello` fetcher emits synthetic +> evidence, so you can watch the whole collect → envelope pipeline before wiring +> up a real service: +> +> ```bash +> paramify run examples/demo.yaml # synthetic evidence — no credentials +> paramify evidence evidence/run-*/demo_hello.json # inspect the enveloped result +> ``` + +--- + +## Drive it with an AI agent + +Every command takes `--json`, and each `paramify manifest` edit returns a stable +`{ok, path, errors}` object — so an agent can assemble a runnable manifest by +reading `errors` and closing each gap, no screen-scraping: + +```bash +paramify catalog --json # discover what's available +paramify manifest add okta_phishing_resistant_mfa --json # → {"ok": false, "errors": [ …missing secrets… ]} +paramify manifest set-secret okta_phishing_resistant_mfa api_token OKTA_API_TOKEN --json +# …repeat until: +paramify validate manifest.yaml --json # → {"ok": true, "errors": []} ``` -Everything goes through one facade, `framework.api` — discovery, manifest -editing, validation, and running. One CLI, `paramify`, sits on top of it and -steers every front-end; because they all share that single code path they behave -identically. Install it once from the repo (editable), then: +The repo also ships Claude Code skills under [`.claude/skills/`](.claude/skills/) — +`create-fetcher`, `wire-manifest`, and `suggest-validator` — so an agent can +scaffold a new fetcher, wire it into a manifest, or propose a validator directly. + +--- + +## Using the CLI + +The same operations, run by hand. `paramify catalog` lists the catalog and +`paramify describe ` shows exactly what any one fetcher needs: + +![Browsing the fetcher catalog with the paramify CLI](docs/demo/catalog.gif) + +A typical run, step by step: ```bash -pip install -e . # installs the `paramify` command - # (use `pip install -e '.[all]'` to add the TUI) +# 1. Browse available fetchers by category +paramify catalog -paramify # human CLI -paramify --json # same commands, machine-readable (for AI/scripts) -paramify tui # interactive terminal UI -``` +# 2. Start a manifest and wire in your fetchers +paramify manifest init +paramify manifest add okta_phishing_resistant_mfa +paramify manifest set-secret okta_phishing_resistant_mfa api_token OKTA_API_TOKEN +paramify manifest set-secret okta_phishing_resistant_mfa org_url OKTA_ORG_URL +# The manifest builder reports missing secrets after each step — +# keep going until it says the manifest is runnable. -> Back-compat: `python -m framework.runner ` and `python -m framework.tui` -> still work and are exactly equivalent to the corresponding `paramify` -> subcommands. +# 3. Set your credentials and run +export OKTA_API_TOKEN= +export OKTA_ORG_URL=https://your-org.okta.com +paramify validate manifest.yaml +paramify run manifest.yaml # evidence → ./evidence/run-/ -`paramify tui` drives that same facade interactively — browse the catalog, build -and validate a manifest, run it, and review evidence without leaving the keyboard: +# 4. Upload to Paramify +export PARAMIFY_UPLOAD_API_TOKEN= # see uploaders/paramify_evidence/README.md for setup +paramify upload # push the latest run +``` -![The paramify terminal UI](docs/demo/tui.gif) +Each service has a credential setup guide in its fetcher directory — for example, [`fetchers/okta/README.md`](fetchers/okta/README.md) covers creating an Okta API token and the required admin role. See [`examples/`](examples/) for complete worked manifests (multi-region AWS, GitLab fanout, etc.) and [`deploy/README.md`](deploy/README.md) for running on a schedule in Docker or Kubernetes. -The CLI command surface: +The full command surface: ```bash paramify list # discovered fetchers (flat) paramify catalog # categories → fetchers → editable fields paramify describe # one fetcher's config / secrets / target fields +paramify ksi # FedRAMP 20x KSI coverage +paramify doctor [manifest] # preflight: Python, required CLIs, manifest secrets paramify manifests # discovered run manifests (manifests/*.yaml) paramify validate # validate a manifest without running paramify run # run it @@ -167,54 +164,24 @@ paramify upload [run-dir] # push a run's evidence to Paramify (default: lat paramify manifest # build/edit a manifest (see below) ``` -Output lands in `/run-/`, one JSON file per fetcher -(or per target for fan-out), alongside a `_run_metadata.json` run index. The -runner wraps each evidence file in an envelope — -`{schema_version, metadata, payload}` — where `metadata` carries the fetcher -name/version/category, run id, target, `collected_at`, status, exit code, and -the `evidence_set` identity; failed invocations also get a `stderr_tail`. The -`_run_metadata.json` index itself is not enveloped. +> Back-compat: `python -m framework.runner ` and `python -m framework.tui` +> still work and are exactly equivalent to the corresponding `paramify` +> subcommands. -A finished evidence file looks like this — an AWS VPC-segmentation run, -abbreviated: +Before a real run, `paramify doctor ` preflights the environment — +Python, the CLIs each category needs, and whether the manifest's secret env vars +are set — and exits non-zero if anything's missing, so it drops straight into CI: -```json -{ - "schema_version": "1.0", - "metadata": { - "fetcher_name": "aws_vpc_network_segmentation", - "fetcher_version": "0.1.0", - "category": "aws", - "run_id": "2026-06-16T15-56-41Z", - "target": { "region": "us-east-1" }, - "collected_at": "2026-06-16T16:00:14Z", - "status": "success", - "exit_code": 0, - "evidence_set": { - "reference_id": "EVD-VPC-SEGMENTATION", - "name": "VPC Network Segmentation", - "instructions": "Script: fetcher.sh. Commands: aws ec2 describe-vpcs, describe-subnets, describe-vpc-peering-connections, describe-vpc-endpoints. Maps to KSI-CNA-03.", - "description": "Lists VPCs, subnets, peering connections, and endpoints to document network topology and segmentation." - } - }, - "payload": { - "metadata": { "account_id": "111122223333", "region": "us-east-1", "datetime": "2026-06-16T16:00:14Z" }, - "results": [ - { "ResourceType": "Vpcs", "Items": [ - { "VpcId": "vpc-0a1b2c3d", "CidrBlock": "172.31.0.0/16", "IsDefault": true, "State": "available" } - ] }, - { "ResourceType": "Subnets", "Items": [ - { "SubnetId": "subnet-0d7e6de0", "VpcId": "vpc-0a1b2c3d", "CidrBlock": "172.31.80.0/20", "AvailabilityZone": "us-east-1b" } - ] } - ] - } -} -``` +```text +$ paramify doctor examples/minimal_run.yaml +✅ Python 3.11.9 (need ≥ 3.10) -The runner owns the `metadata` envelope; the fetcher owns `payload`. The -`evidence_set` block (from `fetcher.yaml`) is what an uploaded file maps to in -Paramify. Note there is no pass/fail verdict — that judgment is Paramify-side, by -design (peering connections and endpoints are omitted above for brevity). +Manifest secrets (examples/minimal_run.yaml): + ❌ okta_phishing_resistant_mfa missing: OKTA_API_TOKEN, OKTA_ORG_URL + ❌ gitlab_ci_cd_pipeline_config missing: GITLAB_TOKEN_1, GITLAB_TOKEN_2 + +Issues found — see above. +``` ### Building a manifest @@ -267,35 +234,88 @@ example glue. --- -## Why the design is strict - -Every fetcher is forced through one contract, validated by JSON Schema, with a -narrow set of allowed shapes. That rigidity is intentional. The previous -generation of fetchers were freeform scripts, and each one invented its own -conventions for config, secrets, and output — which is exactly why none of them -composed and the central catalog had to be hand-maintained in sync. A few -principles keep that from happening again: - -- **One contract, schema-enforced.** A fetcher declares itself in `fetcher.yaml`, - validated at discovery time. Anything not in the schema is not a thing a - fetcher can do. This is what lets the runner treat all 107 fetchers identically. -- **Fetchers run on customer infrastructure**, never Paramify's. So a fetcher - never assumes a Paramify connection, and the framework owns no scheduling. -- **Secrets are source-agnostic.** A fetcher reads `OKTA_API_TOKEN` from the - environment. It never knows or cares whether that came from a `.env` file, - AWS Secrets Manager, Vault, or a CI secret block — because every one of those - already knows how to set an environment variable. We do not write per-provider - secret integrations, and we don't intend to. -- **Collect facts; interpret elsewhere.** A fetcher gathers evidence. Whether - that evidence *satisfies* a control is a Paramify-side mapping, not the - fetcher's job. Keep pass/fail verdicts and compliance thresholds out of - fetchers. -- **One source per fetcher.** Cross-source comparison (e.g. Okta users vs. - Rippling employees) is a separate "comparator" that reads prior outputs — same - contract, different inputs. A fetcher never reads another fetcher's output. - -The full contract is in [`docs/fetcher_contract.md`](docs/fetcher_contract.md); -the rationale is in [`docs/design.md`](docs/design.md). +## How it runs + +Four pieces, kept deliberately separate: + +- **Fetcher** — a small script (`fetcher.py` or `fetcher.sh`) that collects from + *one* source and writes a JSON file. It reads everything it needs from + environment variables and writes only to `EVIDENCE_DIR`. +- **`fetcher.yaml`** — the fetcher's self-description: its name, what secrets and + config it needs, what it outputs, and its `evidence_set` identity. Ships with + the code, validated against a schema. Customers never edit this. +- **Run manifest** — the customer's intent: which fetchers to run, with what + config, against what targets. Lives in the customer's environment, not here. +- **Runner** — reads `fetcher.yaml` files and a manifest, resolves secrets and + config into environment variables, and executes each fetcher. + +```mermaid +flowchart LR + subgraph infra["runs on customer infrastructure"] + direction LR + Y["fetcher.yaml
self-description"] --> R["runner"] + M["run manifest
which fetchers + config"] --> R + R -->|"secrets + config
as env vars"| F["fetcher
one source each"] + F -->|"raw JSON"| R + R -->|"wrap in envelope"| E[("evidence files
one run dir")] + E --> U["uploader"] + end + U -->|"Paramify REST v0 · HTTPS only"| P[("Paramify")] +``` + +Everything goes through one facade, `framework.api` — discovery, manifest +editing, validation, and running. The TUI, the CLI, and the `--json` surface an +agent drives all sit on that single code path, which is why they behave +identically. + +Output lands in `/run-/`, one JSON file per fetcher +(or per target for fan-out), alongside a `_run_metadata.json` run index. The +runner wraps each evidence file in an envelope — +`{schema_version, metadata, payload}` — where `metadata` carries the fetcher +name/version/category, run id, target, `collected_at`, status, exit code, and +the `evidence_set` identity; failed invocations also get a `stderr_tail`. The +`_run_metadata.json` index itself is not enveloped. + +A finished evidence file looks like this — an AWS VPC-segmentation run, +abbreviated: + +```json +{ + "schema_version": "1.0", + "metadata": { + "fetcher_name": "aws_vpc_network_segmentation", + "fetcher_version": "0.1.0", + "category": "aws", + "run_id": "2026-06-16T15-56-41Z", + "target": { "region": "us-east-1" }, + "collected_at": "2026-06-16T16:00:14Z", + "status": "success", + "exit_code": 0, + "evidence_set": { + "reference_id": "EVD-VPC-SEGMENTATION", + "name": "VPC Network Segmentation", + "instructions": "Script: fetcher.sh. Commands: aws ec2 describe-vpcs, describe-subnets, describe-vpc-peering-connections, describe-vpc-endpoints. Maps to KSI-CNA-03.", + "description": "Lists VPCs, subnets, peering connections, and endpoints to document network topology and segmentation." + } + }, + "payload": { + "metadata": { "account_id": "111122223333", "region": "us-east-1", "datetime": "2026-06-16T16:00:14Z" }, + "results": [ + { "ResourceType": "Vpcs", "Items": [ + { "VpcId": "vpc-0a1b2c3d", "CidrBlock": "172.31.0.0/16", "IsDefault": true, "State": "available" } + ] }, + { "ResourceType": "Subnets", "Items": [ + { "SubnetId": "subnet-0d7e6de0", "VpcId": "vpc-0a1b2c3d", "CidrBlock": "172.31.80.0/20", "AvailabilityZone": "us-east-1b" } + ] } + ] + } +} +``` + +The runner owns the `metadata` envelope; the fetcher owns `payload`. The +`evidence_set` block (from `fetcher.yaml`) is what an uploaded file maps to in +Paramify. Note there is no pass/fail verdict — that judgment is Paramify-side, by +design (peering connections and endpoints are omitted above for brevity). > **Status:** pre-1.0 (v0.x). The runner now wraps every output in the > `metadata`+`payload` envelope, but fetchers still write raw evidence dicts and diff --git a/examples/demo.yaml b/examples/demo.yaml new file mode 100644 index 0000000..8a574a4 --- /dev/null +++ b/examples/demo.yaml @@ -0,0 +1,13 @@ +# Zero-credential demo. Run it as-is — no secrets, no cloud account: +# +# paramify run examples/demo.yaml +# paramify evidence evidence/run-*/demo_hello.json +# +# The demo_hello fetcher writes synthetic evidence, so this exercises the whole +# collect → envelope pipeline end to end. Delete once you've seen it work. + +run: + output_dir: ./evidence + + fetchers: + - use: demo_hello diff --git a/fetchers/_categories/demo.yaml b/fetchers/_categories/demo.yaml new file mode 100644 index 0000000..11ad4d7 --- /dev/null +++ b/fetchers/_categories/demo.yaml @@ -0,0 +1,4 @@ +# Demo category — a single credential-free fetcher that emits synthetic +# evidence, so you can exercise the whole run → envelope pipeline with no cloud +# account. Not real evidence; safe to delete. +description: Synthetic evidence for trying the pipeline with no credentials. diff --git a/fetchers/demo/hello/fetcher.py b/fetchers/demo/hello/fetcher.py new file mode 100644 index 0000000..3f71efe --- /dev/null +++ b/fetchers/demo/hello/fetcher.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python3 +""" +Demo fetcher: synthetic MFA-policy evidence. + +Writes a fixed, plausible-looking evidence payload so a newcomer can run the +whole pipeline — collect, envelope, inspect — without any credentials or network +access. This is NOT real evidence; it exists purely to demonstrate the shape of +a run. See `examples/demo.yaml`. +""" + +import json +import logging +import os +from datetime import datetime, timezone +from pathlib import Path + +logger = logging.getLogger("demo_hello") + + +def main(): + logging.basicConfig( + level=os.environ.get("LOG_LEVEL", "INFO"), + format="%(asctime)s %(levelname)s %(name)s %(message)s", + ) + + output_dir = Path(os.environ.get("EVIDENCE_DIR", "./evidence")) + output_dir.mkdir(parents=True, exist_ok=True) + + # A fixed, synthetic snapshot — the shape a real fetcher would collect, + # with obviously fake values. + evidence = { + "metadata": { + "source": "demo", + "note": "Synthetic evidence — not collected from any real system.", + "generated_at": datetime.now(timezone.utc).isoformat(), + }, + "results": [ + { + "policy": "require-phishing-resistant-mfa", + "enabled": True, + "applies_to": "all-users", + "enrolled_users": 42, + "total_users": 42, + }, + { + "policy": "block-legacy-authentication", + "enabled": True, + "applies_to": "all-users", + }, + ], + } + + output_path = output_dir / "demo_hello.json" + with open(output_path, "w") as f: + json.dump(evidence, f, indent=2) + + logger.info("Synthetic evidence saved to %s", output_path) + + +if __name__ == "__main__": + main() diff --git a/fetchers/demo/hello/fetcher.yaml b/fetchers/demo/hello/fetcher.yaml new file mode 100644 index 0000000..a4195c8 --- /dev/null +++ b/fetchers/demo/hello/fetcher.yaml @@ -0,0 +1,24 @@ +name: demo_hello +version: 0.1.0 +description: > + Emits synthetic MFA-policy evidence so you can run the full + collect → envelope pipeline with no credentials. Not real evidence. +category: demo + +runtime: + type: python + entry: fetcher.py + +output: + type: json + path: demo_hello.json + +# The whole point of the demo: no secrets, no config, no targets. +secrets: [] + +evidence_set: + reference_id: EVD-DEMO-HELLO + name: Demo — Synthetic MFA Policy + instructions: > + Script: fetcher.py. Generates a fixed synthetic payload; requires no + credentials or network access. For trying the pipeline only. diff --git a/framework/api.py b/framework/api.py index 656ce2c..9444a1b 100644 --- a/framework/api.py +++ b/framework/api.py @@ -20,6 +20,9 @@ import importlib.util import json import os +import re +import shutil +import sys from datetime import datetime, timezone from pathlib import Path from typing import Any, Callable, Dict, List, Optional @@ -219,6 +222,113 @@ def ksi_coverage(root: Path) -> dict: } +# --------------------------------------------------------------------------- # +# Doctor — preflight environment check +# --------------------------------------------------------------------------- # + +# External CLIs a category's fetchers shell out to. Categories not listed here +# are pure-Python/HTTP fetchers that need no external tool. (AWS fetcher.sh files +# declare "Required tools: aws, jq"; k8s uses kubectl; checkov clones + scans.) +CATEGORY_TOOLS = { + "aws": ["aws", "jq"], + "k8s": ["kubectl"], + "checkov": ["checkov", "git"], +} + +_ENV_REF = re.compile(r"\$\{env:([^}]+)\}") + + +def doctor(root: Path, manifest_path: Optional[Path] = None) -> dict: + """Preflight check for running fetchers here. + + Reports the Python version, whether the external CLIs the relevant categories + need are on PATH, and — if a manifest is given — which secret env vars it + references are actually set. Presentation-agnostic; the CLI and TUI render + this one model. `ok` is a go/no-go: Python must clear the floor, and when a + manifest is supplied its categories' tools must be present and its secret env + vars set. Without a manifest, missing tools are informational (you may run + only some categories), so `ok` reflects the Python check alone. + """ + py = sys.version_info + python = { + "version": f"{py.major}.{py.minor}.{py.micro}", + "required": "3.10", + "ok": (py.major, py.minor) >= (3, 10), + } + + fetchers = discover_fetchers(root) + categories = sorted({f.category for f in fetchers.values() if f.category}) + tools_required = manifest_path is not None + + manifest_report = None + if manifest_path is not None: + data = read_manifest(manifest_path) + entries = (data.get("run") or {}).get("fetchers") or [] + used_cats = sorted({ + c for c in ( + fetchers[e["use"]].category + for e in entries + if e.get("use") in fetchers + ) + if c + }) + if used_cats: + categories = used_cats + + fetcher_reports = [] + manifest_ok = True + for e in entries: + refs: set = set() + for v in (e.get("secrets") or {}).values(): + refs |= set(_ENV_REF.findall(str(v))) + for t in e.get("targets") or []: + for v in (t.get("secrets") or {}).values(): + refs |= set(_ENV_REF.findall(str(v))) + missing = sorted(r for r in refs if not os.environ.get(r)) + manifest_ok = manifest_ok and not missing + fetcher_reports.append({ + "use": e.get("use"), + "env_refs": sorted(refs), + "missing": missing, + "ok": not missing, + }) + manifest_report = { + "path": str(manifest_path), + "fetchers": fetcher_reports, + "ok": manifest_ok, + } + + seen: set = set() + tools = [] + for cat in categories: + for tool in CATEGORY_TOOLS.get(cat, []): + if tool in seen: + continue + seen.add(tool) + resolved = shutil.which(tool) + tools.append({ + "name": tool, + "categories": [c for c in categories if tool in CATEGORY_TOOLS.get(c, [])], + "present": resolved is not None, + "path": resolved, + }) + tools.sort(key=lambda t: str(t["name"])) + + ok = python["ok"] + if tools_required: + tools_ok = all(t["present"] for t in tools) + ok = ok and tools_ok and (manifest_report["ok"] if manifest_report else True) + + return { + "python": python, + "categories": categories, + "tools": tools, + "tools_required": tools_required, + "manifest": manifest_report, + "ok": ok, + } + + # --------------------------------------------------------------------------- # # Manifest read / write # --------------------------------------------------------------------------- # diff --git a/framework/cli.py b/framework/cli.py index 2627382..0adca46 100644 --- a/framework/cli.py +++ b/framework/cli.py @@ -12,6 +12,7 @@ paramify catalog [--json] # categories -> fetchers -> fields paramify describe [--json] paramify ksi [--json] # FedRAMP 20x KSI coverage + paramify doctor [manifest] [--json] # preflight: python, CLIs, secrets paramify manifests [--json] # discovered run manifests paramify runs [--output-dir DIR] [--json] # past runs under an output dir paramify evidence [--json] # read one evidence file @@ -331,6 +332,49 @@ def ksi_cmd(json_out: bool = typer.Option(False, "--json", help="Emit JSON")): typer.echo(f" {u['id']:14s} {', '.join(u['fetchers'])}") +@app.command("doctor") +def doctor_cmd( + manifest: Optional[str] = typer.Argument( + None, help="Manifest to check secret env vars for (optional)" + ), + json_out: bool = typer.Option(False, "--json", help="Emit JSON"), +): + """Preflight: Python version, required CLIs on PATH, and (with a manifest) secrets.""" + root = api.find_repo_root() + rep = api.doctor(root, Path(manifest) if manifest else None) + if json_out: + typer.echo(json.dumps(rep, indent=2)) + raise typer.Exit(0 if rep["ok"] else 1) + + ok_mark, bad_mark = "✅", "❌" + p = rep["python"] + typer.echo(f"{ok_mark if p['ok'] else bad_mark} Python {p['version']} (need ≥ {p['required']})") + + if rep["tools"]: + heading = "Required CLIs" if rep["tools_required"] else "CLIs for discovered categories" + typer.echo(f"\n{heading}:") + for t in rep["tools"]: + mark = ok_mark if t["present"] else bad_mark + where = t["path"] or "not found on PATH" + typer.echo(f" {mark} {t['name']:8s} {where} ({', '.join(t['categories'])})") + if not rep["tools_required"]: + typer.echo(" (informational — you only need the CLIs for categories you run)") + + if rep["manifest"]: + m = rep["manifest"] + typer.echo(f"\nManifest secrets ({m['path']}):") + for fr in m["fetchers"]: + if not fr["env_refs"]: + typer.echo(f" {ok_mark} {fr['use']} (no secrets)") + elif fr["ok"]: + typer.echo(f" {ok_mark} {fr['use']} ({', '.join(fr['env_refs'])})") + else: + typer.echo(f" {bad_mark} {fr['use']} missing: {', '.join(fr['missing'])}") + + typer.echo(f"\n{'All good.' if rep['ok'] else 'Issues found — see above.'}") + raise typer.Exit(0 if rep["ok"] else 1) + + @app.command("manifests") def manifests_cmd(json_out: bool = typer.Option(False, "--json", help="Emit JSON")): """List discovered run manifests (manifests/*.yaml + legacy manifest.yaml).""" diff --git a/tests/test_cli.py b/tests/test_cli.py index 269944b..d7b26c8 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -47,8 +47,8 @@ def _registered(): EXPECTED_TOP = { - "list", "catalog", "describe", "manifests", "runs", "evidence", - "validate", "run", "upload", "manifest", "tui", + "list", "catalog", "describe", "ksi", "doctor", "manifests", "runs", + "evidence", "validate", "run", "upload", "manifest", "tui", } EXPECTED_MANIFEST = { "init", "new", "add", "remove", "set-config", "set-secret", @@ -63,6 +63,26 @@ def test_all_expected_commands_registered(): assert EXPECTED_MANIFEST <= manifest, f"missing manifest subcommands: {EXPECTED_MANIFEST - manifest}" +def test_doctor_json_ok_without_manifest(): + """Without a manifest, doctor is a Python-version gate; tools are advisory.""" + result = runner.invoke(app, ["doctor", "--json"]) + assert result.exit_code == 0, result.output + rep = json.loads(result.output) + assert rep["python"]["ok"] is True + assert rep["ok"] is True + assert rep["tools_required"] is False + assert rep["manifest"] is None + + +def test_doctor_with_credential_free_demo_manifest(): + """The demo manifest has no secrets, so doctor passes with a manifest too.""" + result = runner.invoke(app, ["doctor", "examples/demo.yaml", "--json"]) + assert result.exit_code == 0, result.output + rep = json.loads(result.output) + assert rep["tools_required"] is True + assert rep["manifest"]["ok"] is True + + # --------------------------------------------------------------------------- # # Parity invariant: every api function the TUI calls maps to a CLI command. # Derived from the TUI SOURCE so it can't drift into a tautology. diff --git a/tools/gen_ksi_coverage.py b/tools/gen_ksi_coverage.py new file mode 100644 index 0000000..467883e --- /dev/null +++ b/tools/gen_ksi_coverage.py @@ -0,0 +1,92 @@ +#!/usr/bin/env python3 +"""Regenerate the KSI-coverage block in README.md from live data. + +The FedRAMP 20x KSI coverage numbers rot as fetchers and mappings change, so we +generate them rather than hand-maintain them (the SEC-27 "generate the rot-prone +parts" pattern). This reads `framework.api.ksi_coverage()` — the same model +`paramify ksi` renders — and rewrites the README between the markers: + + ... + +Run it after changing fetcher `ksis` or the KSI reference: + + python tools/gen_ksi_coverage.py + +CI can run it and fail on any diff to guarantee the badge never lies. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(REPO_ROOT)) + +from framework import api # noqa: E402 + +BEGIN = "" +END = "" +BRAND = "1467ff" + + +def build_block(cov: dict) -> str: + s = cov["summary"] + pct = s["coverage_pct"] + badge = ( + f"![FedRAMP 20x KSI coverage]" + f"(https://img.shields.io/badge/FedRAMP_20x_KSI_coverage-{pct}%25-{BRAND})" + ) + + evidenceable_fams = [f for f in cov["families"] if f["evidenceable"] > 0] + organizational_fams = [f for f in cov["families"] if f["evidenceable"] == 0] + + rows = ["| Family | Covered | Gaps |", "|---|---|---|"] + for f in evidenceable_fams: + gaps = ", ".join(f"`{g}`" for g in f["gaps"]) if f["gaps"] else "—" + rows.append(f"| {f['name']} ({f['family']}) | {f['covered']} / {f['evidenceable']} | {gaps} |") + + lines = [ + badge, + "", + f"**{s['covered']} of {s['evidenceable']}** config-evidenceable KSIs covered — " + f"**{pct}%** — plus {s['organizational']} organizational KSIs (evidenced by " + f"HR / training / process, not cloud config). Straight from `paramify ksi`; " + f"regenerate with `python tools/gen_ksi_coverage.py`.", + "", + *rows, + ] + if organizational_fams: + names = ", ".join(f"{f['name']} ({f['family']})" for f in organizational_fams) + lines += ["", f"Organizational-only families (no cloud-config evidence): {names}."] + return "\n".join(lines) + + +def main() -> int: + readme = REPO_ROOT / "README.md" + text = readme.read_text() + if BEGIN not in text or END not in text: + # Coverage publishing is currently parked — the README has no + # ksi-coverage block. We're not yet confident enough in the KSI mapping + # to publish it; `paramify ksi` still shows live numbers. Re-add the + # BEGIN/END markers to the README to resume publishing, then rerun this. + print("ksi-coverage markers not in README.md — coverage publishing is parked; nothing to do.") + return 0 + + cov = api.ksi_coverage(REPO_ROOT) + block = build_block(cov) + + pre, rest = text.split(BEGIN, 1) + _, post = rest.split(END, 1) + new = f"{pre}{BEGIN}\n{block}\n{END}{post}" + + if new == text: + print(f"README.md already up to date ({cov['summary']['coverage_pct']}%).") + return 0 + readme.write_text(new) + print(f"README.md KSI-coverage block updated ({cov['summary']['coverage_pct']}%).") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())