From 4f5ac88f854658992745246da5179f08046a637c Mon Sep 17 00:00:00 2001 From: Tate McCauley Date: Thu, 9 Jul 2026 15:02:00 -0600 Subject: [PATCH 1/2] =?UTF-8?q?docs:=20README=20standout=20features=20?= =?UTF-8?q?=E2=80=94=20coverage,=20demo,=20doctor,=20AI-agent,=20badges?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four additions to make the repo land harder on first read: - FedRAMP 20x KSI coverage in the README (badge + family table + named gaps), generated from `paramify ksi` by tools/gen_ksi_coverage.py between markers so it never drifts (SEC-27 "generate the rot-prone parts" pattern). - Zero-credential demo: a synthetic `demo_hello` fetcher (new `demo` category) + examples/demo.yaml, so `paramify run examples/demo.yaml` exercises the whole collect → envelope pipeline with no cloud account. Doubles as a CI smoke fixture (SEC-34/SEC-38). - `paramify doctor [manifest]` (api.doctor + CLI): preflights Python version, the external CLIs each category needs (aws/jq, kubectl, checkov/git), and — with a manifest — which secret env vars are set; exits non-zero on gaps. - README polish: a badge row (CI/license/python/version) and a "Drive it with an AI agent" section leaning on --json everywhere, the {ok,errors} manifest contract, and the shipped .claude/skills. Tests: EXPECTED_TOP updated (adds ksi + doctor) + two doctor tests. 209 pass; ruff + mypy clean. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 89 ++++++++++++++++++++++++- examples/demo.yaml | 13 ++++ fetchers/_categories/demo.yaml | 4 ++ fetchers/demo/hello/fetcher.py | 61 +++++++++++++++++ fetchers/demo/hello/fetcher.yaml | 24 +++++++ framework/api.py | 110 +++++++++++++++++++++++++++++++ framework/cli.py | 44 +++++++++++++ tests/test_cli.py | 24 ++++++- tools/gen_ksi_coverage.py | 88 +++++++++++++++++++++++++ 9 files changed, 452 insertions(+), 5 deletions(-) create mode 100644 examples/demo.yaml create mode 100644 fetchers/_categories/demo.yaml create mode 100644 fetchers/demo/hello/fetcher.py create mode 100644 fetchers/demo/hello/fetcher.yaml create mode 100644 tools/gen_ksi_coverage.py diff --git a/README.md b/README.md index 438760b..6080c0b 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,12 @@ -[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/paramify/paramify-fetchers) - - # Paramify Fetchers +[![CI](https://github.com/paramify/paramify-fetchers/actions/workflows/ci.yml/badge.svg)](https://github.com/paramify/paramify-fetchers/actions/workflows/ci.yml) +[![License: GPLv3](https://img.shields.io/badge/License-GPLv3-1467ff.svg)](LICENSE) +[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-1467ff.svg)](pyproject.toml) +[![Version](https://img.shields.io/badge/version-0.2.0-1467ff.svg)](CHANGELOG.md) +[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/paramify/paramify-fetchers) + Fetchers are small scripts that collect compliance evidence from your infrastructure and write it to disk as JSON. A separate uploader stage pushes that evidence to Paramify. This repo contains the fetchers, the runner that executes them, and the uploader — the fetchers themselves never talk to Paramify directly. ``` @@ -20,6 +23,22 @@ fetcher's contract: ## Quick start +### See it work first — no credentials + +Curious before wiring up a real service? The bundled demo fetcher emits synthetic +evidence, so the whole collect → envelope pipeline runs with nothing but Python: + +```bash +git clone https://github.com/paramify/paramify-fetchers.git +cd paramify-fetchers +python -m venv .venv && source .venv/bin/activate && pip install -e . + +paramify run examples/demo.yaml # collect synthetic evidence — no credentials +paramify evidence evidence/run-*/demo_hello.json # inspect the enveloped result +``` + +Then come back and wire up your real services below. + **Prerequisites:** Python 3.10+. The CLIs your fetchers need (`aws`, `jq`, `curl`, `kubectl`, etc.) must be on your `PATH` — install only what applies to the categories you'll run. Each service's credential setup guide is in `fetchers//README.md`. ```bash @@ -100,6 +119,33 @@ Azure · and more --- +## FedRAMP 20x coverage + +Every fetcher can declare the FedRAMP 20x Key Security Indicators (KSIs) its +evidence speaks to; `paramify ksi` rolls that up into live coverage (`--json` +emits the full model, per-KSI): + + +![FedRAMP 20x KSI coverage](https://img.shields.io/badge/FedRAMP_20x_KSI_coverage-88.2%25-1467ff) + +**30 of 34** config-evidenceable KSIs covered — **88.2%** — plus 18 organizational KSIs (evidenced by HR / training / process, not cloud config). Straight from `paramify ksi`; regenerate with `python tools/gen_ksi_coverage.py`. + +| Family | Covered | Gaps | +|---|---|---| +| Cloud Native Architecture (CNA) | 5 / 7 | `KSI-CNA-04`, `KSI-CNA-07` | +| Service Configuration (SVC) | 6 / 7 | `KSI-SVC-05` | +| Monitoring, Logging & Auditing (MLA) | 5 / 6 | `KSI-MLA-02` | +| Identity & Access Management (IAM) | 7 / 7 | — | +| Change Management (CMT) | 4 / 4 | — | +| Recovery Planning (RPL) | 1 / 1 | — | +| Policy & Inventory (PIY) | 1 / 1 | — | +| Third-Party Information Resources (TPR) | 1 / 1 | — | + +Organizational-only families (no cloud-config evidence): Cybersecurity Education (CED), Incident Reporting (INR). + + +--- + ## How it runs Four pieces, kept deliberately separate: @@ -158,6 +204,8 @@ The CLI command surface: paramify list # discovered fetchers (flat) paramify catalog # categories → fetchers → editable fields paramify describe # one fetcher's config / secrets / target fields +paramify ksi # FedRAMP 20x KSI coverage +paramify doctor [manifest] # preflight: Python, required CLIs, manifest secrets paramify manifests # discovered run manifests (manifests/*.yaml) paramify validate # validate a manifest without running paramify run # run it @@ -167,6 +215,21 @@ paramify upload [run-dir] # push a run's evidence to Paramify (default: lat paramify manifest # build/edit a manifest (see below) ``` +Before a real run, `paramify doctor ` preflights the environment — +Python, the CLIs each category needs, and whether the manifest's secret env vars +are set — and exits non-zero if anything's missing, so it drops straight into CI: + +```text +$ paramify doctor examples/minimal_run.yaml +✅ Python 3.11.9 (need ≥ 3.10) + +Manifest secrets (examples/minimal_run.yaml): + ❌ okta_phishing_resistant_mfa missing: OKTA_API_TOKEN, OKTA_ORG_URL + ❌ gitlab_ci_cd_pipeline_config missing: GITLAB_TOKEN_1, GITLAB_TOKEN_2 + +Issues found — see above. +``` + Output lands in `/run-/`, one JSON file per fetcher (or per target for fan-out), alongside a `_run_metadata.json` run index. The runner wraps each evidence file in an envelope — @@ -267,6 +330,26 @@ example glue. --- +## Drive it with an AI agent + +Every command takes `--json`, and each `paramify manifest` edit returns a stable +`{ok, path, errors}` object — so an agent can assemble a runnable manifest by +reading `errors` and closing each gap, no screen-scraping: + +```bash +paramify catalog --json # discover what's available +paramify manifest add okta_phishing_resistant_mfa --json # → {"ok": false, "errors": [ …missing secrets… ]} +paramify manifest set-secret okta_phishing_resistant_mfa api_token OKTA_API_TOKEN --json +# …repeat until: +paramify validate manifest.yaml --json # → {"ok": true, "errors": []} +``` + +The repo also ships Claude Code skills under [`.claude/skills/`](.claude/skills/) — +`create-fetcher`, `wire-manifest`, and `suggest-validator` — so an agent can +scaffold a new fetcher, wire it into a manifest, or propose a validator directly. + +--- + ## Why the design is strict Every fetcher is forced through one contract, validated by JSON Schema, with a diff --git a/examples/demo.yaml b/examples/demo.yaml new file mode 100644 index 0000000..8a574a4 --- /dev/null +++ b/examples/demo.yaml @@ -0,0 +1,13 @@ +# Zero-credential demo. Run it as-is — no secrets, no cloud account: +# +# paramify run examples/demo.yaml +# paramify evidence evidence/run-*/demo_hello.json +# +# The demo_hello fetcher writes synthetic evidence, so this exercises the whole +# collect → envelope pipeline end to end. Delete once you've seen it work. + +run: + output_dir: ./evidence + + fetchers: + - use: demo_hello diff --git a/fetchers/_categories/demo.yaml b/fetchers/_categories/demo.yaml new file mode 100644 index 0000000..11ad4d7 --- /dev/null +++ b/fetchers/_categories/demo.yaml @@ -0,0 +1,4 @@ +# Demo category — a single credential-free fetcher that emits synthetic +# evidence, so you can exercise the whole run → envelope pipeline with no cloud +# account. Not real evidence; safe to delete. +description: Synthetic evidence for trying the pipeline with no credentials. diff --git a/fetchers/demo/hello/fetcher.py b/fetchers/demo/hello/fetcher.py new file mode 100644 index 0000000..3f71efe --- /dev/null +++ b/fetchers/demo/hello/fetcher.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python3 +""" +Demo fetcher: synthetic MFA-policy evidence. + +Writes a fixed, plausible-looking evidence payload so a newcomer can run the +whole pipeline — collect, envelope, inspect — without any credentials or network +access. This is NOT real evidence; it exists purely to demonstrate the shape of +a run. See `examples/demo.yaml`. +""" + +import json +import logging +import os +from datetime import datetime, timezone +from pathlib import Path + +logger = logging.getLogger("demo_hello") + + +def main(): + logging.basicConfig( + level=os.environ.get("LOG_LEVEL", "INFO"), + format="%(asctime)s %(levelname)s %(name)s %(message)s", + ) + + output_dir = Path(os.environ.get("EVIDENCE_DIR", "./evidence")) + output_dir.mkdir(parents=True, exist_ok=True) + + # A fixed, synthetic snapshot — the shape a real fetcher would collect, + # with obviously fake values. + evidence = { + "metadata": { + "source": "demo", + "note": "Synthetic evidence — not collected from any real system.", + "generated_at": datetime.now(timezone.utc).isoformat(), + }, + "results": [ + { + "policy": "require-phishing-resistant-mfa", + "enabled": True, + "applies_to": "all-users", + "enrolled_users": 42, + "total_users": 42, + }, + { + "policy": "block-legacy-authentication", + "enabled": True, + "applies_to": "all-users", + }, + ], + } + + output_path = output_dir / "demo_hello.json" + with open(output_path, "w") as f: + json.dump(evidence, f, indent=2) + + logger.info("Synthetic evidence saved to %s", output_path) + + +if __name__ == "__main__": + main() diff --git a/fetchers/demo/hello/fetcher.yaml b/fetchers/demo/hello/fetcher.yaml new file mode 100644 index 0000000..a4195c8 --- /dev/null +++ b/fetchers/demo/hello/fetcher.yaml @@ -0,0 +1,24 @@ +name: demo_hello +version: 0.1.0 +description: > + Emits synthetic MFA-policy evidence so you can run the full + collect → envelope pipeline with no credentials. Not real evidence. +category: demo + +runtime: + type: python + entry: fetcher.py + +output: + type: json + path: demo_hello.json + +# The whole point of the demo: no secrets, no config, no targets. +secrets: [] + +evidence_set: + reference_id: EVD-DEMO-HELLO + name: Demo — Synthetic MFA Policy + instructions: > + Script: fetcher.py. Generates a fixed synthetic payload; requires no + credentials or network access. For trying the pipeline only. diff --git a/framework/api.py b/framework/api.py index 656ce2c..9444a1b 100644 --- a/framework/api.py +++ b/framework/api.py @@ -20,6 +20,9 @@ import importlib.util import json import os +import re +import shutil +import sys from datetime import datetime, timezone from pathlib import Path from typing import Any, Callable, Dict, List, Optional @@ -219,6 +222,113 @@ def ksi_coverage(root: Path) -> dict: } +# --------------------------------------------------------------------------- # +# Doctor — preflight environment check +# --------------------------------------------------------------------------- # + +# External CLIs a category's fetchers shell out to. Categories not listed here +# are pure-Python/HTTP fetchers that need no external tool. (AWS fetcher.sh files +# declare "Required tools: aws, jq"; k8s uses kubectl; checkov clones + scans.) +CATEGORY_TOOLS = { + "aws": ["aws", "jq"], + "k8s": ["kubectl"], + "checkov": ["checkov", "git"], +} + +_ENV_REF = re.compile(r"\$\{env:([^}]+)\}") + + +def doctor(root: Path, manifest_path: Optional[Path] = None) -> dict: + """Preflight check for running fetchers here. + + Reports the Python version, whether the external CLIs the relevant categories + need are on PATH, and — if a manifest is given — which secret env vars it + references are actually set. Presentation-agnostic; the CLI and TUI render + this one model. `ok` is a go/no-go: Python must clear the floor, and when a + manifest is supplied its categories' tools must be present and its secret env + vars set. Without a manifest, missing tools are informational (you may run + only some categories), so `ok` reflects the Python check alone. + """ + py = sys.version_info + python = { + "version": f"{py.major}.{py.minor}.{py.micro}", + "required": "3.10", + "ok": (py.major, py.minor) >= (3, 10), + } + + fetchers = discover_fetchers(root) + categories = sorted({f.category for f in fetchers.values() if f.category}) + tools_required = manifest_path is not None + + manifest_report = None + if manifest_path is not None: + data = read_manifest(manifest_path) + entries = (data.get("run") or {}).get("fetchers") or [] + used_cats = sorted({ + c for c in ( + fetchers[e["use"]].category + for e in entries + if e.get("use") in fetchers + ) + if c + }) + if used_cats: + categories = used_cats + + fetcher_reports = [] + manifest_ok = True + for e in entries: + refs: set = set() + for v in (e.get("secrets") or {}).values(): + refs |= set(_ENV_REF.findall(str(v))) + for t in e.get("targets") or []: + for v in (t.get("secrets") or {}).values(): + refs |= set(_ENV_REF.findall(str(v))) + missing = sorted(r for r in refs if not os.environ.get(r)) + manifest_ok = manifest_ok and not missing + fetcher_reports.append({ + "use": e.get("use"), + "env_refs": sorted(refs), + "missing": missing, + "ok": not missing, + }) + manifest_report = { + "path": str(manifest_path), + "fetchers": fetcher_reports, + "ok": manifest_ok, + } + + seen: set = set() + tools = [] + for cat in categories: + for tool in CATEGORY_TOOLS.get(cat, []): + if tool in seen: + continue + seen.add(tool) + resolved = shutil.which(tool) + tools.append({ + "name": tool, + "categories": [c for c in categories if tool in CATEGORY_TOOLS.get(c, [])], + "present": resolved is not None, + "path": resolved, + }) + tools.sort(key=lambda t: str(t["name"])) + + ok = python["ok"] + if tools_required: + tools_ok = all(t["present"] for t in tools) + ok = ok and tools_ok and (manifest_report["ok"] if manifest_report else True) + + return { + "python": python, + "categories": categories, + "tools": tools, + "tools_required": tools_required, + "manifest": manifest_report, + "ok": ok, + } + + # --------------------------------------------------------------------------- # # Manifest read / write # --------------------------------------------------------------------------- # diff --git a/framework/cli.py b/framework/cli.py index 2627382..0adca46 100644 --- a/framework/cli.py +++ b/framework/cli.py @@ -12,6 +12,7 @@ paramify catalog [--json] # categories -> fetchers -> fields paramify describe [--json] paramify ksi [--json] # FedRAMP 20x KSI coverage + paramify doctor [manifest] [--json] # preflight: python, CLIs, secrets paramify manifests [--json] # discovered run manifests paramify runs [--output-dir DIR] [--json] # past runs under an output dir paramify evidence [--json] # read one evidence file @@ -331,6 +332,49 @@ def ksi_cmd(json_out: bool = typer.Option(False, "--json", help="Emit JSON")): typer.echo(f" {u['id']:14s} {', '.join(u['fetchers'])}") +@app.command("doctor") +def doctor_cmd( + manifest: Optional[str] = typer.Argument( + None, help="Manifest to check secret env vars for (optional)" + ), + json_out: bool = typer.Option(False, "--json", help="Emit JSON"), +): + """Preflight: Python version, required CLIs on PATH, and (with a manifest) secrets.""" + root = api.find_repo_root() + rep = api.doctor(root, Path(manifest) if manifest else None) + if json_out: + typer.echo(json.dumps(rep, indent=2)) + raise typer.Exit(0 if rep["ok"] else 1) + + ok_mark, bad_mark = "✅", "❌" + p = rep["python"] + typer.echo(f"{ok_mark if p['ok'] else bad_mark} Python {p['version']} (need ≥ {p['required']})") + + if rep["tools"]: + heading = "Required CLIs" if rep["tools_required"] else "CLIs for discovered categories" + typer.echo(f"\n{heading}:") + for t in rep["tools"]: + mark = ok_mark if t["present"] else bad_mark + where = t["path"] or "not found on PATH" + typer.echo(f" {mark} {t['name']:8s} {where} ({', '.join(t['categories'])})") + if not rep["tools_required"]: + typer.echo(" (informational — you only need the CLIs for categories you run)") + + if rep["manifest"]: + m = rep["manifest"] + typer.echo(f"\nManifest secrets ({m['path']}):") + for fr in m["fetchers"]: + if not fr["env_refs"]: + typer.echo(f" {ok_mark} {fr['use']} (no secrets)") + elif fr["ok"]: + typer.echo(f" {ok_mark} {fr['use']} ({', '.join(fr['env_refs'])})") + else: + typer.echo(f" {bad_mark} {fr['use']} missing: {', '.join(fr['missing'])}") + + typer.echo(f"\n{'All good.' if rep['ok'] else 'Issues found — see above.'}") + raise typer.Exit(0 if rep["ok"] else 1) + + @app.command("manifests") def manifests_cmd(json_out: bool = typer.Option(False, "--json", help="Emit JSON")): """List discovered run manifests (manifests/*.yaml + legacy manifest.yaml).""" diff --git a/tests/test_cli.py b/tests/test_cli.py index 269944b..d7b26c8 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -47,8 +47,8 @@ def _registered(): EXPECTED_TOP = { - "list", "catalog", "describe", "manifests", "runs", "evidence", - "validate", "run", "upload", "manifest", "tui", + "list", "catalog", "describe", "ksi", "doctor", "manifests", "runs", + "evidence", "validate", "run", "upload", "manifest", "tui", } EXPECTED_MANIFEST = { "init", "new", "add", "remove", "set-config", "set-secret", @@ -63,6 +63,26 @@ def test_all_expected_commands_registered(): assert EXPECTED_MANIFEST <= manifest, f"missing manifest subcommands: {EXPECTED_MANIFEST - manifest}" +def test_doctor_json_ok_without_manifest(): + """Without a manifest, doctor is a Python-version gate; tools are advisory.""" + result = runner.invoke(app, ["doctor", "--json"]) + assert result.exit_code == 0, result.output + rep = json.loads(result.output) + assert rep["python"]["ok"] is True + assert rep["ok"] is True + assert rep["tools_required"] is False + assert rep["manifest"] is None + + +def test_doctor_with_credential_free_demo_manifest(): + """The demo manifest has no secrets, so doctor passes with a manifest too.""" + result = runner.invoke(app, ["doctor", "examples/demo.yaml", "--json"]) + assert result.exit_code == 0, result.output + rep = json.loads(result.output) + assert rep["tools_required"] is True + assert rep["manifest"]["ok"] is True + + # --------------------------------------------------------------------------- # # Parity invariant: every api function the TUI calls maps to a CLI command. # Derived from the TUI SOURCE so it can't drift into a tautology. diff --git a/tools/gen_ksi_coverage.py b/tools/gen_ksi_coverage.py new file mode 100644 index 0000000..98df2fb --- /dev/null +++ b/tools/gen_ksi_coverage.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python3 +"""Regenerate the KSI-coverage block in README.md from live data. + +The FedRAMP 20x KSI coverage numbers rot as fetchers and mappings change, so we +generate them rather than hand-maintain them (the SEC-27 "generate the rot-prone +parts" pattern). This reads `framework.api.ksi_coverage()` — the same model +`paramify ksi` renders — and rewrites the README between the markers: + + ... + +Run it after changing fetcher `ksis` or the KSI reference: + + python tools/gen_ksi_coverage.py + +CI can run it and fail on any diff to guarantee the badge never lies. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(REPO_ROOT)) + +from framework import api # noqa: E402 + +BEGIN = "" +END = "" +BRAND = "1467ff" + + +def build_block(cov: dict) -> str: + s = cov["summary"] + pct = s["coverage_pct"] + badge = ( + f"![FedRAMP 20x KSI coverage]" + f"(https://img.shields.io/badge/FedRAMP_20x_KSI_coverage-{pct}%25-{BRAND})" + ) + + evidenceable_fams = [f for f in cov["families"] if f["evidenceable"] > 0] + organizational_fams = [f for f in cov["families"] if f["evidenceable"] == 0] + + rows = ["| Family | Covered | Gaps |", "|---|---|---|"] + for f in evidenceable_fams: + gaps = ", ".join(f"`{g}`" for g in f["gaps"]) if f["gaps"] else "—" + rows.append(f"| {f['name']} ({f['family']}) | {f['covered']} / {f['evidenceable']} | {gaps} |") + + lines = [ + badge, + "", + f"**{s['covered']} of {s['evidenceable']}** config-evidenceable KSIs covered — " + f"**{pct}%** — plus {s['organizational']} organizational KSIs (evidenced by " + f"HR / training / process, not cloud config). Straight from `paramify ksi`; " + f"regenerate with `python tools/gen_ksi_coverage.py`.", + "", + *rows, + ] + if organizational_fams: + names = ", ".join(f"{f['name']} ({f['family']})" for f in organizational_fams) + lines += ["", f"Organizational-only families (no cloud-config evidence): {names}."] + return "\n".join(lines) + + +def main() -> int: + readme = REPO_ROOT / "README.md" + text = readme.read_text() + if BEGIN not in text or END not in text: + print(f"error: markers {BEGIN} / {END} not found in README.md", file=sys.stderr) + return 1 + + cov = api.ksi_coverage(REPO_ROOT) + block = build_block(cov) + + pre, rest = text.split(BEGIN, 1) + _, post = rest.split(END, 1) + new = f"{pre}{BEGIN}\n{block}\n{END}{post}" + + if new == text: + print(f"README.md already up to date ({cov['summary']['coverage_pct']}%).") + return 0 + readme.write_text(new) + print(f"README.md KSI-coverage block updated ({cov['summary']['coverage_pct']}%).") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From cc775107046c1b60a21e5bd88d4136669aaf75b5 Mon Sep 17 00:00:00 2001 From: Tate McCauley Date: Thu, 9 Jul 2026 15:18:15 -0600 Subject: [PATCH 2/2] =?UTF-8?q?docs:=20lead=20README=20with=20TUI=20?= =?UTF-8?q?=E2=86=92=20AI=20=E2=86=92=20CLI;=20drop=20coverage=20+=20stric?= =?UTF-8?q?t=20sections?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reorder the "how to use it" narrative so the interactive TUI leads, then the AI-agent path, then the manual CLI — consolidating the scattered usage bits (command surface, doctor, manifest building, upload) under one "Using the CLI" section and slimming "How it runs" to architecture. Also, per review: - Remove the FedRAMP 20x coverage section (badge + family table + generated block). We're not yet confident enough in the KSI mapping to publish it; the generator (tools/gen_ksi_coverage.py) now parks gracefully when the README markers are absent, ready to resume later. `paramify ksi` still shows live numbers internally. - Remove the "Why the design is strict" section; preserve the pre-1.0 Status note by relocating it to the end of "How it runs". Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 361 ++++++++++++++++---------------------- tools/gen_ksi_coverage.py | 8 +- 2 files changed, 155 insertions(+), 214 deletions(-) diff --git a/README.md b/README.md index 6080c0b..96e34c1 100644 --- a/README.md +++ b/README.md @@ -14,65 +14,6 @@ Fetchers are small scripts that collect compliance evidence from your infrastruc (on disk, per run) (separate stage) ``` -The `paramify` CLI is the way in — list the catalog, then inspect any one -fetcher's contract: - -![Browsing the fetcher catalog with the paramify CLI](docs/demo/catalog.gif) - ---- - -## Quick start - -### See it work first — no credentials - -Curious before wiring up a real service? The bundled demo fetcher emits synthetic -evidence, so the whole collect → envelope pipeline runs with nothing but Python: - -```bash -git clone https://github.com/paramify/paramify-fetchers.git -cd paramify-fetchers -python -m venv .venv && source .venv/bin/activate && pip install -e . - -paramify run examples/demo.yaml # collect synthetic evidence — no credentials -paramify evidence evidence/run-*/demo_hello.json # inspect the enveloped result -``` - -Then come back and wire up your real services below. - -**Prerequisites:** Python 3.10+. The CLIs your fetchers need (`aws`, `jq`, `curl`, `kubectl`, etc.) must be on your `PATH` — install only what applies to the categories you'll run. Each service's credential setup guide is in `fetchers//README.md`. - -```bash -# 1. Clone and install -git clone https://github.com/paramify/paramify-fetchers.git -cd paramify-fetchers -python -m venv .venv && source .venv/bin/activate # recommended -pip install -e . -# To add the TUI: pip install -e '.[all]' - -# 2. Browse available fetchers by category -paramify catalog - -# 3. Start a manifest and wire in your fetchers -paramify manifest init -paramify manifest add okta_phishing_resistant_mfa -paramify manifest set-secret okta_phishing_resistant_mfa api_token OKTA_API_TOKEN -paramify manifest set-secret okta_phishing_resistant_mfa org_url OKTA_ORG_URL -# The manifest builder reports missing secrets after each step — -# keep going until it says the manifest is runnable. - -# 4. Set your credentials and run -export OKTA_API_TOKEN= -export OKTA_ORG_URL=https://your-org.okta.com -paramify validate manifest.yaml -paramify run manifest.yaml # evidence → ./evidence/run-/ - -# 5. Upload to Paramify -export PARAMIFY_UPLOAD_API_TOKEN= # see uploaders/paramify_evidence/README.md for setup -paramify upload # push the latest run -``` - -Use `paramify describe ` to see exactly what secrets and config any fetcher needs. Each service has a credential setup guide in its fetcher directory — for example, [`fetchers/okta/README.md`](fetchers/okta/README.md) covers creating an Okta API token and the required admin role. See [`examples/`](examples/) for complete worked manifests (multi-region AWS, GitLab fanout, etc.) and [`deploy/README.md`](deploy/README.md) for running on a schedule in Docker or Kubernetes. - --- ## Supported services @@ -119,86 +60,94 @@ Azure · and more --- -## FedRAMP 20x coverage +## Install -Every fetcher can declare the FedRAMP 20x Key Security Indicators (KSIs) its -evidence speaks to; `paramify ksi` rolls that up into live coverage (`--json` -emits the full model, per-KSI): +**Prerequisites:** Python 3.10+. The CLIs your fetchers need (`aws`, `jq`, `curl`, `kubectl`, etc.) must be on your `PATH` — install only what applies to the categories you'll run. Each service's credential setup guide is in `fetchers//README.md`. - -![FedRAMP 20x KSI coverage](https://img.shields.io/badge/FedRAMP_20x_KSI_coverage-88.2%25-1467ff) +```bash +git clone https://github.com/paramify/paramify-fetchers.git +cd paramify-fetchers +python -m venv .venv && source .venv/bin/activate +pip install -e '.[all]' # '[all]' bundles the TUI; use `pip install -e .` for the headless CLI only +``` -**30 of 34** config-evidenceable KSIs covered — **88.2%** — plus 18 organizational KSIs (evidenced by HR / training / process, not cloud config). Straight from `paramify ksi`; regenerate with `python tools/gen_ksi_coverage.py`. +There are three ways to drive it — an interactive **TUI**, an **AI agent**, or the **CLI** directly. All three go through one facade (`framework.api`), so they behave identically; pick whichever fits how you work. -| Family | Covered | Gaps | -|---|---|---| -| Cloud Native Architecture (CNA) | 5 / 7 | `KSI-CNA-04`, `KSI-CNA-07` | -| Service Configuration (SVC) | 6 / 7 | `KSI-SVC-05` | -| Monitoring, Logging & Auditing (MLA) | 5 / 6 | `KSI-MLA-02` | -| Identity & Access Management (IAM) | 7 / 7 | — | -| Change Management (CMT) | 4 / 4 | — | -| Recovery Planning (RPL) | 1 / 1 | — | -| Policy & Inventory (PIY) | 1 / 1 | — | -| Third-Party Information Resources (TPR) | 1 / 1 | — | +--- -Organizational-only families (no cloud-config evidence): Cybersecurity Education (CED), Incident Reporting (INR). - +## The TUI ---- +The fastest way in. `paramify tui` browses the catalog, builds and validates a +manifest, runs it, and reviews evidence — all without leaving the keyboard: -## How it runs +![The paramify terminal UI](docs/demo/tui.gif) -Four pieces, kept deliberately separate: +> **Zero-credential first run:** the bundled `demo_hello` fetcher emits synthetic +> evidence, so you can watch the whole collect → envelope pipeline before wiring +> up a real service: +> +> ```bash +> paramify run examples/demo.yaml # synthetic evidence — no credentials +> paramify evidence evidence/run-*/demo_hello.json # inspect the enveloped result +> ``` -- **Fetcher** — a small script (`fetcher.py` or `fetcher.sh`) that collects from - *one* source and writes a JSON file. It reads everything it needs from - environment variables and writes only to `EVIDENCE_DIR`. -- **`fetcher.yaml`** — the fetcher's self-description: its name, what secrets and - config it needs, what it outputs, and its `evidence_set` identity. Ships with - the code, validated against a schema. Customers never edit this. -- **Run manifest** — the customer's intent: which fetchers to run, with what - config, against what targets. Lives in the customer's environment, not here. -- **Runner** — reads `fetcher.yaml` files and a manifest, resolves secrets and - config into environment variables, and executes each fetcher. +--- -```mermaid -flowchart LR - subgraph infra["runs on customer infrastructure"] - direction LR - Y["fetcher.yaml
self-description"] --> R["runner"] - M["run manifest
which fetchers + config"] --> R - R -->|"secrets + config
as env vars"| F["fetcher
one source each"] - F -->|"raw JSON"| R - R -->|"wrap in envelope"| E[("evidence files
one run dir")] - E --> U["uploader"] - end - U -->|"Paramify REST v0 · HTTPS only"| P[("Paramify")] +## Drive it with an AI agent + +Every command takes `--json`, and each `paramify manifest` edit returns a stable +`{ok, path, errors}` object — so an agent can assemble a runnable manifest by +reading `errors` and closing each gap, no screen-scraping: + +```bash +paramify catalog --json # discover what's available +paramify manifest add okta_phishing_resistant_mfa --json # → {"ok": false, "errors": [ …missing secrets… ]} +paramify manifest set-secret okta_phishing_resistant_mfa api_token OKTA_API_TOKEN --json +# …repeat until: +paramify validate manifest.yaml --json # → {"ok": true, "errors": []} ``` -Everything goes through one facade, `framework.api` — discovery, manifest -editing, validation, and running. One CLI, `paramify`, sits on top of it and -steers every front-end; because they all share that single code path they behave -identically. Install it once from the repo (editable), then: +The repo also ships Claude Code skills under [`.claude/skills/`](.claude/skills/) — +`create-fetcher`, `wire-manifest`, and `suggest-validator` — so an agent can +scaffold a new fetcher, wire it into a manifest, or propose a validator directly. + +--- + +## Using the CLI + +The same operations, run by hand. `paramify catalog` lists the catalog and +`paramify describe ` shows exactly what any one fetcher needs: + +![Browsing the fetcher catalog with the paramify CLI](docs/demo/catalog.gif) + +A typical run, step by step: ```bash -pip install -e . # installs the `paramify` command - # (use `pip install -e '.[all]'` to add the TUI) +# 1. Browse available fetchers by category +paramify catalog -paramify # human CLI -paramify --json # same commands, machine-readable (for AI/scripts) -paramify tui # interactive terminal UI -``` +# 2. Start a manifest and wire in your fetchers +paramify manifest init +paramify manifest add okta_phishing_resistant_mfa +paramify manifest set-secret okta_phishing_resistant_mfa api_token OKTA_API_TOKEN +paramify manifest set-secret okta_phishing_resistant_mfa org_url OKTA_ORG_URL +# The manifest builder reports missing secrets after each step — +# keep going until it says the manifest is runnable. -> Back-compat: `python -m framework.runner ` and `python -m framework.tui` -> still work and are exactly equivalent to the corresponding `paramify` -> subcommands. +# 3. Set your credentials and run +export OKTA_API_TOKEN= +export OKTA_ORG_URL=https://your-org.okta.com +paramify validate manifest.yaml +paramify run manifest.yaml # evidence → ./evidence/run-/ -`paramify tui` drives that same facade interactively — browse the catalog, build -and validate a manifest, run it, and review evidence without leaving the keyboard: +# 4. Upload to Paramify +export PARAMIFY_UPLOAD_API_TOKEN= # see uploaders/paramify_evidence/README.md for setup +paramify upload # push the latest run +``` -![The paramify terminal UI](docs/demo/tui.gif) +Each service has a credential setup guide in its fetcher directory — for example, [`fetchers/okta/README.md`](fetchers/okta/README.md) covers creating an Okta API token and the required admin role. See [`examples/`](examples/) for complete worked manifests (multi-region AWS, GitLab fanout, etc.) and [`deploy/README.md`](deploy/README.md) for running on a schedule in Docker or Kubernetes. -The CLI command surface: +The full command surface: ```bash paramify list # discovered fetchers (flat) @@ -215,6 +164,10 @@ paramify upload [run-dir] # push a run's evidence to Paramify (default: lat paramify manifest # build/edit a manifest (see below) ``` +> Back-compat: `python -m framework.runner ` and `python -m framework.tui` +> still work and are exactly equivalent to the corresponding `paramify` +> subcommands. + Before a real run, `paramify doctor ` preflights the environment — Python, the CLIs each category needs, and whether the manifest's secret env vars are set — and exits non-zero if anything's missing, so it drops straight into CI: @@ -230,55 +183,6 @@ Manifest secrets (examples/minimal_run.yaml): Issues found — see above. ``` -Output lands in `/run-/`, one JSON file per fetcher -(or per target for fan-out), alongside a `_run_metadata.json` run index. The -runner wraps each evidence file in an envelope — -`{schema_version, metadata, payload}` — where `metadata` carries the fetcher -name/version/category, run id, target, `collected_at`, status, exit code, and -the `evidence_set` identity; failed invocations also get a `stderr_tail`. The -`_run_metadata.json` index itself is not enveloped. - -A finished evidence file looks like this — an AWS VPC-segmentation run, -abbreviated: - -```json -{ - "schema_version": "1.0", - "metadata": { - "fetcher_name": "aws_vpc_network_segmentation", - "fetcher_version": "0.1.0", - "category": "aws", - "run_id": "2026-06-16T15-56-41Z", - "target": { "region": "us-east-1" }, - "collected_at": "2026-06-16T16:00:14Z", - "status": "success", - "exit_code": 0, - "evidence_set": { - "reference_id": "EVD-VPC-SEGMENTATION", - "name": "VPC Network Segmentation", - "instructions": "Script: fetcher.sh. Commands: aws ec2 describe-vpcs, describe-subnets, describe-vpc-peering-connections, describe-vpc-endpoints. Maps to KSI-CNA-03.", - "description": "Lists VPCs, subnets, peering connections, and endpoints to document network topology and segmentation." - } - }, - "payload": { - "metadata": { "account_id": "111122223333", "region": "us-east-1", "datetime": "2026-06-16T16:00:14Z" }, - "results": [ - { "ResourceType": "Vpcs", "Items": [ - { "VpcId": "vpc-0a1b2c3d", "CidrBlock": "172.31.0.0/16", "IsDefault": true, "State": "available" } - ] }, - { "ResourceType": "Subnets", "Items": [ - { "SubnetId": "subnet-0d7e6de0", "VpcId": "vpc-0a1b2c3d", "CidrBlock": "172.31.80.0/20", "AvailabilityZone": "us-east-1b" } - ] } - ] - } -} -``` - -The runner owns the `metadata` envelope; the fetcher owns `payload`. The -`evidence_set` block (from `fetcher.yaml`) is what an uploaded file maps to in -Paramify. Note there is no pass/fail verdict — that judgment is Paramify-side, by -design (peering connections and endpoints are omitted above for brevity). - ### Building a manifest `paramify manifest ` edits a manifest file in place (`-f/--file`, default @@ -330,55 +234,88 @@ example glue. --- -## Drive it with an AI agent +## How it runs -Every command takes `--json`, and each `paramify manifest` edit returns a stable -`{ok, path, errors}` object — so an agent can assemble a runnable manifest by -reading `errors` and closing each gap, no screen-scraping: +Four pieces, kept deliberately separate: -```bash -paramify catalog --json # discover what's available -paramify manifest add okta_phishing_resistant_mfa --json # → {"ok": false, "errors": [ …missing secrets… ]} -paramify manifest set-secret okta_phishing_resistant_mfa api_token OKTA_API_TOKEN --json -# …repeat until: -paramify validate manifest.yaml --json # → {"ok": true, "errors": []} +- **Fetcher** — a small script (`fetcher.py` or `fetcher.sh`) that collects from + *one* source and writes a JSON file. It reads everything it needs from + environment variables and writes only to `EVIDENCE_DIR`. +- **`fetcher.yaml`** — the fetcher's self-description: its name, what secrets and + config it needs, what it outputs, and its `evidence_set` identity. Ships with + the code, validated against a schema. Customers never edit this. +- **Run manifest** — the customer's intent: which fetchers to run, with what + config, against what targets. Lives in the customer's environment, not here. +- **Runner** — reads `fetcher.yaml` files and a manifest, resolves secrets and + config into environment variables, and executes each fetcher. + +```mermaid +flowchart LR + subgraph infra["runs on customer infrastructure"] + direction LR + Y["fetcher.yaml
self-description"] --> R["runner"] + M["run manifest
which fetchers + config"] --> R + R -->|"secrets + config
as env vars"| F["fetcher
one source each"] + F -->|"raw JSON"| R + R -->|"wrap in envelope"| E[("evidence files
one run dir")] + E --> U["uploader"] + end + U -->|"Paramify REST v0 · HTTPS only"| P[("Paramify")] ``` -The repo also ships Claude Code skills under [`.claude/skills/`](.claude/skills/) — -`create-fetcher`, `wire-manifest`, and `suggest-validator` — so an agent can -scaffold a new fetcher, wire it into a manifest, or propose a validator directly. +Everything goes through one facade, `framework.api` — discovery, manifest +editing, validation, and running. The TUI, the CLI, and the `--json` surface an +agent drives all sit on that single code path, which is why they behave +identically. ---- +Output lands in `/run-/`, one JSON file per fetcher +(or per target for fan-out), alongside a `_run_metadata.json` run index. The +runner wraps each evidence file in an envelope — +`{schema_version, metadata, payload}` — where `metadata` carries the fetcher +name/version/category, run id, target, `collected_at`, status, exit code, and +the `evidence_set` identity; failed invocations also get a `stderr_tail`. The +`_run_metadata.json` index itself is not enveloped. + +A finished evidence file looks like this — an AWS VPC-segmentation run, +abbreviated: -## Why the design is strict - -Every fetcher is forced through one contract, validated by JSON Schema, with a -narrow set of allowed shapes. That rigidity is intentional. The previous -generation of fetchers were freeform scripts, and each one invented its own -conventions for config, secrets, and output — which is exactly why none of them -composed and the central catalog had to be hand-maintained in sync. A few -principles keep that from happening again: - -- **One contract, schema-enforced.** A fetcher declares itself in `fetcher.yaml`, - validated at discovery time. Anything not in the schema is not a thing a - fetcher can do. This is what lets the runner treat all 107 fetchers identically. -- **Fetchers run on customer infrastructure**, never Paramify's. So a fetcher - never assumes a Paramify connection, and the framework owns no scheduling. -- **Secrets are source-agnostic.** A fetcher reads `OKTA_API_TOKEN` from the - environment. It never knows or cares whether that came from a `.env` file, - AWS Secrets Manager, Vault, or a CI secret block — because every one of those - already knows how to set an environment variable. We do not write per-provider - secret integrations, and we don't intend to. -- **Collect facts; interpret elsewhere.** A fetcher gathers evidence. Whether - that evidence *satisfies* a control is a Paramify-side mapping, not the - fetcher's job. Keep pass/fail verdicts and compliance thresholds out of - fetchers. -- **One source per fetcher.** Cross-source comparison (e.g. Okta users vs. - Rippling employees) is a separate "comparator" that reads prior outputs — same - contract, different inputs. A fetcher never reads another fetcher's output. - -The full contract is in [`docs/fetcher_contract.md`](docs/fetcher_contract.md); -the rationale is in [`docs/design.md`](docs/design.md). +```json +{ + "schema_version": "1.0", + "metadata": { + "fetcher_name": "aws_vpc_network_segmentation", + "fetcher_version": "0.1.0", + "category": "aws", + "run_id": "2026-06-16T15-56-41Z", + "target": { "region": "us-east-1" }, + "collected_at": "2026-06-16T16:00:14Z", + "status": "success", + "exit_code": 0, + "evidence_set": { + "reference_id": "EVD-VPC-SEGMENTATION", + "name": "VPC Network Segmentation", + "instructions": "Script: fetcher.sh. Commands: aws ec2 describe-vpcs, describe-subnets, describe-vpc-peering-connections, describe-vpc-endpoints. Maps to KSI-CNA-03.", + "description": "Lists VPCs, subnets, peering connections, and endpoints to document network topology and segmentation." + } + }, + "payload": { + "metadata": { "account_id": "111122223333", "region": "us-east-1", "datetime": "2026-06-16T16:00:14Z" }, + "results": [ + { "ResourceType": "Vpcs", "Items": [ + { "VpcId": "vpc-0a1b2c3d", "CidrBlock": "172.31.0.0/16", "IsDefault": true, "State": "available" } + ] }, + { "ResourceType": "Subnets", "Items": [ + { "SubnetId": "subnet-0d7e6de0", "VpcId": "vpc-0a1b2c3d", "CidrBlock": "172.31.80.0/20", "AvailabilityZone": "us-east-1b" } + ] } + ] + } +} +``` + +The runner owns the `metadata` envelope; the fetcher owns `payload`. The +`evidence_set` block (from `fetcher.yaml`) is what an uploaded file maps to in +Paramify. Note there is no pass/fail verdict — that judgment is Paramify-side, by +design (peering connections and endpoints are omitted above for brevity). > **Status:** pre-1.0 (v0.x). The runner now wraps every output in the > `metadata`+`payload` envelope, but fetchers still write raw evidence dicts and diff --git a/tools/gen_ksi_coverage.py b/tools/gen_ksi_coverage.py index 98df2fb..467883e 100644 --- a/tools/gen_ksi_coverage.py +++ b/tools/gen_ksi_coverage.py @@ -66,8 +66,12 @@ def main() -> int: readme = REPO_ROOT / "README.md" text = readme.read_text() if BEGIN not in text or END not in text: - print(f"error: markers {BEGIN} / {END} not found in README.md", file=sys.stderr) - return 1 + # Coverage publishing is currently parked — the README has no + # ksi-coverage block. We're not yet confident enough in the KSI mapping + # to publish it; `paramify ksi` still shows live numbers. Re-add the + # BEGIN/END markers to the README to resume publishing, then rerun this. + print("ksi-coverage markers not in README.md — coverage publishing is parked; nothing to do.") + return 0 cov = api.ksi_coverage(REPO_ROOT) block = build_block(cov)