diff --git a/.gitignore b/.gitignore index 25cbe04..1a2a4fc 100644 --- a/.gitignore +++ b/.gitignore @@ -17,6 +17,9 @@ venv/ .env.* !.env.example +# --- Customer-side sync state (validator key -> Paramify id); never commit --- +.paramify/ + # --- Generated collect output (runner writes evidence/run-/) --- /evidence/ diff --git a/CHANGELOG.md b/CHANGELOG.md index d05ec2f..381d26d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,27 @@ schemas and the `paramify` CLI — not the internal code. ## [Unreleased] +### Added + +- Central `validators/` registry: validators are now first-class, deduplicated + objects (`validators//.yaml`, one file each), validated against + the new `framework/schemas/validator_schema.json`. Each validator carries the + Paramify fields as native YAML and owns its fetcher link via an + `evidence_sets` list of `reference_id`s, so a validator shared across fetchers + is one file — never a copy. See [`docs/validators_design.md`](docs/validators_design.md). +- Validator sync (`uploaders/paramify_validators/`, `paramify validators sync`, + and `paramify upload --with-validators`): pushes registry validators to + Paramify (`POST /validators`) and CONNECTs them to evidence sets. **Create-or-skip + by default** — a customer's tuned validator is never overwritten unless + `--update` is passed; `--dry-run` previews. Per-instance ids live in a + gitignored lock, not the shared registry. + +### Removed + +- The inline `validators` block on `fetcher.yaml` (shipped optional in + 0.2.0-beta, populated by no fetcher). Validators moved to the registry above; + `fetcher.yaml` keeps its `evidence_set` identity and `ksis`. + ## [0.2.1-beta] - 2026-07-10 ### Changed diff --git a/docs/fetcher_contract.md b/docs/fetcher_contract.md index cd0fa16..d7c2015 100644 --- a/docs/fetcher_contract.md +++ b/docs/fetcher_contract.md @@ -40,7 +40,7 @@ Every fetcher ships a `fetcher.yaml` in its directory. The schema is enforced; s | `depends_on` | array | Fetcher names this one depends on (not yet honored by the runner) | | `evidence_set` | object | Paramify evidence-set identity: `{reference_id, name, instructions?}`. Carried into envelope metadata and used by the uploader to get-or-create the set. | | `ksis` | array | FedRAMP 20x KSIs this fetcher's evidence speaks to (1+). Intrinsic to the fetcher; per-customer control mappings stay Paramify-side. | -| `validators` | array | Regex checks over the evidence payload that show the control is being implemented. Each entry: `{id, regex, proves?, failure_modes?}` (`id` + `regex` required); each regex matches the whole payload. | +| _validators_ | — | Not in `fetcher.yaml`. Validators are first-class objects in the central `validators/` registry, each linked to a fetcher by its `evidence_set.reference_id`. See [`validators_design.md`](validators_design.md). | --- diff --git a/docs/validators_design.md b/docs/validators_design.md new file mode 100644 index 0000000..70df40a --- /dev/null +++ b/docs/validators_design.md @@ -0,0 +1,210 @@ +# Validators — Design + +**Status:** Schema + registry defined (v0.x, 2026-07-14). +`framework/schemas/validator_schema.json` + the `validators/` registry; +`validators/_template/validator.yaml` is the starting point. +**Date:** 2026-07-14 +**Solves:** validators had only a thin, unused inline sketch in `fetcher.yaml` +(`{id, regex, proves?, failure_modes?}`, populated by zero fetchers) that could +not represent the real Paramify validator, and — being per-fetcher — had no way +to express a validator shared by several fetchers without copying it. + +--- + +## What a validator is + +A validator is a check over a fetcher's collected evidence that asserts the +control is **being implemented** — a collection-side assertion, not a +per-customer compliance judgment. Two kinds: + +- **AUTOMATED** — a `regex` over the evidence envelope whose capture groups are + compared by structured `validation_rules` (e.g. "the encrypted count equals + the total count"). +- **Attestation / manual** — no regex; the check is expressed as + `attestation_rules`. + +The fields mirror Paramify's own validator (`name`, `type`, `statement`, +`regex`, `rules_summary`, `validationRules_json`, `attestationRules_json`, +`evidence_sets`), stored as native YAML rather than stringified JSON. See the +field reference in `framework/schemas/validator_schema.json`. + +--- + +## The model: a deduplicated registry, linked on the validator + +**Decision: validators are first-class objects in a central `validators/` +registry — one file per validator — and each validator names every evidence set +it applies to. They are NOT stored inline in `fetcher.yaml`.** + +``` +validators/ + _template/validator.yaml + aws/alb_encryption_in_transit.yaml + aws/... + okta/... +``` + +Each file: + +```yaml +key: alb_encryption_in_transit # stable id == basename +name: ALB Encryption In Transit +type: AUTOMATED +role: configuration +statement: Ensures that all application load balancers are encrypting data in transit. +regex: '"alb_total":\s*(\d+)[\s\S]*?"alb_encrypted":\s*(\d+)' +rules_summary: MATCH_GROUP[1] EQUALS MATCH_GROUP[2] +validation_rules: + - regexOperation: { type: MATCH_GROUP, groupNumber: 1 } + criteria: EQUALS + value: { type: MATCH_GROUP, groupNumber: 2 } +attestation_rules: [] +evidence_sets: # <- the link lives HERE + - EVD-LB-ENC-STATUS + - EVD-LB-ENC-STATUS-MANUAL +``` + +### Why the link lives on the validator + +The central requirement: **a validator shared by several fetchers must not be +copied.** Because the fetcher→validator relationship is many-to-many (a fetcher +has many validators; a validator can serve many fetchers), putting the link on +the *validator* — a single `evidence_sets` list — means a shared validator is +one file that simply lists more sets. Putting it on the fetcher instead would +force either a copy per fetcher or a second layer of references. This is also +exactly how Paramify models it: the validator owns its evidence-set list (in the +API export, one validator row carries +`Load Balancer Encryption Status | Non-automated Load Balancer Encryption Status`). + +**A fetcher's validators are a reverse lookup:** every registry file whose +`evidence_sets` contains that fetcher's `evidence_set.reference_id`. A generator +or CLI does the walk; nothing verbose lands in `fetcher.yaml`. + +### Why a sidecar registry rather than inline + +Validators carry a regex, a human `rules_summary`, and a structured +`validation_rules`/`attestation_rules` block. Inlining all of that in +`fetcher.yaml` would bury the fetcher's identity under rule blobs and — fatally +— could not dedupe a shared validator. The old inline `validators` block is +removed from `fetcher_schema.json`; `fetcher.yaml` keeps only its `evidence_set` +identity and `ksis`. + +--- + +## Linking by `reference_id` + +`evidence_sets` entries are evidence-set **`reference_id`s** (e.g. +`EVD-LB-ENC-STATUS`), not display names. `reference_id` is the stable +idempotency key the uploader already get-or-creates sets by; display names +drift. A future importer that ingests a Paramify validator export (which lists +sets by name) maps name → `reference_id` via the fetcher registry. + +Some `evidence_sets` entries may name a **manual / non-automated** set that has +no fetcher in this repo (the "Non-automated …" sibling of an automated set). +That is expected: the registry is validator-centric, so it can reference sets +that live only Paramify-side. + +--- + +## Cardinality (recap of the confirmed evidence model) + +- **1 fetcher = 1 (automated) evidence set** — unchanged. When one collection + would feed two sets, split the fetcher, don't fan one artifact out. +- **Each evidence set carries exactly one `completeness` validator** (does the + evidence cover the full population? — often the minimum assessment scope) plus + **any number of `configuration` validators** (posture/config checks). `role:` + records which is which. This is why multiple validators per fetcher is the + norm, not the exception. + +--- + +## Uploading & associating (Paramify REST API v0.6.0) + +The API confirms the sync path — it is the script-sync pattern (decision #122), +just with `subjectType: VALIDATOR` instead of `SCRIPT`: + +- **Upsert the validator** — `POST /validators` (create) / `PATCH + /validators/{id}` (update). The body is a `oneOf` on `type`, and our registry + fields map 1:1: + - `AUTOMATED` → `{name, statement, type, regex, validationRules}` — our + `validation_rules` is the same object shape (`regexOperation`/`criteria`/`value`). + - `ATTESTATION` → `{name, statement, type, attestationRules}` — our + `attestation_rules` is the same shape. +- **Associate to an evidence set** — `POST /evidence/{evidenceId}/associate` + with `{associationType: CONNECT, subjectType: VALIDATOR, subjectId: + }`. Many-to-many falls out for free: one CONNECT per entry in the + validator's `evidence_sets`; `DISCONNECT` reverses it. +- **The evidence set** is get-or-created by `reference_id` via the existing + evidence uploader (`get_or_create_evidence_set`). A manual/non-automated + sibling set that has no fetcher is created (`automated: false`) before CONNECT. + +This is scoped and manifest-driven: sync only the validators whose +`evidence_sets` intersect the evidence sets the user's manifest actually +produces. + +### These are templates — create-or-skip, never clobber + +The shipped validators are **templates, ~80% correct**. A customer tunes them to +their environment (regex thresholds, `statement`, attestation questions) *inside +Paramify*. The sync must never destroy that tuning, so: + +- **Create-or-skip.** Sync creates a validator only when the customer's instance + doesn't already have it. If it exists, the sync **does not PATCH it** by + default — the customer's tuned copy is left exactly as-is. +- **Update is explicit and loud.** A `--update` flag can PATCH an existing + validator from the template (for the not-yet-tuned case), but only opt-in and + with a clear warning; `--dry-run` shows the exact calls first. Default runs + never update. +- **Association is always safe.** CONNECT (`/evidence/{id}/associate`) only wires + the validator to a set; it never touches validator content, so the sync always + ensures the association even when it skips the update. + +### Behavior after a partial upload + +`upload --with-validators` runs the validator sync even when some evidence files +in the run failed to upload, scoped to every evidence-set reference_id the run +*produced*. This is deliberate: a validator attaches to an evidence *set*, which +is get-or-created independently of whether every artifact uploaded, so a single +failed target should not block wiring the set's validators. The combined exit +code still reflects both stages. (If this proves too loose, the alternatives are +to skip the sync unless the upload was fully OK, or to scope only to sets whose +artifacts actually uploaded — both easy to add later.) + +### Per-instance ids live customer-side, not in the registry + +`GET /validators` filters only by `ids`/`type` — no name/reference filter (the +gap script-sync hit). So a validator's Paramify id is **resolved at sync time and +cached in customer-side state** (a gitignored lock, alongside the uploader +config) — never written back into the shared registry file, whose only identity +is `key`. First sync: list + match by `name` → adopt the existing id (no +duplicate) or create. This mirrors evidence sets exactly: the shared stable key +is `reference_id`, and the Paramify id is resolved per instance, never stored in +the repo. (Known limit: if a customer *renames* a validator before its id is +cached, a name-match can miss and create a duplicate — documented, and why the +lock is written on first create.) + +--- + +## What this pass does / does not do + +**Does:** define `validator_schema.json`, stand up the `validators/` registry +with a template, remove the dead inline `validators` block from the fetcher +schema, and document the model. + +**Does not (deferred):** + +- **No importer / backfill.** Converting a Paramify validator export into + registry files (matching `evidence_sets` → `reference_id`, minting `key`s) is + a separate tool. No real validators ship in this pass. +- ~~**No sync tool yet.**~~ **Built.** `uploaders/paramify_validators/syncer.py` + (create-or-skip reconcile + client + customer-side lock) plus `paramify + validators sync` and `paramify upload --with-validators` + (`framework/api.sync_validators`, scoped by manifest or by the reference_ids a + run produced). This intentionally supersedes the "validator linkage stays + manual" scope of decision #122. +- **No contract test yet.** A `validators/**/*.yaml` gate against + `validator_schema.json` (mirroring `tests/test_contracts.py`) should land with + the first real validators, so it doesn't assert on an empty set. +- **`suggest-validator` skill** still frames Paramify as the validator's only + home; update it to point an accepted suggestion at this registry when the + authoring path is built. diff --git a/framework/api.py b/framework/api.py index 9444a1b..773650e 100644 --- a/framework/api.py +++ b/framework/api.py @@ -689,6 +689,80 @@ def _load_paramify_uploader(root: Path): return module +def _load_paramify_validator_syncer(root: Path): + """Load the source-tree validator syncer without requiring uploaders/ to be packaged.""" + path = Path(root) / "uploaders" / "paramify_validators" / "syncer.py" + if not path.exists(): + raise RuntimeError(f"Paramify validator syncer not found at {path}") + spec = importlib.util.spec_from_file_location("paramify_validators_syncer", path) + if spec is None or spec.loader is None: + raise RuntimeError(f"Could not load Paramify validator syncer from {path}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def reference_ids_from_run(run_dir) -> set: + """The evidence-set reference_ids present in a run directory's envelopes. + + Lets `upload --with-validators` scope the sync to exactly the sets this run + produced, rather than a manifest. + """ + refs: set = set() + run_path = Path(run_dir) + if not run_path.is_dir(): + return refs + for p in sorted(run_path.glob("*.json")): + if p.name in ("_run_metadata.json", "upload_log.json"): + continue + try: + env = json.loads(p.read_text()) + except (OSError, json.JSONDecodeError): + continue + if not isinstance(env, dict): + continue + meta = env.get("metadata") + if not isinstance(meta, dict): + continue + es = meta.get("evidence_set") + if isinstance(es, dict) and es.get("reference_id"): + refs.add(es["reference_id"]) + return refs + + +def sync_validators( + root: Path, + manifest_path: Optional[Path] = None, + config_path: Optional[Path] = None, + *, + reference_ids=None, + dry_run: bool = False, + update: bool = False, + lock_path: Optional[str] = None, + on_event: Optional[Callable[[dict], None]] = None, +) -> dict: + """Sync registry validators to Paramify and associate them to evidence sets. + + Scope precedence: explicit `reference_ids` > `manifest_path`'s fetchers' + sets > whole registry. Create-or-skip; associates on create only; `update` + opt-in patches existing. Fires sync_start / sync_validator / sync_complete. + Raises ValueError for setup errors; returns the syncer summary otherwise. + """ + syncer = _load_paramify_validator_syncer(root) + validators = syncer.collect_validators(root, manifest_path, reference_ids) + config: dict = {} + if config_path: + config = yaml.safe_load(Path(config_path).read_text()) or {} + return syncer.sync_validators( + validators, + config=config, + dry_run=dry_run, + update=update, + lock_path=lock_path, + on_event=on_event, + ) + + def upload_preflight( run_dir, root: Path, diff --git a/framework/cli.py b/framework/cli.py index 0adca46..e8a93a8 100644 --- a/framework/cli.py +++ b/framework/cli.py @@ -73,6 +73,12 @@ help="Create/edit a manifest file (-f/--file, default ./manifest.yaml).", ) app.add_typer(manifest_app, name="manifest") +validators_app = typer.Typer( + no_args_is_help=True, + context_settings=_HELP_OPTS, + help="Sync registry validators to Paramify and associate them to evidence sets.", +) +app.add_typer(validators_app, name="validators") # --------------------------------------------------------------------------- # @@ -224,6 +230,41 @@ def on_event(ev: dict) -> None: return on_event +def _human_validator_printer(): + """Return an on_event callback for Paramify validator-sync progress.""" + def on_event(ev: dict) -> None: + kind = ev["event"] + if kind == "sync_start": + mode = " (dry-run)" if ev.get("dry_run") else "" + upd = " (update)" if ev.get("update") else "" + typer.echo(f"Sync {ev['validators']} validator(s) → {ev['base_url']}{mode}{upd}\n") + elif kind == "sync_validator": + outcome = ev.get("outcome") or "" + mark = { + "created": "OK", "updated": "OK", + "skipped_exists": "SKIP", + "would_create": "DRY", "would_update": "DRY", "would_skip_exists": "DRY", + }.get(outcome, "FAIL" if outcome == "error" else "..") + extra = "" + if ev.get("associated"): + extra += f" associated={','.join(ev['associated'])}" + if ev.get("set_not_found"): + extra += f" set_not_found={','.join(ev['set_not_found'])}" + if ev.get("error"): + extra += f" {ev['error']}" + typer.echo(f" [{mark}] {ev.get('key', '?')} {outcome}{extra}") + elif kind == "sync_complete": + typer.echo( + "\nDone: " + f"created={ev['created']} updated={ev['updated']} skipped={ev['skipped']} " + f"associated={ev['associated']} set_not_found={ev['set_not_found']} " + f"errors={ev['errors']}" + ) + if not ev.get("dry_run"): + typer.echo(f"lock → {ev['lock_path']}") + return on_event + + # --------------------------------------------------------------------------- # # Discover / describe # --------------------------------------------------------------------------- # @@ -539,6 +580,7 @@ def upload_cmd( output_dir: str = typer.Option("./evidence", "-o", "--output-dir", help="Base dir to find latest run"), config: Optional[str] = typer.Option(None, "--config", help="Uploader config YAML"), dry_run: bool = typer.Option(False, "--dry-run", help="Resolve and report what would upload; no API calls"), + with_validators: bool = typer.Option(False, "--with-validators", help="After upload, sync validators for the sets this run produced (create-or-skip)"), json_out: bool = typer.Option(False, "--json", help="Emit JSON summary"), ): """Upload one evidence run to Paramify.""" @@ -587,6 +629,65 @@ def upload_cmd( else: _err(f"Upload failed: {e}") raise typer.Exit(1) + + vsummary = None + if with_validators: + try: + refs = list(api.reference_ids_from_run(resolved_run_dir)) + vsummary = api.sync_validators( + root, + reference_ids=refs, + config_path=config_path, + dry_run=dry_run, + on_event=None if json_out else _human_validator_printer(), + ) + except Exception as e: # noqa: BLE001 + if json_out: + typer.echo(json.dumps( + {**summary, "ok": False, "validators": {"ok": False, "error": str(e)}}, + indent=2, default=str, + )) + else: + _err(f"Validator sync failed: {e}") + raise typer.Exit(1) + + # Combined success — top-level `ok` reflects BOTH stages, matching the exit code + # (a consumer reading .ok must not see true when the validator sync failed). + ok = summary["ok"] and (vsummary is None or vsummary["ok"]) + if json_out: + out = dict(summary) if vsummary is None else {**summary, "validators": vsummary} + out["ok"] = ok + typer.echo(json.dumps(out, indent=2, default=str)) + raise typer.Exit(0 if ok else 1) + + +@validators_app.command("sync") +def validators_sync_cmd( + manifest: Optional[str] = typer.Option(None, "-m", "--manifest", help="Scope to validators for this manifest's fetchers (default: whole registry)"), + config: Optional[str] = typer.Option(None, "--config", help="Syncer config YAML (base_url, lock_path)"), + lock: Optional[str] = typer.Option(None, "--lock", help="Lock file path (default ./.paramify/validators-sync.lock.json)"), + update: bool = typer.Option(False, "--update", help="Also PATCH existing validators (overwrites customer tuning)"), + dry_run: bool = typer.Option(False, "--dry-run", help="Report planned actions; make no writes"), + json_out: bool = typer.Option(False, "--json", help="Emit JSON summary"), +): + """Create/associate registry validators in Paramify. Create-or-skip by default.""" + root = api.find_repo_root() + try: + summary = api.sync_validators( + root, + Path(manifest).resolve() if manifest else None, + Path(config).resolve() if config else None, + dry_run=dry_run, + update=update, + lock_path=lock, + on_event=None if json_out else _human_validator_printer(), + ) + except Exception as e: # noqa: BLE001 — surface setup errors to CLI users + if json_out: + typer.echo(json.dumps({"ok": False, "errors": [str(e)]}, indent=2)) + else: + _err(f"Validator sync failed: {e}") + raise typer.Exit(1) if json_out: typer.echo(json.dumps(summary, indent=2, default=str)) raise typer.Exit(0 if summary["ok"] else 1) diff --git a/framework/contract.py b/framework/contract.py index ecde60f..63ddc31 100644 --- a/framework/contract.py +++ b/framework/contract.py @@ -40,6 +40,29 @@ class EvidenceSet: description: Optional[str] = None +@dataclass +class Validator: + """A validator from the central validators/ registry (validators//.yaml). + + A first-class, deduplicated object: defined once, it names every evidence set + it applies to via `evidence_sets` (reference_ids). `key` is the stable + repo-side identity; the Paramify id is resolved per instance at sync time and + never stored here. See framework/schemas/validator_schema.json and + docs/validators_design.md. + """ + key: str + name: str + type: str + statement: str + evidence_sets: List[str] + regex: Optional[str] = None + rules_summary: Optional[str] = None + role: Optional[str] = None + validation_rules: List[Any] = field(default_factory=list) + attestation_rules: List[Any] = field(default_factory=list) + path: Optional[Path] = None + + @dataclass class ConfigField: """A non-secret config knob a fetcher (or platform) accepts. diff --git a/framework/schemas/fetcher_schema.json b/framework/schemas/fetcher_schema.json index 2add6c6..aa96756 100644 --- a/framework/schemas/fetcher_schema.json +++ b/framework/schemas/fetcher_schema.json @@ -109,21 +109,6 @@ "type": "array", "description": "FedRAMP 20x Key Security Indicators this fetcher's evidence speaks to (1 or more). Intrinsic to the fetcher, not a per-customer control mapping (those stay Paramify-side).", "items": { "type": "string" } - }, - - "validators": { - "type": "array", - "description": "Regex checks over this fetcher's evidence that show the control is being implemented (collection-side assertions, not per-customer compliance judgment). Each regex is matched against the fetcher's evidence payload.", - "items": { - "type": "object", - "required": ["id", "regex"], - "properties": { - "id": { "type": "string" }, - "regex": { "type": "string" }, - "proves": { "type": "string", "description": "What passing this validator demonstrates." }, - "failure_modes": { "type": "array", "items": { "type": "string" } } - } - } } } } diff --git a/framework/schemas/validator_schema.json b/framework/schemas/validator_schema.json new file mode 100644 index 0000000..6f08940 --- /dev/null +++ b/framework/schemas/validator_schema.json @@ -0,0 +1,116 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "Validator Schema", + "description": "One validator in the central validators/ registry. A validator is a first-class, DEDUPLICATED object: it is defined exactly once and names — via `evidence_sets` — every evidence set it applies to. This mirrors Paramify's own model (the validator owns its evidence-set list), so a validator shared across fetchers is a single file, never a copy. Validators no longer live inline in fetcher.yaml. See docs/validators_design.md.", + "type": "object", + "additionalProperties": false, + "required": ["key", "name", "type", "statement", "evidence_sets"], + "properties": { + "key": { + "type": "string", + "pattern": "^[a-z0-9]+(?:_[a-z0-9]+)*$", + "description": "Stable repo identity, unique across the registry. Must equal the file's basename (e.g. alb_encryption_in_transit -> validators/aws/alb_encryption_in_transit.yaml). This is what tools reference; it never changes even when `name` does." + }, + "name": { + "type": "string", + "description": "Human-readable validator display name (Paramify `name`)." + }, + "type": { + "type": "string", + "enum": ["AUTOMATED", "ATTESTATION"], + "description": "Paramify validator type (REST API v0.6.0). AUTOMATED = a regex over artifact content evaluated by validation_rules. ATTESTATION = reviewer yes/no questions evaluated by attestation_rules (no regex)." + }, + "role": { + "type": "string", + "enum": ["completeness", "configuration"], + "description": "Repo-side classification of what this validator checks (NOT a Paramify field). `completeness` = 'do I have the full population?' — an evidence set has exactly one. `configuration` = a posture/configuration check — an evidence set may have many. See docs/validators_design.md." + }, + "statement": { + "type": "string", + "description": "The control statement — what passing this validator asserts (Paramify `statement`)." + }, + "regex": { + "type": "string", + "description": "Regex matched against the whole evidence envelope (schema_version + metadata + payload, as written to disk). Its capture groups feed validation_rules. Required for AUTOMATED validators." + }, + "rules_summary": { + "type": "string", + "description": "Human-readable summary of the pass/fail logic, e.g. 'MATCH_GROUP[1] EQUALS MATCH_GROUP[2]' (Paramify `rules_summary`). Derivable from validation_rules; kept for readability." + }, + "validation_rules": { + "type": "array", + "description": "Structured pass/fail logic for AUTOMATED validators. Maps 1:1 to the REST API v0.6.0 `validationRules` (native YAML instead of a JSON string). Rules evaluate in index order.", + "items": { + "type": "object", + "additionalProperties": true, + "required": ["regexOperation", "criteria", "value"], + "properties": { + "regexOperation": { + "type": "object", + "description": "How the regex result is extracted before comparison.", + "required": ["type"], + "properties": { + "type": { "type": "string", "enum": ["MATCH_COUNT", "MATCH_GROUP"], "description": "MATCH_COUNT = number of matches; MATCH_GROUP = contents of one capture group." }, + "groupNumber": { "type": "integer", "description": "Capture group; required when type is MATCH_GROUP." } + }, + "allOf": [ + { "if": { "properties": { "type": { "const": "MATCH_GROUP" } }, "required": ["type"] }, "then": { "required": ["groupNumber"] } } + ] + }, + "criteria": { + "type": "string", + "enum": ["CONTAINS", "DOES_NOT_CONTAIN", "ENDS_WITH", "DOES_NOT_END_WITH", "EQUALS", "NOT_EQUALS", "GREATER_THAN", "GREATER_THAN_OR_EQUAL_TO", "LESS_THAN", "LESS_THAN_OR_EQUAL_TO", "STARTS_WITH", "DOES_NOT_START_WITH"], + "description": "Comparison operator between the extracted value and `value`." + }, + "value": { + "type": "object", + "description": "Right-hand side of the comparison.", + "required": ["type"], + "properties": { + "type": { "type": "string", "enum": ["MATCH_COUNT", "MATCH_GROUP", "CUSTOM_TEXT"], "description": "MATCH_GROUP uses a capture group (groupNumber); CUSTOM_TEXT uses a literal (customText)." }, + "groupNumber": { "type": "integer" }, + "customText": { "type": "string" } + }, + "allOf": [ + { "if": { "properties": { "type": { "const": "MATCH_GROUP" } }, "required": ["type"] }, "then": { "required": ["groupNumber"] } }, + { "if": { "properties": { "type": { "const": "CUSTOM_TEXT" } }, "required": ["type"] }, "then": { "required": ["customText"] } } + ] + } + } + } + }, + "attestation_rules": { + "type": "array", + "description": "Structured logic for ATTESTATION validators. Maps 1:1 to the REST API v0.6.0 `attestationRules` (native YAML). Empty ([]) for AUTOMATED validators.", + "items": { + "type": "object", + "additionalProperties": true, + "required": ["question", "yesDisposition", "noDisposition", "nestedRules"], + "properties": { + "question": { "type": "string", "description": "The yes/no question presented to the reviewer." }, + "yesDisposition": { "type": "string", "enum": ["PASS", "FAIL", "BASED_ON_NESTED"], "description": "Outcome when answered Yes." }, + "noDisposition": { "type": "string", "enum": ["PASS", "FAIL", "BASED_ON_NESTED"], "description": "Outcome when answered No." }, + "nestedRules": { "type": "array", "description": "Follow-on rules when a disposition is BASED_ON_NESTED (nestable up to two more layers). Same item shape.", "items": { "type": "object", "additionalProperties": true } } + } + } + }, + "evidence_sets": { + "type": "array", + "minItems": 1, + "items": { "type": "string" }, + "description": "Every evidence set this validator applies to, by stable `reference_id` (matches a fetcher's evidence_set.reference_id). Listing more than one set here IS the dedup mechanism — one validator, many sets, no copies. A reference_id may name a manual/non-automated set that has no fetcher." + } + }, + "allOf": [ + { + "comment": "AUTOMATED validators must carry the regex + rules the API's POST /validators requires.", + "if": { "properties": { "type": { "const": "AUTOMATED" } }, "required": ["type"] }, + "then": { "required": ["regex", "validation_rules"] } + }, + { + "comment": "ATTESTATION validators must carry attestation_rules (and no regex logic).", + "if": { "properties": { "type": { "const": "ATTESTATION" } }, "required": ["type"] }, + "then": { "required": ["attestation_rules"] } + } + ] +} diff --git a/framework/validators.py b/framework/validators.py new file mode 100644 index 0000000..6b26806 --- /dev/null +++ b/framework/validators.py @@ -0,0 +1,99 @@ +"""Discover and scope validators in the central validators/ registry. + +Walks `validators//.yaml`, skipping any directory or file that +starts with `_` (e.g. `_template`), and validates each against +`validator_schema.json`. Mirrors `config_loader.discover_fetchers`. + +A validator owns its fetcher link via `evidence_sets` (reference_ids), so a +fetcher's validators are a reverse lookup: registry entries whose `evidence_sets` +intersect that fetcher's `evidence_set.reference_id`. See docs/validators_design.md. +""" + +import json +from pathlib import Path +from typing import Dict, Iterable, List, Set + +import yaml +from jsonschema import Draft202012Validator + +from framework.contract import Fetcher, Validator + + +def _load_schema(repo_root: Path) -> dict: + return json.loads( + (repo_root / "framework" / "schemas" / "validator_schema.json").read_text() + ) + + +def _parse_validator(data: dict, path: Path) -> Validator: + return Validator( + key=data["key"], + name=data["name"], + type=data["type"], + statement=data["statement"], + evidence_sets=list(data.get("evidence_sets") or []), + regex=data.get("regex"), + rules_summary=data.get("rules_summary"), + role=data.get("role"), + validation_rules=list(data.get("validation_rules") or []), + attestation_rules=list(data.get("attestation_rules") or []), + path=path.resolve(), + ) + + +def discover_validators(repo_root: Path) -> Dict[str, Validator]: + """Walk validators//.yaml. Returns {key: Validator}. + + Raises ValueError on schema-invalid yaml, a key that does not match its + filename, or a duplicate key. Returns {} when the registry is absent/empty. + """ + validators_root = repo_root / "validators" + if not validators_root.is_dir(): + return {} + + schema = _load_schema(repo_root) + validator = Draft202012Validator(schema) + + out: Dict[str, Validator] = {} + for category_dir in sorted(validators_root.iterdir()): + if not category_dir.is_dir() or category_dir.name.startswith("_"): + continue + for yaml_path in sorted(category_dir.glob("*.yaml")): + if yaml_path.name.startswith("_") or not yaml_path.is_file(): + continue + data = yaml.safe_load(yaml_path.read_text()) + errors = list(validator.iter_errors(data)) + if errors: + detail = "\n".join(f" {e.message}" for e in errors) + raise ValueError(f"{yaml_path}: schema validation failed:\n{detail}") + if data["key"] != yaml_path.stem: + raise ValueError( + f"{yaml_path}: key '{data['key']}' must equal the filename " + f"'{yaml_path.stem}'" + ) + if data["key"] in out: + raise ValueError( + f"Duplicate validator key '{data['key']}': " + f"{out[data['key']].path} and {yaml_path}" + ) + out[data["key"]] = _parse_validator(data, yaml_path) + + return out + + +def manifest_reference_ids(manifest: dict, fetchers: Dict[str, Fetcher]) -> Set[str]: + """The evidence-set reference_ids the manifest's fetchers produce.""" + refs: Set[str] = set() + for entry in (manifest.get("run") or {}).get("fetchers") or []: + name = entry.get("use") if isinstance(entry, dict) else None + fetcher = fetchers.get(name) if name else None + if fetcher and fetcher.evidence_set: + refs.add(fetcher.evidence_set.reference_id) + return refs + + +def select_validators( + validators: Iterable[Validator], reference_ids: Set[str] +) -> List[Validator]: + """Validators whose evidence_sets intersect the given reference_ids.""" + return [v for v in validators if set(v.evidence_sets) & reference_ids] diff --git a/tests/test_validator_api.py b/tests/test_validator_api.py new file mode 100644 index 0000000..33baba0 --- /dev/null +++ b/tests/test_validator_api.py @@ -0,0 +1,76 @@ +"""Facade-level tests for validator sync scoping (no network, no token). + +Covers the pieces Phase 2 added around the reconcile engine: pulling the +evidence-set reference_ids out of a run directory (what `upload --with-validators` +scopes by) and the registry-collection/scoping the syncer performs. The reconcile +engine itself is covered by test_validator_sync.py against a fake client. +""" + +from __future__ import annotations + +import importlib.util +import json +from pathlib import Path + +from framework import api + +REPO_ROOT = Path(__file__).resolve().parent.parent + + +def _load_syncer(): + path = REPO_ROOT / "uploaders" / "paramify_validators" / "syncer.py" + spec = importlib.util.spec_from_file_location("paramify_validators_syncer", path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _write_envelope(path: Path, reference_id: str) -> None: + path.write_text(json.dumps({ + "schema_version": "1.0", + "metadata": { + "fetcher_name": "aws_load_balancer_encryption_status", + "status": "success", + "evidence_set": {"reference_id": reference_id, "name": "Load Balancer Encryption Status"}, + }, + "payload": {"alb_total": 3, "alb_encrypted": 3}, + })) + + +def test_reference_ids_from_run(tmp_path): + _write_envelope(tmp_path / "lb.json", "EVD-LB-ENC-STATUS") + (tmp_path / "_run_metadata.json").write_text("{}") # must be ignored + refs = api.reference_ids_from_run(tmp_path) + assert refs == {"EVD-LB-ENC-STATUS"} + + +def test_reference_ids_from_missing_dir(): + assert api.reference_ids_from_run(REPO_ROOT / "no-such-run") == set() + + +def test_collect_validators_scoped_by_reference_ids(): + syncer = _load_syncer() + hit = syncer.collect_validators(REPO_ROOT, reference_ids=["EVD-LB-ENC-STATUS"]) + assert [v["key"] for v in hit] == ["alb_encryption_in_transit"] + # payload-facing dict maps validation_rules through unchanged + assert hit[0]["validation_rules"] and hit[0]["type"] == "AUTOMATED" + + miss = syncer.collect_validators(REPO_ROOT, reference_ids=["EVD-NOPE"]) + assert miss == [] + + +def test_collect_validators_scoped_by_manifest(tmp_path): + syncer = _load_syncer() + manifest = tmp_path / "m.yaml" + manifest.write_text( + "run:\n output_dir: ./evidence\n fetchers:\n" + " - use: aws_load_balancer_encryption_status\n" + ) + scoped = syncer.collect_validators(REPO_ROOT, manifest_path=manifest) + assert any(v["key"] == "alb_encryption_in_transit" for v in scoped) + + +def test_collect_validators_whole_registry_when_unscoped(): + syncer = _load_syncer() + everything = syncer.collect_validators(REPO_ROOT) + assert any(v["key"] == "alb_encryption_in_transit" for v in everything) diff --git a/tests/test_validator_sync.py b/tests/test_validator_sync.py new file mode 100644 index 0000000..90ba812 --- /dev/null +++ b/tests/test_validator_sync.py @@ -0,0 +1,196 @@ +"""Reconcile-engine unit tests for the validator syncer. + +Drives sync_validators against an in-memory fake client (no network) to pin the +create-or-skip / associate-on-create-only / --update / --dry-run / lock behaviour. +""" + +from __future__ import annotations + +import importlib.util +import json +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent +SYNCER_PATH = REPO_ROOT / "uploaders" / "paramify_validators" / "syncer.py" + + +def _load_syncer(): + spec = importlib.util.spec_from_file_location("paramify_validators_syncer", SYNCER_PATH) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +syncer = _load_syncer() + +BASE = "https://example.test/api/v0" + + +class FakeClient: + def __init__(self, existing=None, evidence_sets=None): + self.existing = existing or [] # [{"id","name"}] + self.evidence_sets = evidence_sets or {} # reference_id -> evidence_id + self.created, self.updated, self.associated = [], [], [] + self._n = 0 + + def list_validators(self): + return self.existing + + def create_validator(self, payload): + self._n += 1 + vid = f"new-{self._n}" + self.created.append((vid, payload)) + return vid + + def update_validator(self, vid, payload): + self.updated.append((vid, payload)) + + def find_evidence_set(self, ref): + return self.evidence_sets.get(ref) + + def associate_validator(self, evidence_id, validator_id): + self.associated.append((evidence_id, validator_id)) + + +def _alb(): + return { + "key": "alb_encryption_in_transit", + "name": "ALB Encryption In Transit", + "type": "AUTOMATED", + "statement": "Ensures ALBs encrypt in transit.", + "regex": r'"alb_total":\s*(\d+)', + "validation_rules": [{"regexOperation": {"type": "MATCH_COUNT"}, "criteria": "GREATER_THAN", "value": {"type": "CUSTOM_TEXT", "customText": "0"}}], + "attestation_rules": [], + "evidence_sets": ["EVD-LB-ENC-STATUS"], + } + + +def _run(validators, client, tmp_path, **kw): + return syncer.sync_validators( + validators, client=client, base_url=BASE, + lock_path=str(tmp_path / "lock.json"), **kw, + ) + + +def test_build_payload_automated_maps_rules(): + p = syncer.build_payload(_alb()) + assert p["type"] == "AUTOMATED" + assert "regex" in p and p["validationRules"] and "validation_rules" not in p + assert "attestationRules" not in p + + +def test_build_payload_attestation(): + v = {"key": "k", "name": "N", "type": "ATTESTATION", "statement": "s", + "attestation_rules": [{"question": "ok?", "yesDisposition": "PASS", "noDisposition": "FAIL", "nestedRules": []}], + "evidence_sets": ["E"]} + p = syncer.build_payload(v) + assert p["type"] == "ATTESTATION" and p["attestationRules"] and "regex" not in p + + +def test_create_and_associate_on_create(tmp_path): + c = FakeClient(evidence_sets={"EVD-LB-ENC-STATUS": "es-1"}) + s = _run([_alb()], c, tmp_path) + assert s["created"] == 1 and s["associated"] == 1 and s["ok"] + assert len(c.created) == 1 and c.associated == [("es-1", "new-1")] + # lock persisted key -> minted id + lock = json.loads((tmp_path / "lock.json").read_text())["validators"] + assert lock["alb_encryption_in_transit"] == "new-1" + + +def test_skip_when_locked_no_writes(tmp_path): + (tmp_path / "lock.json").write_text(json.dumps({"validators": {"alb_encryption_in_transit": "v1"}})) + c = FakeClient(evidence_sets={"EVD-LB-ENC-STATUS": "es-1"}) + s = _run([_alb()], c, tmp_path) + assert s["skipped"] == 1 and s["created"] == 0 + assert not c.created and not c.updated and not c.associated + + +def test_adopt_existing_by_name_no_create_no_associate(tmp_path): + c = FakeClient(existing=[{"id": "ex-9", "name": "ALB Encryption In Transit"}], + evidence_sets={"EVD-LB-ENC-STATUS": "es-1"}) + s = _run([_alb()], c, tmp_path) + assert s["skipped"] == 1 and s["created"] == 0 + assert not c.created and not c.associated # adoption != creation -> no wiring + lock = json.loads((tmp_path / "lock.json").read_text())["validators"] + assert lock["alb_encryption_in_transit"] == "ex-9" # id cached for next run + + +def test_update_patches_but_does_not_associate(tmp_path): + (tmp_path / "lock.json").write_text(json.dumps({"validators": {"alb_encryption_in_transit": "v1"}})) + c = FakeClient(evidence_sets={"EVD-LB-ENC-STATUS": "es-1"}) + s = _run([_alb()], c, tmp_path, update=True) + assert s["updated"] == 1 and s["skipped"] == 0 + assert c.updated == [("v1", syncer.build_payload(_alb()))] and not c.associated + + +def test_dry_run_makes_no_writes(tmp_path): + c = FakeClient(evidence_sets={"EVD-LB-ENC-STATUS": "es-1"}) + s = _run([_alb()], c, tmp_path, dry_run=True) + assert s["created"] == 1 and s["dry_run"] + assert not c.created and not c.associated and not c.updated + assert not (tmp_path / "lock.json").exists() # lock untouched in dry-run + + +def test_set_not_found_still_creates(tmp_path): + c = FakeClient(evidence_sets={}) # set missing + s = _run([_alb()], c, tmp_path) + assert s["created"] == 1 and s["associated"] == 0 and s["set_not_found"] == 1 + assert len(c.created) == 1 and not c.associated + + +def test_per_validator_error_isolated(tmp_path): + class Boom(FakeClient): + def create_validator(self, payload): + raise RuntimeError("api down") + c = Boom() + s = _run([_alb(), {**_alb(), "key": "second", "name": "Second"}], c, tmp_path) + assert s["errors"] == 2 and s["ok"] is False # both isolated, batch completes + + +def test_list_failure_fails_closed_no_duplicate_create(tmp_path): + """If GET /validators fails, reconcile must NOT create duplicates — it fails + closed (every not-yet-locked validator errors, none is created).""" + class ListBoom(FakeClient): + def list_validators(self): + raise RuntimeError("HTTP 500") + c = ListBoom(evidence_sets={"EVD-LB-ENC-STATUS": "es-1"}) + v2 = {**_alb(), "key": "second", "name": "Second"} + s = _run([_alb(), v2], c, tmp_path) # neither in lock -> both consult name_index + assert s["created"] == 0 and s["errors"] == 2 and s["ok"] is False + assert not c.created # crucially, NO spurious creates + + +def test_associate_failure_isolated_not_double_counted(tmp_path): + class AssocBoom(FakeClient): + def associate_validator(self, evidence_id, validator_id): + raise syncer.ParamifyError("assoc failed") + c = AssocBoom(evidence_sets={"EVD-A": "es-a", "EVD-B": "es-b"}) + v = {**_alb(), "evidence_sets": ["EVD-A", "EVD-B"]} + s = _run([v], c, tmp_path) + assert s["created"] == 1 and s["errors"] == 0 and s["associate_errors"] == 2 + assert s["ok"] is False # association failure surfaced, not swallowed + r = s["results"][0] + assert r["outcome"] == "created" and r["validator_id"] == "new-1" # id preserved + assert r.get("associate_failed") == ["EVD-A", "EVD-B"] + lock = json.loads((tmp_path / "lock.json").read_text())["validators"] + assert lock["alb_encryption_in_transit"] == "new-1" # locked -> no re-create + + +def test_update_emits_per_item_event(tmp_path): + (tmp_path / "lock.json").write_text(json.dumps({"validators": {"alb_encryption_in_transit": "v1"}})) + events = [] + c = FakeClient() + s = syncer.sync_validators( + [_alb()], client=c, base_url=BASE, update=True, + lock_path=str(tmp_path / "lock.json"), on_event=events.append, + ) + assert s["updated"] == 1 + upd = [e for e in events if e.get("event") == "sync_validator" and e.get("outcome") == "updated"] + assert len(upd) == 1 and upd[0]["validator_id"] == "v1" + + +def test_missing_key_isolated_not_crash(tmp_path): + c = FakeClient(evidence_sets={"EVD-LB-ENC-STATUS": "es-1"}) + bad = {k: val for k, val in _alb().items() if k != "key"} + s = _run([bad, _alb()], c, tmp_path) # must not raise + assert s["errors"] == 1 and s["created"] == 1 diff --git a/tests/test_validators_registry.py b/tests/test_validators_registry.py new file mode 100644 index 0000000..6686772 --- /dev/null +++ b/tests/test_validators_registry.py @@ -0,0 +1,74 @@ +"""Registry gate: every validators//.yaml satisfies validator_schema.json, +discovery does not raise, and the manifest-scoping helpers select correctly. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +import yaml +from jsonschema import Draft202012Validator + +from framework.config_loader import discover_fetchers +from framework.validators import ( + discover_validators, + manifest_reference_ids, + select_validators, +) + +REPO_ROOT = Path(__file__).resolve().parent.parent +VALIDATORS_ROOT = REPO_ROOT / "validators" +SCHEMA = json.loads((REPO_ROOT / "framework/schemas/validator_schema.json").read_text()) + + +def _validator_yamls() -> list[Path]: + out: list[Path] = [] + for category_dir in sorted(VALIDATORS_ROOT.iterdir()): + if not category_dir.is_dir() or category_dir.name.startswith("_"): + continue + for p in sorted(category_dir.glob("*.yaml")): + if not p.name.startswith("_"): + out.append(p) + return out + + +VALIDATOR_YAMLS = _validator_yamls() + + +def test_registry_not_empty() -> None: + assert VALIDATOR_YAMLS, "no validator files discovered under validators/" + + +@pytest.mark.parametrize( + "yaml_path", VALIDATOR_YAMLS, ids=[str(p.relative_to(REPO_ROOT)) for p in VALIDATOR_YAMLS] +) +def test_validator_matches_schema(yaml_path: Path) -> None: + validator = Draft202012Validator(SCHEMA) + data = yaml.safe_load(yaml_path.read_text()) + errors = sorted(validator.iter_errors(data), key=lambda e: list(e.path)) + msgs = "\n".join(f" {'/'.join(map(str, e.path)) or ''}: {e.message}" for e in errors) + assert not errors, f"{yaml_path.name} violates validator_schema.json:\n{msgs}" + assert data["key"] == yaml_path.stem, "key must equal filename stem" + + +def test_discovery_does_not_raise_and_dedupes() -> None: + registry = discover_validators(REPO_ROOT) + assert "alb_encryption_in_transit" in registry + assert len(registry) == len(VALIDATOR_YAMLS) + + +def test_manifest_scoping_selects_by_reference_id() -> None: + fetchers = discover_fetchers(REPO_ROOT) + registry = discover_validators(REPO_ROOT) + # The ALB validator points at EVD-LB-ENC-STATUS, produced by this fetcher. + manifest = {"run": {"fetchers": [{"use": "aws_load_balancer_encryption_status"}]}} + refs = manifest_reference_ids(manifest, fetchers) + assert "EVD-LB-ENC-STATUS" in refs + selected = select_validators(registry.values(), refs) + assert any(v.key == "alb_encryption_in_transit" for v in selected) + + # A manifest that produces none of the ALB sets selects nothing ALB-ish. + empty = select_validators(registry.values(), {"EVD-NOTHING-HERE"}) + assert all(v.key != "alb_encryption_in_transit" for v in empty) diff --git a/uploaders/paramify_validators/README.md b/uploaders/paramify_validators/README.md new file mode 100644 index 0000000..51e5cd7 --- /dev/null +++ b/uploaders/paramify_validators/README.md @@ -0,0 +1,53 @@ +# paramify_validators — validator sync + +Pushes validators from the central [`validators/`](../../validators/) registry to +Paramify and associates them to evidence sets. Runs as a stage after evidence +upload (the evidence sets it associates to should already exist). See +[`docs/validators_design.md`](../../docs/validators_design.md). + +## What it does, per validator + +1. **Resolve existence** — by a cached id in the lock file, else by matching the + validator's `name` against `GET /validators` (which has no name filter). +2. **Create-or-skip** — creates it (`POST /validators`) only if the instance + lacks it. An existing validator is **not** modified unless `--update` is + passed, so a customer's tuning is preserved. (These validators ship as + ~80%-correct templates that customers tune per environment.) +3. **Associate on create only** — right after creating, it CONNECTs the validator + to each of its `evidence_sets` (`POST /evidence/{id}/associate`, + `subjectType: VALIDATOR`). It never re-asserts wiring on a validator that + already existed. + +## Run it + +```bash +# Everything in the registry (dry-run: reports, writes nothing): +python uploaders/paramify_validators/syncer.py --dry-run + +# Scope to one manifest's fetchers, for real: +python uploaders/paramify_validators/syncer.py --manifest examples/aws_ambient.yaml + +# Pull an improved template onto NOT-yet-tuned validators (overwrites content): +python uploaders/paramify_validators/syncer.py --manifest ... --update +``` + +Auth: `PARAMIFY_UPLOAD_API_TOKEN` (same token as the evidence uploader). +Base URL: `PARAMIFY_API_BASE_URL` (default `https://app.paramify.com/api/v0`). + +## Lock file + +The per-instance `{validator key → Paramify id}` map lives in +`./.paramify/validators-sync.lock.json` (override with `--lock`). It is +**customer-side state, not template content** — gitignored, never in the shared +registry. It makes re-runs idempotent and immune to a validator being renamed in +Paramify after its id is cached. + +## Known limits (v0.x) + +- If a validator is **renamed** in Paramify *before* its id is cached, the + name-match misses and a duplicate is created. The lock exists to prevent this + on every subsequent run. +- **Association happens only at create time.** If a CONNECT fails (e.g. the + evidence set doesn't exist yet), a later run sees the validator as existing and + won't retry the association. Run `paramify upload` first so the sets exist, or + re-create after fixing. diff --git a/uploaders/paramify_validators/syncer.py b/uploaders/paramify_validators/syncer.py new file mode 100644 index 0000000..91c9dc1 --- /dev/null +++ b/uploaders/paramify_validators/syncer.py @@ -0,0 +1,467 @@ +#!/usr/bin/env python3 +"""Sync validators from the central registry to Paramify and associate them. + +A separate stage from evidence upload (per docs/validators_design.md): it reads +the `validators/` registry, scopes to the validators whose `evidence_sets` +intersect the sets a manifest produces, and for each one: + + 1. resolves whether the customer's Paramify instance already has it — by a + cached id (the lock file) or, failing that, by matching `name`, + 2. **creates it only if absent** (`POST /validators`); an existing validator is + never patched unless `--update` is passed, so customer tuning survives, + 3. **associates on create only** — after creating, CONNECTs the validator to + each of its evidence sets (`POST /evidence/{id}/associate`); it never + re-asserts wiring on a validator that already existed. + +The shipped validators are TEMPLATES (~80% right); customers tune them in +Paramify. That is why the default is create-or-skip and why the per-instance +validator id lives in a customer-side lock file, never in the shared registry. + +Auth: PARAMIFY_UPLOAD_API_TOKEN (source-agnostic env — .env, secret manager, CI). +""" + +import argparse +import json +import logging +import os +import sys +from datetime import datetime, timezone +from pathlib import Path +from typing import Callable, Dict, List, Optional +from urllib.parse import urlparse + +import requests +from dotenv import load_dotenv + +logger = logging.getLogger("paramify_validators_syncer") + +DEFAULT_BASE_URL = "https://app.paramify.com/api/v0" +DEFAULT_LOCK_PATH = "./.paramify/validators-sync.lock.json" +_REQUEST_TIMEOUT = 30 + + +def _utc_now() -> str: + return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + + +# --------------------------------------------------------------------------- # +# Paramify API client +# --------------------------------------------------------------------------- # +class ParamifyError(RuntimeError): + pass + + +class ValidatorClient: + """Thin client over the Paramify REST API v0 validator + associate endpoints.""" + + def __init__(self, token: str, base_url: str, timeout: int = _REQUEST_TIMEOUT): + self.base_url = base_url.rstrip("/") + self.timeout = timeout + self.session = requests.Session() + self.session.headers.update({"Authorization": f"Bearer {token}"}) + + def list_validators(self) -> List[Dict]: + """All validators on the instance. GET /validators exposes no name filter, + so name-matching (for reconcile) is done client-side over this list.""" + r = self.session.get(f"{self.base_url}/validators", timeout=self.timeout) + r.raise_for_status() + data = r.json() + return data.get("validators", []) if isinstance(data, dict) else (data or []) + + def create_validator(self, payload: Dict) -> str: + r = self.session.post( + f"{self.base_url}/validators", json=payload, timeout=self.timeout + ) + if r.status_code not in (200, 201): + raise ParamifyError( + f"create validator {payload.get('name')!r} failed " + f"(HTTP {r.status_code}): {r.text[:300]}" + ) + vid = r.json().get("id") + if not vid: + raise ParamifyError( + f"create validator {payload.get('name')!r} returned no id: {r.text[:300]}" + ) + return vid + + def update_validator(self, validator_id: str, payload: Dict) -> None: + r = self.session.patch( + f"{self.base_url}/validators/{validator_id}", + json=payload, + timeout=self.timeout, + ) + if r.status_code not in (200, 201, 204): + raise ParamifyError( + f"update validator {validator_id} failed " + f"(HTTP {r.status_code}): {r.text[:300]}" + ) + + def find_evidence_set(self, reference_id: str) -> Optional[str]: + """Return the evidence-set id for a reference_id, or None. Server-side filter.""" + r = self.session.get( + f"{self.base_url}/evidence", + params={"referenceId": reference_id}, + timeout=self.timeout, + ) + r.raise_for_status() + for ev in r.json().get("evidences", []): + if ev.get("referenceId") == reference_id: + return ev.get("id") + return None + + def associate_validator(self, evidence_id: str, validator_id: str) -> None: + body = { + "associationType": "CONNECT", + "subjectType": "VALIDATOR", + "subjectId": validator_id, + } + r = self.session.post( + f"{self.base_url}/evidence/{evidence_id}/associate", + json=body, + timeout=self.timeout, + ) + if r.status_code not in (200, 201, 204): + raise ParamifyError( + f"associate validator {validator_id} to set {evidence_id} failed " + f"(HTTP {r.status_code}): {r.text[:300]}" + ) + + +# --------------------------------------------------------------------------- # +# Helpers +# --------------------------------------------------------------------------- # +def _base_url_error(base_url: str) -> Optional[str]: + parsed = urlparse(base_url) + if parsed.scheme != "https" and (parsed.hostname or "") not in ("localhost", "127.0.0.1", "::1"): + return ( + "base_url must be https to protect the API token " + f"(got {base_url!r}); only localhost may use http" + ) + return None + + +def build_payload(v: Dict) -> Dict: + """Registry validator dict -> Paramify create/update body. + + Maps validation_rules -> validationRules / attestation_rules -> + attestationRules; repo-side fields (key, role, rules_summary) are not sent. + """ + vtype = v["type"] + payload = {"name": v["name"], "statement": v["statement"], "type": vtype} + if vtype == "AUTOMATED": + payload["regex"] = v.get("regex") + payload["validationRules"] = v.get("validation_rules") or [] + else: # ATTESTATION + payload["attestationRules"] = v.get("attestation_rules") or [] + return payload + + +def load_lock(lock_path: Path) -> Dict[str, str]: + """Read the customer-side {key: paramify_id} lock, or {} if absent/unreadable.""" + try: + data = json.loads(lock_path.read_text()) + except (OSError, json.JSONDecodeError): + return {} + return data.get("validators", {}) if isinstance(data, dict) else {} + + +def write_lock(lock_path: Path, mapping: Dict[str, str]) -> Optional[Path]: + try: + lock_path.parent.mkdir(parents=True, exist_ok=True) + lock_path.write_text( + json.dumps( + {"updated_at": _utc_now(), "validators": mapping}, indent=2, sort_keys=True + ) + ) + return lock_path + except OSError as e: + logger.warning("could not write lock %s (%s)", lock_path, e) + return None + + +def _emit(on_event: Optional[Callable[[dict], None]], event: dict) -> None: + if on_event is not None: + on_event(event) + + +# --------------------------------------------------------------------------- # +# Reconcile engine +# --------------------------------------------------------------------------- # +def sync_validators( + validators: List[Dict], + *, + client: Optional[ValidatorClient] = None, + token: Optional[str] = None, + base_url: Optional[str] = None, + dry_run: bool = False, + update: bool = False, + lock_path: Optional[str] = None, + on_event: Optional[Callable[[dict], None]] = None, + config: Optional[Dict] = None, +) -> Dict: + """Reconcile a list of registry validator dicts against Paramify. + + Create-or-skip; associate on create only; `update` opt-in patches existing; + `dry_run` performs no writes (reads only, to report accurately, when a client + is available). Returns a summary dict; never raises for per-validator errors + (those are isolated and counted), only for setup errors. + """ + load_dotenv() + config = config or {} + paramify_cfg = config.get("paramify") or {} + base_url = ( + paramify_cfg.get("base_url") + or base_url + or os.environ.get("PARAMIFY_API_BASE_URL") + or DEFAULT_BASE_URL + ) + url_error = _base_url_error(base_url) + if url_error: + logger.error(url_error) + raise ValueError(url_error) + + lock_file = Path( + lock_path or (config.get("validators") or {}).get("lock_path") or DEFAULT_LOCK_PATH + ) + lock = load_lock(lock_file) + + token = token or os.environ.get("PARAMIFY_UPLOAD_API_TOKEN") + if client is None: + # Build a client whenever we have a token — even in dry-run, so reads can + # report create-vs-skip precisely. Writes stay gated on `not dry_run`. + if token: + client = ValidatorClient(token, base_url) + elif not dry_run: + msg = "PARAMIFY_UPLOAD_API_TOKEN is not set" + logger.error(msg) + raise ValueError(msg) + + _emit(on_event, { + "event": "sync_start", + "base_url": base_url, + "dry_run": dry_run, + "update": update, + "validators": len(validators), + "lock_path": str(lock_file), + }) + + # Lazily fetched name -> id index over existing validators (for reconcile). + _name_index: Optional[Dict[str, str]] = None + + def name_index() -> Dict[str, str]: + nonlocal _name_index + if _name_index is None: + if client is None: + _name_index = {} + else: + # Build fully, THEN cache. If list_validators() raises, we do NOT + # cache a half/empty index — the exception re-raises per validator + # (each isolated as an error), so a transient GET failure can never + # leave reconcile blind and create duplicates. Fail closed. + index: Dict[str, str] = {} + for ex in client.list_validators(): + if ex.get("name") and ex.get("id"): + index.setdefault(ex["name"], ex["id"]) + _name_index = index + return _name_index + + results: List[Dict] = [] + created = updated = skipped = associated = set_not_found = errors = assoc_errors = 0 + + def add_result(r: Dict) -> None: + results.append(r) + _emit(on_event, {"event": "sync_validator", **r}) + + for v in validators: + key = v.get("key", "") # fallback label if the dict is malformed + try: + key = v["key"] # inside try: a missing key is an isolated error, not a crash + existing_id = lock.get(key) + adopted = False + if existing_id is None and client is not None: + existing_id = name_index().get(v["name"]) + if existing_id: + lock[key] = existing_id + adopted = True + + payload = build_payload(v) + refs = v.get("evidence_sets", []) + + # ---- create (absent) -> create, then associate on create only ---- + if existing_id is None: + if dry_run: + created += 1 + add_result({ + "key": key, "outcome": "would_create", "evidence_sets": refs, + }) + continue + assert client is not None # not dry_run => token present or we raised + vid = client.create_validator(payload) + lock[key] = vid # persist before associating so a re-run won't recreate + created += 1 + # A failed association must NOT undo/duplicate the created validator + # nor mark the whole validator an error — isolate per set. + assoc, missing, failed = [], [], [] + for ref in refs: + try: + eid = client.find_evidence_set(ref) + if eid: + client.associate_validator(eid, vid) + assoc.append(ref) + associated += 1 + else: + missing.append(ref) + set_not_found += 1 + except Exception as ae: + failed.append(ref) + assoc_errors += 1 + logger.error("%s: associate to %s failed: %s", key, ref, ae) + result = { + "key": key, "outcome": "created", "validator_id": vid, + "associated": assoc, "set_not_found": missing, + } + if failed: + result["associate_failed"] = failed + add_result(result) + continue + + # ---- exists -> skip, or (opt-in) update; never re-associate ---- + if update: + if dry_run: + add_result({"key": key, "outcome": "would_update", "validator_id": existing_id}) + else: + assert client is not None # not dry_run => token present or we raised + client.update_validator(existing_id, payload) + add_result({"key": key, "outcome": "updated", "validator_id": existing_id}) + updated += 1 + continue + + skipped += 1 + add_result({ + "key": key, + "outcome": "would_skip_exists" if dry_run else "skipped_exists", + "validator_id": existing_id, + "adopted": adopted, + }) + except Exception as e: # per-validator isolation + logger.error("%s: sync failed: %s", key, e) + errors += 1 + add_result({"key": key, "outcome": "error", "error": str(e)[:300]}) + continue + + lock_written = None + if not dry_run: + lock_written = write_lock(lock_file, lock) + + summary = { + "base_url": base_url, + "dry_run": dry_run, + "update": update, + "validators": len(validators), + "created": created, + "updated": updated, + "skipped": skipped, + "associated": associated, + "set_not_found": set_not_found, + "errors": errors, + "associate_errors": assoc_errors, + "results": results, + "lock_path": str(lock_written) if lock_written else str(lock_file), + "ok": errors == 0 and assoc_errors == 0, + } + logger.info( + "Done: created=%d updated=%d skipped=%d associated=%d set_not_found=%d " + "errors=%d associate_errors=%d%s", + created, updated, skipped, associated, set_not_found, errors, assoc_errors, + " (dry-run)" if dry_run else "", + ) + _emit(on_event, {"event": "sync_complete", **summary}) + return summary + + +# --------------------------------------------------------------------------- # +# Registry collection (standalone use) — imports framework discovery +# --------------------------------------------------------------------------- # +def _validator_to_dict(v) -> Dict: + return { + "key": v.key, + "name": v.name, + "type": v.type, + "statement": v.statement, + "regex": v.regex, + "validation_rules": v.validation_rules, + "attestation_rules": v.attestation_rules, + "evidence_sets": v.evidence_sets, + } + + +def collect_validators( + root: Path, + manifest_path: Optional[Path] = None, + reference_ids: Optional[List[str]] = None, +) -> List[Dict]: + """Discover the registry and scope the selection. + + Precedence: an explicit `reference_ids` set (e.g. what an evidence run + produced) > a `manifest`'s fetchers' sets > the whole registry. + """ + from framework import api + from framework.config_loader import discover_fetchers + from framework.validators import ( + discover_validators, + manifest_reference_ids, + select_validators, + ) + + registry = discover_validators(root) + if reference_ids is not None: + selected = select_validators(registry.values(), set(reference_ids)) + elif manifest_path is not None: + fetchers = discover_fetchers(root) + manifest = api.read_manifest(manifest_path) + refs = manifest_reference_ids(manifest, fetchers) + selected = select_validators(registry.values(), refs) + else: + selected = list(registry.values()) + return [_validator_to_dict(v) for v in selected] + + +# --------------------------------------------------------------------------- # +# Main +# --------------------------------------------------------------------------- # +def main(argv=None) -> int: + logging.basicConfig( + level=os.environ.get("LOG_LEVEL", "INFO"), + format="%(asctime)s %(levelname)s %(name)s %(message)s", + ) + load_dotenv() + + parser = argparse.ArgumentParser(description="Sync registry validators to Paramify") + parser.add_argument("--root", default=".", help="Repo root containing validators/ (default .)") + parser.add_argument("--manifest", help="Scope to validators for this manifest's fetchers") + parser.add_argument("--lock", help=f"Lock file path (default {DEFAULT_LOCK_PATH})") + parser.add_argument("--update", action="store_true", help="Also PATCH existing validators (overwrites tuning)") + parser.add_argument("--dry-run", action="store_true", help="Report planned actions; make no writes") + args = parser.parse_args(argv) + + root = Path(args.root).resolve() + try: + validators = collect_validators(root, Path(args.manifest) if args.manifest else None) + except (ValueError, RuntimeError) as e: + logger.error("could not collect validators: %s", e) + return 1 + if not validators: + logger.info("No validators in scope — nothing to sync.") + return 0 + + try: + summary = sync_validators( + validators, dry_run=args.dry_run, update=args.update, lock_path=args.lock + ) + except ValueError: + return 1 + return 0 if summary["ok"] else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/uploaders/paramify_validators/syncer.yaml b/uploaders/paramify_validators/syncer.yaml new file mode 100644 index 0000000..16ee96e --- /dev/null +++ b/uploaders/paramify_validators/syncer.yaml @@ -0,0 +1,27 @@ +name: paramify_validators +version: 0.1.0 +description: > + Syncs validators from the central validators/ registry to Paramify and + associates them to evidence sets. Create-or-skip (never clobbers customer + tuning); associates on create only. Idempotent via a customer-side lock file. + +runtime: + type: python + entry: syncer.py + +# Source-agnostic: the token can come from .env, a secret manager, CI vars, etc. +# Same token as the evidence uploader, so one credential covers both stages. +secrets: + - name: api_token + env: PARAMIFY_UPLOAD_API_TOKEN + +config_schema: + base_url: + type: string + default: https://app.paramify.com/api/v0 + env: PARAMIFY_API_BASE_URL + description: Paramify REST API v0 base URL. + lock_path: + type: string + default: ./.paramify/validators-sync.lock.json + description: Customer-side {key -> Paramify validator id} map; never committed. diff --git a/validators/README.md b/validators/README.md new file mode 100644 index 0000000..b95f410 --- /dev/null +++ b/validators/README.md @@ -0,0 +1,29 @@ +# Validators registry + +Central, deduplicated home for Paramify **validators** — the regex/attestation +checks that assert a fetcher's evidence structurally shows a control is being +implemented. Validators used to be sketched inline in `fetcher.yaml`; they now +live here as first-class objects. + +## Model + +- **One validator = one file:** `validators//.yaml`, where the + basename equals the file's `key`. +- **The link lives on the validator.** Each validator lists every evidence set + it applies to under `evidence_sets` (by `reference_id`). A validator shared + across fetchers is a **single file with multiple `evidence_sets` entries** — + we never copy a validator. This mirrors how Paramify itself models a + validator (it owns its evidence-set list). +- **A fetcher's validators = reverse lookup:** every registry file whose + `evidence_sets` includes that fetcher's `evidence_set.reference_id`. +- **Two roles** (`role:`): each evidence set has exactly one `completeness` + validator (full-population check) plus any number of `configuration` checks. + +## Authoring + +Copy [`_template/validator.yaml`](_template/validator.yaml) to +`validators//.yaml` and fill it in. `_`-prefixed directories +(like `_template`) are not part of the registry. + +- **Shape / field reference:** [`framework/schemas/validator_schema.json`](../framework/schemas/validator_schema.json) +- **Design & rationale:** [`docs/validators_design.md`](../docs/validators_design.md) diff --git a/validators/_template/validator.yaml b/validators/_template/validator.yaml new file mode 100644 index 0000000..64e48bd --- /dev/null +++ b/validators/_template/validator.yaml @@ -0,0 +1,40 @@ +# Template validator. Copy this file to validators//.yaml and +# fill in the placeholders. See docs/validators_design.md for the full model. +# +# Validated against framework/schemas/validator_schema.json. +# +# A validator is a first-class, DEDUPLICATED object: define it ONCE, then list +# EVERY evidence set it applies to under `evidence_sets`. A validator shared by +# several fetchers is a single file with several entries there — never a copy. + +key: _ # stable id; MUST equal this file's basename +# No Paramify UUID lives here — these files are shared TEMPLATES. The per-instance +# validator id is resolved and cached customer-side at sync time; the customer's +# tuned copy is never overwritten. See docs/validators_design.md. + +name: # e.g. ALB Encryption In Transit +type: AUTOMATED # AUTOMATED (regex-driven) | ATTESTATION (reviewer Q&A) +# role: configuration # optional repo-side tag: completeness | configuration +statement: + +# --- AUTOMATED validators: regex over the evidence envelope + structured logic --- +# (omit this block for ATTESTATION validators) +regex: '' +rules_summary: +validation_rules: + - regexOperation: { type: MATCH_GROUP, groupNumber: 1 } # or { type: MATCH_COUNT } + criteria: EQUALS # CONTAINS | EQUALS | GREATER_THAN | ... (12 operators) + value: { type: MATCH_GROUP, groupNumber: 2 } # or { type: CUSTOM_TEXT, customText: "..." } / { type: MATCH_COUNT } + +# --- ATTESTATION validators: reviewer yes/no questions (use INSTEAD of regex/validation_rules) --- +# attestation_rules: +# - question: +# yesDisposition: PASS # PASS | FAIL | BASED_ON_NESTED +# noDisposition: FAIL +# nestedRules: [] # follow-on rules when a disposition is BASED_ON_NESTED + +# Evidence sets this validator applies to, by reference_id (a fetcher's +# evidence_set.reference_id). Add one line per set that reuses this validator — +# that list IS the dedup mechanism. +evidence_sets: + - diff --git a/validators/aws/alb_encryption_in_transit.yaml b/validators/aws/alb_encryption_in_transit.yaml new file mode 100644 index 0000000..8f41fa3 --- /dev/null +++ b/validators/aws/alb_encryption_in_transit.yaml @@ -0,0 +1,19 @@ +key: alb_encryption_in_transit +name: ALB Encryption In Transit +type: AUTOMATED +role: configuration +statement: Ensures that all application load balancers are encrypting data in transit. + +regex: '"alb_total":\s*(\d+)[\s\S]*?"alb_encrypted":\s*(\d+)' +rules_summary: MATCH_GROUP[1] EQUALS MATCH_GROUP[2] +validation_rules: + - regexOperation: { type: MATCH_GROUP, groupNumber: 1 } + criteria: EQUALS + value: { type: MATCH_GROUP, groupNumber: 2 } + +# Applies to the automated Load Balancer Encryption Status set (the +# aws_load_balancer_encryption_status fetcher). In Paramify this validator is +# also attached to the "Non-automated Load Balancer Encryption Status" sibling +# set; add its reference_id here once that manual set exists. +evidence_sets: + - EVD-LB-ENC-STATUS