From 75fd5262dcbb8aaccc6c4d083b52796e96472291 Mon Sep 17 00:00:00 2001 From: Abhishek Date: Mon, 4 May 2026 16:18:28 +0530 Subject: [PATCH] docs: add configurable pipeline design proposal Signed-off-by: Abhishek --- proposals/configurable-pipeline.md | 176 +++++++++++++++++++++++++++++ 1 file changed, 176 insertions(+) create mode 100644 proposals/configurable-pipeline.md diff --git a/proposals/configurable-pipeline.md b/proposals/configurable-pipeline.md new file mode 100644 index 0000000000..979a88a44c --- /dev/null +++ b/proposals/configurable-pipeline.md @@ -0,0 +1,176 @@ +# Configurable Collector-Certifier-Queryer Pipeline + +**Author:** Community Contribution +**State:** Work in Progress +**Issue:** [#1093](https://github.com/guacsec/guac/issues/1093) + +--- + +## Background + +GUAC aggregates software security metadata into a high-fidelity graph through +Collectors, Certifiers, and a GraphQL Queryer. After hearing feedback from users +deploying GUAC at scale, two recurring concerns have come up. + +### Problem 1 — Operational complexity of multiple binaries + +GUAC currently follows the "do one thing well" Unix philosophy. Each collector +or certifier runs as its own separate binary or process: + +``` +guaccollect osv +guaccollect deps_dev +guaccollect scorecard +guaccollect files +guacone certifier osv +guacone certifier scorecard +guacgql +guaccsub +``` + +While this is architecturally clean, it creates real operational friction: + +- Every collector/certifier needs its own process, container, or Kubernetes pod. +- Operators have to manage lifecycle (start, stop, health checks) across many + separate services. +- Correlating logs across multiple processes when debugging is painful. +- Adding or removing a collector means changing Helm charts or Docker Compose + files, not just a config flag. + +For teams new to GUAC, this is a steep ramp just to get started. + +### Problem 2 — No unified, declarative configuration + +The `guac.yaml` file is a flat list of key-value pairs, and the flag registry in +`pkg/cli/store.go` treats all flags for every component at the same level: + +```yaml +gql-backend: keyvalue +gql-listen-port: 8080 +certifier-batch-size: 60000 +interval: 20m +add-vuln-on-ingest: false +``` + +There is no way for an operator to look at the config and understand what +components are running. There is no concept of "enable the OSV certifier" as a +configuration choice — you have to know to run a separate binary. Enabling or +disabling a component means changing how you invoke processes, not editing a +config file. + +--- + +## Proposal + +Introduce a `pipeline:` configuration block in `guac.yaml` that lets users +declare which collectors, certifiers, and server components should be active. +The idea is that a single `guacone run` (or similar) command could read this +block and start all enabled components in one process. + +A rough sketch of what this could look like: + +```yaml +pipeline: + collectors: + - type: deps_dev + enabled: true + poll: true + retrieve-dependencies: true + + - type: scorecard + enabled: true + poll: true + + - type: files + enabled: false + + certifiers: + - type: osv + enabled: true + interval: 10m + batch-size: 60000 + last-scan: 4 + + - type: clearlydefined + enabled: true + interval: 10m + + - type: scorecard + enabled: false + + server: + gql-backend: keyvalue + gql-listen-port: 8080 + rest-api-server-port: 8081 + + infrastructure: + pubsub-addr: nats://localhost:4222 + blob-addr: file:///tmp/blobstore?no_tmp_dir=true + csub-addr: localhost:2782 + use-csub: true + + ingestion: + add-vuln-on-ingest: false + add-license-on-ingest: false + add-eol-on-ingest: false + add-depsdev-on-ingest: false + + observability: + log-level: Info + enable-prometheus: false + enable-otel: false +``` + +The existing flat keys and all current CLI subcommands (`guaccollect osv`, +`guacone certifier osv`, etc.) would continue to work exactly as before — this +`pipeline:` block would be entirely opt-in. + +### How it would work (rough sketch) + +A new `guacone run` subcommand would: + +1. Read the `pipeline:` block from `guac.yaml`. +2. For each enabled collector, call `collector.RegisterDocumentCollector(...)`. +3. For each enabled certifier, call `certify.RegisterCertifier(...)`. +4. Start the GQL server if configured. +5. Run all components concurrently under a shared context with unified signal + handling for graceful shutdown. + +Override precedence (highest to lowest) would be: + +``` +CLI flags > GUAC_* env vars > pipeline: block > top-level guac.yaml defaults +``` + +### Implementation notes (things to think through) + +- Typed Go structs for `PipelineConfig`, `CollectorConfig`, `CertifierConfig` + would need to live somewhere — possibly a new `pkg/config/pipeline.go`. +- All enabled components would share the same ingestor, GQL transport, and + collectsub client. +- A component failure probably shouldn't crash the whole process — but that + behavior could be configurable. + +--- + +## Open questions + +- Should this be a new `guacone run` subcommand, a separate `guacpipeline` + binary, or something else? +- If one certifier fails, should the rest keep running? +- Should the pipeline support config hot-reload via `SIGHUP`? +- With multiple certifiers in one process, how do Prometheus metrics stay + distinguishable? +- What is the minimum viable set of collectors/certifiers for a first + implementation? + +--- + +## References + +- [GUAC how it works](https://docs.guac.sh/how-guac-works/) +- [`pkg/cli/store.go`](../pkg/cli/store.go) — current flag registry +- [`cmd/guacone/cmd/`](../cmd/guacone/cmd/) — certifier commands +- [`cmd/guaccollect/cmd/`](../cmd/guaccollect/cmd/) — collector commands +- [`guac.yaml`](../guac.yaml) — current config file +- [Original discussion (Google Doc)](https://docs.google.com/document/d/1gyoXic3-UcLj8spgbux4aNDfiEBANikKFN30NDlx-yY/edit?usp=sharing)