feat: opt-in dizzy VictoriaMetrics + Grafana stack and chaos targets for Keystone and Glance

[berendt]

Summary

Adds an opt-in WITH_DIZZY observability stack to the kind dev stack and two
chaos make targets that run dizzy's
create/mutate/delete churn soak against a quick-start ControlPlane. dizzy exports
per-operation metrics over OTLP, which a VictoriaMetrics-single server ingests
natively at /opentelemetry/v1/metrics (no collector), fronted by an
anonymous-Viewer Grafana pre-provisioned with dizzy's three dashboards.

The stack is default-off: with WITH_DIZZY unset the overlay never renders, no
HTTPRoute attaches, and the new kind port mapping and Gateway listener sit idle
but ready, so the flag can be enabled later on any cluster created after this
change without recreating it.

Closes #717

Change set (in commit order)

The branch builds the stack bottom-up: host plumbing first, then the runner, the
overlay, the deploy-script gate, the make targets, and finally the Renovate,
test, and docs wiring.

1. e2557c9c feat(kind): bridge host 8428 to VictoriaMetrics NodePort 30428
   Adds an unconditional extraPortMapping forwarding host 127.0.0.1:8428 →
   container NodePort 30428 — the host→cluster path dizzy (running on the
   developer host) uses to export OTLP metrics. Loopback-only; left in place
   regardless of the opt-in flag so the flag can be turned on later without
   recreating the cluster.

2. 5009774d feat(kind): add https-dizzy Gateway listener and TLS certificate
   Adds an unconditional fourth HTTPS listener (https-dizzy) on the shared
   port 443, routed by SNI to dizzy.127-0-0-1.nip.io. This is the repo's first
   listener that admits routes from another namespace (keyed by the automatic
   kubernetes.io/metadata.name label), so the gated overlay's HTTPRoute
   attaches from the dizzy namespace without a ReferenceGrant. With
   WITH_DIZZY unset no route attaches and the SNI answers 404, mirroring how
   the glance listener exists without a Glance service. Ships a per-hostname
   self-signed Certificate matching the keystone/horizon/glance siblings.

3. 6cdc2558 feat(hack): add dizzy.sh with dashboard staging and chaos runner
   New hack/dizzy.sh wrapping the dizzy tester. DIZZY_VERSION is the single
   pinned string (v0.1.0). stage-dashboards copies dizzy's three Grafana
   dashboards into the git-ignored deploy/kind/dizzy/dashboards/ for the
   overlay's configMapGenerator. chaos <keystone|glance> installs the binary
   via a buildinfo-guarded go install, generates a mode-600 clouds.yaml from
   the ControlPlane admin Secret (missing-Secret and empty-password aborts kept
   distinct), resolves the Keystone auth URL from a docker port probe with a
   :443 fallback, warns without aborting when OTLP ingest looks down, pins the
   OTLP export interval to 15s, and propagates dizzy's own exit code. Both
   subcommands share an atomically-renamed tarball cache under _output/dizzy/.

4. ee1b0331 feat(kind): add opt-in dizzy VictoriaMetrics + Grafana overlay
   The self-contained deploy/kind/dizzy/ overlay
   (LoadRestrictionsRootOnly, no parent-dir refs). Declares two
   HelmRepositories into flux-system from within the overlay, leaving
   deploy/flux-system/** untouched. VictoriaMetrics-single ingests OTLP with
   Prometheus-naming + promote-all-resource-attributes extraArgs, on NodePort
   30428, emptyDir storage, 30d retention (kind throwaway posture).
   Anonymous-Viewer Grafana with the victoriametrics datasource and a
   dashboards ConfigMap. The repo's first static HTTPRoute attaches Grafana to
   the https-dizzy listener at https://dizzy.127-0-0-1.nip.io. Chart values
   were verified with helm template/show values against
   victoria-metrics-single 0.43.0 and grafana 10.5.15; the only correction was
   moving fullnameOverride to the top level so the server Service renders as
   dizzy-victoria-metrics-server, matching the Grafana datasource URL. Chart
   versions use floating ranges, matching the chaos-mesh/prometheus overlay
   precedent.

5. 9f583403 feat(hack): gate the dizzy stack behind WITH_DIZZY in deploy-infra
   Adds the WITH_DIZZY flag (default false), cloned from the
   WITH_PROMETHEUS pattern: doc comment, startup-banner line, a gated Step-3
   block that runs hack/dizzy.sh stage-dashboards immediately before
   kubectl apply -k deploy/kind/dizzy, a docker port probe warning when the
   cluster predates the 8428→30428 mapping (deploy continues), a Phase 3 append
   of dizzy-victoria-metrics + dizzy-grafana to the HelmRelease wait list,
   and a post-wait log printing the Grafana URL (honoring KIND_HOST_PORT).

6. 4edd9c3c feat(make): add dizzy-keystone and dizzy-glance chaos targets
   Two thin wrappers over hack/dizzy.sh chaos <service>. Each runs three
   separated preflights — kubectl reachability, dizzy stack presence, and the
   ControlPlane admin Secret — so failure modes stay distinguishable. The
   version pin stays in hack/dizzy.sh only, so Renovate has exactly one string
   to bump.

7. 88db7eef chore(renovate): track the DIZZY_VERSION pin in hack/dizzy.sh
   A regex customManager over the single DIZZY_VERSION line, resolving from
   the github-releases datasource for B42Labs/dizzy. Major bumps disabled;
   minor/patch automerge after the standard 3-day cooldown. A regression test
   perl-evaluates the matchString regex against the real script so any drift
   fails fast.

8. b3f88678 test: pin WITH_DIZZY gating, overlay render, and default-off posture
   Three tests. The flag test locks the three strict WITH_DIZZY=="true" gates,
   the banner, the staged-apply ordering, the Phase 3 append, and byte-for-byte
   survival of the 8428→30428 mapping under a KIND_HOST_PORT override. The
   overlay test pins the SPDX headers on all seven tracked files, the exact
   rendered resource set, and the fail-when-unstaged staging contract (staging
   developer dashboards aside and restoring via an EXIT trap). The chainsaw
   suite asserts the dizzy Namespace, sources, and releases are absent on a
   default-flags cluster, mirroring no-metrics-server-when-disabled.

9. 06253d11 docs: document the WITH_DIZZY stack and dizzy chaos targets
   Extended quick-start gets an opt-in tip and two architecture-snapshot pod
   rows; the infrastructure-manifests reference documents the overlay (seven
   tracked files, the dashboard staging contract, the always-present listener +
   Certificate vs. the gated HTTPRoute, the 8428→30428 mapping); a new
   dizzy-chaos-testing reference page covers the make runners, environment
   overrides, dashboards, exported metric families, and two troubleshooting
   warnings, wired into the Testing sidebar.

Alignment with the issue

The three decisions settled during elaboration are all honored: host→VictoriaMetrics
OTLP ingest goes through a kind extraPortMapping (not a port-forward or gateway
route); VictoriaMetrics and Grafana install as Flux HelmReleases from the official
charts; dashboards are fetched from the pinned dizzy release at deploy/run time and
staged into a git-ignored directory (not vendored). No divergences from the issue
were observed in the commits.

Verification

Automated coverage lands in commit 8: the deploy-flag test, the overlay-render
test, and the no-dizzy-when-disabled chainsaw suite, plus the Renovate
customManager regression test in commit 7. The overlay chart values were verified
with helm template/show values (noted in commit 4).

---

Implemented by planwerk-agent c8ef501 with Claude:claude-opus-4-8