Meta: consolidated test-coverage roadmap — upgrades, e2e depth, chaos, unit parity, observability

[berendt]

Background

This meta consolidates every open test-coverage issue into one phased roadmap and extends it with the gaps a fresh repo audit (2026-07-11, main @ 6b526377) surfaced across code, upgrades and real-world usage.

Consolidates (closed in favour of this meta — full scenario specs remain readable in the closed issues):

• Meta: Expand E2E test coverage for Keystone operator #277 — Meta: Expand E2E test coverage for Keystone operator. Phase 1 largely landed (test(e2e): Expand webhook/CEL rejection matrix for Keystone CR (#277 Phase 1) #281, test(e2e): Assert semantic invariants on Keystone CR status (#277 Phase 1) #314, test(e2e): Reconcile Keystone CR in PSS-restricted namespace (#277 Phase 1) #317 closed); the open remainder is carried into Phases 1, 5 and 8 here.
• test(e2e): Cover failure & recovery paths for external dependencies (#277 Phase 1) #315 — Failure & recovery paths for external dependencies → Phase 1.
• test(e2e): Cover operator metrics surface and steady-state event budget (#277 Phase 1) #316 — Operator metrics surface + steady-state event budget → Phase 1.
• test(e2e): advanced scenarios and stress tests (upgrade, recovery, rotation, leader election) #369 — Advanced scenarios and stress tests → deduplicated into Phases 2 and 5 (image upgrade, DB failure recovery and leader-election failover have since landed as image-upgrade, mariadb-pod-kill, operator-pod-kill).
• Meta: Expand chaos E2E test coverage (Chaos Mesh) #453 — Meta: Expand chaos E2E test coverage → Phase 6 (scenario specs and conventions stay authoritative in Meta: Expand chaos E2E test coverage (Chaos Mesh) #453).
• Meta: ControlPlane → Keystone chain chaos coverage (K-ORC / c5c3) #550 — Meta: ControlPlane → Keystone chain chaos coverage → Phase 6.
• Rename e2e suite full-controlplane-keystone → full-controlplane (now a multi-service suite) #562 — Rename e2e suite full-controlplane-keystone → full-controlplane → Phase 1 (collision/ambiguity analysis stays in Rename e2e suite full-controlplane-keystone → full-controlplane (now a multi-service suite) #562).

Related, deliberately NOT consolidated: #420 (bug — root cause stays open there; only its untested acceptance criterion lives here as Phase 2 items), #582/#589/#590/#591 (brownfield meta owns its e2e suite), #610/#614 (guides meta owns guide↔suite coupling).

Already fixed — do not re-file: the 2026-06 pre-production audit (#480/#481) landed the keystone operator helm-upgrade-in-place suite, keystone/c5c3 watch-mapper unit tests, chaos pod-leg CI gating (continue-on-error: false), the deletion-stuck-finalizer chaos test and keystone metrics-collector unit tests.

Current baseline (audited): 85 chainsaw e2e tests (49 keystone, 10 horizon, 6 c5c3, 4 infrastructure, 1 keystone-operator, plus SSO + operator-upgrade suites), 13 chaos tests, 127 Go test files across 4 modules (keystone controller 87.2 %, horizon controller 81.6 %, c5c3 controller 82.0 % unit coverage), tempest for 2025.2/2026.1, helm-unittest at parity for all three charts, zero fuzz tests.

Phase 1 — Inherited quick wins & housekeeping (no new infra)

Failure/recovery e2e (specs in #315):

• openbao-sealed — seal all OpenBao replicas → SecretsReady=False/SecretStoreNotReady → unseal → recovery (test(e2e): Cover failure & recovery paths for external dependencies (#277 Phase 1) #315)
• openbao-policy-revoked — revoked keystone-write policy → PushSecret 403 surfaced → restore → recovery (test(e2e): Cover failure & recovery paths for external dependencies (#277 Phase 1) #315)
• eso-down — ESO scaled to 0 → fresh CR must not flip Ready spuriously → scale up → Ready (test(e2e): Cover failure & recovery paths for external dependencies (#277 Phase 1) #315)
• db-expand-failure — failing db-expand Job → DatabaseReady=False/ExpandFailed → revert → recovery (test(e2e): Cover failure & recovery paths for external dependencies (#277 Phase 1) #315)

Observability basics (specs in #316):

• operator-metrics-endpoint — scrape /metrics via curl pod, assert all five metric families with samples (test(e2e): Cover operator metrics surface and steady-state event budget (#277 Phase 1) #316)
• event-budget — ≤ N events per CR over a 5-min steady-state window (test(e2e): Cover operator metrics surface and steady-state event budget (#277 Phase 1) #316)

Housekeeping:

• Rename tests/e2e/c5c3/full-controlplane-keystone/ → full-controlplane/ incl. CR name, OpenBao seed paths, CI/docs references — mind the controlplane name collision and the bundled-ControlPlane grep ambiguity documented in Rename e2e suite full-controlplane-keystone → full-controlplane (now a multi-service suite) #562 (Rename e2e suite full-controlplane-keystone → full-controlplane (now a multi-service suite) #562)
• Fix the make test-shell glob: tests/unit/Makefile/test_e2e_chaos_preflight.sh is outside the {hack,deploy,renovate,docs} directory list and never runs (new)
• Wire or retire tests/ci/verify_*.sh (8 pipeline meta-tests referenced by no workflow or make target) (new)
• Reconcile SC-CHAOS ID drift between chaos test headers and architecture/docs/09-implementation/10-chaos-e2e-testing.md before assigning new IDs (Meta: Expand chaos E2E test coverage (Chaos Mesh) #453)

Phase 2 — Upgrade correctness & data integrity (highest risk)

• Post-upgrade admin auth: release-upgrade and upgrade-flow only do unauthenticated GET /v3; add an authenticated POST /v3/auth/tokens (admin) after the 2025.2 → 2026.1 hop — the exact blind spot behind Cross-release upgrade: keystone-manage bootstrap can't find migrated admin user (get_user_by_name miss); post-upgrade admin auth unverified #420 (Cross-release upgrade: keystone-manage bootstrap can't find migrated admin user (get_user_by_name miss); post-upgrade admin auth unverified #420 acceptance criterion, Meta: Expand E2E test coverage for Keystone operator #277-P4)
• Token continuity: issue a fernet token on 2025.2, assert it validates on 2026.1; assert the fernet-keys Secret survives the upgrade unchanged (Meta: Expand E2E test coverage for Keystone operator #277-P4)
• ControlPlane-driven upgrade e2e: patch openStackRelease 2025.2 → 2026.1 on a live ControlPlane; children upgrade in order, Ready maintained (full-controlplane and multi-controlplane currently pin 2025.2 forever) (test(e2e): advanced scenarios and stress tests (upgrade, recovery, rotation, leader election) #369, new)
• Downgrade rejection e2e: (a) keystone tag revert after a completed upgrade → DowngradeNotSupported (distinct from upgrade-abort, which relies on mid-flight revert being accepted); (b) c5c3 invalid-cr transition fixture for the webhook rejection (feat(api): enforce field immutability via oldSelf CEL rules; reject openStackRelease downgrades #466); (c) decide whether keystone's ValidateUpdate should use oldObj for admission-time rejection instead of condition-only (new)
• Operator-upgrade job hardening: run e2e-operator-upgrade on pushes to main too (currently PR-only); add horizon-operator and c5c3-operator helm-upgrade-in-place suites; pin the baseline to a defined released version instead of "highest published chart + :latest" (new)
• CRD/controller skew window: assert existing CRs stay servable between kubectl apply -f crds/ and the operator upgrade completing (new)
• Upgrade-fixture lockstep: the 2025.2/2026.1 tags are hard-coded in 4+ fixture files; add a wiring check (à la ci-generate-tempest-matrix.sh) tying releases/*/ to the upgrade-suite fixtures (new)
• Tempest on a migrated database: one tempest leg that deploys 2025.2, upgrades to 2026.1, then runs tempest — today tempest only ever sees fresh deployments (new)
• Helm values compat: render the released chart's values against the new chart to catch removed/renamed values before the live upgrade job does (new)

Phase 3 — Activate dormant coverage & prove enforcement

• Un-skip the four dormant c5c3 suites: multi-controlplane, admin-password-scoping, db-credential-scoping, deletion-orchestration always SKIP in CI (presence guards; only full-controlplane runs in the e2e-controlplane job) — multi-tenancy and per-CR credential scoping are currently untested at e2e level in CI. Extend the e2e-controlplane job to run them with E2E_REQUIRE_CONTROLPLANE_STACK=true (new, HIGH)
• NetworkPolicy enforcement: all netpol suites are object-shape only (kindnet ignores NetworkPolicy — self-documented caveat in network-policy-egress). Add a CNI-enabled kind leg (Calico/Cilium) with allow/deny connectivity probes for keystone, horizon and operator egress (Meta: Expand E2E test coverage for Keystone operator #277-P2, new)
• Flux devstack variant: CONTROLPLANE_OPERATORS=flux (the documented quick-start path using published charts) has zero CI coverage — every job pins external + dev images; cover it (scheduled job is fine) and add the missing docs unit test for docs/quick-start-controlplane.md (new)

Phase 4 — Unit & integration parity (code quality)

Horizon (systematic parity gap vs keystone):

• horizon_watches_test.go — both watch mappers at 0 % (keystone/c5c3 mappers were covered by test: pre-production gaps — watch-mapper coverage, operator helm-upgrade e2e, chaos CI gating, stuck-finalizer deletion, metrics lifecycle #480; horizon was missed) (new)
• setupwithmanager_integration_test.go for horizon (SetupWithManager 0 %; the existing integration test re-implements the setup by hand) (new)
• Horizon controller error paths: markConfigFailed, reconcileParallelGroup, registerSecretNameIndex all 0 %; Reconcile 52.9 % vs keystone's 88.4 %; ensureNetworkPolicy apply path 0 % (new)

c5c3:

• Service-account ESO publish path at 0 % (publishServiceAccount, serviceAccountPushSecret, ensureServiceAccountExternalSecret, serviceAccountRemoteKeyFor in reconcile_serviceaccounts.go) — security-relevant credential-distribution flow (new)
• CredentialRotation controller missing from setupwithmanager_test.go; orphan-deletion error paths (deleteOrphanedKeystone/Horizon 58.3 %); Reconcile unit coverage 43.8 % (new)

Keystone:

• KeystoneIdentityBackend reconciler error paths (applyRoleAssignment 26.7 %, ensureProtocol 56 %, teardownFederationObjects 52 %) via the existing mockable identity client — these set MappingsReady (new)
• Federation last-known-good fallbacks (lastKnownGoodSAMLMetadata 50 %, ensureSAMLSPMetadataSecret 53 %, lastKnownGoodProviderMetadata 67 %) — the code that runs exactly when the IdP is down (new)
• keystoneidentitybackend_types_test.go for the pure helpers (FederationRemoteID, EffectiveProtocolID at 0 %); mapGophercloudError table test (42.9 %) (new)

internal/common:

• gateway.EnsureHTTPRoute 0 % + gateway is the only apply-path pkg without an integration_test.go; job.DeleteCronJob 0 %; watch.RegisterSecretNameIndex 0 %; 66.7 % error branches in database/deployment/tls (new)

Cross-cutting:

• CEL envtest parity: only keystone has apiserver-level CEL tests; port the pattern to horizon (8 XValidation markers), c5c3 (14) and internal/common/types (17) for seconds-fast feedback instead of chainsaw-only coverage (new)
• First fuzz targets (repo has zero): federation_mapping_convert JSON conversion, pysettings Python-literal rendering, config INI rendering, release.ParseRelease, webhook string validators (new)
• Convention nits: same-named _test.go for reconcile_upgrade.go (keystone) and reconcile_catalog.go (c5c3) — well covered indirectly, but they trip the sub-reconciler convention audit (new)

Phase 5 — E2E depth on existing features (from #277 Phase 2 remainder + #369)

Verified still open — the existing suites test object shape, not effect:

• Rotation beyond maxActiveKeys prunes the oldest key; tokens signed with the pruned key become 401 (Meta: Expand E2E test coverage for Keystone operator #277-P2)
• Rotation during live auth traffic → zero failed requests (Meta: Expand E2E test coverage for Keystone operator #277-P2)
• Staging secret deleted mid-rotation → operator regenerates, production Secret untouched; concurrencyPolicy serialises manual + CronJob triggers; staging→production promotion rejects invalid base64 (Meta: Expand E2E test coverage for Keystone operator #277-P2)
• Effective topology spread on multi-node kind (current topology-spread suite only asserts spec pass-through) (Meta: Expand E2E test coverage for Keystone operator #277-P2)
• Priority-class preemption on multi-node kind (current suite only asserts field pass-through) (Meta: Expand E2E test coverage for Keystone operator #277-P2)
• Load-driven autoscaling: generator against /v3 → HPA scales up and back down (current suite only asserts HPA object shape) (Meta: Expand E2E test coverage for Keystone operator #277-P2)
• Plugins happy path: registered plugin actually loaded by Keystone at startup (Meta: Expand E2E test coverage for Keystone operator #277-P2)
• Leader-election lease durability: delete the lease → re-election, CR stays Ready (Meta: Expand E2E test coverage for Keystone operator #277-P2)
• Stress: ≥ 10 rapid spec changes converge without leaked owned resources, 3 consecutive flake-free runs (test(e2e): advanced scenarios and stress tests (upgrade, recovery, rotation, leader election) #369)
• ControlPlane config propagation: CP spec change → child CR → ConfigMap → Deployment restart, asserted end-to-end (test(e2e): advanced scenarios and stress tests (upgrade, recovery, rotation, leader election) #369)
• ImagePullSecret flow: private-registry image + pullSecretRef propagated to the Deployment (Meta: Expand E2E test coverage for Keystone operator #277-P4)
• Multi-namespace keystone: 3 CRs in 3 namespaces reconcile in parallel without cross-namespace leaks (concurrent-cr-conflicts covers same-namespace only) (Meta: Expand E2E test coverage for Keystone operator #277-P2, low)

Dropped as since-covered: zero-downtime upgrade under traffic (rolling-update-zero-downtime), Envoy Gateway/HTTPRoute (httproute, gateway-quick-start), SAML/OIDC federation happy paths, brownfield database/cache modes, WATCH_NAMESPACE scoping (namespace-scoped-rbac), PSS-restricted namespace (#317), webhook/CEL rejection matrix (#281), semantic invariants (#314), extraConfig rendering (config-pruning et al.).

Phase 6 — Chaos expansion (scenario specs live in #453 / #550)

• Meta: Expand chaos E2E test coverage (Chaos Mesh) #453 Phase A — catalog completion: cert-manager-pod-kill, memcached-network-partition, multi-dependency-failure, eso-pod-kill, credential-rotation-failure (all five still unimplemented; the 4 suites added since Meta: Expand chaos E2E test coverage (Chaos Mesh) #453 was filed — deletion-stuck-finalizer, horizon-*-pod-kill, keystone-federation — cover none of them)
• Meta: Expand chaos E2E test coverage (Chaos Mesh) #453 Phase B — lifecycle-phase chaos: operator-kill-during-rotation, mariadb-kill-during-provisioning, openbao-network-partition, mariadb-operator-cascading-failure
• Meta: Expand chaos E2E test coverage (Chaos Mesh) #453 Phase C — new fault classes: HTTPChaos on keystone-api, TimeChaos on api + operator pods, IOChaos on the OpenBao PV
• Meta: Expand chaos E2E test coverage (Chaos Mesh) #453 Phase D — infra-prerequisite: DNSChaos (needs Chaos Mesh DNS server), StressChaos/HPA, flux-pod-kill (implement or close with rationale)
• Meta: ControlPlane → Keystone chain chaos coverage (K-ORC / c5c3) #550 — ControlPlane chain: korc-kill-during-controlplane-provisioning, controlplane-status-during-keystone-degradation, korc-network-partition
• c5c3-operator pod-kill mid-ControlPlane-reconcile — zero chaos suites target the umbrella operator today (new)
• Housekeeping: runtime-budget decision (third matrix leg vs. nightly) before Phase C/D; update both scenario-matrix docs per new test. Note: since test: pre-production gaps — watch-mapper coverage, operator helm-upgrade e2e, chaos CI gating, stuck-finalizer deletion, metrics lifecycle #480 the pod leg gates CI (continue-on-error: false); only the network leg stays advisory — Meta: Expand chaos E2E test coverage (Chaos Mesh) #453's "chaos never gates" convention is obsolete

Conventions inherited verbatim from #453: one scenario per sub-issue, <target>-<fault-type> naming, server-side partitions, AllInjected gates, pod-UID kill verification, bounded durations, shared diagnostics.sh, explicit test_dirs in ci.yaml.

Phase 7 — Service parity & observability

Horizon (canonical 8-suite floor is met; day-2 usage trails keystone):

• Horizon e2e for CRD-supported features: autoscaling, logging, missing-secret, config-pruning analogue, image-upgrade/release-upgrade, full gateway-quick-start (smoke-only today) (new)
• tests/e2e/horizon-operator/network-policy-egress counterpart (the guide enable-horizon-operator-networkpolicy.md exists; the suite does not) (new)
• Horizon in the e2e-prometheus job: live ServiceMonitor target-Up + Grafana dashboard assertions (keystone-only today), backing enable-horizon-operator-metrics.md (new)
• Mechanical parity: missing releases/*/test-excludes/horizon.txt; cleanup-images.yaml never prunes horizon-operator packages (new)
• Horizon emits no Kubernetes Events (no EventRecorder) — feature first, then the events e2e analogue (new, product gap)

c5c3-operator cross-cutting:

• pod-security-restricted, healthcheck and metrics e2e for the umbrella operator (new)
• Chart NetworkPolicy template (keystone/horizon charts have one, c5c3 has none) — feature first, then shape + enforcement tests (new, product gap)
• Grafana dashboard + metric-drift test for c5c3 (keystone/horizon have both) (new, product gap)

Alerts:

• No PrometheusRule exists anywhere — alerts are a product gap first; once shipped, add alert-shaped tests (rule syntax, drift, e2e-prometheus firing check) (new, product gap)

Phase 8 — Performance & long-running gates (from #277 Phase 4)

• Commit a performance baseline (p50/p99 for token issue/validate) and wire hack/perf-reconcile-benchmark.sh (exists, deliberately not in CI) into a scheduled job that fails on > 20 % regression (Meta: Expand E2E test coverage for Keystone operator #277-P4)
• Memory soak: 1 h sustained load, operator RSS growth < 10 %, goroutine-leak check via pprof heap diff (Meta: Expand E2E test coverage for Keystone operator #277-P4)
• Secret-leak scan: /proc/<operator-pid>/environ and pod env contain no key material (Meta: Expand E2E test coverage for Keystone operator #277-P4)
• RBAC minimality audit (audit2rbac-style: granted verbs vs. actually-used verbs) — helm role tests pin the contents; minimality is unproven (Meta: Expand E2E test coverage for Keystone operator #277-P4, low)

Conventions for sub-issues

• One cohesive feature area per sub-issue; describe scenario, steps, assertions (condition type/status/reason, resource state).
• Suites live under tests/e2e/<operator>/, tests/e2e/<operator>-operator/, tests/e2e-chaos/, tests/e2e-operator-upgrade/ following the existing three-file convention; respect the chainsaw config contracts (timeouts, parallel: 4) — deviations need justification.
• Every new chaos/e2e directory must be added to the explicit test_dirs of the matching CI matrix leg (include-regex filtering is a no-op).
• DoD per sub-issue: green locally against make kind-up, two consecutive flake-free CI runs, docs matrices updated where applicable, no unreviewed new infra in deploy/kind/.

Out of scope

• Guide↔suite coupling and guide testability — Meta: make every guide reproducible on the tutorial devstacks (ControlPlane-first, kind-tested) #610/Couple every guide to the e2e suite that tests it #614.
• Brownfield/External-mode e2e — Meta: brownfield Keystone adoption — phase 1: identity-only (service-less) ControlPlane against an external Keystone #582/e2e: External-mode ControlPlane suite against a cluster-foreign Keystone #590.
• Cross-release upgrade: keystone-manage bootstrap can't find migrated admin user (get_user_by_name miss); post-upgrade admin auth unverified #420's root cause (migrated admin row vs get_user_by_name) — stays open as a bug; only the missing e2e assertions live here.
• Tempest include-list expansion beyond the per-release lists (unchanged from Meta: Expand E2E test coverage for Keystone operator #277); product features themselves (Events for Horizon, PrometheusRules, c5c3 chart NetworkPolicy) — flagged above as product gap, need feature tickets first, tests follow.