The c5c3-operator's elected leader crash-loops on any cluster that lacks a CRD its ControlPlane controller watches — while the pod reports Ready

[berendt]

Observed behavior

Install the c5c3-operator (chart + its own CRDs) on a cluster that does not carry the Keystone, Horizon, Glance, GlanceBackend, KeystoneIdentityBackend, or K-ORC CRDs. The deployment rolls out, both replicas go Ready 1/1, the admission webhooks serve — and the elected leader dies every two minutes:

{"logger":"controller-runtime.source.Kind","msg":"if kind is a CRD, it should be installed before calling Start",
 "kind":"Keystone.keystone.openstack.c5c3.io","error":"no matches for kind \"Keystone\" in version \"keystone.openstack.c5c3.io/v1alpha1\""}
...
failed to wait for controlplane caches to sync kind source: *v1alpha1.Keystone:
timed out waiting for cache to be synced for kind source: *v1alpha1.Keystone

ControlPlaneReconciler.SetupWithManager (operators/c5c3/internal/controller/controlplane_controller.go) Owns() the Keystone, Horizon, Glance, GlanceBackend, and eight K-ORC kinds and Watches() KeystoneIdentityBackend. controller-runtime informers for a kind whose CRD is absent never sync; the controller start fails on the 2-minute CacheSyncTimeout default, the error tears down the whole manager, and the process exits. The standby replica is never elected leader before the crashed one restarts and re-acquires the lease, so the pattern is: leader restarts every ~2 minutes, standby stays at 0 restarts, ControlPlane CRs are admitted (webhook) but never reconciled — no finalizer, no status, no children, no events.

Why it matters

• The failure is silent. Readiness/liveness probes pass (the probe endpoints and the webhook server come up before controller start), so nothing pages. The only externally visible signals are the leader's restart counter and a ControlPlane CR that never gets a status.
• The install-order dependency is undocumented. A platform team installing the c5c3-operator before the keystone-/horizon-/glance-operators and K-ORC (a natural order for the orchestrator) lands exactly here.
• CI hit it latently. The e2e-operator (c5c3) matrix leg ran the operator in this state on every PR; it went unnoticed because every suite on that leg was webhook-only or skipped, until the dedicated-backing-services suite (feat(c5c3): support opt-in dedicated backing services per ControlPlane service #647) became the first to need the reconciler. That PR gives the leg the watched CRDs, but the operator behavior itself is unchanged.
• It already recurred, for a sibling operator this issue did not originally name. fix(deploy): unsuspend glance-operator on the flux ControlPlane path #690 (merged 2026-07-19) documents the identical failure for the Glance CRDs: the Flux WITH_CONTROLPLANE=true quickstart path left glance-operator suspended, so the Glance CRDs never installed, the c5c3-operator manager crash-looped on no matches for kind ... GlanceBackend, and the applied ControlPlane CR got no status at all. The fix landed only in hack/deploy-infra.sh's unsuspend list — the c5c3-operator itself is unchanged, so the next install-order gap reproduces the same outage.

Reproduction

envtest with only the c5c3 + MariaDB + Memcached + external-secrets + cert-manager CRDs (the deploy-infra baseline), real SetupWithManager, CacheSyncTimeout shortened:

startErr := mgr.Start(ctx)
// startErr: failed to wait for controlplane caches to sync kind source: *v1alpha1.Keystone:
//           timed out waiting for cache to be synced for kind source: *v1alpha1.Keystone

Or on kind: hack/deploy-infra.sh, then OPERATOR=c5c3 IMAGE_REPO=... hack/ci-deploy-operator.sh without the sibling CRDs, and watch the leader's logs for two minutes.

Implementation direction

Conditional watch registration. SetupWithManager probes the manager's REST mapper at startup and registers the Keystone/Horizon/Glance/GlanceBackend/KeystoneIdentityBackend/K-ORC watches only for the CRDs that are actually installed; a missing CRD is skipped rather than crashing the manager. A leader-election-gated background probe re-checks the CRDs that were missing at startup on an interval; the moment one of them appears, the probe returns an error from its Runnable.Start, mgr.Start propagates it as fatal, and the process exits. The Deployment's restart policy brings up a fresh process, which re-evaluates discovery from scratch and registers the now-available watch normally — no re-registration logic inside a running manager, which controller-runtime does not support. This removes the install-order dependency entirely instead of only documenting it.

The other two directions this issue previously listed were considered and are not being built:

• Preflight check + documented prerequisite — failing fast with a "missing required CRDs" error turns the silent failure into a loud one, but a platform team would still have to install the sibling operators in the right order; it does not remove the dependency.
• Ship the watched CRDs with the chart — conflicts with the sibling operators owning their own CRD lifecycle.

---

Clarified by planwerk-agent with Claude:claude-sonnet-5