drain-autoscale-node-storage retrofit never fires: CREATE-only trigger starves mutate-existing UR generation

[devantler]

🤖 Generated by the Daily AI Assistant

Problem. The drain-autoscale-node-storage ClusterPolicy (the mutate-existing retrofit half split out in #2378) has never produced a single UpdateRequest. Live evidence, 2026-07-02 ~22:10Z (~28h after admission, policy Ready since 17:56Z):

• All 4 pre-existing autoscale-cx33-* Longhorn Node CRs still have spec.allowScheduling: true (the policy should set false + evictionRequested: true).
• kubectl get updaterequests -A → 0.
• 24h of kyverno-background-controller logs contain zero lines for this policy, while propagate-reloader-to-flagger-primary (also mutate-existing) logs "successfully created UR on policy update" for its targets every hour on the controller resync.

Root cause. The rule's match — which doubles as the trigger enumeration for mutateExistingOnPolicyUpdate and the hourly background resync — carries operations: [CREATE]. Existing resources enumerated in background context never satisfy a CREATE-only filter, so the retrofit selects nothing, ever. The in-file comment claiming the hourly background scan runs "regardless of triggers" is empirically wrong (see log comparison above). The filterless propagate-reloader-to-flagger-primary match is the proven working pattern.

Impact. The 2026-06-30 FinOps goal is silently unmet: Longhorn replicas stay schedulable on the 4 autoscale nodes, which keeps them CA-unremovable (cluster stuck at 10 nodes; see #2360 aftermath) — ongoing cost.

Proposed direction. Drop the operations: filter from the match (mirror propagate-reloader). Node-CR churn then re-enqueues URs, but the patch is an idempotent 2-field re-assert — same trade the reloader policy already makes cluster-wide. Rough size: XS (one manifest + comment).

[botantler-1]

🎉 This issue has been resolved in version 1.94.2 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

[devantler]

🤖 Generated by the Daily AI Assistant

Reopening — #2393's fix deployed cleanly but the retrofit STILL does not fire; diagnosis corrected

Live evidence (2026-07-02 ~19:52–21:00Z):

• fix: drop CREATE-only trigger so the drain-autoscale retrofit fires #2393 merged 19:53Z; the policy update was seen by the background-controller (policy_controller.go:259 "updating policy" name=drain-autoscale-node-storage at 19:52:50Z), policy generation 2, Ready.
• The current match block has no operations field at all — yet zero UpdateRequests were created in the following hour and the 4 autoscale-* Longhorn Node CRs still have allowScheduling: true.
• Comparison policy propagate-reloader-to-flagger-primary logs mutate.go:56 "successfully created UR on policy update" right after its own "updating policy" line — drain-autoscale never does.
• Background-controller RBAC on nodes.longhorn.io verified: list/get/watch/update/patch all yes.

Code trace (kyverno v1.18.1): the drop is silent by construction at v=2. Path: updatePolicy → syncPolicy → handleMutate (pkg/policy/mutate.go:13) → getTriggers → per-trigger handleUpdateRequest (pkg/policy/policy_controller.go:519) → ApplyBackgroundChecks:

• len(engineResponse.PolicyResponse.Rules) == 0 → return true, nil (silent skip, no log at any level — policy_controller.go:531-533);
• rule status ≠ Pass → logged only at V(4) (policy_controller.go:537).
  So the engine evaluation rejects the rule per trigger (or produces no rule response), and at our log level nothing is emitted. getTriggers' own failure modes (list error) WOULD log an error; none appears — so triggers are likely enumerated and the engine match/evaluation is what rejects.

Discriminators still open (both policies have skipBackgroundRequests: true, both mutate-existing with targets — so it is NOT that field alone): group-qualified custom kind longhorn.io/v1beta2/Node vs plain Deployment in the engine's CheckKind; the names: [autoscale-*] block; patchStrategicMerge vs the reloader's mutation type; spec-level vs rule-level mutateExistingOnPolicyUpdate.

Proposed next steps (in order):

1. Temporarily raise background-controller verbosity to v=4 (helm value) — one policy-update cycle would expose the V(4) "skip creating URs on policy update" rule.Status line and name the failing check directly. Cheap and definitive; revert after capture.
2. Alternatively/afterwards: reproduce on a local kind cluster (kyverno v1.18.1 + a namespaced test CRD + this policy) and bisect the discriminators (drop names, switch kind form, rule-level flag).
3. Upstream: whatever the cause, handleUpdateRequest's zero-rules skip deserves at least a V(2) log with the engine response — candidate kyverno issue once root-caused.

Interim unwedge still stands (maintainer one-click):

kubectl --context admin@prod patch nodes.longhorn.io -n longhorn-system <each autoscale-*> --type=merge -p '{"spec":{"allowScheduling":false,"evictionRequested":true}}'

(Exactly what the merged #2378 intends; Longhorn rebuilds replicas before removal; CA can then consolidate the 4 stranded autoscale nodes.)

[devantler]

🤖 Generated by the Daily AI Assistant

Diagnostic shipped as draft #2396: raises the background controller only to --v=4 via backgroundController.extraArgs (renders after the features-derived --v=2; klog last-wins — verified on a local chart-3.8.1 render). Once deployed, the next policy re-apply / hourly background scan will log which ApplyBackgroundChecks branch rejects drain-autoscale-node-storage, letting us bisect the 4 discriminators vs propagate-reloader. Revert follows once captured.