fix(policy): image-verification webhook timeouts (context deadline exceeded) can wedge the merge queue

[devantler]

🤖 Generated by the Daily AI Assistant

Problem

The verify-app-images Kyverno ImageValidatingPolicy webhook (ivpol.{mutate,validate}.kyverno.svc-fail, timeout=10s, fail-closed) intermittently returns context deadline exceeded while doing keyless cosign verification (which reaches Fulcio/Rekor/ghcr at admission time). When that happens during a deploy-prod merge_group run, legitimate first-party app pods are denied inside the 20-minute apps health-check window → the apps Kustomization times out → the whole merge_group is evicted.

Observed live on the wedding-app rollout: RS wedding-app-684f95779 logged FailedCreate … ivpol.mutate.kyverno.svc-fail … context deadline exceeded ×19 over ~4h, interleaved with clean policy responses. This transient timeout is independent of policy correctness — even a correctly-trusted image can be denied when verification is slow, and in the merge queue that stalls every platform PR, not just the app being rolled out.

Why it matters

A network/latency blip in sigstore or the registry shouldn't be able to wedge the entire platform merge queue or stall an app rollout. Today it can, because image verification is a fail-closed 10s webhook on the hot admission path.

Proposed direction (investigate, pick one)

• Raise the image-verification webhook timeout (10s → ~30s) so cold cosign verifications complete.
• Enable Kyverno's cosign verification-result caching (and/or warm the TUF/Rekor cache) so repeat admissions don't re-hit the network.
• Consider audit-with-retry for first-party images so a transient verify failure doesn't hard-deny a legitimate, already-published image.

Rough size

S–M: a Kyverno HelmRelease values / policy webhook-config change plus a short investigation of which mitigation is safe (must not weaken the security posture the policy exists for).

[devantler]

🤖 Generated by the Daily AI Assistant

Corrected root cause (verified live on prod) — --webhookTimeout does NOT fix this.

I tried the obvious global lever (admissionController.container.extraArgs.webhookTimeout: 30) in #2432 and deployed it. It applied to the cluster, but checking the resulting webhook configs proves it doesn't touch the image-verification path:

webhook	timeoutSeconds after --webhookTimeout=30
validate.kyverno.svc-fail (legacy ClusterPolicy engine)	30 ✅
ivpol.{validate,mutate}.kyverno.svc-fail (ImageValidatingPolicy engine)	10 ❌ (unchanged)

So the flag only raises the legacy policy webhooks. The new policies.kyverno.io IVPOL webhooks — which run our cosign image verification — keep the hardcoded 10s, and wedding-app stayed blocked with …/verify-app-images/verify-ksail-images?timeout=10s: context deadline exceeded. #2432 is reverted to draft.

The IVPOL webhook timeout has only two real levers in Kyverno v1.18.1, both bigger than a values tweak — this needs a call on which way to go:

1. Collapse the two IVPOLs into one and set spec.webhookConfiguration.timeoutSeconds: 30 on it. The finegrained-webhook trap documented in verify-ksail-images.yaml only breaks with multiple IVPOLs (concatenated URL path → "policy not evaluated"), so a single policy should register a clean finegrained webhook. Trade-off: merges two security-critical policies (app + ksail identities) into one CEL set; the "single IVPOL registers cleanly" assumption must be verified on the cluster before trusting it (getting it wrong fail-closes all Pod admission).
2. Bump Kyverno to a release that fixes the multi-IVPOL finegrained webhook bug, then set per-policy webhookConfiguration on both existing IVPOLs. Trade-off: a controller version bump on the prod admission path — needs its own validation.

Both are prod-admission-critical changes I don't want to ship blind from an unattended run. @devantler — which direction do you prefer? I can prepare and validate whichever you pick. Meanwhile the old wedding-app pod is still serving (no user-facing outage); the stuck items are the v1.14.1 rollout and the merge queue (every deploy-prod health-checks wedding-app and evicts).

[devantler]

🤖 Generated by the Daily AI Assistant

Root cause of the CURRENT wedge corrected again — it is the #2429 dual-identity attestor, not the webhook timeout. Live kyverno admission logs show the wedding-app verify completes in ~3s and fails with unsupported: multiple identities are not supported at this time — cosign v3's bundle path (sigstore-go) rejects any attestor with >1 identity entry, so #2429's transition attestor fail-closes every image it gates. Fix: one identity with an alternated subjectRegExp → #2456 (verified: cosign verify of the blocked image against the alternated regexp exits 0). Same bug shape as #2399→#2404 at the Flux OCI layer.

The intermittent context deadline exceeded on ivpol.mutate this issue tracks is real but secondary (self-heals via controller retries; last seen under deploy load). Once #2456 lands, this issue drops from "queue-wedging" to "hardening": the two levers from my earlier comment (merge the IVPOLs + per-policy webhookConfiguration, or a Kyverno bump once a release ships a finegrained fix) remain the options, at normal priority. Note: Kyverno v1.18.1 is still the latest release, so the bump lever isn't available yet.