fix(kubescape): kubevuln produces no CVE data — relevancy ScanCP aborts on partial ApplicationProfiles

[devantler]

🤖 Generated by the Daily AI Assistant

Part of #2447.

Problem

kubevuln is alive and processing its queue (grype DB loads fine at startup; thousands of scan started), but no image carries any vulnerability data. Verified against live prod (2026-07-04):

• vulnerabilitymanifestsummaries: 109/109 show .all=0 and .relevant=0, with vulnerabilitiesRef.all and .relevant both empty (kind:"", name:"").
• vulnerabilitymanifests: 197/197 have zero payload.matches and no tool.name — SBOM shells with no grype match phase ever recorded.
• Dominant kubevuln log (~815×/24h): service warning - ScanCP … container profile is partial (workload restart required).

ScanCP is the scan-with-container-profile (relevancy) path — and it is the only path wired. It aborts before grype runs because the ApplicationProfile never reaches completed (123/288 profiles are still partial), so neither .all nor .relevant is ever written. Real registry errors (ScanCVE i/o timeout on postgres:latest) are rare (3×/24h) — this is not a grype-DB or registry-auth failure.

Consequence: the cluster is not genuinely 0-CVE, and #2149 ("reduce reachable CVEs flagged by Kubescape") currently has nothing to act on — nothing is being flagged.

Proposed direction (CVE = both — settled with the maintainer)

1. Get grype output flowing now — decouple/enable a full non-relevancy scan (the .all view) so grype produces CVE matches regardless of ApplicationProfile completion state. Confirm the kubevuln/node-agent config that governs the relevancy-vs-full scan path (relevancy on but not gating the .all write).
2. Complete the profiles for the reachable view — a controlled, batched rollout of the 123 partial-profile workloads so ScanCP completes and the .relevant view (the ~90%-noise-reduction reachable-CVE list) populates. Batch it; do not mass-restart prod at once.
3. Once data flows, enable VEX generation (capabilities.vexGeneration) so a downstream CI gate can suppress non-reachable CVEs (feeds Reduce reachable base-image CVEs: pin/refresh operator-managed images flagged by Kubescape #2149 and the reachable-CVE CI gate).

Rough size

M (investigation of the scan-path config is the main unknown; the profile rollout is operational and staged).

Acceptance criteria

• vulnerabilitymanifests carry real grype matches and a tool.name; summaries show non-empty .all.
• Partial ApplicationProfiles trend down; .relevant view populates for completed workloads.
• Reduce reachable base-image CVEs: pin/refresh operator-managed images flagged by Kubescape #2149 can proceed from real flagged CVEs.

[devantler]

🤖 Generated by the Daily AI Assistant

Corrected diagnosis (live re-verification 2026-07-04 ~16:45Z): the CVE pipeline is WORKING — this issue's zero-readings were an observation artifact plus a since-healed state.

1. Observation artifact: the kubescape storage APIServer returns a skeleton spec on List (fields present, zeroed/empty) — the real payload only comes back on a direct per-object GET. The "109/109 all=0" / "197/197 zero matches" readings in this issue's body were List reads. Direct GETs today: ghcr.io-devantler-tech-ksail-v7.135.0 carries 10 grype matches (first: CVE-2026-34040, DB built 2026-06-30); a 12-summary sample shows 11/12 with real counts (e.g. external-dns all=47/relevant=47, clickhouse-keeper all=567/relevant=42, velero repo-maintenance all=87/relevant=23).
2. Healed state: the cluster-wide restart from today's ksail cluster update completed many ApplicationProfiles, so ScanCP now writes both views for those workloads.

What genuinely remains: 132/314 profiles are completion=partial (kubevuln still logs ~40 ScanCP partial-aborts/hour), gating the .relevant view for those workloads. Node reboots do NOT fix these — the node-agent DaemonSet races workload startup on a rebooting node, re-creating partials. The fix is in-place workload restarts while node-agent is already up, staged in small batches (learning takes hours, so verify each batch on a later pass).

One-click pilot batch (3 safe stateless workloads whose current-RS profile is partial; my local run was policy-denied):

kubectl --context admin@prod rollout restart deployment/ksail-operator -n ksail-operator
kubectl --context admin@prod rollout restart deployment/external-dns -n external-dns
kubectl --context admin@prod rollout restart deployment/opencost -n opencost

After the pilot's profiles reach completion=complete and their .relevant views populate, subsequent runs continue batch-by-batch (worst offenders: wedding-app, velero, umami, oauth2-proxy/dex, kyverno controllers, flux controllers — stateful sets last, one at a time).

Re-scoped acceptance criteria: (1) partial-profile count trends to ~0 via the staged restarts; (2) .relevant populates for restarted workloads; (3) #2149 proceeds from the now-real flagged CVEs (it is unblocked TODAY on the .all view). VEX generation (item 3 in the original direction) stays as the follow-up once relevancy coverage is broad.

[devantler]

🤖 Generated by the Daily AI Assistant

Pilot batch EXECUTED 2026-07-05 ~05:26Z (the 07-04 policy denial did not recur): rollout restart of ksail-operator, external-dns, opencost — all three rolled out healthy. These are in-place restarts with node-agent already up, so their new ReplicaSets' ApplicationProfiles should learn to completion=complete over the next hours.

Verify on a later pass: the three profiles reach complete and their vulnerability summaries' .relevant view populates. If the pilot proves out, continue batch-by-batch per the plan above (worst offenders next: wedding-app, velero, umami, oauth2-proxy/dex, kyverno controllers, flux controllers — statefuls last, one at a time).

[devantler]

🤖 Generated by the Daily AI Assistant

Pilot verification (2026-07-05 ~08:45Z) — the chain works end to end.

• Profiles: the pilot workloads' post-restart ReplicaSet generations reach completed/complete (e.g. opencost-primary-674c7846ff, asc external-dns-564bfd6ff4); pre-restart generations stay partial as expected. Cluster-wide: 175 complete / 131 partial (the partial set includes many stale RS generations that will GC, so the live gap is smaller).
• CVE data (individual GETs — LISTs return spec-stripped skeletons, per the corrected diagnosis above): .all populates broadly, including never-restarted workloads (velero critical 6, flux-operator critical 8, longhorn-csi-plugin critical 8), and .relevant populates where profiles are complete — opencost-primary shows medium 7 relevant / 10 all, i.e. real reachability filtering, not a copy of .all.

Assessment per AC: AC1 met (real matches, non-empty .all); AC2 in progress and converging — .relevant fills as pods naturally roll, so I do not plan further mass restart batches (natural churn covers it; a targeted batch remains an option if convergence stalls); AC3 shipped as draft #2494 (capabilities.vexGeneration: enable).

Once #2494 lands and VEX documents appear, the remaining consumer work (the reachable-CVE CI gate) belongs to #2149 — this issue can then close.

[devantler]

🤖 Generated by the Daily AI Assistant

AC3 (VEX generation) verified on prod: #2494 merged 09:28Z and reconciled — kubevuln restarted with capabilities.vexGeneration: enable and the cluster now carries 328 openvulnerabilityexchangecontainers.spdx.softwarecomposition.kubescape.io CRs (OpenVEX per image). Combined with the 587th-run evidence (relevancy .all/.relevant populating broadly, profiles reaching complete on natural churn), the CVE surface is healthy end-to-end. Remaining before close: let the 131 stale-generation partials converge on natural churn — no further mass restarts planned.

[devantler]

🤖 Generated by the Daily AI Assistant

Closing — all three acceptance criteria verified on live prod (2026-07-05, ~11:30Z):

1. Real grype output flows. vulnerabilitymanifests carry real matches; summaries show non-empty .all (e.g. flux-system/deployment-flux-operator-manager: critical 8, high 3). 344 summaries + 491 manifests present.
2. Profiles converged. ApplicationProfiles: 307 completed, 23 ready, 0 partial (down from 123/288 partial on 07-04) — the pilot batches plus natural churn closed the gap; .relevant populates for completed workloads.
3. VEX generation live (feat(kubescape): enable VEX generation for reachability-aware CVE triage #2494, merged today): 328 OpenVulnerabilityExchangeContainer docs, with real per-CVE verdicts (sampled doc: 164 statements). Reduce reachable base-image CVEs: pin/refresh operator-managed images flagged by Kubescape #2149 now has real flagged CVEs and a machine-readable suppression source to act on.

Verification note for future readers: the kubescape storage API returns skeleton objects on LIST (payload stripped) — kubectl get -A -o json shows every severity as 0 and every VEX empty. Always GET individual objects to read payloads.

One small residual, tracked separately rather than holding this open: one oversized VEX store failed with HTTP 413 (ghcr.io-cloudnative-pg-postgresql-18.3-system-trixie) — noted on #2451's ingestion lane.

Follow-on work continues in #2149 (reachable-CVE CI gate) and #2451 (ingestion).