fix: gateway ActivityPolicy summary CEL leaks DLQ on events missing responseObject.metadata.name

[ecv]

Summary

DLQSlowLeak firing in prod (prod-infrastructure-control-plane). The
gateway.networking.k8s.io-gateway ActivityPolicy sends Gateway audit events to
the DLQ at a slow, steady rate. ~120 events / 6h and climbing (started as
51 + 69 across the two processor pods). Gateway activity timeline entries are
silently dropped for the affected event shapes.

• Alert: DLQSlowLeak (dlq-health group), severity warning
• Cluster: prod-infrastructure-control-plane, namespace activity-system
• Runbook: docs/runbooks/dlq/dlq-growth.md → docs/runbooks/dlq/policy-dlq-errors.md

Root cause

Processor logs (activity-processor, dlq_retry.go:513), repeating, retries
exhausted at retryCount: 5:

DLQ event re-failed evaluation
  eventType=audit
  policy=gateway.networking.k8s.io-gateway
  errorType=cel_summary
  retryCount=5
  err=rule 0 summary: summary template evaluation failed:
      eval "link(audit.responseObject.metadata.name, audit.objectRef)": no such key: name

Rule 0's summary template dereferences audit.responseObject.metadata.name.
For some Gateway audit events responseObject is present but its metadata
has no name key (DELETE, status subresource, and error/forbidden
responses where the apiserver does not echo a named object). CEL raises
no such key: name, evaluation fails, the event goes to the DLQ, and the
retries fail identically — hence the steady leak rather than a spike.

Only this policy leaks. resourcemanager.miloapis.com-project (the other
audit policy) reports 0:

sum(increase(activity_processor_dlq_events_published_total[6h])) by (policy_name, error_type, kind)
  {error_type="cel_summary", kind="Gateway", policy_name="gateway.networking.k8s.io-gateway"} = 120
  {error_type="cel_summary", kind="Project", policy_name="resourcemanager.miloapis.com-project"} = 0

Where the policy config lives

The failing config is a live ActivityPolicy custom resource named
gateway.networking.k8s.io-gateway on the prod-infrastructure-control-plane
control plane. It is not version-controlled:

• Not shipped as a default/seed in this repo — grep for the policy name and
  kind: ActivityPolicy finds only e2e fixtures under
  test/e2e/activitypolicy/policies/. No bootstrap/seeder creates it.
• Not present in the datum-cloud/infra GitOps repo either.

So its source of truth is external / applied ad-hoc. Gap to fix: this
policy should be checked into a GitOps-managed location so a fix survives and
is reviewable.

Relevant code paths (for reference, not the bug):

• internal/controller/activitypolicy_controller.go — reconciles ActivityPolicy
• internal/cel/environment.go — defines the link(displayText, resourceRef) builtin
• internal/cel/policy.go, internal/processor/policy.go — summary evaluation
• internal/activityprocessor/dlq_retry.go — DLQ retry path (emits the error above)

This repo's own example already uses the correct null-safe form —
examples/basic-kubernetes/networking.yaml:19:

{{ has(audit.objectRef.name) ? link(audit.objectRef.name, audit.objectRef) : '' }}

Fix

Make the summary CEL null-safe. audit.objectRef is always present on audit
events and carries the name, so use it as the source (or fall back to it):

link(
  has(audit.responseObject.metadata.name)
    ? audit.responseObject.metadata.name
    : audit.objectRef.name,
  audit.objectRef
)

Apply to the live CR:

kubectl edit activitypolicy gateway.networking.k8s.io-gateway
# patch rule 0 summary; save → processor immediately retries the DLQ backlog

Secondary bug — the runbook's own fix is wrong

docs/runbooks/dlq/policy-dlq-errors.md documents this remediation:

# Before: audit.responseObject.metadata.name
# After:  has(audit.responseObject) ? audit.responseObject.metadata.name : audit.objectRef.name

This does not fix the present error: when responseObject exists but
metadata.name is absent, has(audit.responseObject) is true and the
expression still evaluates audit.responseObject.metadata.name → same
no such key: name. The guard must be on the full leaf path
(has(audit.responseObject.metadata.name)), not on the root object. Branch
fixes the runbook guidance.

Acceptance

• Live gateway.networking.k8s.io-gateway CR patched; cel_summary DLQ
  rate for this policy returns to 0
• Runbook CEL guidance corrected (has() on the leaf path)
• Gateway policy CR moved into a GitOps-managed source of truth (follow-up)

[ecv]

warnings firing in #prod-warning-alerts

[ecv]

where's this policy even from. – ah. NSO.

[ecv]

Status

Root cause confirmed. The leaking config is the live ActivityPolicy CR gateway.networking.k8s.io-gateway. Source of truth is datum-cloud/network-services-operator → config/milo/activity/policies/gateway-policy.yaml (shipped via NSO's OCI bundle, applied to the milo control plane by the Flux milo-activity-policies Kustomization — not stored in datum-cloud/infra, not in this repo).

The create rule matches on has(audit.requestObject.spec) only, so rejected creates (409/422/admission-deny → responseObject is a Status object with no metadata.name) still match, then the summary derefs responseObject.metadata.name → no such key: name → DLQ. Slow steady leak.

Fixes open:

• 🔧 prod fix (draft): fix: null-guard responseObject derefs across NSO ActivityPolicies (DLQ leak) datum-cloud/network-services-operator#230 — guards the leaf, falls back to audit.objectRef.name. Merges + NSO release → Flux re-applies the corrected CR; no kubectl.
• 📄 runbook fix (draft): fix: correct DLQ runbook CEL null-check guidance #213 — the documented remediation here guarded the root object, not the leaf, so it would not have fixed this error class.

In progress: sweeping all other ActivityPolicies (milo / billing / NSO) for the same unguarded responseObject deref pattern — several look latent. Will report findings + follow-up PRs.

[ecv]

Rollup — root cause + fleet-wide sweep

Confirmed root cause. ActivityPolicy create/update rules match on audit.verb / audit.requestObject.spec, then their summary CEL dereferences audit.responseObject.<leaf>. On a rejected create/update (409 AlreadyExists, 422, admission deny) the apiserver returns a Status object as responseObject — no spec, no metadata.name. CEL raises no such key, the event goes to the DLQ, and retries fail identically → slow steady leak (the prod DLQSlowLeak on gateway.networking.k8s.io-gateway).

Source of truth. These policies are not in the activity repo. Each lives in the originating service repo and ships to the milo control plane via that service's OCI bundle + a Flux Kustomization (milo-activity-policies, billing-milo-activity-policies, NSO auto-roll). No kubectl — merge + cut a release and Flux re-applies the corrected CR, which also triggers the processor to retry the DLQ backlog.

Fixes (all draft)

Repo	PR	Rules
datum-cloud/network-services-operator	#230	gateway create (the prod leak) + backendtls / connector / connectoradvertisement / trafficprotection create + domain create+update
milo-os/milo	#672	project, organization, role, group, iam/serviceaccount, identity/serviceaccount — create
milo-os/billing	#69	billingaccountbinding create-with-refs
milo-os/activity	#213	this repo's DLQ runbook — its documented has(responseObject) ? ... fix guards the root, not the leaf, so it would not have fixed this error class

13 rules. Two fix shapes:

• summary: has(<leaf>) ? link(<leaf>, objectRef) : link(audit.objectRef.name, objectRef) (objectRef.name always present).
• billing: match-tightened to require the .name leaves → malformed routes to the existing create-fallback.

Ruled out (not leaks)

• All delete rules — match already guards has(audit.responseObject.spec).
• httproute/httpproxy delete hostnames[0] — match guards has(...hostnames) && size(...) > 0; audit event is immutable, so no empty-array race.

Sanity check please

@scotwells @mattdjenkinson @JoseSzycho @kevwilliams — you last touched these policies / the DLQ path. Two questions:

1. Is "fall back to audit.objectRef.name" the right display for a rejected create, or should these rules instead not match failed requests at all (e.g. gate on audit.responseStatus.code < 300)? The fallback stops the leak but still emits a "created X" activity for a request that was denied — arguably wrong.
2. Same unguarded link(audit.responseObject.metadata.name, ...) pattern is the default for new create rules. Worth a lint / a shared snippet so the next policy doesn't reintroduce it?

[ecv]

Worth a datum-cloud/actions linter? Meh? Just let activity catch it and rely on the pattern to self-propagate?

[ecv]

Related: #215 is the same bug recurring on additional policies (resourcemanager.miloapis.com-project and networking.datumapis.com-connector) — identical error string eval "link(audit.responseObject.metadata.name, audit.objectRef)": no such key: name and same DLQSlowLeak alert. The systemic prevention for this whole class is tracked in #214.