What's happening
When a user runs a PolicyPreview against an ActivityPolicy whose rules look at the request or response payload (fields like audit.requestObject or audit.responseObject), the request fails with a generic 500 Internal Server Error instead of either succeeding or returning a clear validation message.
The same fields are documented as valid in ActivityPolicy rules, so this is surprising to users — their policy looks correct, but previewing it returns an opaque server error.
How it surfaced
- Alert:
EnvoyService5xxErrors fired in staging at 2026-04-14 16:25 UTC on httproute/datum-system/milo-apiserver/rule/0.
- One real 500 reached an end user (running
datumctl to preview a policy); all other requests on the route were healthy.
- The alert has a low threshold (≥1 5xx in 5m), so every user-visible 500 on this path will page us.
What users see
datumctl / API clients get a 500 with no actionable guidance.- Affects anyone writing or iterating on ActivityPolicies that reference request/response content — a common case.
Expected behavior
- Policies that legitimately reference request/response fields should preview successfully.
- If a policy genuinely has an invalid filter, the API should respond with a 400-class error and a helpful message — not a 500.
Next steps
Two things to fix:
- Make PolicyPreview work for policies that reference request/response content. Today the preview path can't sample against those fields even though ActivityPolicy allows them.
- Stop returning 500 for user-input problems. Validation-style failures should come back as 4xx with a clear message so users can fix their policy without engineering having to investigate each alert.
Context for engineers
Triggering request (one-off, user kwilliams, staging):
- Audit ID:
8ebad696-4620-467f-93c3-f0b46daab58f
- Activity apiserver pod:
activity-apiserver-7cb5fc48bd-mdgpc
- Response message:
failed to auto-fetch sample inputs: failed to fetch audit log samples: Invalid filter ... undeclared reference to 'requestObject'
Starting points in the code:
internal/registry/activity/preview/storage.go — the auto-fetch path that pushes policy rule CEL into the audit-log query filter, and wraps the result as NewInternalError.internal/cel/environment.go / internal/cel/errors.go — the AuditLogQuery CEL schema, which is a subset of the ActivityPolicy CEL schema.
What's happening
When a user runs a PolicyPreview against an ActivityPolicy whose rules look at the request or response payload (fields like
audit.requestObjectoraudit.responseObject), the request fails with a generic500 Internal Server Errorinstead of either succeeding or returning a clear validation message.The same fields are documented as valid in ActivityPolicy rules, so this is surprising to users — their policy looks correct, but previewing it returns an opaque server error.
How it surfaced
EnvoyService5xxErrorsfired in staging at 2026-04-14 16:25 UTC onhttproute/datum-system/milo-apiserver/rule/0.datumctlto preview a policy); all other requests on the route were healthy.What users see
datumctl/ API clients get a 500 with no actionable guidance.Expected behavior
Next steps
Two things to fix:
Context for engineers
Triggering request (one-off, user
kwilliams, staging):8ebad696-4620-467f-93c3-f0b46daab58factivity-apiserver-7cb5fc48bd-mdgpcfailed to auto-fetch sample inputs: failed to fetch audit log samples: Invalid filter ... undeclared reference to 'requestObject'Starting points in the code:
internal/registry/activity/preview/storage.go— the auto-fetch path that pushes policy rule CEL into the audit-log query filter, and wraps the result asNewInternalError.internal/cel/environment.go/internal/cel/errors.go— the AuditLogQuery CEL schema, which is a subset of the ActivityPolicy CEL schema.