Skip to content

ai(observability): the redaction scrubber nulls every AI token count — SECRET_KEY matches /token/i #10211

Description

@JSONbored

Part of epic #8286 (Phase 3 — AI observability). Root cause of #10207.

Problem

The privacy scrubber redacts every property whose key matches SECRET_KEY:

export const SECRET_KEY =
  /(token|secret|key|password|passwd|authorization|auth|dsn|cookie|bearer|credential|private)/i;

/token/i matches PostHog's own $ai_input_tokens and $ai_output_tokens. So scrubRecord — wired as posthog-node's before_send — rewrites both to the "[redacted]" string, and PostHog then coerces that to null on its numerically-typed properties.

Confirmed on a real live event:

{"$ai_http_status":200,"$ai_input_tokens":null,"$ai_is_error":false,"$ai_latency":10.308,
 "$ai_model":"claude-sonnet-5","$ai_output_tokens":null,"$ai_provider":"claude-code",
 "$ai_total_cost_usd":0.44771210000000006, ...}

$ai_total_cost_usd survives because it has no secret-shaped word in it. The token fields do not.

Impact

Not one AI call in the project has ever carried a token count — posthog.ai_events.input_tokens / output_tokens / total_tokens are NULL for every model over the retention window, including claude-sonnet-5 at 2,321 calls and $498.43 of real spend. PostHog's LLM cost views break spend down by token usage, so the ORB shows dollars but no tokens/call, no input-vs-output ratio, and no cost-per-token; $ai_input_cost_usd / $ai_output_cost_usd, which PostHog derives from tokens, cannot be computed at all.

This also silently defeats the miner-side split landed in #10199: those events would be scrubbed the same way.

Fix

A secret-shaped key holding a number is a counter, not a credential. Every secret this module exists to catch — a token, an API key, a password, a cookie, a DSN, a bearer header — is a string; there is no numeric form of one to leak. Skip redaction when the value is a number, and keep redacting every other shape (string, object, array, boolean) exactly as today.

This is deliberately general rather than an allowlist of the two $ai_* keys: an allowlist would go stale the moment PostHog adds $ai_cache_read_input_tokens or the code adds another counter, and it would fail the same way — silently.

Deliverables

  • Numeric values under a secret-shaped key survive scrubRecord, including 0.
  • String values under a secret-shaped key are still redacted.
  • Non-numeric, non-string values (objects, arrays, booleans) under a secret-shaped key are still redacted — the carve-out must not widen into "anything that is not a string".
  • Regression test naming the real properties that were lost.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions