Skip to content

Add importable Datadog dashboard example - #142

Merged
prashar32 merged 2 commits into
mainfrom
docs/datadog-dashboard
Jun 15, 2026
Merged

Add importable Datadog dashboard example#142
prashar32 merged 2 commits into
mainfrom
docs/datadog-dashboard

Conversation

@prashar32

Copy link
Copy Markdown
Owner

Completes the OTel dashboard set — Grafana + Tempo and SigNoz already shipped; this adds the Datadog import so a team already running Datadog can visualize governed runs straight from the OpenTelemetry GenAI spans RiskKernel exports. No extra instrumentation: Datadog ingests the spans over OTLP, and you import one JSON file.

What's here

  • examples/otel/datadog/dashboards/riskkernel-agent-runs.json — Datadog's importable dashboard schema (layout_type: ordered, data_source: spans). Panel parity with Grafana/SigNoz:
    • Total spend, output tokens, model calls, tool calls refused (query-value tiles)
    • Spend over time, spend by run (toplist), p95 latency by model, model-call rate by model, output tokens by model, tool calls by outcome, budget halts by reason
  • examples/otel/datadog/README.md — how to point the OTLP exporter at the Datadog Agent's OTLP receiver (gRPC 4317 / HTTP 4318), import the dashboard, and what each panel queries.

Datadog specifics (called out honestly in the README)

  • The Datadog key lives on the Agent, not RiskKernel. RiskKernel only emits OTLP to the endpoint you give it (no phone-home); DD_API_KEY is standard Agent config.
  • Span attributes are queried with the @ prefix (@riskkernel.cost.usd, @gen_ai.request.model); tags have none. Datadog's OTLP intake preserves the dotted names, so they read identically to the pinned set.
  • @duration is Datadog's intrinsic span-duration measure (nanoseconds) for the latency panel — intrinsic, not a RiskKernel attribute.
  • Indexed spans + retention. data_source: spans panels query retained spans; the README notes keeping a retention filter for RiskKernel's spans, and documents the optional span-based metric upgrade for durable, long-window cost rollups (and the billing note).
  • Service name maps to Datadog's service tag; the dashboard ships a $service template variable defaulting to riskkernel.

Contract fidelity

Every attribute the queries touch matches api/v1/otel-genai.md exactly (verified programmatically): gen_ai.operation.name, gen_ai.request.model, gen_ai.usage.output_tokens, riskkernel.cost.usd, riskkernel.run.id, riskkernel.tool.status, riskkernel.halt.reason — plus the built-in @duration. The JSON parses cleanly. Cross-linked from the Grafana, SigNoz, and main observability READMEs, and a CHANGELOG entry under [Unreleased].

Closes #89

Completes the dashboard set (Grafana + Tempo, SigNoz, now Datadog) so a
team already running Datadog can visualize governed runs from the same
OpenTelemetry GenAI spans RiskKernel exports — no extra instrumentation.

The dashboard JSON is in Datadog's import schema (ordered layout,
data_source: spans) and reaches parity with the Grafana/SigNoz panels:
total/over-time spend, spend by run, halts by reason, tool outcomes,
latency and token burn by model, model-call rate. Every query references
the attribute names pinned in api/v1/otel-genai.md, with Datadog's
@-prefix for span attributes (@riskkernel.cost.usd, @gen_ai.request.model,
...) and the built-in @duration measure for latency.

The README covers pointing the OTLP exporter at the Datadog Agent's OTLP
receiver (gRPC 4317 / HTTP 4318) — the Datadog API key sits on the Agent,
not RiskKernel, which only emits OTLP — plus the @-prefix gotcha, the
service-name template variable, retention/indexed-span behavior, and the
optional span-based-metric path for durable long-window cost rollups.
Cross-linked from the Grafana/SigNoz/main observability READMEs.
@prashar32
prashar32 merged commit 38fee06 into main Jun 15, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Example dashboards for SigNoz and Datadog

1 participant