From fbe8853d7576281981647a1051b4abd7a16aa4d7 Mon Sep 17 00:00:00 2001 From: Adarsh Prashar Date: Tue, 16 Jun 2026 01:12:14 +0530 Subject: [PATCH] docs(otel): add importable Datadog dashboard example MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completes the dashboard set (Grafana + Tempo, SigNoz, now Datadog) so a team already running Datadog can visualize governed runs from the same OpenTelemetry GenAI spans RiskKernel exports — no extra instrumentation. The dashboard JSON is in Datadog's import schema (ordered layout, data_source: spans) and reaches parity with the Grafana/SigNoz panels: total/over-time spend, spend by run, halts by reason, tool outcomes, latency and token burn by model, model-call rate. Every query references the attribute names pinned in api/v1/otel-genai.md, with Datadog's @-prefix for span attributes (@riskkernel.cost.usd, @gen_ai.request.model, ...) and the built-in @duration measure for latency. The README covers pointing the OTLP exporter at the Datadog Agent's OTLP receiver (gRPC 4317 / HTTP 4318) — the Datadog API key sits on the Agent, not RiskKernel, which only emits OTLP — plus the @-prefix gotcha, the service-name template variable, retention/indexed-span behavior, and the optional span-based-metric path for durable long-window cost rollups. Cross-linked from the Grafana/SigNoz/main observability READMEs. --- CHANGELOG.md | 10 + examples/otel/README.md | 3 +- examples/otel/datadog/README.md | 121 ++++ .../dashboards/riskkernel-agent-runs.json | 566 ++++++++++++++++++ examples/otel/grafana/README.md | 3 +- examples/otel/signoz/README.md | 4 +- 6 files changed, 704 insertions(+), 3 deletions(-) create mode 100644 examples/otel/datadog/README.md create mode 100644 examples/otel/datadog/dashboards/riskkernel-agent-runs.json diff --git a/CHANGELOG.md b/CHANGELOG.md index 7904b26..4e209c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,16 @@ surface is governed by [`COMPATIBILITY.md`](COMPATIBILITY.md). ## [Unreleased] ### Added +- **Importable Datadog dashboard.** [`examples/otel/datadog`](examples/otel/datadog) + ships a ready-made Datadog dashboard (spend per run, budget halts by reason, + tool-call outcomes, latency and token burn by model) built from the same + OpenTelemetry GenAI spans RiskKernel already exports — completing the dashboard + set alongside the Grafana + Tempo and SigNoz examples. Panels query the spans + Datadog ingests over OTLP, referencing attributes by their `@`-prefixed names + (e.g. `@riskkernel.cost.usd`) exactly as pinned in + [`api/v1/otel-genai.md`](api/v1/otel-genai.md); a short README covers pointing the + OTLP exporter at the Datadog Agent (the Datadog key lives on the Agent, not + RiskKernel) and the optional span-based-metric path for long-term cost rollups. - **Spend rollup across runs (`riskkernel audit summary`).** Per-run `audit export` has a cross-run companion: `riskkernel audit summary --by >` rolls cost-ledger spend up by a dimension — so spend **by team/user/feature** comes diff --git a/examples/otel/README.md b/examples/otel/README.md index 67c962f..24fe40b 100644 --- a/examples/otel/README.md +++ b/examples/otel/README.md @@ -77,7 +77,8 @@ Header values carry secrets and are never logged. The traces-specific **Want it ready-made?** [`grafana/`](grafana/) ships a provisioned Grafana + Tempo stack with a RiskKernel dashboard (spend, halts, tool outcomes, latency) — one -`docker compose up`, no import step. +`docker compose up`, no import step. The same dashboard is importable on +[`signoz/`](signoz/) and [`datadog/`](datadog/) for teams already on those backends. Because cost and budget live on every span as first-class attributes, you can also build panels directly from spans (e.g. in Grafana over Tempo, or SigNoz): diff --git a/examples/otel/datadog/README.md b/examples/otel/datadog/README.md new file mode 100644 index 0000000..9d8b971 --- /dev/null +++ b/examples/otel/datadog/README.md @@ -0,0 +1,121 @@ +# Cost & governance dashboard (Datadog) + +An importable [Datadog](https://www.datadoghq.com) dashboard for your governed runs +— **spend, token burn, budget halts, tool-call outcomes, and latency by model** — +built entirely from the OpenTelemetry spans RiskKernel already emits. No extra +instrumentation: Datadog ingests the spans over OTLP, aggregates them, and you +import one JSON file. + +> Already on Grafana + Tempo? [`../grafana/`](../grafana/) ships the same panels as +> a provisioned stack. On [SigNoz](https://signoz.io)? [`../signoz/`](../signoz/) is +> the SigNoz import. This is the Datadog equivalent. + +## Point RiskKernel at Datadog + +RiskKernel exports OTLP traces; the **Datadog Agent** ingests OTLP. RiskKernel just +emits OTLP — **the Datadog API key lives on the Agent, not on RiskKernel** (the +runtime never phones home; it only sends spans to the endpoint you point it at). + +First, enable the Agent's OTLP receiver (gRPC `4317` / HTTP `4318`) and give the +Agent your `DD_API_KEY` — this is standard Datadog Agent config, the same key the +Agent already uses to forward everything else: + +```bash +# On the Datadog Agent (Docker example): +docker run -d --name dd-agent \ + -e DD_API_KEY=$DD_API_KEY \ + -e DD_SITE=datadoghq.com \ + -e DD_APM_ENABLED=true \ + -e DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT=0.0.0.0:4317 \ + -e DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT=0.0.0.0:4318 \ + -p 4317:4317 -p 4318:4318 \ + gcr.io/datadoghq/agent:latest +``` + +Then point RiskKernel's exporter at the Agent and start the daemon (your keys, as +always) — the usual one env var: + +```bash +# RiskKernel -> Datadog Agent OTLP (no Datadog key here; it's on the Agent): +export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 +export OTEL_EXPORTER_OTLP_PROTOCOL=grpc +export ANTHROPIC_API_KEY=sk-ant-... +riskkernel serve + +# Then drive some governed traffic — point your app's base URL at the proxy: +export OPENAI_BASE_URL=http://localhost:7070/v1 +# (or run any example under ../../ — the loop-killer, the MCP demo, etc.) +``` + +Use `OTEL_EXPORTER_OTLP_PROTOCOL=http` with endpoint `http://localhost:4318` if you +prefer OTLP/HTTP. See the OTLP export notes in [`../README.md`](../README.md) for +HTTP-vs-gRPC and other backends. + +> **No telemetry by default.** RiskKernel exports *nothing* unless +> `OTEL_EXPORTER_OTLP_ENDPOINT` is set; spans go only to the Datadog Agent you point +> it at. See [`SECURITY.md`](../../../SECURITY.md). + +## Import the dashboard + +In the Datadog UI: **Dashboards → New Dashboard → Import dashboard JSON** (or the +gear menu → **Import dashboard JSON** on an existing one), and upload +[`dashboards/riskkernel-agent-runs.json`](dashboards/riskkernel-agent-runs.json). +Spend, halts, and tool outcomes fill in as runs execute (default range: last 1h — +adjust with the time picker). + +## What each panel is + +Datadog references **span attributes with an `@` prefix** (tags have none), so the +queries read `@riskkernel.cost.usd`, `@gen_ai.request.model`, and friends — exactly +the names RiskKernel emits. + +| Panel | Span query (`data_source: spans`) | +|---|---| +| Total spend / Output tokens / Model calls | `sum(@riskkernel.cost.usd)`, `sum(@gen_ai.usage.output_tokens)`, `count` of `@gen_ai.operation.name:chat` | +| **Tool calls refused** | `count` over `@gen_ai.operation.name:execute_tool -@riskkernel.tool.status:approved` | +| Spend over time | `sum(@riskkernel.cost.usd)` over time | +| Spend by run | `sum(@riskkernel.cost.usd)` group by `@riskkernel.run.id` (toplist) | +| p95 latency by model | `pc95(@duration)` group by `@gen_ai.request.model` | +| Model-call rate by model | `count` of `chat` spans group by `@gen_ai.request.model` | +| Output tokens by model | `sum(@gen_ai.usage.output_tokens)` group by `@gen_ai.request.model` | +| Tool calls by outcome | `count` of `execute_tool` spans group by `@riskkernel.tool.status` (approved / blocked / denied / timeout) | +| Budget halts by reason | `count` over `@riskkernel.halt.reason:*` group by `@riskkernel.halt.reason` | + +Every attribute these queries touch is pinned in +[`api/v1/otel-genai.md`](../../../api/v1/otel-genai.md) — the names are a stable +public contract, so the panels keep working across RiskKernel upgrades. + +## Notes & assumptions + +A few Datadog-specific things to know: + +- **Service name → the `service` tag.** OTLP's `service.name` resource attribute + becomes Datadog's `service`. RiskKernel sets it to `riskkernel`, so the dashboard + ships a `$service` template variable defaulting to `riskkernel`. If you've + overridden the service name, pick yours from the variable (or edit its default); + if you run a single Datadog org for RiskKernel only, you can clear the filter. +- **The `@` prefix.** This is the one thing people trip on: span *attributes* need + the leading `@` (`@riskkernel.cost.usd`); span *tags* (host/env/service) do not. + The dotted attribute names are preserved by Datadog's OTLP intake — they are not + flattened to underscores — so they read identically to the pinned set. +- **`@duration`.** The latency panel uses Datadog's intrinsic span-duration measure + (`@duration`, **nanoseconds**) — span duration is intrinsic, not a RiskKernel + attribute. +- **Indexed spans + retention.** Dashboard widgets with `data_source: spans` query + the spans Datadog **retains** (via your retention filters). Ad-hoc span analytics + has a limited retention window, so make sure a retention filter keeps RiskKernel's + spans (e.g. `service:riskkernel`) if you want history beyond it. +- **Long-term cost aggregation → a span-based metric (optional).** For durable, + long-window spend rollups (and faster dashboards), generate a **span-based custom + metric** from `@riskkernel.cost.usd` once, then point the spend panels at that + metric instead of the raw spans. In **APM → Generate Metrics → New Metric**, set + the query to `service:riskkernel @riskkernel.cost.usd:*`, aggregate the + `@riskkernel.cost.usd` value as `sum`, and add `@riskkernel.run.id` (and any + `@riskkernel.run.meta.*` tag) as a group-by — then the "Spend over time" / "Spend + by run" panels can switch their `data_source` from `spans` to `metrics`. The + shipped JSON uses raw spans so it works on import with no setup; the metric is the + upgrade path for retention and scale. Note that custom metrics are billable. +- **Empty panels?** That almost always means no spans match the filter yet — drive + some governed traffic, widen the time range, confirm the `$service` variable + matches your deployment, and confirm a retention filter is keeping the spans. The + tool/halt panels stay empty until a tool call or a budget halt actually happens. diff --git a/examples/otel/datadog/dashboards/riskkernel-agent-runs.json b/examples/otel/datadog/dashboards/riskkernel-agent-runs.json new file mode 100644 index 0000000..744c76b --- /dev/null +++ b/examples/otel/datadog/dashboards/riskkernel-agent-runs.json @@ -0,0 +1,566 @@ +{ + "title": "RiskKernel — agent runs", + "description": "Spend, token burn, budget halts, tool-call outcomes, and latency by model for your governed agent runs, built from the OpenTelemetry GenAI spans RiskKernel exports. Queries the indexed spans Datadog ingests over OTLP; span attributes are referenced with the @ prefix (e.g. @riskkernel.cost.usd). See api/v1/otel-genai.md for the pinned attribute set, and the README for the optional span-based-metric path for long-term cost aggregation.", + "layout_type": "ordered", + "reflow_type": "fixed", + "tags": [ + "riskkernel", + "agents", + "genai", + "llm" + ], + "template_variables": [ + { + "name": "service", + "prefix": "service", + "available_values": [], + "default": "riskkernel" + } + ], + "widgets": [ + { + "definition": { + "title": "Total spend", + "title_size": "16", + "title_align": "left", + "type": "query_value", + "requests": [ + { + "response_format": "scalar", + "queries": [ + { + "name": "query1", + "data_source": "spans", + "search": { + "query": "service:$service.value @riskkernel.cost.usd:*" + }, + "compute": { + "aggregation": "sum", + "metric": "@riskkernel.cost.usd" + }, + "group_by": [] + } + ], + "formulas": [ + { + "formula": "query1", + "number_format": { + "unit": { + "type": "canonical_unit", + "unit_name": "dollar" + } + } + } + ] + } + ], + "autoscale": true, + "precision": 4 + }, + "layout": { + "x": 0, + "y": 0, + "width": 3, + "height": 2 + } + }, + { + "definition": { + "title": "Output tokens", + "title_size": "16", + "title_align": "left", + "type": "query_value", + "requests": [ + { + "response_format": "scalar", + "queries": [ + { + "name": "query1", + "data_source": "spans", + "search": { + "query": "service:$service.value @gen_ai.usage.output_tokens:*" + }, + "compute": { + "aggregation": "sum", + "metric": "@gen_ai.usage.output_tokens" + }, + "group_by": [] + } + ], + "formulas": [ + { + "formula": "query1" + } + ] + } + ], + "autoscale": true, + "precision": 0 + }, + "layout": { + "x": 3, + "y": 0, + "width": 3, + "height": 2 + } + }, + { + "definition": { + "title": "Model calls", + "title_size": "16", + "title_align": "left", + "type": "query_value", + "requests": [ + { + "response_format": "scalar", + "queries": [ + { + "name": "query1", + "data_source": "spans", + "search": { + "query": "service:$service.value @gen_ai.operation.name:chat" + }, + "compute": { + "aggregation": "count" + }, + "group_by": [] + } + ], + "formulas": [ + { + "formula": "query1" + } + ] + } + ], + "autoscale": true, + "precision": 0 + }, + "layout": { + "x": 6, + "y": 0, + "width": 3, + "height": 2 + } + }, + { + "definition": { + "title": "Tool calls refused", + "title_size": "16", + "title_align": "left", + "type": "query_value", + "requests": [ + { + "response_format": "scalar", + "queries": [ + { + "name": "query1", + "data_source": "spans", + "search": { + "query": "service:$service.value @gen_ai.operation.name:execute_tool -@riskkernel.tool.status:approved" + }, + "compute": { + "aggregation": "count" + }, + "group_by": [] + } + ], + "formulas": [ + { + "formula": "query1" + } + ] + } + ], + "autoscale": true, + "precision": 0 + }, + "layout": { + "x": 9, + "y": 0, + "width": 3, + "height": 2 + } + }, + { + "definition": { + "title": "Spend over time (USD)", + "title_size": "16", + "title_align": "left", + "show_legend": true, + "legend_layout": "auto", + "type": "timeseries", + "requests": [ + { + "response_format": "timeseries", + "queries": [ + { + "name": "query1", + "data_source": "spans", + "search": { + "query": "service:$service.value @riskkernel.cost.usd:*" + }, + "compute": { + "aggregation": "sum", + "metric": "@riskkernel.cost.usd" + }, + "group_by": [] + } + ], + "formulas": [ + { + "formula": "query1" + } + ], + "style": { + "palette": "dog_classic", + "line_type": "solid", + "line_width": "normal" + }, + "display_type": "bars" + } + ] + }, + "layout": { + "x": 0, + "y": 2, + "width": 6, + "height": 3 + } + }, + { + "definition": { + "title": "Spend by run (USD)", + "title_size": "16", + "title_align": "left", + "type": "toplist", + "requests": [ + { + "response_format": "scalar", + "queries": [ + { + "name": "query1", + "data_source": "spans", + "search": { + "query": "service:$service.value @riskkernel.cost.usd:*" + }, + "compute": { + "aggregation": "sum", + "metric": "@riskkernel.cost.usd" + }, + "group_by": [ + { + "facet": "@riskkernel.run.id", + "limit": 20, + "sort": { + "aggregation": "sum", + "metric": "@riskkernel.cost.usd", + "order": "desc" + } + } + ] + } + ], + "formulas": [ + { + "formula": "query1", + "number_format": { + "unit": { + "type": "canonical_unit", + "unit_name": "dollar" + } + } + } + ] + } + ] + }, + "layout": { + "x": 6, + "y": 2, + "width": 6, + "height": 3 + } + }, + { + "definition": { + "title": "p95 latency by model", + "title_size": "16", + "title_align": "left", + "show_legend": true, + "legend_layout": "auto", + "type": "timeseries", + "requests": [ + { + "response_format": "timeseries", + "queries": [ + { + "name": "query1", + "data_source": "spans", + "search": { + "query": "service:$service.value @gen_ai.operation.name:chat" + }, + "compute": { + "aggregation": "pc95", + "metric": "@duration" + }, + "group_by": [ + { + "facet": "@gen_ai.request.model", + "limit": 10, + "sort": { + "aggregation": "pc95", + "metric": "@duration", + "order": "desc" + } + } + ] + } + ], + "formulas": [ + { + "formula": "query1", + "number_format": { + "unit": { + "type": "canonical_unit", + "unit_name": "nanosecond" + } + } + } + ], + "style": { + "palette": "dog_classic", + "line_type": "solid", + "line_width": "normal" + }, + "display_type": "line" + } + ] + }, + "layout": { + "x": 0, + "y": 5, + "width": 6, + "height": 3 + } + }, + { + "definition": { + "title": "Model-call rate by model", + "title_size": "16", + "title_align": "left", + "show_legend": true, + "legend_layout": "auto", + "type": "timeseries", + "requests": [ + { + "response_format": "timeseries", + "queries": [ + { + "name": "query1", + "data_source": "spans", + "search": { + "query": "service:$service.value @gen_ai.operation.name:chat" + }, + "compute": { + "aggregation": "count" + }, + "group_by": [ + { + "facet": "@gen_ai.request.model", + "limit": 10, + "sort": { + "aggregation": "count", + "order": "desc" + } + } + ] + } + ], + "formulas": [ + { + "formula": "query1" + } + ], + "style": { + "palette": "dog_classic", + "line_type": "solid", + "line_width": "normal" + }, + "display_type": "line" + } + ] + }, + "layout": { + "x": 6, + "y": 5, + "width": 6, + "height": 3 + } + }, + { + "definition": { + "title": "Output tokens by model", + "title_size": "16", + "title_align": "left", + "show_legend": true, + "legend_layout": "auto", + "type": "timeseries", + "requests": [ + { + "response_format": "timeseries", + "queries": [ + { + "name": "query1", + "data_source": "spans", + "search": { + "query": "service:$service.value @gen_ai.operation.name:chat @gen_ai.usage.output_tokens:*" + }, + "compute": { + "aggregation": "sum", + "metric": "@gen_ai.usage.output_tokens" + }, + "group_by": [ + { + "facet": "@gen_ai.request.model", + "limit": 10, + "sort": { + "aggregation": "sum", + "metric": "@gen_ai.usage.output_tokens", + "order": "desc" + } + } + ] + } + ], + "formulas": [ + { + "formula": "query1" + } + ], + "style": { + "palette": "dog_classic", + "line_type": "solid", + "line_width": "normal" + }, + "display_type": "bars" + } + ] + }, + "layout": { + "x": 0, + "y": 8, + "width": 6, + "height": 3 + } + }, + { + "definition": { + "title": "Tool calls by outcome", + "title_size": "16", + "title_align": "left", + "show_legend": true, + "legend_layout": "auto", + "type": "timeseries", + "requests": [ + { + "response_format": "timeseries", + "queries": [ + { + "name": "query1", + "data_source": "spans", + "search": { + "query": "service:$service.value @gen_ai.operation.name:execute_tool" + }, + "compute": { + "aggregation": "count" + }, + "group_by": [ + { + "facet": "@riskkernel.tool.status", + "limit": 10, + "sort": { + "aggregation": "count", + "order": "desc" + } + } + ] + } + ], + "formulas": [ + { + "formula": "query1" + } + ], + "style": { + "palette": "dog_classic", + "line_type": "solid", + "line_width": "normal" + }, + "display_type": "bars" + } + ] + }, + "layout": { + "x": 6, + "y": 8, + "width": 6, + "height": 3 + } + }, + { + "definition": { + "title": "Budget halts by reason", + "title_size": "16", + "title_align": "left", + "show_legend": true, + "legend_layout": "auto", + "type": "timeseries", + "requests": [ + { + "response_format": "timeseries", + "queries": [ + { + "name": "query1", + "data_source": "spans", + "search": { + "query": "service:$service.value @riskkernel.halt.reason:*" + }, + "compute": { + "aggregation": "count" + }, + "group_by": [ + { + "facet": "@riskkernel.halt.reason", + "limit": 10, + "sort": { + "aggregation": "count", + "order": "desc" + } + } + ] + } + ], + "formulas": [ + { + "formula": "query1" + } + ], + "style": { + "palette": "warm", + "line_type": "solid", + "line_width": "normal" + }, + "display_type": "bars" + } + ] + }, + "layout": { + "x": 0, + "y": 11, + "width": 12, + "height": 3 + } + } + ] +} diff --git a/examples/otel/grafana/README.md b/examples/otel/grafana/README.md index b82fb2c..221387f 100644 --- a/examples/otel/grafana/README.md +++ b/examples/otel/grafana/README.md @@ -9,7 +9,8 @@ TraceQL metrics, and the dashboard is provisioned so it shows up on first load. > The [Jaeger quick-look](../docker-compose.yaml) one directory up is for reading > individual traces. This is the aggregate view — the panels a platform team watches. > On [SigNoz](https://signoz.io) instead? [`../signoz/`](../signoz/) is the same -> dashboard, importable there. +> dashboard, importable there; on [Datadog](https://www.datadoghq.com)? +> [`../datadog/`](../datadog/) ships it as a Datadog dashboard JSON. ![panels: total spend, output tokens, model calls, tool calls refused, spend over time, spend by run, p95 latency by model, model-call rate, tool calls by outcome, diff --git a/examples/otel/signoz/README.md b/examples/otel/signoz/README.md index 8b06559..4869552 100644 --- a/examples/otel/signoz/README.md +++ b/examples/otel/signoz/README.md @@ -7,7 +7,9 @@ instrumentation, no metrics pipeline: SigNoz aggregates the spans, and you impor one JSON file. > Already on Grafana + Tempo? [`../grafana/`](../grafana/) ships the same panels as -> a provisioned stack. This is the SigNoz equivalent for teams who already run it. +> a provisioned stack. On [Datadog](https://www.datadoghq.com)? +> [`../datadog/`](../datadog/) is the Datadog import. This is the SigNoz equivalent +> for teams who already run it. ## Point RiskKernel at SigNoz