diff --git a/CHANGELOG.md b/CHANGELOG.md index a92b987..274be01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,11 @@ surface is governed by [`COMPATIBILITY.md`](COMPATIBILITY.md). blocks and approval denials are now visible in whatever OTLP backend you already run — a refused call is marked with an error span status so it stands out. See [`api/v1/otel-genai.md`](api/v1/otel-genai.md) and [`examples/otel`](examples/otel). +- **A ready-made cost & governance dashboard** ([`examples/otel/grafana`](examples/otel/grafana)) — + a provisioned Grafana + Tempo stack that turns RiskKernel's spans into panels: + spend over time and per run, token burn, budget halts by reason, tool-call outcomes, + and p95 latency by model. Built from the spans you already emit (Tempo TraceQL + metrics — no extra instrumentation); `docker compose up`, no import step. ## [0.3.0] - 2026-06-06 diff --git a/examples/otel/README.md b/examples/otel/README.md index 42adc81..a16c6ea 100644 --- a/examples/otel/README.md +++ b/examples/otel/README.md @@ -59,7 +59,11 @@ backend prefers OTLP/HTTP. ## Building cost/usage dashboards -Because cost and budget live on every span as first-class attributes, you can build +**Want it ready-made?** [`grafana/`](grafana/) ships a provisioned Grafana + Tempo +stack with a RiskKernel dashboard (spend, halts, tool outcomes, latency) — one +`docker compose up`, no import step. + +Because cost and budget live on every span as first-class attributes, you can also build panels directly from spans (e.g. in Grafana over Tempo, or SigNoz): - **Spend per run** — sum `riskkernel.cost.usd` grouped by `riskkernel.run.id`. diff --git a/examples/otel/grafana/README.md b/examples/otel/grafana/README.md new file mode 100644 index 0000000..d57977b --- /dev/null +++ b/examples/otel/grafana/README.md @@ -0,0 +1,64 @@ +# Cost & governance dashboard (Grafana + Tempo) + +A ready-made Grafana dashboard for your governed runs — **spend, token burn, +budget halts, tool-call outcomes, and latency by model** — built entirely from the +OpenTelemetry spans RiskKernel already emits. No extra instrumentation, no metrics +pipeline: [Grafana Tempo](https://grafana.com/oss/tempo/) aggregates the spans with +TraceQL metrics, and the dashboard is provisioned so it shows up on first load. + +> The [Jaeger quick-look](../docker-compose.yaml) one directory up is for reading +> individual traces. This is the aggregate view — the panels a platform team watches. + +![panels: total spend, output tokens, model calls, tool calls refused, spend over +time, spend by run, p95 latency by model, model-call rate, tool calls by outcome, +budget halts by reason] + +## Run it + +```bash +# 1. Bring up Tempo (OTLP in) + Grafana (dashboard provisioned) +docker compose -f examples/otel/grafana/docker-compose.yaml up -d + +# 2. Point RiskKernel at Tempo and start it (your keys, as always) +export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 +export OTEL_EXPORTER_OTLP_PROTOCOL=grpc +export ANTHROPIC_API_KEY=sk-ant-... +riskkernel serve + +# 3. Drive some governed traffic — point your app's base URL at the proxy: +export OPENAI_BASE_URL=http://localhost:7070/v1 +# (or run any example under ../../ — the loop-killer, the MCP demo, etc.) +``` + +Open **http://localhost:3000** → **Dashboards** → **RiskKernel — agent runs**. +Spend, halts, and tool outcomes fill in as runs execute (default range: last 1h). + +> **No telemetry by default.** RiskKernel exports nothing unless +> `OTEL_EXPORTER_OTLP_ENDPOINT` is set; spans go only to the Tempo you run here. See +> [`SECURITY.md`](../../../SECURITY.md). The Grafana in this stack is login-free for +> convenience — don't expose it as-is. + +## What each panel is + +| Panel | TraceQL metric | +|---|---| +| Total spend / Output tokens / Model calls | `sum_over_time(span.riskkernel.cost.usd)`, `sum_over_time(span.gen_ai.usage.output_tokens)`, `count_over_time()` | +| **Tool calls refused** | `count_over_time()` over `execute_tool` spans where `riskkernel.tool.status != "approved"` | +| Spend over time / Spend by run | `sum_over_time(span.riskkernel.cost.usd)`, grouped `by (span.riskkernel.run.id)` | +| p95 latency by model | `quantile_over_time(duration, .95) by (span.gen_ai.request.model)` | +| Model-call rate by model | `rate() by (span.gen_ai.request.model)` | +| Tool calls by outcome | `count_over_time() by (span.riskkernel.tool.status)` (approved / blocked / denied / timeout) | +| Budget halts by reason | `count_over_time() by (span.riskkernel.halt.reason)` | + +Every attribute these queries touch is pinned in +[`api/v1/otel-genai.md`](../../../api/v1/otel-genai.md). + +## Using it against your own Grafana + +The dashboard is a plain Grafana JSON +([`dashboards/riskkernel-agent-runs.json`](dashboards/riskkernel-agent-runs.json)). +To use your existing stack instead of this compose: import the JSON and point its +**Tempo** data source at any Tempo with +[TraceQL metrics enabled](https://grafana.com/docs/tempo/latest/metrics-from-traces/) +(the `local-blocks` processor — see [`tempo.yaml`](tempo.yaml)). Needs Grafana +10.4+ and Tempo 2.7+ (the dashboard uses `sum_over_time`). diff --git a/examples/otel/grafana/dashboards/riskkernel-agent-runs.json b/examples/otel/grafana/dashboards/riskkernel-agent-runs.json new file mode 100644 index 0000000..5ec3dd0 --- /dev/null +++ b/examples/otel/grafana/dashboards/riskkernel-agent-runs.json @@ -0,0 +1,660 @@ +{ + "annotations": { + "list": [] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 1, + "links": [], + "panels": [ + { + "id": 1, + "type": "stat", + "title": "Total spend", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 0, + "y": 0 + }, + "fieldConfig": { + "defaults": { + "unit": "currencyUSD", + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": null + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "sum" + ], + "fields": "", + "values": false + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto" + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ resource.service.name = \"riskkernel\" } | sum_over_time(span.riskkernel.cost.usd)", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + }, + { + "id": 2, + "type": "stat", + "title": "Output tokens", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 6, + "y": 0 + }, + "fieldConfig": { + "defaults": { + "unit": "short", + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": null + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "sum" + ], + "fields": "", + "values": false + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto" + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ resource.service.name = \"riskkernel\" } | sum_over_time(span.gen_ai.usage.output_tokens)", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + }, + { + "id": 3, + "type": "stat", + "title": "Model calls", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 12, + "y": 0 + }, + "fieldConfig": { + "defaults": { + "unit": "short", + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": null + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "sum" + ], + "fields": "", + "values": false + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto" + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ span.gen_ai.operation.name = \"chat\" } | count_over_time()", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + }, + { + "id": 4, + "type": "stat", + "title": "Tool calls refused", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 18, + "y": 0 + }, + "fieldConfig": { + "defaults": { + "unit": "short", + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1 + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "sum" + ], + "fields": "", + "values": false + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto" + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ span.gen_ai.operation.name = \"execute_tool\" && span.riskkernel.tool.status != \"approved\" } | count_over_time()", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + }, + { + "id": 5, + "type": "timeseries", + "title": "Spend over time (USD)", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 4 + }, + "fieldConfig": { + "defaults": { + "unit": "currencyUSD", + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "showPoints": "auto", + "spanNulls": false, + "axisPlacement": "auto", + "stacking": { + "mode": "none", + "group": "A" + } + }, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "calcs": [] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ resource.service.name = \"riskkernel\" } | sum_over_time(span.riskkernel.cost.usd)", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + }, + { + "id": 6, + "type": "timeseries", + "title": "Spend by run (USD)", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 4 + }, + "fieldConfig": { + "defaults": { + "unit": "currencyUSD", + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "showPoints": "auto", + "spanNulls": false, + "axisPlacement": "auto", + "stacking": { + "mode": "normal", + "group": "A" + } + }, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "calcs": [] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ resource.service.name = \"riskkernel\" } | sum_over_time(span.riskkernel.cost.usd) by (span.riskkernel.run.id)", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + }, + { + "id": 7, + "type": "timeseries", + "title": "p95 latency by model", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 12 + }, + "fieldConfig": { + "defaults": { + "unit": "ns", + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "showPoints": "auto", + "spanNulls": false, + "axisPlacement": "auto", + "stacking": { + "mode": "none", + "group": "A" + } + }, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "calcs": [] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ span.gen_ai.operation.name = \"chat\" } | quantile_over_time(duration, .95) by (span.gen_ai.request.model)", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + }, + { + "id": 8, + "type": "timeseries", + "title": "Model-call rate by model", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 12 + }, + "fieldConfig": { + "defaults": { + "unit": "reqps", + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "showPoints": "auto", + "spanNulls": false, + "axisPlacement": "auto", + "stacking": { + "mode": "none", + "group": "A" + } + }, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "calcs": [] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ span.gen_ai.operation.name = \"chat\" } | rate() by (span.gen_ai.request.model)", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + }, + { + "id": 9, + "type": "timeseries", + "title": "Tool calls by outcome", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 20 + }, + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "showPoints": "auto", + "spanNulls": false, + "axisPlacement": "auto", + "stacking": { + "mode": "normal", + "group": "A" + } + }, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "calcs": [] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ span.gen_ai.operation.name = \"execute_tool\" } | count_over_time() by (span.riskkernel.tool.status)", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + }, + { + "id": 10, + "type": "timeseries", + "title": "Budget halts by reason", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 20 + }, + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "showPoints": "auto", + "spanNulls": false, + "axisPlacement": "auto", + "stacking": { + "mode": "normal", + "group": "A" + } + }, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "calcs": [] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ span.riskkernel.halt.reason != \"\" } | count_over_time() by (span.riskkernel.halt.reason)", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + } + ], + "refresh": "30s", + "schemaVersion": 39, + "tags": [ + "riskkernel", + "agents" + ], + "templating": { + "list": [] + }, + "time": { + "from": "now-1h", + "to": "now" + }, + "timepicker": {}, + "timezone": "", + "title": "RiskKernel \u2014 agent runs", + "uid": "riskkernel-agent-runs", + "version": 1, + "weekStart": "" +} diff --git a/examples/otel/grafana/docker-compose.yaml b/examples/otel/grafana/docker-compose.yaml new file mode 100644 index 0000000..a69182e --- /dev/null +++ b/examples/otel/grafana/docker-compose.yaml @@ -0,0 +1,45 @@ +# A local Grafana + Tempo stack with a ready-made RiskKernel dashboard. +# +# Unlike the Jaeger quick-look one directory up (which is for eyeballing individual +# traces), this turns RiskKernel's spans into cost and governance *panels* — spend +# per run, token burn, budget halts, tool-call outcomes, latency by model — using +# Tempo's TraceQL metrics. The dashboard is provisioned, so it's import-free. +# +# docker compose -f examples/otel/grafana/docker-compose.yaml up -d +# OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \ +# OTEL_EXPORTER_OTLP_PROTOCOL=grpc \ +# ANTHROPIC_API_KEY=sk-ant-... \ +# riskkernel serve +# +# Then drive some governed traffic and open: +# +# http://localhost:3000 → Dashboards → "RiskKernel — agent runs" +# +# RiskKernel emits NOTHING unless OTEL_EXPORTER_OTLP_ENDPOINT is set. See SECURITY.md. + +services: + tempo: + image: grafana/tempo:2.8.1 + command: ["-config.file=/etc/tempo.yaml"] + volumes: + - ./tempo.yaml:/etc/tempo.yaml:ro + ports: + - "4317:4317" # OTLP gRPC — point OTEL_EXPORTER_OTLP_ENDPOINT here + - "4318:4318" # OTLP HTTP + - "3200:3200" # Tempo API (used by Grafana; handy for ad-hoc queries) + + grafana: + image: grafana/grafana:11.3.0 + depends_on: [tempo] + environment: + # Local example only: open, login-free Grafana. Do NOT copy this auth posture + # to anything reachable by others. + - GF_AUTH_ANONYMOUS_ENABLED=true + - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin + - GF_AUTH_DISABLE_LOGIN_FORM=true + - GF_FEATURE_TOGGLES_ENABLE=traceqlEditor + ports: + - "3000:3000" # Grafana UI + volumes: + - ./grafana/provisioning:/etc/grafana/provisioning:ro + - ./dashboards:/var/lib/grafana/dashboards:ro diff --git a/examples/otel/grafana/grafana/provisioning/dashboards/provider.yml b/examples/otel/grafana/grafana/provisioning/dashboards/provider.yml new file mode 100644 index 0000000..db74e26 --- /dev/null +++ b/examples/otel/grafana/grafana/provisioning/dashboards/provider.yml @@ -0,0 +1,10 @@ +# Load every dashboard JSON mounted at /var/lib/grafana/dashboards. +apiVersion: 1 + +providers: + - name: RiskKernel + type: file + allowUiUpdates: true + options: + path: /var/lib/grafana/dashboards + foldersFromFilesStructure: false diff --git a/examples/otel/grafana/grafana/provisioning/datasources/tempo.yml b/examples/otel/grafana/grafana/provisioning/datasources/tempo.yml new file mode 100644 index 0000000..8fe9cf4 --- /dev/null +++ b/examples/otel/grafana/grafana/provisioning/datasources/tempo.yml @@ -0,0 +1,12 @@ +# Auto-provision the Tempo data source so the dashboard works on first load. +apiVersion: 1 + +datasources: + - name: Tempo + type: tempo + uid: tempo + access: proxy + url: http://tempo:3200 + jsonData: + streamingEnabled: + search: true diff --git a/examples/otel/grafana/tempo.yaml b/examples/otel/grafana/tempo.yaml new file mode 100644 index 0000000..609d1bb --- /dev/null +++ b/examples/otel/grafana/tempo.yaml @@ -0,0 +1,52 @@ +# Minimal single-binary Tempo with TraceQL metrics enabled. +# +# TraceQL metrics (sum_over_time / count_over_time / quantile_over_time over span +# attributes) is what powers the dashboard's cost and governance panels. It needs +# the metrics_generator's `local-blocks` processor turned on — that's the only +# non-default piece here. +stream_over_http_enabled: true + +server: + http_listen_port: 3200 + +# Receive RiskKernel's OTLP spans directly (no collector in the middle). +distributor: + receivers: + otlp: + protocols: + grpc: + endpoint: "0.0.0.0:4317" + http: + endpoint: "0.0.0.0:4318" + +ingester: + max_block_duration: 5m # cut blocks quickly so spans are queryable fast + +# TraceQL metrics queries read from the local-blocks processor. +query_frontend: + metrics: + max_duration: 200h + +metrics_generator: + storage: + path: /var/tempo/generator/wal + traces_storage: + path: /var/tempo/generator/traces + processor: + local_blocks: + filter_server_spans: false # keep client/internal spans (our model + tool spans) + flush_to_storage: true # persist so metrics queries see more than the live window + +storage: + trace: + backend: local + wal: + path: /var/tempo/wal + local: + path: /var/tempo/blocks + +# Enable the local-blocks processor for the (single, default) tenant. +overrides: + defaults: + metrics_generator: + processors: [local-blocks]