From 5b9c0093e4fcb7ac75dc76356b9edc6a2f561adb Mon Sep 17 00:00:00 2001 From: Adarsh Prashar Date: Sat, 6 Jun 2026 18:36:48 +0530 Subject: [PATCH] docs(otel): add a provisioned Grafana + Tempo cost & governance dashboard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The OTel export already carries cost, budget headroom, halt reason, and tool-call outcomes on every span, but seeing them meant writing TraceQL by hand. This ships the view ready-made: a Grafana + Tempo compose with the data source and a RiskKernel dashboard both provisioned, so it's `docker compose up` with no import step. Ten panels, all built from the spans RiskKernel already emits (Tempo TraceQL metrics — no extra instrumentation): total spend / tokens / model calls / tool calls refused, spend over time and per run, p95 latency and call rate by model, tool calls by outcome, and budget halts by reason. The dashboard JSON is plain Grafana and works against any Tempo with TraceQL metrics enabled (the local-blocks processor), not just this compose. --- CHANGELOG.md | 5 + examples/otel/README.md | 6 +- examples/otel/grafana/README.md | 64 ++ .../dashboards/riskkernel-agent-runs.json | 660 ++++++++++++++++++ examples/otel/grafana/docker-compose.yaml | 45 ++ .../provisioning/dashboards/provider.yml | 10 + .../provisioning/datasources/tempo.yml | 12 + examples/otel/grafana/tempo.yaml | 52 ++ 8 files changed, 853 insertions(+), 1 deletion(-) create mode 100644 examples/otel/grafana/README.md create mode 100644 examples/otel/grafana/dashboards/riskkernel-agent-runs.json create mode 100644 examples/otel/grafana/docker-compose.yaml create mode 100644 examples/otel/grafana/grafana/provisioning/dashboards/provider.yml create mode 100644 examples/otel/grafana/grafana/provisioning/datasources/tempo.yml create mode 100644 examples/otel/grafana/tempo.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index a92b987..274be01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,11 @@ surface is governed by [`COMPATIBILITY.md`](COMPATIBILITY.md). blocks and approval denials are now visible in whatever OTLP backend you already run — a refused call is marked with an error span status so it stands out. See [`api/v1/otel-genai.md`](api/v1/otel-genai.md) and [`examples/otel`](examples/otel). +- **A ready-made cost & governance dashboard** ([`examples/otel/grafana`](examples/otel/grafana)) — + a provisioned Grafana + Tempo stack that turns RiskKernel's spans into panels: + spend over time and per run, token burn, budget halts by reason, tool-call outcomes, + and p95 latency by model. Built from the spans you already emit (Tempo TraceQL + metrics — no extra instrumentation); `docker compose up`, no import step. ## [0.3.0] - 2026-06-06 diff --git a/examples/otel/README.md b/examples/otel/README.md index 42adc81..a16c6ea 100644 --- a/examples/otel/README.md +++ b/examples/otel/README.md @@ -59,7 +59,11 @@ backend prefers OTLP/HTTP. ## Building cost/usage dashboards -Because cost and budget live on every span as first-class attributes, you can build +**Want it ready-made?** [`grafana/`](grafana/) ships a provisioned Grafana + Tempo +stack with a RiskKernel dashboard (spend, halts, tool outcomes, latency) — one +`docker compose up`, no import step. + +Because cost and budget live on every span as first-class attributes, you can also build panels directly from spans (e.g. in Grafana over Tempo, or SigNoz): - **Spend per run** — sum `riskkernel.cost.usd` grouped by `riskkernel.run.id`. diff --git a/examples/otel/grafana/README.md b/examples/otel/grafana/README.md new file mode 100644 index 0000000..d57977b --- /dev/null +++ b/examples/otel/grafana/README.md @@ -0,0 +1,64 @@ +# Cost & governance dashboard (Grafana + Tempo) + +A ready-made Grafana dashboard for your governed runs — **spend, token burn, +budget halts, tool-call outcomes, and latency by model** — built entirely from the +OpenTelemetry spans RiskKernel already emits. No extra instrumentation, no metrics +pipeline: [Grafana Tempo](https://grafana.com/oss/tempo/) aggregates the spans with +TraceQL metrics, and the dashboard is provisioned so it shows up on first load. + +> The [Jaeger quick-look](../docker-compose.yaml) one directory up is for reading +> individual traces. This is the aggregate view — the panels a platform team watches. + +![panels: total spend, output tokens, model calls, tool calls refused, spend over +time, spend by run, p95 latency by model, model-call rate, tool calls by outcome, +budget halts by reason] + +## Run it + +```bash +# 1. Bring up Tempo (OTLP in) + Grafana (dashboard provisioned) +docker compose -f examples/otel/grafana/docker-compose.yaml up -d + +# 2. Point RiskKernel at Tempo and start it (your keys, as always) +export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 +export OTEL_EXPORTER_OTLP_PROTOCOL=grpc +export ANTHROPIC_API_KEY=sk-ant-... +riskkernel serve + +# 3. Drive some governed traffic — point your app's base URL at the proxy: +export OPENAI_BASE_URL=http://localhost:7070/v1 +# (or run any example under ../../ — the loop-killer, the MCP demo, etc.) +``` + +Open **http://localhost:3000** → **Dashboards** → **RiskKernel — agent runs**. +Spend, halts, and tool outcomes fill in as runs execute (default range: last 1h). + +> **No telemetry by default.** RiskKernel exports nothing unless +> `OTEL_EXPORTER_OTLP_ENDPOINT` is set; spans go only to the Tempo you run here. See +> [`SECURITY.md`](../../../SECURITY.md). The Grafana in this stack is login-free for +> convenience — don't expose it as-is. + +## What each panel is + +| Panel | TraceQL metric | +|---|---| +| Total spend / Output tokens / Model calls | `sum_over_time(span.riskkernel.cost.usd)`, `sum_over_time(span.gen_ai.usage.output_tokens)`, `count_over_time()` | +| **Tool calls refused** | `count_over_time()` over `execute_tool` spans where `riskkernel.tool.status != "approved"` | +| Spend over time / Spend by run | `sum_over_time(span.riskkernel.cost.usd)`, grouped `by (span.riskkernel.run.id)` | +| p95 latency by model | `quantile_over_time(duration, .95) by (span.gen_ai.request.model)` | +| Model-call rate by model | `rate() by (span.gen_ai.request.model)` | +| Tool calls by outcome | `count_over_time() by (span.riskkernel.tool.status)` (approved / blocked / denied / timeout) | +| Budget halts by reason | `count_over_time() by (span.riskkernel.halt.reason)` | + +Every attribute these queries touch is pinned in +[`api/v1/otel-genai.md`](../../../api/v1/otel-genai.md). + +## Using it against your own Grafana + +The dashboard is a plain Grafana JSON +([`dashboards/riskkernel-agent-runs.json`](dashboards/riskkernel-agent-runs.json)). +To use your existing stack instead of this compose: import the JSON and point its +**Tempo** data source at any Tempo with +[TraceQL metrics enabled](https://grafana.com/docs/tempo/latest/metrics-from-traces/) +(the `local-blocks` processor — see [`tempo.yaml`](tempo.yaml)). Needs Grafana +10.4+ and Tempo 2.7+ (the dashboard uses `sum_over_time`). diff --git a/examples/otel/grafana/dashboards/riskkernel-agent-runs.json b/examples/otel/grafana/dashboards/riskkernel-agent-runs.json new file mode 100644 index 0000000..5ec3dd0 --- /dev/null +++ b/examples/otel/grafana/dashboards/riskkernel-agent-runs.json @@ -0,0 +1,660 @@ +{ + "annotations": { + "list": [] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 1, + "links": [], + "panels": [ + { + "id": 1, + "type": "stat", + "title": "Total spend", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 0, + "y": 0 + }, + "fieldConfig": { + "defaults": { + "unit": "currencyUSD", + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": null + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "sum" + ], + "fields": "", + "values": false + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto" + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ resource.service.name = \"riskkernel\" } | sum_over_time(span.riskkernel.cost.usd)", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + }, + { + "id": 2, + "type": "stat", + "title": "Output tokens", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 6, + "y": 0 + }, + "fieldConfig": { + "defaults": { + "unit": "short", + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": null + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "sum" + ], + "fields": "", + "values": false + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto" + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ resource.service.name = \"riskkernel\" } | sum_over_time(span.gen_ai.usage.output_tokens)", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + }, + { + "id": 3, + "type": "stat", + "title": "Model calls", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 12, + "y": 0 + }, + "fieldConfig": { + "defaults": { + "unit": "short", + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": null + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "sum" + ], + "fields": "", + "values": false + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto" + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ span.gen_ai.operation.name = \"chat\" } | count_over_time()", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + }, + { + "id": 4, + "type": "stat", + "title": "Tool calls refused", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 18, + "y": 0 + }, + "fieldConfig": { + "defaults": { + "unit": "short", + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1 + } + ] + } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "sum" + ], + "fields": "", + "values": false + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto" + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ span.gen_ai.operation.name = \"execute_tool\" && span.riskkernel.tool.status != \"approved\" } | count_over_time()", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + }, + { + "id": 5, + "type": "timeseries", + "title": "Spend over time (USD)", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 4 + }, + "fieldConfig": { + "defaults": { + "unit": "currencyUSD", + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "showPoints": "auto", + "spanNulls": false, + "axisPlacement": "auto", + "stacking": { + "mode": "none", + "group": "A" + } + }, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "calcs": [] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ resource.service.name = \"riskkernel\" } | sum_over_time(span.riskkernel.cost.usd)", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + }, + { + "id": 6, + "type": "timeseries", + "title": "Spend by run (USD)", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 4 + }, + "fieldConfig": { + "defaults": { + "unit": "currencyUSD", + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "showPoints": "auto", + "spanNulls": false, + "axisPlacement": "auto", + "stacking": { + "mode": "normal", + "group": "A" + } + }, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "calcs": [] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ resource.service.name = \"riskkernel\" } | sum_over_time(span.riskkernel.cost.usd) by (span.riskkernel.run.id)", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + }, + { + "id": 7, + "type": "timeseries", + "title": "p95 latency by model", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 12 + }, + "fieldConfig": { + "defaults": { + "unit": "ns", + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "showPoints": "auto", + "spanNulls": false, + "axisPlacement": "auto", + "stacking": { + "mode": "none", + "group": "A" + } + }, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "calcs": [] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ span.gen_ai.operation.name = \"chat\" } | quantile_over_time(duration, .95) by (span.gen_ai.request.model)", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + }, + { + "id": 8, + "type": "timeseries", + "title": "Model-call rate by model", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 12 + }, + "fieldConfig": { + "defaults": { + "unit": "reqps", + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "showPoints": "auto", + "spanNulls": false, + "axisPlacement": "auto", + "stacking": { + "mode": "none", + "group": "A" + } + }, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "calcs": [] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ span.gen_ai.operation.name = \"chat\" } | rate() by (span.gen_ai.request.model)", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + }, + { + "id": 9, + "type": "timeseries", + "title": "Tool calls by outcome", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 20 + }, + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "showPoints": "auto", + "spanNulls": false, + "axisPlacement": "auto", + "stacking": { + "mode": "normal", + "group": "A" + } + }, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "calcs": [] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ span.gen_ai.operation.name = \"execute_tool\" } | count_over_time() by (span.riskkernel.tool.status)", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + }, + { + "id": 10, + "type": "timeseries", + "title": "Budget halts by reason", + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 20 + }, + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 10, + "showPoints": "auto", + "spanNulls": false, + "axisPlacement": "auto", + "stacking": { + "mode": "normal", + "group": "A" + } + }, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "calcs": [] + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "tempo", + "uid": "tempo" + }, + "queryType": "traceql", + "query": "{ span.riskkernel.halt.reason != \"\" } | count_over_time() by (span.riskkernel.halt.reason)", + "refId": "A", + "filters": [], + "metricsQueryType": "range", + "tableType": "traces", + "limit": 20 + } + ] + } + ], + "refresh": "30s", + "schemaVersion": 39, + "tags": [ + "riskkernel", + "agents" + ], + "templating": { + "list": [] + }, + "time": { + "from": "now-1h", + "to": "now" + }, + "timepicker": {}, + "timezone": "", + "title": "RiskKernel \u2014 agent runs", + "uid": "riskkernel-agent-runs", + "version": 1, + "weekStart": "" +} diff --git a/examples/otel/grafana/docker-compose.yaml b/examples/otel/grafana/docker-compose.yaml new file mode 100644 index 0000000..a69182e --- /dev/null +++ b/examples/otel/grafana/docker-compose.yaml @@ -0,0 +1,45 @@ +# A local Grafana + Tempo stack with a ready-made RiskKernel dashboard. +# +# Unlike the Jaeger quick-look one directory up (which is for eyeballing individual +# traces), this turns RiskKernel's spans into cost and governance *panels* — spend +# per run, token burn, budget halts, tool-call outcomes, latency by model — using +# Tempo's TraceQL metrics. The dashboard is provisioned, so it's import-free. +# +# docker compose -f examples/otel/grafana/docker-compose.yaml up -d +# OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \ +# OTEL_EXPORTER_OTLP_PROTOCOL=grpc \ +# ANTHROPIC_API_KEY=sk-ant-... \ +# riskkernel serve +# +# Then drive some governed traffic and open: +# +# http://localhost:3000 → Dashboards → "RiskKernel — agent runs" +# +# RiskKernel emits NOTHING unless OTEL_EXPORTER_OTLP_ENDPOINT is set. See SECURITY.md. + +services: + tempo: + image: grafana/tempo:2.8.1 + command: ["-config.file=/etc/tempo.yaml"] + volumes: + - ./tempo.yaml:/etc/tempo.yaml:ro + ports: + - "4317:4317" # OTLP gRPC — point OTEL_EXPORTER_OTLP_ENDPOINT here + - "4318:4318" # OTLP HTTP + - "3200:3200" # Tempo API (used by Grafana; handy for ad-hoc queries) + + grafana: + image: grafana/grafana:11.3.0 + depends_on: [tempo] + environment: + # Local example only: open, login-free Grafana. Do NOT copy this auth posture + # to anything reachable by others. + - GF_AUTH_ANONYMOUS_ENABLED=true + - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin + - GF_AUTH_DISABLE_LOGIN_FORM=true + - GF_FEATURE_TOGGLES_ENABLE=traceqlEditor + ports: + - "3000:3000" # Grafana UI + volumes: + - ./grafana/provisioning:/etc/grafana/provisioning:ro + - ./dashboards:/var/lib/grafana/dashboards:ro diff --git a/examples/otel/grafana/grafana/provisioning/dashboards/provider.yml b/examples/otel/grafana/grafana/provisioning/dashboards/provider.yml new file mode 100644 index 0000000..db74e26 --- /dev/null +++ b/examples/otel/grafana/grafana/provisioning/dashboards/provider.yml @@ -0,0 +1,10 @@ +# Load every dashboard JSON mounted at /var/lib/grafana/dashboards. +apiVersion: 1 + +providers: + - name: RiskKernel + type: file + allowUiUpdates: true + options: + path: /var/lib/grafana/dashboards + foldersFromFilesStructure: false diff --git a/examples/otel/grafana/grafana/provisioning/datasources/tempo.yml b/examples/otel/grafana/grafana/provisioning/datasources/tempo.yml new file mode 100644 index 0000000..8fe9cf4 --- /dev/null +++ b/examples/otel/grafana/grafana/provisioning/datasources/tempo.yml @@ -0,0 +1,12 @@ +# Auto-provision the Tempo data source so the dashboard works on first load. +apiVersion: 1 + +datasources: + - name: Tempo + type: tempo + uid: tempo + access: proxy + url: http://tempo:3200 + jsonData: + streamingEnabled: + search: true diff --git a/examples/otel/grafana/tempo.yaml b/examples/otel/grafana/tempo.yaml new file mode 100644 index 0000000..609d1bb --- /dev/null +++ b/examples/otel/grafana/tempo.yaml @@ -0,0 +1,52 @@ +# Minimal single-binary Tempo with TraceQL metrics enabled. +# +# TraceQL metrics (sum_over_time / count_over_time / quantile_over_time over span +# attributes) is what powers the dashboard's cost and governance panels. It needs +# the metrics_generator's `local-blocks` processor turned on — that's the only +# non-default piece here. +stream_over_http_enabled: true + +server: + http_listen_port: 3200 + +# Receive RiskKernel's OTLP spans directly (no collector in the middle). +distributor: + receivers: + otlp: + protocols: + grpc: + endpoint: "0.0.0.0:4317" + http: + endpoint: "0.0.0.0:4318" + +ingester: + max_block_duration: 5m # cut blocks quickly so spans are queryable fast + +# TraceQL metrics queries read from the local-blocks processor. +query_frontend: + metrics: + max_duration: 200h + +metrics_generator: + storage: + path: /var/tempo/generator/wal + traces_storage: + path: /var/tempo/generator/traces + processor: + local_blocks: + filter_server_spans: false # keep client/internal spans (our model + tool spans) + flush_to_storage: true # persist so metrics queries see more than the live window + +storage: + trace: + backend: local + wal: + path: /var/tempo/wal + local: + path: /var/tempo/blocks + +# Enable the local-blocks processor for the (single, default) tenant. +overrides: + defaults: + metrics_generator: + processors: [local-blocks]