docs(otel): add a provisioned Grafana + Tempo cost & governance dashboard - #75
Merged
Conversation
…oard The OTel export already carries cost, budget headroom, halt reason, and tool-call outcomes on every span, but seeing them meant writing TraceQL by hand. This ships the view ready-made: a Grafana + Tempo compose with the data source and a RiskKernel dashboard both provisioned, so it's `docker compose up` with no import step. Ten panels, all built from the spans RiskKernel already emits (Tempo TraceQL metrics — no extra instrumentation): total spend / tokens / model calls / tool calls refused, spend over time and per run, p95 latency and call rate by model, tool calls by outcome, and budget halts by reason. The dashboard JSON is plain Grafana and works against any Tempo with TraceQL metrics enabled (the local-blocks processor), not just this compose.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A ready-made cost & governance dashboard
The OTel export already puts cost, budget headroom, halt reason, and tool-call outcomes on every span — but to actually see them you had to write TraceQL by hand. This ships that view ready-made.
examples/otel/grafana/is a Grafana + Tempo compose with the Tempo data source and a RiskKernel dashboard both provisioned:docker compose up, point RiskKernel'sOTEL_EXPORTER_OTLP_ENDPOINTat it, and the dashboard is already there. No import step.Ten panels, all from spans RiskKernel already emits (Tempo TraceQL metrics — no extra instrumentation, no metrics pipeline):
The dashboard is plain Grafana JSON and works against any Tempo with TraceQL metrics enabled (the
local-blocksprocessor), not just this compose.Verified
Brought the stack up and drove real spans through RiskKernel's own
otelemitter into Tempo, then confirmed all ten panel queries return correctly-grouped data through the same data-source path the Grafana panels use (/api/metrics/query_range): spend split across runs, tool calls across all four statuses, halts by both reasons, latency per model. Tempo boots cleanly with TraceQL metrics enabled; the data source and dashboard auto-provision (the dashboard lists with all 10 panels). Needs Tempo 2.7+ forsum_over_time— noted in the example README.