Whodunnit

The detective for your AI agents. Track: AI & Agent Observability.

Your agents said they succeeded. They were lying.

When a multi-agent LLM system fails, it usually does not crash. There is no stack trace and no red in any dashboard. The agents hand you a confident, well-written answer that happens to be wrong: a step got skipped, one agent ignored another, the pipeline stopped early. In a normal trace that looks identical to a run that went fine. Same spans, same green checkmarks, similar latency.

Whodunnit catches exactly that: the runs that pass while being wrong. It reconstructs the agent conversation from OpenTelemetry spans in SigNoz, judges it against MAST (a peer-reviewed taxonomy of 14 multi-agent failure modes, Cemri et al., 2025, UC Berkeley), writes a prompt fix, and ships it only after a human approves the diff.

The first time we ran the judge over our own pipeline, it flagged nearly every run: mostly FM-3.2 (No or incorrect verification). The judge was not broken. Our pipeline had no verification step at all. The taxonomy was describing a hole we had designed in from the start. That is the point of this project: a latency graph would never have told us.

How it works

[Multi-agent app (LangGraph) + OTel SDK]
        │  OTLP export (traces + metrics)
        ▼
[SigNoz]  ── ClickHouse + dashboards + query API + MCP
        │  ▲
  pull  │  │ push verdicts back (mast.verdict, tagged trace_id)
        ▼  │
[MAST classifier (Python)]
   - polls SigNoz for completed traces
   - rebuilds the agent transcript from spans
   - LLM-as-a-judge, grounded in the MAST taxonomy
   - emits a verdict, then drafts a prompt fix for human review

Each agent turn is its own span tagged with agent.name, its input/output, and token counts, so the trace is the conversation log. That single decision is what lets the classifier pull a run back out and diagnose it after the fact.

SigNoz surface used

Feature	Where
Traces (OTLP)	Every agent turn is a span with agent.name, I/O, token counts
Metrics (OTLP)	Verdicts pushed back as mast.verdict, tagged trace_id
Query API	Classifier polls /api/v4/query_range, reads spans via /api/v1/traces/{id}
Dashboard	"Whodunnit: Agent Failure Analysis", 4 panels, created via the API
Alerts	v5 rule on mast.verdict: fire when failures spike
MCP server	Deployed by Foundry; lets Claude Code query failures directly

Reproducing the deployment

curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl cast -f casting.yaml

casting.yaml and casting.yaml.lock are committed. Foundry brings up SigNoz and the SigNoz MCP server (spec.mcp.spec.enabled: true). See docs/signoz-mcp.md for the MCP wiring and docs/signoz-selfhost.md for the self-hosted switch.

Layout

backend/
  app/
    agent_app/    LangGraph agents, instrumented with OpenTelemetry
    classifier/   pulls traces from SigNoz, judges them against MAST, pushes verdicts back
    api/          FastAPI backend serving verdicts/stats to the dashboard
    mcp/          SigNoz MCP wiring (lets a coding agent query telemetry)
    types/        shared label sets (the MAST taxonomy)
    utils/        shared paths + OTLP exporter config
  data/           mast_official/ (paper's definitions + examples), prompt_overrides.json, verdicts.db
  requirements.txt
frontend/         Vite + React dashboard (runs, verdicts, failure modes, transcripts)

Run it

The classifier persists verdicts to a local SQLite store (backend/data/verdicts.db), so the dashboard works even before SigNoz is wired up.

# 0. Python env
python -m venv .venv && source .venv/bin/activate
pip install -r backend/requirements.txt

# 1. Env: fill in ANTHROPIC_API_KEY + SIGNOZ_* values
cp .env.example .env

# 2. Bring up SigNoz (Foundry, or self-hosted — see docs/signoz-selfhost.md)
foundryctl cast -f casting.yaml

# 3. Run the instrumented agents (generates traces). Modules live under backend/.
cd backend
python -m app.agent_app.main

# 4. Judge the traces (polls SigNoz, diagnoses, pushes verdicts back)
python -m app.classifier.main

To see it in the UI, start the API and the frontend:

cd backend && python -m uvicorn app.api.main:app --port 8000   # API
cd frontend && npm install && npm run dev                       # UI

• API: http://localhost:8000 (/api/runs, /api/stats, /api/taxonomy)
• Dashboard: http://localhost:5173 (auto-polls the API every 5s)

SigNoz indexes traces ~30 to 60 seconds after a run, so during a demo it is faster to judge on demand than to wait for the poll loop.

Config

All config is via env vars (see .env.example):

• ANTHROPIC_API_KEY — for both the agents and the judge
• OTEL_EXPORTER_OTLP_ENDPOINT — SigNoz OTLP ingest (cloud: https://ingest.<region>.signoz.cloud:443)
• SIGNOZ_INGESTION_KEY — sends telemetry in (SigNoz Cloud)
• SIGNOZ_API_URL — SigNoz query API
• SIGNOZ_API_KEY — reads telemetry back out (needs a viewer/editor/admin role)

A gotcha worth stating: the OTLP HTTP endpoint needs /v1/traces and /v1/metrics appended manually, the SDK does not add them. The ingestion key and the API key are two different things: one sends data in, the other reads it back.

Claude Code ↔ SigNoz MCP

SigNoz ships an MCP server. Point Claude Code at it and the coding agent can query failure data directly ("which traces got an FM-3.2 verdict in the last hour?") and suggest fixes. See docs/signoz-mcp.md.

---

Built for the Agents of SigNoz hackathon by Akarsh Jain and Jahnavi Singh. MAST: arXiv:2503.13657. The taxonomy is the paper's work; the judge that applies it is ours.