From 6a52ca8f0ca2aabb821776769c809ad7819a4acc Mon Sep 17 00:00:00 2001 From: matt butler Date: Thu, 9 Jul 2026 15:19:46 -0400 Subject: [PATCH] docs: manual test runbook for fork-style diagnostic exploration Closes #78. Companion to #67. Documents how to run the coordinator in multi-fork mode (STREAMOPS_AGENT_DIAGNOSTIC_FORKS), what to expect in the logs and the audit trail (with the real emitted strings: fan-out line, per-fork hypothesis spawn lines, merged confidence distribution, cross-fork xf- conflicts, the escalation surfacing), and four concrete ways to prove it actually forked (shared cid with interleaved spawn lines, distinct hypotheses, f{i}: namespaced ids, xf- conflict ids). Includes the single-agent baseline to diff against and pointers to the follow-ups (#91 dynamic hypotheses, #77 cross-cycle change awareness). Also refreshes the README topology: the Monitor->Diagnostic edge now shows the typed DetectedAnomaly handoff (was "full text", stale since #80), the report edge notes it is attributed (#88), and a note + link points to the runbook. --- README.md | 11 +- docs/fork-diagnostics-runbook.md | 169 +++++++++++++++++++++++++++++++ 2 files changed, 178 insertions(+), 2 deletions(-) create mode 100644 docs/fork-diagnostics-runbook.md diff --git a/README.md b/README.md index a93ad72..ba79918 100644 --- a/README.md +++ b/README.md @@ -85,12 +85,19 @@ graph LR REP[Report Agent
Synthesis Only] end - MON -->|"Anomaly context
(full text + schema)"| DIAG + MON -->|"DetectedAnomaly
(typed + schema)"| DIAG DIAG -->|"DiagnosisReport
(claims + sources + conflicts)"| MON MON -->|"DiagnosisReport JSON
(structured, attributed)"| REP - REP -->|"IncidentReport
(severity + actions)"| MON + REP -->|"IncidentReport
(severity + actions, attributed)"| MON ``` +The coordinator can also fan out **multiple diagnostic sub-agents in parallel**, +each seeded with a distinct hypothesis, then merge them (cross-fork +disagreements become escalated conflicts). This is off by default +(`agent_diagnostic_forks = 1`); see the +[fork-diagnostics runbook](docs/fork-diagnostics-runbook.md) for how to enable +and verify it. + ## Agentic Loop The core loop is driven by Claude's `stop_reason`. The agent keeps calling tools until it decides it has enough information. diff --git a/docs/fork-diagnostics-runbook.md b/docs/fork-diagnostics-runbook.md new file mode 100644 index 0000000..731bbef --- /dev/null +++ b/docs/fork-diagnostics-runbook.md @@ -0,0 +1,169 @@ +# Fork-style parallel diagnostic exploration: manual test runbook + +Companion to the fork-diagnostics feature (issue #67). This walks through +running the coordinator in multi-fork mode, what to expect in the logs and +audit trail, and how to prove the diagnosis actually fanned out in parallel +rather than running one agent. + +## What the feature does + +By default the coordinator runs a single diagnostic sub-agent per cycle +(`detect -> diagnose -> report`). One agent is a single line of reasoning, +prone to tunnel vision on an ambiguous anomaly with several plausible causes. + +In multi-fork mode the coordinator spawns **N diagnostic sub-agents +concurrently** from the same detected anomaly, each seeded with a distinct +hypothesis, then **merges the survivors** into one diagnosis. Forks that reach a +different `anomaly_type` than the primary (highest-confidence) fork produce a +cross-fork `ConflictRecord` that is surfaced and escalated, never silently +resolved. + +Fork count is config-gated. Default (`agent_diagnostic_forks = 1`) preserves +single-agent behavior, so nothing changes until you opt in. + +## How to run it locally + +Prerequisites (see the main README): Docker Compose stack up, Flink job +submitted, Java simulator JAR built. + +1. Bring up the stack: + + ``` + docker compose up -d + ``` + +2. Enable fan-out via the environment (the knob is `agent_diagnostic_forks`, + overridden with the `STREAMOPS_` prefix). 3 forks is a good demo value; the + cap is the number of defined hypotheses (currently 4): + + ``` + export STREAMOPS_AGENT_DIAGNOSTIC_FORKS=3 + ``` + +3. Trigger a cycle. Either inject a scenario end-to-end with the demo runner: + + ``` + python scripts/demo_scenario.py latency-spike + ``` + + or run the agent directly for one cycle against whatever the stack is + currently emitting: + + ``` + cd mcp-server + STREAMOPS_AGENT_DIAGNOSTIC_FORKS=3 uv run python -m streamops_mcp.agent.main --single-cycle + ``` + +To get the **single-agent baseline** to diff against, run the same command with +`STREAMOPS_AGENT_DIAGNOSTIC_FORKS=1` (or unset). + +## What to expect in the logs + +Every line from one cycle shares a correlation id (`cid=cyc-...`, see the +correlated-logs feature, issue #84). The log format is: + +``` +