[Node A] Supervisor, State Machine & Reaper Protocol — Full 6-Month Track (M. Yadav)

[mohityadav8]

Owner: M. Yadav · Project Lead / Systems Architect
Hardware: RTX 3050 · 6GB VRAM (Phi-3-Mini 4-bit supervisor model)
Scope: Orchestration, state machine, convergence detection, Reaper Protocol, cluster networking, WebSocket event backend
Duration: 6 Phases / ~26 weeks
Labels: node-a, orchestration, epic, reaper-protocol

---

🎯 Epic Goal

Own the "brain" of HydraNet — the state machine that drives every agent run, the Reaper Protocol that detects and kills dead-end code-fix loops, and the gRPC/WebSocket networking layer that ties Node B (inference) and Node C (sandbox) into one coordinated mesh. This is the single feature (Reaper) that differentiates HydraNet from every other local coding agent — it must work convincingly before anything else matters.

📅 Timeline (6 Months)

gantt
    title Node A — M. Yadav — 6 Month Roadmap
    dateFormat  YYYY-MM-DD
    axisFormat  %b
    section Phase I - Single Node
    Monorepo + State Machine (5 states)   :p1, 2026-07-06, 21d
    section Phase II - Reaper Protocol
    Convergence Scoring + Reaper Hook      :p2, after p1, 21d
    section Phase III - AST Retrieval
    pgvector Graph Store + Query Layer     :p3, after p2, 21d
    section Phase IV - Three-Node Mesh
    gRPC Contracts + Supervisor Router     :p4, after p3, 21d
    section Phase V - War Room Dashboard
    WebSocket Event Server + Auth          :p5, after p4, 21d
    section Phase VI - Benchmark & Ship
    Hardening + Soak Test + Release        :p6, after p5, 21d

Loading

🧭 Definition of Done (Epic-level)

• All 6 phase demo milestones below are recorded and linked in this issue
• hydra fix runs end-to-end across a real 3-laptop cluster without manual intervention
• Reaper Protocol fires correctly on seeded death-spiral bugs with < 5% false-positive rate
• v0.1.0 tagged and pushed to AnthropicBots/HydraNet

---

🟦 PHASE I — Single-Node Agent Loop

Demo Milestone: CLI fixes a real Python bug, end-to-end, on one laptop inside Docker.

• 01. Scaffold monorepo — uv workspaces, packages: orchestrator, inference, sandbox, dashboard, shared; strict pyproject.toml (no floating versions, ruff + mypy configured)
  • Acceptance: uv sync installs clean from a fresh clone; ruff check . and mypy . pass with zero errors on scaffold
• 02. Build core 5-state machine — PLAN → IMPLEMENT → TEST → REFLECT → DONE/ABORT using LangGraph
  • Acceptance: unit test asserts every legal transition; illegal transitions raise a typed exception, not a silent no-op
• 03. Typer CLI entrypoint — hydra fix --repo PATH --issue "description"
  • Acceptance: --help auto-generates from type hints; missing --repo fails fast with a clear error, not a stack trace
• 04. Agent tool set (Phase I scope) — read_file, write_file, list_dir, run_shell
  • Acceptance: every tool call and result is logged with timestamp + duration; write_file refuses paths outside the mounted repo slice
• 05. State transition router — maps tool outputs → next state
  • Acceptance: router is a pure function (state, tool_output) → next_state, unit-testable without any live model or sandbox
• 06. RunLogger — append every state, tool call, output, timestamp to a JSONL file
  • Acceptance: a killed process mid-run still leaves a valid, replayable JSONL (flush after every line, not buffered)
• 07. Hard execution budget — max N steps, max M tokens, abort with clear log entry on breach
  • Acceptance: budget breach produces a distinct ABORT_BUDGET state, not a generic crash
• 08. Config system — hydra.config.json (model endpoints, timeouts, retry limits, budget)
  • Acceptance: invalid config fails validation at startup with a field-level error message, not at runtime mid-fix
• 09. Integration test — seed a known bug in a toy repo, run the CLI, assert the file changed correctly
  • Acceptance: test runs in CI, no external network calls, uses a local toy repo fixture
• 10. Phase I demo — record terminal showing CLI fixing a real bug in a real small repo
  • Acceptance: video uploaded, linked in this issue, includes final git diff on screen

---

🟥 PHASE II — The Reaper Protocol (core feature)

Demo Milestone: Side-by-side recording — baseline agent looping 20+ times vs. HydraNet backtracking after 3 and solving it.

• 01. ConvergenceDetector interface — inputs: diff history, command history, stderr history → output: score 0–1
  • Acceptance: interface documented with a docstring example; implementation swappable behind the interface
• 02. patch_similarity() — normalized Levenshtein edit distance between last two diffs (on changed line sets)
  • Acceptance: identical diffs → 1.0, completely unrelated diffs → ~0.0, unit tests cover both extremes
• 03. command_repetition_rate() — sliding window over last k shell commands, compute repetition fraction
  • Acceptance: k configurable via hydra.config.json, defaults to 3 as per README spec
• 04. error_signature_match() — hash dominant error type + file from stderr, compare across attempts
  • Acceptance: two different stack traces pointing to the same file+line hash identically
• 05. Weighted convergence_score aggregation — configurable α, β, γ weights
  • Acceptance: α·patch_similarity + β·command_repetition_rate + γ·error_signature_match, weights sum to 1, all in config
• 06. ReaperHook — fires when score > threshold (default 0.75), halts branch, emits REAPER_FIRED event
  • Acceptance: threshold configurable; event includes score, attempt number, and triggering diff
• 07. BranchManager — creates a git branch per approach (hydra/attempt-001), tracks genealogy tree as JSON
  • Acceptance: genealogy JSON is queryable and matches the exact tree shown in README's genealogy example
• 08. ReaperNotifier — publishes REAPER_FIRED to the WebSocket stream for the dashboard
  • Acceptance: event delivered within 200ms of the Reaper firing, verified in integration test
• 09. SpiralSimulator test harness — feed the agent a bug with no solution at the file it's looking at; verify Reaper fires at the right threshold
  • Acceptance: harness is reusable for regression testing every future change to convergence scoring
• 10. Phase II demo — side-by-side recording: baseline looping 20+ times vs. HydraNet backtracking after 3 attempts and solving it
  • Acceptance: both runs use the identical seeded bug and identical model; attempt counts and token spend both shown on screen

---

🟩 PHASE III — AST Graph Retrieval

Demo Milestone: One bug that flat search misses, graph walk finds — show the exact query path through the graph.

• 01. Deploy pgvector on Node A Postgres 16 — write schema migration
  • Acceptance: migration is idempotent, runnable via hydra cluster init
• 02. Design graph schema — code_nodes(id, file, type, name, signature, embedding), code_edges(src, dst, edge_type)
  • Acceptance: schema diagram committed to /docs/architecture.md; edge_type enum covers import, call, inherit
• 03. IndexManager — on hydra fix, scan repo, parse every .py file, embed chunks, upsert into pgvector
  • Acceptance: re-running on an unchanged repo is a no-op (see incremental indexing, task 04)
• 04. Incremental indexing — hash each file's content, only re-embed changed files since last run
  • Acceptance: benchmark shows re-index of a 1k-file repo with 1 changed file completes in < 2 seconds
• 05. VectorCacheLayer — LRU cache on the 100 most-recently-retrieved code chunks
  • Acceptance: cache hit avoids a DB round-trip, verified via query count assertion in tests
• 06. RetrievalAPI endpoint — retrieve(error_context, k=10) → list[CodeChunk], called by the orchestrator
  • Acceptance: endpoint returns results ranked, with each chunk's file path and hop-distance from the error
• 07. RetrievalEval harness — 20 seeded bugs, measure recall (did root-cause file appear in top-10?)
  • Acceptance: recall number is written to a report file, not just printed to console
• 08. Stress test — index a 10k-file open-source repo (e.g. Django), measure indexing time and query latency
  • Acceptance: numbers documented in /docs/model-selection.md or equivalent, with hardware specs noted
• 09. IndexHealth CLI — hydra index-status --repo PATH shows stale files, coverage %, last-indexed timestamp
  • Acceptance: command works without needing to run a full hydra fix first
• 10. Phase III demo — one bug flat search misses, graph walk finds; show exact query path through the graph
  • Acceptance: side-by-side output of both retrieval methods on the same bug, recorded

---

🟨 PHASE IV — Three-Node Mesh

Demo Milestone: Phase I CLI command runs across all 3 physical laptops over WiFi.

• 01. gRPC proto definitions — OrchestratorService, InferenceService, SandboxService; commit as v1 contract
  • Acceptance: .proto files reviewed and tagged v1 before any implementation starts against them
• 02. SupervisorRouter — dispatch CodeTask → Node B, TestTask → Node C, based on task type
  • Acceptance: routing logic is table-driven, unit-testable without live gRPC connections
• 03. HeartbeatMonitor — ping each worker every 5s, mark UNHEALTHY after 3 missed beats
  • Acceptance: integration test simulates a dropped worker and confirms detection within 15s
• 04. Worker failover — Node B goes UNHEALTHY mid-task → emit NODE_LOST, pause execution, wait for reconnect
  • Acceptance: no partial/corrupted state written during a pause; run resumes cleanly on reconnect
• 05. NodeRegistry — Node B/C announce via hydra join --host IP --role on startup
  • Acceptance: registry persists across a supervisor restart (not purely in-memory)
• 06. ResourceMatrix — supervisor tracks Node B VRAM usage + Node C container count, routes on capacity
  • Acceptance: routing avoids dispatching to a node reporting > 90% resource utilization
• 07. EventBroadcaster — all significant events (task dispatched, Reaper fired, test passed) → WebSocket stream
  • Acceptance: every event type from Phase V's schema is emitted at least once in an integration test
• 08. ClusterHealthCLI — hydra cluster status shows all connected nodes, load, last heartbeat
  • Acceptance: output matches the exact format shown in the README's "Getting Started" section
• 09. Network load test — sustain a 2-hour run on a real repo, measure packet loss, latency jitter, recovery from WiFi dropout
  • Acceptance: results documented with a graph of latency over time
• 10. Phase IV demo — three laptops on a table, one hydra fix command, all three terminals active, screen recorded
  • Acceptance: video shows all 3 terminals simultaneously and the final successful fix

---

🟪 PHASE V — War Room Dashboard

Demo Milestone: Live UI showing genealogy tree, node meters, state stream.

• 01. WebSocket event server on Node A — one connection per client, one event stream per active run
  • Acceptance: server handles 2+ concurrent client connections without cross-talk between runs
• 02. Full event schema — TASK_DISPATCHED, AGENT_THOUGHT, TOOL_CALLED, TOOL_RESULT, REAPER_FIRED, BRANCH_CREATED, TEST_PASSED, RUN_COMPLETE
  • Acceptance: schema versioned and documented in /packages/shared/types.py
• 03. JWT auth for dashboard WebSocket — hydra dashboard-token generates required token for WS URL
  • Acceptance: connection without a valid token is rejected with a clear 401-equivalent close code
• 04. RunHistoryStore — every completed run written to SQLite; endpoint lists past runs + serves event log
  • Acceptance: history survives a supervisor restart
• 05. LogExportEndpoint — GET /api/runs/{id}/export returns full JSONL event log as downloadable file
  • Acceptance: exported file is byte-identical to the original RunLogger output
• 06. Streaming compression — compress large log payloads before WebSocket send to avoid browser frame drops
  • Acceptance: measured frame time stays smooth on a 10k-event backlog replay
• 07. RateThrottler — batch/debounce if events arrive faster than 60/sec before sending to client
  • Acceptance: client never receives more than 60 discrete WS messages/sec under load test
• 08. FuzzerTest for WebSocket server — 10,000 random events/sec, assert no dropped events or crash
  • Acceptance: test runs in CI on a schedule (nightly), not just locally
• 09. MultiClientBroadcast — two browsers connected simultaneously see the same live run without interference
  • Acceptance: manual + automated test with 2 concurrent WS clients on the same run ID
• 10. Phase V demo — screen record War Room Mode on a live three-node run, genealogy tree growing as Reaper fires
  • Acceptance: video shows convergence gauge, genealogy tree, and node meters all updating live

---

🟧 PHASE VI — Benchmark, Harden, Ship

Demo Milestone: v0.1.0 tagged and released with published benchmark numbers.

• 01. 48-hour soak test — across all three nodes on a real medium-sized repo; log and fix every crash
  • Acceptance: zero unhandled exceptions in the final soak run; a log of all bugs found + fixed is committed
• 02. Dependency pinning + audit — pin pyproject.toml/package.json; run pip-audit and npm audit; fix critical CVEs
  • Acceptance: zero critical/high CVEs remain open at release time
• 03. SetupVerifier CLI — hydra doctor checks Docker, Ollama, Postgres, Node B/C connectivity, reports clearly
  • Acceptance: each check reports PASS/FAIL with an actionable fix suggestion, not just a boolean
• 04. Security hardening doc — sandbox network isolation, API token storage, what data leaves the machine
  • Acceptance: doc explicitly states "nothing leaves the machine unless a cloud fallback is configured," verified against actual code paths
• 05. Single-command installer — install.sh, tested on Ubuntu 22.04, Debian 12, WSL2
  • Acceptance: fresh VM test on all 3 target environments passes without manual intervention
• 06. hydra cluster init — first-time setup, generates pre-shared token, writes hydra.config.json on all three machines
  • Acceptance: re-running init on an already-initialized cluster is safe (idempotent, doesn't rotate token silently)
• 07. hydra update command — pulls latest version, runs DB migrations if schema changed
  • Acceptance: migration failure rolls back cleanly, doesn't leave the DB in a half-migrated state
• 08. Dependency audit — remove anything unused, freeze the rest
  • Acceptance: pyproject.toml and package.json diffs reviewed and documented in the PR description
• 09. Final repo cleanup — remove API keys, temp files, debug branches, TODOs not filed as issues
  • Acceptance: git log and grep -r "TODO" both clean before tagging
• 10. Ship — tag v0.1.0, push, publish to GitHub
  • Acceptance: release notes link back to this issue and to Phase VI benchmark results from Y. Jangra's track

---

🔗 Dependencies

• Depends on Node B track (Y. Jangra) for InferenceClient/gRPC InferenceService before Phase IV routing can be tested live
• Depends on Node C track (B. Kataria) for SandboxManager/gRPC SandboxService before Phase IV routing can be tested live
• Phase V dashboard frontend work is owned by Y. Jangra; this track only owns the WebSocket backend

⚠️ Risks

• Convergence scoring thresholds (α, β, γ, 0.75 cutoff) are guesses until validated against real seeded bugs — budget time in Phase II for tuning, not just implementation
• 6GB VRAM on Node A is tight for Phi-3-Mini + KV cache during long contexts — monitor headroom continuously