Activate v3 framework scaffolds at runtime (deferred during SANS window)

[Juwon1405]

Background

The senior-analyst-v3.yaml playbook (default since v0.5) ships four mature-SOC framework blocks as YAML data scaffolds:

• ads_template — Palantir ADS 9-section detection contract
• magma_ucf — FI-ISAC NL three-tier UCF + CMMI 5-level maturity
• hunt_cycle — TaHiTI H1/H2/H3 hunt phases with designed trigger
• hunting_maturity_model — Bianco HMM 0–4 with self-declared target level

These are inspectable, forkable, citable data. They define the contract a mature-SOC implementation should satisfy. They are not yet referenced from dart_agent or dart_corr — the runtime path remains v2's sequence + next_call_decisions + contradiction_triggers + stop_conditions. QA round 5 (commit pending) corrected README, CHANGELOG, wiki, and the v3 yaml header to honestly frame this as 'data scaffold; runtime activation deferred', tracked here.

Why deferred

Activating any of the four scaffolds at runtime would shift the baseline measured by scripts/measure_accuracy.py. The hackathon submission ships with a stable, reproducible baseline. Doing all four runtime activations together with a single re-baseline after 2026-06-15 is cleaner than doing them piecemeal mid-window and risking a misread regression or artificial improvement.

Acceptance criteria (post-SANS)

1. ADS lint pass

• dart_agent/__init__.py reads ads_template.required_sections from the loaded playbook
• Each finding emitted by the agent is checked against the 9 required ADS sections
• Lint mode permissive / warn / strict controls whether missing sections produce a warning or a hard failure
• Default lint mode upgraded from warn to strict after one re-baseline
• New test in tests/test_qa_pass_regressions.py asserting ADS lint runs on every finding

2. MaGMa CMMI scoring per run

• Agent assigns a CMMI level (1–5) to each completed run based on:
  - Has the run produced findings? (L2 Managed)
  - Are findings ADS-templated? (L3 Defined)
  - Are FP/TP rates measured? (L4 Quantitatively Managed)
  - Is TI feedback actively consumed? (L5 Optimizing)
• CMMI level recorded in progress.jsonl
• CMMI distribution surfaced in scripts/measure_accuracy.py output

3. TaHiTI hunt-mode entry on plateau

• Agent loop detects plateau condition (confidence < 0.6 AND iterations >= 8)
• On plateau, agent transitions to H1 Initiate → H2 Hunt → H3 Finalize
• H1/H2/H3 transitions logged to audit chain
• New stop condition hunt_mode_active AND H3_finalize_complete → emit_with_hunt_findings honored
• New test asserting hunt-mode entry on plateau scenario

4. HMM per-run self-classification

• Agent classifies each run to HMM 0–4 based on:
  - Data sources used (HMM1 if IOC-only, HMM2 if procedure-driven, etc.)
  - Hypothesis source (HMM3 if analyst-formed, HMM4 if automated)
• HMM level recorded in progress.jsonl
• HMM distribution surfaced in scripts/measure_accuracy.py output

5. Re-baseline

• Run scripts/measure_accuracy.py against the existing test corpora and examples/sample-evidence/
• Update baseline numbers in Phase-1.md release-history table
• Update tests/test_qa_pass_regressions.py with the new baseline assertions

6. Documentation honesty re-pass

• README, CHANGELOG, wiki dart-playbook.md, wiki Roadmap.md, v3 yaml header — all 'post-SANS' / 'scaffolded as data' framing flipped to current-state language describing runtime behavior
• Issue Activate three dormant detection patterns (deferred during SANS window) #43 (3 dormant detection patterns) coordinated with this work — single re-baseline covers both

See also

• Issue Activate three dormant detection patterns (deferred during SANS window) #43 — three dormant detection patterns also waiting on the post-SANS re-baseline
• dart_playbook/senior-analyst-v3.yaml — the data scaffolds
• dart_agent/src/dart_agent/__init__.py — where the runtime activation will live
• CHANGELOG [Playbook v3] entry — current 'honest framing' language to be replaced

[Juwon1405]

Update — the playbook sequence is now loaded from the YAML at runtime (commit c2b01d7)

Correcting the premise in the Background above. The issue stated "the runtime path remains v2's sequence + next_call_decisions + contradiction_triggers + stop_conditions." On inspection that wasn't actually true — none of those were read from the YAML. dart_agent/live.py carried a hardcoded 5-line "Your playbook" block inside SYSTEM_PROMPT, and senior-analyst-v3.yaml was never loaded by any code, even though the README and the dart_agent docstring both claimed it was. So the playbook was effectively prompt-first, not architecture-first.

c2b01d7 fixes that for the sequence block specifically:

• A loader (_latest_playbook + _render_playbook) reads the highest-version senior-analyst-v*.yaml and renders its sequence (each phase's rationale + the MCP calls it prescribes) into the system prompt.
• Editing the YAML now actually changes agent behaviour. The YAML is the source of truth for the phase sequence.
• Verified: SYSTEM_PROMPT contains the v3 phase sequence (P0 volatility → P9 finding emission) with per-phase tool lists.

Still deferred — this issue stays open. Scope is unchanged:

• The four framework scaffolds — ads_template, magma_ucf, hunt_cycle, hunting_maturity_model — remain inspectable data, not runtime-active. All six acceptance criteria above are untouched.
• next_call_decisions, contradiction_triggers, and stop_conditions are also still not read from the YAML; only sequence is rendered into the prompt today. Wiring those (plus the four scaffolds) and the single re-baseline is the remaining work here.

Net: the phase sequence is now live and architecture-first; everything this issue was opened to track (scaffold runtime activation + re-baseline) is not. Keeping open.