Phase 2 — Public DFIR dataset integration (NIST CFReDS / Ali Hadi / DFRWS / BOTS)

[Juwon1405]

Why

Agentic-DART's current accuracy measurement (recall=1.0 / FPR=0.0 / hallucination=0) runs against examples/sample-evidence/ (deterministic reference, ~30 lines/file, 100% IOC-loaded) and examples/sample-evidence-realistic/ (same IOCs + 1:30 synthetic benign noise; introduced 2026-05-08). Both are bundled, byte-stable, and good for regression — but neither is a third-party benchmark a reviewer can independently re-score.

Production credibility requires measurement on widely-used, externally-curated DFIR datasets that the community already trusts.

Scope

Integrate Agentic-DART against at least 3 of the following, with each integration producing:

1. A dart_agent --case <id> invocation that runs end-to-end
2. Per-dataset ground-truth definition (which findings are TPs)
3. Measured recall / FPR / hallucination_count
4. Audit chain hash committed for reproducibility
5. Public results in docs/accuracy-report.md under a new "External datasets" section

Candidate datasets

Dataset	Source	Coverage	License
NIST CFReDS Hacking Case	https://cfreds.nist.gov/	Windows host forensics, classic	Public
Ali Hadi forensic challenges	https://ashemery.com/dfir.html	Multi-host, Windows + Linux	Open
DFRWS 2018 Data Leakage Case	https://dfrws.org/	Insider threat scenario	Open for research
Splunk Boss of the SOC v3	https://github.com/splunk/botsv3	Realistic SIEM-scale corpus	Apache 2.0
MalwareTrafficAnalysis.net pcaps	https://malware-traffic-analysis.net/	Network forensics	Open
MITRE Adversary Emulation Plans	https://github.com/center-for-threat-informed-defense	TTP-mapped scenarios	Apache 2.0

Workstream split

• Track A (Windows host): NIST CFReDS Hacking Case — file-system/registry/event log focus. Wires into existing parse_amcache / parse_prefetch / analyze_event_logs etc.
• Track B (multi-host): Ali Hadi case — Windows + Linux + browser. Exercises cross-host correlation.
• Track C (SIEM-scale): Splunk BOTS v3 — production-shape volume. Tests dart_corr DuckDB throughput at 5M+ rows.
• Track D (network): MalwareTrafficAnalysis pcap → drives parse_evtx (issue Add typed parse_evtx (real EVTX format, not pre-extracted JSON) #30) and Phase 2 PCAP primitives.

Why this is Phase 2 (not pre-SANS)

Licensing review, ground-truth annotation, and 3+ dataset benchmarking are 4-6 weeks of work — outside the 2026-06-15 SANS window. Phase 1 establishes the methodology (deterministic + noise-injected variants); Phase 2 operationalizes it on community-trusted benchmarks.

Acceptance

• At least 3 datasets integrated with reproducible accuracy numbers
• docs/accuracy-report.md external-datasets section with measured recall/FPR/hallucination per dataset
• CI job measure-accuracy-external (manual trigger; not on every push due to dataset size)
• Methodology disclosure published: how ground truth was derived, what was scored, what was excluded

Related

• Issue Add accuracy measurement for parse_launchd_plist on Patrick Wardle macOS malware corpus #29 — case classes coverage (this issue extends that with externally-validated cases)
• Issue Phase 2 — Sigma rule synthesis from observed evidence #10 — Phase 2 Sigma rule synthesis (depends on real-corpus audit chains)
• Issue Add typed parse_evtx (real EVTX format, not pre-extracted JSON) #30 — native parse_evtx (Track D dependency)

[Juwon1405]

v1.0.1 status update (infrastructure landed; measurement still Phase 2).

The v1.0.1 platform overhaul delivered the scaffolding this issue scopes, though the actual external-dataset measurement runs remain Phase 2 / post-SANS:

• External-evaluation tier now exists as first-class case dirs: examples/case-studies/external-evaluation/case-01 (NIST CFReDS), case-02 (Ali Hadi web-server), case-03 (Digital Corpora M57-Patents, subject Jo) — each self-contained (README.md + truth.json).
• Downloader hardened (scripts/benchmark/download.py): browser-like headers on every request incl. resumed ranges, pure-Python streaming split-image reassembly, --dry-run / --check-urls. The 8 CFReDS part URLs verified reachable.
• One-command fetch+run: python3 run_eval.py --case external-evaluation/case-01 --download.

Path/name changes since this issue was filed (for the linked CFReDS-gap issues #48/#53/#54/#55 and #1):

• examples/case-studies/case-08-cfreds-hacking-case/ → examples/case-studies/external-evaluation/case-01/
• examples/sample-evidence-realistic/ → examples/case-studies/self-evaluation/case-01/evidence_root/ (now the single canonical bundled evidence; the public --variant selector was removed)
• every ground-truth.json → truth.json

Remaining scope (unchanged, Phase 2): licence review, ground-truth annotation, and the measured recall/FPR/hallucination runs committed to docs/accuracy-report.md. No external-dataset accuracy is claimed at v1.0.1.