refactor: extraction, tooling, and stability#3
Merged
Conversation
…rhaul 8-agent parallel review identified 78 issues (1 critical, 16 high, 42 medium, 19 low). All critical, high, and medium issues implemented across 16 specs. Key improvements: Pipeline efficiency: - Replace LLM utility queries with direct tool calls ($2-8/investigation savings) - Skip redundant extraction when sources already indexed (evidence-path-aware) - Fix N+1 DB queries, double context builds, sequential subprocess calls - Parallelize icat extraction with ThreadPoolExecutor - Reduce prompt token waste (compact results, remove open_case prefix) Reliability: - Fix gates auto-passing on None (retry-aware failure tracking) - Fix race condition in load_case context swap (atomic _ctx replacement) - Protect populated case DBs from being wiped by second server instance - Add thread safety (locks) to 4 module-level mutable caches - Plug resource leaks (FUSE mounts, temp files, pipe deadlock, temp dirs) Security & correctness: - Sanitize evidence-derived content before prompt injection - Accumulate follow-up history across iterations - Capture batch IDs structurally from tool results instead of regex on prose - Fix source_is_cited returning True for empty strings - Add circuit breaker to IOC enrichment, flush audit writes Code quality: - Consolidate duplicated IOC/severity/extension/JSON-parsing utilities - Introduce @tool_access decorator for declarative tool-role mapping - Introduce @audited_tool decorator for tool boilerplate reduction - Decompose runner.py (extract compaction loop, token parsing, panel skip) - Remove dead code (default_registry, orphaned audit prompts) Investigation quality: - Add behavioral context synthesis (correlate user activity with suspicious events) - Add anti-evasion awareness (check for injection within whitelisted JIT/AV) - Fix finding dedup threshold (merge same-event findings with different wording) - Exclude false-positive findings from IOC export appendix - Consolidate registry parser output into one source per hive - Add document/image forensics guidance to extraction planner Testing: - Add 48 new tests (submit_finding, correlator, IOC export, navigator, CLI, extractors) - Prune 20 low-value tests coupled to private internals - Final count: 503 tests, all passing Tool organization: - @tool_access decorator on all 142 MCP tools with Role enum - phases.py builds allowlists dynamically from registry (no more manual lists) - Prompt/tool list contradictions resolved (get_tool_guide, submit_finding, parse_plist)
…access - Store EVTX extraction dir in DB so index_evtx_file works across server restarts - Store TSK extraction dirs in DB for same reason - DB is the single source of truth for all investigation state
… in DB tool_response now returns status, source, preview (500 chars), and search hint when a source is provided (indicating data was indexed). Full extraction and analysis output is only accessible via search/get_raw_output. Control tools (batch, parallel, case management) and read tools (without a source) unchanged and return full results. Reduces context pressure on agents by 10-50x per tool call. - Refactored hayabusa, hindsight, and mvt to use tool_response consistently - Volatility batch passes source=None since its per-plugin coordination data is not a single indexed source (individual plugins are indexed separately) - windowed_response (READ path) unaffected
- Executor allowed_tools restricted to plan contents + control essentials - Reduces executor context from 74 tools to ~15 per session - Standardize top 30 tool docstrings with what/when/output structure
…on only) - Remove pre-population from planner output (unreliable, causes stale entries) - Tasks appear only when actually dispatched or called - Clear task panel on phase retry to prevent accumulation - Consistent real-time updates from job completion markers
- Remove SleuthKitExtractor class (extractors/sleuthkit.py) - Remove BulkExtractorExtractor class (extractors/bulk.py) - Remove PlasoExtractor class (extractors/plaso.py) - Remove LogFileExtractor class (extractors/logs.py) - Remove EZToolsExtractor class (extractors/eztools.py) - Remove DiskImageExtractor class and its private helpers from disk.py - Remove VolatilityExtractor class from volatility.py - Remove Extractor protocol and ExtractorRegistry (extractors/base.py) - Remove ExtractionResult dataclass (unused by server or orchestrator) - Remove tests/test_extractors.py (tested only dead extractor classes) Retained from disk.py: _parse_evtx_file, _mount_image, _unmount_image (used by server/tools/extract/evtx.py and server/extract_helpers.py). Retained from volatility.py: _find_vol_binary, _plugin_short_name (used by server/tools/extract/volatility.py and server/tools/yara.py). Retained: extractors/classifier.py (EvidenceClassifier used by MCP tools).
- Query enrichment.iocs source for threat intel metadata - Add context column to IOC tables showing country, ASN, abuse score - Works in both markdown and HTML reports
…tate - Update CLI usage with required case_id argument - Document tool_access decorator pattern for contributors - Update architecture with dynamic allowlists, composite indexing, gate changes - Add investigation quality improvements section
- Step-by-step walkthrough from function creation to registration - Covers tool_access, indexing, skip logic, and testing - Linked from README
…nsaw - run_hayabusa falls back to DB when in-memory EVTX path unavailable - run_chainsaw gets same treatment if affected - Consistent with index_evtx_file fix from earlier
- Match system names against relative path instead of filename only in _build_evidence_context, so files in nested subdirectories (e.g. /evidence/system-10.3.58.4/system-disk/system.E01) are discovered even when the system name only appears in a parent directory - Intersect dynamic executor allowlist with role-based permissions, preventing the planner from granting the executor access to tools outside its declared role (e.g. scan_evidence leaking from catalog) - Enforce MULDER_CASE_ID in create_case, refusing to create per-system databases when an orchestrator-level case ID is set - Enforce MULDER_CASE_ID in scan_evidence over any explicit case_id argument, preventing agents from bypassing the enforced case - Fix test mocks for _run_analyst task_system parameter
mmls consistently exits with code 1 and empty stderr on images that lack a partition table (partition dumps, single-filesystem images). The previous error response was generic and gave no recovery guidance. Introduce _classify_mmls_failure to categorize mmls failures into three types: no_partition_table (empty stderr), ewf_unsupported (E01/libewf keywords in stderr), and generic mmls_failed. Each type returns a structured error_type and actionable suggestion directing the agent to use run_fls with partition_offset=0.
…allel _handle_job_completion broadcast each JOB_COMPLETE event to every active system, so when system A's run_fls finished it also marked system B's run_fls as done. Add complete_one_running_task to the dashboard, which updates exactly one task in "running" state per completion event, and switch the log tailer to use it instead of the broadcast loop.
- Equal task lines per system regardless of task count - Fix log panel to fixed height to prevent layout jumps - Show running tasks first, cap done tasks per system
…refreshes - Always render two-column layout (never switch between 1 and 2 columns) - Empty task panel when no tasks instead of removing the column - All panels have fixed heights that never change between refreshes
The Rich Console was initialized with stderr=True, tying its output to fd 2. Any stray write to fd 2 from the Claude Agent SDK subprocess or native code would land at the cursor position below the Live panel, persisting for one refresh cycle (~0.5s) before the next redraw overwrote it. Changes: - Duplicate fd 2 at init so the Console always reaches the terminal through a private fd, independent of the process-wide stderr. - On start(), redirect fd 1 and fd 2 to /dev/null; restore on stop(). This silences ALL subprocess and native-code writes at the OS level. - Remove the dead _refresh() method (Live.update sets _renderable which get_renderable never reads; the auto-refresh callback was always the sole render path). - Increase refresh_per_second from 2 to 4 for smoother visual updates. - Fix pre-existing SIM102 lint errors in runner.py and yara.py.
finalize_report reads model_usage.json for real token counts instead of falling back to audit log estimates. The sidecar file is now written before the report phase starts so it exists at render time.
- Removed outdated image from README and consolidated documentation sections. - Introduced a new glossary file to define project terminology and concepts for contributors and users. - Updated README to include links to comprehensive documentation resources, including architecture, tool addition guides, and a tool manifest.
- Added `case_id` argument to the `investigate` command for better tracking of investigations. - Updated documentation to clarify the use of `case_id` in CLI commands. - Improved terminal size handling in the dashboard to prevent errors when retrieving terminal dimensions. - Adjusted prompt templates to include case ID instructions for clarity. - Refactored tests for better readability and consistency.
- Correct CLI option docs (export-iocs, export-navigator, report) - Fix architecture claims (retry scope, throttling, PDF, exports, enrichment) - Fix adding-tools guide (variable names, pattern conflicts, verification steps) - Correct tool-manifest types and role tags - Fix glossary source references
…t fails When run_fls succeeds with an explicit partition offset but mounting fails (multi-segment E01s, missing FUSE, container restrictions), the partition offset was lost. Downstream tools (registry, EVTX, Amcache, ShimCache, MFT, Prefetch) that fell back to icat extraction would use offset 0, causing all extractions to fail. - Store partition offset in DB kv_store on successful run_fls - Add _resolve_partition_offset helper (kv_store -> tsk.partitions -> mmls) - Shared helper for cross-tool offset reuse in _tsk_extract_files, _extract_evtx_from_image, _run_fls_inline, and run_mft_parser
…offset failure - Fix mmls output regex to handle standard TSK 4.x format with spaced slot fields (e.g. "002: 000:000") and optional leading whitespace; the previous regex only matched compact "002:000" format, silently returning offset 0 even when mmls found a valid partition table - Restructure run_fls to be self-contained: check kv_store first, try offset 0, and on failure run mmls inline to detect and retry with the correct offset, eliminating race condition with parallel run_mmls - Consolidate _detect_partition_offset to delegate to _parse_partition_offset, adding linux and "basic data" (GPT) partition type detection
Reverse the extraction strategy for registry, prefetch, amcache, shimcache, and MFT parsers: attempt TSK extraction (fls + icat) first since it reads E01 images directly without FUSE, then fall back to mount-based extraction as a bonus path. This eliminates the container dependency on ewfmount/guestmount which require privileged mode.
- Defer temp dir creation in _tsk_extract_files until first pattern match to prevent leaked empty directories when fls data exists but no files match - Log icat timeout and failure in run_mft_parser so fallback to mount is observable in logs - Update docstrings to reflect TSK-first extraction order
run_fls now analyzes all non-trivial partitions, not just the largest. Each partition's files indexed separately with partition-aware source names (tsk.filelist for primary, tsk.filelist.p1/p2/etc. for secondary). Primary partition offset stored in kv_store for downstream tools; secondary partitions accessible via search on their source names. Downstream consumers (_tsk_extract_files, _extract_evtx_from_image, _find_inodes_by_pattern, list_files, get_deleted_files) updated to search all tsk.filelist* sources and use the correct per-partition offset for icat extraction.
…tence, and staging Add three methodology improvements to analyst prompts based on gap analysis: - Authentication pattern analysis: guide analysts to examine failure/success clusters and verify source addresses before dismissing logon events as normal - Persistence coverage: ensure all relevant log sources (including system-level logs) are indexed before concluding no persistence was found - Data staging detection: search filesystem metadata for archive files in staging locations, not just execution artifacts for archiving tools
…stomp detection - find_file_staging searches MFT/filelist for archive creation and deletion patterns - index_evtx_file auto-indexes System.evtx alongside Security for persistence coverage - detect_timestomping identifies SI vs FN timestamp mismatches in MFT data
- Add multi-partition, TSK-first extraction, new tools to architecture - Add find_file_staging and detect_timestomping to tool manifest - Update glossary with new terms - Verify README accuracy
- Indexes Autoruns CSV as searchable source for persistence detection - Auto-discovers autoruns CSV files in evidence directories - Updated tool manifest, glossary, and source prefix registry
…when suspicious binaries are found.
… to resolve compatibility issues with Suricata installation on amd64
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tool_accesscontrols.Key Changes
E01 Disk Image Extraction
Investigation Quality
Tool Architecture
tool_accessdecorator usage for consistent access enforcement.New Tools
find_file_staging.detect_timestomping.parse_autoruns.UI Stabilization
Documentation and Reliability Fixes