Parallel worker-thread scanning + shareable scan cache

[cursor]

Summary

Speed up scanning on large monorepos via worker-thread parallel detector execution and a shareable (content-addressed) cache key, building on the existing scan cache.

type: rfc — settle the parallelism model and cache key/versioning contract before implementation.

Motivation

DebtLens runs detectors in-process; on large repos this is single-threaded and re-parses unchanged files each run unless the local cache hits. Two improvements compound: (1) parallelize across files/detectors using worker threads, and (2) make the scan cache shareable across machines/CI so a warm cache from one runner benefits others.

Proposed behavior

• Parallel execution: shard files across a pool of worker_threads, run detectors per shard, then merge ScanResults deterministically (stable ordering regardless of worker scheduling). --concurrency <n> with a sensible default (CPU-based); --concurrency 1 preserves today's behavior.
• Shareable cache: extend the current cache (src/core/scanCache.ts) with a content-addressed key (file hash + rule config hash + DebtLens version) and an optional --cache-dir so CI can restore/save it as an artifact. Atomic writes are already used; keep them.
• Determinism and parallelism-safety are first-class requirements (cross-file rules like duplicate-logic, import-cycle, duplicated-literal need a merge/aggregation phase, not naive per-file sharding).

Implementation surface

• New src/core/parallelScan.ts orchestrating worker_threads; refactor src/core/scan.ts to separate per-file detection from cross-file aggregation so the latter runs after merge.
• Extend src/core/scanCache.ts with the content-addressed key and --cache-dir.
• CLI flags in src/cli/commands/scan.ts / src/cli/scanPipeline.ts.
• Benchmarks: extend scripts/benchmark.mjs to compare serial vs parallel; assert identical findings.
• Tests: determinism (parallel result == serial result), cache hit/miss, key invalidation on config/version change.

Acceptance criteria

• RFC defines the sharding model, cross-file aggregation phase, and cache key contract.
• Parallel scan produces byte-for-byte identical findings to a serial scan on the benchmark fixtures.
• --concurrency and --cache-dir work; --concurrency 1 matches current behavior.
• Benchmarks show a speedup on the large fixture without correctness regressions.
• Docs + CHANGELOG updated; npm run test:all passes.

Difficulty: large.