diff --git a/docs/adrs/ADR-0002-dream-cycle-security-adversarial-entrypoint-liveness.md b/docs/adrs/ADR-0002-dream-cycle-security-adversarial-entrypoint-liveness.md new file mode 100644 index 0000000..3af08c6 --- /dev/null +++ b/docs/adrs/ADR-0002-dream-cycle-security-adversarial-entrypoint-liveness.md @@ -0,0 +1,107 @@ +# ADR-0002: Evaluator entrypoints must be classified live/blocked/suspicious-silent before an EVALUATED verdict is recorded + +- **Status**: Proposed +- **Date**: 2026-08-13 +- **Related**: ADR-0001 §2.3 ("Evaluation is delegated, never reimplemented"), §2.4 ("the engine must never hard-depend on [evaluation backends]"), §5 Test Contract item 4 (optional-backend degradation) +- **Deciders**: dream-cycle nightly session (security-adversarial, SCAN=redblue/supply-chain), 2026-08-13 +- **Tags**: dream-cycle, security-adversarial, supply-chain, evaluator-trust, witness-every-quantitative-claim + +## 1. Context + +Tonight's SCAN=redblue probe reproduced a real, 100%-reliable failure in +`npx @metaharness/redblue` (the SCAN=redblue evaluator entrypoint this repo's +own `dream.config.json` declares): the command exits `0` with **zero bytes** +on both stdout and stderr, for every subcommand tried. Root cause: the +package's CLI guards its dispatch with `import.meta.url === +\`file://${process.argv[1]}\``, a comparison that fails once the executable +is reached through the symlink `npm`/`npx` always create for a package's +`bin` entry — so the dispatch body silently never runs. + +ADR-0001 §2.3 composes `@metaharness/{flywheel,darwin,redblue}` as the +pipeline's evaluation backends and explicitly says the engine must treat them +as optional (§2.4) — but "optional" was only specified for the *absent* +case (package not installed, no credentials → `LLM_EVAL=blocked`). Nothing in +ADR-0001 addresses the case discovered tonight: a backend that *is* installed, +*is* invoked correctly, and *exits 0* — while having silently done nothing. +An exit code alone cannot distinguish this from a genuine clean pass, and +STEP 5-9 of the compiled nightly prompt ("Do not infer results from logs. +Preserve the real receipt") assumes a human/agent will catch this by hand +every time. Tonight's session did, by accident of manual probing — that is +not a mechanism, it's luck. + +## 2. Decision + +Ship `@dream-machine/cli`'s `classifyEntrypointResult` (`packages/cli/src/entrypoint.ts`) +as the canonical, deterministic classifier for any evaluator-entrypoint +result in this engine, with three outcomes: + +- **live** — exit 0 with non-empty stdout or stderr. +- **blocked** — nonzero exit (the tool ran and told us something is wrong; + the reason is in stderr). +- **suspicious-silent** — exit 0 with empty stdout *and* empty stderr. Never + equated with a clean pass; never sufficient on its own to record + `EVALUATED=yes`. + +Exposed tonight as `dream-machine verify-entrypoint