From 1991962da9dd101d92252be43f47d68b93e2060b Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 21:53:10 +0000 Subject: [PATCH 1/3] security-adversarial: classify evaluator entrypoint liveness (redblue silent no-op) Reproduced a real, 100%-reliable silent failure in this repo's own SCAN=redblue evaluator entrypoint (dream.config.json): `npx @metaharness/redblue ` exits 0 with zero bytes on both stdout and stderr, for every subcommand tried. Root cause: the package's CLI guards its dispatch with the classic `import.meta.url === file://${process.argv[1]}` idiom, which fails once the executable is reached through the symlink npm/npx create for a package's bin entry, so the dispatch body never runs. Exit 0 + silence is indistinguishable from "ran clean, zero findings" to anything that only checks the exit code. Adds classifyEntrypointResult (packages/cli/src/entrypoint.ts) and a `dream-machine verify-entrypoint` CLI command so this pipeline can separate live / blocked / suspicious-silent before ever recording EVALUATED=yes. Includes two fixes from an independent adversarial critique of this diff: unquoted multi-word --cmd used to silently truncate and produce a false "live" verdict (fixed with a hard usage-error guard); child_process.exec's error.code can be a non-number on maxBuffer overflow (fixed with explicit buffer size + defensive coercion). npm test: 85 -> 96 (all new, 0 removed/modified). ADR-0002 documents the entrypoint-trust convention. Full evidence in docs/dream-cycle/2026-08-13-security-adversarial-report.md. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01UAJ2FdMooTevrJ34FW3mpr --- ...ecurity-adversarial-entrypoint-liveness.md | 107 +++++++ docs/adrs/INDEX.md | 1 + .../2026-08-13-security-adversarial-report.md | 273 ++++++++++++++++++ packages/cli/src/bin.ts | 20 ++ packages/cli/src/entrypoint.test.ts | 39 +++ packages/cli/src/entrypoint.ts | 51 ++++ packages/cli/src/index.test.ts | 58 ++++ packages/cli/src/index.ts | 35 +++ 8 files changed, 584 insertions(+) create mode 100644 docs/adrs/ADR-0002-dream-cycle-security-adversarial-entrypoint-liveness.md create mode 100644 docs/dream-cycle/2026-08-13-security-adversarial-report.md create mode 100644 packages/cli/src/entrypoint.test.ts create mode 100644 packages/cli/src/entrypoint.ts diff --git a/docs/adrs/ADR-0002-dream-cycle-security-adversarial-entrypoint-liveness.md b/docs/adrs/ADR-0002-dream-cycle-security-adversarial-entrypoint-liveness.md new file mode 100644 index 0000000..3af08c6 --- /dev/null +++ b/docs/adrs/ADR-0002-dream-cycle-security-adversarial-entrypoint-liveness.md @@ -0,0 +1,107 @@ +# ADR-0002: Evaluator entrypoints must be classified live/blocked/suspicious-silent before an EVALUATED verdict is recorded + +- **Status**: Proposed +- **Date**: 2026-08-13 +- **Related**: ADR-0001 §2.3 ("Evaluation is delegated, never reimplemented"), §2.4 ("the engine must never hard-depend on [evaluation backends]"), §5 Test Contract item 4 (optional-backend degradation) +- **Deciders**: dream-cycle nightly session (security-adversarial, SCAN=redblue/supply-chain), 2026-08-13 +- **Tags**: dream-cycle, security-adversarial, supply-chain, evaluator-trust, witness-every-quantitative-claim + +## 1. Context + +Tonight's SCAN=redblue probe reproduced a real, 100%-reliable failure in +`npx @metaharness/redblue` (the SCAN=redblue evaluator entrypoint this repo's +own `dream.config.json` declares): the command exits `0` with **zero bytes** +on both stdout and stderr, for every subcommand tried. Root cause: the +package's CLI guards its dispatch with `import.meta.url === +\`file://${process.argv[1]}\``, a comparison that fails once the executable +is reached through the symlink `npm`/`npx` always create for a package's +`bin` entry — so the dispatch body silently never runs. + +ADR-0001 §2.3 composes `@metaharness/{flywheel,darwin,redblue}` as the +pipeline's evaluation backends and explicitly says the engine must treat them +as optional (§2.4) — but "optional" was only specified for the *absent* +case (package not installed, no credentials → `LLM_EVAL=blocked`). Nothing in +ADR-0001 addresses the case discovered tonight: a backend that *is* installed, +*is* invoked correctly, and *exits 0* — while having silently done nothing. +An exit code alone cannot distinguish this from a genuine clean pass, and +STEP 5-9 of the compiled nightly prompt ("Do not infer results from logs. +Preserve the real receipt") assumes a human/agent will catch this by hand +every time. Tonight's session did, by accident of manual probing — that is +not a mechanism, it's luck. + +## 2. Decision + +Ship `@dream-machine/cli`'s `classifyEntrypointResult` (`packages/cli/src/entrypoint.ts`) +as the canonical, deterministic classifier for any evaluator-entrypoint +result in this engine, with three outcomes: + +- **live** — exit 0 with non-empty stdout or stderr. +- **blocked** — nonzero exit (the tool ran and told us something is wrong; + the reason is in stderr). +- **suspicious-silent** — exit 0 with empty stdout *and* empty stderr. Never + equated with a clean pass; never sufficient on its own to record + `EVALUATED=yes`. + +Exposed tonight as `dream-machine verify-entrypoint