Rotation
DEEP=security-adversarial, SCAN=redblue,supply-chain (slot 3, DAYINT % 5). No bonus deep dive triggered (DAYINT % 25 = 13, % 75 = 13).
Ledger Check
docs/dream-cycle/LEDGER.md has 6 rows (verified structurally OK via dream-machine ledger verify), but rows referencing issues/PRs #3, #4, #6, #9, #10, #12, #13, #15, #16, #18 and dates 2026-08-14 through 2026-08-19 (which postdate tonight, 2026-08-13) do not correspond to real GitHub state — confirmed via mcp__github__list_issues/list_pull_requests: 0 real issues, only 5 open dependabot PRs (#1-5). These are seed/example rows, not real run history. Flagged here so a future night doesn't try to "re-check the fate" of fictional issues. Not corrected in this PR (out of scope for tonight's candidate).
ledger signals: zeroMergeStreak=true (6 nights considered) — but this is computed against the same seed data above, so treat with the same caution; it happens to also match this candidate's small/reviewable scope anyway.
Deep Dive
Reproduced (grade A, first-hand, not sourced): npx -y @metaharness/redblue <init|run|attack|help|bare|--version> all produce 0-byte stdout, 0-byte stderr, exit 0 — every subcommand tried. Root-caused via direct inspection of the installed package (~/.npm/_npx/*/node_modules/@metaharness/redblue/dist/cli/index.js line ~489): the CLI guards its dispatch with const isMain = import.meta.url === \file://${process.argv[1]}`;, a comparison that fails once the executable is reached through the symlink npm/npx always create for a package's bin entry (node_modules/.bin/redblue→@metaharness/redblue/dist/cli/index.js), so the dispatch body never runs. This repo's own dream.config.json:17 documents exactly this broken invocation ("redblue": "npx @metaharness/redblue"`) as the SCAN=redblue evaluator entrypoint every future security-adversarial night is instructed to trust.
Also found: "flywheel": "npx @metaharness/flywheel" fails too, but loudly (@metaharness/flywheel ships no bin field, so npx errors immediately, visibly) — safe by comparison, but still a stale entrypoint doc.
@metaharness/darwin is genuinely invocable (npx -y @metaharness/darwin evolve|bench|security), but --version isn't a recognized flag and exits 1 with usage text — correctly classified tonight as blocked (visible reason), demonstrating the classifier doesn't just rubber-stamp everything nonzero as bad, it distinguishes "wrong invocation with a visible reason" from "wrong invocation with zero visibility."
Hypothesis
Given a nightly pipeline that shells out to third-party evaluator entrypoints and trusts their exit code as a pass/fail signal, when a deterministic classifyEntrypointResult function is interposed between the raw exec result and the pipeline's EVALUATED=yes/no/blocked decision, then it should correctly separate LIVE (real output, exit 0) from BLOCKED (nonzero exit, has a reason) from SUSPICIOUS-SILENT (exit 0, zero output on both streams) — a distinction the raw exit code alone cannot make — subject to: no change to any existing test, no change to the bench: npm test evaluator's own behavior, and the classifier must be pure/deterministic.
Frozen before implementation; not modified after evaluation began.
Evaluation Receipt
Real evaluator: npm test (vitest), this repo's own bench entrypoint.
|
Baseline (parent, commit 3de0107) |
Candidate |
| Tests |
85 |
96 (+11, 0 removed, 0 modified) |
| Result |
85 passed |
96 passed |
Live receipt against this repo's real dream.config.json evaluatorEntrypoints tonight — see full transcript in docs/dream-cycle/2026-08-13-security-adversarial-report.md ("Benchmarks / Evaluation" section). Summary: bench→live, flywheel→blocked, redblue→suspicious-silent, darwin --version→blocked. All four match ground truth verified by hand.
Darwin Results
Not run — DARWIN=not-applicable. No evolvable population for a single, already-minimal, pure classification function with an exhaustive 3-branch structure; bounded Darwin's generations×children search needs a mutable candidate population this doesn't have. Not a stop condition.
Evidence
OBSERVATION / MEASUREMENT / INFERENCE / DECISION trail in full in the committed report. Key: npm test 85/85 → 96/96, 0 regressions; root cause traced to nodejs/node#57616 (grade B) and Node's native fix import.meta.main (grade A, nodejs.org/api/esm.html, stable-track ≥22.18/24.2).
Reward-Hack Check
Independent critic (separate agent, not this candidate's author): verdict CLEAR of reward-hacking (only the 5 intended files changed, all additive, no gold/threshold/test weakening). Found and this PR fixes two real defects before finalizing: (1) an unquoted multi-word --cmd used to silently truncate and produce a false live verdict — fixed by rejecting stray positionals as a hard usage error, regression test added; (2) child_process.exec's error.code can be a string on maxBuffer overflow, violating the number contract, with no maxBuffer override — fixed with explicit 10MiB buffer + defensive coercion. One risk flagged and correctly left unfixed as out-of-scope: io.exec shells out an unsanitized string; not exploitable today (cmd only comes from a human-typed CLI flag), but the natural next automation (auto-feeding dream.config.json#evaluatorEntrypoints into this) would pipe a repo-modifiable value into an unsanitized shell exec — MUST be addressed (allowlist or execFile with argv array) before anyone wires that automation, not before tonight's tiny diff.
Security Review
io.exec in bin.ts shells via child_process.exec (/bin/sh -c); cmd today only ever comes from a trusted, human-typed local CLI invocation — no new remote/network attack surface. No credentials handled. No LLM calls in this candidate (N/A for prompt injection). IO.exec is optional (exec?), so no existing call site gains a new capability.
Scan Findings
redblue: see Deep Dive above — upstream package (ruvnet/agent-harness-generator), out of this repo's scope to patch directly; recommend filing there separately (not done tonight).
supply-chain: @metaharness/{flywheel,darwin,redblue} are legitimate, correctly-scoped, single-maintainer (ruvnet) npm packages — no typosquat/dependency-confusion risk. The real supply-chain gap found tonight is trust, not identity: no mechanism existed to detect a correctly-resolved, legitimately-published dependency silently doing nothing. This PR closes that gap.
Competitors
Sakana AI "The AI Scientist" (B), OpenHands (C), SWE-agent/SWE-bench (C, exit-code fail2pass has known ~0.90-0.93 precision per arXiv:2503.15223), DSPy/GEPA (B, absence confirmed via official docs) — none document a comparable tool-liveness check. Closest prior art: pytest's dedicated exit code 5 for "ran clean but did zero real work" (B). Full table in the committed report.
Gist
No gh gist create tonight — this session has GitHub API access via MCP tools for issues/PRs/branches but no gist-creation tool was available (gh CLI itself is also not installed in this environment). Report committed instead at docs/dream-cycle/2026-08-13-security-adversarial-report.md. GIST=LOCAL.
Witness
report_sha256 : 8984e0cf5bcd850e6122aaec2831b16b74307a2a51e4e0b708118513fe24d64a
session_commit: 3de01079abe64e4a1f2d3fe3b758523705b3bf47
witness : ec2052aa3ce57068bba6f723ab0a00f9cde7b7a42e5abed71ea4388a43e4a4a4
Verify: sha256sum docs/dream-cycle/2026-08-13-security-adversarial-report.md then printf '%s%s' "<that hash>" "3de01079abe64e4a1f2d3fe3b758523705b3bf47" | sha256sum must equal the witness above. Independently confirmed tonight via dream-machine witness verify.
Recommendation
evaluated: accepted — human review of the draft PR. Not done tonight, flagged as follow-ups: (1) wire evaluatorEntrypoints through this classifier automatically inside the compiled prompt's STEP 5-9 guidance; (2) file upstream against @metaharness/redblue recommending import.meta.main; (3) the unsanitized-shell-exec risk noted in Reward-Hack Check must be closed before any automation feeds config-sourced commands into io.exec; (4) the fabricated ledger seed rows noted in Ledger Check above should probably be cleaned up or clearly marked as fixtures.
Rotation
DEEP=security-adversarial, SCAN=redblue,supply-chain (slot 3, DAYINT % 5). No bonus deep dive triggered (DAYINT % 25 = 13, % 75 = 13).
Ledger Check
docs/dream-cycle/LEDGER.mdhas 6 rows (verified structurally OK viadream-machine ledger verify), but rows referencing issues/PRs#3, #4, #6, #9, #10, #12, #13, #15, #16, #18and dates2026-08-14through2026-08-19(which postdate tonight, 2026-08-13) do not correspond to real GitHub state — confirmed viamcp__github__list_issues/list_pull_requests: 0 real issues, only 5 open dependabot PRs (#1-5). These are seed/example rows, not real run history. Flagged here so a future night doesn't try to "re-check the fate" of fictional issues. Not corrected in this PR (out of scope for tonight's candidate).ledger signals:zeroMergeStreak=true(6 nights considered) — but this is computed against the same seed data above, so treat with the same caution; it happens to also match this candidate's small/reviewable scope anyway.Deep Dive
Reproduced (grade A, first-hand, not sourced):
npx -y @metaharness/redblue <init|run|attack|help|bare|--version>all produce 0-byte stdout, 0-byte stderr, exit 0 — every subcommand tried. Root-caused via direct inspection of the installed package (~/.npm/_npx/*/node_modules/@metaharness/redblue/dist/cli/index.jsline ~489): the CLI guards its dispatch withconst isMain = import.meta.url === \file://${process.argv[1]}`;, a comparison that fails once the executable is reached through the symlink npm/npx always create for a package'sbinentry (node_modules/.bin/redblue→@metaharness/redblue/dist/cli/index.js), so the dispatch body never runs. This repo's owndream.config.json:17documents exactly this broken invocation ("redblue": "npx @metaharness/redblue"`) as the SCAN=redblue evaluator entrypoint every future security-adversarial night is instructed to trust.Also found:
"flywheel": "npx @metaharness/flywheel"fails too, but loudly (@metaharness/flywheelships nobinfield, so npx errors immediately, visibly) — safe by comparison, but still a stale entrypoint doc.@metaharness/darwinis genuinely invocable (npx -y @metaharness/darwin evolve|bench|security), but--versionisn't a recognized flag and exits 1 with usage text — correctly classified tonight asblocked(visible reason), demonstrating the classifier doesn't just rubber-stamp everything nonzero as bad, it distinguishes "wrong invocation with a visible reason" from "wrong invocation with zero visibility."Hypothesis
Given a nightly pipeline that shells out to third-party evaluator entrypoints and trusts their exit code as a pass/fail signal, when a deterministic
classifyEntrypointResultfunction is interposed between the raw exec result and the pipeline'sEVALUATED=yes/no/blockeddecision, then it should correctly separate LIVE (real output, exit 0) from BLOCKED (nonzero exit, has a reason) from SUSPICIOUS-SILENT (exit 0, zero output on both streams) — a distinction the raw exit code alone cannot make — subject to: no change to any existing test, no change to thebench: npm testevaluator's own behavior, and the classifier must be pure/deterministic.Frozen before implementation; not modified after evaluation began.
Evaluation Receipt
Real evaluator:
npm test(vitest), this repo's ownbenchentrypoint.Live receipt against this repo's real
dream.config.jsonevaluatorEntrypoints tonight — see full transcript indocs/dream-cycle/2026-08-13-security-adversarial-report.md("Benchmarks / Evaluation" section). Summary:bench→live,flywheel→blocked,redblue→suspicious-silent,darwin --version→blocked. All four match ground truth verified by hand.Darwin Results
Not run —
DARWIN=not-applicable. No evolvable population for a single, already-minimal, pure classification function with an exhaustive 3-branch structure; bounded Darwin's generations×children search needs a mutable candidate population this doesn't have. Not a stop condition.Evidence
OBSERVATION / MEASUREMENT / INFERENCE / DECISION trail in full in the committed report. Key:
npm test85/85 → 96/96, 0 regressions; root cause traced tonodejs/node#57616(grade B) and Node's native fiximport.meta.main(grade A,nodejs.org/api/esm.html, stable-track ≥22.18/24.2).Reward-Hack Check
Independent critic (separate agent, not this candidate's author): verdict CLEAR of reward-hacking (only the 5 intended files changed, all additive, no gold/threshold/test weakening). Found and this PR fixes two real defects before finalizing: (1) an unquoted multi-word
--cmdused to silently truncate and produce a falseliveverdict — fixed by rejecting stray positionals as a hard usage error, regression test added; (2)child_process.exec'serror.codecan be a string on maxBuffer overflow, violating thenumbercontract, with nomaxBufferoverride — fixed with explicit 10MiB buffer + defensive coercion. One risk flagged and correctly left unfixed as out-of-scope:io.execshells out an unsanitized string; not exploitable today (cmd only comes from a human-typed CLI flag), but the natural next automation (auto-feedingdream.config.json#evaluatorEntrypointsinto this) would pipe a repo-modifiable value into an unsanitized shell exec — MUST be addressed (allowlist orexecFilewith argv array) before anyone wires that automation, not before tonight's tiny diff.Security Review
io.execinbin.tsshells viachild_process.exec(/bin/sh -c);cmdtoday only ever comes from a trusted, human-typed local CLI invocation — no new remote/network attack surface. No credentials handled. No LLM calls in this candidate (N/A for prompt injection).IO.execis optional (exec?), so no existing call site gains a new capability.Scan Findings
redblue: see Deep Dive above — upstream package (
ruvnet/agent-harness-generator), out of this repo's scope to patch directly; recommend filing there separately (not done tonight).supply-chain:
@metaharness/{flywheel,darwin,redblue}are legitimate, correctly-scoped, single-maintainer (ruvnet) npm packages — no typosquat/dependency-confusion risk. The real supply-chain gap found tonight is trust, not identity: no mechanism existed to detect a correctly-resolved, legitimately-published dependency silently doing nothing. This PR closes that gap.Competitors
Sakana AI "The AI Scientist" (B), OpenHands (C), SWE-agent/SWE-bench (C, exit-code fail2pass has known ~0.90-0.93 precision per arXiv:2503.15223), DSPy/GEPA (B, absence confirmed via official docs) — none document a comparable tool-liveness check. Closest prior art: pytest's dedicated exit code 5 for "ran clean but did zero real work" (B). Full table in the committed report.
Gist
No
gh gist createtonight — this session has GitHub API access via MCP tools for issues/PRs/branches but no gist-creation tool was available (ghCLI itself is also not installed in this environment). Report committed instead atdocs/dream-cycle/2026-08-13-security-adversarial-report.md.GIST=LOCAL.Witness
Verify:
sha256sum docs/dream-cycle/2026-08-13-security-adversarial-report.mdthenprintf '%s%s' "<that hash>" "3de01079abe64e4a1f2d3fe3b758523705b3bf47" | sha256summust equal the witness above. Independently confirmed tonight viadream-machine witness verify.Recommendation
evaluated: accepted— human review of the draft PR. Not done tonight, flagged as follow-ups: (1) wireevaluatorEntrypointsthrough this classifier automatically inside the compiled prompt's STEP 5-9 guidance; (2) file upstream against@metaharness/redbluerecommendingimport.meta.main; (3) the unsanitized-shell-exec risk noted in Reward-Hack Check must be closed before any automation feeds config-sourced commands intoio.exec; (4) the fabricated ledger seed rows noted in Ledger Check above should probably be cleaned up or clearly marked as fixtures.