docs(design): record success-semantics and evaluation decisions

[fazpu]

What

Adds a design document — design/designs/success-semantics-and-evaluation.md — that writes down two deliberate, load-bearing decisions that were previously only implicit in the code. During the July 2026 review, an outside reading of the code (including an independent parallel analysis pass) mistook each for a defect. They are not defects. This doc exists so the next reader — human or agent — does not "fix" them by accident.

Both decisions share one principle: do not infer semantic success from noisy mechanical signals; delegate the success/acceptance decision to an explicit evaluation layer.

Decision 1 — Iteration success means "the assignment ran", not "the work is good"

IterationResult.success is True whenever a team-harness run returns normally, False only when the harness itself raises. It is not a judgment about whether the work was accomplished, and TeamHarnessResult.agents (worker exit codes) is intentionally not consulted — worker exit codes are a noisy proxy (a coordinator can legitimately return after a worker fails). Semantic success is delegated to control.json (the sole stop switch) and goal_check.json (evidence only). This has been unconditional since the first commit of harness_runner.py (a4cca5e) — original intent, not drift. The doc records the one bounded consequence (scheduler cadence keys off mechanical success) and an opt-in fix for it, should it ever matter.

Decision 2 — Evaluation is LLM-as-judge; agents don't author deterministic checks

The packaged inner_outer_eval eval workflows create only harness_judge checks on described outcomes and forbid agent-authored deterministic checks — because in practice, agent-authored deterministic checks were brittle and self-gameable (the implementer inventing its own pass/fail lets it game itself). The doc records the important boundary: the failure mode was agent-authored checks, not repo-owned contract tests. A target repo that already owns a trustworthy suite (e.g. a future ugm_wp) should add a deterministic backstop under the judge rather than drop the judge.

Also included

design/analysis/ — the July 2026 review that produced these decisions, including an independent parallel analysis pass. Kept as the decision trail and to make the design doc's cross-references valid.

Not included

No code changes. A follow-up PR will add a separate proposals design doc for the forward-looking improvements (double-loop hardening, human-in-the-loop pause state, per-child budgets, events/cost ledger, etc.).

🤖 Generated with Claude Code

https://claude.ai/code/session_01PN8aFwwxA8FzMy9LgpbQFt