diff --git a/design/analysis/codex_parallel_analysis.md b/design/analysis/codex_parallel_analysis.md new file mode 100644 index 0000000..95b79d2 --- /dev/null +++ b/design/analysis/codex_parallel_analysis.md @@ -0,0 +1,665 @@ +# Executive summary + +`loopy-loop` is a promising durable control shell for long-running agent work, but it is not yet a trustworthy autonomous project manager for a system as large as UGM. Its strongest idea is the right one: continuity belongs in repository files, git, explicit workflow prompts, and inspectable artifacts rather than in one ever-growing chat. Its weakest seam is also clear in the code: mechanical completion is too easily promoted to semantic success. `team-harness` may return normally after failed or prematurely abandoned workers; `loopy_loop.harness_runner._normalize_harness_result()` then marks the iteration successful; and the stock eval workflow asks an agent to author and judge high-level LLM checks while explicitly forbidding deterministic checks. Before adding parallel loopy workers or a dashboard, the maintainer should harden the state machine, make active-child recovery durable, introduce typed outcomes and an append-only event/cost model, and make externally owned contract tests the authority for completion. UGM is an excellent dogfood target only as a sequence of narrow, human-gated work-package pilots—starting with one bounded Phase 0 package—not as a 120/160-turn autonomous run over the roadmap. The overall bet on long-horizon agent coding is sound as supervised, evidence-gated automation; the claim of unattended end-to-end autonomy is not yet supported. + +# Q1. Future improvements to loopy-loop + +## Current technical assessment + +The codebase is small enough to reason about: roughly 2,769 lines under `src/loopy_loop`, with 116 tests. I ran the suite: all 116 passed. Ruff and `uv run pyright` also pass. The fundamentals are better than the Alpha label might suggest: + +- `StateStore.mutate()` holds a file lock and commits `state.json` with `os.replace()`, giving coordinator transitions a useful atomic core. +- Root configuration is captured in `LoopState.config_snapshot`, so a worker does not silently take a new model setting from the root YAML halfway through a session. +- The `pending_finished_request.json` plus `result.json` recovery path closes an important crash window between harness completion and `/finished` acknowledgement. +- The scheduler has explicit and well-tested cadence semantics, including failed-workflow retry without incorrectly unlocking `must_follow` dependants. +- Signals use strict Pydantic schemas and schema versions. + +Those strengths should be preserved. The next work should focus on correctness and proof, not breadth. + +## Ranked priorities + +Effort estimates assume one maintainer already familiar with the code, include tests and docs, and exclude elaborate UI polish. + +| Rank | Improvement | Rough effort | Why now | +|---|---|---:|---| +| 1 | Make dispatch and session-stack recovery a real state machine | 2–3 weeks | There are crash/restart paths that can lose the active child or duplicate live work. This is correctness, not ergonomics. | +| 2 | Define typed execution, acceptance, and failure outcomes | 1–2 weeks | A normal harness return is currently equivalent to success even if the work is bad or workers failed. Failure strings are too lossy for policy. | +| 3 | Implement the event/usage/cost ledger, then a live status view | 2–4 weeks | `events.jsonl` is empty by design today. Without a canonical event stream, operations, debugging, budgets, and a TUI would all invent separate truth. | +| 4 | Replace the stock eval policy with deterministic-first evidence gates | 4–7 days | The current template's “only harness_judge; deterministic forbidden” rule is backwards for software delivery and unsafe as a stop gate. | +| 5 | Finish the operator experience: `doctor`, `run`, session-aware `status`, pause/resume, and force-stop | 1–2 weeks | The current CLI hides active children and cannot interrupt a running harness. It is too easy to operate the wrong session. | +| 6 | Version and validate configuration/runtime compatibility | 1–2 weeks | Model IDs and CLI contracts churn across three repos; state/config schemas have no explicit migration layer. | +| 7 | Refactor `CoordinatorService` around one transition function | 3–5 days | It lowers the defect rate for ranks 1–3. It is valuable, but refactoring alone is not the product priority. | +| 8 | Strengthen child-session contracts and budgets | 1–2 weeks | Child requests lack stable request IDs, per-child turn/cost/model budgets, explicit outcomes, and durable active-child ownership. | +| 9 | Add fault-injection and real dependency integration tests | 1–2 weeks initially, ongoing | The unit suite is strong on known branches but misses restart, race, corruption, disk, and live-CLI boundaries. | +| 10 | Reconsider multi-worker loopy-loop execution only after isolation exists | 4–8 weeks | Parallel work on one checkout would trade elapsed time for nondeterministic repository corruption and merge conflict. It is not the next bottleneck. | + +### 1. Make dispatch and session-stack recovery a real state machine + +The biggest immediate problem is that `/register` is both “give me work” and “proof that the previous worker is dead.” In `CoordinatorService.register_worker()`, any existing `current_task` is recovered from local artifacts or immediately recorded as `error="abandoned"`. A second accidentally started worker, a worker reconnecting during a network partition, or a worker still running while the coordinator is restarted can therefore cause duplicate work. The first worker may still be editing the same checkout when the second one receives a replacement task. Removing leases simplified v0.2.0, but it also removed the coordinator's basis for deciding that a task is actually orphaned. + +Do not restore the old distributed worker system wholesale. For the intended single-worker product, make the invariant explicit: + +- A normal `/register` while `current_task` exists returns `busy`/the existing assignment without mutating it, or returns HTTP 409 with an actionable message. +- Recovery is a separate explicit operation: `resume-task`, `force-abandon`, or coordinator startup recovery after a configurable grace period and process/heartbeat evidence. +- Add an `attempt_id` UUID. The current key `(session_id, workflow_id, iteration)` cannot distinguish a legitimate retry from the original execution. +- Persist `last_worker_contact_at`, execution PID/host when local, and a recovery decision with reason. Worker identity need not return as a scheduling abstraction; a per-attempt owner token is enough to prevent stale writers. + +Child recovery is more urgent. While a child runs, the active state is held only by `CoordinatorService.state_store` pointing at the child's `state.json`. The parent has `children.json`, but no durable `active_child_session_id`. After a coordinator process crash, a new `StateStore(repo_root=...)` chooses `latest_top_level_state_path()`, so `--resume` reopens the parent. The consumed child request is gone, the child can remain marked `running`, and the parent can dispatch new work. There is no test for restart during an active child. + +Persist the session stack in the root state (or a small root `runtime.json`) and make startup reconstruct and validate it. A transition should atomically mean “parent suspended on child X; child X is active.” On restart, terminal children should be finalized and the parent resumed; non-terminal children should remain active. `children.json` should be an index/projection, not the only bridge. + +Other recovery details to cover in the same work: + +- Write `children.json`, `result.json`, `pending_finished_request.json`, and control/eval artifacts atomically. Currently only `state.json` gets the temp-file/replace treatment. +- Give invalid child request files a terminal rejection record instead of silently leaving them in the directory. +- Enforce at most one outstanding child request rather than relying on prompts. +- Define what happens when the repository has changes from an abandoned attempt. Retrying is not automatically safe merely because the harness result file is absent. +- Add `state_schema_version` and migration code. Relying on Pydantic dropping old fields, as the v0.2.0 changelog describes, is not a durable resume strategy. + +### 2. Define typed execution, acceptance, and failure outcomes + +`run_harness_iteration()` distinguishes exceptions, but `_normalize_harness_result()` creates `IterationResult(success=True, ...)` for every normal `TeamHarnessResult`. That is only “the coordinator model stopped without raising.” It does not mean the requested change was completed, tests passed, or every worker succeeded. `TeamHarnessResult.agents` is discarded even though it contains statuses and exit codes. + +Replace the Boolean with at least three layers: + +1. `execution_outcome`: `completed | coordinator_error | worker_error | timed_out | cancelled | protocol_error`. +2. `artifact_outcome`: whether required files/signals/reports exist and validate. +3. `acceptance_outcome`: `accepted | rejected | inconclusive | not_evaluated`, set only by a configured verifier or deterministic contract. + +Scheduling should key off explicit policy. An implementation workflow may be mechanically complete but still require review; a failed worker may be tolerable if another worker delivered and verification passed; an empty coordinator final response should be inconclusive. Preserve the full agent summary and structured `TeamHarnessError.detail` in `IterationResult` and history. + +The failure taxonomy should include, at minimum: + +- coordinator auth/rate-limit/provider/transient exhaustion; +- worker binary missing, worker auth/API failure, non-zero exit, timeout, killed-on-finalization; +- malformed coordinator response/tool protocol; +- invalid/missing control, goal-check, child-request, or eval report; +- repository conflict/dirty-state policy violation; +- verification failed vs. verification unavailable; +- operator cancellation, stale attempt, forced abandonment; +- infrastructure unavailable and human-decision required. + +Each class needs retryability, backoff, maximum attempts, and whether it consumes the work package's turn budget. Today a generic harness exception becomes a failed iteration and the scheduler can retry until `max_turns`, spending money without distinguishing a transient 429 from a deterministic test failure. + +### 3. Implement events, usage, and cost before building a dashboard + +`sessions.create_session_dir()` creates `events.jsonl`, and the docs call it reserved, but no code appends to it. Fill that contract before choosing a TUI framework. Emit one versioned event per significant transition: + +- session created/resumed/paused/terminated; +- task selected/started/heartbeat/completed/accepted/rejected/abandoned; +- child requested/rejected/started/completed/parent-resumed; +- control and goal-check signal read/invalid/applied; +- retry/backoff and structured failure; +- operator update/stop/force-stop; +- git baseline/head/dirty summary; +- coordinator and worker model identities, provider/CLI versions, token usage, duration, and cost estimate. + +Events need an `event_id`, `schema_version`, UTC timestamp, root/child session IDs, task and attempt IDs, causation ID, and a small typed payload. Append under the same session lock; tolerate a truncated final line. Derive status views from state plus events, not from ad-hoc directory scanning. + +`team-harness` already records coordinator usage per turn in its `run.json`, but does not expose aggregate usage in `TeamHarnessResult`; worker CLI usage is provider-specific and often only present in stream output. Add a usage adapter with `known/unknown` fields rather than pretending all costs are measurable. Store price-table version and currency beside estimates because model prices change. Budgets should be enforceable at session, child, workflow, model, and wall-clock levels, checked before dispatch and after every harness result. Unknown usage should be visible and policy-configurable, never silently zero. + +Once the event model exists, a useful first UI is modest: + +- `loopy status --watch` with root/active-child tree, current attempt age, last event, spend, and next eligible workflow; +- `loopy events --follow --json` for automation; +- later, a Rich/Textual TUI reading exactly the same event/state APIs. + +A web dashboard is not justified yet. A local TUI covers the actual solo-maintainer deployment without adding another service and authentication surface. + +### 4. Change the eval template's philosophy + +The `inner_outer_eval/eval_reviewer` prompt says “Only create harness_judge checks” and “Deterministic checks are forbidden.” I disagree with this strongly. In software engineering, objective checks should dominate whenever possible: tests, static analysis, import boundaries, migrations, command exit codes, schema invariants, and golden-set metrics. LLM judges are useful for residual qualitative properties, not as substitutes for facts a program can establish. + +The stock policy should be: + +1. Run repository-owned deterministic contracts first. +2. Fail closed on missing, changed, or errored required checks. +3. Use a judge only for explicitly qualitative criteria. +4. Use a different model family from the implementing agent where practical. +5. Require repeated/consensus judgments for a terminal stop, or retain human approval for high-impact goals. +6. Prevent the implementation branch from weakening its own required checks without a separately reviewed change. + +For eval-banana, the eval runner should parse `report.json` itself and derive `goal_check.json`; the agent should not be trusted to paraphrase console output into the stop signal. A report path and hash should appear in the goal-check artifact. + +### 5. Finish the operator experience + +The existing `init`, `status`, and `stop` commands are a good skeleton but misleading around child sessions: + +- `status` uses the latest top-level state, so while the service is executing a child it can show the suspended parent and no current task. +- `stop` sets `stop_requested` on that top-level parent. It does not interrupt or even directly target the active child; the request only takes effect after the child eventually terminates and the parent resumes. +- A running `TeamHarness.run()` has no cancellation channel from loopy-loop. Stop is observed between iterations only. + +Add: + +- `loopy doctor`: Python/package versions, coordinator model capability, configured worker binaries/auth, eval-banana availability, git status, writable session directory, port availability, and config compatibility. +- `loopy run`: supervise coordinator plus one worker locally, handle signals, print the session ID, and keep the two-process mode available for remote execution. +- `loopy sessions list/show/tree`, with explicit `--session` everywhere. +- `loopy status --json/--watch`, including active child and stale-attempt diagnosis. +- `loopy pause` (finish current task, dispatch nothing), `resume`, `stop` (cooperative), and `stop --force` (cancel harness/process with a recorded outcome). +- `loopy validate` for workflow graphs, template dependencies, signal schemas, and required external tools without starting a session. + +The packaged `pm_planner_dispatcher` template currently tells the dispatcher to request workflow set `inner_outer_eval`, but `loopy init --template pm_planner_dispatcher` copies only the planner and dispatcher workflows. The child set is absent unless the user separately initializes/copies it. The current template preflight test checks only the parent. This should be fixed by bundling the child set or by making workflow-set dependencies declarative and installable; it is a concrete onboarding failure. + +### 6. Stabilize configuration without freezing model IDs + +Model names are repeated in `config.py`, `cli.py`, packaged YAML, README/docs, `.team-harness/config.toml`, `.eval-banana/config.toml`, and the two sibling projects. That guarantees drift. The answer is not a forever-stable default model name—there will not be one. + +Use named execution profiles, for example `balanced`, `high_assurance`, and `cheap`, resolved in one project-local file to exact provider/model/effort values. Scaffold a profile with comments and require `doctor` to validate it. Package defaults should prefer the provider/CLI's supported default or a centrally declared compatibility recipe, not duplicate a model string across templates and prose. Snapshot the resolved exact values into the session. + +Pin the critical dependency as `team-harness>=0.2.10,<0.3` while both projects are pre-1.0 and breaking changes are expected. Add a tested compatibility matrix and an adapter layer in loopy-loop, rather than constructor-signature introspection as the long-term compatibility policy. Eval-banana is not a hard dependency, so templates that require it must either declare/install an extra (`loopy-loop[eval]`) or fail preflight with a clear instruction. + +Also snapshot workflow definitions and prompt hashes. Root config is snapshotted today, but the worker reloads `prompt.txt` and per-workflow `config.yaml` from disk, and the coordinator reloads workflow definitions. A resume after prompt/scheduling edits is therefore not a replay of the original session. Either deliberately version/live-reload these files with change events, or store immutable copies in the session and make `--resume-with-current-workflows` explicit. + +### 7. Refactor coordinator duplication as part of the state work + +The repeated blocks in `register_worker()` and both stale/matched branches of `finish_assignment()` all do: + +1. apply stop conditions; +2. dispatch a requested child; +3. choose a workflow; +4. set `current_task`; +5. build a run response. + +Extract a single `_advance(state, cause, now) -> TaskResponse` after defining the state-transition table. Keep record/recovery parsing separate from advancement. This will remove roughly three copies of terminal/no-eligible/new-task behavior and make event emission occur once. I would not do a cosmetic extraction first: encode the new invariants and table-driven transition tests at the same time, or the refactor merely centralizes current ambiguities. + +### 8. Improve child sessions before allowing arbitrary nesting + +Depth-first, one-child-at-a-time is a sound v1 choice. It matches a shared checkout and makes parent review comprehensible. Do not add breadth-first child parallelism yet. + +What is missing is contract richness: + +- stable `request_id` and parent work-item ID; +- required acceptance artifact names; +- child `max_turns`, wall-clock, token/cost budget, execution profile, and retry policy; +- `requested | accepted | rejected | running | paused | terminal` lifecycle; +- explicit outcome and evidence bundle returned to the parent; +- durable active-child pointer and nested session stack; +- maximum depth and total descendant budget. + +The current child inherits root configuration through `_preflight_for()`; the request schema contains only `workflow_set`, `goal`, and `schema_version`. A PM parent cannot set a small child budget or a different verifier/model profile. That is a practical blocker for UGM. + +### 9. Testing gaps + +The 116 tests cover the intended scheduler and HTTP branches well, but confidence is concentrated in mocked, single-process happy paths. Add tests for: + +- coordinator restart while a child is running and after the child becomes terminal but before parent resumption; +- two `/register` calls while a task is live, late `/finished` from an old attempt, and network partition timing; +- stop/status while a child is active and force-cancellation during a harness run; +- crash/disk-full/partial JSON at every artifact handoff; +- workflow/config edits between start and resume; +- invalid and duplicate child requests; +- state migration from every released schema; +- real subprocess integration against a fake agent CLI, plus a scheduled smoke matrix for actual supported CLIs; +- team-harness returning failed/killed agents with a normal final response; +- property-based scheduler/state-machine invariants; +- packaging tests proving every scaffolded template can execute its first child/eval dependency from a clean temp repo. + +### 10. Do not prioritize parallel loopy workers + +This is where I disagree with the obvious “scale means more workers” answer. `team-harness` already provides parallelism inside an iteration. Loopy-loop coordinates stateful repository mutations in a single checkout. Two loopy assignments editing overlapping files, migrations, design status, or git branches would require worktree isolation, declared dependency/resource locks, deterministic merge/review, and conflict recovery. Reintroducing leases is the smallest part of that job. + +Parallel loopy workers become worthwhile only when tasks are explicitly read-only or allocated isolated worktrees/branches with a merge coordinator. UGM's early work packages are highly serial: schema, ports, worker substrate, and eval harness establish contracts consumed by everything else. The near-term throughput constraint will be human decisions and evidence quality, not worker occupancy. Keep one loopy assignment at a time and make team-harness's internal delegation observable and budgeted. + +# Q2. Is loopy-loop up for implementing UGM? + +## Verdict + +Not end to end, unattended. Yes for a deliberately narrow pilot with human-owned gates and deterministic acceptance. + +UGM is not merely a large coding task. It is a sequence of architecture, data-quality, infrastructure, model-selection, human-labeling, and operations decisions. The roadmap has nine numbered phases (0 through 8), dozens of work packages, a 2,375-line Postgres schema design, about 6,938 lines of design documents, 64 recorded decisions, and open decisions/spikes that deliberately block implementation. The current code is six executable statements plus two smoke tests. I ran its current checks: Ruff, formatting, pyright, and both tests pass, but there is effectively no product implementation. + +Loopy-loop can provide repetition and durable artifacts. It cannot turn unresolved product authority into an autonomous decision without violating UGM's own design. + +## The stock `pm_planner_dispatcher` is directionally right but insufficient + +The parent-planner/child-implementation shape is better than running `inner_outer_eval` over “build UGM.” One work package per child is the maximum sensible scope. The parent can review terminal child evidence and keep the larger roadmap out of the child's immediate context. + +I would not use the stock template unchanged: + +- Its child workflow dependency is not installed by the PM template, as noted above. +- The parent and child share one root config and therefore one `max_turns` and execution profile unless the runtime is extended. +- Work-item identity exists only in prompt-maintained Markdown, not in `ChildSessionRequest` or `children.json`. +- The planner is allowed to create/refine backlog items from a prose goal. UGM already has an authoritative roadmap; the agent must not invent an alternative backlog. +- The generic inner/outer prompts are extremely long and themselves instruct team-harness workers to create further agent teams, research, plan, review, execute, open PRs, and merge. Nested delegation can multiply cost and make responsibility unclear. +- The eval reviewer forbids deterministic tests, directly conflicting with UGM's contract-test and measured-golden-set philosophy. +- Parent stop/pause and active-child restart semantics are not strong enough for multi-day WPs. + +Build a custom `ugm_phase_driver` parent workflow set derived from the PM template, with these workflows: + +1. `gate_reconciler`: compare the phase entry gates and WP dependency/status projection against authoritative decisions and repository evidence; pause on unresolved human gates. +2. `wp_selector`: select exactly one already-defined eligible WP. It may narrow execution into PR-sized slices, but cannot alter acceptance criteria or mark gates resolved. +3. `dispatcher`: create a typed child request containing phase/WP ID, design reads, exact acceptance contracts, immutable check set/hash, and budgets. +4. `evidence_reviewer`: import child artifacts and independently run/inspect acceptance. Only this workflow updates WP status or accepted-completion records. + +The child should be a smaller `ugm_wp` workflow set, not the generic 160-turn loop: + +- `plan`: read only the WP's `Reads` list plus `concepts.md §0`, produce a bounded plan and surface design conflicts. +- `implement`: implement one PR-sized slice. +- `contract_eval`: run repository-owned commands and parse machine reports; `emits_goal_check: true`. +- `review`: inspect the diff and evidence with a different model family if an LLM review is useful. + +For small WPs, run the contract eval after every implementation success, not after ten successful `inner` iterations as the stock cadence does. + +## Where the model breaks on UGM + +### Human decisions are not failures + +UGM has legitimate decision gates: model seats, PageIndex deployment, budget/corpus assumptions, HA/observability, belief scope, hard-delete design, rename/CLA. Loopy-loop only understands terminal goal success, terminal unresolvable error, stop request, max turns, or scheduling failure. It needs a non-terminal `paused`/`waiting_for_human` state. + +Add a versioned `gate_request.json` contract: + +```json +{ + "schema_version": 1, + "gate_id": "llm-stage-models", + "blocks": ["WP-1.3"], + "question": "Which extraction model/profile is approved for the Phase 1 pilot?", + "options": ["..."], + "recommendation": "...", + "evidence_paths": ["..."], + "requested_at": "..." +} +``` + +The coordinator should stop dispatching, keep the session resumable, show the gate in `status`, and accept an operator decision with author/timestamp. Do not encode this as `unresolvable_error`; that makes a normal governance checkpoint look like a failed terminal session. + +### The roadmap is already stale enough to fool an autonomous planner + +The repository provides direct evidence of the danger: + +- `plan/plans/phase-0-foundations.md` still marks `WP-0.1` as `blocked(stack-conventions)`, while commits `d6abccf`, `cba62f4`, and `ec5ce3a` merged the package/tooling/CI scaffold into `main`. +- `roadmap.md` and `phase-1-walking-skeleton.md` still list embedding choice #3 as an entry blocker, while `questions.md` marks it resolved by D63 and `decisions.md` contains D63. +- `questions.md` says observability stack #10 should be decided “before the first worker,” while the roadmap gate table maps it to Phase 7. + +An agent following the phase file literally will stop on resolved work; an agent following git literally may declare a WP done without updating the authoritative status. Before automation, add a machine-readable status/gate projection and CI consistency checks. Markdown can remain canonical, but a parser should validate that resolved question references, gate register status, WP statuses, merged evidence links, and phase entry gates agree. The driver should fail closed on inconsistency and request reconciliation, not guess which document wins. + +### Infrastructure and money require policy boundaries + +Phase 0's reference path includes Postgres, GCP Cloud Tasks/Run, GCS/gcsfuse, and later Hetzner. A headless agent can write Terraform, compose files, adapters, and mocked/service-container tests. It should not silently create billable cloud resources, choose regions, modify production IAM, buy a Hetzner server, or decide a secrets policy. The stock prompts' broad permission language is too permissive for this project. + +For the pilot: + +- use the self-host profile and ephemeral Postgres/MinIO service containers; +- make cloud adapters contract-testable with fakes/emulators; +- require an explicit `external_action_request` for billable or destructive operations; +- pass credentials only through UGM's required `pydantic-settings`/`SecretStr` surfaces; +- prohibit agent-authored secrets and record every external mutation. + +### Golden sets need humans and protected data + +D22 explicitly requires human-adjudicated measurement labels and guards against circularity. Agents can propose pairs, create labeling tools, and run metrics. They cannot manufacture the gold labels and then claim measured quality. Real corpus data may also be unavailable or privacy-sensitive. + +The UGM repo should distinguish: + +- frozen, human-owned golden labels; +- synthetic test fixtures agents may create; +- agent-proposed labels awaiting adjudication; +- measured thresholds and the exact dataset/model/config version that produced them. + +Do not let an implementation child edit a required golden set, acceptance threshold, or eval definition in the same acceptance transaction. Changes to those assets need a separate review path. + +### Design conflict must pause implementation + +The roadmap correctly says that a WP requiring deviation from design is a design change and must stop. The child prompt must carry this as a hard output contract, not just prose. A `design_conflict.json` should name the WP, cited sections, conflict, options, and recommended amendment. The parent pauses or dispatches a design-only child; it never allows implementation to quietly choose an architecture. + +## What to add before a serious UGM run + +### In loopy-loop + +- Durable active-child/session-stack recovery and task attempt ownership. +- `paused/waiting_for_human` plus typed gate and external-action requests. +- Per-child turn, time, cost, model, and retry budgets. +- Structured outcomes and acceptance evidence, not Boolean harness success. +- Event/cost ledger and session-aware status/stop. +- Workflow-set dependencies so the PM template is executable from a clean init. +- Frozen workflow/prompt/check hashes per session. +- A direct deterministic verifier hook that does not require another LLM to translate reports. + +### In UGM + +- A reconciled gate register/WP status with evidence links to the already merged scaffold. +- A machine-validated projection of gates, WP IDs, dependencies, statuses, design reads, acceptance commands, and protected eval assets. +- `.loopy_loop/workflow_sets/ugm_phase_driver` and `ugm_wp` prompts that obey the roadmap rather than inventing a parallel plan. +- One goal file per phase or, preferably, per WP; never “implement UGM.” +- Exact `make`/`uv` acceptance commands and report formats for each WP. +- Local disposable infrastructure recipes before any reference-cloud execution. +- Human-decision and external-action inboxes with explicit ownership. +- A branch/worktree policy. One child at a time should own the checkout; evidence review should use the resulting commit/PR, not race it. + +## Recommended pilot + +Start with one Phase 0 work package, not all of Phase 0 and not the Phase 1 walking skeleton. First reconcile and close `WP-0.1` against the work already merged. Then use **WP-0.4 (ports plus import-linter contracts)** as the coding pilot: it is bounded, does not need billable infrastructure or golden labels, has a deterministic acceptance criterion (“an illegal import fails CI”), and exercises architecture, tests, review, PR delivery, and durable evidence. + +Do not start the Phase 1 walking skeleton until Phase 0's migrations, local queue/profile, and eval harness exist and the extractor model seat (#4) is explicitly chosen. Phase 1 is a good second pilot because it is vertically integrated, but it already requires Postgres, Lance, embedding configuration, extraction calls, and scenario contracts; using it first would make failures hard to attribute. + +An immediate single-WP root configuration could look like this using the current schema and the model name currently used by the checked-in templates (the exact model should be validated by `doctor`, not copied blindly): + +```yaml +goal_file: ".loopy_loop/goals/wp-0.4.txt" +workflow_set: "ugm_wp" +max_turns: 12 +goal_check_consecutive_failures_cap: 2 + +team_harness_provider: "codex" +team_harness_model: "gpt-5.5" +team_harness_agents: + - "codex" + - "claude" +team_harness_agent_models: + codex: "gpt-5.5" +team_harness_agent_reasoning_efforts: + codex: "high" + +team_harness_api_base: "https://openrouter.ai/api/v1" +team_harness_api_key_env: "OPENROUTER_API_KEY" +team_harness_system_prompt_extension: | + This session implements only WP-0.4. + plan/plans/phase-0-foundations.md and the WP Reads list are binding. + A design conflict pauses the session; do not improvise architecture. + Required checks and import contracts may not be weakened to obtain a pass. +``` + +The corresponding goal should name only WP-0.4, quote its deliverable and acceptance criteria, list exact allowed design reads, prohibit cloud mutations, and require `uv run ruff`, `uv run pyright`, `uv run pytest`, import-linter, diff review, and PR evidence. Configure `contract_eval` after every implementation success. Twelve turns is still generous; a healthy pilot should finish in fewer. + +If testing the parent/child design is itself the goal, use a parent capped around 8–12 PM turns and a child capped around 8–12 turns—but current loopy-loop cannot set those independently. Add per-child budgets first rather than giving a child the PM template's current `max_turns: 120`. + +# Q3. Documentation and an official loopy-loop launch + +## Launch position + +The README is strong for an Alpha project, the two contract documents are useful, the release pipeline uses PyPI trusted publishing, and the changelog is candid about v0.2.0's breaking API. That is enough for an early adopter release, not an “official launch.” The product still has a correctness gap in child resume, an incomplete PM scaffold, no live operational view, no cost story, and stale Agent Skill instructions. + +The Agent Skill is a pre-launch blocker. `skills/loopy-loop/SKILL.md` says state lives at `.loopy_loop/state.json`, describes “one or more blocking workers” polling HTTP, shows inline `goal`, and documents the removed `.loopy_loop/workflows//` layout. The current runtime requires a goal file, workflow sets, session-local state, and a single-worker ping-pong protocol. A skill that teaches agents an obsolete API is more damaging than missing documentation because it confidently generates broken setups. + +The README also says “one or more workers” even though `coordinator_app.py` is explicitly safe only under its single-worker assumption and a second register can abandon live work. Correct that claim before public promotion. + +## Missing documentation + +### A real end-to-end tutorial + +Create a copy/paste tutorial against a tiny maintained demo repository. It should show: + +- prerequisites and exact tested versions; +- install and `loopy doctor` output; +- init, the generated files, and the edits the user makes; +- coordinator/worker or `loopy run` terminal output; +- expected `status` output before, during, and after an iteration; +- the resulting session tree and how to follow harness/eval evidence; +- stopping, crashing, and resuming; +- expected token/time/cost range from measured runs; +- cleanup and common failure output. + +The current README explains components but does not let a newcomer compare their terminal to a known-good transcript. + +### Concept and architecture guide + +Explain the three nested control layers clearly: + +```text +loopy-loop session/workflow scheduler + -> one team-harness coordinator run per workflow iteration + -> external coding-agent CLI subprocesses + -> optional eval-banana checks as evidence +``` + +State which layer owns durability, retries, concurrency, cancellation, model selection, and success. Many users will otherwise assume “worker” means a team-harness worker or that loopy-loop resumes model conversations. Include the coordinator state transition diagram, crash windows, root/child session stack, and trust/security model. + +### Workflow authoring guide + +Document scheduling semantics with worked traces, not just field definitions. Include: + +- priority/tie behavior; +- `run_every`, `must_follow`, `run_on_start`, and `run_after_successes` interactions; +- failure/retry behavior; +- `emits_goal_check` vs. actual stop control; +- workflow-set dependencies and child request lifecycle; +- prompt ownership, artifact contracts, and idempotency rules; +- patterns for implement/review/eval, human gates, and read-only research workflows; +- anti-patterns: overlapping writers, self-authored acceptance, huge goals, and prompt-only locks. + +Ship a config schema/reference generated from Pydantic so prose does not drift. + +### Troubleshooting and recovery + +Cover missing binaries/auth, unsupported model IDs, API retries, invalid signals, stale tasks, dirty repos, corrupt state/artifacts, coordinator vs. worker crashes, active children, exhausted max turns, and how to decide between resume, retry, force-abandon, and a fresh session. Every error should point to this guide and the relevant artifact. + +### Cost, security, and operating expectations + +Publish measured ranges for each template: number of harness coordinator calls, likely spawned workers, eval frequency, typical wall time, and known unknowns in worker-token accounting. Explain that default worker templates use approval/sandbox bypass flags and therefore execute with the user's filesystem and credentials. Document safe use in disposable containers/worktrees, secret handling, network access, PR/merge permissions, and why untrusted repositories/prompts are unsafe. + +### Versioning and compatibility + +Declare which surfaces are public: + +- CLI commands/exit codes; +- root/workflow YAML schema; +- HTTP contract; +- session layout and event/signal schemas; +- Python SDK, if any; +- template semantics. + +For Alpha, say they may break on minor releases with migration notes. For Beta, commit to migrations for session state and deprecation for config/CLI. For 1.0, use SemVer for public surfaces, support reading at least the previous major session schema or provide an offline migrator, and publish a team-harness/eval-banana/CLI compatibility matrix. + +## Positioning against alternatives + +Do not position loopy-loop as a better coding agent, a general multi-agent SDK, or a distributed workflow engine. + +- Ralph-style loops already offer fresh-context iterations with git and small progress/PRD files. Their advantage is radical simplicity. The public [snarktank Ralph implementation](https://github.com/snarktank/ralph) explicitly uses git history, `progress.txt`, and `prd.json`; loopy-loop must justify its extra machinery with typed workflows, inspectable per-iteration artifacts, crash recovery, child sessions, and evidence gates. +- OpenHands provides a broad software-agent SDK, state lifecycle, remote workspaces, server isolation, and scaling. Its [official SDK overview](https://docs.openhands.dev/sdk/index) positions it as a production-ready agent engine for local or cloud use. Loopy-loop should present itself as a durable repository-level workflow controller that can call such engines/CLIs, not compete on agent runtime breadth. +- Native Codex/Claude/Gemini automation and CI coding agents own their ecosystems and UX. Loopy-loop's differentiation is model-agnostic orchestration across repeated fresh runs with repo-native state and user-authored workflow/eval contracts. + +A concise position would be: **“A durable, inspectable control loop for repository-scale agent work. Bring your coding agents; loopy-loop schedules repeatable workflows, preserves evidence, and resumes from files and git.”** Avoid “autonomous software engineer” claims until measured long-run evidence exists. + +## Concrete pre-launch checklist + +### Release blockers + +- [ ] Fix active-child restart/recovery and second-register semantics. +- [ ] Make stop/status session-stack aware and document cancellation limits. +- [ ] Introduce typed outcomes; do not equate normal team-harness return with accepted work. +- [ ] Populate versioned `events.jsonl` and expose basic usage/duration. +- [ ] Change stock evals to deterministic-first and parse report artifacts directly. +- [ ] Make `pm_planner_dispatcher` install/declare its `inner_outer_eval` dependency. +- [ ] Rewrite and test the Agent Skill against a clean generated repo. +- [ ] Remove all multi-worker claims unless/until they are true. +- [ ] Pin/test the pre-1.0 team-harness compatibility range and check eval-banana availability. +- [ ] Add a clean-install packaging/template smoke job in CI. + +### Documentation and proof + +- [ ] Publish the end-to-end tutorial with expected output. +- [ ] Publish architecture/state-machine, workflow authoring, troubleshooting, cost, and security guides. +- [ ] Create one demo repo with a pinned goal, deterministic checks, a redacted successful session bundle, and a short video/GIF. +- [ ] Publish at least three dogfood case studies: small task, crash/resume, and multi-WP parent/child; include elapsed time, tokens/cost coverage, failures, and human interventions. +- [ ] Run the UGM WP-0.4 pilot and publish the honest trace even if it fails. +- [ ] Document supported OS/Python/CLI/model combinations. + +### Project/community surface + +- [ ] Add `CONTRIBUTING.md`, `SECURITY.md`, code of conduct, support policy, issue/bug templates, and a minimal public roadmap. +- [ ] Enable GitHub Discussions or name one support channel; define response expectations suitable for one maintainer. +- [ ] Label good first issues only after contributor setup is reproducible. +- [ ] Add privacy/telemetry policy (prefer no telemetry by default). +- [ ] Automate stale docs/model-string checks and link validation. + +## Suggested sequencing and version labels + +1. **Hardening release (0.3.x Alpha):** state stack, typed outcomes, events, deterministic eval policy, PM template, skill correction. +2. **Dogfood cycle:** run several bounded real projects, especially UGM WP-0.4; fix failure modes and collect measured cost/time data. +3. **Documentation/demo release:** tutorial, architecture, workflow authoring, troubleshooting, security/cost, demo artifacts. +4. **0.4.0 Beta:** only after active-session migrations work, the event/outcome schemas are versioned, the compatibility matrix is CI-tested, and several multi-day resumes have succeeded. Beta means config/CLI/session changes get deprecations or migrators. +5. **Public “official launch”:** launch the Beta, not 1.0. Be explicit that it is a single-checkout, single-loopy-worker system with internal team-harness parallelism. +6. **1.0:** after at least a few months of Beta use, no known state-loss bugs, documented recovery/cancellation, stable public contracts, bounded spend controls, and a repeatable benchmark/case-study suite. Parallel loopy workers are not required for 1.0. + +# Q4. State of team-harness and eval-banana + +## team-harness 0.2.10 + +### Architecture and strengths + +The installed v0.2.10 package is about 8,673 Python LOC. I inspected that installed code directly. For test breadth I also used the nearby source checkout: it still declares 0.2.10 but contains a small unreleased Antigravity change, so its test count is evidence about the current development line rather than a byte-for-byte claim about the published wheel. The architecture is intelligible: + +- `harness.py` owns lifecycle/config/client/tool registration/finalization and exposes a small SDK. +- `coordinator/loop.py` implements the tool-call conversation, retries, usage/context tracking, and compaction. +- `agents/template.py`, `registry.py`, and `spawner.py` turn configured agent types into argv/env and launch them with `asyncio.create_subprocess_exec()`. +- `tools/agent_tools.py` exposes spawn/status/read/wait/kill and records worker metadata. +- `tracking/` writes run turns, agent sessions, failures, and resumability metadata. +- Per-run tool-binding closures isolate cursor state across concurrent SDK runs. + +Specific strengths in the actual code: + +- Worker subprocesses use argument arrays rather than a shell, avoiding a large injection class in the worker launch path. +- Worker labels are validated and resolved under the session output directory, a good fix documented in 0.2.10. +- Output reads are bounded, incremental cursor state is shared with wait snapshots, and worker stdout/stderr are preserved. +- Codex/Gemini/Claude session IDs are captured for explicit resume; unsupported resume is represented in the manifest. +- Coordinator 429/5xx/network failures get exponential backoff and structured `RunFailureRecord` diagnostics. +- Shutdown finalizes logs and writes a worker-session manifest even on exceptions. +- The patience policy, API failure classifier, context tracker, and prompt warnings show operational experience rather than a toy orchestration loop. +- The development source suite is substantial: 448 tests ran in the available environment. 446 passed; two Rich styling tests failed because expected bold ANSI codes were absent. Those are UI-only failures, but a release suite should still be green under declared dependency ranges. + +This is solid Alpha engineering. It is considerably more mature than its 8.7k LOC size implies. + +### Critical failure modes + +#### Normal return is not successful execution + +`TeamHarness.run()` returns `TeamHarnessResult` if the coordinator loop ends without `run_log.error`. Failed workers remain summaries; they do not automatically fail the harness. A coordinator can synthesize a final answer after a failed worker, or simply decide it has enough. That flexibility is reasonable for an orchestrator, but consumers must not interpret it as task acceptance. Loopy-loop currently does. + +The result needs an explicit orchestration outcome and policy fields such as `all_required_workers_terminal`, `failed_agents`, `killed_agents`, `empty_final`, and `coordinator_declared_complete`. Better yet, let the caller supply an acceptance callback/required artifact contract. + +#### Premature coordinator finalization kills live work + +The coordinator run ends whenever the model emits an assistant response with no tool calls. `_finalize_run()` then waits only `shutdown_timeout_s` (default 10 seconds) for running workers before terminating them. This conflicts with the system prompt's expectation that Codex tasks commonly take 20–45 minutes and should not be killed before 600 seconds. The minimum lifetime protects the model-visible `kill_agent` tool, but not finalization cleanup. + +A single premature final answer can therefore kill useful workers and still return a normal result. Enforce “no final while required agents run” in code, or return an explicit incomplete/error outcome. Prompt discipline is insufficient. + +#### No run budget or termination bound + +`coordinator.loop.run()` is an unbounded `while True` until the model stops making tool calls. There is no maximum coordinator turn count, wall-clock deadline, spawn count, concurrency cap, token budget, or cost budget. `bash` has a 120-second command timeout, but the orchestration run and worker agents do not have equivalent caller policy. One confused model can loop, overspawn, or wait indefinitely. + +These controls belong in team-harness so every caller benefits; loopy-loop's outer `max_turns` does not bound cost inside one harness iteration. + +#### Headless auto-compaction appears ineffective during a tool chain + +`_should_compact()` requires the last message role to be `user`. In a headless single-shot run, the initial user task is followed by assistant tool calls and `tool` results. Subsequent turns normally end in `tool`, so the compaction predicate will not run as the tool chain grows. Compaction is well tested for interactive/new-user boundaries, but the loopy-loop workload is a long single user turn. That path needs a dedicated test and a compaction strategy that can preserve an in-flight tool-call transcript safely. + +#### CLI integrations are a volatile compatibility surface + +Agent templates hard-code flags, session event shapes, model injection, and auth environment behavior for multiple independently changing CLIs. The abstraction is thoughtfully implemented, but unit tests mostly mock subprocesses and the one named integration test uses a fake coordinator API. A scheduled live smoke matrix is necessary. Missing binaries currently warn during template validation; a coordinator can still choose one and fail at runtime. + +#### Security is intentionally broad + +The built-in Codex/Claude/Gemini templates use sandbox/approval bypass modes. Coordinator shell and filesystem tools are also unrestricted, and a spawn `cwd` is resolved but not confined to the configured project. The system prompt says the coordinator is not an implementer, but tools do not enforce that boundary. This is acceptable for a trusted local power tool, not for untrusted repos or multi-tenant use. Documentation and optional workspace confinement are required. + +#### Tracking durability and code complexity + +`RunLogWriter._flush()` rewrites `run.json` directly rather than temp-file/replace, so a process or disk failure can corrupt the primary run log. `agent_tools.py` is 1,319 lines and contains both legacy global tool functions and near-duplicate per-run closure implementations; the duplicate `except asyncio.TimeoutError` in the global `wait_for_agents()` path is a small sign of drift. Refactoring around a per-run tool context would reduce risk. + +#### Cost visibility is incomplete + +Coordinator usage is recorded per turn and context limits are handled thoughtfully, but `TeamHarnessResult` exposes no usage aggregate and no cost. Worker usage is not normalized. For UGM, where every LLM worker must have an append-only ledger (D52) and budgets must be enforced, this is insufficient without additional instrumentation. + +### Test and maturity verdict + +The test volume is a genuine strength: config merging, command rendering, session capture, retries, compaction, output cursors, shutdown, manifests, and error rendering all receive attention. However: + +- tests are mostly unit/mocked integration; +- no cross-provider live CLI contract is demonstrated by the suite I ran; +- pyright is `basic` with many report categories disabled; +- dependency lower bounds have no upper bounds, and the two Rich failures demonstrate drift risk; +- v0.2.0 intentionally shipped breaking SDK/env changes, appropriate for Alpha but a warning to loopy-loop's unbounded dependency spec. + +**Is it solid enough for serious UGM work?** As a supervised delegation engine inside one bounded WP, yes. As an authority that says a WP is complete, no. Use it to obtain plans, diffs, reviews, and artifacts; use UGM's deterministic contracts and human decisions to accept them. Add run budgets and incomplete-worker enforcement before a multi-day autonomous pilot. + +## eval-banana 0.3.0 + +### Architecture and strengths + +Eval-banana is deliberately compact: about 2,030 source LOC. The v0.3.0 source suite has 115 tests, all of which passed in the available Python 3.13 environment. It has a clean pipeline: + +1. discover YAML checks; +2. parse strict discriminated Pydantic definitions (`extra="forbid"`, `schema_version: 1`); +3. select deterministic or harness-judge runner; +4. normalize binary results; +5. calculate threshold/error-aware `run_passed`; +6. write human and machine reports plus per-check stdout/stderr. + +Good decisions include fail-fast validation when a selected judge lacks a configured harness, path-specific load errors, duplicate-ID detection, explicit `error` distinct from `failed`, and making any errored check block `run_passed` even if the numeric threshold is met. The last-valid-JSON parser handles chatty agent stdout better than expecting a pristine response. + +For deterministic checks and reporting, it is a useful lightweight framework. It can serve UGM as a runner/report format, though UGM's pytest-native golden-set harness will be richer than a generic binary check list. + +### Judge reliability and pitfalls + +`harness_judge` is not reliable enough to be the sole stop condition of a long-running coding loop. + +#### Non-determinism and no calibration + +Each check gets one binary verdict from one agent invocation. There is no seed/temperature surface, repetition, majority vote, confidence, disagreement record, calibration set, or flakiness history. Binary scoring is easier to reason about than a 1–5 scale, but it does not make a single stochastic judgment deterministic. + +For release gating, run qualitative checks multiple times or across model families and record disagreement. Calibrate each judge prompt against human-labeled pass/fail examples before trusting it. + +#### The judge can modify what it judges + +The coding agent runs in the project root with built-in approval/sandbox bypass flags. The prompt asks it to read and evaluate, but no code makes the workspace read-only. A judge can modify implementation, tests, or eval definitions before returning a verdict. Even accidental formatter/fix behavior compromises the measurement. + +Run judges in a read-only snapshot/container or detached worktree and verify the tree hash before/after. Network and secrets should be minimized. This is essential if the verdict can stop an autonomous loop. + +#### Prompt injection and gaming + +The agent reads repository files under natural-language instructions. Repository content can instruct it to ignore the evaluator or emit a passing JSON object. More subtly, the same implementation agent can weaken checks or tailor prose to the judge. The loopy-loop stock eval reviewer magnifies this by letting an agent author the high-level checks that later decide completion. + +Keep required check definitions outside the implementation's writable scope or require independent review of check changes. Prefer behavioural tests over judging self-authored claims. Use a different model family for judging and include adversarial calibration cases. + +#### Exit-code handling is too permissive + +`run_harness_judge_check()` parses stdout even when the agent exits non-zero; the test `test_non_zero_exit_with_valid_json_still_passes` explicitly codifies that a valid `score: 1` can pass despite exit code 1. This salvages useful final output, but for a gate it can turn a crashed agent's stale/partial verdict into success. A non-zero exit should be at least `inconclusive/error` by default, with an opt-in salvage mode. + +#### Judge provenance is incomplete + +The check result records model, agent type, raw stdout, and exit code, but not the exact rendered prompt hash, agent CLI version, template/config hash, repository commit/tree hash, environment identity, or judge repetition. Those are necessary to reproduce and audit a terminal decision. + +#### Dead/stale configuration and documentation + +v0.3.0 removed `target_paths`, instructing the real coding agent to find files named in `instructions`. `Config.llm_max_input_chars` remains parsed and documented as a per-target-file truncation control, but it is not used anywhere in the judge runner. The `harness_judge` docstring also says it builds a prompt from target-file contents, while `_build_judge_prompt()` contains only description and instructions. This is concrete API/docs drift. + +#### Execution limits + +Judge subprocesses have a hard-coded 300-second timeout, not a per-check/configurable value. Deterministic checks have no timeout at all; one hung script blocks the entire sequential run. Checks run serially, so a qualitative suite can be slow and expensive. Add per-check timeouts, a safe global limit, and controlled parallelism for independent read-only checks. + +#### Equal weighting and thresholds + +Every check has equal one-point weight. A `pass_threshold` below 1.0 can allow a critical check to fail as long as enough minor checks pass. Introduce required/critical checks or named suites; terminal loop completion should require all mandatory contracts regardless of aggregate threshold. + +### Recommended role in loopy-loop and UGM + +Use eval-banana for: + +- orchestrating deterministic scripts and normalizing reports; +- lightweight qualitative smoke checks; +- preserving stdout/stderr and a simple pass/fail artifact; +- developer feedback and regression trends. + +Do not use one harness-judge pass as a stop switch. For UGM, acceptance should be layered: + +1. required pytest/import/migration/schema/golden-set metrics pass; +2. required eval-banana deterministic checks pass; +3. qualitative judge checks, if any, pass under an independent, read-only, calibrated policy; +4. human resolves explicit architecture/model/infra/golden-label gates; +5. the loopy-loop verifier derives the stop artifact from those reports. + +## Foundation risk across the three tools + +The stack has good conceptual separation: + +- loopy-loop supplies long-horizon durable workflow/session state; +- team-harness supplies short-horizon model/worker orchestration; +- eval-banana supplies check discovery and report artifacts; +- git and project-owned tests remain the durable implementation evidence. + +The coupling is nevertheless risky. All three are Alpha, maintained by one person, share hard-coded model/CLI assumptions, and can evolve together without an external compatibility constraint. Loopy-loop currently depends on `team-harness>=0.2.10` with no upper bound even though team-harness has already made breaking pre-1.0 changes. Eval-banana is conventionally required by the recommended template but not installed or validated as a dependency. Integration tests do not prove a clean installed trio executes a full template with real CLIs. + +Treat the trio as one release train for now: compatibility matrix, pinned ranges, end-to-end clean-environment test, shared outcome/event vocabulary, and coordinated release notes. Decouple later when the contracts have survived real use. + +# Bottom line + +The bet is sound in its restrained form: fresh agent contexts plus durable repository state, explicit workflows, git, and independently verified artifacts are a credible way to extend useful coding work beyond one chat. The biggest risk is **false closure**—agents and harnesses producing plausible completion prose, altered evals, or mechanically successful runs that the system promotes to accepted work. It is not primarily the single-worker ceiling or context-window size. The single highest-leverage next move is to run one UGM Phase 0 work package—preferably WP-0.4—through a hardened, single-worker pilot whose acceptance tests are frozen and deterministic, while recording every transition, failure, token/cost observation, human intervention, and final diff. Use that evidence to define the shared outcome contract across loopy-loop, team-harness, and eval-banana before investing in parallelism or a polished dashboard. diff --git a/design/analysis/loopy-loop-state-and-roadmap.md b/design/analysis/loopy-loop-state-and-roadmap.md new file mode 100644 index 0000000..4fd8087 --- /dev/null +++ b/design/analysis/loopy-loop-state-and-roadmap.md @@ -0,0 +1,445 @@ +# loopy-loop — State, Roadmap, and the ugm Question + +**Author:** Claude (Opus 4.8), as co-maintainer +**Date:** 2026-07-12 +**Scope:** Future improvements · fitness for building `ugm` · documentation & launch · the state of `team-harness` and `eval-banana` + +> A second, independent analysis was run in parallel by Codex (gpt-5.6-sol, xhigh +> reasoning) and lives beside this file at +> [`codex_parallel_analysis.md`](./codex_parallel_analysis.md). Read both; where +> they disagree, that disagreement is signal. +> +> **⚠️ Read [`synthesis.md`](./synthesis.md) first.** The parallel pass verified +> several concrete in-code defects this doc reasoned around, and corrected two of +> my claims. Specifically: (1) the biggest risk is **false closure** +> (`harness_runner.py:190` marks every non-raising harness run `success=True`), not +> maintainer concentration as I state below; and (2) §4 below praises eval-banana +> for mixing deterministic + judge checks — true of the *framework*, but the +> shipped `inner_outer_eval` template **forbids deterministic checks** +> (`eval_reviewer/prompt.txt:56`), which inverts the point. The synthesis reconciles +> both passes into one action plan; this doc is my independent first pass, kept as a +> record. + +--- + +## Executive summary + +loopy-loop is a genuinely well-built alpha: durable file-based state, a pure +scheduler, atomic writes under a file lock, schema-versioned signal files, +explicit crash recovery, 116 fast tests, clean CI. The code is not the problem. +The gaps are **operational** (no live observability, thin failure taxonomy, no +cost/token accounting) and **strategic** (single maintainer, dormant since late +May, single-worker ceiling, and a dependency surface — `team-harness`, +`eval-banana`, hardcoded model IDs — that churns). + +On the headline question: **loopy-loop is architecturally a strong fit for +driving `ugm`, but only for the code-and-contract-test slices of the work.** `ugm` +is a design-first, 8-phase system whose own roadmap is explicitly gated on human +decisions and measured golden-set spikes, and whose Phase 0 needs real infra +(Postgres, queues, object store). A headless agent loop can implement work +packages and pass deterministic contract tests; it cannot make the embedding-model +decision, stand up Hetzner Postgres, or manufacture a labeled golden set. The +`pm_planner_dispatcher` template is the right shape, but loopy-loop is **missing a +first-class human-in-the-loop pause state**, and that is the single most important +thing to add before pointing it at `ugm`. + +--- + +## 1. Future improvements (ranked) + +Ranked by leverage-per-effort. "Effort" is rough: S = a day or two, M = a week, +L = multi-week. + +### Tier 1 — do these before any serious autonomous run + +**1.1 First-class human-in-the-loop pause state. (M) — highest leverage** +Today a workflow that hits a decision it can't make has two options: fail the +loop (`control.json` → `unresolvable_error`) or guess. The planner prompt says +"mark blocked" but the loop has no `paused_awaiting_human` status — a blocked +item either stalls silently or stops the whole session. Add a real terminal-ish +state: `status: "paused"`, `stop_reason: "awaiting_human"`, with a structured +`questions_for_human.md` / `pending_decisions.json` the coordinator surfaces via +`loopy status`, plus a clean `--resume` that re-reads `updates_from_user.md` and +continues. This is the gating capability for `ugm` (see §2) and useful for every +non-trivial goal. + +**1.2 Live observability: `events.jsonl` + `loopy watch`. (M)** +`events.jsonl` is documented as "reserved" but not written. Right now the only +way to see what a running loop is doing is to tail files by hand. Implement the +append-only event stream (iteration start/finish, workflow chosen, stop-reason, +control/goal-check transitions, child spawn/return) and a `loopy watch` TUI +(rich already a dependency) that renders the session timeline, current task, +history, and last result. For long-horizon runs this is the difference between +"trust the loop" and "babysit the loop." + +**1.3 Cost & token accounting. (S–M)** +There is no record of what a session *cost*. team-harness has a `tracking/` +module — surface its per-iteration token/cost data into the iteration directory +and aggregate it into `session.json`. Add a `max_cost_usd` stop condition +alongside `max_turns`. Without this, an autonomous multi-day `ugm` run is an +uncapped spend. + +### Tier 2 — robustness and clarity + +**1.4 Richer failure taxonomy. (S)** +Failure currently collapses to `abandoned`, `invalid_goal_check_output`, +`invalid_control_output`, `goal_check_broken`, `no_eligible_workflow`, +`unresolvable_error`. Missing: distinguishing *transient* worker/API failure +(retryable) from *deterministic* workflow failure (a prompt that always fails), +and a consecutive-failure circuit breaker per workflow (analogous to +`goal_check_consecutive_failures_cap`) so a wedged `inner` doesn't burn all +`max_turns`. Add `stop_reason: "workflow_failure_cap"`. + +**1.5 De-duplicate the coordinator mutators. (S)** +`coordinator_app.py` repeats the "check stop → maybe dispatch child → choose next +→ set current_task → build run response" sequence three times (`register_worker`, +the `finish_assignment` no-current-task branch, and the matched branch). Extract a +single `_advance(state, now) -> TaskResponse` helper. It's ~706 lines today; this +is the one spot where drift will cause a real bug, because the three copies must +stay behaviorally identical. + +**1.6 Config: stop hardcoding model IDs in templates. (S)** +Templates pin `gpt-5.5`, `claude-opus-4-8`, `gemini-3.5-flash`; git history is +already full of "bump gpt-5.4 → 5.5" commits. Introduce a `models:` alias block +(e.g. `strong`, `cheap`, `judge`) resolved once, and/or read defaults from +environment, so a model rev is one edit, not N. Document the supported IDs in one +place with a "last verified" date. + +### Tier 3 — scaling and reach (only if the use case demands it) + +**1.7 Parallel workers — deliberately *not* now. (L)** +v0.2.0 intentionally removed leases/worker-identity for a clean single-worker +model, and much of the recovery logic is commented "safe in the single-worker +model." Re-introducing concurrency is a redesign of the state machine, not a flag. +`ugm`'s child-session model is depth-first anyway, and one honest worker per +session is easier to reason about. **Recommendation: keep single-worker; instead +invest in child-session breadth (below).** + +**1.8 Child sessions: breadth and durability. (M–L)** +v1 is depth-first, one child at a time, and a child's `parent_session_id` linkage +plus `_resume_parent_if_active_child_completed` is the fragile seam. For `ugm`, +one-WP-at-a-time is actually fine (see §2), so this is lower priority than it +looks — but if PM throughput ever matters, allowing N independent children with a +join barrier is the natural next step. Add tests for parent/child crash recovery +*across* the boundary first; that path is under-tested relative to its risk. + +**1.9 DX polish. (S, ongoing)** +`loopy status` should show cost, current workflow, and pending human questions; +`loopy stop` should be graceful (finish current iteration) vs `--now`. A +`loopy doctor` that validates config + workflow graph (`must_follow` / +`run_after_successes` referential integrity is checked at preflight — surface it +as a standalone command) would catch template mistakes before a run. + +### Testing gaps worth closing +- Parent↔child crash recovery across the boundary (highest-risk under-tested path). +- `run_after_successes` cadence interacting with `must_follow` (combinatorial; + the scheduler is pure — property-based tests would pay off here). +- Concurrency: there's a file lock but no test that simulates two workers racing + `/register` (even if unsupported, it should fail safely, not corrupt state). + +--- + +## 2. Is loopy-loop up for building `ugm`? + +**Short answer: yes for the code, no for the whole thing unaided — and that's fine +if you scope it right.** Use `pm_planner_dispatcher`, but add a human-gate and +pilot on one phase. + +### Why the fit is genuinely good + +`ugm` is almost eerily well-matched to the pm/dispatcher model: + +- Its roadmap is already **work packages (WP-N.x) that are "pointers with + contracts": one-line goal, a minimal reading list, dependencies, a deliverable, + and acceptance criteria drawn from the designs' own contract tests.** That is + *exactly* the shape of a good child-session goal. You barely have to translate. +- Acceptance is tied to **deterministic contract tests** (import-linter dependency + arrows, grain CI, envelope invariants, canaries, schema migrations). A child + `inner_outer_eval` session can implement a WP and an `eval_runner` can *actually + decide* pass/fail on these — no LLM judgment required for the gate. +- The whole project is **designed to be executed by coding agents** "after reading + only the documents named in its *Reads* column." Someone already did the hard + work of making this agent-consumable. +- Continuity-in-files matches `ugm`'s own "designs say how, plans say order, + git is truth" philosophy. + +The planner → dispatcher → child(`inner_outer_eval`) loop maps cleanly: +`planner` maintains PM state mirroring the phase file's WP status table; `dispatcher` +turns the selected WP into a child goal (goal text = WP goal + Reads list + +acceptance criteria); the child implements and self-evaluates against the contract +tests; `planner` reviews the child's evidence (PR, CI status, eval report) and +marks the WP `accepted` or `needs_rework`. + +### Where it will break — be honest about these + +1. **Human-decision gates.** `ugm`'s roadmap has an explicit *gate register*: + embedding model + dimension (blocks Phase 1 entry, "hardest-to-change choice in + the system"), LLM-per-stage, PageIndex hosted vs self-hosted, HA/observability + choices, and an owner-provided "stack conventions" slot that **blocks Phase 0 + WP-0.1 today**. A headless loop must not guess these. Without the pause state + from §1.1, the planner will either stall or hallucinate a decision. **This is + the blocking gap.** + +2. **Infra standup.** Phase 0 needs Postgres + Alembic migrations + queues + (`LISTEN/NOTIFY` + `SKIP LOCKED`), Phase 1 pulls in LanceDB, later phases GCS, + LadybugDB, PageIndex. An agent can *write* the migration and the adapter code; + it cannot reliably provision a database, hold cloud credentials, or run a real + `gcsfuse` mount from inside a sandboxed `codex` subprocess. Anything whose + acceptance requires a live external service will fail the eval even when the + code is correct. + +3. **Measured spikes.** "The eval harness precedes everything tunable" — nearly + every design defers its numbers to golden-set measurement (D17 thresholds, D35 + recall, adjudicator gates). Those WPs need labeled data, real models, and money, + and their acceptance is a *number a human interprets*, not a green test. These + are human-run experiments, not autonomous work packages. + +4. **Cross-WP coherence / architectural drift.** The planner reviewing one WP at a + time can lose the forest. `ugm` mitigates this with import-linter and contract + CI (drift shows up as a failing build), which is a real safety net — but the + planner prompt should be told to treat a full-suite CI run, not just the WP's + own tests, as part of acceptance. + +### Concrete recommendation + +**Pilot, don't boil the ocean. Drive Phase 0 + the Phase 1 walking skeleton, +agent-executing only the code-and-deterministic-test WPs, with humans owning gates +and infra.** Specifically: + +**A. Unblock and pre-decide before you start.** +- Fill `ugm`'s "stack conventions" slot yourself (uv, ruff, pyright strict, GitHub + Actions, secrets handling) so Phase 0 WP-0.1 isn't `blocked(stack-conventions)` + on turn one. It's already the house style — write it into a decision. +- For the pilot, use a **local docker-compose Postgres** and a **local MinIO** so + the agent has real-but-disposable infra to test against; provide connection + details in the session config / a `.env` the workers can read. + +**B. Add the human-gate mechanism (§1.1).** Until it exists, give the planner an +explicit contract: *"If a WP is `blocked()` in the roadmap and the gate is +unresolved in `decisions.md`, do NOT dispatch it. Mark it `blocked`, write the +exact decision needed to `questions_for_human.md`, and select the next unblocked +WP. If no unblocked WP remains, stop with `unresolvable_error` and a clear +summary."* Then the human answers, updates `decisions.md` + `updates_from_user.md`, +and `--resume`s. + +**C. Wire the goal, don't restate it.** The session goal file should point at the +phase, not re-explain the system: +> "Implement Phase 0 of ugm per `plan/plans/phase-0-foundations.md`. Source of +> truth: the designs it references. Complete WPs in dependency order. A WP is done +> only when its acceptance criteria pass in CI (import-linter arrows, migrations +> apply cleanly, contract tests green). Do not implement WPs marked `blocked(...)` +> whose gate is unresolved in `decisions.md` — surface them for human decision." + +**D. Sketch config for the pilot** (`ugm/loopy_loop_config.yaml`): +```yaml +goal_file: loopy_loop_goal.txt # points at Phase 0, per (C) +workflow_set: pm_planner_dispatcher +max_turns: 120 +goal_check_consecutive_failures_cap: 3 +team_harness_provider: "codex" +team_harness_model: "gpt-5.5" # planner/dispatcher coordinator +team_harness_agents: ["codex", "claude"] +team_harness_agent_models: + codex: "gpt-5.5" + claude: "claude-opus-4-8" +team_harness_agent_reasoning_efforts: + codex: "high" +# add when §1.3 lands: +# max_cost_usd: 50 +``` +Child sessions run `inner_outer_eval`; each child goal = one WP with its Reads +list and acceptance criteria pasted in. + +**E. Success criterion for the pilot itself:** can the loop take Phase 0 from +scaffolding to "migrations apply + import-linter passes + the harness skeleton +runs" with the human only answering gate questions and provisioning infra? If yes, +graduate to Phase 1. If the planner keeps needing rescue on non-gate work, the +problem is prompt/eval design, not the phase — fix that before scaling. + +**Bottom line for §2:** loopy-loop is up for the *implementation* of `ugm`, phase +by phase, if you (1) add the human-gate pause, (2) keep infra + measured spikes in +human hands, and (3) lean on `ugm`'s already-excellent contract-test acceptance as +the eval gate. Do not point it at "build all of ugm" and walk away. + +--- + +## 3. Documentation & official launch + +loopy-loop is already on PyPI (0.2.1, Alpha) with a strong README, an HTTP +contract doc, a session-layout doc, an Agent Skill, and a real CHANGELOG. That's a +better starting position than most "launched" tools. What's missing is the +**narrative middle** between "here's the reference" and "here's how I actually use +this." + +### What's missing (in priority order) + +1. **An end-to-end tutorial with real expected output.** Pick one small, real + goal (e.g. "add a `/health` endpoint with a test"), run it start to finish, and + show the actual session directory that results — the iterations, the + `goal_check.json`, the stop. The README explains the machinery; nobody has seen + it *move*. This is the single highest-value doc. +2. **A concepts / architecture page.** One diagram: coordinator ↔ worker ↔ + team-harness ↔ agent CLIs, and the session-directory as the durable spine. + Explain *why* file-state-not-chat, the two-endpoint contract, and the + single-worker model as a deliberate choice. This is where you win over the + skeptical reader. +3. **"Write your own workflow set."** The scheduler semantics (`priority`, + `run_every`, `must_follow`, `run_after_successes`, `emits_goal_check`, + `run_on_start`) are powerful and non-obvious. A cookbook — "how do I express + 'run X after every 10 successes of Y'", "how do I gate a stop on an eval" — + turns the config reference into something people can build with. +4. **Troubleshooting / operations.** What `stop_reason`s mean and what to do about + each; how to resume; how to read a failed iteration; how to interpret + `goal_check_broken`; expected cost ranges. Ties directly to §1.2/§1.3. +5. **A demo/example repo.** A tiny target repo pre-initialized with a template and + a goal that a reader can clone and `loopy coordinator` against, plus a recorded + run. Removes all setup friction. + +### Launch risks to close first + +- **Model-ID churn (§1.6) is a launch-day footgun.** A new user who installs today + and hits a decommissioned `gpt-5.5` sees a broken tool. Alias the models and + document supported IDs with a verified date before you promote it. +- **The team-harness / eval-banana coupling is under-explained.** A launched + loopy-loop implies "these work." Pin known-good versions, and state plainly in + the README what auth/keys each needs (Codex login vs `OPENROUTER_API_KEY`). +- **No stability promise.** Alpha is honest today. Decide what "beta" means + (config schema frozen? HTTP contract frozen?) and put a one-line stability policy + in the README so early adopters know what can change under them. + +### Suggested sequencing + +1. **0.3 (docs + safety):** tutorial, concepts page, model-alias config, + troubleshooting, pinned dependency versions. No API changes. → announce as a + *soft* launch (a blog post / README badge), still labeled Alpha. +2. **0.4 (observability):** `events.jsonl` + `loopy watch` + cost tracking (§1.2/ + §1.3). This is what makes a public demo compelling. +3. **0.5 → beta:** freeze the config schema and HTTP contract, publish the + stability policy, ship the demo repo, write the "workflow set cookbook." + *Beta = "we won't break your config without a major bump."* +4. **1.0:** only after a real external project (ideally `ugm`, or a public pilot) + has been driven end-to-end and you can point to it. **Don't call it 1.0 until + something real was built with it.** That artifact *is* the launch. + +The most credible launch is not a version number — it's "here is a non-trivial +system that loopy-loop built, here's the session directory, here's what it cost." +`ugm`'s Phase 0/1 pilot could be exactly that story. + +--- + +## 4. State of team-harness and eval-banana + +The user asked for detail here, because these are the foundation loopy-loop stands +on. (Codex's parallel report reads their source independently — cross-check.) + +### team-harness (v0.2.10, ~8.7k LOC, Alpha) — the critical dependency + +**What it is:** a model-agnostic orchestration harness where a *coordinator LLM* +spawns external worker CLIs (Codex, Gemini, Claude Code, opencode, pi, OpenHands) +as tool-use actions. Structure is sensible and modular: `coordinator/` (loop, +client, `codex_client`, auth, system_prompt, protocols), `agents/` (manager, +registry, spawner, `session_capture`, `api_error_classifier`, template), plus +`tools/`, `tracking/`, `ui/`, `skills/`. It supports both an OpenAI-compatible +coordinator and an experimental `codex` provider. + +**Strengths:** +- Clear separation between the coordinator loop and the worker-CLI spawning; a + registry/spawner pattern makes new agent CLIs pluggable. +- An explicit `api_error_classifier` and retry budget (`max_retries`, and + loopy-loop exposes `team_harness_max_retries` / backoff) — transient-error + handling is a first-class concern, which is exactly what a long-horizon loop + needs. +- A `tracking/` module exists — the raw material for the cost/token accounting + loopy-loop should surface (§1.3). +- Same house style as loopy-loop (Pydantic, typed, ruff/pyright, Apache-2.0), + which lowers the cost of fixing it when something breaks. + +**Risks / concerns:** +- **It is the single biggest external risk.** loopy-loop pins + `team-harness>=0.2.10`, and its whole value proposition ("run assignments + through a harness that spawns agent CLIs") is delegated here. A regression or a + breaking change in team-harness breaks loopy-loop silently. For a serious `ugm` + run, **pin an exact known-good version**, not `>=`. +- **The worker CLIs are themselves moving targets.** team-harness shells out to + `codex`, `claude`, `gemini` binaries whose flags, auth, and model names change + (loopy-loop's own Makefile shows `codex --yolo --model gpt-5.6`). The harness is + only as stable as the least stable CLI it wraps. +- **`max_depth = 3` recursion** (coordinator agents spawning th-run agents) is + powerful and a place where cost and confusion compound; for autonomous runs, + understand and bound it. +- Alpha, same solo maintainer, same bus-factor as loopy-loop. The three repos rise + and fall together. + +**Verdict:** solid enough in architecture to build `ugm` on, *if* you pin versions +and treat the worker-CLI layer as the fragile edge. It is the right abstraction; +its risk is operational stability, not design. + +### eval-banana (~0.3.0, actively developed) — the loop's judgment + +**What it is:** a lightweight YAML eval framework. Deterministic checks plus +`harness_judge` (LLM-as-judge, run via a coding agent — `codex`/`claude`/etc.), +writing `report.json` / `report.md` with a configurable `pass_threshold` +(default 1.0 = every check must pass). Used by loopy-loop's `inner_outer_eval` +template as the mechanism that produces `goal_check.json`. Notably it is **not a +hard dependency** of loopy-loop — it's a convention the template adopts — which is +good decoupling. + +**Strengths:** +- Actively versioned (0.0.2 → 0.3.0), the most alive of the three by release + cadence. +- The mix matters: **deterministic checks for things that can be checked + deterministically, `harness_judge` only for things that genuinely need + judgment.** That's the correct design — and it's why `ugm` is a good fit, because + most of `ugm`'s acceptance is deterministic contract tests that don't need the + judge at all. +- `pass_threshold` and truncation controls are sensible operational knobs. + +**Risks / concerns — especially for gating a loop's stop condition:** +- **LLM-as-judge is non-deterministic and gameable.** If `harness_judge` decides + whether a loop *stops*, a flaky judge either stops too early (declares a goal met + that isn't) or never (burns `max_turns`). loopy-loop already anticipates the + second failure mode with `goal_check_consecutive_failures_cap` / + `goal_check_broken`, which is the right instinct — but the *false-positive* case + (judge wrongly says "done") is the dangerous one and has no guard. +- **Cost.** Every `harness_judge` check is an agent invocation. In a loop that + evaluates every N iterations over a growing artifact, judge cost is a real line + item — reinforcing §1.3. +- **The judge shares a model family with the implementer**, which invites + correlated blind spots (the same model that wrote the code judging the code). + `ugm`'s own design mandates *cross-family checkers* (D53) for exactly this + reason — mirror that in eval config: judge with a different family than you + implement with. + +**Verdict:** fine as a *component* of the stop decision, dangerous as the *sole* +gate. The loopy-loop design already agrees — "a valid `goal_check.json` does not +stop the loop by itself; it is evidence; stopping is controlled by `control.json`." +Keep it that way: use `eval-banana` deterministic checks as hard gates, use +`harness_judge` as advisory evidence a human or the planner weighs, and never let +a single LLM judgment flip a loop to `goal_met` on high-stakes work. + +--- + +## Cross-cutting read: is the bet sound? + +These three tools plus `ugm` are a coherent bet on **autonomous long-horizon agent +coding**: durable state (loopy-loop) + multi-agent execution (team-harness) + +objective acceptance (eval-banana / contract tests), pointed at a large, already +agent-consumable design (`ugm`). + +- **The bet is sound in shape.** The insight that long-horizon agent work should be + durable, file-based, and gated on *objective* acceptance rather than a chat + transcript is correct, and it's better-engineered here than in most attempts. +- **The single biggest risk is not the code — it's the concentration.** One + maintainer, three interlocking alpha repos, all dormant since late May, plus a + worker-CLI/model-ID layer that churns weekly. The system's stability is bounded + by the least stable moving part, and there's no second person to catch drift. +- **The single highest-leverage next move** is the §1.1 human-in-the-loop pause + state, executed as the enabling step for a **narrow `ugm` Phase 0/1 pilot**. That + one capability unblocks the flagship use case, produces the artifact that makes + the launch credible (§3), and stress-tests team-harness/eval-banana on real work + — three birds. Everything else (observability, cost, dedup) is in service of + making that pilot legible and safe. + +**One-sentence bottom line:** the foundation is real and well-built; the work now +is operational hardening (pause/observe/cost) and proving it on one honest slice +of `ugm`, not adding capability for its own sake. diff --git a/design/analysis/synthesis.md b/design/analysis/synthesis.md new file mode 100644 index 0000000..1081b9a --- /dev/null +++ b/design/analysis/synthesis.md @@ -0,0 +1,274 @@ +# Synthesis — Two independent analyses, reconciled + +**Date:** 2026-07-12 +**Inputs:** +[`loopy-loop-state-and-roadmap.md`](./loopy-loop-state-and-roadmap.md) (Claude / Opus 4.8) +and [`codex_parallel_analysis.md`](./codex_parallel_analysis.md) (Codex / gpt-5.6-sol, xhigh). + +This is the capstone. Read it first, then the two source analyses for depth. It +records (1) what the parallel pass found that the first pass missed — **each claim +re-verified against the code**, (2) where the two agree (treat as high-confidence), +(3) where I changed my mind, and (4) one merged, prioritized action plan. + +**Honest note on method:** the Codex pass was stronger than my first pass on the +foundation question because it read the actual `team-harness` and `eval-banana` +source (not just the installed wheel), **ran all four test suites**, and inspected +`ugm`'s roadmap for internal inconsistencies. It surfaced concrete, in-code +defects my pass reasoned around instead of opening. I've verified its headline +findings myself rather than take them on faith; results below. + +--- + +## 1. Concrete findings the parallel pass surfaced — all verified + +Each of these I checked against the code today. All confirmed. + +### 1.1 "False closure" is DELIBERATE, not a bug — reclassified ⚠️ CORRECTED +Both analyses initially read `harness_runner.py:190` (`_normalize_harness_result()` +returns `success=True` for any `TeamHarnessResult` that didn't throw; `result.agents` +discarded) as a defect. **The maintainer confirms it is intentional, and git +confirms it: `success=True` has been unconditional since the first commit of the +file (`a4cca5e`, Apr 19) — it was never a regression.** + +The coherent design: `IterationResult.success` means *"the assignment ran to +completion without the harness itself erroring"* (transport/config/exception → +`success=False`), **not** "the work was good." Semantic success is deliberately +delegated to the workflow-written artifacts (`control.json`, `goal_check.json`), +per the README. Inferring real success from `result.agents` exit codes is +unreliable — team-harness's coordinator can legitimately finish *after* a worker +fails — so the success decision was pushed to an explicit eval layer instead. Same +instinct as the LLM-judge decision (§1.2). + +**Residual risk (narrow):** the scheduler keys cadence (`run_every`, `must_follow`, +`run_after_successes`) off `entry.success`, so a run where a worker actually died +still advances those counters. Minor scheduling inaccuracy, bounded by +`control.json`/`goal_check.json` being the true gates. **Optional fix:** let cadence +require an *accepted* eval, not just a completed run. Not a rearchitecture. + +**The real consequence** is that the entire correctness burden rests on the eval +layer being run and honest — and that layer is LLM-as-judge by deliberate choice. +That concentration, not a harness-runner bug, is the thing to reason about (§3.1). + +### 1.2 The stock eval template forbids deterministic checks — ON PURPOSE ⚠️ CORRECTED +`templates/.../eval_reviewer/prompt.txt:54,56`: *"Only create harness_judge +checks"* and *"Deterministic checks are forbidden."* Both analyses flagged this as +backwards. **The maintainer explains why it's deliberate: when agents were allowed +to *author* deterministic checks, they invented nonsense — brittle, wrong-target, +gameable checks. LLM-as-judge on a described outcome stopped agents from gaming +their own checks.** That is a valid lesson; keep the judge. + +**The key distinction the blanket rule misses** — the failure mode was +*agent-authored* checks, not deterministic checks per se: +- *Agent invents a check* → garbage (what burned us; avoid). +- *Run a check the repo already owns* (`uv run pytest`, `import-linter`, + `alembic upgrade`, `make test`, exit-code pass/fail) → deterministic but NOT + agent-invented; it's the project's own contract. + +**Implication for `ugm` (important):** `ugm`'s acceptance criteria *are* pre-written +repo-owned contract tests, so you don't have to choose. Keep the LLM judge for the +qualitative "did this achieve the described outcome," and *also* gate on `ugm`'s own +suite via an eval-banana **deterministic check that shells out to the existing +command** (not an agent-authored one). That gives a deterministic backstop under the +judge — a mis-judged `goal_met` can't pass a red suite — without reintroducing the +agent-authoring problem. The stock "deterministic forbidden" rule is right for +generic repos and wrong for `ugm`; the `ugm_wp` child set should override it. + +### 1.3 `pm_planner_dispatcher` does not bundle its child workflow set ✅ VERIFIED +`loopy init --template pm_planner_dispatcher` copies only `planner/` and +`dispatcher/` (confirmed: those are the only two workflow dirs in the template). +But the dispatcher's whole job is to spawn child sessions running +`inner_outer_eval`, which is **absent** from a clean init. The PM template is not +executable as scaffolded. Directly blocks the `ugm` plan. **Fix:** bundle the child +set, or make workflow-set dependencies declarative + installable, and extend the +preflight test to cover the child. Codex §Q1.5. + +### 1.4 The Agent Skill teaches a removed API ✅ VERIFIED +`skills/loopy-loop/SKILL.md` still documents top-level `.loopy_loop/state.json`, +"one or more blocking workers poll it," inline `goal`, and the +`.loopy_loop/workflows//` layout — all of which v0.2.0 removed (state is now +session-local; workers are single, ping-pong not polling; the README explicitly +says the old workflows layout "is not loaded"). A skill that confidently generates +the *old* setup is worse than no skill. **Pre-launch blocker.** Codex §Q3. + +### 1.5 Active-child crash recovery can lose the child ✅ VERIFIED (by reading) +While a child runs, the "active" pointer is only +`CoordinatorService.state_store` aiming at the child's `state.json`; there is no +durable `active_child_session_id` in root state. After a coordinator crash, a fresh +`StateStore` picks `latest_top_level_state_path()` → reopens the **parent**, the +consumed child request is already deleted, and the child can be left `running` +while the parent dispatches new work. There is no test for "restart while a child +is active." My first pass flagged this path as "under-tested"; Codex identified the +actual failure mechanism. **Fix:** persist the session stack in root state; make +startup reconstruct and validate it. Codex §Q1.1. + +### 1.6 `ugm`'s own roadmap is already internally inconsistent ✅ VERIFIED (spot-checked) +`ugm` is design-first and mostly hand-maintained markdown, and it has drifted: +`phase-0-foundations.md` still marks `WP-0.1` `blocked(stack-conventions)` though +the scaffold merged to `main`; `roadmap.md` still lists the embedding-model choice +as a Phase-1 entry blocker though `questions.md`/`decisions.md` record it resolved +(D63). **An autonomous planner reading the phase files literally will either stall +on already-done work or declare things done without updating the source of truth.** +This is a first-order risk for pointing a loop at `ugm`, and neither my prose fix +(§2) nor the stock template addresses it. **Fix in `ugm`:** a machine-readable +gate/WP-status projection with a CI consistency check; the driver fails closed on +disagreement and asks for reconciliation rather than guessing which doc wins. Codex +§Q2. + +**team-harness / eval-banana specifics** (Codex read the source; I worked from the +wheel — defer to its detail, §Q4). Highest-signal, worth knowing before an +autonomous run: +- team-harness: normal return ≠ success (mirrors 1.1); **premature coordinator + finalization kills live workers after `shutdown_timeout_s`=10s** even though + Codex tasks routinely run 20–45 min; the coordinator loop is an unbounded + `while True` with **no turn/wall-clock/spawn/token/cost cap inside one harness + iteration** (loopy's outer `max_turns` does not bound intra-iteration cost); + headless auto-compaction likely never fires (`_should_compact` needs the last + message role to be `user`, but tool chains end in `tool`); `run.json` is rewritten + without atomic replace. +- eval-banana: a **non-zero agent exit can still pass** if stdout has valid JSON + (there's a test codifying it); the judge runs writable in the project root, so it + **can modify what it's judging**; `pass_threshold < 1.0` lets a *critical* check + fail as long as enough minor ones pass; some dead/stale config (`llm_max_input_chars`, + `target_paths`). + +--- + +## 2. Where both analyses independently agree — high confidence + +Two models reasoning separately reached the same conclusions here. Treat these as +settled: + +1. **Add a first-class human-in-the-loop pause state** (`paused` / + `waiting_for_human`) with a typed gate-request artifact — do **not** encode a + governance checkpoint as `unresolvable_error`. *(My §1.1 = Codex §Q1 rank-adjacent + + §Q2.)* Both of us call this the enabling capability for `ugm`. +2. **Keep single-worker; do not build parallel loopy workers now.** Both of us + argue this explicitly and against the obvious "scale = more workers" answer: + parallelism already lives inside team-harness; parallel loopy workers on one + checkout buys nondeterministic corruption. *(My §1.7 = Codex §Q1.10.)* +3. **Implement the reserved `events.jsonl` + cost/token ledger before any + dashboard.** *(My §1.2–1.3 = Codex §Q1.3.)* +4. **De-duplicate the three coordinator mutator blocks into one `_advance()`** — + but fold it into the state-machine hardening, not as a cosmetic refactor. + *(My §1.5 = Codex §Q1.7.)* +5. **Stop hardcoding model IDs; use named execution profiles; pin team-harness to + `>=0.2.10,<0.3`; treat eval-banana as a declared extra.** *(My §1.6 = Codex §Q1.6.)* +6. **Judge is evidence, never the sole stop gate; judge with a different model + family than you implement with** (`ugm` D53). *(My §4 = Codex §Q4.)* +7. **`ugm` via a PM/dispatcher shape is directionally right, but only as narrow, + human-gated, deterministic-acceptance pilots — never "build ugm" unattended.** + *(My §2 = Codex §Q2.)* +8. **Launch = docs (tutorial with real output, architecture, workflow-authoring, + troubleshooting, cost/security) + a demo repo + dogfood case studies; call it + Beta not 1.0; 1.0 only after something real was built with it.** *(My §3 = Codex + §Q3.)* + +--- + +## 3. Where I changed my mind + +### 3.1 The #1 risk is a single un-backstopped judgment, not a harness bug +My first pass named *concentration*; the parallel pass named *false closure* and I +briefly adopted it. After the maintainer's clarification (§1.1/§1.2) the accurate +framing is neither: the harness-runner behavior is deliberate, and LLM-as-judge is a +considered choice made because agent-authored deterministic checks failed. The real +#1 risk is what those two deliberate choices *combine* into: **acceptance for a +whole iteration rests on a single LLM judgment with no deterministic backstop.** For +low-stakes goals that's a fine, conscious tradeoff. For high-stakes work (`ugm`) +it's the thing to fix — not by reverting either decision, but by adding a +deterministic gate *under* the judge using the repo's **own** contract tests (§1.2). +That preserves both maintainer decisions and removes the single point of failure. +Maintainer concentration remains the #2 (slow, org-level) risk. + +### 3.2 The pilot should be one work package (WP-0.4), not "Phase 0 + Phase 1" +I recommended piloting Phase 0 plus the Phase 1 walking skeleton. Codex's narrower +call is better: **reconcile `WP-0.1` against the already-merged scaffold, then pilot +`WP-0.4` (ports + import-linter contracts) only** — it's bounded, needs no billable +infra and no golden labels, and has a crisp deterministic acceptance ("an illegal +import fails CI"). Save the Phase-1 walking skeleton for the *second* pilot, once +Phase-0 infra and the extractor model seat exist, so failures are attributable. I'm +adopting WP-0.4 as the pilot. + +### 3.3 The child needs its own budgets — a concrete blocker I under-weighted +Both passes note child sessions inherit root config, but Codex makes the +consequence sharp: a PM parent **cannot** give a child a small `max_turns`/cost/model +profile — the child would inherit `max_turns: 120`. For `ugm` that's a real blocker, +not a nicety. Per-child budgets move up my list (into Tier 1). + +*(Where we still differ in emphasis: I lean slightly more toward shipping +observability early as the thing that makes the pilot legible; Codex leans toward +state-machine/outcome correctness first. These aren't in conflict — correctness +first, then observe — and the merged plan below sequences them that way.)* + +--- + +## 4. Merged, prioritized action plan + +### loopy-loop — do in this order + +**P0 — correctness & trust (before any autonomous `ugm` run):** +1. **Durable session-stack / active-child recovery** + attempt IDs; a live + `/register` while a task exists returns busy/409 instead of abandoning it. + *(fixes §1.5 — the untested double-loop gap)* +2. **`paused` / `waiting_for_human` state + typed `gate_request.json` + + `external_action_request`** for billable/destructive ops. *(§2 enabler)* +3. **Per-child budgets** (turns, wall-clock, cost, model profile) in + `ChildSessionRequest` — today a child inherits `max_turns: 120`. *(fixes §3.3)* +4. **Deterministic backstop under the judge, for `ugm`** — an `ugm_wp` eval that + shells out to the repo's OWN suite (`pytest`/`import-linter`/`alembic`), + deterministic but not agent-authored, so a mis-judged `goal_met` can't pass a red + suite. Keep the LLM judge; do NOT globally drop the "deterministic forbidden" + rule (§1.2). *(This is the fix that removes the §3.1 single-point-of-failure.)* + +**P1 — legibility & safety:** +5. **`events.jsonl` + usage/cost ledger**, then `loopy status --watch` reading it. +6. **Fix the PM template**: bundle/declare the `inner_outer_eval` child set; extend + preflight to the child (the double loop can't run from a clean init today). + *(fixes §1.3)* +7. **Rewrite + test the Agent Skill** against a clean generated repo; remove + "one or more workers" claims from README/skill. *(fixes §1.4)* +8. *(optional, low priority)* Let scheduler cadence require an *accepted* eval, not + just a completed run — closes the narrow §1.1 residual. Not a rearchitecture. + +**P2 — hardening & DX:** +9. Named model profiles; pin `team-harness<0.3`; eval-banana as `[eval]` extra. +10. `_advance()` refactor folded into the P0 state-machine work. +11. Failure taxonomy + per-workflow failure cap + retryability/backoff classes. +12. `loopy doctor` / `validate` / session-aware `stop --force`; clean-install + template smoke test in CI. +13. Fault-injection tests (restart-with-active-child, double-register, partial + JSON at every handoff, schema migration, fake-CLI subprocess integration). + +**Explicitly deferred:** parallel loopy workers; web dashboard; breadth-first child +sessions. + +### `ugm` — prerequisites before pointing a loop at it +- Reconcile the roadmap: close `WP-0.1`, correct the resolved gates, and add a + **machine-readable gate/WP-status projection with a CI consistency check** (the + driver fails closed on markdown/git disagreement). *(fixes §1.6)* +- Pre-fill the owner "stack conventions" slot (uv, ruff, strict pyright, GH Actions, + secrets) so Phase 0 isn't self-blocked. +- Custom `ugm_phase_driver` (parent: `gate_reconciler` → `wp_selector` → + `dispatcher` → `evidence_reviewer`) and a small `ugm_wp` child set + (`plan`/`implement`/`contract_eval` with `emits_goal_check`/`review`) — not the + stock 120/160-turn loop. Only `evidence_reviewer` may change WP status. +- One goal file per WP (never "implement ugm"); exact `uv`/`make` acceptance + commands per WP; local disposable Postgres/MinIO; human inboxes for decisions and + external actions; a one-child-owns-the-checkout branch policy. +- **Pilot = reconcile WP-0.1, then WP-0.4**, contract-eval after every implement + success, child capped ~8–12 turns (needs P0.4 first). + +--- + +## 5. Bottom line + +Both passes agree the bet is sound *in its restrained form* — fresh agent contexts ++ durable repo state + git + independently verified artifacts is a credible way to +push useful coding work past one chat. The correction I'm accepting from the +parallel pass is the framing of the central risk: **it is false closure, not +concentration.** The highest-leverage next move is therefore not a feature — it's +(a) make completion *typed and evidence-derived* so the loop can't lie to itself, +then (b) prove it on `ugm` **WP-0.4** under a hardened, human-gated, deterministic +pilot, recording every transition, cost, and intervention. That single honest trace +is simultaneously the correctness proof, the launch artifact, and the compatibility +test for the whole trio. diff --git a/design/designs/success-semantics-and-evaluation.md b/design/designs/success-semantics-and-evaluation.md new file mode 100644 index 0000000..9dad9d0 --- /dev/null +++ b/design/designs/success-semantics-and-evaluation.md @@ -0,0 +1,191 @@ +# Design: Success Semantics and Evaluation Strategy + +**Status:** Accepted (documenting decisions already in the codebase) +**Date recorded:** 2026-07-12 +**Applies to:** `loopy-loop` coordinator/worker loop and the packaged +`inner_outer_eval` workflow set. + +This document records two design decisions that are **deliberate and load-bearing**, +but were until now implicit in the code rather than written down. Both were +re-examined during the July 2026 review (see +[`../analysis/synthesis.md`](../analysis/synthesis.md)); an outside reading of the +code mistook each for a defect. They are not defects. This document exists so the +next reader — human or agent — does not "fix" them by accident. + +Both decisions share one principle: + +> **Do not infer semantic success from noisy mechanical signals. Push the +> success/acceptance decision to an explicit, purpose-built evaluation layer.** + +--- + +## Decision 1 — Iteration success means "the assignment ran," not "the work is good" + +### Decision + +`IterationResult.success` is `True` whenever a `team-harness` run returns normally, +and `False` only when the harness itself raises (`ConfigError`, `TeamHarnessError`, +or an unexpected exception). It is **not** a judgment about whether the requested +work was actually accomplished. `TeamHarnessResult.agents` (per-worker statuses and +exit codes) is intentionally **not** consulted to decide iteration success. + +Reference: `src/loopy_loop/harness_runner.py` — `_normalize_harness_result()` +returns `success=True`; the `success=False` paths live only in `run_harness_iteration()`'s +exception handlers. + +**Semantic success is decided elsewhere**, by artifacts the workflow writes: + +- `control.json` — the session stop switch (`running` → `stopped` with a + `stop_reason`). This, and only this, stops the loop. +- `goal_check.json` — per-iteration evidence (`goal_met` / `reason`). Evidence only; + a valid `goal_check.json` does **not** by itself stop the loop. + +### Context / why + +`team-harness`'s coordinator is an orchestrator, not a build system. It can +legitimately return a normal result after a worker has failed — it may synthesize a +final answer, decide it has enough information, or route around a dead worker. Worker +exit codes are therefore a **noisy** proxy for "did the assignment succeed": a +non-zero worker can accompany a perfectly good outcome, and an all-green set of +workers can accompany a useless one. Mapping those signals to a boolean would +manufacture false precision. + +So `loopy-loop` draws the line at the only thing it can observe reliably — *did the +assignment run to completion without the harness itself erroring* — and delegates the +"was it any good" question to an explicit evaluation step that produces +`goal_check.json`, with the human/agent-owned `control.json` as the actual gate. + +This has been the behavior since the first commit of `harness_runner.py` +(`a4cca5e`, 2026-04-19); it is original design intent, not drift. + +### Consequences + +- **The evaluation layer is the real arbiter of completion**, not the harness return + value. Everything downstream depends on that layer being run and being honest + (see Decision 2, and the "known limitation" below). +- **The scheduler keys cadence off mechanical success.** `run_every`, `must_follow`, + and `run_after_successes` all read `HistoryEntry.success`. A run where a worker + actually failed but the harness returned normally still advances these counters. + This is an **accepted, bounded inaccuracy**: `control.json`/`goal_check.json` remain + the true gates, so the worst case is a slightly-off cadence, not a false "goal met." +- **Crash recovery treats a locally-written result as authoritative.** The + `pending_finished_request.json` / `result.json` recovery path trusts the recorded + iteration result; it does not re-derive success from worker artifacts. + +### Known limitation (documented, not a call to revert) + +Because acceptance for an entire iteration ultimately rests on the evaluation layer, +and that layer is LLM-as-judge by design (Decision 2), a whole iteration's +"success" can rest on a single model judgment with no deterministic backstop. For +low-stakes goals this is an acceptable, conscious trade. For high-stakes work it +should be **backstopped**, not reverted — see the proposals doc and the note in +Decision 2 about repo-owned deterministic checks. + +### Alternatives considered and rejected + +- *Derive iteration success from worker exit codes / `result.agents`.* Rejected: + unreliable for the reasons above; produces false negatives (good outcome, failed + worker) and false positives (all-green, useless outcome). +- *Make `goal_check.json` directly stop the loop.* Rejected: conflates evidence with + control. Keeping `control.json` as the sole stop switch means a workflow (or human) + must make an explicit stop decision, which is auditable and reversible. + +### When to revisit + +If the cadence inaccuracy ever causes a concrete problem, add an **opt-in** for +cadence to require an *accepted* eval rather than a merely *completed* run. That is a +small, additive change; it does not require touching the success boundary itself. + +--- + +## Decision 2 — Evaluation is LLM-as-judge; agents do not author deterministic checks + +### Decision + +In the packaged `inner_outer_eval` workflow set, the eval workflows create **only** +`harness_judge` (LLM-as-judge) checks that describe *desired outcomes*. Authoring +deterministic checks is explicitly forbidden in the stock template. + +Reference: +`src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_reviewer/prompt.txt` +— "Only create harness_judge checks"; "Do not create deterministic checks. +Deterministic checks are forbidden." + +### Context / why + +This rule comes from direct experience, not theory. When agents were allowed to +**author** deterministic checks, they produced bad ones: brittle string-matching, +checks that tested the wrong thing, checks that passed for the wrong reason, and +checks an agent could trivially satisfy without doing the real work. In practice, +letting the implementer invent its own pass/fail criteria let it game itself. + +`harness_judge` on a described *outcome* removes that failure mode: the check states +what good looks like in natural language, and a judge evaluates against it. The +implementer cannot quietly redefine "done" into something it already produced. + +### Scope and boundary (important — read before applying to other repos) + +The thing that failed was **agent-authored** checks, not deterministic checks as a +category. Two very different things get conflated under "deterministic check": + +- **Agent invents a check** → the failure mode above. Correctly forbidden. +- **Run a check the repo already owns** → e.g. `uv run pytest`, `import-linter`, + `alembic upgrade`, `make test`, evaluated on exit code. The agent did not invent + these; they are the project's own contract. Running them is deterministic but is + **not** the failure mode this rule targets. + +Therefore the "deterministic forbidden" rule is correct **for generic target repos +where the only deterministic checks would be agent-invented**. For a target repo that +already owns a trustworthy contract-test suite, the right configuration is *both*: +LLM-as-judge for the qualitative "did this achieve the outcome," **and** a +deterministic gate that shells out to the repo's own suite as a backstop under the +judge. That backstop does not reintroduce the agent-authoring problem, and it removes +the single-judgment point of failure noted in Decision 1. Workflow sets targeting +such repos (e.g. a future `ugm_wp`) should override the stock rule accordingly. + +### Consequences + +- **Evaluation is outcome-focused and resistant to self-gaming**, at the cost of the + usual LLM-as-judge properties: non-determinism, per-check inference cost, and the + judge as a point of trust. +- **The judge should not share failure modes with the implementer.** Prefer judging + with a different model family than the one that implemented (cf. `ugm`'s D53 + cross-family principle). +- **A single judge pass is evidence, not a hard gate for high-stakes stops.** Keep + `control.json` as the stop switch (Decision 1); for high-stakes goals, require + repeated/independent judgments or a deterministic backstop before a terminal + `goal_met`. + +### Alternatives considered and rejected + +- *Let agents author deterministic checks (the prior state).* Rejected on evidence: + produced nonsensical, gameable checks. +- *Deterministic-first everywhere, judge as residual (the "obvious" best practice).* + Rejected **as a blanket rule** because in generic repos the only deterministic + checks available are the agent-authored ones that failed. It is the *right* rule + only where the deterministic checks are repo-owned (see boundary above). + +### When to revisit + +Revisit per target repo, not globally: when a target owns a trustworthy contract-test +suite, add the deterministic backstop (do not remove the judge). When judge cost or +flakiness becomes material, add repetition/consensus and cross-family judging rather +than abandoning the approach. + +--- + +## Summary + +| | Decision 1 | Decision 2 | +|---|---|---| +| **What** | Iteration success = harness completed, not work-is-good | Eval = LLM-as-judge on outcomes; agents don't author deterministic checks | +| **Why** | Worker exit codes are a noisy proxy for real success | Agent-authored deterministic checks were gameable nonsense | +| **True gate** | `control.json` (stop) + `goal_check.json` (evidence) | The judge's verdict, recorded as evidence | +| **Shared principle** | Delegate the success decision to an explicit eval layer | Same | +| **Backstop for high-stakes** | Optionally require *accepted* eval for cadence | Add repo-owned deterministic check under the judge | + +Both decisions are sound. Neither should be reverted. The one thing worth adding — +for high-stakes targets only — is a deterministic backstop built from the target +repo's **own** contract tests, which strengthens both decisions without undoing +either. That, and other forward-looking proposals, are covered separately in the +proposals design doc.