/map-efficient: parallel-wave execution shipped but dark — sequential by default at every gate

[azalio]

Summary

The full parallel-execution machinery from #284 (per-subtask worktree isolation, wave planner, merge_wave_worktrees coordinator) is built and tested, but it is opt-in at every layer. For a typical run, /map-efficient executes strictly sequentially even when the plan is perfectly parallelizable (a wave with ≥2 independent, disjoint-file subtasks). The feature is effectively dark.

mapify_version: 3.20.0

Observed

A decomposition that yields a wave of ≥2 independent subtasks still runs one Actor at a time. Parallelism never engages on the default path.

Root cause is a chain of three closed gates, each defaulting to sequential, plus two latent causes:

Gate 1 — default loop is the sequential walker; wave-loop is opt-in.

• Main loop drives off get_next_step (single-subtask walker): src/mapify_cli/templates_src/skills/map-efficient/SKILL.md.jinja:135,409,446,464
• Explicit: "the skill defaults to the sequential walker for simplicity. Switch to the wave-loop when (a) waves have ≥2 subtasks AND (b) … disjoint files" — SKILL.md.jinja:217-225
• So even when set_waves produced a ≥2 wave, the default path never reads it back via get_wave_step as a parallel batch.

Gate 2 — concurrent Actor dispatch is hard-coupled to worktree.isolation, which is OFF.

• Default: worktree.isolation → "false" — map_step_runner.py.jinja:15325
• When off, create_subtask_worktree returns {"status":"disabled","reason":"worktree.isolation is off"} — map_step_runner.py.jinja:15420
• Concurrent dispatch requires mode:"parallel" AND isolation enabled — efficient-reference.md.jinja:572-573
• This coupling is correct in principle: Claude Code gives subagents context isolation but NOT filesystem isolation — two Actors writing the same working tree concurrently corrupt it. So parallelism is structurally gated on the opt-in worktree feature.

Gate 3 — policy text repeatedly biases toward sequential.

• parallel_tool_policy: guarded_wave_only (SKILL.md.jinja:23), "Keep execution sequential by default" (:27), "Sequential subtask execution is default" (:35), "Default to sequential execution … or explicit user-requested parallel execution" (:210).

Latent A — decomposer over-serialization. Parallelism is derived post-hoc from dependencies via Kahn (dependency_graph.py::compute_waves). Any false "logical-ordering" dependency collapses waves to size 1 → mode always "sequential". Guardrails exist (task-decomposer.md "Minimize Dependencies for Parallelism") but it's an LLM judgment, not a guarantee.

Latent B — no explicit multi-Task() dispatch mechanism. Reference says "dispatch the Actors concurrently (separate Task agents)" (efficient-reference.md.jinja:574) but never states the actual mechanism: emit one Task(actor) tool_use per subtask in a single assistant message. Claude Code has no special syntax for this — concurrency is purely the model emitting multiple tool calls in one turn. After reading "sequential by default" five times, the operator model won't.

Expected

When a wave legitimately contains ≥2 independent, disjoint-file subtasks, /map-efficient should engage parallel execution without the operator manually flipping config + switching dispatchers.

Suggested levers (each can be a slice)

1. Auto-switch to the wave-loop when set_waves yields any multi-element wave (currently opt-in "for simplicity") — opens Gate 1.
2. Reconsider the worktree.isolation default (or auto-enable it when a parallel wave is about to dispatch) — opens Gate 2. Needs a fallback when not in a git repo.
3. Add an explicit SKILL instruction + example: "emit one Task(actor) per wave subtask in one message" — closes Latent B.
4. Optionally surface a one-line diagnostic when a parallelizable wave is detected but executed sequentially (so the dark path is observable, not silent).

Notes

• This is a deliberate-but-too-conservative rollout of GSD-style fresh-context-per-task: worktree isolation for autonomous execution #284, not a crash. Filing so the activation work isn't lost.
• Implementation has documented eval risk (changes Actor dispatch + Monitor/merge flow) → consult llm-council before building.

[azalio]

llm-council verdict — how to implement (deep mode, conv d5ff5181-a11c-4a2a-8f4e-14087c0ae830)

Council split 2–2 on two sub-points (flagged below), but converged strongly on the architecture. Models: minimax-m3, gpt-5.5, qwen3.7-max, deepseek-v4-pro.

Unanimous decisions

1. Wave-loop becomes the default path — "opt-in for simplicity" IS the bug (Gate 1). Legacy get_next_step walker stays as the instant rollback.
2. Do NOT globally flip worktree.isolation to true — it breaks non-git / shallow / submodule third-party repos. Auto-enable per-dispatch only when a parallel group is about to run; degrade to sequential, don't hard-fail (Gate 2).
3. Delete the 5× "sequential by default" repetitions in the skill doc — repeated language overrides a single new paragraph; it's active anti-signal (Gate 3 + Latent B).
4. Dependency-graph lint gate, warn-only by default; never silently auto-delete edges (Latent A).
5. Concurrency in Claude Code = N Task tool_use blocks in ONE assistant message. No parallel keyword exists.

Per-question recommendations

Q1 — Wave-loop default. Make it default, but key the predicate on the color group, not the wave. A 5-subtask wave may split into 3 color groups where only some have ≥2 members; a wave-size predicate triggers overhead on all-singleton groups. Predicate: engage wave-mode iff any(len(color_group) >= 2) across the plan. Config execution.wave_mode: off|auto|on (default auto); off = instant rollback.

Q2 — Worktree isolation. Replace the boolean with an enum: worktree.isolation: off | auto | required (default auto).

• auto — create per-subtask worktrees only when a parallel-ready group dispatches; if unavailable → sequential + loud warning.
• required — hard-fail before dispatch if unavailable (for first-party repos).
• Fallback matrix (auto): not-a-git-repo / worktree-unsupported / create-failed / dirty-target → sequential + warning. required → hard-fail on all.
• Probe safely: no universal dry-run for git worktree add; do a minimal git worktree add --detach .map/worktrees/.probe HEAD round-trip in try/finally, cache per session. Verify primary worktree via git rev-parse --show-toplevel vs git worktree list.
• Add require_clean_merge_target: true (refuse dispatch if git status --porcelain dirty).
• Cleanup hygiene: unconditionally remove rejected/orphaned worktrees from .map/worktrees/ after merge — never litter third-party repos.

Q3 — Reliable N-concurrent dispatch (highest leverage).

• Delete the 5 repetitions; replace with one rule: mode follows the plan's color-group shape.
• Add a CRITICAL section with a literal example using the real Task params (subagent_type/description/prompt), three Task(...) calls shown in one message, plus an explicit INCORRECT contrast (one Task per turn).
• Condition dispatch on parallel_ready: true (unique worktrees ready), NOT on raw wave size — otherwise the doc tells the LLM to parallelize unsafely.
• Anti-patterns callout: one Task per turn across N turns, "thinking through" each prompt before dispatching, TodoWrite between dispatches, waiting for one result before the next.
• max_actors cap (default 4–8); oversized groups pre-split into sequential batches surfaced as batch: 1/3 — doc must not contradict the cap.
• Self-audit: orchestrator emits a checkable line ("I will emit {n} Task(actor) calls in one message") reconciled via a record_dispatch_actual(...) call — converts an invisible instruction-following failure into a detectable event (the Python coordinator can't introspect tool calls live).
• Actor prompt template pins: work only inside /abs/.map/worktrees/<id>, frozen base SHA, never touch main tree / sibling worktrees, completion report (path, base SHA, final SHA, files changed).

Q4 — Decomposer over-serialization lint. Two layers:

• Layer A (hard error, always on): self-loops, cycles, dep-on-unknown-subtask, duplicate edges → block + force planner retry.
• Layer B (warn-only default): require a typed basis per edge (file_write_read, interface_contract, test_dependency, schema_or_migration, generated_artifact, external_constraint, …); mechanical check — if A.outputs ∩ B.inputs == ∅ AND A.affected_files ∩ B.affected_files == ∅, flag the edge regardless of wording; soft-phrase blacklist ("logical ordering", "do this first for safety", "natural order"); same-file-touch is a coloring concern, not a dep edge (flag INFO); plan-shape metric — N>=4 with max width ==1 → "fully serialized" warning.
• NOTE: Kahn sort does NOT do transitive reduction; redundant edges are harmless → log INFO, not a defect.
• Enforcement off|warn|repair_once|strict, default warn. Auto-prune stays opt-in experimental (decomposition.auto_prune_weak_deps: false) — most evaluators consider auto-prune-by-default unsafe (a lint false-positive silently deletes a real dep). If active correction is wanted, prefer capped repair_once (ask the planner to revise its own graph once, accept only if still a valid DAG and parallelism not worsened).

Q5 — Observability (never silent). Invariant: if color_group.size > 1 and actual dispatch was sequential → warning with a stable reason code, no exceptions.

• 3 levels: (1) plan summary after set_waves+split (subtasks/edges/waves/max-width/color-group breakdown, loud "fully serialized" if width 1); (2) per-group decision log (planned vs actual mode, worktree status, base SHA, dispatch count, degradation reason); (3) machine-readable artifact .map/runs/<run_id>/parallelism.json.
• Stable reason codes: not_git_repo, worktree_unsupported, worktree_create_failed, dirty_merge_target, dispatch_serial, parallel_capped_by_max_actors, monitor_rejected_subtask, merge_conflict, post_wave_gate_failed.
• Detection: use the runtime's tool-call count from the dispatching message as the primary signal, NOT time-overlap of sessions (unreliable across boundaries). Skip the "Nx speedup" metric — planned-vs-actual dispatch accounting is the signal that matters.
• Optional parallel.strict (default false) promotes degradation warnings to errors — but max_actors batching must never count as a violation.

Recommended ordered rollout (eval-harness-first, behavior-neutral until late)

• Slice 0 — Eval harness + scaffolding (no behavior change). Add all new config keys defaulting to current behavior. Regression suite of canned plans: linear, 2-wave parallel, conflict-split, non-git, shallow clone, submodule, network-fs, dirty repo. Gates every later promotion.
• Slice 1 — Dependency lint, warn-only. DAG hard-errors + justification warnings. Burn-in to measure false-positive rate before any pruning. Escape: lint_deps=off.
• Slice 2 — Worktree probe + clean-target + cleanup hygiene (flag-gated). try/finally probe; require_clean_merge_target; orphan cleanup. Test matrix clean/dirty/shallow/submodule/non-git/symlink-heavy.
• Slice 3 — Wave-loop default, flag-gated, sequential inside. Route through get_wave_step/validate_wave_step/advance_wave but execute groups sequentially — exercises merge/rollback/gate at full fidelity on the larger N-subtask diff (post-wave gate now sees a bigger diff → may need tolerance tuning) WITHOUT adding concurrency. Isolates wave-coordinator bugs from dispatch bugs. Escape: wave_mode=off.
• Slice 4 — Skill-doc surgery, synchronized with wave-loop promotion. Delete the 5 repetitions; add literal Task example, decision table, anti-patterns, self-audit, actor-boundary template, parallel_ready conditioning, max_actors batching. Must ship in lockstep with wave-mode being on — shipping "emit N parallel Tasks" before the state machine supports it tells the LLM to fan out into a sequential coordinator.
• Slice 5 — Auto-worktree + concurrent dispatch (flag-gated). worktree.isolation: auto per color group + real N-Task fanout, conditioned on parallel_ready. Actor-failure policy: any failure/timeout/Monitor-reject rolls back the WHOLE wave (never merge a successful subset — preserves GSD-style fresh-context-per-task: worktree isolation for autonomous execution #284 invariant). retry_degraded_once: false default. Promote after eval matrix green and the serial-dispatch reason code fires correctly on induced regression.
• Slice 6 — Default flips. Flip wave_mode=auto, worktree.isolation=auto, parallel execution on, after a 2–4 week soak. Permanent escape hatches: MAP_EFFICIENT_SEQUENTIAL_ONLY=1 env + config overrides; auto_prune and strict stay opt-in.
• Slice 7 (optional) — Decomposer prompt tuning from accumulated lint data; graduate repair_once if validated.

Contested points (council genuinely divided — decide at impl time)

• "Always wave-loop" vs predicate-gated: one model wanted the wave-loop to be the only scheduler (sequential = behavior within it) for a single code path + consistent telemetry; others route linear plans to the legacy walker to avoid coordinator overhead. Both defensible.
• Backward-compat migration: change default only when the worktree.isolation key is absent (surprises nobody but leaves already-installed repos dark) vs migrate generated false→auto / true→required with a MIGRATE-001 warning. Choose based on whether false was installer-emitted (migrate) or a deliberate user choice (preserve + nudge).

Bottom line: wave-loop default gated on color-group concurrency; worktree.isolation → off|auto|required enum default auto; doc surgery with a literal multi-Task example; DAG lint warn-only (auto-prune opt-in); never-silent reason codes via tool-call-count detection; roll out eval-harness-first with concurrency behind a flag until the matrix is green.

[azalio]

Беру в работу. Council идентифицировал баг: "opt-in for simplicity" IS the bug (Gate 1).

Начинаю с Slice 0 по рекомендованному council rollout-плану:

• Добавление новых config keys (execution.wave_mode: off|auto|on, worktree.isolation: off|auto|required) без изменения поведения
• Regression suite для canned plans (linear, 2-wave parallel, non-git, dirty repo и т.д.)

Слайс 0 не меняет поведение — это eval harness + scaffolding, который гейтирует все последующие слайсы.

---

Generated by Claude Code

[azalio]

Taking Slice 0 of the 7-slice rollout identified in the council's analysis.

Slice 0 scope (config scaffolding, no behavior change):

• Migrate worktree_isolation from bool → 3-value enum string: off / auto / required
• Add execution_wave_mode field: off / auto / on (default auto)
• Backward-compat: YAML boolean false→"off", true→"required" via migration in load_map_config
• YAML 1.1 coercion for bare off/on parsed as Python booleans
• Update _wt_isolation_enabled in step runner to handle new enum values
• Full test coverage: enum validation, backward-compat migration, YAML 1.1 edge cases

PR incoming.

---

Generated by Claude Code

[azalio]

Slice 5 — implementation decisions (llm-council deep, conv d5ff5181)

Slices 0–4 shipped behavior-neutral (#304 + #305). Before building Slice 5 (the first behavior-changing slice — auto-worktree + concurrent dispatch), consulted the council on the concrete implementation (the architecture was already settled in the earlier d5ff5181 round). Verdict: split Slice 5 into 5a/5b plus several concrete decisions.

Decisions

• Q1 — concurrency_enabled ownership. Coordinator-owned (Python), never the LLM skill. Split into concurrency_allowed (pure function of select_execution_strategy: strategy==wave_loop AND isolation in {auto,required} AND any color-group width>=2) + concurrency_ready (runtime check the coordinator makes against its own worktree state after create_subtask_worktree: path registered, HEAD==base SHA, clean tree). No skill-reported parallel_ready flag across the LLM trust boundary. get_wave_step returns dispatch_mode + a stable reason code; degrade is never silent.
• Q2 — record_dispatch_actual self-audit. Two signals, kept separate. N distinct worktree SHAs off one wave base prove artifact isolation, NOT same-turn concurrent dispatch (sequential dispatch through N worktrees produces identical SHA evidence). Tag audit status same_turn_verified / artifact_verified / self_report_only. Self-report alerts on mismatch, never hard-fails. Handle no-op (no commit = no_op_passed), multi-commit, empty-commit edge cases.
• Q3 — sub-slice. 5a = wire worktree.isolation in {auto,required} to actually create per-color-group worktrees and route the wave through merge_wave_worktrees, but dispatch Actors sequentially — exercises real worktree creation + atomic wave-merge + whole-wave rollback at full fidelity without concurrency. NOT redundant with shipped Slice 3 (which routes the wave-loop but never creates worktrees). 5b = flip concurrent_dispatch so the skill emits N Task(actor) blocks in one message. Skip the synthetic-diff shadow phase (the GSD-style fresh-context-per-task: worktree isolation for autonomous execution #284 merge path is already built+tested). concurrent_dispatch fails closed when isolation is off.
• Q4 — max_actors. Default 4, clamped [1,8] and against provider/repo limits. Batch within color groups into chunks of cap (deterministic order by (priority, id)); each sub-batch branches from the prior sub-batch's post-merge commit. Capacity batching is NOT a parallel.strict violation (a true violation is: fits in one batch but dispatched serially anyway).
• Q5 — whole-wave failure. Discard-all + re-run-from-base as the Slice 5 default — matches the existing GSD-style fresh-context-per-task: worktree isolation for autonomous execution #284 atomic invariant and retry_degraded_once=false. Any Actor failure / Monitor reject → merge no output, discard/quarantine all group worktrees, reset to wave base (GSD-style fresh-context-per-task: worktree isolation for autonomous execution #284 already does this), retry from base (max 1–2), then degrade to sequential, then surface. Partial preservation (--partial-preserve) ships later as opt-in only.
• Q6 — eval landmines the Slices-0–4 sequential harness misses (→ a separate parallel-aware harness): (1) phantom parallelism (skill serializes despite the instruction; code is correct so every gate passes — the landmine most specific to Slice 5; catch via planned-vs-observed dispatch + same_turn_verified); (2) shared state outside the git worktree (/tmp, ~/.cache, .venv, ports, local DBs, Docker — worktrees isolate repo files only); (3) generated-file / implicit conflicts (package-lock.json, generated clients, *.proto, migration registries, barrel exports look file-disjoint but collide at merge → conflict-expansion rules + post-completion git diff --name-only base...HEAD audit); (4) post-merge integrity (per-worktree passes, merged tree fails — run full suite on the merge commit); (5) non-deterministic merge ordering (enforce deterministic order by task id); (6) worktree-lock contention on .git/worktrees; (7) cleanup leaks on partial failure (assert zero orphaned worktrees after every run incl. simulated crash).

Highest-leverage correction: stop treating worktree-SHA evidence as proof of concurrency — it proves isolation, not same-turn dispatch; conflating them is exactly how a "parallel" wave silently runs sequential while every gate stays green.

Rollout

Proceeding with Slice 5a now (flag-gated, behavior-neutral by default since worktree.isolation defaults off). Slice 5b (concurrent fanout) lands after 5a's parallel-aware eval matrix is green. Slice 6 (default flips) only after a soak. No shadow-mode rollout.

Council: d5ff5181-a11c-4a2a-8f4e-14087c0ae830. Models: minimax-m3, gpt-5.5, qwen3.7-max, deepseek-v4-pro.

[azalio]

Slice 5a — landed in #306 (CI green)

Auto-worktree wave-loop with sequential dispatch, flag-gated (default worktree.isolation=off → byte-identical legacy sequential path). 7 subtasks via /map-efficient, all Monitor valid, independent final-verifier PASS, full make check green (3057 passed).

• ST-001 concurrency_allowed (pure, coordinator-owned) + select_execution_strategy CLI
• ST-002 get_wave_step → structured dispatch_mode/isolation_active/reason (replaces bare WAVE_CONCURRENCY_ENABLED echo; dispatch_mode always sequential in 5a)
• ST-003 concurrency_ready — read-only coordinator worktree-readiness check (council Q1; no skill-reported flag)
• ST-004 skill/reference docs: 5a sequential-through-worktree flow; concurrent fanout relabeled 5b-only; codex provider-neutral
• ST-005 dormant max_actors(4,[1,8]) + retry_degraded_once(false) config + clamp (no 5a consumer)
• ST-006 parallel-aware Q6 harness: orphan-cleanup, generated-file conflict rollback (attribution names both), post-wave-gate rollback (HC-4), no-op classification, deterministic sorted-id order — found no bug in merge_wave_worktrees
• ST-007 HC-1 default-config behavior-neutrality proof (non-tautological: seeds a width≥2 wave so the isolation gate, not an empty plan, holds the sequential path)

HC-1..HC-5 all proven by tests. #284 atomic whole-wave invariant preserved (HC-4).

Remaining: 5b (real concurrent N-Task fanout + record_dispatch_actual + max_actors batch-split) after this matrix soaks; Slice 6 (default flips) after that. No shadow-mode rollout.