Implement workflow orchestration engine (P0 + P1)

CHANGELOG.md

@@ -6,6 +6,129 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+### Added
+
+- **Workflow orchestration engine (experimental).** akm can now execute
+  multi-step workflows as deterministic **YAML programs**, driven either by a
+  native engine or by any agent session. This is a new, self-contained
+  surface; classic linear **markdown workflows and the stable workflow CLI
+  contract (`start`/`next`/`complete`/`status`/`list`) are unchanged**. What
+  ships:
+  - **Authoring.** Orchestrated workflows are YAML programs
+    (`workflows/*.yaml`, `version: 1`) validated against a published JSON
+    Schema (`schemas/akm-workflow.json`) by `akm workflow validate`; scaffold
+    one with `akm workflow template --yaml` or `akm workflow create
+    <name>.yaml`. A closed `${{ … }}` expression language (exactly
+    `params.<name>`, `steps.<id>.output.<path>`, `item`, `item_index`, parsed
+    once into an AST) wires steps together. `validate` also surfaces non-fatal
+    **warnings** (a step with no typed `output:` schema; a `${{ params.<name> }}`
+    reference to a param the declared `params:` block omits) that never change
+    the frozen plan or its hash. Creating a workflow whose canonical name
+    collides with an existing asset of a **different** extension (`foo.yaml`
+    while `foo.md` exists, or vice-versa) is refused, since the two would
+    silently shadow each other.
+  - **Compilation + frozen plans.** `akm workflow start` compiles the program
+    into a backend-agnostic Workflow Plan Graph IR (`src/workflows/ir/`) and
+    freezes it on the run row (`plan_json` + `plan_hash`); a run executes the
+    plan compiled at start, and edits to the source file require a new run.
+  - **Per-step orchestration.** A step can declare a runner, model, timeout,
+    fan-out (`map`/`over` with a `concurrency` cap and a `collect` | `vote`
+    reducer), a typed `output` JSON Schema (validated via a `runStructured`
+    retry-with-feedback loop), `env` bindings (resolved through the existing
+    `akm env run` machinery — secret tokens, dangerous-key policy, keys-only
+    audit events), classify-and-dispatch `route` steps, and `depends_on`
+    ordering.
+  - **Determinism + replay.** Journaled unit identity is content-derived
+    (`<step>:<sha256(item)[:12]>`, `:solo` for a single unit), so cached
+    results survive item-list reordering; a completed unit whose recorded
+    inputs differ on replan is a hard **replay-divergence** failure naming the
+    unit, never a silent re-dispatch. Every unit is recorded in the new
+    `workflow_run_units` table behind a serialized writer queue.
+  - **Execution (`akm workflow run`).** A semaphore-bounded scheduler fans a
+    step's units out (concurrency defaults to 1 per the local-model
+    LLM-defaults rule, capped by the engine-wide `workflow.maxConcurrency`
+    config knob — unset = `min(16, max(1, cores − 2))`, explicit values
+    clamped to `[1, 64]`), enforces per-unit
+    timeouts (default 10 m) and run **budget ceilings** (`budget.max_tokens` /
+    `budget.max_units`, seeded from the journal so they span resumes), and
+    advances the run **strictly through `completeWorkflowStep`** so completion
+    gates are never bypassed. Every dispatched unit gets a standard akm
+    preamble (run/unit ids, knowledge + env/secret + reporting contract).
+  - **Typed artifacts + honest gates.** A step's promoted artifact is
+    validated against its declared `output` schema before completion; a
+    criteria-bearing gate judges that **artifact** (canonical JSON, clipped)
+    rather than machine prose, and each engine-driven evaluation is journaled
+    as a gate unit row. `gate.max_loops` bounds an evaluator-optimizer retry
+    loop (feedback threaded into re-dispatched unit prompts); `gate.required`
+    (or the run-wide `--require-gates`) makes a gate with no available judge
+    **block** for a human instead of failing open.
+  - **Failure policy.** Per-unit `on_error: fail | continue` (fail-fast
+    default) plus bounded `retry: { max, on: [<failure_reason>…] }` keyed on
+    the persisted failure taxonomy.
+  - **Isolation + leases.** `isolation: worktree` runs each file-mutating unit
+    in a fresh detached git worktree (journaled path; clean trees
+    auto-removed, dirty ones retained). A run **lease** (`engine_lease_*`)
+    ensures a run is driven by exactly one engine or one external driver at a
+    time; manual `complete` is refused while a live engine lease is held.
+  - **Harness-neutral driver protocol.** An orchestrated run can be driven by
+    ANY agent session (Claude Code, opencode, Codex, a human at a shell), not
+    only the native engine. **`akm workflow brief <run>`** is read-only (takes
+    no lease, mutates nothing) and emits the active step's expected work-list —
+    per-unit content-derived id, resolved instructions + input hash
+    (byte-identical to the engine's dispatch), output schema, env binding
+    NAMES only, and the exact `report` command lines. **`akm workflow report
+    <run> --unit <id> --status completed|failed|running`** is the one mutating
+    verb, ingesting a unit's result through the SAME shared step semantics the
+    engine uses (idempotent same-hash re-report, replay-divergence on a
+    differing hash, budget enforcement, schema validation, and the
+    artifact-judged gate/`max_loops` completion path). `--status running`
+    claims/heartbeats a unit for stale-driver detection without advancing the
+    spine; `--rerun` records a fresh attempt for a failed unit (carrying its
+    prior token total forward). Every report command carries `--expect-step`
+    (refused if the spine has moved since the brief), and `report --settle`
+    (no `--unit`) advances a step that dispatches no reportable units — a
+    params-only route, an empty fan-out, or an all-unresolvable work-list — so
+    a driver is never wedged. The engine and the brief/report surfaces are
+    proven to produce **identical unit graphs**
+    (`tests/workflows/conformance/driver-parity.test.ts`).
+  - **Observability.** `akm workflow watch <run>` tails the run's `workflow_*`
+    / `workflow_unit_*` events as NDJSON (`--stream` foreground-polls to a
+    terminal status, no daemon); `akm workflow status --units` lists per-unit
+    diagnostics (failure reason + result/error text) without feeding them into
+    the deterministic artifact graph; unit lifecycle emits
+    `workflow_unit_started` / `workflow_unit_finished` events carrying
+    ids/status/enums only. `akm show workflow:<name>` summarizes each step's
+    orchestration.
+  - **Harness adapters.** Seven local coding-agent CLIs are first-class
+    dispatch targets — Codex, Copilot CLI, Pi, Gemini, Aider, Amazon Q, and
+    OpenHands — each registered in `HARNESS_REGISTRY` with a command builder +
+    result extractor; agent-identity detection and the session-log provider
+    list are derived from the registry, and harness-native session ids are
+    journaled opportunistically for future session reuse.
+  - **Storage.** Additive `workflow.db` migrations 004–009 (unit journal,
+    harness session ids, frozen plans + run leases, check-in heartbeats,
+    attempt counter, unit claims); migrations 001–003 are untouched and linear
+    workflows behave exactly as before.
+
+  See "Orchestrated steps" and "Driving a run from any agent" in
+  `docs/features/workflows.md`, the redesign addendum in
+  `docs/technical/akm-workflows-orchestration-plan.md`, and `STABILITY.md`
+  (Experimental).
+- **`fable` built-in model alias** — resolves to `claude-fable-5`
+  (`opencode/claude-fable-5` on opencode); recommended resolution target for
+  the `deep` workflow model tier.
+
+### Fixed
+
+- **Check-in directives now survive plain-text output and `workflow
+  status`** (check-in review C2/M1): `formatWorkflowNextPlain` and
+  `formatWorkflowStatusPlain` render the `CONTINUE` directive, and every
+  run-detail response (status/start/complete) evaluates the check-in instead
+  of only `workflow next`.
+- Workflow frontmatter validator error message now lists the actually-allowed
+  keys (`name`, `updated` were missing); removed the documented-but-nonexistent
+  `akm workflow step` alias from `docs/features/workflows.md`.
+
 ## [0.9.0] — 2026-06-30
 
 ### Fixed

STABILITY.md

@@ -95,6 +95,49 @@ for scripted use.
 - **Memory belief-state transitions** — `captureMode`, `beliefState`,
   contradiction edges, and the consolidate journal are observable but
   the algorithm that writes them is tuning across patch releases.
+- **`akm workflow run` + YAML workflow programs** — orchestrated workflows
+  are written as YAML programs (`workflows/*.yaml`, `version: 1`, validated
+  against `schemas/akm-workflow.json`) with `${{ … }}` expressions, per-step
+  fan-out, routing, frozen per-run plans, and an explicit failure policy,
+  executed engine-driven by `akm workflow run`. The R2 engine rework adds
+  journaled replay with content-derived unit identity (input divergence is a
+  hard error), single-driver run leases, typed step artifacts,
+  artifact-judged gates with bounded `max_loops` and required gates
+  (`gate.required`, or the run-wide `akm workflow run --require-gates`, which
+  BLOCK for a human instead of failing open when no judge is available), run
+  budget ceilings (`budget.max_tokens`/`max_units`), the engine concurrency
+  cap knob (`workflow.maxConcurrency` config; unset =
+  `min(16, max(1, cores − 2))`, explicit values clamped to `[1, 64]`),
+  `akm workflow watch`
+  (NDJSON event tail, `--stream`), and `isolation: worktree`. The R3 rework
+  adds a
+  **harness-neutral driver protocol** so any agent session (Claude Code,
+  opencode, Codex, a human at a shell) can drive a run instead of the native
+  engine: **`akm workflow brief <run>`** (read-only; takes no lease and
+  mutates nothing) emits the active step's expected work-list — per-unit
+  resolved instructions, output schema, env binding NAMES only, timeout,
+  and the exact report command lines — and **`akm workflow report <run>
+  --unit <id> --status completed|failed|running`** (the one mutating verb)
+  ingests a unit's result through the SAME shared step semantics the engine
+  uses, enforcing input-hash idempotency/replay-divergence, output-schema
+  validation, budget ceilings, and the artifact-judged gate/`max_loops`
+  completion path. A same-hash re-report of a completed unit is an idempotent
+  no-op; `--rerun` records a fresh attempt for a failed unit. `--status
+  running` claims/heartbeats a unit (a claim holder + expiry, and
+  `last_checkin_at`) for stale-driver detection without advancing the spine.
+  Every report command carries `--expect-step` (refused if the spine has
+  moved), and `report --settle` (no `--unit`) advances a step that dispatches
+  no reportable units — a params-only route, an empty fan-out, or an
+  all-unresolvable work-list — so a driver never wedges.
+  A run is driven by one engine OR one external driver
+  at a time (the run lease arbitrates; `report` is refused while a live
+  engine lease exists), and the two surfaces produce identical unit graphs.
+  The YAML format, its schema, the `run`/`watch`/`brief`/`report` flags, and
+  all JSON output shapes (including `workflow-brief`/`workflow-report`) may
+  all change while the orchestration engine matures. (This format replaced
+  the never-released P1 markdown orchestration subsections.) Classic
+  **linear markdown workflows are unchanged and stable**, as is the workflow
+  CLI contract (`start`/`next`/`complete`/`status`/`list`).
 
 ## On the horizon
 

biome.json

@@ -15,15 +15,20 @@
     }
   },
   "files": {
-    "includes": ["src/**", "tests/**", "!src/assets/templates/html/**"]
+    "includes": ["src/**", "tests/**", "!src/assets/templates/html"]
   },
   "overrides": [
     {
       "includes": [
         "tests/env-secret-tokens.test.ts",
         "tests/env-path-run.test.ts",
         "tests/integration/env-run.test.ts",
-        "tests/commands/env-set-unset.test.ts"
+        "tests/commands/env-set-unset.test.ts",
+        "tests/workflows/program-expressions.test.ts",
+        "tests/workflows/program-parser.test.ts",
+        "tests/workflows/ir-compile.test.ts",
+        "tests/workflows/native-executor.test.ts",
+        "tests/workflows/conformance/conformance.test.ts"
       ],
       "linter": { "rules": { "suspicious": { "noTemplateCurlyInString": "off" } } }
     }

docs/cli.md

@@ -524,7 +524,7 @@ Subcommands:
 | `next <run-id\|ref>` | Return the current actionable step; resumes active runs and starts a new run when the ref has no active run |
 | `complete <run-id> --step <step-id>` | Update the current pending step on an active run and persist status, notes, and evidence |
 | `status <run-id\|ref>` | Show the full run state, including all step statuses |
-| `list` | List workflow runs (optionally filtered by `--ref` and `--active`) |
+| `list` | List workflow runs (optionally filtered by `--ref`; `--active` shows only `status=active` runs, excluding `blocked`/`failed`/`completed`) |
 | `resume <run-id>` | Flip a `blocked` or `failed` run back to `active`. Completed runs cannot be resumed |
 
 Workflow runs are scoped to the current working context, not globally across all

docs/configuration-agent-profiles.md

@@ -141,20 +141,22 @@ binary is on `PATH`):
 | --- | --- | --- | --- | --- |
 | `opencode` | `opencode` | `["run"]` | `interactive` | `opencode` |
 | `claude` | `claude` | `[]` | `interactive` | `claude` |
-| `codex` | `codex` | `[]` | `interactive` | *(none — dispatch errors)* |
+| `codex` | `codex` | `[]` | `interactive` | `codex` |
 | `gemini` | `gemini` | `[]` | `interactive` | *(none — dispatch errors)* |
 | `aider` | `aider` | `["--no-auto-commits"]` | `interactive` | *(none — dispatch errors)* |
 
 Each has a `-headless` variant (e.g. `opencode-headless`) that sets
 `stdio: "captured"` and `parseOutput: "json"` for automation contexts.
 
-The `codex`, `gemini`, and `aider` profiles have no dedicated command builder
+The `gemini` and `aider` profiles have no dedicated command builder
 yet: dispatching them with a prompt/model/system-prompt raises a `ConfigError`
 instead of silently building a command with the wrong flag shapes (aider, for
-example, treats positional args as file names). Interactive launch (no
-dispatch flags) still works. If your CLI is flag-compatible with opencode or
-claude, set `commandBuilder` on a custom profile; dedicated builders for these
-CLIs arrive with the workflow-orchestration harness adapters.
+example, treats positional args as file names). The `codex` profile has a
+dedicated builder that injects `--sandbox workspace-write` automatically.
+Interactive launch (no dispatch flags) still works. If your CLI is
+flag-compatible with opencode or claude, set `commandBuilder` on a custom
+profile; dedicated builders for the remaining CLIs arrive with the
+workflow-orchestration harness adapters.
 
 In v2 config the built-in profile names still resolve — but to use them from
 `profiles.agent`, declare them explicitly so the profile pool is self-contained.

docs/configuration.md

@@ -134,6 +134,7 @@ in-place rewrite. Set `AKM_NO_AUTO_MIGRATE=1` to suppress the rewrite.
 | `feedback.allowedFailureModes` | array | curated set | Restrict the accepted `--failure-mode` values. |
 | `improve.utilityDecay` / `calibration` / `exploration` / `salience` | object | — | Improve-pipeline tuning. See [Advanced improve tuning](#advanced-improve-tuning). |
 | `improve.eventRetentionDays` | number | `90` | Retention window (days) for `state.db` `events`. `0` disables purging. |
+| `workflow.maxConcurrency` | integer | `min(16, max(1, cores − 2))` | Engine-wide ceiling on concurrent units for native workflow fan-out (`akm workflow run`). A step's declared `concurrency:` is clamped to it. Unset = the CPU-derived default; an explicit value is clamped at read time to `[1, 64]`. Does not affect the `akm workflow brief`/`report` driver surface (drivers own their own parallelism). |
 | `stashDir` | string | platform default | Path to the working stash. |
 | `search.minScore` | number | `0.2` | Minimum score floor for semantic-only hits. |
 | `search.graphBoost.directBoostPerEntity` | number | `0.25` | Additive direct-match graph boost per matched entity. |

docs/data-and-telemetry.md

@@ -123,7 +123,8 @@ An append-only log of every mutating action you perform with AKM. Events are sto
 | `proposal_expired` | Proposal expired | `ref` |
 | `events_purged` | Old events deleted by maintenance | `purgedCount`, `retentionDays` |
 | `workflow_started` | `akm workflow start <ref>` | `ref`, `runId` |
-| `workflow_step_completed` | `akm workflow next` | `ref`, `runId`, `stepId` |
+| `workflow_step_completed` | `akm workflow next` (genuine `completed` transition only) | `ref`, `runId`, `stepId`, `status` |
+| `workflow_step_updated` | `akm workflow next` (non-`completed` transitions: `failed`/`skipped`/`blocked`) | `ref`, `runId`, `stepId`, `status` |
 | `workflow_finished` | `akm workflow complete` | `ref`, `runId` |
 | `schema_repair_invoked` | `akm lint --repair` triggered schema repair | `ref` |
 | `archive_cleanup` | Archive cleanup during consolidation | |

docs/features/agent-integration.md

@@ -149,11 +149,12 @@ akm agent opencode agent:code-reviewer --model opencode/claude-opus-4-7 --prompt
 akm agent opencode agent:architect
 ```
 
-**Built-in model aliases** — `opus`, `sonnet`, and `haiku` are resolved
-per platform automatically:
+**Built-in model aliases** — `fable`, `opus`, `sonnet`, and `haiku` are
+resolved per platform automatically:
 
 | Alias | opencode | claude |
 | --- | --- | --- |
+| `fable` | `opencode/claude-fable-5` | `claude-fable-5` |
 | `opus` | `opencode/claude-opus-4-7` | `claude-opus-4-7` |
 | `sonnet` | `opencode/claude-sonnet-4-6` | `claude-sonnet-4-6` |
 | `haiku` | `opencode/claude-haiku-4-5` | `claude-haiku-4-5-20251001` |