diff --git a/CONTEXT.md b/CONTEXT.md index aa878db..4477166 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -16,7 +16,7 @@ ## Autonomous CLI Testing -- Agent-driven e2e runbooks live in `e2e/` (start with `e2e/RULES.md`): scenario documents an agent executes through `scripts/lash-operator.py`, judging CLI semantics against this file's Operator UI contracts. Scripted harnesses (`e2e/restate-postgres-workers`) remain deterministic gate evidence; runbooks are the judged layer on top. +- Agent-driven e2e runbooks live in `runbooks/` (start with `runbooks/RULES.md`): scenario documents an agent executes through `scripts/lash-operator.py`, judging CLI semantics against this file's Operator UI contracts. The scripted CLI E2E harness remains deterministic gate evidence; runbooks are the judged layer on top. - Run `cargo test -p lash-cli --features test-provider --test cli_e2e` to exercise the real `lash` binary without live provider credentials. - The PTY smoke test launches interactive mode with a deterministic `test` provider, types a prompt, waits for rendered output, exits with `/exit`, and validates the generated UI trace/snapshot. - Run `scripts/lash-operator.py --provider test` for an agent-operated PTY session. It builds/launches the real `lash` binary with an isolated deterministic provider, then accepts commands such as `expect 15 Idle`, `type hello`, `key enter`, `screen`, and `lash-exit`. diff --git a/runbooks/RULES.md b/runbooks/RULES.md new file mode 100644 index 0000000..81d66ec --- /dev/null +++ b/runbooks/RULES.md @@ -0,0 +1,154 @@ +# E2E Runbook Rules + +Read this before running any scenario in `runbooks/`. Each runbook links here and does +**not** repeat these rules — it only adds its scenario-specific purpose, golden rules, +phases, and scorecard. + +CLI E2E has **two layers**. The scripted deterministic harness (`cargo test -p lash-cli +--features test-provider --test cli_e2e`) is gate **evidence**: it drives the real binary +and asserts exact outcomes, and it stays code. `runbooks/` is the **agent-judged semantic +layer** on top: you (the agent) drive the real `lash` binary through the PTY operator and +judge the result with your own reasoning, gating on what the CLI actually renders. Keep +the layers separate — a runbook never re-implements a scripted harness, and a scripted +harness never asks for judgement. + +These are **agent-driven runbooks**, not scripts. Use judgement freely — but never skip a +scenario's verification gates or the Abort rule below. + +## What you're testing + +You are testing **lash's operator surface**, not the model and not your own operator +script. The scenario is only valid if the **CLI** — its input handling, runtime, and +render — produces the observed result, not the literal text you typed. When a runbook +asserts a rendered label (`◆ Will send in this turn`), the gate is that lash's queue +projector drew it in response to the ingress you drove, not that the string appears +because you typed it into the draft. When a scenario calls for isolation (a fresh session +that must not see another session's transcript), a leak **voids the run**. + +The contract source for every operator behavior a runbook asserts is **CONTEXT.md → +"Operator UI"**. Cross-check a claim against CONTEXT.md before gating on it. **Do not edit +CONTEXT.md** — a divergence between it and the CLI is a finding to report, never to +reconcile by editing the doc. + +## The operator surface + +CLI scenarios are driven through `scripts/lash-operator.py`, which launches the real `lash` +binary in a child PTY and reads line commands from stdin or `--script FILE`. + +**Launch lines** + +- `scripts/lash-operator.py --provider test` — build + launch with an **isolated + deterministic** provider: a temp `LASH_HOME`, `config.json` pinned to + `{"active_provider":"test"}`, model `test/cli-e2e-model`, scenario `standard-echo`. +- `--scenario ` selects the deterministic test-provider scenario. Available: + `standard-echo` (echoes `test-provider echo: `; a bare prompt echoes + `interactive prompt`, the literal `hello from pty` echoes itself), `standard-slow-echo` + (as echo, but the prompt `slow initial prompt` sleeps ~2s before replying — a bounded + active-turn window), `standard-gated-escape` (the prompt `gated initial prompt` blocks + the turn **indefinitely** until a `gated-initial-release` marker file appears in + `LASH_HOME` — an unbounded active-turn window), and the RLM scenarios + `rlm-subagent-smoke` / `rlm-workspace-smoke` / `rlm-nonzero-exit-smoke` (need `-- -em rlm`). +- `--no-build` launches the existing `target/debug/lash` instead of rebuilding. +- `--lash-home DIR` fixes `LASH_HOME` so sessions and artifacts **persist across runs** + (sessions live in `DIR/sessions/*.db`). Omit it for a throwaway temp home. +- `--trace PATH` passes `--debug-ui-trace PATH` to lash: a replayable UI-trace JSON plus a + final snapshot — the same on-disk artifact the PTY smoke test asserts on. +- `-- ` — everything after `--` is forwarded to `lash` (e.g. `-- -em rlm`). +- `scripts/lash-operator.py --provider real -- --model ` drives the user's + configured provider/credentials. **Deliberate only** — it spends real tokens. + +**Commands** (one per line; `#` and blank lines are ignored) + +- `type TEXT` — type literal text into the draft (no newline). +- `send ESCAPED` — write raw bytes; `\xNN`/`\t`/`\r` escapes are decoded (e.g. `send \x03`). +- `key NAME [COUNT]` — send a named key `COUNT` times. Known: `enter`/`return`, `tab`, + `backtab`, `esc`/`escape`, `ctrl-c`, `ctrl-d`, `backspace`, `delete`, `up`, `down`, + `left`, `right`, `alt-up`, `alt-down`, `pageup`, `pagedown`, `home`, `end`. An unknown + key name is an operator error → Abort. +- `expect [SECS] TEXT` — poll the rendered screen until `TEXT` appears (substring, matched + against raw or ANSI-stripped output). Default 10s. **This is a gate.** A timeout, or + lash exiting before the text appears, raises and stops the run. +- `expect-re [SECS] REGEX` — as `expect`, but a multiline regex over the stripped screen. +- `wait SECS` — sleep. **Never a gate** (see below). +- `screen [LINES]` / `raw [LINES]` — print the last N stripped / raw lines for inspection. +- `clear` — drop the capture buffer (so the next `screen`/`expect` only sees new frames). +- `status` — print the child exit code (`None` while running). +- `lash-exit [SECS]` — send `/exit`, then wait up to SECS for a clean process exit. +- `kill` — SIGTERM/SIGKILL the child process group. `quit` — stop the driver. + +**Pre-flight** (every runbook's Phase 0): launch, and confirm the launched binary carries +the deterministic provider. A startup failure `provider 'test' is not supported by this +CLI build` means the binary was compiled **without** `--features test-provider` — rebuild +(`cargo build -p lash-cli --features test-provider`) and make sure the launched +`target/debug/lash` is that fresh artifact, then Abort/RCA the run and note it as a +harness gap. Confirm the idle prompt renders (`expect 20 Message · / for commands`) before +driving anything. If a scenario needs a key, scenario, or surface the operator cannot +reach, that missing capability → Abort/RCA, noted as a harness gap (do not pretend around +it). + +## Poll, don't sleep + +Turns, tool calls, subagent spawns, and queued-turn dispatch are all async and render over +several frames. Gate on the **rendered outcome** with `expect ` — never `wait +N`. Submit a turn, then `expect `; the footer status walks +`Idle → Working → Thinking → Responding → Idle` (and `Running tool · ` for tool +calls), so gate on the terminal render, not a fixed sleep. `wait` is only for letting the +render settle before a `screen`, or spacing session-file writes so timestamp filenames +don't collide — **never** to decide that async work finished. An `expect` timeout at a gate +is a hard failure → Abort/RCA. `expect` also fails fast if lash exits first (`lash exited +with N before '' appeared`) — that is itself an Abort trigger. + +## Gate objectively before you judge + +Prefer an objective signal over eyeballing. In order of authority: + +1. **Rendered screen** — the `expect`/`screen` gate: footer status labels, the user `●` / + assistant `■` markers, queue previews (`◆ Will send in this turn` / `◇ Queued for next + turn`), overlay boxes (`Resume Session (n/m)`, the suggestion popup, `/help` / `/info` + document overlays). This is what the operator surface actually shows a user. +2. **UI trace / snapshot** (`--trace PATH`) — the durable record of what was *submitted*, + independent of what rendered: ops like `user_turn`, `queue_current_turn_input` (Early + Injection), `queue_turn` (Next Full Turn). Use it to prove intent-vs-render (e.g. Enter + mid-turn recorded exactly one `queue_current_turn_input` and zero `queue_turn`). +3. **On-disk state** — `LASH_HOME/sessions/*.db` (durable session evidence; message counts + drive the resume picker), and `LASH_HOME/test-provider-requests.jsonl` (the visible user + texts each provider call actually saw — objective proof of what reached the model, and + how many times). + +Run the structural gate **before** judging behavior. If the objective signal is missing +(the label never rendered, the trace op is absent), the failure is upstream of anything you +would judge — Abort/RCA, don't score the vibe. + +## When to STOP (Abort triggers) + +Stop immediately on **any** of: + +- an operator command error (an `ERROR ...` line / non-zero driver exit); +- an `expect` / `expect-re` timeout at a gate; +- lash exiting unexpectedly before a gate (`lash exited with N before ...`); +- a **contract violation** — an isolation or invariant break: a supposedly-hidden empty + session surfacing in the resume picker, a rendered queue label that contradicts the + ingress you drove, a Runtime Process ending merely because its session was deleted, or a + document-overlay that won't close on `Esc`/`Ctrl+C`; +- an assertion that contradicts the scenario's answer key. + +Do not push through, do not paper over, do not attempt a fix as part of the run. + +## How to REPORT + +**On abort — RCA, then stop:** +1. **Stop.** Do not continue the scenario. +2. **Capture evidence** — the failing operator command and its `ERROR` tail; the last + `screen`; the child `status`; the `LASH_HOME` path and any `--trace` artifact / session + db involved; and the exact gate string that failed. +3. **RCA** — symptom → the CLI stage it broke at (input handling / suggestion+modal + dismissal / turn submit / runtime execution / render / session persistence) → root + cause → the evidence that proves it. Never stop at "the expect timed out." +4. **Report and stop.** This is a diagnosis, not a repair. A divergence between an observed + behavior and CONTEXT.md or the docs is reported as a finding — **do not** edit the doc + or the code to make the run pass. + +**On success — score, don't vibe:** for each scored item, name the **specific rendered +string** (or trace op / on-disk fact) the gate matched — no credit for vibes. Mark the +objective gate (screen / trace / disk) separately from any judged behavior. Fill the +scenario's scorecard verbatim. diff --git a/runbooks/cancel-semantics/runbook.md b/runbooks/cancel-semantics/runbook.md new file mode 100644 index 0000000..820a385 --- /dev/null +++ b/runbooks/cancel-semantics/runbook.md @@ -0,0 +1,138 @@ +# E2E Scenario: Cancel Semantics — The Ctrl+C Ladder + +> **Read [../RULES.md](../RULES.md) first** — operator surface, poll-don't-sleep, stop +> triggers, and reporting/RCA conventions. This runbook only adds the scenario-specific +> parts. + +**Purpose.** Prove the inline operator's cooperative cancel surface walks its documented +`Ctrl+C` ladder, one rung per press, in order: +**close a suggestion/overlay → cancel the active turn → clear a non-empty draft → quit only +from an idle empty prompt.** Each rung is gated objectively, and a single escalating +sequence of four presses peels all four layers so the *ordering* — not just each behavior in +isolation — is what's under test. + +**Why this matters.** `Ctrl+C` is overloaded on purpose (CONTEXT.md → "Operator UI": +"reserved for cancel/dismiss/quit semantics"). If the rungs fired out of order — quitting +while a turn ran, or cancelling a turn when the user only meant to close a popup — the +operator would destroy work on a keystroke people press reflexively. Copy is `Ctrl+Shift+C` +precisely so `Ctrl+C` can own this ladder. + +## Scenario-specific golden rules + +1. **One rung per press, in order.** With a suggestion popup open, an active turn, and a + non-empty draft all present, the four presses must fire suggestion-dismiss → turn-cancel + → draft-clear → quit — never skip a rung, never quit early. +2. **Cancel must reach a terminal state.** Rung 2 is not "the turn looked interrupted" — it + is the turn committing to `Manually interrupted.` and the footer returning to `Idle`. +3. **Quit only from idle + empty.** The final press exits the process (clean status 0) only + because the prompt is idle and empty; any earlier press that exits is a violation. +4. **Interrupted means a committed Lash terminal.** The rendered interruption is never + inferred from killing an engine invocation. Engine break-glass cancellation is an + operational failure and must not be presented as `Cancelled`. Exact-address durable + request and evidence gates live in the Lash repository's scripted Restate harness and + companion `runbooks/workbench-durable-stop/runbook.md` browser runbook. + +## Working material + +- Scenario `standard-gated-escape`, holding prompt `gated initial prompt` (an unbounded + active turn, so rung 2 can't win a race by accident). Typing `/` opens the slash-command + suggestion popup for rung 1; that `/` draft is the non-empty draft rung 3 clears. + +## Phase 0 — Pre-flight + +Per [../RULES.md](../RULES.md). Launch `scripts/lash-operator.py --provider test --scenario +standard-gated-escape`, confirm the deterministic provider, gate the idle prompt +(`expect 20 Message · / for commands`). + +## Phase 1 — Arm all four layers + +Start the holding turn, confirm it is active, then open the suggestion popup on top of it: + +``` +type gated initial prompt +key enter +expect 20 gated initial prompt +expect 15 Thinking +type / +expect 8 Reset conversation +``` + +State now: active turn (`Thinking`), a non-empty draft (`/`), and the suggestion popup open +(the bordered list showing `/clear Reset conversation`, `/controls`, …). All four ladder +layers are armed. + +## Phase 2 — Walk the ladder, one `Ctrl+C` at a time + +**Rung 1 — close the suggestion overlay.** The popup closes; the `/` draft and the active +turn survive. + +``` +key ctrl-c +wait 1 +screen 20 +``` + +Gate: the suggestion border is gone, the input still shows `❯ /`, and the footer still reads +`Thinking`. (Absence of the popup — inspect the `screen`; the border box and `Reset +conversation` must be gone.) + +**Rung 2 — cancel the active turn.** The turn commits to its interrupted terminal state. + +``` +key ctrl-c +expect 20 Manually interrupted +expect 10 Idle +``` + +Gate: `Manually interrupted.` renders and the footer returns to `Idle`. A turn that keeps +running, or errors instead of interrupting, → Abort/RCA (turn cancel). The `/` draft still +survives. + +**Rung 3 — clear the non-empty draft.** The `/` draft is discarded; the idle placeholder +returns. + +``` +clear +key ctrl-c +wait 1 +expect 5 Message · / for commands +``` + +Gate: the input returns to the `❯ Message · / for commands` placeholder (empty draft). The +process is **still running** — confirm with `status` (`None`). + +**Rung 4 — quit from idle + empty.** Only now does `Ctrl+C` exit. + +``` +key ctrl-c +wait 1 +status +``` + +Gate: the child has exited — `status` reports `STATUS 0`. (The `wait` only lets the process +reap so `status` doesn't poll a still-shutting-down child; the exit — not the sleep — is the +gate. A *clean* exit here is the expected outcome of this rung, not an Abort trigger.) + +## Phase 3 — Score + +| Item | Objective gate | Verdict | Notes | +|------|----------------|---------|-------| +| Rung 1: suggestion closes | popup + `Reset conversation` gone; `/` draft + `Thinking` survive | | | +| Rung 2: active turn cancels | `Manually interrupted.` + footer `Idle` | | | +| Rung 3: draft clears | `❯ Message · / for commands` returns; process still alive | | | +| Rung 4: quit from idle empty | child exits `0` | | | +| Order held | exactly four presses, one rung each, none out of order | | | +| Semantic layer | interruption is a committed cooperative turn result, not engine break-glass | | rendered terminal + companion gate references | + +**Aggregate:** did four presses peel four layers in the documented order, with the turn +reaching a real interrupted terminal state and the quit firing only from idle + empty. + +This CLI scenario deliberately does not invent an external cancellation API. The Lash +repository additionally gates the first-party primitive with `just +restate-postgres-workers-e2e` (exact request, replay, race, terminal evidence, and +break-glass distinction) and `runbooks/workbench-durable-stop/runbook.md` (browser/API +agreement across web-process restart). + +--- + +_Stop triggers and the Abort/RCA + reporting protocol are in [../RULES.md](../RULES.md)._ diff --git a/runbooks/process-dock-lifecycle/runbook.md b/runbooks/process-dock-lifecycle/runbook.md new file mode 100644 index 0000000..6602fec --- /dev/null +++ b/runbooks/process-dock-lifecycle/runbook.md @@ -0,0 +1,137 @@ +# E2E Scenario: Process Dock Lifecycle — Visibility, Cancel, and the Deletion Invariant + +> **Read [../RULES.md](../RULES.md) first** — operator surface, poll-don't-sleep, stop +> triggers, and reporting/RCA conventions. This runbook only adds the scenario-specific +> parts. + +**Purpose.** Prove the CLI process dock's contract: a **Runtime Process** shows in the dock +(header `Background`), the focused process can be cancelled (`Delete`), and — the invariant +— **ending or deleting a session never ends a process by itself**. Only runtime processes +appear in the dock; a subagent spawn does not. + +**Why this matters.** The Lash repository's CONTEXT.md → "Runtime Process" says its +lifecycle is independent of any session and only runtime processes appear in the CLI +process dock. That independence is the whole point of a durable process; if session +teardown silently killed it, the durability guarantee would be a lie. + +## Pre-flight harness gap (read before scoring) + +**A Runtime Process visible in the dock is not reachable under `--provider test`.** Verified +by dry-run against every deterministic scenario: + +- There is **no `/process` slash command** — the builtin set is `/clear /compact /controls + /fork /tree /version /info /model /variant /mode /provider /logout /retry /resume /skills + /help /exit` (`crates/lash-cli/src/command.rs`). The dock is a render surface fed by + `session.processes().list()`, not a command. +- The RLM `rlm-subagent-smoke` scenario **does** spawn a subagent, but it renders as an + **inline tool activity** — `◆ spawn subagent · …` with footer `Running tool · + spawn_agent` — and the `Background` dock header never appears. A subagent is not a dock + Runtime Process (consistent with the contract: "Only runtime processes appear"). +- The deterministic test provider only returns canned responses, so no scenario leaves a + durable Runtime Process running; the dock stays empty (`app.processes.is_empty()` → + `crates/lash-cli/src/render/sections/docks.rs` draws nothing). A **Process Engine** that + produces dock processes is contributed by an installed plugin, not wired by the test + provider. + +Per [../RULES.md](../RULES.md)'s "missing capability → note as harness gap": this runbook +runs the **drivable negative-space gates** below under `--provider test`, and specifies the +**full positive procedure** for `--provider real` with a process-engine plugin. Do not +fabricate a dock process that the test provider cannot produce. + +## Scenario-specific golden rules + +1. **Subagent ≠ dock process.** A spawned subagent is an inline tool activity; it must + **not** raise the `Background` dock. Confirming that separation is a real gate, not a + consolation prize. +2. **The deletion invariant is the crown jewel.** In the positive (real-provider) run, the + process must **survive** its originating session being ended/deleted. A process that dies + with its session is a hard fail. +3. **Don't invent a process.** If the dock is empty, gate the emptiness honestly; don't + read a tool activity as a dock entry. + +## Phase 0 — Pre-flight + +Per [../RULES.md](../RULES.md). For the negative-space gates, launch +`scripts/lash-operator.py --provider test --scenario rlm-subagent-smoke -- -em rlm` and +confirm the deterministic provider and idle prompt. + +## Phase 1 — Drivable gates under `--provider test` (negative space) + +**Subagent renders as a tool activity, not a dock process.** + +``` +type Does your subagent tool work +key enter +expect 15 Running tool +expect 45 subagent-ok +screen 40 +``` + +Gate: the spawn shows as `◆ spawn subagent · …` and the footer passes through `Running tool +· spawn_agent`; the settled value is `■ subagent-ok`. The `Background` dock header is +**absent** the entire time (inspect the `screen`). A `Background` dock appearing here would +itself be a contract surprise → investigate/report. Then `lash-exit 10`. + +**Empty dock has nothing to focus.** Relaunch `--scenario standard-echo`. With an idle empty +prompt and no processes, the dock-focus keys have no target: + +``` +expect 20 Message · / for commands +key tab +wait 1 +screen 18 +``` + +Gate: no `Background` dock, and `Tab` opens no process overview (the dock-focus binding — +`docs/index.html`: "With an empty prompt, cycle focus through the dock of background +processes" — falls through when the dock is empty; note that `Shift+Tab` here falls through +to the plan-mode toggle, further evidence there is no dock to cycle). The process **cancel** +(`Delete`) and **overview** (`Enter`) rungs cannot be exercised — no process exists to +focus. Record this as the harness gap. Then `lash-exit 10`. + +## Phase 2 — Full positive procedure (`--provider real`, needs a process-engine plugin) + +Not runnable under the deterministic provider; specified so a real-provider run (RLM mode +plus a plugin whose tool starts a **durable Runtime Process**, e.g. a long-running +background job that yields a process handle) can execute it verbatim. Launch +`scripts/lash-operator.py --provider real -- --model -em rlm` (spends +tokens — deliberate). + +1. **Start a process.** Drive a turn whose tool starts a durable background process. Gate: + the `Background` dock renders a row `◆ running · ·