diff --git a/CHANGELOG.md b/CHANGELOG.md index 7ae903b..c451351 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,9 @@ surface is governed by [`COMPATIBILITY.md`](COMPATIBILITY.md). agent whose daemon is `kill -9`'d mid-run resumes from its last checkpoint and finishes without re-spending; `./demo.sh` scripts the whole crash-and-recover and proves the loop counter doesn't double. Key-free. +- **Crash-resume guide** ([`docs/RESUME.md`](docs/RESUME.md)) โ€” the full model: what's + restored, the exact-once budget guarantee, writing a resumable agent with + `resume_run`, and the one thing that's yours (idempotent side effects). ### Fixed - **Resume is exact-once across a mid-step crash.** If the daemon died after a step diff --git a/README.md b/README.md index ca531f9..6d1d986 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ It is **not** another gateway (LiteLLM/Portkey own routing), **not** another obs | ๐Ÿ’ธ **Hard cost ceiling per run** | A run that hits its dollar/token budget is killed cleanly, state persisted. Safe defaults out of the box ([the budget contract](docs/BUDGETS.md)). | | ๐Ÿ” **Hard loop-iteration cap** | No more infinite agent loops. | | โฑ๏ธ **Hard wall-clock budget** | Runs that exceed their time budget halt. | -| ๐Ÿ’พ **Crash-resumable checkpoints** | `kill -9` the daemon mid-run; it reloads with the budget already spent and resumes from the last checkpoint โ€” without re-spending ([the flagship demo](examples/kill-9-resume)). | +| ๐Ÿ’พ **Crash-resumable checkpoints** | `kill -9` the daemon mid-run; it reloads with the budget already spent and resumes from the last checkpoint โ€” without re-spending. [Guide](docs/RESUME.md) ยท [demo](examples/kill-9-resume). | | โœ‹ **Framework-agnostic approval gates** | Side-effecting tool calls pause for human approval โ€” CLI, local web, or webhook. | | ๐Ÿง  **Memory you own** | Git-native markdown/YAML on your disk; episodic state in your SQLite. | | ๐Ÿ“ก **OpenTelemetry GenAI** | Emits `gen_ai.*` spans to *your* backend (Grafana/SigNoz/Datadog/Langfuse). | diff --git a/docs/RESUME.md b/docs/RESUME.md new file mode 100644 index 0000000..8477577 --- /dev/null +++ b/docs/RESUME.md @@ -0,0 +1,110 @@ +# Crash-resume โ€” the moat + +A long agent run that crashes shouldn't restart from zero, re-doing work you've +already paid for. RiskKernel persists enough state that a killed run **resumes from +where it left off, without re-spending** โ€” the budget meter is restored exactly, so +a crash can't hand the run a fresh budget, and the agent picks its work back up from +its last checkpoint. + +This is the differentiator. This guide is the full model: what's restored, how to +write a resumable agent, the exact-once guarantee, and the one thing that's *your* +responsibility (idempotent side effects). + +> Want to just *see* it? [`examples/kill-9-resume`](../examples/kill-9-resume) +> `kill -9`s the daemon mid-run and resumes โ€” `./demo.sh` scripts the whole thing. + +## The model โ€” two cooperating halves + +**1. The daemon (automatic).** As a run progresses, the daemon writes its state to +the SQLite file you own โ€” the run row (budget, cumulative usage, status), an +append-only cost ledger, a step row per iteration, and a **checkpoint** after every +model call (and every `run.checkpoint(...)` you make). On startup it **reloads** +every non-terminal run, reconstructing the governor with the budget and usage it had +already spent. Enforcement just continues: a `SIGKILL` can't reset the meter. + +**2. The agent (cooperative).** The runtime can restore the *budget* on its own, but +only your agent knows what *work* it had done. So you **checkpoint your progress** +(a cursor, the messages so far โ€” whatever you need to continue) and, on resume, +**re-attach to the same run** and read that checkpoint back. + +```python +import riskkernel as rk +rt = rk.Runtime() + +with rt.resume_run(run_id) as run: # attach to the existing run (no new run, no cancel) + cp = run.latest_checkpoint() # the state you saved before the crash + start = cp["payload"]["cursor"] if cp else 0 + for i in range(start, total): # skip the steps you already finished + run.step() # counts against the SAME budget + ... do the work ... + run.checkpoint("progress", {"cursor": i + 1}) +``` + +The run id is the only thing your agent must keep across a restart โ€” persist it +wherever you like (a file, your job queue, a DB row). [`resume_run`](../sdks/python/README.md#resume-after-a-crash) +is the SDK entry point; over the API it's just `GET /v1/runs/{id}` + +`GET /v1/checkpoints/{id}` and reusing the id. + +## What's restored + +Everything the budget enforces, exactly as it stood: + +| Restored | Not restored | +|---|---| +| Spent **tokens, dollars, loops** | In-flight work the agent didn't checkpoint | +| The per-run **budget** (and `policyRef`) | The wall-clock **time** budget's clock โ€” it restarts (it meters one active session, not downtime) | +| The cost **ledger** and step history | โ€” | +| Your last **checkpoint payload** | โ€” | + +A resumed run enforces against what it had *already* spent, so it can't overspend by +restarting: if it had burned `$4.20` of a `$5` budget, it resumes at `$4.20`, not +`$0`. + +## Exact-once, even mid-step + +A step is counted (and the run row persisted) in `run.step()` *before* its work runs +and checkpoints. If the daemon dies in that window, the run row is briefly one loop +ahead of the last durable checkpoint. On restart the daemon **reloads from the last +checkpoint**, rolling that partial step back โ€” so resume re-attempts it and the +**loop and dollar budgets are charged exactly once**, never twice. + +What *is* re-attempted is the interrupted step's **work**: at most one step runs +again (never the whole run). Which leads to the one rule that's yours, not ours. + +## Your part: make side-effecting steps idempotent + +Because the interrupted step can run a second time, a step that *does something to +the outside world* โ€” create a PR, charge a card, send an email โ€” must be safe to +re-run. RiskKernel helps but can't do this for you: + +- **It gates side effects.** Side-effecting tool calls route through the + [approval gate](../sdks/python/README.md#human-in-the-loop-tools); a human (or + policy) decides before they run. +- **It records every call.** The cost ledger and the `tool_calls` audit trail + (`riskkernel audit tools `) show exactly what ran, so you can reconcile. + +Design the work itself to be idempotent โ€” an idempotency key, a check-before-write, +or a "have I already done step N?" guard keyed on your checkpoint cursor. The safest +agents checkpoint *after* the side effect commits, so a re-attempt sees it's done. + +## What can't be resumed + +Resume is for *interrupted* runs, not *finished* ones. A run that **halted on its +budget** (`token_/dollar_/loop_/time_budget_exceeded`) or was **cancelled** is +terminal โ€” it stays halted. If a budget halt is what you hit, decide deliberately and +start a **new** run with a bigger budget; RiskKernel won't silently keep spending. + +Check a run's resumability without starting it: + +```bash +riskkernel runs resume # reports: spent / budget / remaining / last checkpoint +riskkernel runs list # every run, its status, and what it's spent +``` + +## Stability + +Once you depend on resume, the on-disk checkpoint format is a compatibility surface +โ€” it's versioned and forward-migratable like the rest of the SQLite schema (see +[`COMPATIBILITY.md`](../COMPATIBILITY.md)). The `Budget` semantics, the +`resume_run` / checkpoint SDK methods, and the `/v1/checkpoints/{id}` and +`/v1/runs/{id}` endpoints are stable across v0.x minor versions. diff --git a/examples/kill-9-resume/README.md b/examples/kill-9-resume/README.md index bc1df04..65a37ff 100644 --- a/examples/kill-9-resume/README.md +++ b/examples/kill-9-resume/README.md @@ -98,10 +98,11 @@ watch `riskkernel audit export ` before and after the crash. ## A note on the crash instant If the daemon dies *in the middle of a step* (after the loop is counted but before -that step's checkpoint), resume re-attempts exactly that one step โ€” at most one -step of work is repeated, never the whole run. Tightening that boundary (partial -tool calls, mid-stream responses) is ongoing hardening; the budget is always -restored exactly. +that step's checkpoint), the daemon rolls that partial step back on restart โ€” so the +**budget is charged exactly once**, never twice. At most one step's *work* is +re-attempted, never the whole run; because that step can run twice, side-effecting +tools should be idempotent. The full model โ€” what's restored, exact-once semantics, +and idempotency โ€” is in the [crash-resume guide](../../docs/RESUME.md). ## Tuning for a recording