prashar32 · prashar32 · Jun 5, 2026 · Jun 5, 2026
@@ -15,6 +15,10 @@ surface is governed by [`COMPATIBILITY.md`](COMPATIBILITY.md).
   Python agent can pick its work back up from the last checkpoint after a `SIGKILL`.
   The run keeps its server-side budget and already-spent usage, so it can't
   overspend by restarting. See the [SDK README](sdks/python/README.md#resume-after-a-crash).
+- **`examples/kill-9-resume`** — the flagship crash-resume demo. A checkpointing
+  agent whose daemon is `kill -9`'d mid-run resumes from its last checkpoint and
+  finishes without re-spending; `./demo.sh` scripts the whole crash-and-recover and
+  proves the loop counter doesn't double. Key-free.
 
 ## [0.2.0] - 2026-06-04
 

@@ -36,7 +36,7 @@ It is **not** another gateway (LiteLLM/Portkey own routing), **not** another obs
 | 💸 **Hard cost ceiling per run** | A run that hits its dollar/token budget is killed cleanly, state persisted. Safe defaults out of the box ([the budget contract](docs/BUDGETS.md)). |
 | 🔁 **Hard loop-iteration cap** | No more infinite agent loops. |
 | ⏱️ **Hard wall-clock budget** | Runs that exceed their time budget halt. |
-| 💾 **Crash-resumable checkpoints** | `SIGKILL` a run; `riskkernel runs resume <id>` picks up from the last step. |
+| 💾 **Crash-resumable checkpoints** | `kill -9` the daemon mid-run; it reloads with the budget already spent and resumes from the last checkpoint — without re-spending ([the flagship demo](examples/kill-9-resume)). |
 | ✋ **Framework-agnostic approval gates** | Side-effecting tool calls pause for human approval — CLI, local web, or webhook. |
 | 🧠 **Memory you own** | Git-native markdown/YAML on your disk; episodic state in your SQLite. |
 | 📡 **OpenTelemetry GenAI** | Emits `gen_ai.*` spans to *your* backend (Grafana/SigNoz/Datadog/Langfuse). |
@@ -111,6 +111,10 @@ Want to *see* the headline feature? [`examples/codebase-qa`](examples/codebase-q
 is a runnable agent that loops over a codebase until the governor kills it on its
 loop/dollar budget — the deterministic kill, end to end, with a real model.
 
+And the moat: [`examples/kill-9-resume`](examples/kill-9-resume) `kill -9`s the
+daemon mid-run and resumes without re-spending — `./demo.sh` scripts the whole
+crash-and-recover and proves the counter doesn't double, key-free.
+
 Brand new to the SDK? [`examples/wrap-your-agent`](examples/wrap-your-agent) is the
 no-key, two-minute version — a generic Python loop the governor caps at a loop
 budget, the deterministic kill with nothing running but the daemon.

@@ -0,0 +1,110 @@
+# kill-9-resume — a crashed run resumes without re-spending
+
+**The flagship.** An agent does expensive, checkpointed work under a governed run.
+The RiskKernel daemon is **`kill -9`'d mid-run** — a hard crash, no graceful
+shutdown. On restart the daemon reloads the run with the budget and usage it had
+already spent; re-running the agent attaches to that same run (`resume_run`), reads
+its last checkpoint, and finishes the work — **without redoing (re-paying for) the
+steps already done.**
+
+The proof is one number: across the crash the governor's loop counter ends at
+**10** (one per step of work), not 15 — it never restarts from zero, and never
+re-spends.
+
+## Run it (one command)
+
+`demo.sh` scripts the whole thing reproducibly: start daemon → 5 of 10 steps →
+`kill -9` the daemon → restart → resume → finish → show the proof.
+
+```bash
+cd examples/kill-9-resume
+pip install -r requirements.txt           # the RiskKernel SDK (stdlib-only)
+
+# needs the daemon binary + a python with the SDK:
+RISKKERNEL_BIN=../../riskkernel ./demo.sh
+#   (or, with the CLI on your PATH:  ./demo.sh)
+```
+
+## What you'll see
+
+```
+── 2. agent does 5 of 10 steps, checkpointing each ────────────────────
+▶ FRESH run 4e25ad71-…   (budget: loops=50)
+  step  1/10 done   (checkpointed cursor=1)
+  …
+  step  5/10 done   (checkpointed cursor=5)
+⏸  did 5 steps, then stopping (the demo crashes the daemon here).
+
+── 3. kill -9 the daemon  (a hard crash — no graceful shutdown) ───────
+   daemon killed. restarting it…
+   ✓ msg="resumed runs from store" count=1
+
+── 4. re-run the agent: it RESUMES and finishes ───────────────────────
+↻ RESUMING run 4e25ad71-…
+  the governor already counts 5 spent steps — resuming at cursor 5, not redoing them.
+  step  6/10 done   (checkpointed cursor=6)
+  …
+  step 10/10 done   (checkpointed cursor=10)
+✅ completed all 10 steps. governor loop counter = 10 — exactly one per step of
+   work. The steps finished before the crash were neither redone nor re-paid.
+
+── 5. proof — the run did 10 steps total across the crash, not 15 ─────
+  …  kill-9-resume  running  …  LOOPS=10  …
+```
+
+## Do it by hand (for a live recording)
+
+```bash
+# terminal 1 — the daemon
+riskkernel serve
+
+# terminal 2 — the agent. it checkpoints each step.
+python agent.py
+#   …step 1…2…3…  ← now CRASH the daemon: in terminal 1, Ctrl-C twice or `kill -9 <pid>`
+#   the agent prints: "💥 the daemon is gone … restart and re-run to resume"
+
+# terminal 1 — restart it; it logs  "resumed runs from store count=1"
+riskkernel serve
+
+# terminal 2 — re-run the SAME script. it resumes from the last checkpoint:
+python agent.py
+#   "↻ RESUMING … not redoing them"  → finishes → loop counter = number of steps, not double
+```
+
+## How it works
+
+Three pieces, all already in the runtime — the example just wires them:
+
+1. **Checkpoint each step.** `run.checkpoint("progress", {"cursor": i+1})` durably
+   saves *where you are* in the run's SQLite state (the daemon also snapshots
+   cumulative usage after every step).
+2. **Reload on restart.** The daemon reloads non-terminal runs on boot,
+   reconstructing each governor with the budget and usage it had already spent —
+   so enforcement continues; a SIGKILL can't reset the meter.
+3. **Attach and continue.** `with rt.resume_run(run_id) as run:` re-attaches to the
+   run (it neither creates a new one nor cancels it); `run.latest_checkpoint()`
+   gives the cursor to resume from. The remaining steps run against the **same
+   budget**.
+
+## The "$ not double-counting" headline
+
+This demo counts **loops** so it needs no API key. The **dollar** counter behaves
+identically: `Reload` restores spent dollars and tokens too, so a run that had
+burned `$4.20` of a `$5` budget resumes at `$4.20`, not `$0` — it can't get a fresh
+budget by crashing. To see it on real spend, route your model through the run's
+proxy (`run.proxy_config()`, as in [`examples/codebase-qa`](../codebase-qa)) and
+watch `riskkernel audit export <run-id>` before and after the crash.
+
+## A note on the crash instant
+
+If the daemon dies *in the middle of a step* (after the loop is counted but before
+that step's checkpoint), resume re-attempts exactly that one step — at most one
+step of work is repeated, never the whole run. Tightening that boundary (partial
+tool calls, mid-stream responses) is ongoing hardening; the budget is always
+restored exactly.
+
+## Tuning for a recording
+
+- `RK_TOTAL_STEPS` (default 10), `RK_WORK_SECONDS` (default 0.4) — total work and
+  per-step delay. `RK_STOP_AFTER` makes the agent stop cleanly after N steps (the
+  orchestrator uses it to crash at a known point).
@@ -0,0 +1,101 @@
+#!/usr/bin/env python3
+"""kill-9-resume — a crashed run resumes without re-spending. THE flagship.
+
+An agent does expensive, checkpointed work under a governed run. The RiskKernel
+daemon is `kill -9`'d mid-run — a hard crash, no graceful shutdown. On restart the
+daemon reloads the run with the budget and usage it had already spent; re-running
+this agent attaches to that same run (``resume_run``), reads its last checkpoint,
+and finishes the work — WITHOUT redoing (re-paying for) the steps already done.
+
+The proof: across the crash the governor's loop counter ends at exactly the number
+of steps of *work* (10), not double (it never restarts from zero).
+
+Run it twice with a daemon crash in between — or just run ``./demo.sh``, which
+scripts the whole thing. This agent owns one piece of state: a file holding the
+run id to resume (``$RK_RUN_ID_FILE``, default ``.resume-run-id``).
+"""
+
+from __future__ import annotations
+
+import os
+import time
+
+import riskkernel as rk
+
+DAEMON_URL = os.environ.get("RISKKERNEL_BASE_URL", "http://localhost:7070")
+TOTAL_STEPS = int(os.environ.get("RK_TOTAL_STEPS", "10"))
+WORK_SECONDS = float(os.environ.get("RK_WORK_SECONDS", "0.4"))
+RUN_ID_FILE = os.environ.get("RK_RUN_ID_FILE", ".resume-run-id")
+# Demo knob: stop cleanly after this many total steps (so the orchestrator can
+# crash the daemon at a known point). 0 = run to completion.
+STOP_AFTER = int(os.environ.get("RK_STOP_AFTER", "0")) or TOTAL_STEPS
+
+
+def do_expensive_work(step: int) -> None:
+    """Stand-in for the per-step work you don't want to pay for twice — a model
+    call, a tool invocation, a long computation. Here it just sleeps."""
+    time.sleep(WORK_SECONDS)
+
+
+def run_loop(run: "rk.Run", start: int) -> int:
+    """Do steps [start, TOTAL_STEPS), checkpointing each. Returns the next cursor."""
+    i = start
+    while i < TOTAL_STEPS:
+        if i >= STOP_AFTER:
+            print(f"\n⏸  did {i} steps, then stopping (the demo crashes the daemon here).")
+            return i
+        run.step()                                   # one governed step (counts vs the budget)
+        do_expensive_work(i)
+        run.checkpoint("progress", {"cursor": i + 1})  # save WHERE we are, durably
+        print(f"  step {i + 1:>2}/{TOTAL_STEPS} done   (checkpointed cursor={i + 1})")
+        i += 1
+    return i
+
+
+def _finish(run: "rk.Run") -> None:
+    loops = run.status().get("usage", {}).get("loops")
+    if os.path.exists(RUN_ID_FILE):
+        os.remove(RUN_ID_FILE)
+    print(f"\n✅ completed all {TOTAL_STEPS} steps. governor loop counter = {loops} "
+          f"— exactly one per step of work. The steps finished before the crash were "
+          f"neither redone nor re-paid.")
+
+
+def main() -> int:
+    rt = rk.Runtime(base_url=DAEMON_URL)
+    try:
+        if os.path.exists(RUN_ID_FILE):
+            # ── RESUME path: attach to the existing run after the crash ──
+            run_id = open(RUN_ID_FILE).read().strip()
+            with rt.resume_run(run_id) as run:
+                cp = run.latest_checkpoint()
+                start = int(cp["payload"]["cursor"]) if cp and cp.get("payload") else 0
+                spent = run.status().get("usage", {}).get("loops", start)
+                print(f"↻ RESUMING run {run_id}\n"
+                      f"  the governor already counts {spent} spent steps — resuming at "
+                      f"cursor {start}, not redoing them.\n")
+                end = run_loop(run, start)
+                if end >= TOTAL_STEPS:
+                    _finish(run)
+        else:
+            # ── FRESH path: open a new governed run ──
+            budget = rt.budget(loops=TOTAL_STEPS * 5, seconds=3600)  # generous; the demo is about resume, not halt
+            with rt.governed_run(name="kill-9-resume", budget=budget) as run:
+                open(RUN_ID_FILE, "w").write(run.id)
+                print(f"▶ FRESH run {run.id}   (budget: loops={TOTAL_STEPS * 5})\n")
+                end = run_loop(run, 0)
+                if end >= TOTAL_STEPS:
+                    _finish(run)
+        return 0
+    except rk.APIError as e:
+        if e.code == "connection_error":
+            print("\n💥 the daemon is gone (kill -9?). Your progress is safe — it's "
+                  "checkpointed in the run's SQLite state.\n"
+                  "   Restart the daemon and re-run this script: it RESUMES from the last "
+                  "checkpoint, no work re-paid.")
+            return 1
+        raise
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+# kill-9-resume — scripts the flagship demo end to end, reproducibly:
+#   start daemon → agent does 5/10 steps → kill -9 the daemon → restart →
+#   agent RESUMES and finishes → prove the loop counter is 10, not 15.
+#
+# Prereqs: the `riskkernel` binary and a Python with the SDK installed.
+#   RISKKERNEL_BIN  path to the daemon binary   (default: riskkernel on PATH)
+#   PYTHON          python with the SDK          (default: python3)
+# e.g.   RISKKERNEL_BIN=../../riskkernel PYTHON=.venv/bin/python ./demo.sh
+set -uo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+BIN="${RISKKERNEL_BIN:-riskkernel}"
+PY="${PYTHON:-python3}"
+PORT="${RK_PORT:-7070}"
+DATA="$(mktemp -d)"
+export RISKKERNEL_BASE_URL="http://localhost:${PORT}"
+export RK_RUN_ID_FILE="${DATA}/run-id"
+export RK_PORT="${PORT}"
+
+cleanup() { [[ -n "${PID:-}" ]] && kill "${PID}" 2>/dev/null; rm -rf "${DATA}"; }
+trap cleanup EXIT
+
+start_daemon() {
+  RISKKERNEL_DATA_DIR="${DATA}" RISKKERNEL_PORT="${PORT}" "${BIN}" serve >>"${DATA}/serve.log" 2>&1 &
+  PID=$!
+  for _ in $(seq 1 40); do
+    curl -sf "http://localhost:${PORT}/healthz" >/dev/null 2>&1 && return 0
+    sleep 0.25
+  done
+  echo "✗ daemon didn't come up — see ${DATA}/serve.log" >&2; exit 1
+}
+
+echo "── 1. start the daemon ────────────────────────────────────────────────"
+start_daemon
+echo "   up (pid ${PID})"
+
+echo
+echo "── 2. agent does 5 of 10 steps, checkpointing each ────────────────────"
+RK_STOP_AFTER=5 "${PY}" "${HERE}/agent.py"
+
+echo
+echo "── 3. kill -9 the daemon  (a hard crash — no graceful shutdown) ───────"
+kill -9 "${PID}"; sleep 0.5
+echo "   daemon killed. restarting it…"
+start_daemon
+grep -i "resumed runs" "${DATA}/serve.log" | tail -1 | sed 's/^/   ✓ /' || true
+
+echo
+echo "── 4. re-run the agent: it RESUMES and finishes ───────────────────────"
+"${PY}" "${HERE}/agent.py"
+
+echo
+echo "── 5. proof — the run did 10 steps total across the crash, not 15 ─────"
+RISKKERNEL_DATA_DIR="${DATA}" "${BIN}" runs list 2>/dev/null | grep -v "state store ready" || true
+echo
+echo "   loops = 10 (one per step of work). Re-running from zero would read 15."
@@ -0,0 +1,6 @@
+# The only dependency is the RiskKernel Python SDK (stdlib-only). It isn't on PyPI
+# yet, so it's installed from source. resume_run() needs SDK >= 0.2.0.
+riskkernel @ git+https://github.com/prashar32/riskkernel.git#subdirectory=sdks/python
+
+# Working inside a clone and want your local SDK instead? From THIS directory:
+#     pip install ../../sdks/python