From 10ab9a94e543fc173ff50ddb4bb16924089f6cae Mon Sep 17 00:00:00 2001 From: Adarsh Prashar Date: Fri, 5 Jun 2026 20:36:43 +0530 Subject: [PATCH] docs(examples): kill -9 -> resume without re-spending (the flagship) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The crash-resumable moat, made into a recordable, reproducible demo. A checkpointing agent runs under a governed run; the daemon is kill -9'd mid-run; on restart it reloads the run with the budget it had already spent, and re-running the agent (via resume_run) picks up from the last checkpoint and finishes β€” without redoing the paid steps. demo.sh scripts the whole money shot deterministically: 5 of 10 steps -> kill -9 the daemon -> restart -> resume -> finish, then proves the governor's loop counter is 10, not 15. Key-free: the loop counter is the proof; the README shows the dollar counter behaves identically (Reload restores spent dollars/tokens too). agent.py is the resumable agent β€” create-or-resume via a run-id file, checkpoint each step, and a clear "the daemon is gone, restart and re-run to resume" message when it can't reach the daemon. Linked prominently from the README as the moat. --- CHANGELOG.md | 4 + README.md | 6 +- examples/kill-9-resume/README.md | 110 ++++++++++++++++++++++++ examples/kill-9-resume/agent.py | 101 ++++++++++++++++++++++ examples/kill-9-resume/demo.sh | 57 ++++++++++++ examples/kill-9-resume/requirements.txt | 6 ++ 6 files changed, 283 insertions(+), 1 deletion(-) create mode 100644 examples/kill-9-resume/README.md create mode 100644 examples/kill-9-resume/agent.py create mode 100755 examples/kill-9-resume/demo.sh create mode 100644 examples/kill-9-resume/requirements.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index 00bbc39..77a713d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,10 @@ surface is governed by [`COMPATIBILITY.md`](COMPATIBILITY.md). Python agent can pick its work back up from the last checkpoint after a `SIGKILL`. The run keeps its server-side budget and already-spent usage, so it can't overspend by restarting. See the [SDK README](sdks/python/README.md#resume-after-a-crash). +- **`examples/kill-9-resume`** β€” the flagship crash-resume demo. A checkpointing + agent whose daemon is `kill -9`'d mid-run resumes from its last checkpoint and + finishes without re-spending; `./demo.sh` scripts the whole crash-and-recover and + proves the loop counter doesn't double. Key-free. ## [0.2.0] - 2026-06-04 diff --git a/README.md b/README.md index 229deb7..ca531f9 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ It is **not** another gateway (LiteLLM/Portkey own routing), **not** another obs | πŸ’Έ **Hard cost ceiling per run** | A run that hits its dollar/token budget is killed cleanly, state persisted. Safe defaults out of the box ([the budget contract](docs/BUDGETS.md)). | | πŸ” **Hard loop-iteration cap** | No more infinite agent loops. | | ⏱️ **Hard wall-clock budget** | Runs that exceed their time budget halt. | -| πŸ’Ύ **Crash-resumable checkpoints** | `SIGKILL` a run; `riskkernel runs resume ` picks up from the last step. | +| πŸ’Ύ **Crash-resumable checkpoints** | `kill -9` the daemon mid-run; it reloads with the budget already spent and resumes from the last checkpoint β€” without re-spending ([the flagship demo](examples/kill-9-resume)). | | βœ‹ **Framework-agnostic approval gates** | Side-effecting tool calls pause for human approval β€” CLI, local web, or webhook. | | 🧠 **Memory you own** | Git-native markdown/YAML on your disk; episodic state in your SQLite. | | πŸ“‘ **OpenTelemetry GenAI** | Emits `gen_ai.*` spans to *your* backend (Grafana/SigNoz/Datadog/Langfuse). | @@ -111,6 +111,10 @@ Want to *see* the headline feature? [`examples/codebase-qa`](examples/codebase-q is a runnable agent that loops over a codebase until the governor kills it on its loop/dollar budget β€” the deterministic kill, end to end, with a real model. +And the moat: [`examples/kill-9-resume`](examples/kill-9-resume) `kill -9`s the +daemon mid-run and resumes without re-spending β€” `./demo.sh` scripts the whole +crash-and-recover and proves the counter doesn't double, key-free. + Brand new to the SDK? [`examples/wrap-your-agent`](examples/wrap-your-agent) is the no-key, two-minute version β€” a generic Python loop the governor caps at a loop budget, the deterministic kill with nothing running but the daemon. diff --git a/examples/kill-9-resume/README.md b/examples/kill-9-resume/README.md new file mode 100644 index 0000000..bc1df04 --- /dev/null +++ b/examples/kill-9-resume/README.md @@ -0,0 +1,110 @@ +# kill-9-resume β€” a crashed run resumes without re-spending + +**The flagship.** An agent does expensive, checkpointed work under a governed run. +The RiskKernel daemon is **`kill -9`'d mid-run** β€” a hard crash, no graceful +shutdown. On restart the daemon reloads the run with the budget and usage it had +already spent; re-running the agent attaches to that same run (`resume_run`), reads +its last checkpoint, and finishes the work β€” **without redoing (re-paying for) the +steps already done.** + +The proof is one number: across the crash the governor's loop counter ends at +**10** (one per step of work), not 15 β€” it never restarts from zero, and never +re-spends. + +## Run it (one command) + +`demo.sh` scripts the whole thing reproducibly: start daemon β†’ 5 of 10 steps β†’ +`kill -9` the daemon β†’ restart β†’ resume β†’ finish β†’ show the proof. + +```bash +cd examples/kill-9-resume +pip install -r requirements.txt # the RiskKernel SDK (stdlib-only) + +# needs the daemon binary + a python with the SDK: +RISKKERNEL_BIN=../../riskkernel ./demo.sh +# (or, with the CLI on your PATH: ./demo.sh) +``` + +## What you'll see + +``` +── 2. agent does 5 of 10 steps, checkpointing each ──────────────────── +β–Ά FRESH run 4e25ad71-… (budget: loops=50) + step 1/10 done (checkpointed cursor=1) + … + step 5/10 done (checkpointed cursor=5) +⏸ did 5 steps, then stopping (the demo crashes the daemon here). + +── 3. kill -9 the daemon (a hard crash β€” no graceful shutdown) ─────── + daemon killed. restarting it… + βœ“ msg="resumed runs from store" count=1 + +── 4. re-run the agent: it RESUMES and finishes ─────────────────────── +↻ RESUMING run 4e25ad71-… + the governor already counts 5 spent steps β€” resuming at cursor 5, not redoing them. + step 6/10 done (checkpointed cursor=6) + … + step 10/10 done (checkpointed cursor=10) +βœ… completed all 10 steps. governor loop counter = 10 β€” exactly one per step of + work. The steps finished before the crash were neither redone nor re-paid. + +── 5. proof β€” the run did 10 steps total across the crash, not 15 ───── + … kill-9-resume running … LOOPS=10 … +``` + +## Do it by hand (for a live recording) + +```bash +# terminal 1 β€” the daemon +riskkernel serve + +# terminal 2 β€” the agent. it checkpoints each step. +python agent.py +# …step 1…2…3… ← now CRASH the daemon: in terminal 1, Ctrl-C twice or `kill -9 ` +# the agent prints: "πŸ’₯ the daemon is gone … restart and re-run to resume" + +# terminal 1 β€” restart it; it logs "resumed runs from store count=1" +riskkernel serve + +# terminal 2 β€” re-run the SAME script. it resumes from the last checkpoint: +python agent.py +# "↻ RESUMING … not redoing them" β†’ finishes β†’ loop counter = number of steps, not double +``` + +## How it works + +Three pieces, all already in the runtime β€” the example just wires them: + +1. **Checkpoint each step.** `run.checkpoint("progress", {"cursor": i+1})` durably + saves *where you are* in the run's SQLite state (the daemon also snapshots + cumulative usage after every step). +2. **Reload on restart.** The daemon reloads non-terminal runs on boot, + reconstructing each governor with the budget and usage it had already spent β€” + so enforcement continues; a SIGKILL can't reset the meter. +3. **Attach and continue.** `with rt.resume_run(run_id) as run:` re-attaches to the + run (it neither creates a new one nor cancels it); `run.latest_checkpoint()` + gives the cursor to resume from. The remaining steps run against the **same + budget**. + +## The "$ not double-counting" headline + +This demo counts **loops** so it needs no API key. The **dollar** counter behaves +identically: `Reload` restores spent dollars and tokens too, so a run that had +burned `$4.20` of a `$5` budget resumes at `$4.20`, not `$0` β€” it can't get a fresh +budget by crashing. To see it on real spend, route your model through the run's +proxy (`run.proxy_config()`, as in [`examples/codebase-qa`](../codebase-qa)) and +watch `riskkernel audit export ` before and after the crash. + +## A note on the crash instant + +If the daemon dies *in the middle of a step* (after the loop is counted but before +that step's checkpoint), resume re-attempts exactly that one step β€” at most one +step of work is repeated, never the whole run. Tightening that boundary (partial +tool calls, mid-stream responses) is ongoing hardening; the budget is always +restored exactly. + +## Tuning for a recording + +- `RK_TOTAL_STEPS` (default 10), `RK_WORK_SECONDS` (default 0.4) β€” total work and + per-step delay. `RK_STOP_AFTER` makes the agent stop cleanly after N steps (the + orchestrator uses it to crash at a known point). diff --git a/examples/kill-9-resume/agent.py b/examples/kill-9-resume/agent.py new file mode 100644 index 0000000..77f749d --- /dev/null +++ b/examples/kill-9-resume/agent.py @@ -0,0 +1,101 @@ +#!/usr/bin/env python3 +"""kill-9-resume β€” a crashed run resumes without re-spending. THE flagship. + +An agent does expensive, checkpointed work under a governed run. The RiskKernel +daemon is `kill -9`'d mid-run β€” a hard crash, no graceful shutdown. On restart the +daemon reloads the run with the budget and usage it had already spent; re-running +this agent attaches to that same run (``resume_run``), reads its last checkpoint, +and finishes the work β€” WITHOUT redoing (re-paying for) the steps already done. + +The proof: across the crash the governor's loop counter ends at exactly the number +of steps of *work* (10), not double (it never restarts from zero). + +Run it twice with a daemon crash in between β€” or just run ``./demo.sh``, which +scripts the whole thing. This agent owns one piece of state: a file holding the +run id to resume (``$RK_RUN_ID_FILE``, default ``.resume-run-id``). +""" + +from __future__ import annotations + +import os +import time + +import riskkernel as rk + +DAEMON_URL = os.environ.get("RISKKERNEL_BASE_URL", "http://localhost:7070") +TOTAL_STEPS = int(os.environ.get("RK_TOTAL_STEPS", "10")) +WORK_SECONDS = float(os.environ.get("RK_WORK_SECONDS", "0.4")) +RUN_ID_FILE = os.environ.get("RK_RUN_ID_FILE", ".resume-run-id") +# Demo knob: stop cleanly after this many total steps (so the orchestrator can +# crash the daemon at a known point). 0 = run to completion. +STOP_AFTER = int(os.environ.get("RK_STOP_AFTER", "0")) or TOTAL_STEPS + + +def do_expensive_work(step: int) -> None: + """Stand-in for the per-step work you don't want to pay for twice β€” a model + call, a tool invocation, a long computation. Here it just sleeps.""" + time.sleep(WORK_SECONDS) + + +def run_loop(run: "rk.Run", start: int) -> int: + """Do steps [start, TOTAL_STEPS), checkpointing each. Returns the next cursor.""" + i = start + while i < TOTAL_STEPS: + if i >= STOP_AFTER: + print(f"\n⏸ did {i} steps, then stopping (the demo crashes the daemon here).") + return i + run.step() # one governed step (counts vs the budget) + do_expensive_work(i) + run.checkpoint("progress", {"cursor": i + 1}) # save WHERE we are, durably + print(f" step {i + 1:>2}/{TOTAL_STEPS} done (checkpointed cursor={i + 1})") + i += 1 + return i + + +def _finish(run: "rk.Run") -> None: + loops = run.status().get("usage", {}).get("loops") + if os.path.exists(RUN_ID_FILE): + os.remove(RUN_ID_FILE) + print(f"\nβœ… completed all {TOTAL_STEPS} steps. governor loop counter = {loops} " + f"β€” exactly one per step of work. The steps finished before the crash were " + f"neither redone nor re-paid.") + + +def main() -> int: + rt = rk.Runtime(base_url=DAEMON_URL) + try: + if os.path.exists(RUN_ID_FILE): + # ── RESUME path: attach to the existing run after the crash ── + run_id = open(RUN_ID_FILE).read().strip() + with rt.resume_run(run_id) as run: + cp = run.latest_checkpoint() + start = int(cp["payload"]["cursor"]) if cp and cp.get("payload") else 0 + spent = run.status().get("usage", {}).get("loops", start) + print(f"↻ RESUMING run {run_id}\n" + f" the governor already counts {spent} spent steps β€” resuming at " + f"cursor {start}, not redoing them.\n") + end = run_loop(run, start) + if end >= TOTAL_STEPS: + _finish(run) + else: + # ── FRESH path: open a new governed run ── + budget = rt.budget(loops=TOTAL_STEPS * 5, seconds=3600) # generous; the demo is about resume, not halt + with rt.governed_run(name="kill-9-resume", budget=budget) as run: + open(RUN_ID_FILE, "w").write(run.id) + print(f"β–Ά FRESH run {run.id} (budget: loops={TOTAL_STEPS * 5})\n") + end = run_loop(run, 0) + if end >= TOTAL_STEPS: + _finish(run) + return 0 + except rk.APIError as e: + if e.code == "connection_error": + print("\nπŸ’₯ the daemon is gone (kill -9?). Your progress is safe β€” it's " + "checkpointed in the run's SQLite state.\n" + " Restart the daemon and re-run this script: it RESUMES from the last " + "checkpoint, no work re-paid.") + return 1 + raise + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/examples/kill-9-resume/demo.sh b/examples/kill-9-resume/demo.sh new file mode 100755 index 0000000..675e782 --- /dev/null +++ b/examples/kill-9-resume/demo.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# kill-9-resume β€” scripts the flagship demo end to end, reproducibly: +# start daemon β†’ agent does 5/10 steps β†’ kill -9 the daemon β†’ restart β†’ +# agent RESUMES and finishes β†’ prove the loop counter is 10, not 15. +# +# Prereqs: the `riskkernel` binary and a Python with the SDK installed. +# RISKKERNEL_BIN path to the daemon binary (default: riskkernel on PATH) +# PYTHON python with the SDK (default: python3) +# e.g. RISKKERNEL_BIN=../../riskkernel PYTHON=.venv/bin/python ./demo.sh +set -uo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" +BIN="${RISKKERNEL_BIN:-riskkernel}" +PY="${PYTHON:-python3}" +PORT="${RK_PORT:-7070}" +DATA="$(mktemp -d)" +export RISKKERNEL_BASE_URL="http://localhost:${PORT}" +export RK_RUN_ID_FILE="${DATA}/run-id" +export RK_PORT="${PORT}" + +cleanup() { [[ -n "${PID:-}" ]] && kill "${PID}" 2>/dev/null; rm -rf "${DATA}"; } +trap cleanup EXIT + +start_daemon() { + RISKKERNEL_DATA_DIR="${DATA}" RISKKERNEL_PORT="${PORT}" "${BIN}" serve >>"${DATA}/serve.log" 2>&1 & + PID=$! + for _ in $(seq 1 40); do + curl -sf "http://localhost:${PORT}/healthz" >/dev/null 2>&1 && return 0 + sleep 0.25 + done + echo "βœ— daemon didn't come up β€” see ${DATA}/serve.log" >&2; exit 1 +} + +echo "── 1. start the daemon ────────────────────────────────────────────────" +start_daemon +echo " up (pid ${PID})" + +echo +echo "── 2. agent does 5 of 10 steps, checkpointing each ────────────────────" +RK_STOP_AFTER=5 "${PY}" "${HERE}/agent.py" + +echo +echo "── 3. kill -9 the daemon (a hard crash β€” no graceful shutdown) ───────" +kill -9 "${PID}"; sleep 0.5 +echo " daemon killed. restarting it…" +start_daemon +grep -i "resumed runs" "${DATA}/serve.log" | tail -1 | sed 's/^/ βœ“ /' || true + +echo +echo "── 4. re-run the agent: it RESUMES and finishes ───────────────────────" +"${PY}" "${HERE}/agent.py" + +echo +echo "── 5. proof β€” the run did 10 steps total across the crash, not 15 ─────" +RISKKERNEL_DATA_DIR="${DATA}" "${BIN}" runs list 2>/dev/null | grep -v "state store ready" || true +echo +echo " loops = 10 (one per step of work). Re-running from zero would read 15." diff --git a/examples/kill-9-resume/requirements.txt b/examples/kill-9-resume/requirements.txt new file mode 100644 index 0000000..747087e --- /dev/null +++ b/examples/kill-9-resume/requirements.txt @@ -0,0 +1,6 @@ +# The only dependency is the RiskKernel Python SDK (stdlib-only). It isn't on PyPI +# yet, so it's installed from source. resume_run() needs SDK >= 0.2.0. +riskkernel @ git+https://github.com/prashar32/riskkernel.git#subdirectory=sdks/python + +# Working inside a clone and want your local SDK instead? From THIS directory: +# pip install ../../sdks/python