Skip to content

Add a timed kill -9 recovery benchmark (exact-once spend) - #137

Merged
prashar32 merged 2 commits into
mainfrom
feat/benchmark-recovery
Jun 14, 2026
Merged

Add a timed kill -9 recovery benchmark (exact-once spend)#137
prashar32 merged 2 commits into
mainfrom
feat/benchmark-recovery

Conversation

@prashar32

Copy link
Copy Markdown
Owner

The cost benchmark measures one dimension — dollars saved when a runaway
loop is capped. This adds the other: crash recovery, with a number.

benchmark/recovery.py runs a governed, checkpointing run through RiskKernel,
hard kill -9s the daemon mid-run (no graceful shutdown), restarts it on the
same durable data dir, and:

  • times recovery — from the kill to the daemon being healthy and the run
    reloaded with its prior spend (the resumed runs from store startup line is the
    daemon confirming it restored from the store);
  • proves exact-once spend — the meter after the restart equals the meter
    before the crash (no reset, no double-count), and the run still halts at the
    original budget. A reset would have re-paid the pre-crash calls; the harness
    prints that counterfactual.

It mirrors the cost benchmark exactly: key-free and deterministic (the same mock
provider, a dummy key, spend read straight from RiskKernel’s own ledger via
GET /v1/runs/{id}), waits for /healthz before driving calls, and cleans up
its processes and temp data dir on exit. Reproducible on a clean machine with no
API key.

results.json gains a recovery key without disturbing the existing cost object,
and the benchmark README gets a "Recovery time" section with the methodology.

Sample run on a laptop:

  RECOVERY TIME              56 ms   (kill -9 -> healthy + run reloaded)
  spend (ledger)                 dollars   tokens  loops
  before crash                   $0.1000    16000      8
  after restart (reloaded)       $0.1000    16000      8
  final (budget halt)            $0.2500    40000     20
  EXACT-ONCE across crash    PASS

Closes #98

Add a recovery-time benchmark alongside the cost benchmark. It interrupts a
governed, checkpointing run with a hard kill -9 of the daemon mid-run, restarts
on the same durable data dir, and times how long until the daemon is healthy with
the run reloaded. It then proves the cost meter continues from where it was — no
reset, no double-count — and still halts at the original budget, i.e. exact-once
spend across the crash.

Key-free and deterministic like the cost benchmark: the same mock provider, a
dummy key, spend read from RiskKernel's own ledger (GET /v1/runs/{id}). Numbers
are merged into results.json under a recovery key without touching the existing
cost object; the benchmark README gains a Recovery time section.
@prashar32
prashar32 merged commit bedeaaa into main Jun 14, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Benchmark: add the recovery-time dimension (timed kill -9 -> resume)

1 participant