Skip to content

Loop/time-budget halts (and normal completion) aren't persisted — run status stays "running" #34

Description

@prashar32

Summary

A run that the governor halts on its loop or time budget keeps status: "running" and an empty haltReason in the store, even though it is terminally halted in memory (context cancelled, further calls correctly refused with 402). GET /v1/runs/{id}, riskkernel runs list, and the SDK all then report a halted run as still running.

Reproduce

Run examples/codebase-qa in --mode runaway (loop budget = 4), then:

$ riskkernel runs list
ID         NAME                 STATUS   ... LOOPS  HALT
5b9a...    codebase-qa-runaway  running  ... 4      -      # <- should be halted / loop_budget_exceeded

The proxy correctly returns 402 loop_budget_exceeded for the 5th call — only the persisted status is wrong.

Cause

GET /v1/runs/{id} reads from the store (runViewBodystore.GetRun). Token/dollar halts persist because they happen in RecordCallpersistCall(r, c, err). But loop/time halts happen in Run.BeginStepgovernor.PreStep, which sets r.halted in memory and returns the HaltError before anything is persisted:

// internal/runs/manager.go
func (r *Run) BeginStep() (int32, error) {
    if err := r.gov.PreStep(); err != nil {
        return 0, err            // <- halts the governor, but persists nothing
    }
    ...
}

The gateway's budgetError path returns the 402 without persisting the run either.

Fix sketch

When BeginStep (or the gateway) observes a PreStep halt, persist the run with its halt reason and halted status (mirroring what RecordCall does for token/dollar halts). One option: have BeginStep call r.mgr.persistRun(r) on the halt branch before returning the error.

Related / secondary

A run that completes normally also stays running — there's no "finish run" signal from the SDK/governed_run to the daemon. Worth deciding whether a terminal completed status should exist (e.g. a POST /v1/runs/{id}/complete, or the SDK marking it on clean context exit). Separate from the halt-persistence bug above but the same observable symptom (status never leaves running).

Impact

Observability/audit correctness: runs list and dashboards misreport halted runs as running. The enforcement itself is correct. Good candidate to bundle into the v0.1.1 alongside #32.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions