diff --git a/CHANGELOG.md b/CHANGELOG.md index 9ec1e0c..635e570 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,14 @@ surface is governed by [`COMPATIBILITY.md`](COMPATIBILITY.md). ## [Unreleased] ### Added +- **Troubleshooting guide.** [`docs/TROUBLESHOOTING.md`](docs/TROUBLESHOOTING.md) + maps the errors new users actually hit — each as symptom → cause → fix — to the + exact messages the daemon emits: a missing/invalid provider key, port 7070 already + in use, the **expected** HTTP 402 budget halt (with the `HaltReason` it carries — + not a bug), a killed run that didn't resume, the schema-newer-than-binary + downgrade guard, a 401 from API auth, an unreachable OTLP endpoint, and OTLP + ingress 400s. It leads with `riskkernel doctor` and is linked from the README + quickstart. - **One-command docker-compose quickstart.** [`examples/quickstart-compose`](examples/quickstart-compose) brings up the daemon, a stand-in mock LLM, and a tiny looping agent with a single `docker compose up` — so a newcomer watches the deterministic loop budget hard-stop diff --git a/README.md b/README.md index 01b16ab..8180925 100644 --- a/README.md +++ b/README.md @@ -137,6 +137,10 @@ Governing tools over MCP? [`examples/mcp`](examples/mcp) puts the MCP gateway in front of a stub server and shows a tool blocked by the allowlist, a side-effecting tool held for approval, and the audit trail — key-free. +Hit a snag? `riskkernel doctor` diagnoses most setups, and the +[troubleshooting guide](docs/TROUBLESHOOTING.md) maps the common errors — +missing key, port in use, the expected 402 budget halt — to fixes. + ## Design principles - **Deterministic core in Go.** All enforcement (budgets, kill switches, gating, routing, retries, checkpointing) lives in compiled, statically-typed code — never in an LLM. diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md new file mode 100644 index 0000000..98325cc --- /dev/null +++ b/docs/TROUBLESHOOTING.md @@ -0,0 +1,304 @@ +# Troubleshooting + +The errors a new RiskKernel user actually hits, each as **symptom → cause → +fix**. If your problem isn't here, [open an issue](https://github.com/prashar32/riskkernel/issues) +— a confusing error is a docs bug as much as a code bug. + +> **Run this first.** `riskkernel doctor` is the built-in diagnostics command. It +> checks your data dir, provider credential, default budget, API token, policy +> file, and probes a running daemon — and exits non-zero if anything hard-fails. +> Most of what's below, it catches before you hit it. +> +> ```bash +> riskkernel doctor +> ``` + +--- + +## Provider API key missing + +**Symptom.** A model call (proxy, `riskkernel chat`, or the SDK) fails with: + +``` +anthropic: missing API key +openai: missing API key +``` + +`riskkernel doctor` flags it ahead of time: + +``` +⚠ default provider (anthropic) — ANTHROPIC_API_KEY not set — model calls will fail +``` + +and `riskkernel serve` logs a startup warning: + +``` +ANTHROPIC_API_KEY is not set — model calls will fail until a key is provided +``` + +**Cause.** The default provider needs a key, and none is in the environment. +RiskKernel reads keys from the environment / `.env` only — it never stores them in +state. + +**Fix.** Set the key for your default provider before starting the daemon: + +```bash +export ANTHROPIC_API_KEY=sk-ant-... # or OPENAI_API_KEY for OpenAI +``` + +If you scaffolded with `riskkernel init`, put it in the generated `.env` (the +daemon reads it). Local models via Ollama are key-free — set +`RISKKERNEL_DEFAULT_PROVIDER=ollama` and no key is needed. + +--- + +## Provider API key invalid (or wrong provider) + +**Symptom.** The key is set, but a proxy call returns **HTTP 502** with: + +```json +{ "code": "provider_error", "message": "anthropic: invalid x-api-key (authentication_error, http 401)" } +``` + +The exact message is the upstream provider's, surfaced verbatim in the form +`anthropic: (, http )` (same shape for `openai:`). + +**Cause.** RiskKernel forwarded the call to the real provider with your key, and +the provider rejected it — a bad/expired key, the wrong provider's key, or a +model your account can't access (`http 401` / `403` / `invalid_request_error`). +The governor did its job; the failure is upstream. + +**Fix.** Verify the key works against the provider directly, then confirm the +model id routes to the provider whose key you set — `claude-*` routes to +Anthropic, `gpt-*` / `o1` / `o3` to OpenAI. A `claude-*` model with only +`OPENAI_API_KEY` set will fail this way. `riskkernel chat "hi"` is the fastest +isolated check of the provider path. + +--- + +## Port already in use (7070) + +**Symptom.** `riskkernel serve` exits immediately with: + +``` +listen tcp :7070: bind: address already in use +``` + +**Cause.** Another process — often a RiskKernel daemon you already started — holds +port 7070 (the default daemon port). + +**Fix.** Find and stop the other listener, or run on a different port: + +```bash +lsof -i :7070 # see what's holding it +riskkernel doctor # "daemon (:7070) — responding on /healthz" means one is already up +``` + +To move ports, set `RISKKERNEL_PORT` (and update `OPENAI_BASE_URL` / +`OTEL_EXPORTER_OTLP_ENDPOINT` to match). With Docker, change the host side of the +mapping, e.g. `-p 8070:7070`. + +--- + +## A run was halted — HTTP 402 (this is expected, not a bug) + +**Symptom.** A proxy call returns **HTTP 402** with a body and a header: + +```json +{ "code": "dollar_budget_exceeded", "message": "run halted: dollar_budget_exceeded" } +``` + +``` +X-RiskKernel-Halt-Reason: dollar_budget_exceeded +``` + +`riskkernel runs list` shows the run as `halted`. + +**Cause.** This is RiskKernel **working as designed** — the headline feature, not +an error. The run hit one of its four budgets and the governor stopped it before +the next call. The `code` / `X-RiskKernel-Halt-Reason` is one of: + +| Halt reason | Tripped budget | +|---|---| +| `dollar_budget_exceeded` | `dollars` — cumulative USD from the cost ledger | +| `token_budget_exceeded` | `tokens` — prompt + completion across the run | +| `loop_budget_exceeded` | `loops` — agent loop iterations / steps | +| `time_budget_exceeded` | `seconds` — wall-clock duration | + +Unconfigured, every run gets a **safe default budget** ($5 / 100 loops / 1 hour), +so a 402 on a brand-new setup usually means the safe default did its job on a +runaway loop. See [the budget contract](BUDGETS.md). + +**Fix.** Decide whether the halt was correct. If it was a runaway loop, it just +saved you money — nothing to fix. If the limit was genuinely too low, **start a +new run with a bigger budget** (raise `RISKKERNEL_DEFAULT_DOLLARS` / +`_LOOPS` / `_TOKENS` / `_SECONDS`, set a per-run `budget`, or use a policy +bundle). A budget-halted run stays halted by design — it does not resume; that's +crash-resume's job (next entry), not the kill switch's. + +--- + +## A killed run didn't resume / re-spent + +**Symptom.** You `kill -9`'d the daemon mid-run and expected it to pick up where it +left off, but it looks like it started over (or didn't resume at all). + +**Cause.** Resume is automatic on the next `riskkernel serve` — the daemon reloads +mid-flight runs from the durable store on startup. If you don't see it, +the run had no checkpoint yet, or the state file isn't the same one +(`RISKKERNEL_DATA_DIR` / the mounted volume differs between runs). + +**Fix.** Restart with the **same** data dir / volume. On startup the daemon logs: + +``` +resumed runs from store count=1 +``` + +A resumed run enforces against what it had **already** spent — if it had burned +$4.20 of a $5 budget, it resumes at $4.20, never $0, so it can't overspend by +restarting. Re-attaching from the SDK is `rt.resume_run(run_id)`; over the API +it's `GET /v1/runs/{id}` + `GET /v1/checkpoints/{id}` reusing the same id. See +[the crash-resume guide](RESUME.md). + +--- + +## Daemon refuses to start: schema is newer than the binary + +**Symptom.** `riskkernel serve` exits at startup with: + +``` +storage: on-disk schema is newer than this binary; upgrade riskkernel (on-disk v5 > binary v4) +``` + +**Cause.** **Downgrade protection.** Your state file (SQLite or Postgres) was +written by a *newer* RiskKernel than the binary you're now running. Rather than +risk corrupting your data with an older binary, the daemon refuses to start. This +is intentional — see [`COMPATIBILITY.md`](../COMPATIBILITY.md). + +**Fix.** Run a RiskKernel **at least as new** as the one that last touched the +state. Migrations are forward-only and run automatically on startup, so upgrading +is the supported path; there is no downgrade migration. If you genuinely need the +old binary, point it at a **fresh** data dir (`RISKKERNEL_DATA_DIR`) — but you'll +lose the existing runs/ledger/checkpoints. + +--- + +## API returns 401 Unauthorized + +**Symptom.** Requests to the daemon's API/proxy return **401** with: + +```json +{ "code": "unauthorized", "message": "missing or invalid bearer token" } +``` + +even though the daemon is up and the provider key is fine. + +**Cause.** You set `RISKKERNEL_API_TOKEN`, which turns on single-tenant bearer +auth on the API — and the request didn't send the token (or sent the wrong one). + +**Fix.** Send the bearer token on every request: + +```bash +curl -H "Authorization: Bearer $RISKKERNEL_API_TOKEN" http://localhost:7070/v1/... +``` + +The Python SDK reads `RISKKERNEL_API_TOKEN` from the environment, or takes it as +`Runtime(token=...)`. If you *didn't* mean to require auth, the daemon warns at +startup when it's unset — + +``` +RISKKERNEL_API_TOKEN is not set — the API is unauthenticated; do not expose this port to an untrusted network +``` + +— which is the opposite case: fine on localhost, dangerous on an exposed port. +Either set a token, or keep the port local. + +--- + +## OTLP endpoint unreachable / no spans showing up + +**Symptom.** You pointed RiskKernel at an OTLP backend but no spans arrive in +Grafana/Tempo/SigNoz/Honeycomb/Datadog, or the daemon logs export errors. + +**Cause.** OTel export is **off until you configure an endpoint** — RiskKernel +never emits telemetry on its own. If it's on, the usual culprits are the wrong +endpoint URL, the wrong protocol (`grpc` vs `http`), or the collector not +listening. + +**Fix.** Set the endpoint (and protocol if not gRPC) before starting the daemon: + +```bash +export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 +export OTEL_EXPORTER_OTLP_PROTOCOL=grpc # default; use "http" for OTLP/HTTP +``` + +When export is enabled the daemon logs it at startup: + +``` +otel export enabled endpoint=http://localhost:4317 protocol=grpc +``` + +If that line is missing, the endpoint env var isn't set in the daemon's +environment. RiskKernel reads `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` first, then +`OTEL_EXPORTER_OTLP_ENDPOINT`. See [`METRICS.md`](METRICS.md) for the local +`/metrics` scrape, which needs no exporter at all. + +--- + +## OTLP ingress returns 400 Bad Request + +**Symptom.** An app exporting traces *into* RiskKernel (Surface 3 ingress) gets: + +``` +invalid OTLP protobuf: ... +invalid OTLP JSON: ... +``` + +**Cause.** The body sent to `POST /v1/traces` wasn't valid OTLP in the declared +encoding (a content-type / payload mismatch — RiskKernel accepts +`application/x-protobuf`, the OTLP default, and JSON). Note the ingress receiver +is **off** unless `RISKKERNEL_OTEL_INGRESS_ENABLED` is truthy. + +**Fix.** Enable ingress and make sure your exporter targets the RiskKernel +endpoint with a matching content-type: + +```bash +export RISKKERNEL_OTEL_INGRESS_ENABLED=1 +# in the producing app: +export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:7070 +``` + +See [`OTLP_INGRESS.md`](OTLP_INGRESS.md) for the spans RiskKernel meters and how +runs are attributed. + +--- + +## Unknown provider / model not routing + +**Symptom.** A proxy call returns **HTTP 400**: + +```json +{ "code": "unknown_provider", "message": "..." } +``` + +**Cause.** The `model` in the request didn't route to a configured provider. +RiskKernel routes by model-id prefix — `claude-*` → Anthropic, `gpt-*` / `o1` / +`o3` → OpenAI — and anything else falls through to the default provider, which may +not be registered. + +**Fix.** Use a model id that routes to a provider you've configured a key for, or +set `RISKKERNEL_DEFAULT_PROVIDER` to the one you want unprefixed models to use. +For providers RiskKernel doesn't implement natively, front them with LiteLLM as an +upstream forwarder. + +--- + +## Where to look next + +- [`BUDGETS.md`](BUDGETS.md) — the budget contract: dimensions, precedence, safe + defaults, and exact halt semantics. +- [`RESUME.md`](RESUME.md) — crash-resume mechanics and what's restored. +- [`METRICS.md`](METRICS.md) / [`OTLP_INGRESS.md`](OTLP_INGRESS.md) — observability. +- [`POSTGRES.md`](POSTGRES.md) — the opt-in Postgres backend. +- [`COMPATIBILITY.md`](../COMPATIBILITY.md) — what's stable across versions and the + downgrade-protection policy.