From 05706926bd0e0ccb79146d321dcb1422917fd55a Mon Sep 17 00:00:00 2001 From: Adarsh Prashar Date: Mon, 15 Jun 2026 00:46:39 +0530 Subject: [PATCH] docs: add a troubleshooting guide for common setup errors New users were hitting the same handful of errors and had nowhere to look: a missing/invalid provider key, port 7070 already in use, the expected 402 budget halt (which reads like a failure but is the kill switch working), the schema-newer-than-binary downgrade guard, API-auth 401s, and OTLP export/ingress misconfiguration. Add docs/TROUBLESHOOTING.md as a skimmable symptom -> cause -> fix guide. Every entry quotes the exact message the daemon emits (grepped from the code, not invented) so users can match what they see, and it leads with 'riskkernel doctor' since that catches most of these before they bite. Link it from the README quickstart. --- CHANGELOG.md | 8 ++ README.md | 4 + docs/TROUBLESHOOTING.md | 304 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 316 insertions(+) create mode 100644 docs/TROUBLESHOOTING.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 9ec1e0c..635e570 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,14 @@ surface is governed by [`COMPATIBILITY.md`](COMPATIBILITY.md). ## [Unreleased] ### Added +- **Troubleshooting guide.** [`docs/TROUBLESHOOTING.md`](docs/TROUBLESHOOTING.md) + maps the errors new users actually hit — each as symptom → cause → fix — to the + exact messages the daemon emits: a missing/invalid provider key, port 7070 already + in use, the **expected** HTTP 402 budget halt (with the `HaltReason` it carries — + not a bug), a killed run that didn't resume, the schema-newer-than-binary + downgrade guard, a 401 from API auth, an unreachable OTLP endpoint, and OTLP + ingress 400s. It leads with `riskkernel doctor` and is linked from the README + quickstart. - **One-command docker-compose quickstart.** [`examples/quickstart-compose`](examples/quickstart-compose) brings up the daemon, a stand-in mock LLM, and a tiny looping agent with a single `docker compose up` — so a newcomer watches the deterministic loop budget hard-stop diff --git a/README.md b/README.md index 01b16ab..8180925 100644 --- a/README.md +++ b/README.md @@ -137,6 +137,10 @@ Governing tools over MCP? [`examples/mcp`](examples/mcp) puts the MCP gateway in front of a stub server and shows a tool blocked by the allowlist, a side-effecting tool held for approval, and the audit trail — key-free. +Hit a snag? `riskkernel doctor` diagnoses most setups, and the +[troubleshooting guide](docs/TROUBLESHOOTING.md) maps the common errors — +missing key, port in use, the expected 402 budget halt — to fixes. + ## Design principles - **Deterministic core in Go.** All enforcement (budgets, kill switches, gating, routing, retries, checkpointing) lives in compiled, statically-typed code — never in an LLM. diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md new file mode 100644 index 0000000..98325cc --- /dev/null +++ b/docs/TROUBLESHOOTING.md @@ -0,0 +1,304 @@ +# Troubleshooting + +The errors a new RiskKernel user actually hits, each as **symptom → cause → +fix**. If your problem isn't here, [open an issue](https://github.com/prashar32/riskkernel/issues) +— a confusing error is a docs bug as much as a code bug. + +> **Run this first.** `riskkernel doctor` is the built-in diagnostics command. It +> checks your data dir, provider credential, default budget, API token, policy +> file, and probes a running daemon — and exits non-zero if anything hard-fails. +> Most of what's below, it catches before you hit it. +> +> ```bash +> riskkernel doctor +> ``` + +--- + +## Provider API key missing + +**Symptom.** A model call (proxy, `riskkernel chat`, or the SDK) fails with: + +``` +anthropic: missing API key +openai: missing API key +``` + +`riskkernel doctor` flags it ahead of time: + +``` +⚠ default provider (anthropic) — ANTHROPIC_API_KEY not set — model calls will fail +``` + +and `riskkernel serve` logs a startup warning: + +``` +ANTHROPIC_API_KEY is not set — model calls will fail until a key is provided +``` + +**Cause.** The default provider needs a key, and none is in the environment. +RiskKernel reads keys from the environment / `.env` only — it never stores them in +state. + +**Fix.** Set the key for your default provider before starting the daemon: + +```bash +export ANTHROPIC_API_KEY=sk-ant-... # or OPENAI_API_KEY for OpenAI +``` + +If you scaffolded with `riskkernel init`, put it in the generated `.env` (the +daemon reads it). Local models via Ollama are key-free — set +`RISKKERNEL_DEFAULT_PROVIDER=ollama` and no key is needed. + +--- + +## Provider API key invalid (or wrong provider) + +**Symptom.** The key is set, but a proxy call returns **HTTP 502** with: + +```json +{ "code": "provider_error", "message": "anthropic: invalid x-api-key (authentication_error, http 401)" } +``` + +The exact message is the upstream provider's, surfaced verbatim in the form +`anthropic: (, http )` (same shape for `openai:`). + +**Cause.** RiskKernel forwarded the call to the real provider with your key, and +the provider rejected it — a bad/expired key, the wrong provider's key, or a +model your account can't access (`http 401` / `403` / `invalid_request_error`). +The governor did its job; the failure is upstream. + +**Fix.** Verify the key works against the provider directly, then confirm the +model id routes to the provider whose key you set — `claude-*` routes to +Anthropic, `gpt-*` / `o1` / `o3` to OpenAI. A `claude-*` model with only +`OPENAI_API_KEY` set will fail this way. `riskkernel chat "hi"` is the fastest +isolated check of the provider path. + +--- + +## Port already in use (7070) + +**Symptom.** `riskkernel serve` exits immediately with: + +``` +listen tcp :7070: bind: address already in use +``` + +**Cause.** Another process — often a RiskKernel daemon you already started — holds +port 7070 (the default daemon port). + +**Fix.** Find and stop the other listener, or run on a different port: + +```bash +lsof -i :7070 # see what's holding it +riskkernel doctor # "daemon (:7070) — responding on /healthz" means one is already up +``` + +To move ports, set `RISKKERNEL_PORT` (and update `OPENAI_BASE_URL` / +`OTEL_EXPORTER_OTLP_ENDPOINT` to match). With Docker, change the host side of the +mapping, e.g. `-p 8070:7070`. + +--- + +## A run was halted — HTTP 402 (this is expected, not a bug) + +**Symptom.** A proxy call returns **HTTP 402** with a body and a header: + +```json +{ "code": "dollar_budget_exceeded", "message": "run halted: dollar_budget_exceeded" } +``` + +``` +X-RiskKernel-Halt-Reason: dollar_budget_exceeded +``` + +`riskkernel runs list` shows the run as `halted`. + +**Cause.** This is RiskKernel **working as designed** — the headline feature, not +an error. The run hit one of its four budgets and the governor stopped it before +the next call. The `code` / `X-RiskKernel-Halt-Reason` is one of: + +| Halt reason | Tripped budget | +|---|---| +| `dollar_budget_exceeded` | `dollars` — cumulative USD from the cost ledger | +| `token_budget_exceeded` | `tokens` — prompt + completion across the run | +| `loop_budget_exceeded` | `loops` — agent loop iterations / steps | +| `time_budget_exceeded` | `seconds` — wall-clock duration | + +Unconfigured, every run gets a **safe default budget** ($5 / 100 loops / 1 hour), +so a 402 on a brand-new setup usually means the safe default did its job on a +runaway loop. See [the budget contract](BUDGETS.md). + +**Fix.** Decide whether the halt was correct. If it was a runaway loop, it just +saved you money — nothing to fix. If the limit was genuinely too low, **start a +new run with a bigger budget** (raise `RISKKERNEL_DEFAULT_DOLLARS` / +`_LOOPS` / `_TOKENS` / `_SECONDS`, set a per-run `budget`, or use a policy +bundle). A budget-halted run stays halted by design — it does not resume; that's +crash-resume's job (next entry), not the kill switch's. + +--- + +## A killed run didn't resume / re-spent + +**Symptom.** You `kill -9`'d the daemon mid-run and expected it to pick up where it +left off, but it looks like it started over (or didn't resume at all). + +**Cause.** Resume is automatic on the next `riskkernel serve` — the daemon reloads +mid-flight runs from the durable store on startup. If you don't see it, +the run had no checkpoint yet, or the state file isn't the same one +(`RISKKERNEL_DATA_DIR` / the mounted volume differs between runs). + +**Fix.** Restart with the **same** data dir / volume. On startup the daemon logs: + +``` +resumed runs from store count=1 +``` + +A resumed run enforces against what it had **already** spent — if it had burned +$4.20 of a $5 budget, it resumes at $4.20, never $0, so it can't overspend by +restarting. Re-attaching from the SDK is `rt.resume_run(run_id)`; over the API +it's `GET /v1/runs/{id}` + `GET /v1/checkpoints/{id}` reusing the same id. See +[the crash-resume guide](RESUME.md). + +--- + +## Daemon refuses to start: schema is newer than the binary + +**Symptom.** `riskkernel serve` exits at startup with: + +``` +storage: on-disk schema is newer than this binary; upgrade riskkernel (on-disk v5 > binary v4) +``` + +**Cause.** **Downgrade protection.** Your state file (SQLite or Postgres) was +written by a *newer* RiskKernel than the binary you're now running. Rather than +risk corrupting your data with an older binary, the daemon refuses to start. This +is intentional — see [`COMPATIBILITY.md`](../COMPATIBILITY.md). + +**Fix.** Run a RiskKernel **at least as new** as the one that last touched the +state. Migrations are forward-only and run automatically on startup, so upgrading +is the supported path; there is no downgrade migration. If you genuinely need the +old binary, point it at a **fresh** data dir (`RISKKERNEL_DATA_DIR`) — but you'll +lose the existing runs/ledger/checkpoints. + +--- + +## API returns 401 Unauthorized + +**Symptom.** Requests to the daemon's API/proxy return **401** with: + +```json +{ "code": "unauthorized", "message": "missing or invalid bearer token" } +``` + +even though the daemon is up and the provider key is fine. + +**Cause.** You set `RISKKERNEL_API_TOKEN`, which turns on single-tenant bearer +auth on the API — and the request didn't send the token (or sent the wrong one). + +**Fix.** Send the bearer token on every request: + +```bash +curl -H "Authorization: Bearer $RISKKERNEL_API_TOKEN" http://localhost:7070/v1/... +``` + +The Python SDK reads `RISKKERNEL_API_TOKEN` from the environment, or takes it as +`Runtime(token=...)`. If you *didn't* mean to require auth, the daemon warns at +startup when it's unset — + +``` +RISKKERNEL_API_TOKEN is not set — the API is unauthenticated; do not expose this port to an untrusted network +``` + +— which is the opposite case: fine on localhost, dangerous on an exposed port. +Either set a token, or keep the port local. + +--- + +## OTLP endpoint unreachable / no spans showing up + +**Symptom.** You pointed RiskKernel at an OTLP backend but no spans arrive in +Grafana/Tempo/SigNoz/Honeycomb/Datadog, or the daemon logs export errors. + +**Cause.** OTel export is **off until you configure an endpoint** — RiskKernel +never emits telemetry on its own. If it's on, the usual culprits are the wrong +endpoint URL, the wrong protocol (`grpc` vs `http`), or the collector not +listening. + +**Fix.** Set the endpoint (and protocol if not gRPC) before starting the daemon: + +```bash +export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 +export OTEL_EXPORTER_OTLP_PROTOCOL=grpc # default; use "http" for OTLP/HTTP +``` + +When export is enabled the daemon logs it at startup: + +``` +otel export enabled endpoint=http://localhost:4317 protocol=grpc +``` + +If that line is missing, the endpoint env var isn't set in the daemon's +environment. RiskKernel reads `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` first, then +`OTEL_EXPORTER_OTLP_ENDPOINT`. See [`METRICS.md`](METRICS.md) for the local +`/metrics` scrape, which needs no exporter at all. + +--- + +## OTLP ingress returns 400 Bad Request + +**Symptom.** An app exporting traces *into* RiskKernel (Surface 3 ingress) gets: + +``` +invalid OTLP protobuf: ... +invalid OTLP JSON: ... +``` + +**Cause.** The body sent to `POST /v1/traces` wasn't valid OTLP in the declared +encoding (a content-type / payload mismatch — RiskKernel accepts +`application/x-protobuf`, the OTLP default, and JSON). Note the ingress receiver +is **off** unless `RISKKERNEL_OTEL_INGRESS_ENABLED` is truthy. + +**Fix.** Enable ingress and make sure your exporter targets the RiskKernel +endpoint with a matching content-type: + +```bash +export RISKKERNEL_OTEL_INGRESS_ENABLED=1 +# in the producing app: +export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:7070 +``` + +See [`OTLP_INGRESS.md`](OTLP_INGRESS.md) for the spans RiskKernel meters and how +runs are attributed. + +--- + +## Unknown provider / model not routing + +**Symptom.** A proxy call returns **HTTP 400**: + +```json +{ "code": "unknown_provider", "message": "..." } +``` + +**Cause.** The `model` in the request didn't route to a configured provider. +RiskKernel routes by model-id prefix — `claude-*` → Anthropic, `gpt-*` / `o1` / +`o3` → OpenAI — and anything else falls through to the default provider, which may +not be registered. + +**Fix.** Use a model id that routes to a provider you've configured a key for, or +set `RISKKERNEL_DEFAULT_PROVIDER` to the one you want unprefixed models to use. +For providers RiskKernel doesn't implement natively, front them with LiteLLM as an +upstream forwarder. + +--- + +## Where to look next + +- [`BUDGETS.md`](BUDGETS.md) — the budget contract: dimensions, precedence, safe + defaults, and exact halt semantics. +- [`RESUME.md`](RESUME.md) — crash-resume mechanics and what's restored. +- [`METRICS.md`](METRICS.md) / [`OTLP_INGRESS.md`](OTLP_INGRESS.md) — observability. +- [`POSTGRES.md`](POSTGRES.md) — the opt-in Postgres backend. +- [`COMPATIBILITY.md`](../COMPATIBILITY.md) — what's stable across versions and the + downgrade-protection policy.