From cc4af026e374614efa6d0aa368d954d457c9d904 Mon Sep 17 00:00:00 2001 From: Adarsh Prashar Date: Tue, 16 Jun 2026 01:10:25 +0530 Subject: [PATCH] docs: front the long tail of providers with LiteLLM upstream Document calling the 100+ providers not implemented natively (Gemini, Cohere, Mistral, Bedrock, ...) by putting a LiteLLM OpenAI-compatible proxy upstream of RiskKernel via the existing RISKKERNEL_OPENAI_BASE_URL override. RiskKernel governs every call (budgets, approvals, audit, checkpoints, cost metering, OTel); LiteLLM does the provider routing and holds the real keys. Adds docs/PROVIDERS.md with a topology diagram, a copy-pasteable docker-compose, a minimal LiteLLM config.yaml, and a governed-call example. Honest about the model-name routing rule (RISKKERNEL_DEFAULT_PROVIDER=openai for unmatched names) and the cost caveat (long-tail models need a RISKKERNEL_PRICING_FILE rate to count toward the dollar budget). Links it from the README proxy/providers bullet. --- CHANGELOG.md | 11 +++ README.md | 2 +- docs/PROVIDERS.md | 205 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 217 insertions(+), 1 deletion(-) create mode 100644 docs/PROVIDERS.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 7904b26..c6d0f3f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,17 @@ surface is governed by [`COMPATIBILITY.md`](COMPATIBILITY.md). ## [Unreleased] ### Added +- **Guide: front the long tail of providers with LiteLLM upstream.** + [`docs/PROVIDERS.md`](docs/PROVIDERS.md) documents calling the 100+ providers + RiskKernel doesn't implement natively (Gemini, Cohere, Mistral, Bedrock, …) by + putting a LiteLLM OpenAI-compatible proxy *upstream* of RiskKernel via the existing + `RISKKERNEL_OPENAI_BASE_URL` override — RiskKernel governs every call (budgets, + approvals, audit, checkpoints, cost metering, OTel) and LiteLLM does the routing, + with the real provider keys living in LiteLLM. Copy-pasteable docker-compose + a + minimal LiteLLM `config.yaml`, and honest notes on the model-name routing rule + (`RISKKERNEL_DEFAULT_PROVIDER=openai` for unmatched names) and the cost caveat + (long-tail models need a `RISKKERNEL_PRICING_FILE` rate to count toward the dollar + budget). - **Spend rollup across runs (`riskkernel audit summary`).** Per-run `audit export` has a cross-run companion: `riskkernel audit summary --by >` rolls cost-ledger spend up by a dimension — so spend **by team/user/feature** comes diff --git a/README.md b/README.md index 10b1935..d9a61db 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ It is **not** another gateway (LiteLLM/Portkey own routing), **not** another obs ## Three ways to adopt — pick the one that fits -1. **Proxy (zero code).** Set one env var: `OPENAI_BASE_URL=http://localhost:7070/v1` (or `ANTHROPIC_BASE_URL` for `/v1/messages`). Every call — streaming or not — is intercepted, budgeted, logged, checkpointed, and forwarded to the real provider with your key. Native providers: Anthropic, OpenAI, and Ollama (local). +1. **Proxy (zero code).** Set one env var: `OPENAI_BASE_URL=http://localhost:7070/v1` (or `ANTHROPIC_BASE_URL` for `/v1/messages`). Every call — streaming or not — is intercepted, budgeted, logged, checkpointed, and forwarded to the real provider with your key. Native providers: Anthropic, OpenAI, and Ollama (local); front the long tail (Gemini, Cohere, Mistral, Bedrock, …) with [LiteLLM upstream](docs/PROVIDERS.md). 2. **SDK (deep control).** `pip install riskkernel` (Python) or `npm install @riskkernel/sdk` (TypeScript), then governed runs, per-step loop/time budgets, checkpoints, and approval gates. Framework adapters for the Claude Agent SDK, OpenAI Agents SDK, LangChain, LlamaIndex, CrewAI, AutoGen, and PydanticAI (Python), and the Vercel AI SDK (TypeScript). 3. **OpenTelemetry (universal).** RiskKernel is an OTLP endpoint *and* emitter — ingest GenAI spans (`POST /v1/traces`) to meter apps already instrumented with OpenLLMetry / the OpenAI Agents SDK / the Vercel AI SDK, and export cost/halt/tool spans to the backend you already run. diff --git a/docs/PROVIDERS.md b/docs/PROVIDERS.md new file mode 100644 index 0000000..a28767b --- /dev/null +++ b/docs/PROVIDERS.md @@ -0,0 +1,205 @@ +# Providers — native, and the long tail via LiteLLM + +RiskKernel implements the top providers **natively** in Go: Anthropic, OpenAI, and +Ollama (local). For those, point your app at the proxy and you're done — no extra +moving parts. + +The other 100+ providers (Google Gemini, Cohere, Mistral, Groq, Together, Azure +OpenAI, AWS Bedrock, OpenRouter, …) are not reimplemented inside RiskKernel — that +isn't the product. Instead you front them with **[LiteLLM](https://github.com/BerriAI/litellm)**, +which already speaks all of them through one OpenAI-compatible endpoint, and put +RiskKernel **in front of LiteLLM**. RiskKernel governs every call; LiteLLM does the +provider routing. + +``` +your app → RiskKernel → LiteLLM proxy → the real provider + (budgets, approvals, (routing to 100+ (Gemini, Cohere, + audit, checkpoints, providers, keys) Mistral, …) + cost metering, OTel) +``` + +This is the same shape the [`examples/quickstart-compose`](../examples/quickstart-compose) +demo already proves — it points `RISKKERNEL_OPENAI_BASE_URL` at a one-reply mock +upstream. LiteLLM is just the real-world version of that upstream. + +## When to use which + +| You want to call… | Do this | +|---|---| +| Anthropic (`claude-*`) | Native. Set `ANTHROPIC_API_KEY` on the daemon; use a `claude-*` model. | +| OpenAI (`gpt-*`, `o1`, `o3`) | Native. Set `OPENAI_API_KEY` on the daemon; use a `gpt-*`/`o1`/`o3` model. | +| A local Ollama model | Native. Set `RISKKERNEL_OLLAMA_BASE_URL`; use a model the routing sends to Ollama. | +| Anything else (Gemini, Cohere, Mistral, Groq, Bedrock, …) | Front it with **LiteLLM** as described below. | + +You only need LiteLLM for the long tail. If your stack is purely Anthropic/OpenAI/ +Ollama, skip this page. + +## How it works + +RiskKernel's native **OpenAI provider** has a configurable upstream base URL, +`RISKKERNEL_OPENAI_BASE_URL`. The OpenAI provider POSTs to +`/v1/chat/completions`. LiteLLM's proxy serves exactly that path on +`/v1/chat/completions`, so pointing `RISKKERNEL_OPENAI_BASE_URL` at the LiteLLM +proxy makes RiskKernel forward every "openai" call to LiteLLM, which then routes to +the real provider using the keys **stored in LiteLLM** (never in RiskKernel). + +``` +RISKKERNEL_OPENAI_BASE_URL=http://litellm:4000 + └────────────────────┘ + RiskKernel POSTs to http://litellm:4000/v1/chat/completions +``` + +> **Routing caveat — read this.** RiskKernel decides which native provider handles a +> request **by model-name prefix**: `claude-*` → Anthropic, `gpt-*`/`o1`/`o3` → +> OpenAI, everything else → the **default provider**. A long-tail LiteLLM model name +> like `gemini-1.5-pro` or `command-r` does **not** match the `gpt-*` prefix, so it +> would fall through to the default provider. To send arbitrary model names to the +> OpenAI provider (and thus to LiteLLM), set **`RISKKERNEL_DEFAULT_PROVIDER=openai`** +> so the fall-through lands on the LiteLLM-backed OpenAI provider. Then any model +> name LiteLLM understands is forwarded as-is. (If you still want native Anthropic +> alongside, `claude-*` names keep routing natively; only the unmatched names go to +> LiteLLM.) + +> **The OpenAI provider must be activated.** RiskKernel only registers the OpenAI +> provider when `OPENAI_API_KEY` is set. With LiteLLM upstream you still set it — +> use LiteLLM's **master key** (e.g. `sk-litellm-...`) so RiskKernel authenticates to +> the LiteLLM proxy, not to OpenAI. The real provider keys live in LiteLLM's config, +> not here. + +## Configuration + +Three knobs on the RiskKernel daemon, one on your app: + +| Where | Variable | Value | Why | +|---|---|---|---| +| RiskKernel | `RISKKERNEL_OPENAI_BASE_URL` | `http://litellm:4000` | Forward "openai" calls to the LiteLLM proxy. | +| RiskKernel | `OPENAI_API_KEY` | LiteLLM master key (or any non-empty value if LiteLLM has no auth) | Activates the OpenAI provider; sent to LiteLLM as the bearer token. | +| RiskKernel | `RISKKERNEL_DEFAULT_PROVIDER` | `openai` | Route long-tail model names to the LiteLLM-backed OpenAI provider (see the routing caveat). | +| Your app | `OPENAI_BASE_URL` | `http://localhost:7070/v1` | The usual one env var — your app calls RiskKernel unchanged. | + +LiteLLM holds the **real** provider keys (`GEMINI_API_KEY`, `COHERE_API_KEY`, +`MISTRAL_API_KEY`, …) in its own environment / config — RiskKernel never sees them. + +### docker-compose example + +```yaml +services: + litellm: + image: ghcr.io/berriai/litellm:main-latest + command: ["--config", "/etc/litellm/config.yaml"] + volumes: + - ./litellm-config.yaml:/etc/litellm/config.yaml:ro + environment: + # The real long-tail provider keys live HERE, not in RiskKernel. + GEMINI_API_KEY: ${GEMINI_API_KEY} + COHERE_API_KEY: ${COHERE_API_KEY} + LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY} # e.g. sk-litellm-... + + riskkernel: + image: ghcr.io/prashar32/riskkernel:latest + depends_on: [litellm] + ports: + - "7070:7070" + volumes: + - ./data:/data + # Optional: prices for long-tail models (see "Cost accuracy" below). + - ./pricing.json:/etc/riskkernel/pricing.json:ro + environment: + RISKKERNEL_OPENAI_BASE_URL: "http://litellm:4000" + OPENAI_API_KEY: ${LITELLM_MASTER_KEY} # authenticates to LiteLLM + RISKKERNEL_DEFAULT_PROVIDER: "openai" + RISKKERNEL_DEFAULT_DOLLARS: "5" # a hard $5/run ceiling + RISKKERNEL_PRICING_FILE: "/etc/riskkernel/pricing.json" +``` + +### Minimal LiteLLM `config.yaml` + +```yaml +model_list: + - model_name: gemini-1.5-pro # the name your app sends to RiskKernel + litellm_params: + model: gemini/gemini-1.5-pro # how LiteLLM routes it + api_key: os.environ/GEMINI_API_KEY + - model_name: command-r + litellm_params: + model: cohere/command-r + api_key: os.environ/COHERE_API_KEY + +general_settings: + master_key: os.environ/LITELLM_MASTER_KEY +``` + +### A governed call + +Your app changes nothing but `OPENAI_BASE_URL`; the model name is whatever LiteLLM +exposes: + +```bash +export OPENAI_BASE_URL=http://localhost:7070/v1 + +curl -s -D- http://localhost:7070/v1/chat/completions \ + -H 'content-type: application/json' \ + -H 'X-RiskKernel-Run-Id: gemini-demo' \ + -d '{"model":"gemini-1.5-pro","messages":[{"role":"user","content":"hi"}]}' +# → X-RiskKernel-Cost-Usd, X-RiskKernel-Tokens, X-RiskKernel-Step … +# the run is killed with HTTP 402 the moment it exceeds the budget. +``` + +The call is budgeted, metered, and audited by RiskKernel, then forwarded to LiteLLM, +which calls Gemini with the key it holds. + +## What governs what — be honest about the split + +| Concern | Owner | +|---|---| +| Cost / token / loop / time budgets (hard ceilings) | **RiskKernel** | +| Kill switch, crash-resume / checkpoints | **RiskKernel** | +| Human-in-the-loop approval gates | **RiskKernel** | +| Cost ledger, audit trail, OTel GenAI spans | **RiskKernel** | +| Routing to 100+ providers, provider failover, load balancing | **LiteLLM** | +| Holding the real provider API keys | **LiteLLM** | + +RiskKernel meters cost from the **token usage LiteLLM passes back** in the response — +the same mechanism as a native call — so spend is metered for any provider LiteLLM +returns usage for. RiskKernel does **not** rely on LiteLLM's own spend tracking; it +prices the usage itself. + +### Cost accuracy — the one caveat + +RiskKernel prices a call by multiplying provider-reported tokens by a **per-model +rate from its own token→$ table** (see [`docs/BUDGETS.md`](BUDGETS.md)). The built-in +rates only cover the native families (`claude-*`, `gpt-*`). A long-tail model +RiskKernel doesn't have a rate for is metered at **$0** and recorded with +`priced: false`: its tokens still count toward the **token** budget, but it can't +count toward the **dollar** budget until you add a rate. + +Close that gap with a `RISKKERNEL_PRICING_FILE` — a JSON file of model→rate overrides +(USD per 1M tokens), keyed by model name or prefix: + +```json +{ + "gemini-1.5-pro": { "inputPerM": 1.25, "outputPerM": 5.0 }, + "command-r": { "inputPerM": 0.5, "outputPerM": 1.5 } +} +``` + +Use the model names your app sends (the `model_name` from LiteLLM's config — that's +what comes back in the response and gets priced). The daemon refuses to start on a +malformed pricing file, and logs how many overrides it loaded. See the pricing +section of [`docs/BUDGETS.md`](BUDGETS.md) for the full format and stability promise. + +## Limitations, stated plainly + +- **One OpenAI provider slot.** RiskKernel has a single "openai" provider, and its + upstream is either the real OpenAI API or your LiteLLM proxy — not both at once. + When you point `RISKKERNEL_OPENAI_BASE_URL` at LiteLLM, *all* "openai" routing + (including `gpt-*` names) goes to LiteLLM. Expose OpenAI models through LiteLLM too + if you need them. There's no per-model upstream routing inside RiskKernel beyond + the prefix rule today; native `claude-*` continues to route to Anthropic + independently of this. +- **No dollar budget without a rate.** Long-tail models need a pricing override to + participate in the dollar budget (above). Token/loop/time budgets work regardless. +- **You run LiteLLM.** This adds a hop and a process to operate. For purely + Anthropic/OpenAI/Ollama stacks, native providers need no LiteLLM at all. +- **Streaming** works end-to-end (LiteLLM is OpenAI-compatible SSE), and the call is + metered from the stream's final usage chunk — same as a native streamed call.