Skip to content

feat: reproducible cost benchmark + provider base-URL override - #79

Merged
prashar32 merged 2 commits into
mainfrom
feat/provider-base-url
Jun 7, 2026
Merged

feat: reproducible cost benchmark + provider base-URL override#79
prashar32 merged 2 commits into
mainfrom
feat/provider-base-url

Conversation

@prashar32

@prashar32 prashar32 commented Jun 6, 2026

Copy link
Copy Markdown
Owner

Two things, shipped together because the second is what the first needs.

1. Reproducible cost benchmark (benchmark/)

The flagship "dollars saved" measurement. The same looping agent runs twice against a deterministic mock provider — once ungoverned, once through RiskKernel with a hard dollar budget — and the spend is compared, read straight from RiskKernel's own ledger:

baseline (no governance)  50 calls  $0.6250
governed (RiskKernel)     20 calls  $0.2500   ← halted: dollar_budget_exceeded
dollars saved             $0.3750  (60%)

Key-free and reproducible (python3 benchmark/benchmark.py): the mock returns fixed token usage and pricing.json pins the model price, so the only variable is whether the budget stopped the loop. The governed spend is measured, not modelled (GET /v1/runs/{id}usage.dollars). The cap is flat no matter how long the runaway would have continued — the saving grows without bound; see the README methodology.

2. Provider base-URL override (the enabler)

RISKKERNEL_OPENAI_BASE_URL / RISKKERNEL_ANTHROPIC_BASE_URL point a native provider at a gateway, proxy, or local mock instead of the default endpoint — a chainable WithBaseURL(...), wired from config. RiskKernel-namespaced so it never collides with the caller-facing OPENAI_BASE_URL used to point an app at RiskKernel (which would make RiskKernel forward to itself in a shared shell).

Verified

WithBaseURL trims/defaults correctly and a Chat call routes to the override (asserted against a mock); config reads the namespaced vars; go test ./... green (13 packages). The benchmark runs end-to-end and halts the governed loop at exactly the budget.

prashar32 added 2 commits June 7, 2026 04:02
Add RISKKERNEL_OPENAI_BASE_URL / RISKKERNEL_ANTHROPIC_BASE_URL to point a native
provider at an OpenAI-compatible gateway, a corporate proxy, or a local mock
instead of its default API endpoint — via a chainable WithBaseURL on each
provider, wired from config.

Namespaced deliberately: the bare OPENAI_BASE_URL / ANTHROPIC_BASE_URL are what a
caller sets to point an app *at* RiskKernel, so reusing them for the upstream
would make RiskKernel forward to itself in a shared shell.

Tested: WithBaseURL trims and defaults correctly, a Chat call routes to the
overridden base, and config reads the namespaced env vars.
…ling

Runs the same looping agent twice against a deterministic mock provider — once
ungoverned, once through RiskKernel with a hard dollar budget — and reports the
spend saved, read from RiskKernel's own cost ledger. Key-free and reproducible:
the mock returns fixed token usage and pricing.json pins the model price, so the
only variable is whether the budget stopped the loop. Uses the namespaced
RISKKERNEL_OPENAI_BASE_URL override to point RiskKernel at the mock.
@prashar32 prashar32 changed the title feat(provider): override a provider's upstream base URL feat: reproducible cost benchmark + provider base-URL override Jun 6, 2026
@prashar32
prashar32 merged commit 23b0062 into main Jun 7, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant