feat: reproducible cost benchmark + provider base-URL override - #79
Merged
Conversation
Add RISKKERNEL_OPENAI_BASE_URL / RISKKERNEL_ANTHROPIC_BASE_URL to point a native provider at an OpenAI-compatible gateway, a corporate proxy, or a local mock instead of its default API endpoint — via a chainable WithBaseURL on each provider, wired from config. Namespaced deliberately: the bare OPENAI_BASE_URL / ANTHROPIC_BASE_URL are what a caller sets to point an app *at* RiskKernel, so reusing them for the upstream would make RiskKernel forward to itself in a shared shell. Tested: WithBaseURL trims and defaults correctly, a Chat call routes to the overridden base, and config reads the namespaced env vars.
…ling Runs the same looping agent twice against a deterministic mock provider — once ungoverned, once through RiskKernel with a hard dollar budget — and reports the spend saved, read from RiskKernel's own cost ledger. Key-free and reproducible: the mock returns fixed token usage and pricing.json pins the model price, so the only variable is whether the budget stopped the loop. Uses the namespaced RISKKERNEL_OPENAI_BASE_URL override to point RiskKernel at the mock.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things, shipped together because the second is what the first needs.
1. Reproducible cost benchmark (
benchmark/)The flagship "dollars saved" measurement. The same looping agent runs twice against a deterministic mock provider — once ungoverned, once through RiskKernel with a hard dollar budget — and the spend is compared, read straight from RiskKernel's own ledger:
Key-free and reproducible (
python3 benchmark/benchmark.py): the mock returns fixed token usage andpricing.jsonpins the model price, so the only variable is whether the budget stopped the loop. The governed spend is measured, not modelled (GET /v1/runs/{id}→usage.dollars). The cap is flat no matter how long the runaway would have continued — the saving grows without bound; see the README methodology.2. Provider base-URL override (the enabler)
RISKKERNEL_OPENAI_BASE_URL/RISKKERNEL_ANTHROPIC_BASE_URLpoint a native provider at a gateway, proxy, or local mock instead of the default endpoint — a chainableWithBaseURL(...), wired from config. RiskKernel-namespaced so it never collides with the caller-facingOPENAI_BASE_URLused to point an app at RiskKernel (which would make RiskKernel forward to itself in a shared shell).Verified
WithBaseURLtrims/defaults correctly and aChatcall routes to the override (asserted against a mock); config reads the namespaced vars;go test ./...green (13 packages). The benchmark runs end-to-end and halts the governed loop at exactly the budget.