feat(gateway): streaming proxy with budget enforcement (OpenAI) - #124
Merged
Conversation
Support stream:true on POST /v1/chat/completions. The budget is enforced before the stream opens (BeginStep + CanProceed → 402 if over); the provider's SSE is forwarded to the client verbatim — authentic chunks, no re-serialization — while token usage is metered from the final usage chunk; and the run's context (time budget, kill switch, or client disconnect) cuts a live stream. The streamed call is recorded to the ledger at stream end, so the next call is refused if it went over budget. Dollar/token are pre-checked + post-recorded (OpenAI only reports usage at the end of a stream, so true mid-stream dollar cutting isn't possible — the time budget / kill switch are what cut a live stream). Adds an optional provider.Streamer interface; OpenAI implements it (asking for stream_options.include_usage). Providers without it, and the Anthropic /v1/messages endpoint, return a clear 501 instead of silently buffering — native Anthropic SSE streaming is the documented next step. Tests: provider-level (mock SSE upstream → chunks forwarded verbatim, usage + model parsed), gateway-level (client receives the SSE and the call is metered to the run; the budget refuses a second streamed call at 402 before the provider is hit; a non-streaming provider returns 501). Race detector clean on the streaming path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The most-requested adoption gap:
stream:trueonPOST /v1/chat/completions. Closes #22.Behavior
BeginStep+CanProceed; if the run is over budget, a402is returned and the provider is never hit.Honest scope: OpenAI reports usage only in the final stream chunk, so true mid-stream dollar/token cutting isn't possible — those are pre-checked + post-recorded; the time budget / kill switch are what cut a live stream. A new optional
provider.Streamerinterface gates this; OpenAI implements it. Providers without it, and the Anthropic/v1/messagesendpoint, return a clear501rather than silently buffering — native Anthropic SSE streaming is the documented next step.Tests
usage/modelparsed (incl. thestream+include_usagerequest shape).gofmt/go vetclean; full suite green; race detector clean on the gateway + provider streaming path.Built alongside (parallel, in isolated worktrees) #122 (shell completions) and #123 (
/metrics); all three touch disjoint files. CHANGELOG entries will need trivial merge ordering.