Skip to content

feat(gateway): streaming proxy with budget enforcement (OpenAI) - #124

Merged
prashar32 merged 2 commits into
mainfrom
feat/streaming-proxy
Jun 13, 2026
Merged

feat(gateway): streaming proxy with budget enforcement (OpenAI)#124
prashar32 merged 2 commits into
mainfrom
feat/streaming-proxy

Conversation

@prashar32

Copy link
Copy Markdown
Owner

The most-requested adoption gap: stream:true on POST /v1/chat/completions. Closes #22.

Behavior

  • Budget enforced before the stream opensBeginStep + CanProceed; if the run is over budget, a 402 is returned and the provider is never hit.
  • Authentic SSE pass-through — the provider's SSE chunks are forwarded to the client verbatim (no parse-and-reserialize), so an OpenAI client sees a real OpenAI stream. Token usage is teed off the final usage chunk for metering.
  • Live-stream cut — the call runs under the run's context, so the time budget, kill switch, or client disconnect interrupts a stream in flight.
  • Metered at stream end — the streamed call is recorded to the cost ledger, so the next call is refused if this one went over budget.

Honest scope: OpenAI reports usage only in the final stream chunk, so true mid-stream dollar/token cutting isn't possible — those are pre-checked + post-recorded; the time budget / kill switch are what cut a live stream. A new optional provider.Streamer interface gates this; OpenAI implements it. Providers without it, and the Anthropic /v1/messages endpoint, return a clear 501 rather than silently buffering — native Anthropic SSE streaming is the documented next step.

Tests

  • Provider: mock SSE upstream → chunks forwarded verbatim, usage/model parsed (incl. the stream+include_usage request shape).
  • Gateway: client receives the SSE and the call is metered to the run; a second streamed call is refused at 402 before the provider is hit (loop budget); a non-streaming provider returns 501.
  • gofmt/go vet clean; full suite green; race detector clean on the gateway + provider streaming path.

Built alongside (parallel, in isolated worktrees) #122 (shell completions) and #123 (/metrics); all three touch disjoint files. CHANGELOG entries will need trivial merge ordering.

Support stream:true on POST /v1/chat/completions. The budget is enforced before
the stream opens (BeginStep + CanProceed → 402 if over); the provider's SSE is
forwarded to the client verbatim — authentic chunks, no re-serialization — while
token usage is metered from the final usage chunk; and the run's context (time
budget, kill switch, or client disconnect) cuts a live stream. The streamed call
is recorded to the ledger at stream end, so the next call is refused if it went
over budget.

Dollar/token are pre-checked + post-recorded (OpenAI only reports usage at the
end of a stream, so true mid-stream dollar cutting isn't possible — the time
budget / kill switch are what cut a live stream). Adds an optional
provider.Streamer interface; OpenAI implements it (asking for
stream_options.include_usage). Providers without it, and the Anthropic
/v1/messages endpoint, return a clear 501 instead of silently buffering — native
Anthropic SSE streaming is the documented next step.

Tests: provider-level (mock SSE upstream → chunks forwarded verbatim, usage +
model parsed), gateway-level (client receives the SSE and the call is metered to
the run; the budget refuses a second streamed call at 402 before the provider is
hit; a non-streaming provider returns 501). Race detector clean on the streaming
path.
@prashar32
prashar32 merged commit e8a1921 into main Jun 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Proxy: streaming (SSE) support with mid-stream budget enforcement

1 participant