Skip to content

feat: stream SSE tokens in-flight with delta content fallback#6

Merged
kaissb merged 1 commit into
stagingfrom
feature/0.1.1-streaming-token-counting
Apr 10, 2026
Merged

feat: stream SSE tokens in-flight with delta content fallback#6
kaissb merged 1 commit into
stagingfrom
feature/0.1.1-streaming-token-counting

Conversation

@kaissb

@kaissb kaissb commented Apr 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Zero-copy token accounting for streaming responses: chunks are
processed as they arrive via SSEAccumulator rather than buffering
the full response body.

Changes

  • Adds SSEAccumulator in internal/pricing/streaming.go that processes SSE response chunks byte-by-byte as they arrive, building lines across chunk boundaries without buffering the full response body
  • Updates budgetTrackingBody in proxy.go to route streaming reads through SSEAccumulator instead of b.buffer, preserving zero-copy semantics while tracking costs in-flight
  • Priority chain for token resolution: usage field in final SSE chunk (exact) → sum of choices[0].delta.content bytes (estimated, output only) → body-length estimation fallback (existing)
  • Delta-content estimation is significantly more accurate than body-length for SSE responses because it ignores the data:, JSON structure, and newline overhead that inflated previous estimates
  • Adds scripts/dev/test-streaming-tokens.sh integration script covering three cases: streaming with usage field, streaming without usage field (delta fallback), and non-streaming baseline

Testing

  • go test ./... passes
  • go vet ./... passes
  • bash scripts/dev/test-streaming-tokens.sh — all 3 tests pass and spent_usd > 0 for each agent
  • Unit tests in pricing/streaming_test.go: usage field extraction, delta content fallback, malformed chunks, partial lines spanning chunk boundaries
  • Existing pricing_test.go non-streaming path unchanged

Checklist

  • PR title follows Conventional Commits format (feat:, fix:, docs:, etc.)
  • One logical change per PR
  • No new dependencies added (or explicitly approved in description)
  • No sensitive data in code or logs
  • Doc comments added for new exported symbols

Zero-copy token accounting for streaming responses: chunks are
processed as they arrive via SSEAccumulator rather than buffering
the full response body.

- Add SSEAccumulator that parses SSE lines byte-by-byte across
  chunk boundaries, extracting usage fields or summing delta.content
  bytes as fallback
- Update budgetTrackingBody to route streaming reads through
  SSEAccumulator instead of the buffer, preserving zero-copy semantics
- Fall back to delta content byte estimation when no usage field is
  present (more accurate than full-body estimation for SSE)
- Add test-streaming-tokens.sh integration script covering: streaming
  with usage field, streaming without usage field, and non-streaming
  baseline
@kaissb
kaissb merged commit 7a26596 into staging Apr 10, 2026
5 checks passed
@kaissb
kaissb deleted the feature/0.1.1-streaming-token-counting branch April 10, 2026 22:10
@kaissb
kaissb restored the feature/0.1.1-streaming-token-counting branch May 31, 2026 17:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant