Skip to content

fix: refund users when model returns zero tokens with non-zero cost#489

Open
sh1ftred wants to merge 1 commit into
mainfrom
fix/refund-zero-token-responses
Open

fix: refund users when model returns zero tokens with non-zero cost#489
sh1ftred wants to merge 1 commit into
mainfrom
fix/refund-zero-token-responses

Conversation

@sh1ftred

@sh1ftred sh1ftred commented May 6, 2026

Copy link
Copy Markdown
Contributor

Problem

When a model produces empty output (zero tokens reported) but the cost calculation returns a non-zero USD cost, the previous behavior was to charge the full amount as output_msats. This means users were being billed for responses where the model produced nothing.

Fix

In the else branch of calculate_cost() (when no tokens are reported):

  • Before: output_msats = cost_in_msats (user gets charged the full amount)
  • After: Set input_msats = 0, output_msats = 0, cost_in_msats = 0 — issuing a full refund to the user
  • Added a logger.warning with usd_cost and model details for debugging/monitoring

File Changed

  • routstr/payment/cost_calculation.py

Testing

The change is a simple conditional branch update. When usage.input_tokens and usage.output_tokens are both 0 but usd_cost > 0, the user now receives a full refund instead of being charged.

When a model produces empty output (zero tokens reported) but the cost
calculation returns a non-zero USD cost, the previous behavior was to
charge the full amount as output_msats. This is incorrect — if no tokens
were produced, the user should not be charged.

Now the else branch:
- Logs a warning with the usd_cost and model for debugging
- Sets input_msats, output_msats, and cost_in_msats to 0 (full refund)
@9qeklajc

Copy link
Copy Markdown
Contributor

some upstream will have the problem of not setting the token count for some reasons even when the user gets a response which in that case implies that the request is free

@shroominic

Copy link
Copy Markdown
Contributor

Heads up: I opened a clean re-implementation of this fix against current main#561. This PR is 110 commits behind a refactored main (conflicting) and ships no tests; the new one re-applies the fix to the current _calculate_from_usd_cost code, tightens the gate so legitimately-billed cache-read-only turns aren't refunded (this PR refunds whenever prompt+completion==0, which over-fires on cache-only turns), and adds regression tests (truly-empty → refunded, verified fail-without-fix; cache-only → still billed). Suggest closing this in favour of #561 once reviewed.

shroominic added a commit that referenced this pull request Jun 20, 2026
When an upstream reports a non-zero USD cost but the response carries no
tokens at all (input, output, cache-read and cache-creation all zero), the
USD path billed the full USD-derived amount for an empty response. Return an
all-zero cost (full refund) for that case only.

The gate is tightened relative to the superseded PR #489: a cache-read- or
cache-creation-only turn legitimately reports zero prompt/completion tokens
with a real cost and must still be billed, so the refund only fires when every
token bucket is zero.

Adds unit tests covering both the refund and the still-billed cache-only path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants