feat: report what a run spent on the model - #180
Merged
Conversation
Nothing in Alphoryn reported token usage or spend. Establishing what the
2026-08-13 run cost meant querying Cloud Logging by hand, and the answer was
uncomfortable: ~$2.90 on the model while the entire trading P&L moved $0.59.
A number nobody can see is a number nobody manages.
Both agents now accumulate per-call usage and emit TOKEN_USAGE telemetry, and
the run prints a per-agent and per-run summary when it ends:
gemini-2.5-pro: Model usage: 116 calls, 979,824 in (576,558 cached) /
207,375 out, 99% reasoning -> ~$2.76
Three things worth knowing about the shape of this:
Output is derived as total - prompt rather than read from
candidates_token_count, because whether that field includes thought tokens has
changed between API versions. The subtraction cannot drift.
Usage is recorded before the failure paths, not after. A session that failed
still spent what it spent - the live run burned three retries on a session that
produced nothing, and that is exactly the spend you want to see. The summary
prints from the run loop's `finally` for the same reason: both live runs so far
ended by being killed.
format_summary takes a cost rather than a model so a total spanning two
differently priced models is summed by the caller. Pricing a mixed total at one
model's rate is how a cost report starts lying. An unpriced model reports its
tokens and says "cost unknown" instead of guessing.
Verified against the real run-4 figures: the summary reproduces the $2.89 that
the manual Cloud Logging query produced.
mohammadp1001
force-pushed
the
feat/report-run-cost
branch
from
August 13, 2026 22:18
bf3ceb3 to
1b995b2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Nothing in Alphoryn reported what a run cost. Establishing the 2026-08-13 figure
meant querying Cloud Logging by hand, and the answer was uncomfortable:
The run cost more than the entire P&L swing, in both directions. The manual dig
also turned up the thing that actually matters: 94% of output tokens were
reasoning tokens, and reasoning bills at the output rate - 8x the input rate.
Thinking is the cost.
The fix
Both agents accumulate per-call usage and emit
TOKEN_USAGEtelemetry. The runprints a per-agent and per-run summary when it ends:
That
$2.89is the same number the manual Cloud Logging query produced forrun-4, which is how this was checked.Three decisions worth flagging
Output is
total - prompt, notcandidates_token_count. Whether that fieldincludes thought tokens has changed between API versions; the subtraction cannot
drift, because everything that is not prompt bills at the output rate.
Usage is recorded before the failure paths. A session that failed still spent
what it spent - the live run burned three retries on a session that produced
nothing, and that is precisely the spend you want visible. The run summary prints
from the loop's
finallyfor the same reason: both live runs so far ended bybeing killed, and an aborted run is when you most want the number.
format_summarytakes a cost, not a model, so a total spanning twodifferently priced models is summed per-model by the caller. Pricing a mixed
total at one model's rate is how a cost report starts lying. An unpriced model
reports its tokens and says
cost unknownrather than guessing.usage_from_eventis a single shared reader for both agents, deliberately - thelast time this kind of event-reading logic was written twice it drifted and cost
the feedback agent a month of silent failure (#178).
What this does not do
It does not reduce the cost. Note for whoever picks that up: do not reach for
disabling
include_thoughts- peralphoryn/agents/thinking.py, 2.5 modelsreason either way, so you would still pay for the tokens and merely lose the
trace. The real levers are a thinking budget cap or a cheaper model for
main_agent.
Verification
ruff check alphoryn/ tests/cleanusage/tokens.py39 stmts / 6 branches, all covered)GOOGLE_APPLICATION_CREDENTIALS=/nonexistent/adc.json python -m pytest -q