Skip to content

feat: report what a run spent on the model - #180

Merged
mohammadp1001 merged 1 commit into
mainfrom
feat/report-run-cost
Aug 13, 2026
Merged

feat: report what a run spent on the model#180
mohammadp1001 merged 1 commit into
mainfrom
feat/report-run-cost

Conversation

@mohammadp1001

Copy link
Copy Markdown
Owner

The problem

Nothing in Alphoryn reported what a run cost. Establishing the 2026-08-13 figure
meant querying Cloud Logging by hand, and the answer was uncomfortable:

Model spend ~$2.90
Trading P&L that day -$0.59

The run cost more than the entire P&L swing, in both directions. The manual dig
also turned up the thing that actually matters: 94% of output tokens were
reasoning tokens
, and reasoning bills at the output rate - 8x the input rate.
Thinking is the cost.

The fix

Both agents accumulate per-call usage and emit TOKEN_USAGE telemetry. The run
prints a per-agent and per-run summary when it ends:

  gemini-2.5-pro: Model usage: 116 calls, 979,824 in (576,558 cached) / 207,375 out, 99% reasoning  ->  ~$2.76
  gemini-2.5-pro: Model usage: 14 calls, 8,581 in (0 cached) / 12,212 out, 0% reasoning  ->  ~$0.13
Run Model usage: 130 calls, 988,405 in (576,558 cached) / 219,587 out, 93% reasoning  ->  ~$2.89

That $2.89 is the same number the manual Cloud Logging query produced for
run-4, which is how this was checked.

Three decisions worth flagging

Output is total - prompt, not candidates_token_count. Whether that field
includes thought tokens has changed between API versions; the subtraction cannot
drift, because everything that is not prompt bills at the output rate.

Usage is recorded before the failure paths. A session that failed still spent
what it spent - the live run burned three retries on a session that produced
nothing, and that is precisely the spend you want visible. The run summary prints
from the loop's finally for the same reason: both live runs so far ended by
being killed, and an aborted run is when you most want the number.

format_summary takes a cost, not a model, so a total spanning two
differently priced models is summed per-model by the caller. Pricing a mixed
total at one model's rate is how a cost report starts lying. An unpriced model
reports its tokens and says cost unknown rather than guessing.

usage_from_event is a single shared reader for both agents, deliberately - the
last time this kind of event-reading logic was written twice it drifted and cost
the feedback agent a month of silent failure (#178).

What this does not do

It does not reduce the cost. Note for whoever picks that up: do not reach for
disabling include_thoughts
- per alphoryn/agents/thinking.py, 2.5 models
reason either way, so you would still pay for the tokens and merely lose the
trace. The real levers are a thinking budget cap or a cheaper model for
main_agent.

Verification

  • ruff check alphoryn/ tests/ clean
  • 734 tests, 100% coverage (usage/tokens.py 39 stmts / 6 branches, all covered)
  • Reproduced the CI environment: GOOGLE_APPLICATION_CREDENTIALS=/nonexistent/adc.json python -m pytest -q

Nothing in Alphoryn reported token usage or spend. Establishing what the
2026-08-13 run cost meant querying Cloud Logging by hand, and the answer was
uncomfortable: ~$2.90 on the model while the entire trading P&L moved $0.59.
A number nobody can see is a number nobody manages.

Both agents now accumulate per-call usage and emit TOKEN_USAGE telemetry, and
the run prints a per-agent and per-run summary when it ends:

  gemini-2.5-pro: Model usage: 116 calls, 979,824 in (576,558 cached) /
                  207,375 out, 99% reasoning  ->  ~$2.76

Three things worth knowing about the shape of this:

Output is derived as total - prompt rather than read from
candidates_token_count, because whether that field includes thought tokens has
changed between API versions. The subtraction cannot drift.

Usage is recorded before the failure paths, not after. A session that failed
still spent what it spent - the live run burned three retries on a session that
produced nothing, and that is exactly the spend you want to see. The summary
prints from the run loop's `finally` for the same reason: both live runs so far
ended by being killed.

format_summary takes a cost rather than a model so a total spanning two
differently priced models is summed by the caller. Pricing a mixed total at one
model's rate is how a cost report starts lying. An unpriced model reports its
tokens and says "cost unknown" instead of guessing.

Verified against the real run-4 figures: the summary reproduces the $2.89 that
the manual Cloud Logging query produced.
@mohammadp1001
mohammadp1001 merged commit 906bb41 into main Aug 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant