feat(cost): estimated cost tracking in bundle/Observer/serve + fail-closed maxUsd cap (0.19.0) - #294
Merged
Merged
Conversation
Add src/pricing.ts: a single, dated, operator-editable per-model + E2B desktop rate table plus two pure, deterministic estimators (estimateActorCost / estimateDesktopCost) with injectable rates for tests. Unknown model/no-usage/no-duration => DECLARED ABSENT (estimatedCostUsd: null + a reason), never a guessed or silent-zero cost. Add the additive, optional ActorTrace.estimatedCost field (humanish.actor-estimated-cost.v1), kept distinct from the reserved provider-returned tokenUsage.costUsd so an estimate is never confused with an authoritative charge. Export the pricing surface from src/index.ts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…beling Add the optional, additive RunBundle.cost (humanish.run-cost-summary.v1) with RunCostLine/RunCostSummary: the sum of every lane's model-token estimate plus the E2B desktop-minute estimate, carrying the terminal ledger's null-discipline (a present-but-unpriceable line is null + a reason, contributes nothing to the total; an all-null summary has a null total, never 0). humanish.run-bundle.v1 stays v1 — every field is additive/optional so old bundles remain byte-stable and loadable. verifyPreparedRun gains a "cost estimate labeling" check (costLabelingFindings): absence PASSES (fail-open on display), but a CLAIMED dollar figure without its ratesAsOf date + source, or a total that does not equal round6(sum of only the non-null lines), FAILS. Magnitude is never inspected — a correctly-labeled huge estimate still passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add optional maxUsd + an injected pure estimateTurnCostUsd to CuaLoopOptions (threaded through CuaActorSessionOptions). When maxUsd is set, the loop aborts (completionReason "budget_reached") the moment the running estimated spend crosses it — placed right after per-turn usage accumulation and BEFORE the next provider.nextTurn, so a model stuck retrying cannot spend one extra turn past the cap (the runaway-retry guard). Absent maxUsd is a no-op (byte-stable), and a null/unpriceable estimate can never trip the cap (never a silent-zero abort). The estimator is injected so the loop stays free of the operator rate table and the cap is deterministic in tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…unpriced-cap preflight Attach a token-derived estimatedCost to each CUA lane's trace at the lab boundary (before persist), measure the host-side E2B desktop create->teardown span via an injected clock (deterministic in tests), and assemble a run-level RunCostSummary (per-lane model-token lines + one aggregate desktop-minute line) on both the single-lane and fan-out bundles. The null-discipline is honored end-to-end: an unpriced model yields a present-but-null line that contributes nothing to the total; dry-run and in-process lanes invent no spend (cost omitted, byte-stable). Wire execution.caps.maxUsd into the loop as the fail-closed abort (with an injected pure per-turn estimator), reusing the terminal lane's LabScenarioCaps shape (parsed via the existing parseCaps; inert/warned off the CUA route). Resolve the invariant tension fail-closed at PREFLIGHT: a maxUsd cap on a model src/pricing.ts cannot price is refused (HUMANISH_CUA_LAB_UNPRICED_CAP) before any sandbox rather than run uncapped. Thread the clock into the concurrent shared-world lane too. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Project the run-level cost summary through buildObserverData to ObserverData.cost, and render it in the Observer: a run-total stat in the status bar + run-details popover, and a per-lane figure in the focus side panel. Every dollar reads "~$X estimated (rates as of <date>)" (never a bare "$X"), tags a placeholder rate, and reads "not estimated (<reason>)" when declared absent. The serve library index + history feed carry a labeled per-run "~$X est." token with a rates-as-of tooltip; buildHistoryIndex projects estimatedTotalUsd/ratesAsOf/placeholder and the LibraryHistory row widens to match. Pages stay self-contained (no new assets/fetch). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bump package.json 0.18.0 -> 0.19.0 (minor: additive/back-compatible schema + a fail-closed cap that defaults OFF, so no existing run changes behavior; humanish.run-bundle.v1 stays v1). Update the release-test pin and the four doc version pins. Document the three new .v1 schema tags (humanish.pricing.v1, run-cost-summary.v1, actor-estimated-cost.v1), the declared-absent null discipline, the mandatory ratesAsOf/source provenance, the placeholder flag, and that verify asserts labeling not magnitude (schemas.md, run-bundle.md). Add the estimatedCost-vs-costUsd distinction to the actor-contract architecture doc, a README cost-tracking + execution.caps.maxUsd section pointing at the operator-editable src/pricing.ts, and a 0.19.0 capability-state entry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…iled; per-lane cap honesty; conservative aggregate asOf - computer-use: the fail-closed maxUsd break now mirrors the wall-clock path — budget_reached (passed) only when materialActions>0; a zero-action runaway that crosses the cap ends gave_up (failed) instead of greenlighting the exact runaway the cap exists to catch. Both branches cite the estimate + the cap. - cua-actor-lab: preflight run-level WARNING when maxUsd is set and >1 lane — maxUsd is a PER-LANE cap; N lanes may spend up to N × maxUsd. Documented as per-lane in README, schemas.md, and the LabExecution.caps config help. - cua-actor-lab: aggregate run-cost-summary ratesAsOf now takes the MIN (oldest) contributing asOf, not MAX — an aggregate is only as fresh as its stalest input; per-line breakdown keeps its own true asOf. - docs: fix the self-contradictory Run Cost Summary example (fullyEstimated:true since both lines are priced). - tests: zero-action maxUsd:0 abort -> FAILED/gave_up; material-then-cap -> budget_reached/passed with the cap cited; aggregate asOf reports the OLDER date; fan-out per-lane cap warning cites the ~N × cap ceiling. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds estimated cost tracking across lanes, surfaced in the Observer and the
serverun-library, plus a fail-closedmaxUsdspend cap for the computer-use lane. Ships as 0.19.0. Motivated directly by a session where ~7 live computer-use runs were fired with no visible running tab — the token counts were in the bundles, but nothing computed dollars or showed them.Cost is always an ESTIMATE, never an authoritative charge
The load-bearing discipline (claims-match-mechanism, invariant 6):
src/pricing.ts— a single dated, operator-editable rate table (per-model token rates + E2B desktop $/minute), each entry carrying asourceandasOfdate, with a prominent "these are estimates, update them" header. The OpenAI CU-model default and the E2B desktop rate ship as clearly-flagged placeholders for the maintainer to confirm against the live pricing pages.estimatedCostUsd(the real, provider-returnedcostUsdon the codex path is left untouched, so a barecostUsdstill means "the provider billed this").null+ areason), never a guessed or silent-zero cost. A run total isnullif every line is null; a number equals the rounded sum of only the priced lines. AggregateratesAsOfis the oldest contributing date (an estimate is only as fresh as its stalest rate).Surfaces
humanish.run-bundle.v1, additive/back-compatible): per-actorestimatedCoston the trace + a run-levelcostsummary (token cost per lane + E2B desktop-minutes), with full provenance.servelibrary: per-run estimated cost.verify: never fails on missing cost (advisory), but fails closed if a cost is claimed without the estimated-labeling/provenance.--maxUsdspend cap (the runaway-retry guard)Per-lane, opt-in (
execution.caps.maxUsd, reusing the terminal lane's config shape). The computer-use loop aborts fail-closed the moment the running estimate crosses the cap. Honest classification: crossing the cap after material work isbudget_reached(completed); crossing it with zero progress isgave_up(failed) — so a stuck runaway is caught, not blessed as a pass. Documented as a per-lane cap, with a preflight warning that an N-lane fan-out can reach ~N × the cap.How the blocker was caught
The adversarial review found — and this PR fixes — a real blocker: the spend cap initially reported a zero-progress runaway as a passed lane (mapping
budget_reached→passed with no material-progress gate), silently greenlighting the exact runaway the cap exists to stop (maxUsd:0made it deterministic). Fixed by mirroring the wall-clock budget rule and verified by a dedicated test; a zero-action abort now surfaces asfailed.Verification
pnpm release:checkgreen: 1176 tests pass (public-surface scan 476 files / 2 binaries, pack dry-run good). No new dependencies.🤖 Generated with Claude Code