diff --git a/README.md b/README.md index f93e044..074ec20 100644 --- a/README.md +++ b/README.md @@ -327,6 +327,27 @@ the run bundle; the trace stores only the matched rule id and primitive names. B and text observation requires a Chrome/Chromium CDP session in the desktop. For deterministic browser-observed stops, set `execution.desktop.browser: chrome` or `chromium`. +**Cost tracking (estimated).** Computer-use run bundles carry an advisory `cost` block: a +per-lane token-derived model estimate plus one aggregate E2B desktop-minute estimate. Every +dollar figure is an ESTIMATE, never a provider charge — it is a rate-table multiply, always +surfaced as "~$X estimated (rates as of ``)" in the Observer and the run library, and it +carries the pricing date + source so a token-derived number is never mistaken for an +authoritative bill. Unknown model/rate is declared absent (`null` + a reason), never guessed or +silently zeroed; dry-runs invent no spend. The rates live in +[`src/pricing.ts`](src/pricing.ts) as **operator-editable, dated estimates** — some are +`placeholder` stand-ins (the `gpt-5.5` model rate and the E2B desktop rate); update the numbers +AND the `asOf` date when providers change pricing. + +**Fail-closed spend cap.** Set `execution.caps.maxUsd` on a computer-use lab to abort a session +the moment its running estimated spend crosses the cap — the runaway-retry guard (mirrors the +terminal lane's `scenario.caps.maxUsd`). It is a **per-lane** cap: enforced inside each lane's loop, +so an N-lane fan-out can spend up to N × `maxUsd` before any lane aborts (the run bundle warns with +the true ~N × cap ceiling; a shared run-level budget is future work). A lane that did real work then +hits its cap passes (`budget_reached`); a zero-action runaway that crosses it fails (`gave_up`). +Absent = uncapped (the historical CUA behavior); `maxUsd: 0` = no-spend. A cap on a model +`src/pricing.ts` cannot price is refused at preflight (`HUMANISH_CUA_LAB_UNPRICED_CAP`) rather than +run uncapped — an unenforceable cap is more dangerous than none, so add a rate or drop the cap. + **Failed-lane reruns.** Multi-lane CUA fan-out can be rerun surgically without mutating the source run: diff --git a/docs/architecture/actor-contract.md b/docs/architecture/actor-contract.md index a3a74a9..b53f458 100644 --- a/docs/architecture/actor-contract.md +++ b/docs/architecture/actor-contract.md @@ -165,6 +165,7 @@ export interface ActorTrace { counts: Record; items: ActorTraceItem[]; tokenUsage?: { input?: number; output?: number; total?: number; costUsd?: number }; + estimatedCost?: ActorEstimatedCost; // humanish.actor-estimated-cost.v1 (additive) capabilities: ActorCapabilities; } @@ -232,6 +233,17 @@ export interface Actor { target URLs, or unredacted provider payloads in the trace. - **Capabilities.** Declare them honestly; the registry uses them to refuse unsuitable dispatch. +- **Cost (estimate vs. charge).** `tokenUsage.costUsd` stays RESERVED for a + real, provider-returned charge (the codex/agent-SDK path) — a bare `costUsd` + always means "the provider billed this". The optional `estimatedCost` + (`humanish.actor-estimated-cost.v1`) is a SEPARATE, differently-named field: a + token-derived rate-table multiply from the operator-editable `src/pricing.ts`, + labeled honestly as an estimate and projected up into `RunBundle.cost` (see + [`../contracts/schemas.md`](../contracts/schemas.md) → Run Cost Summary And + Estimated Actor Cost). The CUA lab computes and attaches `estimatedCost` at the + lab boundary before persisting the trace, so the pure computer-use loop never + depends on the pricing table. An unknown model yields + `estimatedCostUsd: null` + a `reason`, never a guessed charge. ## The scripted-browser lane (shipped) diff --git a/docs/contracts/run-bundle.md b/docs/contracts/run-bundle.md index ea30811..213dd6f 100644 --- a/docs/contracts/run-bundle.md +++ b/docs/contracts/run-bundle.md @@ -112,6 +112,29 @@ instead (a dirty working tree cannot be commit-pinned), and `app-url` carries no code pin at all. No path, basename, or other host-machine string ever enters this field; identity is digests, a sha, a boolean, and counts. +## Cost Estimate (advisory) + +`cost` is optional and additive (`humanish.run-cost-summary.v1`): the +computer-use lane's run-level cost ESTIMATE — the sum of each lane's +token-derived model cost plus one aggregate E2B desktop-minute figure. It is an +ESTIMATE, never authoritative: every dollar is a rate-table multiply from the +operator-editable `src/pricing.ts`, carries the pricing `ratesAsOf` date and +`source`, and is surfaced with the "estimated (rates as of ``)" label — +never a bare charge. It follows the same **declared-absent** discipline as the +terminal cost ledger: an unpriceable line stays present with +`estimatedCostUsd: null` + a `reason` and contributes nothing; +`estimatedTotalUsd` is `null` iff every line is null (never coerced to `0`). +Dry-runs and lanes that spend nothing omit `cost` entirely, so pre-existing +bundles stay byte-stable. Each lane's own estimate also rides its +`stream.actor.estimatedCost` (`humanish.actor-estimated-cost.v1`), kept distinct +from the reserved provider-returned `tokenUsage.costUsd`. See +[`schemas.md`](schemas.md) → Run Cost Summary And Estimated Actor Cost. + +`humanish verify` treats cost as ADVISORY on magnitude and FAIL-CLOSED on +labeling: absence passes, but a claimed dollar figure without its `ratesAsOf` +date + `source`, or a total that does not match its known lines, fails. Verify +never inspects the magnitude — a correctly-labeled large estimate still passes. + ## Adapter Score `adapterScore` is optional and namespaced. It lets a downstream adapter summarize diff --git a/docs/contracts/schemas.md b/docs/contracts/schemas.md index 440d1a3..a284aaa 100644 --- a/docs/contracts/schemas.md +++ b/docs/contracts/schemas.md @@ -3,7 +3,7 @@ Date: 2026-06-02 (current-state note updated 2026-07-14) Status: reference map for the major contracts shipped through source version -`0.18.0`; it is not an exhaustive inventory of command/result envelopes. Exported types, +`0.19.0`; it is not an exhaustive inventory of command/result envelopes. Exported types, schema constants, parsers, and validators in `src/` are authoritative. Rows marked "reserved" name layering intent only — no code emits or validates them yet. Do not emit a reserved schema. @@ -43,6 +43,9 @@ workflow without leaking private upstream truth into core. | Feedback | `humanish.feedback.v1` | `public-safe-feedback` | | Terminal cost ledger | `humanish.terminal-cost-ledger.v1` | see Terminal Cost Ledger below | | Terminal no-spend proof | `humanish.terminal-no-spend-proof.v1` | see Terminal Cost Ledger below | +| Pricing (operator-editable rates) | `humanish.pricing.v1` (`src/pricing.ts`; dated per-model + E2B desktop rates) | see Run Cost Summary And Estimated Actor Cost below | +| Run cost summary | `humanish.run-cost-summary.v1` (additive `RunBundle.cost`; estimate, never a charge) | see Run Cost Summary And Estimated Actor Cost below | +| Estimated actor cost | `humanish.actor-estimated-cost.v1` (additive `ActorTrace.estimatedCost`) | see Run Cost Summary And Estimated Actor Cost below | | Adapter score | `humanish.adapter-score.v1` (`RunBundle.adapterScore`; namespaced; route-specific acceptance semantics) | see Product-Adapter Extension Seam below | | Adapter artifact | `humanish.adapter-artifact.v1` (`RunBundle.adapterArtifacts[]`; namespaced; local relative proof references) | see Product-Adapter Extension Seam below | | Shared-world evidence | `humanish.shared-world.v1` (additive `RunBundle.sharedWorld` + `RunBundle.attributionClass`; `topologyMode: sequential \| concurrent`) | see Shared-World Evidence below | @@ -607,6 +610,14 @@ Core-owned fields: distinct from `timed_out`, which stays reserved for a zero-progress deadline hit and remains a failure) - `ids`, `counts`, `items[]`, optional `tokenUsage`, `capabilities` +- optional `estimatedCost` (`humanish.actor-estimated-cost.v1`): a token-derived + cost ESTIMATE for this lane (see Run Cost Summary And Estimated Actor Cost). + It is deliberately a DIFFERENT field from `tokenUsage.costUsd`: a bare + `costUsd` is RESERVED for a real provider-returned charge, while + `estimatedCost.estimatedCostUsd` is a rate-table multiply, named honestly as + an estimate so a reader can never confuse the two (invariant 6). Absent on + codex/scripted lanes and on every pre-existing bundle; a `null` + `estimatedCostUsd` is DECLARED ABSENT (unknown rate / no usage), never 0. Unexpected actor-loop diagnostics live inside `items[]` as `kind: notice`, `status: error` rows. They are public-safe evidence, not crash @@ -661,7 +672,7 @@ mode (`loopback | exposed | share-safe-open`), the loopback host/port, `allowEmails`, `allowDomains` — operator-supplied allow rules, public-safe to echo to the operator's own stdout, never persisted into any bundle), runs listed, computed warnings, and the `ServeErrorCode` union. Exposure auth is -tunnel-edge only — as of 0.18.0 there are no `capabilityUrl`/`publicCapabilityUrl` +tunnel-edge only — as of 0.19.0 there are no `capabilityUrl`/`publicCapabilityUrl` /`ttlMinutes` fields, no `--auth`/`--ttl` flags, and no `capability-link` mode (the in-process `observer-auth.ts` capability-link was removed as a pre-1.0 breaking change). @@ -743,6 +754,82 @@ measure) and never grant a green pass (they surface as unmeasured). `verifyRun` fails closed when a live bundle lacks the cost ledger or no-spend proof, when the proof claims zero on a `null` line, or when known spend exceeds the declared cap. +## Run Cost Summary And Estimated Actor Cost + +The computer-use (CUA) lane surfaces an ADVISORY, additive cost ESTIMATE. It is +never authoritative: every dollar figure is a rate-table multiply, labeled +"estimated (rates as of ``)", and is NEVER presented as a provider charge +(invariant 6). Three new `.v1` schema tags ship, all additive and optional so +`humanish.run-bundle.v1` stays v1 and every pre-existing bundle is byte-stable: + +- `humanish.pricing.v1` — the OPERATOR-EDITABLE rate table in `src/pricing.ts`: + dated per-model input/output USD-per-token rates and an E2B desktop + USD-per-minute rate, each with a public pricing-page `source` and an `asOf` + date. A prominent banner says these are estimates to update when providers + change pricing. Some entries are `placeholder: true` stand-ins (the shipped + `gpt-5.5` model rate and the E2B desktop rate) — an operator MUST confirm them + before trusting the magnitude; the flag propagates into every estimate so a + stand-in is never mistaken for a live rate. An UNKNOWN model/desktop rate is + DECLARED ABSENT (`estimatedCostUsd: null` + a `reason`), never guessed. +- `humanish.actor-estimated-cost.v1` — `ActorTrace.estimatedCost`: one lane's + token-derived model cost, with `estimatedCostUsd` (or `null` + `reason` + `no_rate_for_model`/`no_token_usage`), `ratesAsOf`, `source`, `modelId`, + optional `placeholder`, and a `breakdown`. +- `humanish.run-cost-summary.v1` — `RunBundle.cost`: the sum of every lane's + `model-tokens` line PLUS one aggregate `desktop-minutes` line. + +The summary follows the SAME null discipline as the terminal cost ledger above. +`estimatedTotalUsd` sums ONLY the non-null `breakdown` lines and is `null` iff +EVERY line is null (never coerced to `0`); a present-but-unpriceable line stays +in `breakdown` with `estimatedCostUsd: null` + a `reason` (it records that we +tried and could not price it) and contributes nothing. `fullyEstimated` is +`false` when any applicable line is null (the total is then a lower bound); +`placeholder` is true when any contributing rate is a stand-in; `ratesAsOf` is +the MIN (oldest) `asOf` across contributing rates — an aggregate is only as fresh +as its stalest input, so MAX would overclaim freshness (each `breakdown` line +keeps its own true `asOf`). `desktopMinutes` is a HOST-SIDE +create→teardown span — an approximation of E2B's server-side billed lifetime, so +the desktop dollar figure is doubly an estimate. + +```yaml +schema: humanish.run-cost-summary.v1 +currency: usd +estimatedTotalUsd: 11.60167 # sum of KNOWN lines only; null iff every line null +ratesAsOf: "2026-08-01" +fullyEstimated: true # both breakdown lines are priced (no null line) +placeholder: true # a stand-in rate contributed +breakdown: + - { kind: model-tokens, laneId: lane-01, modelId: computer-use-preview, + estimatedCostUsd: 11.60, ratesAsOf: "2026-08-01", source: "openai.com/api/pricing" } + - { kind: desktop-minutes, estimatedCostUsd: 0.00167, ratesAsOf: "2026-08-01", + source: "…e2b.dev/pricing", placeholder: true } +tokenUsage: { input: 3843523, output: 5869, total: 3849392 } +desktopMinutes: 1 +note: "Estimated 11.60167 USD total…" +``` + +`verifyRun` asserts LABELING/provenance, never MAGNITUDE. Absence PASSES +(fail-open on display): a bundle with no cost, a null estimate, or a lane without +`estimatedCost` verifies fine. A CLAIMED number FAILS closed when it lacks its +`ratesAsOf` date or `source`, or when `estimatedTotalUsd` does not equal the +rounded sum of its non-null lines (a null line coerced to 0 is a mechanism +mismatch). A correctly-labeled huge estimate still passes. Cost is neither a +secret nor a share-blocker, so it never affects `shareSafety`. + +**Fail-closed spend cap.** `execution.caps.maxUsd` (the terminal lane's +`LabScenarioCaps` shape, consumed on the CUA route) aborts a session the moment +its running ESTIMATED spend crosses the cap — the runaway-retry guard. It is a +**PER-LANE** cap: enforced INSIDE each lane's loop independently, so an N-lane +fan-out can spend up to N × `maxUsd` before any lane aborts (the run bundle +warns with the true ~N × cap ceiling; a shared run-level budget is future work). +A lane that does real work THEN crosses its cap ends `budget_reached` (passed); a +zero-action runaway that crosses it ends `gave_up` (failed) — the cap classifies +its outcome honestly rather than greenlighting the runaway it exists to catch. +Absent = uncapped (the historical behavior); `maxUsd: 0` = no-spend. A cap on a +model `src/pricing.ts` cannot price is REFUSED at preflight +(`HUMANISH_CUA_LAB_UNPRICED_CAP`) before any sandbox rather than run uncapped — +an unenforceable cap is more dangerous than none. + ## Product-Adapter Extension Seam The terminal-product and browser/computer-use lanes let an adopter attach diff --git a/docs/goals/current.md b/docs/goals/current.md index 3f46d26..79ec51f 100644 --- a/docs/goals/current.md +++ b/docs/goals/current.md @@ -16,7 +16,7 @@ Humanish should be the open-source CLI that lets a maintainer ask: The answer should be observable, verifiable, public-safe, and easy to turn into actionable feedback. -## Current Program Truth (source `0.18.0`) +## Current Program Truth (source `0.19.0`) The package source and repository implementation in this tree agree on these points: diff --git a/docs/goals/proof-roadmap/README.md b/docs/goals/proof-roadmap/README.md index 6132954..aae7293 100644 --- a/docs/goals/proof-roadmap/README.md +++ b/docs/goals/proof-roadmap/README.md @@ -18,6 +18,15 @@ file records current implementation truth without rewriting that packet. schema implementation. Its adopter-need and maintainer-review gate is closed. - Public out-of-tree actor registration and conformance certification remain unbuilt. +- Cost tracking (ESTIMATED) shipped in `0.19.0`: a dated, operator-editable + pricing table (`src/pricing.ts`), a per-lane + run-level cost ESTIMATE on the + computer-use bundle (`humanish.run-cost-summary.v1` / + `humanish.actor-estimated-cost.v1`, additive so `humanish.run-bundle.v1` stays + v1), an `execution.caps.maxUsd` fail-closed abort for the CUA lane (default + uncapped; an unpriced cap is refused at preflight), and Observer/library + labeling ("estimated (rates as of ``)", never a bare charge). Every + dollar figure is an estimate; `verify` asserts its labeling, never its + magnitude. ## Proof state diff --git a/docs/ramp/README.md b/docs/ramp/README.md index a401c6c..db8cff0 100644 --- a/docs/ramp/README.md +++ b/docs/ramp/README.md @@ -2,7 +2,7 @@ Status: public-safe contributor and agent ramp. -Package/source version in this tree: `0.18.0` (2026-08-02). The containment boundary introduced in +Package/source version in this tree: `0.19.0` (2026-08-02). The containment boundary introduced in `0.15.1` remains in force: managed run and output paths bind to validated physical filesystem identities, and stored provider IDs are evidence, not cleanup authority. The bundled OSS meta-lab is dry-run only until diff --git a/package.json b/package.json index 83a57a5..8b1cee9 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "humanish", - "version": "0.18.0", + "version": "0.19.0", "description": "Open-source-safe CLI for persona simulation, observer review, and public-safe feedback drafts.", "author": "Daniel G Wilson ", "keywords": [ diff --git a/src/actor-contract.ts b/src/actor-contract.ts index 93af3bb..f5f4ca5 100644 --- a/src/actor-contract.ts +++ b/src/actor-contract.ts @@ -1,4 +1,5 @@ import type { CodexAppServerRunResult, CodexAppServerStatus, CodexAppServerTrace } from "./codex-app-server.js"; +import type { ActorEstimatedCost } from "./pricing.js"; import { redactText } from "./redaction.js"; // The provider-neutral evidence schema. Codex item/* events, Claude @@ -129,6 +130,15 @@ export interface ActorTrace { counts: Record; items: ActorTraceItem[]; tokenUsage?: ActorTokenUsage; + /** + * ADDITIVE + OPTIONAL token-derived cost ESTIMATE for this lane (humanish.actor-estimated-cost.v1). + * Distinct from `tokenUsage.costUsd`, which is RESERVED for a real provider-returned charge: a + * bare `costUsd` always means "the provider billed this", while `estimatedCost.estimatedCostUsd` + * is a rate-table multiply named honestly as an estimate (invariant 6). Absent on codex/scripted + * lanes and on every pre-existing bundle — its absence is tolerated by verify (fail-open on + * display). A `null` estimatedCostUsd is DECLARED ABSENT (unknown rate / no usage), never 0. + */ + estimatedCost?: ActorEstimatedCost; capabilities: ActorCapabilities; } diff --git a/src/computer-use-actor.ts b/src/computer-use-actor.ts index f64b2c1..805fd6e 100644 --- a/src/computer-use-actor.ts +++ b/src/computer-use-actor.ts @@ -75,6 +75,10 @@ export interface CuaActorSessionOptions { writeScreenshot?: (name: string, bytes: Buffer) => Promise; /** Deterministic harness-owned stop guards evaluated between model turns. */ stopWhen?: StopWhen; + /** FAIL-CLOSED spend cap (USD) threaded to the loop; absent = uncapped. See CuaLoopOptions.maxUsd. */ + maxUsd?: number; + /** Injected pure per-turn cost estimator paired with `maxUsd`. See CuaLoopOptions.estimateTurnCostUsd. */ + estimateTurnCostUsd?: (input: number, output: number) => number | null; } export async function runCuaActorSession(options: CuaActorSessionOptions): Promise { @@ -96,7 +100,9 @@ export async function runCuaActorSession(options: CuaActorSessionOptions): Promi ...(options.redactScreenshots === undefined ? {} : { redactScreenshots: options.redactScreenshots }), ...(options.scrubText === undefined ? {} : { scrubText: options.scrubText }), ...(options.writeScreenshot === undefined ? {} : { writeScreenshot: options.writeScreenshot }), - ...(options.stopWhen === undefined ? {} : { stopWhen: options.stopWhen }) + ...(options.stopWhen === undefined ? {} : { stopWhen: options.stopWhen }), + ...(options.maxUsd === undefined ? {} : { maxUsd: options.maxUsd }), + ...(options.estimateTurnCostUsd === undefined ? {} : { estimateTurnCostUsd: options.estimateTurnCostUsd }) }; return runComputerUseLoop(loopOptions); diff --git a/src/computer-use.ts b/src/computer-use.ts index c8c1c4c..653d246 100644 --- a/src/computer-use.ts +++ b/src/computer-use.ts @@ -215,6 +215,22 @@ export interface CuaLoopOptions { * wandering after the product already reached an app-visible endpoint. */ stopWhen?: StopWhen; + /** + * FAIL-CLOSED spend cap (USD). When set, the loop aborts (completionReason "budget_reached") + * the moment the running ESTIMATED spend crosses it, BEFORE the next provider turn — the + * runaway-retry-loop guard. Absent = uncapped (the historical CUA behavior). maxUsd: 0 means + * no-spend (any measurable estimate > 0 aborts). Enforcement needs a measurable estimate, so + * the lab refuses a cap on an unpriced model at PREFLIGHT rather than running uncapped. + */ + maxUsd?: number; + /** + * Injected PURE per-turn cost estimator (keeps the loop free of the operator rate table and + * makes the cap deterministic in tests). Given running (input, output) token totals, returns the + * estimated USD, or null when unpriceable. Only consulted when `maxUsd` is set. A null estimate + * mid-run cannot trip the cap — preflight already guaranteed a rate exists, so a null here is a + * vanished-rate harness condition, not a silent uncapped pass. + */ + estimateTurnCostUsd?: (input: number, output: number) => number | null; } export interface CuaLoopResult { @@ -432,7 +448,9 @@ export async function runComputerUseLoop(options: CuaLoopOptions): Promise text, writeScreenshot = async (name) => `screenshots/${name}`, - stopWhen + stopWhen, + maxUsd, + estimateTurnCostUsd } = options; const noProgressRecoverySteps = Math.min(Math.max(1, noProgressSteps - 1), 3); const idleRecoverySteps = Math.min(Math.max(1, idleSteps - 1), 3); @@ -599,6 +617,30 @@ export async function runComputerUseLoop(options: CuaLoopOptions): Promise maxUsd) { + if (materialActions > 0) { + completionReason = "budget_reached"; + reason = `estimated spend $${running} crossed execution.caps.maxUsd=$${maxUsd} after productive activity (${materialActions} material action(s), ${counts.turns} turn(s)); aborted fail-closed before the next model turn`; + } else { + completionReason = "gave_up"; + reason = `estimated spend $${running} crossed execution.caps.maxUsd=$${maxUsd} with no material progress; aborted fail-closed before the next model turn`; + } + break; + } + } if (turn.reasoning) { items.push({ id: nextId("reasoning"), diff --git a/src/concurrent-shared-world-lab.ts b/src/concurrent-shared-world-lab.ts index 2ae2f75..b5267f1 100644 --- a/src/concurrent-shared-world-lab.ts +++ b/src/concurrent-shared-world-lab.ts @@ -713,6 +713,7 @@ export async function runConcurrentSharedWorld(options: RunConcurrentSharedWorld redactScreenshots, scrubKnownValues, runSession, + now, hooks: cuaHooks, // Concurrent lanes are independent evidence seats: a requested-vs-verified screen // mismatch is recorded as separate facts + a warning instead of failing the lane's diff --git a/src/cua-actor-lab.ts b/src/cua-actor-lab.ts index 8e646cb..90da9bd 100644 --- a/src/cua-actor-lab.ts +++ b/src/cua-actor-lab.ts @@ -40,6 +40,7 @@ import { } from "./browser-evidence-hygiene.js"; import type { CuaActorSessionOptions } from "./computer-use-actor.js"; import type { CuaExecutor, CuaLoopResult, CuaProvider } from "./computer-use.js"; +import { DEFAULT_OPENAI_CU_MODEL } from "./openai-responses-cu.js"; import type { E2BDesktopLike } from "./e2b-desktop-executor.js"; import { createDesktopSandbox, @@ -114,9 +115,12 @@ import { type RunSimulationStatus, type RunStream, type RunProviderResource, + type RunCostLine, + type RunCostSummary, type RunSubjectProvenance, type RunSubjectStateStepRecord } from "./run.js"; +import { estimateActorCost, estimateDesktopCost, MODEL_RATES, round6 } from "./pricing.js"; export const CUA_ACTOR_LAB_SCHEMA = "humanish.cua-lab-result.v2"; @@ -259,6 +263,10 @@ export interface CuaActorLabHooks extends BrowserLabAdapterHooks { buildProvider?: (ctx: { config: LabConfig; actor: CuaActorDescriptor }) => Promise; env?: Record; renderObserverFn?: typeof renderObserver; + /** Injected clock (ms) for the host-side E2B desktop create->teardown span measurement that + * feeds the desktop-minute cost estimate. Defaults to Date.now; tests inject a frozen/stepped + * clock so the desktop-minute line is deterministic. */ + now?: () => number; /** Injected clock/sleep for the detached-step polling (tests only). */ detachedTimers?: DetachedTimers; /** @@ -390,6 +398,10 @@ export type CuaActorLabErrorCode = | "HUMANISH_CUA_LAB_FANOUT_INVALID" | "HUMANISH_CUA_LAB_RERUN_INVALID" | "HUMANISH_CUA_LAB_DEVICE_GEOMETRY" + // A fail-closed spend cap (execution.caps.maxUsd) was set but src/pricing.ts has no rate for the + // resolved model, so the cap could not be enforced. Refused at preflight (before any sandbox) + // rather than run uncapped — an unenforceable cap is more dangerous than none. + | "HUMANISH_CUA_LAB_UNPRICED_CAP" // watch --expose (tunnel-edge auth) validation + tunnel-startup failures surfaced by runCuaBackend // before or around the run. Carried on the CUA lab envelope so `watch --expose` refusals // render through the same formatter as any other CUA lab failure. @@ -910,6 +922,9 @@ export interface CuaLaneDeps { redactScreenshots: boolean; scrubKnownValues: (text: string) => string; runSession: (options: CuaActorSessionOptions) => Promise; + /** Injected clock (ms). Used to measure the host-side E2B desktop create->teardown span so the + * desktop-minute cost estimate is deterministic in tests. Defaults to Date.now. */ + now: () => number; hooks: CuaActorLabHooks; /** Lane-0 only: signal the pipeline gate after provisioning succeeds (true) or fails (false). */ signalProvisioned?: (ok: boolean) => void; @@ -929,6 +944,10 @@ export interface LaneRunOutcome { session?: CuaLoopResult; sessionError?: string; sandboxId?: string; + /** Host-side E2B desktop create->teardown span (ms). An APPROXIMATION of E2B's server-side + * billed lifetime (server-side kill-on-timeout can extend it) — so the derived dollar figure is + * doubly an estimate. Absent on the in-process route (no sandbox) and on dry-run. */ + desktopDurationMs?: number; killed: boolean; streamUrlPresent: boolean; screenshots: string[]; @@ -1644,6 +1663,11 @@ export async function runCuaLane(spec: CuaLaneSpec, deps: CuaLaneDeps): Promise< let sessionError: string | undefined; let failureCode: CuaActorLabErrorCode | undefined; let sandboxId: string | undefined; + // Host-side E2B desktop billed-span endpoints, measured via the injected clock. Captured right + // after create() succeeds and again in the finally after teardown resolves (both the killed and + // kept-for-debug paths), so the desktop-minute cost estimate reflects the honest lifetime. + let sandboxCreatedAtMs: number | undefined; + let sandboxTornDownAtMs: number | undefined; let killed = false; let streamUrl: string | undefined; let subjectCommit: string | undefined; @@ -1694,6 +1718,8 @@ export async function runCuaLane(spec: CuaLaneSpec, deps: CuaLaneDeps): Promise< lifecycle: { onTimeout: "kill" } }, config.execution?.desktop?.template); sandboxId = desktop.sandboxId; + // The billed span starts the instant the sandbox exists. + sandboxCreatedAtMs = deps.now(); if (deps.hooks.prepareDesktop) { await deps.hooks.prepareDesktop(desktop, { laneId: spec.laneId, laneIndex: spec.laneIndex, laneCount: deps.laneCount }); @@ -1817,6 +1843,12 @@ export async function runCuaLane(spec: CuaLaneSpec, deps: CuaLaneDeps): Promise< warnings.push(`Live desktop stream unavailable (run continues; evidence still captured): ${redactText(deps.scrubKnownValues(toErrorMessage(error)))}`); } + // The FAIL-CLOSED spend cap (execution.caps.maxUsd) is wired into the loop as maxUsd + an + // injected pure per-turn estimator keyed on the resolved model. Preflight already refused a + // cap on an unpriced model, so the estimate is measurable whenever a cap is in force. The + // model id here matches provider.version (openai-responses-cu resolves the default when unset). + const capModelId = config.actors[0]?.model ?? DEFAULT_OPENAI_CU_MODEL; + const maxUsd = config.execution?.caps?.maxUsd; const sessionOptions: CuaActorSessionOptions = { instructions: spec.instructions, persona: spec.persona, @@ -1825,6 +1857,13 @@ export async function runCuaLane(spec: CuaLaneSpec, deps: CuaLaneDeps): Promise< apiKey: deps.openaiApiKey, ...(config.actors[0]?.model ? { model: config.actors[0]!.model } : {}) }, + ...(maxUsd === undefined + ? {} + : { + maxUsd, + estimateTurnCostUsd: (input: number, output: number): number | null => + estimateActorCost({ input, output }, capModelId).estimatedCostUsd + }), desktop: desktop as unknown as E2BDesktopLike, ...(launchedBrowserFamily === "chromium" ? { @@ -1910,10 +1949,24 @@ export async function runCuaLane(spec: CuaLaneSpec, deps: CuaLaneDeps): Promise< } else { warnings.push("Installed @e2b/desktop SDK does not expose Sandbox.kill; server-side kill-on-timeout will reclaim the sandbox."); } + // Close the billed span for BOTH the killed and kept-for-debug paths (a kept sandbox is + // still billed until its server-side timeout, so the honest span ends here either way). + sandboxTornDownAtMs = deps.now(); } } + // Host-side approximation of the E2B desktop's billed lifetime; feeds the desktop-minute cost + // estimate. Never negative. + const desktopDurationMs = sandboxCreatedAtMs !== undefined && sandboxTornDownAtMs !== undefined + ? Math.max(0, sandboxTornDownAtMs - sandboxCreatedAtMs) + : undefined; + if (session) { + // Per-lane model-token cost ESTIMATE, attached to the trace before it is persisted (the model + // id is authoritative here — provider.version). Kept at the lab boundary so the pure loop + // never depends on the operator rate table. estimateActorCost declares absent (null) for an + // unknown rate / missing usage rather than guessing. + session.trace.estimatedCost = estimateActorCost(session.trace.tokenUsage, session.trace.ids.model); await writeContainedOutputFile(deps.artifactRoot, spec.traceArtifactPath, `${JSON.stringify(session.trace, null, 2)}\n`, "utf8"); if (session.trace.redaction.screenshots === "raw") { warnings.push("Screenshots are full-fidelity (raw) for local use — the bundle stays in gitignored .humanish and nothing scans these pixels; review them before sharing anywhere. Set policies.redactScreenshots: true to blur a share-as-is bundle."); @@ -1944,6 +1997,7 @@ export async function runCuaLane(spec: CuaLaneSpec, deps: CuaLaneDeps): Promise< ...(session ? { session } : {}), ...(sessionError === undefined ? {} : { sessionError }), ...(sandboxId === undefined ? {} : { sandboxId }), + ...(desktopDurationMs === undefined ? {} : { desktopDurationMs }), killed, streamUrlPresent: streamUrl !== undefined, screenshots, @@ -2470,6 +2524,22 @@ export async function runCuaActorLab(options: RunCuaActorLabOptions): Promise 1) { + aggregateWarnings.push( + `execution.caps.maxUsd ($${perLaneCapUsd}) is a PER-LANE cap; ${laneCount} lanes may spend up to ${laneCount} × $${perLaneCapUsd} (~$${round6(perLaneCapUsd * laneCount)} total) before any lane aborts. A shared run-level budget is future work.` + ); + } const aggregateSubject = ((): CuaSubjectProjection => { const first = laneSubjects[0]!; if (first.source !== "clone") { @@ -2962,6 +3043,9 @@ function buildSingleLaneBundle(args: { }), ...(args.localAppSubject || args.inProcessRoute ? { entryKind: "local-app" as const } : {}), ...(outcome?.session ? { traceArtifactPath: spec.traceArtifactPath } : {}), + ...(desktopSpanToMinutes(outcome?.desktopDurationMs) === undefined + ? {} + : { desktopMinutes: desktopSpanToMinutes(outcome?.desktopDurationMs)! }), phaseEvents: outcome?.phaseRecords ?? [] }); } @@ -3379,6 +3463,105 @@ function tailOf(log: string): string { * `stream.actor = session.trace` — the provider-neutral ActorTrace seam the Observer renders. * Exported for the bundle-builder tests. */ +/** + * Assemble the run-level cost ESTIMATE from each lane's persisted per-actor estimate + * (trace.estimatedCost, set at the lab boundary) plus the aggregated E2B desktop-minute span. + * Returns undefined (cost OMITTED) when nothing was priceable AND no sandbox ran — a pure dry-run + * or an in-process lane (no trace.estimatedCost, no desktop) stays byte-stable with no cost block. + * The null-discipline mirrors the terminal ledger: a present-but-unpriceable line is null + a + * reason and contributes NOTHING to estimatedTotalUsd (never coerced to 0); an all-null summary + * has a null total. Every non-null figure carries its ratesAsOf date + source (invariant 6). + */ +export function buildCuaCostSummary(args: { + lanes: Array<{ laneId?: string; trace: ActorTrace }>; + desktopMinutes: number | undefined; +}): RunCostSummary | undefined { + const breakdown: RunCostLine[] = []; + let sumInput = 0; + let sumOutput = 0; + + for (const lane of args.lanes) { + const usage = lane.trace.tokenUsage; + if (usage) { + sumInput += usage.input ?? 0; + sumOutput += usage.output ?? 0; + } + const est = lane.trace.estimatedCost; + if (!est) { + continue; + } + breakdown.push({ + kind: "model-tokens", + ...(lane.laneId === undefined ? {} : { laneId: lane.laneId }), + ...(est.modelId === undefined ? {} : { modelId: est.modelId }), + estimatedCostUsd: est.estimatedCostUsd, + ...(est.reason === undefined ? {} : { reason: est.reason }), + ratesAsOf: est.ratesAsOf, + ...(est.source === undefined ? {} : { source: est.source }), + ...(est.placeholder ? { placeholder: true } : {}) + }); + } + + if (args.desktopMinutes !== undefined) { + const desktop = estimateDesktopCost(args.desktopMinutes); + breakdown.push({ + kind: "desktop-minutes", + estimatedCostUsd: desktop.estimatedCostUsd, + ...(desktop.reason === undefined ? {} : { reason: desktop.reason }), + ratesAsOf: desktop.ratesAsOf, + ...(desktop.source === undefined ? {} : { source: desktop.source }), + ...(desktop.placeholder ? { placeholder: true } : {}) + }); + } + + if (breakdown.length === 0) { + return undefined; + } + + let knownSum = 0; + let anyKnown = false; + let anyNull = false; + let placeholder = false; + // Aggregate freshness is CONSERVATIVE: an aggregate estimate is only as current as its OLDEST + // contributing rate, so ratesAsOf takes the MIN (oldest) asOf — MAX would overclaim freshness the + // moment operator-edited rates in src/pricing.ts diverge. Each breakdown line keeps its own true asOf. + let minRatesAsOf: string | null = null; + for (const line of breakdown) { + if (line.estimatedCostUsd === null) { + anyNull = true; + continue; + } + anyKnown = true; + knownSum += line.estimatedCostUsd; + if (line.placeholder) placeholder = true; + if (line.ratesAsOf !== null && (minRatesAsOf === null || line.ratesAsOf < minRatesAsOf)) { + minRatesAsOf = line.ratesAsOf; + } + } + const estimatedTotalUsd = anyKnown ? round6(knownSum) : null; + const note = estimatedTotalUsd === null + ? `No priced spend lines this run — every cost line is DECLARED ABSENT (unknown rate / no usage / no duration); nothing is guessed. Add a rate to src/pricing.ts to estimate this model.` + : `Estimated ${estimatedTotalUsd} USD total${anyNull ? " (LOWER BOUND — some lines unmeasured/unpriced)" : ""}${placeholder ? "; includes PLACEHOLDER rate(s) — confirm before trusting the magnitude" : ""}. Every figure is an ESTIMATE (rates as of ${minRatesAsOf} — the OLDEST contributing rate, since an aggregate is only as fresh as its stalest input), a rate-table multiply, NOT an authoritative provider charge.`; + + return { + schema: "humanish.run-cost-summary.v1", + currency: "usd", + estimatedTotalUsd, + ratesAsOf: minRatesAsOf, + fullyEstimated: !anyNull, + placeholder, + breakdown, + tokenUsage: { input: sumInput, output: sumOutput, total: sumInput + sumOutput }, + desktopMinutes: args.desktopMinutes ?? null, + note + }; +} + +// Convert a host-side desktop span (ms) into billed minutes, or undefined when no sandbox ran. +function desktopSpanToMinutes(desktopDurationMs: number | undefined): number | undefined { + return desktopDurationMs === undefined ? undefined : desktopDurationMs / 60_000; +} + export function buildCuaBundle(args: { actorId: string; appUrl: string; @@ -3431,8 +3614,16 @@ export function buildCuaBundle(args: { /** Completed subject-phase records (clone/upload/extract/install/build/ready/state groups) * to fold into bundle.events, so run.json carries real phase timing after the fact. */ phaseEvents?: SubjectPhaseEvent[]; + /** Host-side E2B desktop billed span for this lane, in minutes (from LaneRunOutcome + * desktopDurationMs). Absent when no sandbox ran (in-process/dry-run) → no desktop cost line. */ + desktopMinutes?: number; }): RunBundle { const publicAppUrl = publicSafeAppUrlLabel(args.appUrl); + // Run-level cost ESTIMATE (advisory; omitted when nothing was priced and no sandbox ran). + const cost = buildCuaCostSummary({ + lanes: args.session ? [{ ...(args.laneId === undefined ? {} : { laneId: args.laneId }), trace: args.session.trace }] : [], + desktopMinutes: args.desktopMinutes + }); const status: RunSimulationStatus = args.inProgress === true ? "running" : args.session @@ -3712,7 +3903,8 @@ export function buildCuaBundle(args: { // honest on app-url bundles too — the caller minted the URL, its state is the caller's. // CuaSubjectProvenanceArg's two variants (clone, local-tree) are already RunSubjectProvenance- // shaped, so no reconstruction is needed beyond the app-url fallback. - subject: args.subjectProvenance ?? { source: "app-url", state: { provenance: "undeclared" } } + subject: args.subjectProvenance ?? { source: "app-url", state: { provenance: "undeclared" } }, + ...(cost === undefined ? {} : { cost }) }; } @@ -4113,6 +4305,18 @@ export function buildCuaFanoutBundle(args: { laneId: outcome.spec.laneId })); + // Run-level cost ESTIMATE: one model-token line per lane that ran a session (from its persisted + // trace.estimatedCost) + one aggregate desktop-minutes line summing each lane's OWN sandbox span + // (per-lane worlds => no shared provisioning to double-count). Omitted on a pure dry-run. + const costLanes = specs + .map((spec, index) => ({ laneId: spec.laneId, outcome: outcomes?.[index] })) + .filter((entry): entry is { laneId: string; outcome: LaneRunOutcome } => entry.outcome?.session !== undefined) + .map((entry) => ({ laneId: entry.laneId, trace: entry.outcome.session!.trace })); + const desktopMinutesTotal = (outcomes ?? []).some((outcome) => outcome.desktopDurationMs !== undefined) + ? (outcomes ?? []).reduce((sum, outcome) => sum + (outcome.desktopDurationMs ?? 0), 0) / 60_000 + : undefined; + const cost = buildCuaCostSummary({ lanes: costLanes, desktopMinutes: desktopMinutesTotal }); + return { schema: RUN_BUNDLE_SCHEMA, runId: args.runId, @@ -4169,7 +4373,8 @@ export function buildCuaFanoutBundle(args: { ? {} : { desktopBrowser: { requested: configuredBrowser, ...(unanimousResolvedBrowser === undefined ? {} : { resolved: unanimousResolvedBrowser }) } }), ...(providerResources.length === 0 ? {} : { providerResources }), - subject: args.aggregateSubject + subject: args.aggregateSubject, + ...(cost === undefined ? {} : { cost }) }; } diff --git a/src/index.ts b/src/index.ts index 112e79d..9480a71 100644 --- a/src/index.ts +++ b/src/index.ts @@ -63,6 +63,19 @@ export type { BrowserLabScoringContext } from "./adapter-extension.js"; export type { RedactionHooks } from "./redaction.js"; +export { + DESKTOP_RATE, + MODEL_RATES, + PRICING_SCHEMA, + estimateActorCost, + estimateDesktopCost +} from "./pricing.js"; +export type { + ActorEstimatedCost, + DesktopCostEstimate, + DesktopRate, + ModelRate +} from "./pricing.js"; export { normalizeCliArgv } from "./argv.js"; export { CODEX_APP_SERVER_UI_SCHEMA, @@ -137,6 +150,8 @@ export type { RunAdapterScore, RunAttributionClass, RunBundle, + RunCostLine, + RunCostSummary, RunDesktopGeometry, RunEvent, RunFeedbackCandidate, diff --git a/src/lab-config.ts b/src/lab-config.ts index a9b0d83..992c050 100644 --- a/src/lab-config.ts +++ b/src/lab-config.ts @@ -445,6 +445,18 @@ export interface LabExecution { /** FORWARD-DECLARED. */ concurrency?: number; desktop?: LabExecutionDesktop; + /** + * Blast-radius budget for the computer-use lane. CONSUMED on the CUA route: `caps.maxUsd`, when + * set, is a FAIL-CLOSED abort — the session stops the moment its running ESTIMATED spend crosses + * it (the runaway-retry guard), and a cap on a model src/pricing.ts cannot price is REFUSED at + * preflight rather than run uncapped. It is a PER-LANE cap: enforced inside each lane's loop, so + * an N-lane fan-out can spend up to N × maxUsd before any lane aborts (the run warns with the + * true ~N × cap ceiling; a shared run-level budget is future work). Absent = UNCAPPED (the + * historical CUA behavior); maxUsd: 0 = no-spend (any measurable estimate > 0 aborts). Inert + * (warned) on non-CUA routes. Reuses the same LabScenarioCaps shape as the terminal lane's + * `scenario.caps` (not a fork). + */ + caps?: LabScenarioCaps; /** `terminal-product` route: the terminal transport + stdin posture. Consumed on that route. */ terminal?: LabExecutionTerminal; /** `terminal-product` route: the in-sandbox agent's runtime-auth channel. Live runs inject the @@ -1234,6 +1246,9 @@ function forwardDeclaredWarnings(config: LabConfig): string[] { // execution.concurrency is CONSUMED on the cua route (it bounds in-flight fan-out lanes); // inert (warned) everywhere else. if (config.execution?.concurrency !== undefined && !routesToCua) inert.push("execution.concurrency"); + // execution.caps is CONSUMED on the cua route (maxUsd is the fail-closed spend abort); inert + // (warned) everywhere else so a misplaced budget field is never trusted to cap a route it cannot. + if (config.execution?.caps && !routesToCua) inert.push("execution.caps (the fail-closed spend abort is a computer-use route capability; needs a computer-use actor on e2b-desktop)"); // terminal-product consumes subject.product, scenario.caps, execution.{terminal,runtimeAuth}: // dry-run records the contract; live execution enforces caps and command-scoped auth. On every // OTHER route they are inert and must warn so a @@ -2041,6 +2056,13 @@ function parseExecution(raw: unknown): { ok: true; value: LabExecution | undefin return desktopResult; } if (desktopResult.value) execution.desktop = desktopResult.value; + // Reuse the terminal lane's caps parser (same shape, not a fork) — a malformed budget is a hard + // error, never silently dropped (a cap that silently does nothing would be a safety lie). + const capsResult = parseCaps(raw.caps); + if (!capsResult.ok) { + return capsResult; + } + if (capsResult.value) execution.caps = capsResult.value; const terminalResult = parseTerminal(raw.terminal); if (!terminalResult.ok) { return terminalResult; diff --git a/src/observer-assets.ts b/src/observer-assets.ts index 0c7c4e3..e8698f5 100644 --- a/src/observer-assets.ts +++ b/src/observer-assets.ts @@ -1223,6 +1223,30 @@ export function observerClientJs(): string { function laneEvents(s) { return s.timeline || []; } function laneArtifacts(s) { return s.artifacts || []; } + // ---------------------------------------------------------------- cost (ALWAYS labeled estimated) + function fmtCostUsd(v) { + if (typeof v !== "number" || !isFinite(v)) return null; + return v >= 0.01 ? v.toFixed(2) : String(v); + } + // A single honest label for any cost figure. A number ALWAYS reads "~$X estimated (rates as of + // )" (never a bare "$X"); a placeholder rate is tagged; a null is "not estimated ()". + function costText(usd, ratesAsOf, placeholder, reason) { + if (usd == null) return "not estimated (" + (reason || "unmeasured") + ")"; + var s = "~$" + (fmtCostUsd(usd) || "0") + " estimated (rates as of " + (ratesAsOf || "unknown") + ")"; + if (placeholder) s += " · placeholder rate"; + return s; + } + function runCostText() { + var c = currentData.cost; + if (!c) return ""; + return costText(c.estimatedTotalUsd, c.ratesAsOf, c.placeholder, "no priced lines"); + } + function laneCostText(s) { + var est = s && s.actor && s.actor.estimatedCost; + if (!est) return ""; + return costText(est.estimatedCostUsd, est.ratesAsOf, est.placeholder, est.reason); + } + function geometrySize(value) { var width = value && Number(value.width); var height = value && Number(value.height); @@ -2192,7 +2216,9 @@ export function observerClientJs(): string { + '
Persona
' + esc((run.persona && run.persona.name) || "Synthetic persona") + ' is attempting this lane as a realistic setup operator.
' + '
Scenario
' + esc((run.scenario && run.scenario.goal) || laneSummary(s)) + '
' + '
' + pip(s.status, live) + '' + (live ? "Now" : "Last step") + '
' - + '
' + esc(laneStep(s)) + '
' + buildMeaningfulUseCard(s) + '' + + '
' + esc(laneStep(s)) + '
' + + (laneCostText(s) ? '
Est. cost
' + esc(laneCostText(s)) + '
' : "") + + buildMeaningfulUseCard(s) + '' + '
' + tabs.map(function (t) { return ''; }).join("") + '
' @@ -2231,6 +2257,7 @@ export function observerClientJs(): string { + '' + '' + counts.map(function (c) { return '' + c.count + '' + c.label + ''; }).join("") + '' + '' + esc("mode: " + (run.mode || "unknown") + " · " + (run.runId || "")) + '' + + (currentData.cost ? '' + esc("cost: " + runCostText()) + '' : "") + '
' + ''; + + '' + esc(meta) + ''; }).join(""); return '