Context
Desktop agent sessions now record provider-reported usage buckets (input, output, reasoning, cache_read, cache_write) and derive display cost from per-turn { model, usage }. This surfaced that cache accounting is wired, but the main OpenRouter/Claude path does not appear to be actively using prompt caching.
Current code facts:
packages/grida-ai-agent/src/providers/byok.ts sets includeUsage: true for OpenRouter/OpenAI-compatible streams, so usage chunks are requested.
packages/grida-ai-agent/src/runtime/run-agent.ts maps AI SDK cachedInputTokens into persisted MessageUsage.cache_read.
packages/grida-ai-agent/src/session/cost.ts prices cache_read / cache_write with the model catalog rates.
packages/grida-ai-agent/src/agent/index.ts sets Anthropic thinking options, but does not configure prompt-cache breakpoints.
- We currently do not pass OpenRouter
session_id / x-session-id for sticky routing.
- We currently do not pass OpenAI
prompt_cache_key.
Observed locally on existing large sessions: cache_read=0 and cache_write=0, so the tracking path is present but no cache benefit is showing up.
Why this matters
Agent sessions repeatedly send large stable prefixes: tool definitions, system prompt, project instructions, skill index, and accumulated conversation history. On supported providers this should reduce input cost and latency. Without explicit configuration, Anthropic via OpenRouter may miss most of that benefit.
Research notes
- OpenAI prompt caching is automatic for supported models once prompts are eligible, but hit rates can be improved with stable prefixes / cache keys.
- OpenRouter documents prompt caching support, and for Anthropic recommends
cache_control plus sticky routing; session_id can be supplied as a top-level field or x-session-id header.
- Anthropic supports automatic and explicit cache breakpoints, with provider/model minimum token thresholds.
Proposed follow-up
- Add provider-specific prompt-cache configuration for OpenRouter + Anthropic agent runs. Candidate: top-level
cache_control: { type: "ephemeral" } when the selected model/provider supports it.
- Pass a stable OpenRouter
session_id derived from our sessionId so requests stay on a sticky provider endpoint.
- Consider OpenAI-compatible
prompt_cache_key for direct OpenAI/Gateway paths where supported.
- Keep the prompt prefix stable: system instructions, tool definitions, skill index, and project instructions before variable per-turn content.
- Add a live/manual verification path that checks
chat_sessions.cache_read > 0 after repeated large turns on a supported model.
- Ensure cost display continues to derive from
{ model, usage }, not persisted calculated cost.
Acceptance criteria
- Repeated turns in a large supported OpenRouter/Claude desktop agent session show nonzero
cache_read.
- The context/cost popover reflects cheaper cached-input pricing via derived session cost.
- Unsupported endpoints/models degrade cleanly with zero cache fields and no request errors.
- Tests or smoke docs cover the provider-specific request options enough to prevent accidental removal.
Context
Desktop agent sessions now record provider-reported usage buckets (
input,output,reasoning,cache_read,cache_write) and derive display cost from per-turn{ model, usage }. This surfaced that cache accounting is wired, but the main OpenRouter/Claude path does not appear to be actively using prompt caching.Current code facts:
packages/grida-ai-agent/src/providers/byok.tssetsincludeUsage: truefor OpenRouter/OpenAI-compatible streams, so usage chunks are requested.packages/grida-ai-agent/src/runtime/run-agent.tsmaps AI SDKcachedInputTokensinto persistedMessageUsage.cache_read.packages/grida-ai-agent/src/session/cost.tspricescache_read/cache_writewith the model catalog rates.packages/grida-ai-agent/src/agent/index.tssets Anthropic thinking options, but does not configure prompt-cache breakpoints.session_id/x-session-idfor sticky routing.prompt_cache_key.Observed locally on existing large sessions:
cache_read=0andcache_write=0, so the tracking path is present but no cache benefit is showing up.Why this matters
Agent sessions repeatedly send large stable prefixes: tool definitions, system prompt, project instructions, skill index, and accumulated conversation history. On supported providers this should reduce input cost and latency. Without explicit configuration, Anthropic via OpenRouter may miss most of that benefit.
Research notes
cache_controlplus sticky routing;session_idcan be supplied as a top-level field orx-session-idheader.Proposed follow-up
cache_control: { type: "ephemeral" }when the selected model/provider supports it.session_idderived from oursessionIdso requests stay on a sticky provider endpoint.prompt_cache_keyfor direct OpenAI/Gateway paths where supported.chat_sessions.cache_read > 0after repeated large turns on a supported model.{ model, usage }, not persisted calculated cost.Acceptance criteria
cache_read.