Skip to content

Enable and verify prompt caching for desktop agent runs #966

Description

@softmarshmallow

Context

Desktop agent sessions now record provider-reported usage buckets (input, output, reasoning, cache_read, cache_write) and derive display cost from per-turn { model, usage }. This surfaced that cache accounting is wired, but the main OpenRouter/Claude path does not appear to be actively using prompt caching.

Current code facts:

  • packages/grida-ai-agent/src/providers/byok.ts sets includeUsage: true for OpenRouter/OpenAI-compatible streams, so usage chunks are requested.
  • packages/grida-ai-agent/src/runtime/run-agent.ts maps AI SDK cachedInputTokens into persisted MessageUsage.cache_read.
  • packages/grida-ai-agent/src/session/cost.ts prices cache_read / cache_write with the model catalog rates.
  • packages/grida-ai-agent/src/agent/index.ts sets Anthropic thinking options, but does not configure prompt-cache breakpoints.
  • We currently do not pass OpenRouter session_id / x-session-id for sticky routing.
  • We currently do not pass OpenAI prompt_cache_key.

Observed locally on existing large sessions: cache_read=0 and cache_write=0, so the tracking path is present but no cache benefit is showing up.

Why this matters

Agent sessions repeatedly send large stable prefixes: tool definitions, system prompt, project instructions, skill index, and accumulated conversation history. On supported providers this should reduce input cost and latency. Without explicit configuration, Anthropic via OpenRouter may miss most of that benefit.

Research notes

  • OpenAI prompt caching is automatic for supported models once prompts are eligible, but hit rates can be improved with stable prefixes / cache keys.
  • OpenRouter documents prompt caching support, and for Anthropic recommends cache_control plus sticky routing; session_id can be supplied as a top-level field or x-session-id header.
  • Anthropic supports automatic and explicit cache breakpoints, with provider/model minimum token thresholds.

Proposed follow-up

  1. Add provider-specific prompt-cache configuration for OpenRouter + Anthropic agent runs. Candidate: top-level cache_control: { type: "ephemeral" } when the selected model/provider supports it.
  2. Pass a stable OpenRouter session_id derived from our sessionId so requests stay on a sticky provider endpoint.
  3. Consider OpenAI-compatible prompt_cache_key for direct OpenAI/Gateway paths where supported.
  4. Keep the prompt prefix stable: system instructions, tool definitions, skill index, and project instructions before variable per-turn content.
  5. Add a live/manual verification path that checks chat_sessions.cache_read > 0 after repeated large turns on a supported model.
  6. Ensure cost display continues to derive from { model, usage }, not persisted calculated cost.

Acceptance criteria

  • Repeated turns in a large supported OpenRouter/Claude desktop agent session show nonzero cache_read.
  • The context/cost popover reflects cheaper cached-input pricing via derived session cost.
  • Unsupported endpoints/models degrade cleanly with zero cache fields and no request errors.
  • Tests or smoke docs cover the provider-specific request options enough to prevent accidental removal.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions