Upstream caching for OpenAI Responses API and Anthropic Messages API

[astsiapanay]

Upstream caching for OpenAI Responses API and Anthropic Messages API

Context

DIAL Core already implements context caching end-to-end for OpenAI chat completions
(83de30f9, plus the cacheRead/cacheWrite pricing follow-up in 3916d3e4). Core computes
rolling SHA-1 prefix hashes over the request body, stores hash -> (upstream endpoint, upstream id, prefix path, extra metadata) in Redis with a TTL, and pins subsequent requests carrying the same
prefix to the same upstream so the provider's own prompt cache actually hits. Policy is chosen with
X-DIAL-CACHE-POLICY, and adapters drive cache creation via the X-DIAL-CACHE-BREAKPOINT-PATH /
-EXPIRE-AT / -EXTRA-METADATA response headers.

Everything below the hash layer is already shape-agnostic — the balancer, Redis service, retry/pinning
logic, header plumbing, token accounting and pricing all work for any API. What is missing is the
hash layer itself for the two newer interfaces:

• ResponsesApiRequest.buildMessageCacheKeys() / buildToolCacheKeys() throw
  UnsupportedOperationException (ResponsesApiRequest.java:49-56).
• MessagesApiRequest likewise (MessagesApiRequest.java:49-56).
• BuildUpstreamCacheFn is registered only in DeploymentPostController
  (line 72), so
  context.getCacheBreakpointContext() is always null for the other two controllers.
• The prefix-path grammar is chat-shaped: ^prefix\.body\.(tools|messages)$
  (UpstreamCacheService.java:36).

Outcome: Anthropic Messages and OpenAI Responses requests get the same cache-aware upstream pinning
that chat completions has today, so multi-turn conversations on those interfaces stop paying full
input-token price on every turn.

This work also fixes a latent defect in the existing implementation. The digest is currently
per-node, not cumulative. MessageDigest is created inside buildCacheKeys(name)
(ChatCompletionRequest.java:81),
which is called once per node — so messages[3]'s hash does not include tools. Providers cache the
whole rendered prefix (Anthropic documents tools → system → messages), so a request with the same
messages but a different system prompt currently reuses the same Redis entry and the same
extra_metadata. That is wrong for providers that return an explicit cache id. Fixed for all three
APIs.

Nothing is needed for pricing or usage: MessagesTokenUsageParser already maps
cache_read_input_tokens → cachedTokens and cache_creation_input_tokens → cacheWriteTokens,
Responses rides TokenUsage's @JsonAlias({"input_tokens_details","prompt_tokens_details"}), and
ModelCostCalculator already prices cacheRead/cacheWrite.

Decisions

Area	Decision
Breakpoint source	Anthropic: client's native cache_control blocks (explicit) and auto-caching. Responses: auto-caching only — OpenAI prompt caching has no breakpoint concept. No new DIAL field; both bodies stay pass-through.
Granularity	Block-level for Anthropic: messages[i].content[j].
Digest	One rolling MessageDigest across the entire node order, for all three APIs.
Node order	Hardcoded per API. No new config: fieldsHashingOrder stays chat-completions-only, and the two new interfaces use built-in orders fixed by their wire formats.
Anthropic nodes	tools → system → messages (with .content[j]).
Responses nodes	tools → instructions → input.
cache_control	Excluded from the digest (skipped during iteration, never removed from the tree).
Candidate cap	None — every element/block is a candidate under auto-caching, as today.
Scalars	A scalar node behaves as a one-element array: system:"x" → system[0], string content → content[0], instructions → instructions[0], string input → input[0].
Feature flags	Stay global on Features (cacheSupported / autoCachingSupported). No per-interface flags.
Path notation	Extend the existing grammar with a nesting segment: prefix.body.messages[1].content[2]. A strict superset of today's form — no legacy parsing, no canonicalization, no deprecation window.
count_tokens	Excluded from cache routing.
Contract	Single List<CacheKey> buildCacheKeys(List<String> nodeOrder); CacheKey gains path.

Three rationale notes worth keeping:

• Why the new APIs don't read fieldsHashingOrder. It has a non-null default of
  ["prefix.body.tools", "prefix.body.messages"] (Model.java:19-20).
  Letting Anthropic fall back to it would silently drop system from every hash. It stays a
  chat-completions-only setting; the new interfaces use built-in orders.
• Why no per-interface config knob. The setting exists for the Llama-3.2 case, where tools are
  embedded in the user prompt — a chat-shaped problem. Anthropic's wire format fixes
  tools→system→messages and OpenAI fixes instructions→input, so the knob would only matter for a
  backend that speaks one of those wire formats while rendering prompts in a different order. Nobody
  has reported that, and adding the override later is purely additive.
• Why not JSONPath for paths. Core never evaluates the path against the body — it formats the
  string, keys prefixToHash with it, and matches whatever the adapter returns by exact string
  equality (UpstreamRoute.java:158).
  Only adapters interpret it. A JSONPath library would therefore help nobody on the Core side, while
  its dot-vs-bracket spellings are string-unequal under exact matching. Extending the existing grammar
  keeps every deployed adapter and existing model config working untouched.

Implementation

1. Node order

• config/.../InterfaceType.java: give each constant its built-in hashing order, since the order is a
  property of the wire format:
  • OPENAI_CHAT_COMPLETIONS → prefix.body.tools, prefix.body.messages
  • ANTHROPIC_MESSAGES → prefix.body.tools, prefix.body.system, prefix.body.messages
  • OPENAI_RESPONSES → prefix.body.tools, prefix.body.instructions, prefix.body.input
• config/.../Model.java: add resolveFieldsHashingOrder(InterfaceType) returning
  fieldsHashingOrder for OPENAI_CHAT_COMPLETIONS and the interface's built-in order otherwise.
  No new config field anywhere.

2. Path grammar

Widen UpstreamCacheService's PREFIX_PATH into a small parser/formatter (new class, e.g.
server/.../data/cache/CachePrefixPath.java) handling two related forms:

• Node designator, as used in fieldsHashingOrder config and the built-in orders:
  prefix.body.<node> where node is one of tools, messages, system, input, instructions.
• Concrete path, as emitted and echoed: prefix.body.<node>[i] with an optional .content[j]
  segment.

Today's prefix.body.tools and prefix.body.messages[3] remain valid verbatim, so no compat layer,
dual-form parsing, or deprecation window is needed. The only additions adapter authors must learn are
the three new node names and the .content[j] segment.

3. Cache-key contract

• server/.../function/request/CacheKey.java: record CacheKey(String path, String hash, boolean hasBreakpoint).
• server/.../function/request/RequestObject.java: drop buildMessageCacheKeys() /
  buildToolCacheKeys(); add default List<CacheKey> buildCacheKeys(List<String> nodeOrder) { return List.of(); }.
  The default removes both UnsupportedOperationException stubs and means future request shapes need
  no boilerplate.
• New shared helper (e.g. server/.../function/request/CacheKeyBuilder.java) owning the rolling
  digest, the JsonUtil.sort canonicalization, hex encoding (currently duplicated in
  ChatCompletionRequest.toString), scalar normalization, and path formatting. The three impls
  supply only: node order matching, which sub-fields to skip, and whether an element carries a
  breakpoint. Keeps the three implementations thin and the hashing rules in one place.

Do not mutate the tree. Skip cache_control and custom_fields during iteration rather than
removing them — Anthropic bodies are forwarded verbatim. Note JsonUtil.sort already rewrites nested
arrays in place (JsonUtil.java:114-128);
that only reorders object keys so it is semantically inert, but do not add to it.

4. The three request implementations

• ChatCompletionRequest: same field rules as today (skip custom_fields; from custom_content
  include only attachments), now driven by the shared builder with one rolling digest and pathed
  keys. Breakpoint flag stays custom_fields.cache_breakpoint. Emitted paths are unchanged from
  today.
• MessagesApiRequest: implement tools[i], system (string → system[0]; array → system[i]),
  and messages[i].content[j] (string content → content[0]). The message envelope fields
  (role, …) feed the digest before the blocks, so a role change changes the hash. Breakpoint flag
  = the block carries cache_control; cache_control itself is excluded from the digest so a client
  moving its marker forward each turn (the Anthropic cookbook pattern) does not invalidate earlier
  prefixes.
• ResponsesApiRequest: implement tools[i], instructions[0], input[i] (string input →
  input[0]). hasBreakpoint is always false — candidates come solely from
  autoCachingSupported. input last guarantees monotonicity: appending a turn never perturbs an
  earlier prefix.

5. Service and wiring

• UpstreamCacheService.buildCacheBreakpointContext(...): take the InterfaceType, resolve the node
  order via Model.resolveFieldsHashingOrder, make a single request.buildCacheKeys(order) call, and
  build breakpoints / prefixToHash from the returned pathed keys. The per-node dispatch and the
  "messages".equals(nodeName) branch go away. getCacheEntry (longest-prefix-first, RBatch of 64)
  and updateEntry are unchanged.
• BuildUpstreamCacheFn: accept an InterfaceType at construction and pass it through. The global
  cacheSupported || autoCachingSupported gate is unchanged.
• Register the fn, with the matching interface type, in:
  • DeploymentPostController (existing entry, now parameterized OPENAI_CHAT_COMPLETIONS)
  • ResponsesController (chain at lines 64-69) — OPENAI_RESPONSES
  • MessagesController only — not the shared MessagesBaseController chain, so count_tokens
    is excluded. The base assigns a final field in its constructor
    (MessagesBaseController.java:58-66),
    so convert it to a protected overridable builder method and have MessagesController append the fn.

Notable behavior changes to call out in the changelog

• Existing chat-completion prefix hashes change, because the digest becomes cumulative across nodes.
  Redis entries carry a 10-minute default TTL, so the effect is a short window of cache misses and
  extra cache writes — no wrong answers. Emitted path strings for chat completions are unchanged.
• X-DIAL-CACHE-POLICY now takes effect on /openai/v1/responses and /anthropic/v1/messages.
  CachePolicy.fromString already 400s on an unknown value, so those endpoints inherit that.
• Auto-caching at block granularity is uncapped by decision: a cold or expired cache on a long
  conversation costs a full candidate scan (batched 64 per Redis round trip) plus one prefixToHash
  entry per block.

Docs and schemas

• docs/dynamic-settings/models.md — note at fieldsHashingOrder (line 55) that it applies to chat
  completions only, and document the built-in orders used by the two new interfaces.
• docs/open_api_core.yaml — the X-DIAL-CACHE-POLICY parameter (576-590, 700-708) now applies to
  the two new operations.
• openapi-generator/src/main/resources/schemas/ — document that Anthropic uses native
  cache_control rather than CacheBreakpoint, and that Responses is auto-caching only.
• Adapter-facing contract: the three new node names and the .content[j] segment. Purely additive —
  existing chat-completions adapters need no changes.

Verification

Unit and integration:

./gradlew :server:test --tests "com.epam.aidial.core.server.service.UpstreamCacheServiceTest" --tests "com.epam.aidial.core.server.upstream.*" --tests "com.epam.aidial.core.server.function.BuildUpstreamCacheFnTest"

Extend the existing suites rather than adding parallel ones:

• UpstreamCacheServiceTest — cumulative-across-nodes hashing (changing system must change every
  later messages hash); order resolution per interface, including that Anthropic uses its built-in
  order and never inherits Model.fieldsHashingOrder even when that is explicitly configured;
  parsing of node designators and of nested concrete paths; scalar-normalized paths; existing
  prefix.body.tools / prefix.body.messages[3] inputs still resolving.
• New request-impl tests for MessagesApiRequest (string vs array system and content,
  cache_control excluded from the digest, marker moving forward across turns leaves earlier hashes
  stable, role change alters the hash) and ResponsesApiRequest (instructions[0], string input,
  hasBreakpoint always false, appending input leaves earlier prefixes stable).
• UpstreamRouteTest — succeed(...) writing an entry from a nested messages[i].content[j]
  breakpoint header.
• Controller tests asserting count_tokens produces no CacheBreakpointContext while
  POST /anthropic/v1/messages and POST /openai/v1/responses do.

End-to-end, using the existing integration harness (embedded Redis + OkHttp MockWebServer): two
upstreams, a model with autoCachingSupported, a mock adapter echoing
X-DIAL-CACHE-BREAKPOINT-PATH + X-DIAL-CACHE-EXPIRE-AT on the first call. Assert turn 2 lands on
the same upstream as turn 1, and that under X-DIAL-CACHE-POLICY: cache-priority a 429 retries the
same upstream while availability-priority moves on.

./gradlew :server:test
./gradlew checkstyleMain checkstyleTest

Note: full-suite runs in this repo are flaky in ways unrelated to a change under test — re-run or
isolate a failing class before attributing it.