From 06540936d20c88132d08bc67d50a00827f5d6076 Mon Sep 17 00:00:00 2001 From: Kyle Corbitt Date: Thu, 26 Mar 2026 14:16:10 -0700 Subject: [PATCH 1/2] docs: add inference prefix caching page --- docs.json | 1 + .../response-settings/prefix-caching.mdx | 96 +++++++++++++++++++ 2 files changed, 97 insertions(+) create mode 100644 inference/response-settings/prefix-caching.mdx diff --git a/docs.json b/docs.json index 08f4b95890..a490b26052 100644 --- a/docs.json +++ b/docs.json @@ -1052,6 +1052,7 @@ "inference/response-settings/json-mode", "inference/response-settings/reasoning", "inference/response-settings/streaming", + "inference/response-settings/prefix-caching", "inference/response-settings/structured-output", "inference/response-settings/tool-calling" ] diff --git a/inference/response-settings/prefix-caching.mdx b/inference/response-settings/prefix-caching.mdx new file mode 100644 index 0000000000..b6bc907384 --- /dev/null +++ b/inference/response-settings/prefix-caching.mdx @@ -0,0 +1,96 @@ +--- +title: "Prefix Caching" +description: "Reduce latency for repeated prompts with prefix caching, and isolate cache reuse with cache_salt when needed." +--- + +W&B Inference uses prefix caching on supported hosted models to speed up repeated requests with identical prompt prefixes. + +When a request shares the same prompt prefix as an earlier request on the same backend, the model can reuse previously computed KV cache for that prefix instead of recomputing it from scratch. This can reduce latency for repeated prompts, long system prompts, and workloads with a stable shared prefix. + +Prefix caching is automatic on supported models. You do not need to enable it in your request. + +## When prefix caching helps + +Prefix caching is most useful when you repeatedly send requests that share a long common prefix, such as: + +- A large system prompt reused across many requests. +- A long shared document followed by different user questions. +- Repeated evaluation prompts with only small per-request changes. +- Multi-turn workloads where much of the conversation history stays the same. + +## Cache Isolation + +By default, identical prompt prefixes may reuse cache on shared infrastructure when the backend allows it. + +If you want to isolate cache reuse to a specific trust boundary, set the `cache_salt` request parameter. Requests only reuse prefix cache when both the prompt prefix and the `cache_salt` match. + +Use `cache_salt` when you want cache reuse within a single user, tenant, session, or application boundary, but do not want reuse across other callers. + +### How it works + +- Same prompt prefix, no `cache_salt`: cache may be reused across matching requests. +- Same prompt prefix, same `cache_salt`: cache can be reused. +- Same prompt prefix, different `cache_salt`: cache is isolated and will not be reused across salts. + + +`cache_salt` must be a non-empty string when provided. + + +## Examples + + + + ```python + import openai + + client = openai.OpenAI( + base_url="https://api.inference.wandb.ai/v1", + api_key="", + ) + + response = client.chat.completions.create( + model="moonshotai/Kimi-K2.5", + messages=[ + { + "role": "system", + "content": "You are a careful assistant that answers concisely." + }, + { + "role": "user", + "content": "Summarize this document in one sentence: " + }, + ], + cache_salt="tenant-a-user-123-secret", + ) + + print(response.choices[0].message.content) + ``` + + + + ```bash + curl https://api.inference.wandb.ai/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer " \ + -d '{ + "model": "moonshotai/Kimi-K2.5", + "messages": [ + { "role": "system", "content": "You are a careful assistant that answers concisely." }, + { "role": "user", "content": "Summarize this document in one sentence: " } + ], + "cache_salt": "tenant-a-user-123-secret" + }' + ``` + + + +## Response behavior + +On some models, usage details may include cached token counts in `usage.prompt_tokens_details.cached_tokens` when prefix cache is reused. Availability of that field may vary by model and backend. + +## Related pages + +- [Chat Completions](/inference/api-reference/chat-completions) +- [Enable streaming responses](/inference/response-settings/streaming) +- [Structured output](/inference/response-settings/structured-output) +- [JSON mode](/inference/response-settings/json-mode) From 0b22697dad533a224f3bca9c15ed7d484764107c Mon Sep 17 00:00:00 2001 From: John Mulhausen Date: Wed, 29 Apr 2026 11:56:47 -0400 Subject: [PATCH 2/2] Apply Google Developer Style Guide to prefix caching docs Style-only changes, no technical content added or removed: - Fix title and heading to sentence case - Add document purpose statement, section intros, and list lead-ins - Expand KV abbreviation on first use - Use present tense, contractions, and active voice - Replace idiom with plain language for global audience - Tighten parallel structure in bullet lists --- .../response-settings/prefix-caching.mdx | 26 ++++++++++++------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/inference/response-settings/prefix-caching.mdx b/inference/response-settings/prefix-caching.mdx index b6bc907384..d9029334b0 100644 --- a/inference/response-settings/prefix-caching.mdx +++ b/inference/response-settings/prefix-caching.mdx @@ -1,13 +1,13 @@ --- -title: "Prefix Caching" +title: "Prefix caching" description: "Reduce latency for repeated prompts with prefix caching, and isolate cache reuse with cache_salt when needed." --- W&B Inference uses prefix caching on supported hosted models to speed up repeated requests with identical prompt prefixes. -When a request shares the same prompt prefix as an earlier request on the same backend, the model can reuse previously computed KV cache for that prefix instead of recomputing it from scratch. This can reduce latency for repeated prompts, long system prompts, and workloads with a stable shared prefix. +When a request shares the same prompt prefix as an earlier request on the same backend, the model reuses the previously computed key-value (KV) cache instead of recomputing the entire prefix. This reduces latency for repeated prompts, long system prompts, and workloads with a stable shared prefix. -Prefix caching is automatic on supported models. You do not need to enable it in your request. +Prefix caching is automatic on supported models, so you don't need to enable it in your request. This page describes when prefix caching is most effective and how to control cache isolation with `cache_salt`. ## When prefix caching helps @@ -18,19 +18,23 @@ Prefix caching is most useful when you repeatedly send requests that share a lon - Repeated evaluation prompts with only small per-request changes. - Multi-turn workloads where much of the conversation history stays the same. -## Cache Isolation +## Cache isolation -By default, identical prompt prefixes may reuse cache on shared infrastructure when the backend allows it. +In some environments, you might need to prevent cache reuse across different users or applications. The `cache_salt` parameter provides this control. -If you want to isolate cache reuse to a specific trust boundary, set the `cache_salt` request parameter. Requests only reuse prefix cache when both the prompt prefix and the `cache_salt` match. +By default, requests with identical prompt prefixes may reuse cache on shared infrastructure when the backend allows it. -Use `cache_salt` when you want cache reuse within a single user, tenant, session, or application boundary, but do not want reuse across other callers. +To isolate cache reuse to a specific trust boundary, set the `cache_salt` request parameter. Requests only reuse prefix cache when both the prompt prefix and the `cache_salt` match. + +Use `cache_salt` when you want cache reuse within a single user, tenant, session, or application boundary, but don't want reuse across other callers. ### How it works +The presence and value of `cache_salt` affect cache reuse as follows: + - Same prompt prefix, no `cache_salt`: cache may be reused across matching requests. - Same prompt prefix, same `cache_salt`: cache can be reused. -- Same prompt prefix, different `cache_salt`: cache is isolated and will not be reused across salts. +- Same prompt prefix, different `cache_salt`: cache is isolated and not reused across salts. `cache_salt` must be a non-empty string when provided. @@ -38,6 +42,8 @@ Use `cache_salt` when you want cache reuse within a single user, tenant, session ## Examples +The following examples show how to send a chat completion request with `cache_salt` to isolate cache reuse. + ```python @@ -86,10 +92,12 @@ Use `cache_salt` when you want cache reuse within a single user, tenant, session ## Response behavior -On some models, usage details may include cached token counts in `usage.prompt_tokens_details.cached_tokens` when prefix cache is reused. Availability of that field may vary by model and backend. +To confirm that prefix caching is active for a request, check the response usage details. On some models, usage details may include cached token counts in `usage.prompt_tokens_details.cached_tokens` when prefix cache is reused. The availability of that field varies by model and backend. ## Related pages +The following pages cover related topics: + - [Chat Completions](/inference/api-reference/chat-completions) - [Enable streaming responses](/inference/response-settings/streaming) - [Structured output](/inference/response-settings/structured-output)