Add W&B Inference prefix caching docs

docs.json

@@ -1090,6 +1090,7 @@
                   "inference/response-settings/json-mode",
                   "inference/response-settings/reasoning",
                   "inference/response-settings/streaming",
+                  "inference/response-settings/prefix-caching",
                   "inference/response-settings/structured-output",
                   "inference/response-settings/tool-calling"
                 ]

inference/response-settings/prefix-caching.mdx

@@ -0,0 +1,104 @@
+---
+title: "Prefix caching"
+description: "Reduce latency for repeated prompts with prefix caching, and isolate cache reuse with cache_salt when needed."
+---
+
+W&B Inference uses prefix caching on supported hosted models to speed up repeated requests with identical prompt prefixes.
+
+When a request shares the same prompt prefix as an earlier request on the same backend, the model reuses the previously computed key-value (KV) cache instead of recomputing the entire prefix. This reduces latency for repeated prompts, long system prompts, and workloads with a stable shared prefix.
+
+Prefix caching is automatic on supported models, so you don't need to enable it in your request. This page describes when prefix caching is most effective and how to control cache isolation with `cache_salt`.
+
+## When prefix caching helps
+
+Prefix caching is most useful when you repeatedly send requests that share a long common prefix, such as:
+
+- A large system prompt reused across many requests.
+- A long shared document followed by different user questions.
+- Repeated evaluation prompts with only small per-request changes.
+- Multi-turn workloads where much of the conversation history stays the same.
+
+## Cache isolation
+
+In some environments, you might need to prevent cache reuse across different users or applications. The `cache_salt` parameter provides this control.
+
+By default, requests with identical prompt prefixes may reuse cache on shared infrastructure when the backend allows it.
+
+To isolate cache reuse to a specific trust boundary, set the `cache_salt` request parameter. Requests only reuse prefix cache when both the prompt prefix and the `cache_salt` match.
+
+Use `cache_salt` when you want cache reuse within a single user, tenant, session, or application boundary, but don't want reuse across other callers.
+
+### How it works
+
+The presence and value of `cache_salt` affect cache reuse as follows:
+
+- Same prompt prefix, no `cache_salt`: cache may be reused across matching requests.
+- Same prompt prefix, same `cache_salt`: cache can be reused.
+- Same prompt prefix, different `cache_salt`: cache is isolated and not reused across salts.
+
+<Note>
+`cache_salt` must be a non-empty string when provided.
+</Note>
+
+## Examples
+
+The following examples show how to send a chat completion request with `cache_salt` to isolate cache reuse.
+
+<Tabs>
+  <Tab title="Python">
+    ```python
+    import openai
+
+    client = openai.OpenAI(
+        base_url="https://api.inference.wandb.ai/v1",
+        api_key="<your-api-key>",
+    )
+
+    response = client.chat.completions.create(
+        model="moonshotai/Kimi-K2.5",
+        messages=[
+            {
+                "role": "system",
+                "content": "You are a careful assistant that answers concisely."
+            },
+            {
+                "role": "user",
+                "content": "Summarize this document in one sentence: <long shared prefix here>"
+            },
+        ],
+        cache_salt="tenant-a-user-123-secret",
+    )
+
+    print(response.choices[0].message.content)
+    ```
+  </Tab>
+
+  <Tab title="Bash">
+    ```bash
+    curl https://api.inference.wandb.ai/v1/chat/completions \
+      -H "Content-Type: application/json" \
+      -H "Authorization: Bearer <your-api-key>" \
+      -d '{
+        "model": "moonshotai/Kimi-K2.5",
+        "messages": [
+          { "role": "system", "content": "You are a careful assistant that answers concisely." },
+          { "role": "user", "content": "Summarize this document in one sentence: <long shared prefix here>" }
+        ],
+        "cache_salt": "tenant-a-user-123-secret"
+      }'
+    ```
+  </Tab>
+</Tabs>
+
+## Response behavior
+
+To confirm that prefix caching is active for a request, check the response usage details. On some models, usage details may include cached token counts in `usage.prompt_tokens_details.cached_tokens` when prefix cache is reused. The availability of that field varies by model and backend.
+
+## Related pages
+
+The following pages cover related topics:
+
+- [Chat Completions](/inference/api-reference/chat-completions)
+- [Enable streaming responses](/inference/response-settings/streaming)
+- [Structured output](/inference/response-settings/structured-output)
+- [JSON mode](/inference/response-settings/json-mode)