diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index cddda4638..c05221918 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -2,13 +2,17 @@ name: PR on: pull_request: - types: [opened, edited, synchronize] + types: [opened, edited, synchronize, ready_for_review] permissions: contents: read jobs: checklist-completed: + # Draft PRs are still in progress, so don't require a completed checklist + # until they're marked ready for review. The ready_for_review trigger above + # re-runs this job when that happens. + if: github.event.pull_request.draft == false permissions: # The action reads the PR body through the issues API, so it needs issues # access on top of pull-requests. The repo's default workflow token is diff --git a/design/disaggregation.md b/design/disaggregation.md new file mode 100644 index 000000000..07aeced01 --- /dev/null +++ b/design/disaggregation.md @@ -0,0 +1,288 @@ +# Prefill/Decode Disaggregation + +**Status:** Draft +**Date:** June 2026 +**Author:** Dennis Ramdass + +This document proposes prefill/decode disaggregation for Modelplane. It builds +on the base design in [design.md](./design.md) and the routing it relies on. + +## Summary + +LLM inference has two phases with opposite hardware profiles. Prefill processes +the whole prompt at once and is compute-bound; it sets time-to-first-token. +Decode generates one token at a time and is memory-bandwidth-bound; it sets +inter-token latency. Run on the same pods, a prefill burst stalls in-flight +decodes and neither phase can be tuned independently. + +Disaggregation runs the two phases as separate pod sets. A prefill instance +processes the prompt and transfers its KV cache to a decode instance, which +generates the output. Modelplane expresses this with a `prefill` block on the +deployment: the top-level `workers` is the decode (or unified) role, and adding +a `prefill` block makes the deployment disaggregated. + +It pays off for large models under load with strict TTFT/ITL targets, long +context, and a fast interconnect, where the two phases' loads are large and +skewed enough to tune separately; for small models, short context, or low +traffic the KV-transfer overhead outweighs the benefit and aggregated serving +(optionally with chunked prefill) is simpler. The choice is the operator's: +Modelplane serves unified by default and disaggregates only when a `prefill` +block is set. + +```yaml +apiVersion: modelplane.ai/v1alpha1 +kind: ModelDeployment +metadata: + name: llama-405b + namespace: ml-team +spec: + replicas: 1 + modelCacheRef: + name: llama-405b + # Top-level workers: the decode role (memory-bandwidth-bound). + workers: + count: 3 + topology: + tensor: 8 + template: + spec: + containers: + - name: engine + image: vllm/vllm-openai:v0.9.1 + args: + - "--model=/mnt/models" + - '--kv-transfer-config={"kv_connector":"NixlConnector","kv_role":"kv_consumer"}' + # Decode's hardware. nodeSelector is required (a list of DRA device requests); + # here a high-VRAM GPU plus the InfiniBand fabric the KV transfer needs. + nodeSelector: + devices: + - name: gpu + count: 8 + selectors: + - cel: device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("141Gi")) >= 0 + - name: nic + count: 1 + selectors: + - cel: device.attributes["nic.nvidia.com"].linkType == "infiniband" + # Prefill role. Self-contained, with its own (compute-bound, single-GPU) hardware. + prefill: + workers: + count: 5 + topology: + tensor: 1 + template: + spec: + containers: + - name: engine + image: vllm/vllm-openai:v0.9.1 + args: + - "--model=/mnt/models" + - '--kv-transfer-config={"kv_connector":"NixlConnector","kv_role":"kv_producer"}' + nodeSelector: + devices: + - name: gpu + count: 1 + selectors: + - cel: device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("80Gi")) >= 0 + - name: nic + count: 1 + selectors: + - cel: device.attributes["nic.nvidia.com"].linkType == "infiniband" + # Routing: required for a disaggregated deployment. The endpoint picker (EPP) + # that sequences prefill -> decode, given explicitly like the engine — no + # guessed default. routing.template is a curated PodSpec subset (the EPP + # container); the llm-d inference-scheduler is the usual choice. + routing: + template: + spec: + containers: + - name: epp + image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.8.0 + args: ["--config-file=/config/epp.yaml"] +``` + +## The prefill block + +The `prefill` block is self-contained: its own `workers.count`, `topology`, +`template`, and `nodeSelector`. It repeats settings rather than inheriting from +the root, because explicit repetition is easier to reason about than an implicit +merge. This matches the shape design.md already sketches for disaggregation. + +The prefill:decode ratio is the two `workers.count` values — a topology +parameter fixed per deployment, not a scaling knob. Both are stated explicitly; +there is no default ratio. Defaulting one block of a disaggregated deployment +would be asymmetrical with the rest — we don't guess an engine config or the +`routing` EPP if you omit them — and the ratio is a deliberate topology choice. + +Because the block carries its own `nodeSelector` and `topology`, an operator can +place prefill and decode on different GPU classes through the normal +capability-matching mechanism. Prefill is compute-bound and suits high-FLOPS +GPUs; decode is memory-bandwidth-bound and suits high-bandwidth GPUs. Modelplane +does not choose that hardware. It exposes the knob, and the in-cluster scheduler +places the pods, the same as the unified path. Prefill and decode of a replica +stay on one InferenceCluster, since KV transfer needs co-location, so distinct +hardware means different pools within that cluster rather than different +clusters. + +A deployment without a `prefill` block is unified serving and is unaffected. + +## KV cache transfer + +The prefill engine produces the KV cache and the decode engine consumes it, +configured through the engine's `--kv-transfer-config` (`NixlConnector`, with +`kv_role` `kv_producer` on prefill and `kv_consumer` on decode). NIXL moves the +cache between the two over the fastest available interconnect. + +KV cache size grows roughly linearly with input length, on the order of 0.1 GB +per 1K input tokens for an 8B model, so under load the transfer can reach tens +of GB/s. That is comfortable over NVLink within a node and over RDMA/InfiniBand +across nodes, but it saturates PCIe or plain ethernet. Where the engine supports +it, the transfer is hidden behind compute (layer by layer, asynchronous, +chunked), keeping the disaggregation overhead small. + +## Routing + +Disaggregation needs to route a request to a prefill instance, transfer the KV +cache, then have a decode instance generate from it. Modelplane does this with a +Gateway API Inference Extension (GAIE) `InferencePool` fronted by a swappable +endpoint-picker (EPP), typically the llm-d inference-scheduler — no bespoke +proxy. + +An [`InferencePool`][gaie] is a Gateway API backend (used in an `HTTPRoute` like +a `Service`) that groups model-serving pods and delegates the per-request +endpoint choice to an EPP over the Endpoint Picker Protocol. It is the standard +seam where inference-aware routing — prefix-cache locality, load, prefill/decode +sequencing — plugs in, instead of the round-robin a plain `Service` does. + +[gaie]: https://gateway-api-inference-extension.sigs.k8s.io/ + +**One `InferencePool` fronts both roles.** Its selector matches a deployment's +prefill and decode pods alike; the EPP partitions them internally by a role +label (`llm-d.ai/role: prefill|decode`) its prefill/decode filters select on. +Modelplane stamps that `llm-d.ai/role` label on each role's pods — alongside the +`modelplane.ai/pd-role` label its own compositions use internally, since the EPP +does not read Modelplane's label. Per request the EPP picks a decode pod, +then a prefill pod, and passes the decode pod the chosen prefill's address (an +`x-prefiller-host-port` header). A small routing sidecar on the decode pod +forwards the prompt to that prefill, which runs prefill and transfers its KV +cache over NIXL; the decode engine then generates. The EPP's prefix-cache scorer +still applies, so cache-aware placement carries over. (An earlier sketch assumed +a decode-only pool with the EPP pair-picking across two pools; the llm-d +mechanism is the single-pool, role-partitioned form above.) + +The EPP is configured through `routing.template` — a curated PodSpec subset, the +same shape and owner as the engine — given **explicitly** on a disaggregated +deployment (the example above). There is no default EPP: we don't guess the +routing component any more than we guess the engine. `routing` is required when a +`prefill` block is set. + +The EPP is a lightweight CPU controller — it watches the deployment's pods and +scores requests; it serves no model and holds no GPU. So it runs as a small +Deployment on ordinary nodes in the serving namespace (no GPU, no special +`nodeSelector`), one per deployment's `InferencePool`. Its pod shape comes from +`routing.template`, the same way the engine's comes from `workers.template`. + +A deployment with a `prefill` block selects the multi-pod (llm-d) backend even +at `pipeline: 1`, because disaggregation needs cross-pod coordination regardless +of the per-role topology. + +**Gateway.** `InferencePool` as an `HTTPRoute` backend needs a GAIE-conformant +gateway. ServingStack installs **Envoy AI Gateway** as the default — it's +Envoy-based, purpose-built for LLM traffic, and serves `InferencePool` v1 — so +disaggregation works out of the box on a Modelplane-provisioned cluster. But the +gateway is a swappable seam, like the EPP: any GAIE-conformant gateway works +(Istio, kgateway, GKE Gateway, …), so a customer can plug in their own — for a +BYO cluster that already runs one, or simply a different preference. (Plain +Envoy Gateway is not conformant on its own; `InferencePool` support comes from +the AI Gateway layer, which is why we install that rather than bare Envoy +Gateway.) Unified serving keeps its plain `Service` route for now; the +`InferencePool`/EPP path is used by disaggregated serving (and can later carry +KV-/load-aware routing for unified serving too). + +## Constraints + +These are documented now and enforced as the matching and validation surfaces +mature. + +- **Co-location.** A replica's prefill and decode must be schedulable on one + InferenceCluster. The fleet scheduler rejects the deployment if no matched + cluster can host both roles. +- **Interconnect.** KV transfer needs NVLink within a node or RDMA/InfiniBand + across nodes; over PCIe or ethernet it bottlenecks. The fabric is modeled as a + `Synthetic` device on the InferenceClass (e.g. a `nic` device with + `claim: Synthetic`, since no DRA driver claims it) and matched by a + `nodeSelector` device request, the same way as a claimable GPU — see the + example above. +- **Connector and model compatibility.** Both roles run a compatible KV + connector (`NixlConnector`, paired `kv_role`) on the same model and dtype, + with compatible parallelism so the KV layout matches. +- **Both counts explicit.** A disaggregated deployment sets both `workers.count` + and `prefill.workers.count`; there is no default ratio. +- **Routing explicit.** A disaggregated deployment sets `routing`; the EPP is + not defaulted. + +## Alternatives considered + +### Two ModelDeployments (one prefill, one decode) + +Disaggregation could be expressed as two separate ModelDeployments — one for +prefill, one for decode — reusing existing primitives with no new `prefill` +block. It is close to what's proposed and appealingly minimal, but the single-MD +form is better on two counts: + +- **Co-location.** With one MD the scheduler has everything it needs to place + prefill and decode workers on the same cluster sensibly. With two MDs the + author must either get crafty with `nodeSelector`s to force co-location, or we + add cross-MD co-scheduling hints (and the scheduler would then have to reason + about all MDs together). +- **It's still a model deployment.** A disaggregated MD is conceptually one + thing — "a model deployment." A prefill-only MD isn't one; it's an + implementation half that only makes sense paired with a decode MD. + +### KServe prefill section + +The original sketch (issue #34) expressed disaggregation through KServe's +`LLMInferenceService.prefill` section. Modelplane dropped KServe for a backend +dispatcher (native and llm-d), so disaggregation now lives in the `prefill` +block on the deployment and is emitted by the llm-d backend. The concept carries +over; the resource does not. + +### A bespoke prefill/decode proxy + +vLLM and Ray ship a small proxy that sequences prefill and decode. Running our +own proxy would work, but the GAIE `InferencePool` plus a swappable EPP is the +standard seam for prefix- and KV-aware routing. Reusing it means one routing +component we can extend to unified serving later, rather than a +disaggregation-only proxy. + +### A routing discriminator instead of a template + +The EPP could be selected by a `picker` enum instead of the `routing.template` +shown in the example above: + +```yaml +spec: + routing: + picker: llm-d # enum: llm-d | ...; each value hard-codes an EPP +``` + +A discriminator would force Modelplane to enumerate and version every supported +picker. The template treats the EPP as what it is — a container — and lets a +user swap or tune it (different image, extra scorer args) without an API change, +matching the engine convention and design.md's preference against gratuitous +discriminators. + +### Modelplane choosing per-role hardware + +Modelplane could read the compute-bound and bandwidth-bound profiles and place +each role on a chosen GPU class. It does not. Placement stays a user-declared +`nodeSelector` resolved by the in-cluster scheduler, the same as every other +workload. Modelplane exposes the knob and guards correctness; it does not make +in-cluster scheduling decisions. + +### An implicit prefill:decode ratio + +A default ratio — say 1:1 when the counts are omitted — would be convenient. We +require both counts instead: defaulting one block of a disaggregated deployment +is asymmetrical with the rest (we don't guess an engine config or the `routing` +EPP), and the ratio is a deliberate topology choice, not a sensible default. diff --git a/docs/content/concepts.md b/docs/content/concepts.md index 4d482618c..83103cb0f 100644 --- a/docs/content/concepts.md +++ b/docs/content/concepts.md @@ -136,6 +136,34 @@ across GPUs within a node. Multi-node deployments require a [ModelCache](#modelcache) referenced via `spec.modelCacheRef.name`. +## Disaggregated Serving + +Prefill (processing the whole prompt) and decode (generating tokens one at a +time) have opposite hardware profiles. Prefill is compute-bound and sets +time-to-first-token; decode is memory-bandwidth-bound and sets inter-token +latency. On shared pods a prefill burst stalls in-flight decodes, and neither +phase can be tuned on its own. + +Set a `prefill` block on the ModelDeployment to split them. The top-level +`workers` becomes the decode role and `prefill` is its own self-contained role, +with its own `workers.count`, `topology`, `template`, and `nodeSelector`, so +each phase can land on a different GPU class through the usual capability +matching. The prefill engine transfers its KV cache to a decode engine over +NIXL, configured through the engine's `--kv-transfer-config`. Like multi-node +serving, a disaggregated deployment requires a [ModelCache](#modelcache); both +roles mount the same PVC and stay co-located on one cluster, since KV transfer +needs a fast interconnect (NVLink within a node, RDMA across nodes). + +Disaggregation runs on the multi-node (llm-d) path. A request is routed to a +prefill instance and then to the decode instance holding its KV cache by the +same endpoint picker that fronts multi-node serving. A deployment without a +`prefill` block is unified serving and is unaffected. + +Disaggregation pays off for large models under load with strict latency targets +and long context. For small models or low traffic the KV-transfer overhead +outweighs the benefit, so aggregated serving (optionally with chunked prefill) +is the default. + ## ModelCache A ModelCache stages a model artifact on workload-cluster storage as a @@ -153,11 +181,11 @@ engine container's `env`). Each cache has: -- A **source**: a discriminated union of where to fetch the artifact, with - source-specific fields (e.g. `huggingFace.repo` and `huggingFace.sizeGiB` - today). Future types (`dragonfly` for P2P distribution, `oci` for NIM-style - bundled artifacts) will declare different fields under their own - discriminator. +- A **source**: a required `source` enum naming the kind, with the matching + source object set alongside it (e.g. `source: HuggingFace` selects + `spec.huggingFace`, which carries `repo` and `sizeGiB`). `HuggingFace` is the + only value today; future sources add an enum value and a sibling object + (`Dragonfly` for P2P distribution, `OCI` for NIM-style bundled artifacts). - An optional **clusterSelector** to scope replication. Omitting `spec.clusterSelector` stages the cache on every matched cluster; setting `matchLabels` restricts it to clusters carrying those labels.