What problem are you facing?
Modelplane only wires an endpoint picker (EPP + InferencePool) for disaggregated (PrefillDecode) serving. Unified serving falls through to a plain Kubernetes Service (compose-model-replica/function/routing.py, apply() gates the EPP on serving.mode == "PrefillDecode").
So whenever an aggregated model has more than one interchangeable serving pod in a cluster (copies > 1, or co-located replicas of the same model), requests are sprayed round-robin with no prefix-cache or load awareness. Each pod recomputes prefixes another pod already has cached.
How could Modelplane help?
Give aggregated serving the same KV-/load-aware endpoint picking that disaggregated serving already gets, whenever a model has more than one serving pod in a cluster. Open question: how the picker should be scoped (per replica vs per model-per-cluster) given co-located replicas fragment the view.
What problem are you facing?
Modelplane only wires an endpoint picker (EPP + InferencePool) for disaggregated (
PrefillDecode) serving.Unifiedserving falls through to a plain Kubernetes Service (compose-model-replica/function/routing.py,apply()gates the EPP onserving.mode == "PrefillDecode").So whenever an aggregated model has more than one interchangeable serving pod in a cluster (
copies > 1, or co-located replicas of the same model), requests are sprayed round-robin with no prefix-cache or load awareness. Each pod recomputes prefixes another pod already has cached.How could Modelplane help?
Give aggregated serving the same KV-/load-aware endpoint picking that disaggregated serving already gets, whenever a model has more than one serving pod in a cluster. Open question: how the picker should be scoped (per replica vs per model-per-cluster) given co-located replicas fragment the view.