What problem are you facing?
When a model is running, Modelplane treats the cluster's capacity as a fixed number taken from the InferenceCluster spec (nodePools[].nodeCount, or the autoscaling bounds), not as something it observes. The published status.gpuPools[].nodes is derived from that declared number.
This becomes a problem after a model is already serving and the cluster loses capacity that the model needs. Nodes can disappear for several reasons. A GPU node can fail and be replaced by the platform's node repair. A node can be preempted or drained. An operator can scale a pool down. In each case the declared nodeCount is unchanged, so Modelplane keeps reporting the original capacity. Placement does not take the cluster's real state into account. It neither moves the affected replica onto healthy capacity nor reports that the cluster has fallen below the capacity the model requires.
Today the only signal that something is wrong is ReplicasReady going false, because the underlying pods stop being ready. That tells an operator only that the model is not ready. It does not say that the cluster lost capacity the model depends on, and the two need different responses. The gap is sharper for multi-node workers, where a model needs a fixed set of nodes to run at all (a gang). Losing one node of the gang leaves the whole replica unable to schedule, while the reported capacity still claims the nodes are present.
This affects every cluster source. The gap is most acute for Existing (bring your own) clusters, where there is no provisioned node pool whose count Modelplane could consult, so the declared number is the only capacity input.
How could Modelplane help solve your problem?
For a running model, capacity and placement should be based on the cluster's actual, observed capacity rather than a static declared number. When a cluster drops below the capacity a model requires, Modelplane should make that visible in the InferenceCluster and ModelDeployment conditions, with a reason that separates a model that is still starting from a cluster that lost capacity the model needs. An operator or an automation can then act on it, rather than wait for the pods to fail.
The signal should be provider agnostic. It needs to work the same for provisioned clusters (GKE, EKS) and for Existing clusters, since a bring your own cluster is the case where a declared count is least trustworthy.
This describes the outcome rather than a mechanism. The balance to weigh is responsiveness against stability. Reacting to every brief dip, such as a single node restarting, would make deployments flap, so the behavior should separate a recoverable blip from a sustained shortfall below what the model requires.
What problem are you facing?
When a model is running, Modelplane treats the cluster's capacity as a fixed number taken from the
InferenceClusterspec (nodePools[].nodeCount, or the autoscaling bounds), not as something it observes. The publishedstatus.gpuPools[].nodesis derived from that declared number.This becomes a problem after a model is already serving and the cluster loses capacity that the model needs. Nodes can disappear for several reasons. A GPU node can fail and be replaced by the platform's node repair. A node can be preempted or drained. An operator can scale a pool down. In each case the declared
nodeCountis unchanged, so Modelplane keeps reporting the original capacity. Placement does not take the cluster's real state into account. It neither moves the affected replica onto healthy capacity nor reports that the cluster has fallen below the capacity the model requires.Today the only signal that something is wrong is
ReplicasReadygoing false, because the underlying pods stop being ready. That tells an operator only that the model is not ready. It does not say that the cluster lost capacity the model depends on, and the two need different responses. The gap is sharper for multi-node workers, where a model needs a fixed set of nodes to run at all (a gang). Losing one node of the gang leaves the whole replica unable to schedule, while the reported capacity still claims the nodes are present.This affects every cluster source. The gap is most acute for
Existing(bring your own) clusters, where there is no provisioned node pool whose count Modelplane could consult, so the declared number is the only capacity input.How could Modelplane help solve your problem?
For a running model, capacity and placement should be based on the cluster's actual, observed capacity rather than a static declared number. When a cluster drops below the capacity a model requires, Modelplane should make that visible in the
InferenceClusterandModelDeploymentconditions, with a reason that separates a model that is still starting from a cluster that lost capacity the model needs. An operator or an automation can then act on it, rather than wait for the pods to fail.The signal should be provider agnostic. It needs to work the same for provisioned clusters (GKE, EKS) and for
Existingclusters, since a bring your own cluster is the case where a declared count is least trustworthy.This describes the outcome rather than a mechanism. The balance to weigh is responsiveness against stability. Reacting to every brief dip, such as a single node restarting, would make deployments flap, so the behavior should separate a recoverable blip from a sustained shortfall below what the model requires.