What problem are you facing?
Engine images dominate replica cold start. vllm/vllm-openai is ~10 GB compressed, ~20 GB unpacked on a fresh node the pull-and-unpack takes minutes, often longer than loading the model itself once a ModelCache serves the weights. ModelCache solved the weights half of cold start, the image half is untouched.
Today the pull starts only when the first replica lands on a node, so every new node pays it in the serving path.
How could Modelplane help solve your problem?
Proposal: an ImageCache, the image sibling of ModelCache. A namespaced XR that pre-pulls a list of images onto every node of matched clusters, so the image is already unpacked when a replica schedules there.
apiVersion: modelplane.ai/v1alpha1
kind: ImageCache
metadata:
name: engines
namespace: ml-team
spec:
images:
- vllm/vllm-openai:v0.23.0
# Optional, same semantics as ModelCache: omit to cache on every cluster.
inferenceClusterSelector:
matchLabels:
modelplane.ai/region: eu-north
Composition: per matched cluster, a DaemonSet applied through the cluster's provider-kubernetes ProviderConfig (the ModelCache pattern).
One initContainer per image with command ["true"] and imagePullPolicy IfNotPresent, then a pause main container. kubelet pulls the images as the DaemonSet pod starts, which on a new node is at node birth, in parallel with driver install and NFD labeling rather than in the serving path. The long-running pod keeps the images referenced so kubelet image GC doesn't evict them under disk pressure. The DaemonSet must tolerate the nvidia.com/gpu taint; the GPU nodes are the whole point (see the NFD and CSI plugin tolerations for precedent).
Honest limits:
- Scale-from-zero gains nothing. When a replica triggers the node scale-up, the pre-pull and the workload pull start together. ImageCache helps standing pools, engine version upgrades staged before a rollout, and re-scheduling; it does not make autoscaling cold starts faster.
- Disk: ~20 GB per engine image per node. Node pools default to something like 100 GB boot disks, two engine images plus the base system gets tight. The docs should say to size diskSizeGb for the images cached.
- The image list is manual in v1alpha1. Deriving it automatically from ModelDeployments referencing the cache's namespace is a possible v2alpha1, but pre-pulling only helps when it happens before the deployment exists, so an explicit list is not just simpler, it's more correct.
Alternatives considered:
- Spegel (in-cluster p2p registry mirror): makes the second node's pull fast by fetching from peers, but the first pull still hits the registry, and it adds a component to every cluster. Complementary rather than competing; could be a later serving-stack addition.
- OpenKruise ImagePullJob: purpose-built CRDs, but pulls in the Kruise controller as a dependency for something a plain DaemonSet does.
What problem are you facing?
Engine images dominate replica cold start.
vllm/vllm-openaiis ~10 GB compressed, ~20 GB unpacked on a fresh node the pull-and-unpack takes minutes, often longer than loading the model itself once a ModelCache serves the weights. ModelCache solved the weights half of cold start, the image half is untouched.Today the pull starts only when the first replica lands on a node, so every new node pays it in the serving path.
How could Modelplane help solve your problem?
Proposal: an ImageCache, the image sibling of ModelCache. A namespaced XR that pre-pulls a list of images onto every node of matched clusters, so the image is already unpacked when a replica schedules there.
Composition: per matched cluster, a DaemonSet applied through the cluster's provider-kubernetes ProviderConfig (the ModelCache pattern).
One initContainer per image with command ["true"] and imagePullPolicy IfNotPresent, then a pause main container. kubelet pulls the images as the DaemonSet pod starts, which on a new node is at node birth, in parallel with driver install and NFD labeling rather than in the serving path. The long-running pod keeps the images referenced so kubelet image GC doesn't evict them under disk pressure. The DaemonSet must tolerate the nvidia.com/gpu taint; the GPU nodes are the whole point (see the NFD and CSI plugin tolerations for precedent).
Honest limits:
Alternatives considered: