Can vllm serving clients by using multiple model instances?

[aoyulong]

Based on the examples, vllm can launch a server with a single model instances. Can vllm serving clients by using multiple model instances? With multiple model instances, the sever will dispatch the requests to different instances to reduce the overhead.

[zhuohan123]

Right now vLLM is a serving engine for a single model. You can start multiple vLLM server replicas and use a custom load balancer (e.g., nginx load balancer). Also feel free to checkout FastChat and other multi-model frontends (e.g., aviary). vLLM can be a model worker of these libraries to support multi-replica serving.

[hex-plex]

This is just a example of load balancer that I use hope its useful for others

Load Balancer Gist

[hughesadam87]

Is this still the case? If so, why does the API support a model parameter if the intents is not to host multiple models?

[dmarx]

maybe to be compliant with the OpenAI API spec, possibly also future proofing

[khurramkhalil]

Can someone please let me know what is best practice to serve multiple (different) models with vllm?

[huiyeruzhou]

有人可以告诉我使用 vllm 服务多个（不同）模型的最佳做法是什么吗？

Ray serve shold have been the best practice, However it seems that ray serve framework and ray infrastration used in AsyncLLMEngine are conflicting and result in a No GPU Error.

[Yixin-L]

Can vllm serving clients by using multiple model instances now?

[kinthaiofficial]

Multiple model instances for serving is a natural question when you're scaling up, but the more interesting design question for agent systems is: how do you route intelligently across those instances?

The naive approach is round-robin across instances, which works for load balancing but ignores:

KV cache affinity — if the same agent always calls the same instance, the prefix cache hit rate is dramatically higher. A routing layer that assigns agent-id → instance-id can turn 80% cache misses into 80% cache hits for agents with recurring prefixes.

Model tier differentiation — you might want different instances to specialize: one runs a fast small model (Haiku-class), another runs a capable mid-tier model. The routing layer then needs cost-aware dispatch: cheap tasks go to the fast instance, reasoning tasks go to the capable instance. We settled on ~58% fast / 31% mid-tier / 11% reasoning in our blended traffic, which roughly mirrors how we configure the capacity ratio across instances.

Budget-weighted priority queues — in multi-tenant serving, a high-budget agent should get priority over a low-budget one. This needs to be a first-class concept in the instance selection logic, not just a queue depth check.

For the embeddings question (#310): embeddings are a perfect candidate for a separate dedicated instance since the compute profile is very different from generation — no KV cache, no sampling, pure throughput. Mixing embedding and generation traffic on the same instance tends to hurt both.

Write-up on the multi-tier routing model: https://blog.kinthai.ai/openclaw-multi-tenancy-why-vm-per-user-doesnt-scale

Are you thinking about stateless round-robin across instances, or looking for affinity-based routing?

[kinthaiofficial]

Serving clients using multiple model instances is exactly what vLLM is designed for — the key is configuring the router layer correctly. A few patterns:

Horizontal scaling with a load balancer:
Run multiple vLLM instances (one per GPU or GPU cluster) behind a reverse proxy. Nginx, Envoy, or a simple round-robin load balancer routes requests to available instances. Each instance serves the same model but handles independent request queues.

upstream vllm_backend {
    least_conn;
    server 127.0.0.1:8000;
    server 127.0.0.1:8001;
    server 127.0.0.1:8002;
}

Multiple models on the same instance:
vLLM's --model flag runs one model per process. For multiple models, you run multiple vLLM processes (one per model) on different ports, then route via a proxy based on the requested model name.

Model routing middleware:
A thin FastAPI proxy that reads the model field from the request body and routes to the appropriate vLLM backend:

@app.post("/v1/completions")
async def route_completion(request: Request):
    body = await request.json()
    model = body.get("model", "default")
    backend_url = model_to_backend_map.get(model, default_backend)
    # forward to appropriate vLLM instance

Tensor parallel vs model parallel:
For a single large model: use tensor parallelism (--tensor-parallel-size N) to spread across N GPUs. For serving multiple smaller models: run each model on separate GPUs, route at the proxy layer.

For agent workloads:
Smart routing based on task type — route reasoning tasks to a larger model, text extraction to a smaller one. The routing overhead (1-2ms) is negligible vs model inference time.

[AkshatRaj00]

Yes! vLLM supports serving clients using multiple model instances in several ways:

1. Multi-GPU Tensor Parallelism (Single instance, multiple GPUs)

One model instance split across multiple GPUs:

bash
vllm serve meta-llama/Llama-2-70b-hf \
  --tensor-parallel-size 4  # splits across 4 GPUs

2. Multiple Separate Instances (True multi-instance)

Run multiple vLLM server processes on different ports and load balance in front:

bash
# Instance 1
vllm serve mistralai/Mistral-7B-v0.1 --port 8000

# Instance 2
vllm serve mistralai/Mistral-7B-v0.1 --port 8001

Then use Nginx / HAProxy / any load balancer in front to distribute traffic.

3. LLM Engine with AsyncLLMEngine (Programmatic)

python
from vllm import AsyncLLMEngine, AsyncEngineArgs

engine = AsyncLLMEngine.from_engine_args(
    AsyncEngineArgs(model="mistralai/Mistral-7B-v0.1")
)

vLLM's async engine handles concurrent requests automatically via continuous batching — so a single instance already serves many clients simultaneously without spinning up multiple processes.

4. Pipeline Parallelism (Multi-node)

For very large models across multiple nodes:

bash
vllm serve meta-llama/Llama-2-70b-hf \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 2  # 2 nodes x 4 GPUs = 8 GPUs total

---

Which approach to use?

Scenario | Recommendation -- | --

---

Bottom line

A single vLLM instance already handles multiple concurrent clients efficiently via continuous batching + PagedAttention. For higher availability or scaling beyond one machine, run multiple instances behind a load balancer. 🚀