External prefix cache hit rate stuck at 100% with LMCache PD disaggregation on RTX PRO 6000 GPUs

[nokxxxxx]

I'm using the ds4dspark-v10.md deployment specification with the Docker image:
voipmonitor/vllm:fathomless-firmament-ds4-v10-vllmadf15ca-b12x90172a5-fi2cba2f7-cu132-20260712

My deployment follows the disaggregated prefill architecture documented in the LMCache official guide:
https://docs.lmcache.ai/zh_CN/mp/disaggregated_prefill.html

lmcache server --port 6555 --http-port 8090 \
    --l1-size-gb 100 --eviction-policy LRU --chunk-size 256 --instance-id prefiller

CUDA_VISIBLE_DEVICES=6 VLLM_NIXL_SIDE_CHANNEL_HOST=127.0.0.1 VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
UCX_NET_DEVICES=all NCCL_CUMEM_ENABLE=1 \
vllm serve <model> --port 8001 --enforce-eager --max-model-len 16384 --gpu-memory-utilization 0.4 \
    --kv-transfer-config '{"kv_connector":"MultiConnector","kv_role":"kv_producer","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail"},{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.host":"tcp://localhost","lmcache.mp.port":6555}}]}}'

lmcache server --port 6556 --http-port 8091 \
    --l1-size-gb 100 --eviction-policy LRU --chunk-size 256 --instance-id decoder

CUDA_VISIBLE_DEVICES=7 VLLM_NIXL_SIDE_CHANNEL_HOST=127.0.0.1 VLLM_NIXL_SIDE_CHANNEL_PORT=5558 \
UCX_NET_DEVICES=all NCCL_CUMEM_ENABLE=1 \
vllm serve <model> --port 8002 --enforce-eager --max-model-len 16384 --gpu-memory-utilization 0.4 \
    --kv-transfer-config '{"kv_connector":"MultiConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail"},{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.host":"tcp://localhost","lmcache.mp.port":6556}}]}}'

vllm-router --policy round_robin --vllm-pd-disaggregation \
    --prefill http://localhost:8001 --decode http://localhost:8002 --host 0.0.0.0 --port 30000

Issue Description

I’ve encountered an abnormal behavior in the decode-stage logs: the External prefix cache hit rate metric is stuck at 100% from the very start of service startup. Relevant log snippets are attached below:

(APIServer pid=7) INFO 07-16 02:41:18 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=7) INFO 07-16 02:41:18 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=7) INFO:     Started server process [7]
(APIServer pid=7) INFO:     Waiting for application startup.
(APIServer pid=7) INFO:     Application startup complete.
(Worker_TP0 pid=199) INFO 07-16 02:47:00 [base_worker.py:645] NIXL compatibility check passed (hash: f74175fccc1f2b38848b17c3fc7a424a757de81bc3b207c31df67276b3311d3d)
(Worker_TP0 pid=199) INFO 07-16 02:47:00 [base_worker.py:1577] Transfer plan: TransferTopology(tp_ratio=1, num_kv_heads=1, local_tp=2, remote_tp=2, remote_pp=0, local_rank=0, remote_block_len=1039680)
(Worker_TP1 pid=200) INFO 07-16 02:47:00 [base_worker.py:645] NIXL compatibility check passed (hash: f74175fccc1f2b38848b17c3fc7a424a757de81bc3b207c31df67276b3311d3d)
(Worker_TP1 pid=200) INFO 07-16 02:47:00 [base_worker.py:1577] Transfer plan: TransferTopology(tp_ratio=1, num_kv_heads=1, local_tp=2, remote_tp=2, remote_pp=0, local_rank=1, remote_block_len=1039680)
(APIServer pid=7) INFO 07-16 02:47:01 [loggers.py:277] Engine 000: Avg prompt throughput: 1.0 tokens/s, Avg generation throughput: 89.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%, External prefix cache hit rate: 100.0%
(APIServer pid=7) INFO 07-16 02:47:01 [metrics.py:103] KV Transfer metrics: NixlPullConnector={'Num successful transfers': 2, 'Avg xfer time (ms)': np.float64(38.618), 'P90 xfer time (ms)': 42.228, 'Avg post time (ms)': np.float64(0.533), 'P90 post time (ms)': 0.555, 'Avg MB per transfer': np.float64(5.949), 'Throughput (MB/s)': np.float64(154.048), 'Avg number of descriptors': np.float64(1.0)}
(APIServer pid=7) INFO 07-16 02:47:02 [loggers.py:277] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 79.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, External prefix cache hit rate: 100.0%
(APIServer pid=7) INFO 07-16 02:47:03 [loggers.py:277] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, External prefix cache hit rate: 100.0%

Deployment Constraints & Business Goal

Our hardware is limited to RTX PRO 6000 GPUs with constrained VRAM capacity. I intend to combine LMCache with the Prefill-Decode (PD) disaggregation deployment pattern to boost overall inference throughput, but all my attempts to get this stack working properly have failed so far.

Could you please assist me in troubleshooting and resolving this issue? This feature is critically important for our production workloads.

---