vllm-prefill
--served-model-name=glm5.1
--tensor-parallel-size=8
--enforce-eager
--trust-remote-code
--return-tokens-as-token-ids
--gpu-memory-utilization=0.5
--kv-cache-dtype=fp8_ds_mla
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'
--kv-transfer-config '{ "kv_connector": "TileRTConnector", "kv_connector_module_path": "tilert.pd_vllm.prefill_connector", "kv_role": "kv_producer", "kv_connector_extra_config":{ "tilert_host":"[10.91.0.187]", "tilert_ctrl_port":5556, "tilert_model":"glm5", "tilert_max_seq_len":202752 } }'

tilert-decoder
--engine=tilert
--model=glm5
--with-mtp
--max-seq-len=202752
--kv-cache-dtype=fp8
--ctrl-port=5556
--http-port=5557

tilert router:
--vllm-url http://10.91.0.187:40046
--decode 10.91.0.187:5556:5557

log:

glm5-ewem3_2026-07-31_23-24-30.txt
glm5.1-prefill-upvqa_2026-07-31_23-24-28.txt
glm5.1-cv5ee_2026-07-31_23-24-32.txt

i know the reason. because rilert router now only support stream:false , error:"{"error":"streaming is supported on /v1/chat/completions"}"

but vllm bench serve only support stream=true benchmark ,so i change the test tool to 'evalscope[perf]' to test ,it support stream:false test

64K/1K*10

evalscope perf
--url http://10.91.0.187:40040/v1/completions
--api openai
--model glm5.1
--dataset random
--tokenizer-path /var/lib/gpustack/cache/model_scope/ZhipuAI/GLM-5.1-FP8
--min-prompt-length 64000 --max-prompt-length 64000
--max-tokens 3000
--number 10 --parallel 1
--no-stream