在zhilight0.4.7镜像环境下，使用 4张4090 GPU QwQ_72B_GPTQ prompt为4k 1并发 没有复现TTFT为1000左右的情况

[DreamTeamWangbowen]

服务端启动命令：

CUDA_VISIBLE_DEVICES=4,5,6,7 python -m zhilight.server.openai.entrypoints.api_server   --host 0.0.0.0   --port 6005   --served-model-name QwQ_72B_GPTQ_4090_zhilight   --model-path /mnt/host_root/mnt/public_data/Qwen/Qwen2.5-72B-Instruct-GPTQ-Int4_main --env "HIGH_PRECISION=0;CPM_FUSE_QKV=1;CPM_FUSE_FF_IN=2;REDUCE_TP_INT8_THRES=100;DUAL_STREAM=1" --dyn-max-batch-size 16

测试命令：

python3 inference_benchmark.py --base-url http://10.72.1.119:6005 --model QwQ_72B_GPTQ_4090_zhilight --input-len 4096 --output-len 1024 --max-concurrency 1 --stop-slo "ttft:10000" "throughput:10" --goodput "ttft:10000" "throughput:10" --metadata arch=x86 gpu="NVIDIA 4090 24G" gpu_num=4 replicas=1 backend=zhilight0.4.17_DUAL_STREAM_0 other_params="{params:--max_model_len 10240 --no-enable-prefix-caching}"

测试结果：

mean_input_tokens	mean_output_tokens	mean_TTFT	mean_prefill_throughput	mean_decode_throughput	output_throughput	mean_TPOT	mean_ITL	mean_E2EL	request_throughput
4044	1024	4017.89	1006.5	45.17	38.4	22.14	22.02	26666.46	0.04

[DreamTeamWangbowen]

mean_TTFT为4000多，是硬件或是参数哪里有问题没生效吗？

[spetrel]

can you run nccl-test on your machine?

[DreamTeamWangbowen]

can you run nccl-test on your machine?

The image I use is ghcr.io/zhihu/zhilight/zhilight:0.4.8-cu124. There seems to be no nccl-test in this container.

vLLM is using nccl==2.21.5

[DreamTeamWangbowen]

can you run nccl-test on your machine?
The output of nccl-test is as follows：

CUDA_VISIBLE_DEVICES=7 ./build/all_reduce_perf -b 8 -e 2048M -f 2 -g 1
# nThread 1 nGpus 1 minBytes 8 maxBytes 2147483648 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid  41304 on  localhost device  0 [0000:d6:00] NVIDIA GeForce RTX 4090
#
#                                                              out-of-place                       in-place          
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       
           8             2     float     sum      -1     6.88    0.00    0.00      0     0.30    0.03    0.00      0
          16             4     float     sum      -1     7.04    0.00    0.00      0     0.30    0.05    0.00      0
          32             8     float     sum      -1     6.65    0.00    0.00      0     0.28    0.11    0.00      0
          64            16     float     sum      -1     6.52    0.01    0.00      0     0.29    0.22    0.00      0
         128            32     float     sum      -1     6.54    0.02    0.00      0     0.29    0.45    0.00      0
         256            64     float     sum      -1     6.71    0.04    0.00      0     0.29    0.89    0.00      0
         512           128     float     sum      -1     5.93    0.09    0.00      0     0.26    2.00    0.00      0
        1024           256     float     sum      -1     5.94    0.17    0.00      0     0.26    4.01    0.00      0
        2048           512     float     sum      -1     6.06    0.34    0.00      0     0.26    7.94    0.00      0
        4096          1024     float     sum      -1     6.31    0.65    0.00      0     0.26   15.96    0.00      0
        8192          2048     float     sum      -1     5.97    1.37    0.00      0     0.26   31.68    0.00      0
       16384          4096     float     sum      -1     5.99    2.74    0.00      0     0.26   62.91    0.00      0
       32768          8192     float     sum      -1     6.13    5.34    0.00      0     0.26  125.12    0.00      0
       65536         16384     float     sum      -1     5.99   10.95    0.00      0     0.26  252.69    0.00      0
      131072         32768     float     sum      -1     5.98   21.91    0.00      0     0.24  540.61    0.00      0
      262144         65536     float     sum      -1     6.04   43.40    0.00      0     0.22  1172.64    0.00      0
      524288        131072     float     sum      -1     5.36   97.87    0.00      0     0.20  2558.13    0.00      0
     1048576        262144     float     sum      -1     5.03  208.43    0.00      0     0.19  5416.20    0.00      0
     2097152        524288     float     sum      -1     5.36  391.21    0.00      0     0.17  12434.94    0.00      0
     4194304       1048576     float     sum      -1     9.70  432.25    0.00      0     0.16  26051.58    0.00      0
     8388608       2097152     float     sum      -1    17.77  471.95    0.00      0     0.15  56507.97    0.00      0
    16777216       4194304     float     sum      -1    35.64  470.76    0.00      0     0.11  151692.73    0.00      0
    33554432       8388608     float     sum      -1    71.55  468.96    0.00      0     0.11  301748.49    0.00      0
    67108864      16777216     float     sum      -1    147.6  454.70    0.00      0     0.11  605675.67    0.00      0
   134217728      33554432     float     sum      -1    293.2  457.69    0.00      0     0.11  1211898.22    0.00      0
   268435456      67108864     float     sum      -1    585.6  458.41    0.00      0     0.12  2286503.03    0.00      0
   536870912     134217728     float     sum      -1   1171.7  458.20    0.00      0     0.11  4909656.26    0.00      0
  1073741824     268435456     float     sum      -1   2340.9  458.68    0.00      0     0.11  9739154.87    0.00      0
  2147483648     536870912     float     sum      -1   4678.5  459.01    0.00      0     0.27  7962490.35    0.00      0
# Out of bounds values : 0 OK
# Avg

[spetrel]

you should run nccl-test with 4 GPUs.

[DreamTeamWangbowen]

you should run nccl-test with 4 GPUs.

root@sangfor:/home/wbw/nccl-tests# ./build/all_reduce_perf -b 8 -e 2048M -f 2 -g 4
--------------------------------------------------------------------------
Open MPI failed an OFI Libfabric library call (fi_getinfo).  This is highly
unusual; your job may behave unpredictably (and/or abort) after this.

  Local host: sangfor
  Location: mtl_ofi_component.c:737
  Error: Function not implemented (38)
--------------------------------------------------------------------------
# nThread 1 nGpus 4 minBytes 8 maxBytes 2147483648 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid 292670 on    sangfor device  0 [0000:18:00] NVIDIA GeForce RTX 4090
#  Rank  1 Group  0 Pid 292670 on    sangfor device  1 [0000:19:00] NVIDIA GeForce RTX 4090
#  Rank  2 Group  0 Pid 292670 on    sangfor device  2 [0000:38:00] NVIDIA GeForce RTX 4090
#  Rank  3 Group  0 Pid 292670 on    sangfor device  3 [0000:39:00] NVIDIA GeForce RTX 4090
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    20.51    0.00    0.00      0    20.20    0.00    0.00      0
          16             4     float     sum      -1    20.49    0.00    0.00      0    19.79    0.00    0.00      0
          32             8     float     sum      -1    19.86    0.00    0.00      0    19.97    0.00    0.00      0
          64            16     float     sum      -1    20.24    0.00    0.00      0    20.20    0.00    0.00      0
         128            32     float     sum      -1    19.96    0.01    0.01      0    20.31    0.01    0.01      0
         256            64     float     sum      -1    20.16    0.01    0.02      0    19.63    0.01    0.02      0
         512           128     float     sum      -1    19.92    0.03    0.04      0    20.19    0.03    0.04      0
        1024           256     float     sum      -1    20.01    0.05    0.08      0    20.60    0.05    0.07      0
        2048           512     float     sum      -1    20.51    0.10    0.15      0    20.20    0.10    0.15      0
        4096          1024     float     sum      -1    20.70    0.20    0.30      0    20.28    0.20    0.30      0
        8192          2048     float     sum      -1    20.36    0.40    0.60      0    20.30    0.40    0.61      0
       16384          4096     float     sum      -1    20.53    0.80    1.20      0    20.38    0.80    1.21      0
       32768          8192     float     sum      -1    21.05    1.56    2.33      0    21.08    1.55    2.33      0
       65536         16384     float     sum      -1    25.71    2.55    3.82      0    25.30    2.59    3.89      0
      131072         32768     float     sum      -1    41.60    3.15    4.73      0    40.82    3.21    4.82      0
      262144         65536     float     sum      -1    75.76    3.46    5.19      0    73.87    3.55    5.32      0
      524288        131072     float     sum      -1    100.2    5.23    7.85      0    99.80    5.25    7.88      0
     1048576        262144     float     sum      -1    147.2    7.12   10.69      0    144.5    7.26   10.89      0
     2097152        524288     float     sum      -1    240.2    8.73   13.10      0    238.8    8.78   13.17      0
     4194304       1048576     float     sum      -1    458.4    9.15   13.73      0    460.9    9.10   13.65      0
     8388608       2097152     float     sum      -1    911.3    9.20   13.81      0    911.7    9.20   13.80      0
    16777216       4194304     float     sum      -1   1814.8    9.24   13.87      0   1816.9    9.23   13.85      0
    33554432       8388608     float     sum      -1   3618.1    9.27   13.91      0   3612.3    9.29   13.93      0
    67108864      16777216     float     sum      -1   7202.7    9.32   13.98      0   7187.1    9.34   14.01      0
   134217728      33554432     float     sum      -1    14348    9.35   14.03      0    14354    9.35   14.03      0
   268435456      67108864     float     sum      -1    28663    9.37   14.05      0    28638    9.37   14.06      0
   536870912     134217728     float     sum      -1    57198    9.39   14.08      0    57166    9.39   14.09      0
  1073741824     268435456     float     sum      -1   114269    9.40   14.09      0   114310    9.39   14.09      0
  2147483648     536870912     float     sum      -1   228452    9.40   14.10      0   228394    9.40   14.10      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 6.553
#

Is this OK? I am not running the Zhilight server.

[spetrel]

What tool you used to run benchmark?
--max-concurrency 1 is too small.

[DreamTeamWangbowen]

What tool you used to run benchmark? --max-concurrency 1 is too small.

batch	mean_input_tokens	mean_output_tokens	mean_TTFT	mean_prefill_throughput	mean_decode_throughput	output_throughput	mean_TPOT	mean_ITL	mean_E2EL	request_throughput
1	4044	1024	4061.54	995.68	45.35	38.47	22.05	22.03	26617.08	0.04
2	4005	1024	8159.27	490.92	38.2	58.61	26.18	26.15	34937.47	0.06
3	3886	1024	11770.09	330.22	39.5	51.59	25.32	25.12	27819.57	0.08
4	3846	1024	16083.18	239.18	37.78	66.43	26.48	26.29	34500.6	0.09

This is the result of testing multiple concurrency.

[DreamTeamWangbowen]

What tool you used to run benchmark? --max-concurrency 1 is too small.

The tool is adapted and modified based on the official script of vllm, and can be used to test frameworks such as vllm and sglang.

vllm official tool link: https://github.com/vllm-project/vllm/blob/main/benchmarks/benchmark_serving.py

[yabea]

@DreamTeamWangbowen 可以参考：https://github.com/zhihu/ZhiLight/blob/main/docs/benchmarks/README.md

[DreamTeamWangbowen]

@DreamTeamWangbowen 可以参考：https://github.com/zhihu/ZhiLight/blob/main/docs/benchmarks/README.md

How to ensure data authenticity when using closed-source tools? We used open-source tools to test but could not reproduce these results.

[DreamTeamWangbowen]

@DreamTeamWangbowen 可以参考：https://github.com/zhihu/ZhiLight/blob/main/docs/benchmarks/README.md

这个是我刚刚采用docker run --network host -it mirrors.sangfor.com/zhihu/zhilight/benchmark:1.0.1 ./main --min_qps 0.10 --max_qps 0.30 --qps_step 0.10 --server_url http://127.0.0.1:6005/v1 --min_duration_s 300 运行的结果，TTFT也是一样的效果
No warnings encountered during benchmark.

Benchmark Results Summary
┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Target QPS ┃ Actual QPS ┃ TTFT Mean(ms) ┃ TTFT P95(ms) ┃ TPOT Mean(ms) ┃ TPOT P95(ms) ┃ Output TPS ┃ Input Tokens ┃ Output Tokens ┃ Request Completed ┃ Request Failed ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ 0.1 │ 0.09 │ 4919.74 │ 8578.32 │ 49.43 │ 85.36 │ 17.71 │ 3851 │ 696 │ 31 │ 0 │
├────────────┼────────────┼───────────────┼──────────────┼───────────────┼──────────────┼────────────┼──────────────┼───────────────┼───────────────────┼────────────────┤
│ 0.2 │ 0.1 │ 61077.93 │ 143959.59 │ 499.21 │ 690.84 │ 1.71 │ 3851 │ 698 │ 58 │ 0 │
├────────────┼────────────┼───────────────┼──────────────┼───────────────┼──────────────┼────────────┼──────────────┼───────────────┼───────────────────┼────────────────┤
│ 0.3 │ 0.14 │ 58242.28 │ 217293.65 │ 524.75 │ 1135.87 │ 0.75 │ 2227 │ 402 │ 83 │ 35 │
└────────────┴────────────┴───────────────┴──────────────┴───────────────┴──────────────┴────────────┴──────────────┴───────────────┴───────────────────┴────────────────┘