MSCCL all-to-all performance did not improve compared with NCCL

[Musisoul]

Hi, I have tried nccl-alltoall_perf-tests on 1/2/8 nodes with 8xA100 GPUs and found that the performance of msccl(in-place) did not imporve compared with nccl(out-of-place). My MSCCL_XML_FILES were generated by python msccl-tools/examples/mscclang/alltoall_a100_two_step.py.py --protocol=LL 8 8 > two_step_64.xml. I also tried alltoall_a100_three_step.py and alltoall_allpairs.py, they all behaved similarly.
The test code is nccl-tests/build/alltoall_perf -b 1MB -e 1024MB -f 2 -g 1 -n 100 -w 100, and I used 8/16/64 GPUs to run it, corresponding to 1/2/8 nodes.
The alltoall-test result of 8 nodes is like this:

#                                                              out-of-place                       in-place          
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       
     1048576          4096     float    none      -1   9012.2    0.12    0.11      0    561.5    1.87    1.84    N/A
     2097152          8192     float    none      -1   1067.7    1.96    1.93      0   1046.3    2.00    1.97    N/A
     4194304         16384     float    none      -1   2010.8    2.09    2.05      0   2023.0    2.07    2.04    N/A
     8388608         32768     float    none      -1   5698.5    1.47    1.45      0   4261.4    1.97    1.94    N/A
    16777216         65536     float    none      -1   8339.5    2.01    1.98      0   8211.3    2.04    2.01    N/A
    33554432        131072     float    none      -1    16235    2.07    2.03      0    16281    2.06    2.03    N/A
    67108864        262144     float    none      -1    32252    2.08    2.05      0    51440    1.30    1.28    N/A
   134217728        524288     float    none      -1    63877    2.10    2.07      0    83221    1.61    1.59    N/A
   268435456       1048576     float    none      -1   147334    1.82    1.79      0   142747    1.88    1.85    N/A
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 1.77934 

I also find that the Avg bus bandwidth drops sharply on multi-nodes(2/8) compared with one node. I have attached the logs of 8/16/64 GPUs below. Thank you!
gpu8-two_step.log
gpu16-two_step.log
gpu64-two_step.log

[saeedmaleki]

Hi @Musisoul, I think you used the original nccl-tests which doesn't have ncclAllToAll and therefore, all of your logs correspond to NCCL's alltoall algorithm. Please change the nccl-tests alltoall.cu's way of calling alltoall to this way. Alternatively, you could just use this forked version of nccl-tests to test alltoall performance.

Regarding the performance drop: clearly for cross-node communication, some networking interface will be used. If you can provide your log with NCCL_DEBUG=INFO environment variable, I would be able to help you better.

Lastly, please note that in-place alltoall is impossible to perform correctly. As you can see in your logs, the #wrong column is N/A for in-place alltoall even for NCCL runs. The algorithms that you generated with msccl-tools is only valid for out-of-place as well.

I hope this helps!

[Musisoul]

Hi @Musisoul, I think you used the original nccl-tests which doesn't have ncclAllToAll and therefore, all of your logs correspond to NCCL's alltoall algorithm. Please change the nccl-tests alltoall.cu's way of calling alltoall to this way. Alternatively, you could just use this forked version of nccl-tests to test alltoall performance.

Regarding the performance drop: clearly for cross-node communication, some networking interface will be used. If you can provide your log with NCCL_DEBUG=INFO environment variable, I would be able to help you better.

Lastly, please note that in-place alltoall is impossible to perform correctly. As you can see in your logs, the #wrong column is N/A for in-place alltoall even for NCCL runs. The algorithms that you generated with msccl-tools is only valid for out-of-place as well.

I hope this helps!

Thanks for your reply! I'll change the version of nccl-tests and try again.
As for the performance drop, I have attached the logs using NCCL_DEBUG=INFO below.
gpu8-two_step_complete.log
gpu64-two_step_complete.log

[Jack47]

Thanks for the kindly reply. We test according to msccl readme. the changes in alltoall.cu means msccl generated new API ncclAllToAll, which is currently not compatible with nccl API? nccl currently doesn't have this API.

[Jack47]

I found msccl2DAllToAll in msccl , seems there are two kinds of all2all in msccl?

1. native implemented msccl2DAllToAll
2. algorithms generated using msccl-tools

Three questions:

1. what's the correct way to let msccl use native msccl2DAllToAll?
2. is NCCL_ALGO used to specify the priority order of available algos?
3. when generating allreduce algos in msccl-tools, we should specify num nodes, this means whenever I changed the training scale, I should first generate coordinate msccl xmls?

[Musisoul]

Hi @Musisoul, I think you used the original nccl-tests which doesn't have ncclAllToAll and therefore, all of your logs correspond to NCCL's alltoall algorithm. Please change the nccl-tests alltoall.cu's way of calling alltoall to this way. Alternatively, you could just use this forked version of nccl-tests to test alltoall performance.

Regarding the performance drop: clearly for cross-node communication, some networking interface will be used. If you can provide your log with NCCL_DEBUG=INFO environment variable, I would be able to help you better.

Lastly, please note that in-place alltoall is impossible to perform correctly. As you can see in your logs, the #wrong column is N/A for in-place alltoall even for NCCL runs. The algorithms that you generated with msccl-tools is only valid for out-of-place as well.

I hope this helps!

Hi, I have tried the forked version of nccl-tests, the result is different from previous, but the in-place result is extremely high compared with out-of-place. That seems to be abnormal. Here is the 2-nodes(16 GPUs) all-to-all test result:

#                                                     out-of-place                       in-place          
#       size         count    type   redop     time   algbw   busbw  error     time   algbw   busbw  error
#        (B)    (elements)                     (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       
     1048576         16384   float            571.5    1.83    1.72  0e+00     0.50  2087.88  1957.39  1e+00
     2097152         32768   float           1043.0    2.01    1.88  0e+00     0.21  9985.96  9361.84  1e+00
     4194304         65536   float           1972.5    2.13    1.99  0e+00     0.22  18941.04  17757.23  1e+00
     8388608        131072   float           3909.3    2.15    2.01  0e+00     0.51  16433.75  15406.64  1e+00
    16777216        262144   float           7596.3    2.21    2.07  0e+00     0.45  37659.30  35305.59  1e+00
    33554432        524288   float            14892    2.25    2.11  0e+00     0.17  196339.57  184068.34  1e+00
    67108864       1048576   float            29605    2.27    2.13  0e+00     0.21  320099.52  300093.30  1e+00
   134217728       2097152   float            33285    4.03    3.78  0e+00     0.49  271437.56  254472.71  1e+00
   268435456       4194304   float            76900    3.49    3.27  0e+00     0.14  1938582.05  1817420.67  1e+00
   536870912       8388608   float           181940    2.95    2.77  0e+00     0.40  1341070.90  1257253.97  1e+00
  1073741824      16777216   float           285011    3.77    3.53  0e+00     0.47  2292116.18  2148858.92  1e+00
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 274636 

The complete logs are attached below.
n8-two_step-complete.log
n16-two_step-complete.log

What's the correct usage of all-to-all tests? Does it differ from allreduce-tests, like generating xmls in msccl-tools and the startup code(e.g. NCCL_ALGO=MSCCL,RING,TREE) in nccl-tests?
Thank you!

[saeedmaleki]

Hi @Musisoul, I think you used the original nccl-tests which doesn't have ncclAllToAll and therefore, all of your logs correspond to NCCL's alltoall algorithm. Please change the nccl-tests alltoall.cu's way of calling alltoall to this way. Alternatively, you could just use this forked version of nccl-tests to test alltoall performance.
Regarding the performance drop: clearly for cross-node communication, some networking interface will be used. If you can provide your log with NCCL_DEBUG=INFO environment variable, I would be able to help you better.
Lastly, please note that in-place alltoall is impossible to perform correctly. As you can see in your logs, the #wrong column is N/A for in-place alltoall even for NCCL runs. The algorithms that you generated with msccl-tools is only valid for out-of-place as well.
I hope this helps!

Thanks for your reply! I'll change the version of nccl-tests and try again. As for the performance drop, I have attached the logs using NCCL_DEBUG=INFO below. gpu8-two_step_complete.log gpu64-two_step_complete.log

This is an interesting topology you have! 8xA100s with only a single IB! I think your IB has 200Gbps BW which explains your numbers. I think one can design a much better algorithm for your specific topology but the current one you are currently using seems to be delivering a pretty good performance. If you really want the maximum performance, we can help you design the algorithm for your topology.

[saeedmaleki]

Thanks for the kindly reply. We test according to msccl readme. the changes in alltoall.cu means msccl generated new API ncclAllToAll, which is currently not compatible with nccl API? nccl currently doesn't have this API.

That's correct. NCCL made the decision not to make alltoall a collective with an API but RCCL (AMD's version of NCCL) and any MPI implementation has alltoall as an API. Because of this, we apply a patch to PyTorch to change alltoall's implementation with a call to our API. New PyTorch has this API for RCCL in the latest versions.

[saeedmaleki]

Thanks for the kindly reply. We test according to msccl readme. the changes in alltoall.cu means msccl generated new API ncclAllToAll, which is currently not compatible with nccl API? nccl currently doesn't have this API.

That's correct. NCCL made the decision not to make alltoall a collective with an API but RCCL (AMD's version of NCCL) and any MPI implementation has alltoall as an API. Because of this, we apply a patch to PyTorch to change alltoall's implementation with a call to our API. New PyTorch has this API for RCCL in the latest versions.

[saeedmaleki]

Hi @Musisoul, I think you used the original nccl-tests which doesn't have ncclAllToAll and therefore, all of your logs correspond to NCCL's alltoall algorithm. Please change the nccl-tests alltoall.cu's way of calling alltoall to this way. Alternatively, you could just use this forked version of nccl-tests to test alltoall performance.
Regarding the performance drop: clearly for cross-node communication, some networking interface will be used. If you can provide your log with NCCL_DEBUG=INFO environment variable, I would be able to help you better.
Lastly, please note that in-place alltoall is impossible to perform correctly. As you can see in your logs, the #wrong column is N/A for in-place alltoall even for NCCL runs. The algorithms that you generated with msccl-tools is only valid for out-of-place as well.
I hope this helps!

Hi, I have tried the forked version of nccl-tests, the result is different from previous, but the in-place result is extremely high compared with out-of-place. That seems to be abnormal. Here is the 2-nodes(16 GPUs) all-to-all test result:

#                                                     out-of-place                       in-place          
#       size         count    type   redop     time   algbw   busbw  error     time   algbw   busbw  error
#        (B)    (elements)                     (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       
     1048576         16384   float            571.5    1.83    1.72  0e+00     0.50  2087.88  1957.39  1e+00
     2097152         32768   float           1043.0    2.01    1.88  0e+00     0.21  9985.96  9361.84  1e+00
     4194304         65536   float           1972.5    2.13    1.99  0e+00     0.22  18941.04  17757.23  1e+00
     8388608        131072   float           3909.3    2.15    2.01  0e+00     0.51  16433.75  15406.64  1e+00
    16777216        262144   float           7596.3    2.21    2.07  0e+00     0.45  37659.30  35305.59  1e+00
    33554432        524288   float            14892    2.25    2.11  0e+00     0.17  196339.57  184068.34  1e+00
    67108864       1048576   float            29605    2.27    2.13  0e+00     0.21  320099.52  300093.30  1e+00
   134217728       2097152   float            33285    4.03    3.78  0e+00     0.49  271437.56  254472.71  1e+00
   268435456       4194304   float            76900    3.49    3.27  0e+00     0.14  1938582.05  1817420.67  1e+00
   536870912       8388608   float           181940    2.95    2.77  0e+00     0.40  1341070.90  1257253.97  1e+00
  1073741824      16777216   float           285011    3.77    3.53  0e+00     0.47  2292116.18  2148858.92  1e+00
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 274636 

The complete logs are attached below. n8-two_step-complete.log n16-two_step-complete.log

What's the correct usage of all-to-all tests? Does it differ from allreduce-tests, like generating xmls in msccl-tools and the startup code(e.g. NCCL_ALGO=MSCCL,RING,TREE) in nccl-tests? Thank you!

These numbers make sense to me given your topology. Regarding in-place: note that the error column is non-zero which means that the implementation is incorrect. The reason is that in the forked repo, we completely disabled in-place alltoall as no one expect in-pace alltoall to work correctly. That's why the in-place numbers are crazy high since they do not perform any communication and it immediately returns.

[saeedmaleki]

I found msccl2DAllToAll in msccl , seems there are two kinds of all2all in msccl?

1. native implemented msccl2DAllToAll
2. algorithms generated using msccl-tools

Three questions:

1. what's the correct way to let msccl use native msccl2DAllToAll?
2. is NCCL_ALGO used to specify the priority order of available algos?
3. when generating allreduce algos in msccl-tools, we should specify num nodes, this means whenever I changed the training scale, I should first generate coordinate msccl xmls?

I found msccl2DAllToAll in msccl , seems there are two kinds of all2all in msccl?

1. native implemented msccl2DAllToAll
2. algorithms generated using msccl-tools

Three questions:

1. what's the correct way to let msccl use native msccl2DAllToAll?
2. is NCCL_ALGO used to specify the priority order of available algos?
3. when generating allreduce algos in msccl-tools, we should specify num nodes, this means whenever I changed the training scale, I should first generate coordinate msccl xmls?

Great questions!

AllToAll: msccl2DAllToAll is triggered if the name filed inside XML is just 2D. However, it only performs better than the one you are generating with msccl-tools when there are >1K GPUs. This was a hacky way for us to compare the performance of the two approaches it will be removed in the future releases. NCCL_ALGO doesn't need to change.

msccl.init from here is an automatic way to manage these XMLs. But given that you have a unique topology, you won't necessarily get the maximum bang for the buck with the same algorithms. The ones that we have studied the performance for were on 8xA100 nodes with 8xIBs such as NDv4 SKU offered on Azure.

I hope this helps.

[Jack47]

If you really want the maximum performance, we can help you design the algorithm for your topology

sure, we really want the maximum performance under 42 A100 nodes with only 1 200Gbps IB per node. it would be great if you guys can help design the algorithm.

BTW, I've read the gc3 paper and find it's inspiring, it's very useful and effective in this case.

[saeedmaleki]

What input sizes do you need the alltoall for? Your algBW at max should be 25GBps/(N-1). For example, for 8 nodes, the maximum you can get at very large sizes is 3.57GBps algBW.

Regarding the proper algorithm for your use case: you might wanna use something like the 3-step algorithm for alltoall except that the relay logic needs to change. This means that we need to (1) gather cross-node traffic to a local GPU (let's say local GPU 0) and (2) then GPU 0 sends cross-node data to local GPU 0 on other nodes. (3) Lastly, GPU 0 scatters the data to everyone. Thus, the name 3-step algorithm.

Hope this helps.

[Jack47]

What input sizes do you need the alltoall for?

256MiB.

About the relay logic, do we need to find proper local GPU? I don't know how to find the GPU which is nearest to IB, bypeps paper says this will be faster?

[Musisoul]

What input sizes do you need the alltoall for? Your algBW at max should be 25GBps/(N-1). For example, for 8 nodes, the maximum you can get at very large sizes is 3.57GBps algBW.

Regarding the proper algorithm for your use case: you might wanna use something like the 3-step algorithm for alltoall except that the relay logic needs to change. This means that we need to (1) gather cross-node traffic to a local GPU (let's say local GPU 0) and (2) then GPU 0 sends cross-node data to local GPU 0 on other nodes. (3) Lastly, GPU 0 scatters the data to everyone. Thus, the name 3-step algorithm.

Hope this helps.

As Jack47 says, the size we need is 256MiB. BTW, I have tried the nccl-tests(forked version) and compared the time and busbw between NCCL and MSCCL algos, both at out-of-place. It seems that MSCCL algos did not improve. I have used 8 nodes to test and the size was 256MiB. What do you mean by quoting

the current one you are currently using seems to be delivering a pretty good performance.

Thank you!

	time(us)	busbw(GB/s)
NCCL(out of place)	123058	2.15
MSCCL two_step(out of place)	165042	1.60
MSCCL three_step(out of place)	150965	1.78