How to set the environment variables for all_to_all_perf profiling?

[Chen-Chang]

Hi! Thank you for developing this helpful profiling tool.

I would like to profile the detailed information of alltoall communication. However, I noticed that the default environment variable settings are configured for allreduce. I have experimented with several settings, and so far, only "-DENABLE_NPKIT_EVENT_NET_SEND_ENTRY xxx" has produced useful profiling results.

Could you please provide the appropriate variable settings for alltoall and send-receive profiling on multiple nodes? The test bin is the official nccl-test alltoall_perf.

Thank you for your assistance!

[yzygitzh]

Hi,

For alltoall, I would recommend using

-DENABLE_NPKIT_EVENT_SEND_ENTRY
-DENABLE_NPKIT_EVENT_SEND_EXIT
-DENABLE_NPKIT_EVENT_RECV_ENTRY
-DENABLE_NPKIT_EVENT_RECV_EXIT

for GPU events, and

-DENABLE_NPKIT_EVENT_NET_SEND_ENTRY
-DENABLE_NPKIT_EVENT_NET_SEND_EXIT
-DENABLE_NPKIT_EVENT_NET_RECV_ENTRY
-DENABLE_NPKIT_EVENT_NET_RECV_EXIT

for net events.

[Chen-Chang]

I have tested the settings for GPU events, but the json output is "{"traceEvents": [], "displayTimeUnit": "ns"}" and npkit_dump dir is empty.
Should I change the NCCL_PROTO and NCCL_ALGO settings?

[yzygitzh]

There should always be files in NPKIT_DUMP_DIR. Which NPKit version did you use?

[Chen-Chang]

NPKit_NCCL

[yzygitzh]

Got it. That version only LL and LL128 protocol are supported for those events.

We recommend using the NPKit for MSCCL instead of NPKit for NCCL because the latter is no longer actively maintained. MSCCL and NPKit for it are being actively developed, and MSCCL in terms of functionality is a superset of NCCL.

[Chen-Chang]

Thanks a lot! Can I profile the trace file of the original nccl implementation in msccl framework?

[yzygitzh]

It’s better to process the trace files using their corresponding scripts.

[Chen-Chang]

I have changed the NCCL_PROTO from "Simple" to "LL". The output trace of NET events is normal, but there are some errors when generating GPU events.