Undefined GLOO_USE_ROCM error together with HIP

[geraldog]

gloo/gloo/cuda.h

Line 23 in 70dc360

#if !GLOO_USE_CUDA

This line I believe is wrong and should be #if !defined(GLOO_USE_CUDA) instead.

The problem manifests when compiling PyTorch with HIP backend. Compiler spits out that GLOO_USE_ROCM isn't defined when obviously it is, because it is defined to 1 and ! negates that 1 with a 0 which leads to a false evaluation.

[dolpm]

gloo/gloo/cuda.h

Line 23 in 70dc360

#if !GLOO_USE_CUDA
This line I believe is wrong and should be #if !defined(GLOO_USE_CUDA) instead.

The problem manifests when compiling PyTorch with HIP backend. Compiler spits out that GLOO_USE_ROCM isn't defined when obviously it is, because it is defined to 1 and ! negates that 1 with a 0 which leads to a false evaluation.

if my understanding is correct, i think we'd want:

#if !GLOO_USE_CUDA && !GLOO_USE_ROCM
#error "Expected GLOO_USE_CUDA or GLOO_USE_ROCM to be defined"
#endif

some paths set these to zero, so it doesn't make sense to only check for the existence of the defn.

i also don't understand:

because it is defined to 1 and ! negates that 1 with a 0 which leads to a false evaluation.

#define GLOO_USE_CUDA 1
#if !GLOO_USE_CUDA
...
#endif

will not evaluate

do you have the actual stack trace on hand? maybe i'm missing something.

[geraldog]

i also don't understand:

because it is defined to 1 and ! negates that 1 with a 0 which leads to a false evaluation.

#define GLOO_USE_CUDA 1
#if !GLOO_USE_CUDA
...
#endif

will not evaluate

Oh, sorry @dolpm , you're correct on this, we actually want a false evaluation there. It was a bit late here in the night.

[dolpm]

i also don't understand:

because it is defined to 1 and ! negates that 1 with a 0 which leads to a false evaluation.

#define GLOO_USE_CUDA 1
#if !GLOO_USE_CUDA
...
#endif
will not evaluate

Oh, sorry @dolpm , you're correct on this, we actually want a false evaluation there. It was a bit late here in the night.

nw - lmk if #507 is sufficient to fix your issue?

[geraldog]

No #507 is not correct I think @dolpm because tools/amd_build/build_amd.py already correctly hipifies cuda.h into hip.h and takes care of that define.

My point is that I do not understand properly how the ! logical negation operator together with a literal value of 1 for the #define is somehow executing specifically inside pytorch build, but changing that to a #if !defined(GLOO_USE_CUDA) with the proper definition check preprocessor macro fixes the error.

EDIT: hipifies cuda.h into hip.h

[dolpm]

No #507 is not correct I think @dolpm because tools/amd_build/build_amd.py already correctly hipifies cuda.h into cuda.h and takes care of that define.

My point is that I do not understand properly how the ! logical negation operator together with a literal value of 1 for the #define is somehow executing specifically inside pytorch build, but changing that to a #if !defined(GLOO_USE_CUDA) with the proper definition check preprocessor macro fixes the error.

i would need more information here in order to reproduce (i.e., cmd, hashes, e.g., output etc.) - just built torch+hip w/o issue locally

[geraldog]

OK, here we go. HEAD is at 5cdb4b0418db ("Remove dead THNN/UCC/TH references from CMakeLists (#185062)").

PyTorch build command is USE_GLOO=1 USE_KINETO=0 USE_ROCM=1 USE_CUDA=0 ROCM_PATH=/usr PYTORCH_ROCM_ARCH=gfx900 pip install -e . -v --user --break-system-packages --no-build-isolation

And the error is:

In file included from /usr/include/gloo/hip_allreduce_ring_chunked.h:14, from /home/dumbo/pytorch/torch/csrc/distributed/c10d/ProcessGroupGlooCuda.cpp:6:
  /usr/include/gloo/hip.h:24:2: error: #error "Expected GLOO_USE_ROCM to be defined"
   24 | #error "Expected GLOO_USE_ROCM to be defined"

[dolpm]

hip_allreduce_ring_chunked

ahh... a few things i noticed here:

1. /usr/include/gloo/hip.h -- this is a system gloo install (as opposed to the one in pytorch/third_party/...) that's been hipified (it's hip.h not cuda.h)
2. GLOO_USE_ROCM=0 (i assume the source of truth is at /usr/include/gloo/config.h)

i've repro'd this locally by:

1. installing system gloo w/ -DUSE_ROCM=ON
2. reinstall system gloo w/o removing prev. w/ -DUSE_ROCM=ON
3. this leaves the hipified headers, but changes the GLOO_USE_ROCM value.

from our side, perhaps we should check whether the system GLOO_USE_ROCM/GLOO_USE_CUDA matches the values inline with the ongoing build - but to fix this, either uninstall the system gloo (and use the one that comes with pytorch) or do a fresh install w/ -DUSE_ROCM=ON