So, there are no issues when CUDA_VISIBLE_DEVICES is set to a single integer index such as 0.
The issue is when CUDA_VISIBLE_DEVICES is set to a UUID which is also a valid way of setting it as per NVIDIA documentation:
https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-environment-variables
https://docs.nvidia.com/deploy/topics/topic_5_2_1.html
If CUDA_VISIBLE_DEVICES is set to a UUID, the program will fail with error:
RuntimeError: Attempting to deserialize object on CUDA device 0 but torch.cuda.device_count() is 0. Please use torch.load with map_location to map your storages to an existing device.
If I then set CUDA_VISIBLE_DEVICES to 0 and re-run again, the program works fine.
I can imagine when running this locally this wouldn't be too much of an issue, but when running in a HPC system this poses an issue.
For example, in a server with 8 GPU's, if you're assigned a UUID of a given GPU, and the system utilises cgroups, it's very difficult to know which GPU ID the given UUID is coming from.
If the program correctly parses UUID's as well as integers indices that would solve the issue.
So, there are no issues when
CUDA_VISIBLE_DEVICESis set to a single integer index such as0.The issue is when
CUDA_VISIBLE_DEVICESis set to a UUID which is also a valid way of setting it as per NVIDIA documentation:https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-environment-variables
https://docs.nvidia.com/deploy/topics/topic_5_2_1.html
If
CUDA_VISIBLE_DEVICESis set to a UUID, the program will fail with error:RuntimeError: Attempting to deserialize object on CUDA device 0 but torch.cuda.device_count() is 0. Please use torch.load with map_location to map your storages to an existing device.If I then set
CUDA_VISIBLE_DEVICESto 0 and re-run again, the program works fine.I can imagine when running this locally this wouldn't be too much of an issue, but when running in a HPC system this poses an issue.
For example, in a server with 8 GPU's, if you're assigned a UUID of a given GPU, and the system utilises cgroups, it's very difficult to know which GPU ID the given UUID is coming from.
If the program correctly parses UUID's as well as integers indices that would solve the issue.