When running the mimic training task for the h1_2 robot on NVIDIA H800 with --num-envs 2048, IsaacGym crashes immediately with GPU PhysX internal errors and segmentation fault.
The code works with smaller numbers of environments (1024), but crashes with just num-envs=2048
Command：
CUDA_VISIBLE_DEVICES=7 python protomotions/train_agent.py
--robot-name h1_2
--simulator isaacgym
--experiment-path examples/experiments/mimic/mlp.py
--experiment-name mimic
--motion-file data/motion_for_trackers/h1_2_random_subset_tiny.pt
--num-envs 2048
--batch-size 2048
--ngpu 1
Error Log：
/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (54) : internal error : SynchronizeStreams cuEventRecord failed
/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (60) : internal error : SynchronizeStreams cuStreamWaitEvent failed
/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2408) : internal error : memcpy failed fail!
700
/buildAgent/work/99bede84aa0a52c2/source/gpucommon/src/PxgCudaMemoryAllocator.cpp (59) : warning : Failed to allocate pinned memory.
Segmentation fault (core dumped)
When running the mimic training task for the h1_2 robot on NVIDIA H800 with --num-envs 2048, IsaacGym crashes immediately with GPU PhysX internal errors and segmentation fault.
The code works with smaller numbers of environments (1024), but crashes with just num-envs=2048
Command：
CUDA_VISIBLE_DEVICES=7 python protomotions/train_agent.py
--robot-name h1_2
--simulator isaacgym
--experiment-path examples/experiments/mimic/mlp.py
--experiment-name mimic
--motion-file data/motion_for_trackers/h1_2_random_subset_tiny.pt
--num-envs 2048
--batch-size 2048
--ngpu 1
Error Log：
/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (54) : internal error : SynchronizeStreams cuEventRecord failed
/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (60) : internal error : SynchronizeStreams cuStreamWaitEvent failed
/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2408) : internal error : memcpy failed fail!
700
/buildAgent/work/99bede84aa0a52c2/source/gpucommon/src/PxgCudaMemoryAllocator.cpp (59) : warning : Failed to allocate pinned memory.
Segmentation fault (core dumped)