The config has users specify the --gpus-per-node parameter however in the code it defaults to 8. This breaks on newer systems such as GB200 w/ 4 gpu's per node.
Error message is:
sbatch: error: Batch job submission failed: Requested node configuration is not available
I'll create a PR to update.
The config has users specify the
--gpus-per-nodeparameter however in the code it defaults to 8. This breaks on newer systems such as GB200 w/ 4 gpu's per node.training/large_language_model_pretraining/nemo/pretrain_llama31.py
Line 78 in 751d033
Error message is:
I'll create a PR to update.