Benchmarking full N-body simulation on GPU clusters

[EiffL]

This issue is to track the work on benchmarking the new Horovod backend for GPU clusters and getting profiling information for FlowPM.

We want to do the following things:

• : Run profiler on https://github.com/DifferentiableUniverseInitiative/flowpm/blob/mesh_update/scripts/mesh_nbody_benchmark.py and https://github.com/DifferentiableUniverseInitiative/flowpm/blob/mesh_update/scripts/pyramid_nbody.py
• : Compare the performance of both implementations as a function of the mesh size
• : Identify the most costly parts of the computation

To learn how to do this profiling, keep an eye on DifferentiableUniverseInitiative/IDRIS-hackathon#2

[santiagocasas]

We will make a new bash profiling script based on the message sent by @EiffL on #cosmostat slack on Monday:

#!/bin/bash
#SBATCH --job-name=fft_benchmark     # nom du job
##SBATCH --partition=gpu_p2          # de-commente pour la partition gpu_p2
#SBATCH --ntasks=8                   # nombre total de tache MPI (= nombre total de GPU)
#SBATCH --ntasks-per-node=4          # nombre de tache MPI par noeud (= nombre de GPU par noeud)
#SBATCH --gres=gpu:4                 # nombre de GPU par nœud (max 8 avec gpu_p2)
#SBATCH --cpus-per-task=10           # nombre de coeurs CPU par tache (un quart du noeud ici)
##SBATCH --cpus-per-task=3           # nombre de coeurs CPU par tache (pour gpu_p2 : 1/8 du noeud)
# /!\ Attention, "multithread" fait reference a l'hyperthreading dans la terminologie Slurm
#SBATCH --hint=nomultithread         # hyperthreading desactive
#SBATCH --time=00:10:00              # temps d'execution maximum demande (HH:MM:SS)
#SBATCH --output=fft_benchmark%j.out # nom du fichier de sortie
#SBATCH --error=fft_benchmark%j.out  # nom du fichier d'erreur (ici commun avec la sortie)
#SBATCH -A ftb@gpu                   # specify the project
#SBATCH --qos=qos_gpu-dev            # using the dev queue, as this is only for profiling
# nettoyage des modules charges en interactif et herites par defaut
module purge
# chargement des modules
module load tensorflow-gpu/py3/2.4.1+nccl-2.8.3-1
# echo des commandes lancees
set -x
# JZ FIX
export TMPDIR=$JOBSCRATCH
ln -s $JOBSCRATCH /tmp/nvidia
# execution du code avec binding via bind_gpu.sh : 1 GPU pour 1 tache MPI.
srun --unbuffered --mpi=pmi2 -o fft_%t.log /gpfslocalsup/pub/idrtools/bind_gpu.sh nsys profile --stats=true -t nvtx,cuda,mpi -o result-%q{SLURM_TASK_PID} python -u fft_benchmark.py --mesh_shape="b1:2,b2:4" -
-layout="nx:b1,tny:b1,ny:b2,tnz:b2"

[santiagocasas]

We created a new script to benchmark pyramid_nbody.py and it works.
Here the script

#!/bin/bash
#SBATCH --job-name=pyramid_benchmark   # nom du job
##SBATCH --partition=gpu_p2          # de-commente pour la partition gpu_p2
#SBATCH --ntasks=4                  # nombre total de tache MPI (= nombre total de GPU)
#SBATCH --ntasks-per-node=4          # nombre de tache MPI par noeud (= nombre de GPU par noeud)
#SBATCH --gres=gpu:4                 # nombre de GPU par nœud (max 8 avec gpu_p2)
#SBATCH --cpus-per-task=10           # nombre de coeurs CPU par tache (un quart du noeud ici)
##SBATCH --cpus-per-task=3           # nombre de coeurs CPU par tache (pour gpu_p2 : 1/8 du noeud)
# /!\ Attention, "multithread" fait reference a l'hyperthreading dans la terminologie Slurm
#SBATCH --hint=nomultithread         # hyperthreading desactive
#SBATCH --time=00:10:00              # temps d'execution maximum demande (HH:MM:SS)
#SBATCH --output=pyramid_benchmark_new_%j.out # nom du fichier de sortie
#SBATCH --error=pyramid_benchmark_new_%j.out  # nom du fichier d'erreur (ici commun avec la sortie)
#SBATCH -A ftb@gpu                   # specify the project
#SBATCH --mail-user=denise.lanzieri@cea.fr    # send mail to user
#SBATCH --qos=qos_gpu-dev            # using the dev queue, as this is only for profiling

# nettoyage des modules charges en interactif et herites par defaut
module purge

# chargement des modules
module load tensorflow-gpu/py3/2.4.1+nccl-2.8.3-1

# echo des commandes lancees
set -x

# JZ FIX
export TMPDIR=$JOBSCRATCH
ln -s $JOBSCRATCH /tmp/nvidia
# execution du code avec binding via bind_gpu.sh : 1 GPU pour 1 tache MPI.

# execution du code avec binding via bind_gpu.sh : 1 GPU pour 1 tache MPI.
srun --unbuffered --mpi=pmi2 -o pyramid_%t.log /gpfslocalsup/pub/idrtools/bind_gpu.sh nsys profile --stats=true -t nvtx,cuda,mpi -o result-pyramid-%q{SLURM_TASK_PID} python -u pyramid_nbody.py

[santiagocasas]

We created a new script to benchmark mesh_nbody_benchmark.py and it works.
Here the script:

#!/bin/bash
#SBATCH --job-name=nbody_benchmark   # nom du job
##SBATCH --partition=gpu_p2          # de-commente pour la partition gpu_p2
#SBATCH --ntasks=4                  # nombre total de tache MPI (= nombre total de GPU)
#SBATCH --ntasks-per-node=4          # nombre de tache MPI par noeud (= nombre de GPU par noeud)
#SBATCH --gres=gpu:4                 # nombre de GPU par nœud (max 8 avec gpu_p2)
#SBATCH --cpus-per-task=10           # nombre de coeurs CPU par tache (un quart du noeud ici)
##SBATCH --cpus-per-task=3           # nombre de coeurs CPU par tache (pour gpu_p2 : 1/8 du noeud)
# /!\ Attention, "multithread" fait reference a l'hyperthreading dans la terminologie Slurm
#SBATCH --hint=nomultithread         # hyperthreading desactive
#SBATCH --time=00:10:00              # temps d'execution maximum demande (HH:MM:SS)
#SBATCH --output=nbody_benchmark_new_%j.out # nom du fichier de sortie
#SBATCH --error=nbody_benchmark_new_%j.out  # nom du fichier d'erreur (ici commun avec la sortie)
#SBATCH -A ftb@gpu                   # specify the project
#SBATCH --mail-user denise.lanzieri@cea.fr    # send mail to user
#SBATCH --qos=qos_gpu-dev            # using the dev queue, as this is only for profiling

# nettoyage des modules charges en interactif et herites par defaut
module purge

# chargement des modules
module load tensorflow-gpu/py3/2.4.1+nccl-2.8.3-1

# echo des commandes lancees
set -x

# JZ FIX
export TMPDIR=$JOBSCRATCH
ln -s $JOBSCRATCH /tmp/nvidia
# execution du code avec binding via bind_gpu.sh : 1 GPU pour 1 tache MPI.

# execution du code avec binding via bind_gpu.sh : 1 GPU pour 1 tache MPI.
srun --unbuffered --mpi=pmi2 -o mesh_nbody_%t.log /gpfslocalsup/pub/idrtools/bind_gpu.sh nsys profile --stats=true -t nvtx,cuda,mpi -o result-%q{SLURM_TASK_PID} python -u mesh_nbody_benchmark.py --nc=512 --batch_size=1 --nx=4 --ny=4 --hsize=32

[dlanzieri]

@EiffL @santiagocasas I have obtained some .qdrep files for our benchmark. But there are several things that I still don't understand.
In the path /gpfswork/rech/ftb/ulm75uc/repo/flowpm/scripts/mesh_nbody_out_3 there is the first, not-empty .qdrep file (result-75171.qdrep ).
My first question is: what are those other empty files ?
The mesh_nbody_benchmark_1751099.out looks like :

Loading tensorflow-gpu/py3/2.4.1+nccl-2.8.3-1
Loading requirement: gcc/8.3.1 cuda/10.2 nccl/2.8.3-1-cuda
cudnn/8.0.4.30-cuda-10.2 openmpi/4.0.2-cuda
+ export TMPDIR=/gpfsssd/jobscratch/1751099/
+ TMPDIR=/gpfsssd/jobscratch/1751099/
+ ln -s /gpfsssd/jobscratch/1751099/ /tmp/nvidia
+ srun --unbuffered --mpi=pmi2 -o mesh_nbody_%t.log /gpfslocalsup/pub/idrtools/bind_gpu.sh nsys profile --stats=true -t nvtx,cuda,mpi -o 'result-%q{SLURM_TASK_PID}' python -u mesh_nbody_benchmark.py --nc=512 --batch_size=1 --nx=2 --ny=2 --hsize=32
srun: error: r10i0n8: task 3: Exited with exit code 255
srun: Terminating job step 1751099.0
srun: error: r10i0n8: tasks 0-2: Terminated
srun: Force Terminated job step 1751099.0

Also, ~2 h later , I made the same mesh_nbody_benchmark_idris.sh run again, I got the following mesh_nbody_benchmark_ 1753425.out, but I got also a core-mesh_nbody_benc-10550-6 file and I didn't get the empty files or the .png file.

Loading tensorflow-gpu/py3/2.4.1+nccl-2.8.3-1
 Loading requirement: gcc/8.3.1 cuda/10.2 nccl/2.8.3-1-cuda
 cudnn/8.0.4.30-cuda-10.2 openmpi/4.0.2-cuda
+ export TMPDIR=/gpfsssd/jobscratch/1753425/
+ TMPDIR=/gpfsssd/jobscratch/1753425/
+ ln -s /gpfsssd/jobscratch/1753425/ /tmp/nvidia
+ srun --unbuffered --mpi=pmi2 -o mesh_nbody_%t.log /gpfslocalsup/pub/idrtools/bind_gpu.sh nsys profile --stats=true -t nvtx,cuda,mpi -o 'result-%q{SLURM_TASK_PID}' python -u mesh_nbody_benchmark.py --nc=512 --batch_size=1 --nx=2 --ny=2 --hsize=32
srun: error: r11i5n7: task 3: Exited with exit code 6
srun: Terminating job step 1753425.0
srun: error: r11i5n7: tasks 0-2: Terminated
srun: Force Terminated job step 1753425.0

However, I got a result-10423.sqlite not-empty, that I opened. All the results of this run are in the path gpfswork/rech/ftb/ulm75uc/repo/flowpm/scripts/mesh_nbody_out_4.

Then, I made the pyramid_nbody_benchmark_idris.sh run twice .
The first time I mistyped the full name of the module in the pyramid_nbody_benchmark_idris.sh file, but , nevertheless,I got the result-pyramid-76305.qdrep file (that you can find in /gpfswork/rech/ftb/ulm75uc/repo/flowpm/scripts/pyramid_nbody_out_3).
After correcting the .sh file, I got the result-pyramid-49953.qdrep (that you can find here /gpfswork/rech/ftb/ulm75uc/repo/flowpm/scripts/pyramid_nbody_out_4).

The .out files are :

Loading tensorflow-gpu/py3/2.4.1+nccl-2.8.3-1
Loading requirement: gcc/8.3.1 cuda/10.2 nccl/2.8.3-1-cuda
cudnn/8.0.4.30-cuda-10.2 openmpi/4.0.2-cuda
 
ERROR: Unable to locate a modulefile for 'nvidia-nsight-systems/2021'
+ export TMPDIR=/gpfsssd/jobscratch/1751703/
+ TMPDIR=/gpfsssd/jobscratch/1751703/
+ ln -s /gpfsssd/jobscratch/1751703/ /tmp/nvidia
+ srun --unbuffered --mpi=pmi2 -o pyramid_%t.log /gpfslocalsup/pub/idrtools/bind_gpu.sh nsys profile --stats=true -t nvtx,cuda,mpi -o 'result-pyramid-%q{SLURM_TASK_PID}' python -u pyramid_nbody.py
srun: error: r10i0n8: task 1: Exited with exit code 9
srun: Terminating job step 1751703.0
srun: error: r10i0n8: tasks 0,2-3: Terminated
srun: Force Terminated job step 1751703.0

and

Loading tensorflow-gpu/py3/2.4.1+nccl-2.8.3-1
 Loading requirement: gcc/8.3.1 cuda/10.2 nccl/2.8.3-1-cuda
cudnn/8.0.4.30-cuda-10.2 openmpi/4.0.2-cuda
+ export TMPDIR=/gpfsssd/jobscratch/1767083/
+ TMPDIR=/gpfsssd/jobscratch/1767083/
+ ln -s /gpfsssd/jobscratch/1767083/ /tmp/nvidia
+ srun --unbuffered --mpi=pmi2 -o pyramid_%t.log /gpfslocalsup/pub/idrtools/bind_gpu.sh nsys profile --stats=true -t nvtx,cuda,mpi -o 'result-pyramid-%q{SLURM_TASK_PID}' python -u pyramid_nbody.py
srun: error: r13i1n4: task 3: Exited with exit code 9
srun: Terminating job step 1767083.0
srun: error: r13i1n4: tasks 0-2: Terminated
srun: Force Terminated job step 1767083.0
 

I hope to understand:

1. If ad how the mistyped of the full name of the module effected the final .qdrep file
2. Why only 3 tasks are terminated (except in one .out file , the task 3 is not terminated)
3. The different exit codes
4. The empty files
5. Why I got different results with the same mesh_nbody_benchmark_idris.sh file

[EiffL]

ok, so first comment, to understand what's going on, you probably should look at the log files, they should be called something like mesh_nbody_%t.log because in the srun command we redirect all outputs to these logs, so you are not seeing here where the code actually crashes, and if it gives you particular errors.

Don't worry about the exit codes,they are not very important for us, and they don't tell us why the job is canceled, the log file should have more info towards the end.

A core- file is an indication of a segmentation fault that's a pretty big error happening somewhere in the code, but again, the log file should tell us more.

[dlanzieri]

I think I found it (mesh_nbody_3.log in /gpfswork/rech/ftb/ulm75uc/repo/flowpm/scripts/mesh_nbody_out_4)

bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-05-18 23:19:47.622246: W ./tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 17179869184
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Fatal Python error: Aborted

[EiffL]

ah that's interesting! this means that the simulation is running out of memory and dying, because not enough space on the GPU. You could try with a smaller simulation.

[dlanzieri]

ok, I heard that story once!

[dlanzieri]

Daily Report:
We made the mesh_nbody_new_benchmark_idris.job run with different configurations.
What follows is the summary of the results and the setting of our tests.

#SBATCH --ntasks=1                 
#SBATCH --nodes=1                 
#SBATCH --ntasks-per-node=1      
#SBATCH --gres=gpu:1                
#SBATCH --cpus-per-task=10           

with

--nc=128 --batch_size=1 --nx=1 --ny=1 --hsize=32 --nsteps=1

#SBATCH --ntasks=2                
#SBATCH --nodes=1                 
#SBATCH --ntasks-per-node=2      
#SBATCH --gres=gpu:2               
#SBATCH --cpus-per-task=10           

with

--nc=128 --batch_size=1 --nx=2--ny=1 --hsize=32 --nsteps=1

#SBATCH --ntasks=4                
#SBATCH --nodes=1                 
#SBATCH --ntasks-per-node=4     
#SBATCH --gres=gpu:4               
#SBATCH --cpus-per-task=10           

with

--nc=128 --batch_size=1 --nx=4--ny=1 --hsize=32 --nsteps=1

Run time: 1:35

#SBATCH --ntasks=4                
#SBATCH --nodes=1                 
#SBATCH --ntasks-per-node=4     
#SBATCH --gres=gpu:4               
#SBATCH --cpus-per-task=10           

with

--nc=128 --batch_size=1 --nx=1--ny=4 --hsize=32 --nsteps=1

Run time : 0:35

#SBATCH --ntasks=4                
#SBATCH --nodes=1                 
#SBATCH --ntasks-per-node=4     
#SBATCH --gres=gpu:4               
#SBATCH --cpus-per-task=10           

with

--nc=128 --batch_size=1 --nx=2--ny=2 --hsize=32 --nsteps=1

Run time: 1:25

We noticed :

• The artifacts arise when we start to use 4 gpu
• In the configurations nx=2 ny=2 these artifacts are visible also in the initial conditions
• With the same number of gpu the run time changes with different mesh configurations

Please @santiagocasas add some more If I forgot somethings or if I was wrong with some configurations descriptions!

[EiffL]

@b-remy in case you have not seen it, this is super useful I think, it shows you what configurations give weird results

[dlanzieri]

@EiffL I almost forgot, @santiagocasas and I also noticed that if we use (for instance) the following configuration:

#SBATCH --job-name=mesh_nbody_benchmark   # nom du job
##SBATCH --partition=gpu_p2          # de-commente pour la partition gpu_p2
#SBATCH --ntasks=4                  # nombre total de tache MPI (= nombre total de GPU)
#SBATCH --nodes=1                  # number of nodes
#SBATCH --ntasks-per-node=4          # nombre de tache MPI par noeud (= nombre de GPU par noeud)
#SBATCH --gres=gpu:4                 # nombre de GPU par n?~Sud (max 8 avec gpu_p2)
#SBATCH --cpus-per-task=10           # nombre de coeurs CPU par tache (un quart du noeud ici)

We find 80 CPUS (shouldn't we find 40? ):

Thu May 20 17:27:03 2021
JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON) CPUS
1781699 gpu_p13 mesh_nbo ucd93sf COMPLETI 0:35 15:00 1 r11i3n2 80