From 58a7e6cecaeca06c7306622d20aa28831870cede Mon Sep 17 00:00:00 2001 From: Mark Dixon Date: Tue, 30 Sep 2025 15:33:06 +0100 Subject: [PATCH 1/2] Added new Grace Hopper node with 2x superchips Contains 2x superchips, each with its own 72 core Grace CPU, 480GB system memory and Hopper GPU. Unlike the existing Grace Hoppers, these Hoppers have 144GB instead of 96GB RAM. --- hardware/index.rst | 11 +++++++++++ usage/index.rst | 28 +++++++++++++++++++++------- 2 files changed, 32 insertions(+), 7 deletions(-) diff --git a/hardware/index.rst b/hardware/index.rst index 615d48a..8d16e70 100644 --- a/hardware/index.rst +++ b/hardware/index.rst @@ -59,6 +59,17 @@ There are: - 480 GB LPDDR5X RAM - 1x Mellanox CONNECTX-7 NDR200 (100Gb/s due to existing network) InfiniBand port +- 1x ``gh`` node containing + + - 2x `NVIDIA Grace Hopper Superchips `_ (GH200 144G HBM3e) + + - 1x NVIDIA Grace aarch64 CPU @ 3.483 GHz (72 Arm Neoverse V2 cores) + - 1x NVIDIA H100 144GB with 900 GB/s NVLink-C2C + + - 960 GB LPDDR5X RAM + - 2x Mellanox CONNECTX-7 NDR200 (100Gb/s due to existing network) InfiniBand port + + The Mellanox EDR InfiniBand interconnect is organised in a 2:1 block fat tree topology. GPUDirect RDMA transfers are supported on the 32 ``gpu`` diff --git a/usage/index.rst b/usage/index.rst index 95aa896..738e09a 100644 --- a/usage/index.rst +++ b/usage/index.rst @@ -344,11 +344,11 @@ Part of, or an entire node #SBATCH --time=0:15:0 # Run for a max of 15 minutes # Node resources: - # 1 gpu per node + # (choose between 1-2 gpus per node) #SBATCH --partition=ghtest # Choose either "gh" or "ghtest" #SBATCH --nodes=1 # Resources from a single node - #SBATCH --gres=gpu:1 # One GPU per node (plus 100% of node CPU and RAM per GPU) + #SBATCH --gres=gpu:1 # One GPU per node (plus 72 CPU cores and 480GB RAM) # Run commands: @@ -406,7 +406,7 @@ CPUs/GPUs across one or more machines, via ``bede-mpirun``: #SBATCH --partition=gh # Choose either "gh" or"ghtest" partition type #SBATCH --nodes=2 # Resources from two nodes - #SBATCH --gres=gpu:1 # 1 GPU per node (plus 100% of node CPU and RAM per node) + #SBATCH --gres=gpu:1 # One GPU per node (plus 72 CPU cores and 480GB RAM) # Run commands: @@ -416,7 +416,7 @@ CPUs/GPUs across one or more machines, via ``bede-mpirun``: .. note:: - There are only ``2`` ``gh`` nodes currently available for batch jobs in Bede. As a result multi-node Grace-Hopper jobs may queue for a significant time. + There are only ``8`` ``gh`` nodes currently available for batch jobs in Bede. As a result multi-node Grace-Hopper jobs may queue for a significant time. The ``bede-mpirun`` command takes both ordinary ``mpirun`` arguments and the special ``--bede-par `` option, allowing control over how @@ -506,11 +506,25 @@ Jobs are scheduled subject to Slurm's `Multifactor Priority Plugin `_, containing two 72-core 64-bit ARM CPUs. Further details are listed on the :ref:`hardware` page. -Each Grace-Hopper node contains a single `Grace Hopper Superchip `_, containing one 72-core 64-bit ARM CPU and one 96GB Hopper GPU with NVLink-C2C providing 900GB/s of bidirectional bandwidth between the CPU and GPU. Further details are listed on the :ref:`hardware` page. +There are two types of Grace-Hopper node. 7 nodes contain: + +- 1x `Grace Hopper Superchip `_ + + - one 72-core 64-bit ARM CPU + - one 96GB Hopper GPU + - NVLink-C2C providing 900GB/s of bidirectional bandwidth between the CPU and GPU. + +- 2x `Grace Hopper Superchip `_ + + - one 72-core 64-bit ARM CPU + - one 144GB Hopper GPU + - NVLink-C2C providing 900GB/s of bidirectional bandwidth between the CPU and GPU. + +Further details are listed on the :ref:`hardware` page. Connecting to the ``ghlogin`` node @@ -528,7 +542,7 @@ Batch Jobs To submit a job to a Grace Hopper compute node, you can use ``sbatch`` or ``srun`` as normal from within a ``ghlogin`` session. Alternatively, use the ``ghbatch`` or ``ghrun`` commands from a Bede login node. -Your job submission scripts should specify the ``--partition=gh`` or ``--partition=ghtest``. +Your job submission scripts should specify the ``--partition=gh`` or ``--partition=ghtest``. Jobs will be allocated a Grace-Hopper with either 96GB or 144GB of GPU memory. To specifically request a type, replace ``--gres=gpu:1`` in your submission script with ``--gres=gpu:gh200:1`` (for 96GB), or ``--gres=gpu:gh200_144g:1`` (for 144GB). Further details and example batch job submission scripts are provided in the :ref:`usage_requesting_resources` section above. From ce9fb89281c6e328e69e571a63128e969f773ed6 Mon Sep 17 00:00:00 2001 From: Mark Dixon Date: Tue, 30 Sep 2025 16:01:50 +0100 Subject: [PATCH 2/2] Corrected a set of hardware list bullet points --- usage/index.rst | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/usage/index.rst b/usage/index.rst index 738e09a..ac54923 100644 --- a/usage/index.rst +++ b/usage/index.rst @@ -510,19 +510,23 @@ Bede contains 1 NVIDIA Grace-CPU node and 8 NVIDIA Grace-Hopper nodes. Each Grace-CPU node node contains a single `Grace Superchip `_, containing two 72-core 64-bit ARM CPUs. Further details are listed on the :ref:`hardware` page. -There are two types of Grace-Hopper node. 7 nodes contain: +There are two types of Grace-Hopper node: -- 1x `Grace Hopper Superchip `_ +- 7 nodes: - - one 72-core 64-bit ARM CPU - - one 96GB Hopper GPU - - NVLink-C2C providing 900GB/s of bidirectional bandwidth between the CPU and GPU. + - 1x `Grace Hopper Superchip `_, each with: -- 2x `Grace Hopper Superchip `_ + - one 72-core 64-bit ARM CPU + - one 96GB Hopper GPU + - NVLink-C2C providing 900GB/s of bidirectional bandwidth between the CPU and GPU. - - one 72-core 64-bit ARM CPU - - one 144GB Hopper GPU - - NVLink-C2C providing 900GB/s of bidirectional bandwidth between the CPU and GPU. +- 1 node: + + - 2x `Grace Hopper Superchip `_, each with: + + - one 72-core 64-bit ARM CPU + - one 144GB Hopper GPU + - NVLink-C2C providing 900GB/s of bidirectional bandwidth between the CPU and GPU. Further details are listed on the :ref:`hardware` page.