diff --git a/hardware/index.rst b/hardware/index.rst
index 615d48a..8d16e70 100644
--- a/hardware/index.rst
+++ b/hardware/index.rst
@@ -59,6 +59,17 @@ There are:
- 480 GB LPDDR5X RAM
- 1x Mellanox CONNECTX-7 NDR200 (100Gb/s due to existing network) InfiniBand port
+- 1x ``gh`` node containing
+
+ - 2x `NVIDIA Grace Hopper Superchips `_ (GH200 144G HBM3e)
+
+ - 1x NVIDIA Grace aarch64 CPU @ 3.483 GHz (72 Arm Neoverse V2 cores)
+ - 1x NVIDIA H100 144GB with 900 GB/s NVLink-C2C
+
+ - 960 GB LPDDR5X RAM
+ - 2x Mellanox CONNECTX-7 NDR200 (100Gb/s due to existing network) InfiniBand port
+
+
The Mellanox EDR InfiniBand interconnect is organised in a 2:1 block fat
tree topology. GPUDirect RDMA transfers are supported on the 32 ``gpu``
diff --git a/usage/index.rst b/usage/index.rst
index 95aa896..ac54923 100644
--- a/usage/index.rst
+++ b/usage/index.rst
@@ -344,11 +344,11 @@ Part of, or an entire node
#SBATCH --time=0:15:0 # Run for a max of 15 minutes
# Node resources:
- # 1 gpu per node
+ # (choose between 1-2 gpus per node)
#SBATCH --partition=ghtest # Choose either "gh" or "ghtest"
#SBATCH --nodes=1 # Resources from a single node
- #SBATCH --gres=gpu:1 # One GPU per node (plus 100% of node CPU and RAM per GPU)
+ #SBATCH --gres=gpu:1 # One GPU per node (plus 72 CPU cores and 480GB RAM)
# Run commands:
@@ -406,7 +406,7 @@ CPUs/GPUs across one or more machines, via ``bede-mpirun``:
#SBATCH --partition=gh # Choose either "gh" or"ghtest" partition type
#SBATCH --nodes=2 # Resources from two nodes
- #SBATCH --gres=gpu:1 # 1 GPU per node (plus 100% of node CPU and RAM per node)
+ #SBATCH --gres=gpu:1 # One GPU per node (plus 72 CPU cores and 480GB RAM)
# Run commands:
@@ -416,7 +416,7 @@ CPUs/GPUs across one or more machines, via ``bede-mpirun``:
.. note::
- There are only ``2`` ``gh`` nodes currently available for batch jobs in Bede. As a result multi-node Grace-Hopper jobs may queue for a significant time.
+ There are only ``8`` ``gh`` nodes currently available for batch jobs in Bede. As a result multi-node Grace-Hopper jobs may queue for a significant time.
The ``bede-mpirun`` command takes both ordinary ``mpirun`` arguments and
the special ``--bede-par `` option, allowing control over how
@@ -506,11 +506,29 @@ Jobs are scheduled subject to Slurm's `Multifactor Priority Plugin `_, containing two 72-core 64-bit ARM CPUs. Further details are listed on the :ref:`hardware` page.
-Each Grace-Hopper node contains a single `Grace Hopper Superchip `_, containing one 72-core 64-bit ARM CPU and one 96GB Hopper GPU with NVLink-C2C providing 900GB/s of bidirectional bandwidth between the CPU and GPU. Further details are listed on the :ref:`hardware` page.
+There are two types of Grace-Hopper node:
+
+- 7 nodes:
+
+ - 1x `Grace Hopper Superchip `_, each with:
+
+ - one 72-core 64-bit ARM CPU
+ - one 96GB Hopper GPU
+ - NVLink-C2C providing 900GB/s of bidirectional bandwidth between the CPU and GPU.
+
+- 1 node:
+
+ - 2x `Grace Hopper Superchip `_, each with:
+
+ - one 72-core 64-bit ARM CPU
+ - one 144GB Hopper GPU
+ - NVLink-C2C providing 900GB/s of bidirectional bandwidth between the CPU and GPU.
+
+Further details are listed on the :ref:`hardware` page.
Connecting to the ``ghlogin`` node
@@ -528,7 +546,7 @@ Batch Jobs
To submit a job to a Grace Hopper compute node, you can use ``sbatch`` or ``srun`` as normal from within a ``ghlogin`` session.
Alternatively, use the ``ghbatch`` or ``ghrun`` commands from a Bede login node.
-Your job submission scripts should specify the ``--partition=gh`` or ``--partition=ghtest``.
+Your job submission scripts should specify the ``--partition=gh`` or ``--partition=ghtest``. Jobs will be allocated a Grace-Hopper with either 96GB or 144GB of GPU memory. To specifically request a type, replace ``--gres=gpu:1`` in your submission script with ``--gres=gpu:gh200:1`` (for 96GB), or ``--gres=gpu:gh200_144g:1`` (for 144GB).
Further details and example batch job submission scripts are provided in the :ref:`usage_requesting_resources` section above.