Memory Requirements

[ellsong]

What are the system requirements outside of the GPU? I am running out of RAM on my 64 GB machine when running the benchmark with the following arguments:

sh evaluate.sh pistachio_reachable f2f

In WSL and PopOS it fails on a 44 GiB memory allocation for a numpy array with the below traceback:

Traceback (most recent call last):
  File "/home/gellson/code/desp/desp/experiments/evaluate.py", line 62, in <module>
    fwd_predictor = ForwardPredictor(
                    ^^^^^^^^^^^^^^^^^
  File "/home/gellson/code/desp/desp/experiments/../../desp/inference/forward_predictor.py", line 63, in __init__
    self.bb_predictor = BuildingBlockPredictor(
                        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gellson/code/desp/desp/experiments/../../desp/inference/building_block_predictor.py", line 49, in __init__
    bb_tensor = bb_fps.toarray().astype(np.float32)
                ^^^^^^^^^^^^^^^^
  File "/home/gellson/anaconda3/envs/desp/lib/python3.12/site-packages/scipy/sparse/_compressed.py", line 1181, in toarray
    out = self._process_toarray_args(order, out)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gellson/anaconda3/envs/desp/lib/python3.12/site-packages/scipy/sparse/_base.py", line 1301, in _process_toarray_args
    return np.zeros(self.shape, dtype=self.dtype, order=order)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 44.0 GiB for an array with shape (23079928, 256) and data type int64

[ykevu]

I see, loading the building block tensor definitely takes up quite a bit of CPU memory. Can you try going to the file desp/inference/building_block_predictor.py and changing line 49 to bb_tensor = bb_fps.astype(np.float32).toarray()?

[ellsong]

I am no longer seeing that error, but the process simply ends with the terminal saying Killed. Above that it shows bb_checkpoint = torch.load(model_path, map_location="cpu")

[ykevu]

I checked the peak CPU memory usage and it was a bit under 50 GB, which I'm assuming is too much for your machine? I will look into how to lower this since once the building blocks get shuttled to GPU the CPU memory footprint is only around 9 GB.

[ellsong]

Curious, I have 64 GB and have tried running it in a clean linux image. Appreciate the help!

[randyshee]

I also encountered the same issue with a 256 GB clean Linux image. I’ve included my solution in PR #3, which resolves the problem by converting the array in smaller chunks. Line 49, bb_tensor = bb_fps.toarray().astype(np.float32), is changed to the following:

# bb_tensor = bb_fps.toarray().astype(np.float32)
# Convert in smaller chunks to avoid memory spike
chunk_size = 10000  # Adjust this based on your memory/matrix dimensions
total_rows = bb_fps.shape[0]
bb_tensor = np.zeros((total_rows, bb_fps.shape[1]), dtype=np.float32)

for i in range(0, total_rows, chunk_size):
    end_idx = min(i + chunk_size, total_rows)
    bb_tensor[i:end_idx] = bb_fps[i:end_idx].toarray()
    if i % (chunk_size * 10) == 0:
        print(f"Processed {i}/{total_rows} rows...")