Skip to content

Shared memory issue (#120)#139

Open
SCiarella wants to merge 4 commits into
mainfrom
oom
Open

Shared memory issue (#120)#139
SCiarella wants to merge 4 commits into
mainfrom
oom

Conversation

@SCiarella

Copy link
Copy Markdown
Collaborator

This PR targets issue #120.

The test run (on Discord) with 128 MPI ranks fails because all ranks simultaneously load large (~1.2 GB) HDF5 opacity files into private RAM. With 5 molecules and 128 ranks, that's about 768 GB allocated on one node.

Every rank called self._spec_dict["xsecarr"][()] which reads the entire dataset into private RAM before moving it to shared memory. Fix: only the shared-memory root rank (rank 0 per node) loads the data from disk. Non-root ranks pass None to allocate_as_shared() and attach to shared memory without ever holding a private copy.

Also this line mem = GlobalCache()["xsec_in_memory"] or True is problematic because it always returns True, so it was impossible to ever disable in-memory loading.

I have also added some explicit close() for the HDF5 files.


With these modifications, the test does not fail anymore of oom issues, so I invite you to test this fix.


(Summary) Memory impact:

  • Before: ~768 GB (128 ranks x 5 molecules x ~1.2 GB each)
  • After: ~6 GB (1 copy per molecule in shared memory, shared across all ranks on the node)

@HannoSpreeuw

Copy link
Copy Markdown
Collaborator
  • Before: ~768 GB (128 ranks x 5 molecules x ~1.2 GB each)

  • After: ~6 GB (1 copy per molecule in shared memory, shared across all ranks on the node)

Wow, that's impressive.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants