[WIP] Reduce extra memory usage in scGW and GW+EDMFT workflows

[weiyiguo9]

Important

Work in progress. Implementation validation and memory profiling are still ongoing.

Goal

Eliminate avoidable extra and peak memory usage throughout the GW and EDMFT workflows, especially for large distributed calculations where MPI communication can create substantial temporary allocations.

Main changes

• make memory-intensive MPI collective paths more memory friendly
• split large MPI all-reduce operations into bounded chunks to limit collective scratch memory
• stream distributed-array redistribution through bounded peer-to-peer tiles instead of allocating full send and receive pack buffers
• reuse redistribution metadata and avoid full-size intermediate arrays in the GW Sigma-divergence correction
• bound temporary memory in one-body upfolding, two-body downfolding, partial inverse Fourier transforms, and DIIS residual storage
• add focused regression coverage for distributed layouts, shared-array reductions, embedding projectors, and bounded DIIS history

Status

This PR is still under active development. Focused regression tests have been added, but end-to-end GW and EDMFT workflow validation and memory profiling are not yet complete.

Current capacity milestone

Preliminary WIP result: an SrVO3 calculation with a 10x10x10 k-point grid and 300 bands can currently run within the following regular-node allocation:

• 8 Snellius Genoa regular (thin CPU) nodes
• 144 MPI processes per node, 1,152 MPI processes in total
• 2.2 GB memory budget per MPI process/allocated core, approximately 316.8 GB per node

For context, the current Snellius genoa partition provides 192 CPU cores and 336 GiB of allocatable RAM per node. The underlying regular Genoa node has two 96-core AMD EPYC 9654 processors at 2.4 GHz, 384 GiB of physical DDR5 memory, and ConnectX-7 NDR networking (200 Gbps within a rack and 100 Gbps across racks). See the official Snellius partition documentation and hardware documentation.

This records the current runnable memory envelope only. End-to-end validation, memory profiling, scaling measurements, and scientific acceptance are still in progress.