Skip to content

CUDA out of memory #16

Description

@talonmyburgh

It would be better if using --cupy handled chunking loads to device rather than failing due to Out of memory allocation. This could maybe be easily done by moving to Jax or an alternative GPU acceleration library.

Traceback (most recent call last):
  File "/home/cosmic/anaconda3/envs/blri/bin/blri_pycorr", line 6, in <module>
    sys.exit(correlate_cli())
             ^^^^^^^^^^^^^^^
  File "/home/cosmic/anaconda3/envs/blri/lib/python3.12/site-packages/blri/entrypoints/correlate.py", line 584, in correlate_cli
    correlate(
  File "/home/cosmic/anaconda3/envs/blri/lib/python3.12/site-packages/blri/entrypoints/correlate.py", line 403, in correlate
    for correlation_tuple in correlation_iter.data():
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cosmic/anaconda3/envs/blri/lib/python3.12/site-packages/blri/entrypoints/correlate.py", line 221, in data
    datablock = dsp.correlate(datablock, conjugation_convention_flip=self.invert_correlation_conjugation)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cosmic/anaconda3/envs/blri/lib/python3.12/site-packages/blri/dsp.py", line 198, in correlate
    corr = compy.zeros(
           ^^^^^^^^^^^^
  File "/home/cosmic/anaconda3/envs/blri/lib/python3.12/site-packages/cupy/_creation/basic.py", line 250, in zeros
    a = cupy.ndarray(shape, dtype, order=order)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "cupy/_core/core.pyx", line 167, in cupy._core.core.ndarray.__new__
  File "cupy/_core/core.pyx", line 254, in cupy._core.core._ndarray_base._init
  File "cupy/cuda/memory.pyx", line 875, in cupy.cuda.memory.alloc
  File "cupy/cuda/memory.pyx", line 1579, in cupy.cuda.memory.MemoryPool.malloc
  File "cupy/cuda/memory.pyx", line 1600, in cupy.cuda.memory.MemoryPool.malloc
  File "cupy/cuda/memory.pyx", line 1271, in cupy.cuda.memory.SingleDeviceMemoryPool.malloc
  File "cupy/cuda/memory.pyx", line 1292, in cupy.cuda.memory.SingleDeviceMemoryPool._malloc
  File "cupy/cuda/memory.pyx", line 1499, in cupy.cuda.memory.SingleDeviceMemoryPool._try_malloc
  File "cupy/cuda/memory.pyx", line 1511, in cupy.cuda.memory.SingleDeviceMemoryPool._try_malloc
cupy.cuda.memory.OutOfMemoryError: Out of memory allocating 10,905,169,920 bytes (allocated so far: 11,325,009,920 bytes, limit set to: 16,891,248,640 bytes).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions