memory issue

[LongjunGao]

"Hello, when I was running your code, I encountered a situation where the memory usage kept increasing until it filled up my entire memory, and then the process was killed. My server has 128GB of RAM and 8x A100 40GB GPUs. Did you encounter the same issue when you ran the code? How should I solve this?"issue

[Jamin-Yang]

I think the OOM error is caused by an uncontrolled caching mechanism in the data loading process. Because the cache is not size-limited, it continuously accumulates data throughout training.