RAM vs. Disk Dilemma #63
Answered
by
Eamon2009
eamonsippy
asked this question in
Q&A
|
If i feed a 50GB directory of .bin shard files into the DataLoader, how much physical RAM does the program actually consume to load the dataset, and what specific OS mechanism makes this possible? |
Answered by
Eamon2009
Jul 28, 2026
Replies: 1 comment
|
If you load a 50GB directory of shards, the program uses almost zero additional physical RAM for the dataset itself. It achieves this using OS-level memory mapping (mmap on Linux/macOS, MapViewOfFile on Windows). The OS creates a virtual memory space pointing to the files on disk, and physical RAM is only consumed page-by-page (usually in ~10KB chunks) at the exact moment the CPU requests those specific tokens during training. |
0 replies
Answer selected by
eamonsippy
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
If you load a 50GB directory of shards, the program uses almost zero additional physical RAM for the dataset itself. It achieves this using OS-level memory mapping (mmap on Linux/macOS, MapViewOfFile on Windows). The OS creates a virtual memory space pointing to the files on disk, and physical RAM is only consumed page-by-page (usually in ~10KB chunks) at the exact moment the CPU requests those specific tokens during training.