Training benchmark shows zero storage traffic from Epoch 2 but reports misleadingly high AU — possible caching issue

[litianqi00315]

Description:

I'm testing the unet3d training benchmark and observed the following behavior:

• Epoch 1: Read traffic to the storage system is visible (confirmed via network card monitoring)
• Epochs 2–5: Zero network/storage traffic, yet the final reported AU (Accelerator Utilization) is very high

Environment:

• Number of client nodes: 16
• Memory per client: 256 GB
• Total dataset size: 37 TB (generated by 16 clients)
• Model: unet3d
  Questions:

1. With 37 TB of data across 16 clients (2.3 TB per client, 256 GB RAM each), the data should not fit in client memory. Could this be caused by storage-server-side caching (e.g., storage array
   DRAM/NVMe cache)? Or is the DLIO data loader holding internal buffers across epochs that bypass actual I/O?
2. The drop_caches call runs on a single rank and fails silently — is this a known limitation? Should it be executed on all nodes, and should it fail loudly when sudo is unavailable?
3. Should the training run command validate at startup whether the dataset meets the 5× memory requirement, and refuse to run if it doesn't?
4. For distributed/shared filesystem scenarios, is there a recommended way for users to verify that results are not contaminated by caching? (e.g., checking potential_caching in summary.json, or
   independently monitoring storage-side traffic?)

Thanks for any guidance!

[wolfgang-desalvador]

This relates to #455

[FileSystemGuy]

Thanks for the careful observations and the math — they're pointing at three separable problems. Direct answers, then proposed fixes.

Q1 — where's the data coming from on epochs 2–5?

Three candidates, in decreasing likelihood given your numbers:

1. Storage-server-side cache. 37 TB / 16 = 2.3 TB working set per client, well over your 256 GB RAM, so client page cache can't be holding it. But a modern array with multi-TB DRAM + NVMe tier can absolutely cache the entire 37 TB. DLIO can only flush client OS page cache; it has no visibility into the array.
2. drop_caches is running on only one node (see Q2). The other 15 nodes' page caches are never flushed. That doesn't fully explain it at 2.3 TB > 256 GB, but it means epoch-N reads on those 15 nodes are at least partially served from page cache — and if your per-rank working set is smaller than you expect (e.g., the rank's file shard is much smaller than total_files/comm_size, or shuffle is repeating a hot subset), it could fit.
3. Local-disk vs. remote-storage confusion. "Zero network traffic" is consistent with the data living on local disks. If you generated the dataset onto each client's local FS (rather than the remote storage system under test), there is no network traffic on any epoch — including epoch 1. Worth ruling out by checking where the data was actually written.

DLIO does not hold internal cross-epoch buffers. Worker processes are torn down and re-spawned per epoch by reconfigure() / init_loader(), so there's no Python-level data cache surviving between epochs.

Q2 — drop_caches on a single rank, silent failure: known bug, two parts

Confirmed both, in DLIO main.py:458-475 (currently pinned in storage):

• Part A — runs on global rank 0 only, not local rank 0 per node. In your 16-client run, only the launcher node's page cache is dropped. The other 15 nodes' page caches persist across epochs. This needs to be local_rank() == 0 (per-node leader) plus a barrier so no rank starts reading before all nodes' caches are flushed.
• Part B — failure is partly silent. When sudo -n fails, a logger.warning(...) fires on rank 0, then drop_caches_disabled = True and the run continues — silently — with stale cache. The warning is one line on rank 0 and easy to miss, and nothing in summary.json or the submission checker records that drop_caches never ran. Right behavior: also persist a drop_caches_failed count into summary.json and have the storage run-checker reject the result for CLOSED submissions (unless an explicit --allow-cache-failure is set for debug).

I'll send a fix to the DLIO fork for Part A + Part B.

Q3 — should the run command validate 5× at startup?

Yes, and today it doesn't. The check exists at datagen/datasize time (mlpstorage_py/rules/utils.py:69 — dataset_size_bytes = 5 * total_mem_bytes), but at run time mlpstorage_py/validation_helpers.py:386 is just a comment: "This is a soft check - full validation happens after the run." It should compute cluster memory × 5 from --client-host-memory-in-gb × num_hosts and refuse to start the run for CLOSED if dataset bytes < 5× — same arithmetic as datasize, no excuse not to gate it.

Q4 — how to verify results aren't contaminated by cache?

Today, two channels:

• summary.json → potential_caching is already populated by DLIO (statscounter.py:113-137). It compares per-node data_size_per_host_GB against host_memory_GB. With your numbers it should report 0 per node — confirm that to rule out client-side caching.
• Independently monitor storage-side traffic, which is what you've done. There is currently no way for mlpstorage to detect storage-server caching on its own; the array's behavior is opaque to the client.

What's missing on the storage side: mlpstorage_py/rules/run_checkers/training.py:200 has a placeholder check_file_system_caching() that does nothing. It should consume summary.json[potential_caching] per node and fail CLOSED if any node reports 1, and fail CLOSED if drop_caches_failed > 0 once Part B lands.

Concrete fixes I'll send

DLIO fork (mlcommons/DLIO_local_changes):

1. main.py — change drop_caches gate from self.my_rank == 0 to local_rank() == 0; preserve the existing "warn once" behavior per node; record drop_caches_failed per node into summary.json.

Storage repo:
2. mlpstorage_py/rules/run_checkers/training.py:check_file_system_caching — read summary.json[potential_caching] and the new drop_caches_failed; fail CLOSED if either flags caching.
3. mlpstorage_py/validation_helpers.py — implement the startup 5× check using client-host-memory-in-gb × num_hosts × 5; block CLOSED at run start if the dataset doesn't meet it (it'd just have hit the datasize check anyway — this catches users who skip datagen).

Wolfgang's note that this relates to #455: agreed loosely — his wdesalvador/fix-lazy-file-list branch also addresses per-node memory budget (#448), and once that lands the cluster-memory accounting these checks rely on becomes more trustworthy. I'm tracking that on #449.

Will reply with PR links once the DLIO change is up.

[russfellows]

This should be fixed with PR 26 and 27 in DLIO_local_changes:
mlcommons/DLIO_local_changes#26
mlcommons/DLIO_local_changes#27