infinite s3dlio list during training (retinanet)

[rasar07]

Issue description: 
We have 4 clients with 755GB memory per client. This results in 50 million .3MiB Jpg files to be ingested to read back for training. s3dlio lists back the bucket and it has taken more than 12 hours and it is still listing.

1. Datasize:
   mlpstorage closed training retinanet datasize --client-host-memory-in-gb 754 --max-accelerators 16 --accelerator-type b200 --num-client-hosts 4

gives:

RESULT: Number of training files: 50136788

2. Data generation:

mlpstorage closed training retinanet datagen object
--num-processes 64
--hosts gizmoc5,gizmoc6,gizmoc7,gizmoc8
--data-dir retinanet_64p
--params dataset.num_files_train=50136788
--params storage.storage_type=s3
--params storage.storage_root=retinanet1
--mpi-params "-x AWS_ACCESS_KEY_ID -x AWS_SECRET_ACCESS_KEY -x AWS_REGION -x S3_ENDPOINT_URIS -x S3DLIO_MAX_CONCURRENCY -x S3DLIO_MULTIPART_THRESHOLD -x S3DLIO_PART_SIZE -x S3DLIO_CONNECTION_TIMEOUT -x S3DLIO_READ_TIMEOUT -x AWS_S3_MULTIPART_THRESHOLD -x RUST_LOG --map-by slot --mca pml ob1 --mca btl tcp,self --mca btl_openib_allow_ib 0 --mca btl_openib_warn_no_device_params_found 0"
--allow-run-as-root[11:59 AM]export AWS_ACCESS_KEY_ID=user1

export AWS_SECRET_ACCESS_KEY=iOpHmc2afcbYuAfAapfWljfxtbHM6XYalMC2GwD6
export AWS_REGION=us-east-1
export S3_ENDPOINT_URIS= http://ip1:9020,http://ip2:9020,http://ip3:9020,http://ip4:9020
export S3DLIO_MAX_CONCURRENCY=32
export S3DLIO_MULTIPART_THRESHOLD=1073741824
export S3DLIO_PART_SIZE=268435456
export S3DLIO_CONNECTION_TIMEOUT=300
export S3DLIO_READ_TIMEOUT=300
export RUST_LOG=s3dlio=debug,aws_sdk_s3=debug

3. Training:

mlpstorage open training retinanet run object
--data-dir retinanet_64p
--results-dir /tmp/result.20260618.033618/mlperf/training/retinanet/run_20260618_033720
--accelerator-type b200
--num-accelerators 256
--client-host-memory-in-gb 755
--hosts gizmoc5,gizmoc6,gizmoc7,gizmoc8
--params storage.storage_type=s3
--params storage.storage_root=retinanet1
--params dataset.num_files_train=50204282
--mpi-params "-x AWS_ACCESS_KEY_ID -x AWS_SECRET_ACCESS_KEY -x AWS_REGION -x S3_ENDPOINT_URIS -x S3DLIO_MAX_CONCURRENCY -x S3DLIO_MULTIPART_THRESHOLD -x S3DLIO_PART_SIZE -x S3DLIO_CONNECTION_TIMEOUT -x S3DLIO_OPERATION_TIMEOUT_SECS -x S3DLIO_READ_TIMEOUT -x RUST_LOG --map-by slot --mca pml ob1 --mca btl tcp,self --mca btl_openib_allow_ib 0 --mca btl_openib_warn_no_device_params_found 0"
--allow-run-as-root

export AWS_ACCESS_KEY_ID=user1
export AWS_SECRET_ACCESS_KEY=iOpHmc2afcbYuAfAapfWljfxtbHM6XYalMC2GwD6
export AWS_REGION=us-east-1
export S3_ENDPOINT_URIS=http://ip1:9020,http://ip2:9020,http://ip3:9020,http://ip4:9020
export S3DLIO_MAX_CONCURRENCY=32
export S3DLIO_MULTIPART_THRESHOLD=1073741824
export S3DLIO_PART_SIZE=268435456
export S3DLIO_CONNECTION_TIMEOUT=600
export S3DLIO_OPERATION_TIMEOUT_SECS=600
export S3DLIO_READ_TIMEOUT=600
export RUST_LOG=s3dlio=info,aws_sdk_s3=warn

[russfellows]

This seems related to issues #449 and issues #466 , both of which were identified by @wolfgang-desalvador .

There is a PR that Wolfgang is working on which should address those issues, and hopefully this issue as well.

[russfellows]

The solution for this particular issue is two fold:

1. Generate the naming deterministically, just as the file / object generation is done
2. Check only a limited subset of files / objects, first, last and every 1,000th item.

This reduces the number of list requests by a factor of 1,000 or more, which should enable listing a subset of 50M items in approximately 1 or 2 minutes, which is reasonable.

Testing the solution now, current output looks like following:

[INFO] 2026-06-19 14:32:01 skip_listing [train]: validating 50,001 of 50,136,788 files
(first, last, every 1,000) via HEAD requests ...
[INFO] 2026-06-19 14:32:11 skip_listing [train]: 5,000/50,001 checked (10%) — 483 checks/s — ETA 93s — 0 failed so far
[INFO] 2026-06-19 14:32:22 skip_listing [train]: 10,000/50,001 checked (20%) — 489 checks/s — ETA 82s — 0 failed so far
...
[INFO] 2026-06-19 14:33:44 skip_listing [train]: validation complete — all 50,001 samples exist
(103.6s, 483 checks/s); 12,534,197 URIs ready for rank 0
(50,136,788 total across all ranks)

[russfellows]

This should be fixed with PR 26 and 27 in DLIO_local_changes:
mlcommons/DLIO_local_changes#26
mlcommons/DLIO_local_changes#27