S3 Environment variables are not being used

[keithpij]

The DLIO_benchmark code is not honoring environment variables containing S3 connection info. I had to edit the yaml and put my connection information in the yaml to run successfully.

The environment variables are:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_ENDPOINT_URL

If you run mlpstorage, you get an error saying it needs a .env file, but even after setting up the .env file, it still does not work.

[FileSystemGuy]

Thanks @keithpij — confirmed two underlying bugs that together produce what you saw:

Bug 1 — .env is required even when AWS_* are already exported

mlpstorage_py/benchmarks/dlio.py:159-174 (in _apply_object_storage_params()) raises FileNotFoundError unconditionally when load_dotenv() finds no file, with no check for "are the required envs already in os.environ?". So even after export AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… AWS_ENDPOINT_URL=…, you still hit the .env error.

Bug 2 — MPI launcher doesn't forward AWS_* to ranks

The MPI command builder at mlpstorage_py/utils.py:573-624 adds -n, -host, --bind-to, --map-by, --mca btl etc., but never -x for any S3/AWS env var. On multi-node runs, OpenMPI does not propagate env vars from the launcher to remote ranks by default — they have to be explicitly forwarded. So after .env loads creds into the launcher's os.environ, ranks on remote hosts still don't see them, and DLIO reads only what's in the YAML (which is why putting the creds in YAML worked for you).

Proposed fix — security-aware

The obvious "fix" is -x AWS_SECRET_ACCESS_KEY=… — but that puts the secret value on argv. Anyone with ps auxe on the launcher and every remote host can see it, and our run logfile (which gets archived into the submission tarball) would contain it too. We're not going to do that.

OpenMPI's -x VAR (name-only form, no =VALUE) does the right thing: mpirun reads VAR from its own environment, ships it to remote orte daemons over its internal channel, and sets it in the spawned rank's environment. The value never appears on a command line. The literal command we build and log contains only the name (-x AWS_SECRET_ACCESS_KEY), which is safe to log.

Residual exposure (unavoidable with any env-var-based credential delivery): once a rank is running, ps auxe on its host shows the secret in the process env. If your threat model doesn't allow this, the right answer is AWS_PROFILE + ~/.aws/credentials (or IAM instance roles on EC2) — neither requires a secret on the wire. We'll make sure AWS_PROFILE is also forwarded so this path works.

Five concrete changes

1. mlpstorage_py/utils.py — in the MPI prefix builder, when data_access_protocol != 'file', append name-only -x VAR flags for each var that is actually set in os.environ. Candidate set: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_ENDPOINT_URL, AWS_ENDPOINT_URL_S3, AWS_REGION, AWS_CA_BUNDLE, AWS_PROFILE, AWS_SHARED_CREDENTIALS_FILE, BUCKET, STORAGE_LIBRARY, STORAGE_URI_SCHEME, S3_ENDPOINT_URIS, S3_ENDPOINT_TEMPLATE, S3_ENDPOINT_FILE, S3_ENDPOINT, S3_LOAD_BALANCE_STRATEGY. Never the =VALUE form.

2. mlpstorage_py/benchmarks/dlio.py:_apply_object_storage_params() — skip the .env-required path when either AWS_ACCESS_KEY_ID or AWS_PROFILE is already set in the shell. If .env is present but doesn't supply the missing vars, fail with a specific message naming which vars are still unset (instead of a generic "no .env").

3. mlpstorage_py/storage_config.py:_redact() — replace the single length-only redaction with per-var policy:

   Variable	Log form
   AWS_SECRET_ACCESS_KEY	[SET — 40 chars, sha256:8a1f…] (length + 8-hex fingerprint — lets you diff against "the key I expected" without revealing it)
   AWS_ACCESS_KEY_ID	AKIA****XYZW (first-4 + last-4)
   AWS_ENDPOINT_URL	full value (it's a hostname, not a secret — and this is the field most likely to be wrong, so showing it solves 80% of "did .env load right?" questions)
   AWS_REGION, BUCKET, STORAGE_LIBRARY, STORAGE_URI_SCHEME, AWS_CA_BUNDLE, AWS_PROFILE	full value

4. mlpstorage_py/run_summary.py — the comment at lines 7-8 admits the summary prints before .env is loaded. Move the summary call (or add a second post-load summary) so the resolved S3 config — using the redactions above — gets printed after _apply_object_storage_params() runs. That gives you a clear "credentials detected: AWS_ACCESS_KEY_ID=AKIA****XYZW, AWS_ENDPOINT_URL=https://…" banner at startup so you can immediately see whether .env parsing worked.

5. mlpstorage_py/utils.py (logging) — log the built mpirun command exactly once at info level. The command will contain only -x VAR names, which is safe to archive in the run logfile.

What I need from you to verify

• Single-node or multi-node setup? (single-node usually inherits env via fork; multi-node is where -x becomes mandatory and is my strongest suspect for "still does not work")
• Roughly what your .env looked like (KEY=VALUE format, any quoting/spaces around =, file made on Windows / shared via copy-paste from a Mac — python-dotenv parses some of these differently than bash export and silently loads empty values). A redacted snippet is enough.

Treating this as the storage-side fix tracker; will open a PR with the five changes above once we've confirmed the multi-node detail.

[russfellows]

This should be fixed with PR 26 and 27 in DLIO_local_changes:
mlcommons/DLIO_local_changes#26
mlcommons/DLIO_local_changes#27