run_custom_training packages parameter fails on OpenShift (read-only `/.local`)

[abhijeet-dhumal]

Description

When using run_custom_training(packages=["torch", "transformers"], ...) on OpenShift, the pre-script pip install step runs pip install --user which writes to /.local. Under OpenShift's restricted SCC, the root filesystem is read-only and /.local is not backed by a writable volume — the job fails immediately with:

PermissionError: [Errno 13] Permission denied: '/.local'

The volumes parameter does auto-inject a writable emptyDir at /workspace, but the emptyDir volumes defined by the user (e.g. for /.local) are not mounted as volumeMounts on the training container — only /workspace is.

Steps to Reproduce

1. On an OpenShift cluster with restricted SCC
2. Submit:

run_custom_training(
    runtime="torch-distributed",
    script="import torch; print(torch.__version__)",
    packages=["torch"],
    confirmed=True
)

3. Check logs → PermissionError: /.local

Expected Behavior

Packages are installed successfully and the training script runs.

Actual Behavior

Job fails at pip install with PermissionError.

Workaround

Do NOT use packages. Install inside the script to the writable /workspace/lib:

import subprocess, sys, os
lib_dir = '/workspace/lib'
os.makedirs(lib_dir, exist_ok=True)
subprocess.run([
    sys.executable, '-m', 'pip', 'install',
    '--target', lib_dir, '--quiet',
    'transformers', 'peft', 'trl'
], check=True)
sys.path.insert(0, lib_dir
)

MCP Server Version

No response

Python Version

Python-3.11

Kubernetes Version

No response

MCP Client

None

References

Proposed Fixes

• Documentation: Update platform-fixes.md resource to document this gap and the workaround
• System prompt: Add guidance to INSTRUCTION_SECTIONS["training"] telling agents not to use packages on OpenShift
• Log analysis: Add OpenShift-specific failure pattern to monitoring.py (_FAILURE_PATTERNS) so get_training_logs returns actionable hints
• Long-term: Change the SDK's pip install step to use --target=/workspace/lib instead of --user when running on OpenShift

[abhijeet-dhumal]

/good-first-issue

[google-oss-prow]

@abhijeet-dhumal:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

Details

In response to this:

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[priyank766]

/assign