test(e2e): define GPU validation image artifacts

[elezar]

Description

Define the driver-agnostic GPU validation image artifacts needed by GPU e2e tests.

Issue #1472 will add sandbox-side GPU execution validation. Those tests should consume prebuilt sandbox images rather than building validation images during the test run. This task defines the image contract, repository layout, and initial image artifacts that can validate the execution flow before a CUDA-specific reference image is enforced in GPU CI.

Context

Parent roadmap: #1444
Related GPU validation test issue: #1472

The validation image contract should stay independent of the compute driver. Docker is the first driver integration for #1472, but the same image contract should be usable later by Podman, Kubernetes, and VM-backed sandboxes.

The image base is not part of the contract. The important properties are that the image is usable directly as an OpenShell sandbox image, runs without network access inside the sandbox, and emits stable output that the e2e harness can assert.

Proposed Scope

Add GPU validation image specifications under a driver-neutral location:

e2e/gpu/
  images/
    smoke-pass/
      README.md
      Dockerfile
    smoke-fail/
      README.md
      Dockerfile
    cuda-basic/
      README.md
      Dockerfile

Start with simple fixture images to validate the basic sandbox execution flow:

• smoke-pass: default command prints OPENSHELL_GPU_WORKLOAD_SUCCESS and exits 0.
• smoke-fail: default command prints a stable diagnostic marker and exits non-zero.

These fixtures do not prove GPU functionality. They exist to validate image publishing, sandbox-image compatibility, and e2e assertion plumbing.

Define the first CUDA validation image contract separately in cuda-basic/. The CUDA image should eventually run a deterministic CUDA workload, emit OPENSHELL_GPU_WORKLOAD_SUCCESS on success, and return non-zero with useful diagnostics on failure.

Definition of Done

• Document the GPU validation image contract: default command, success marker, failure behavior, and stdout/stderr expectations.
• Add the driver-neutral e2e/gpu/images/ layout for validation image specs.
• Add a simple always-pass fixture image that can run directly as an OpenShell sandbox image.
• Add a simple always-fail fixture image, or document why the pass fixture is sufficient for the initial flow validation.
• Define the initial cuda-basic validation image requirements.
• Document how validation images are built and published.
• Document how tests should reference published images by immutable tag or digest.

[elezar]

📋 triage-agent

Triage Assessment

Classification: feature-valid

Summary

This is a valid prerequisite feature for the GPU execution validation work in #1472. Defining driver-neutral workload image artifacts first gives the e2e tests a concrete sandbox image contract to consume, and the proposed smoke fixtures let the test plumbing be validated before a CUDA-specific reference image is enforced in CI.

Investigation

• There is no existing e2e/gpu/ image contract or OPENSHELL_GPU_WORKLOAD_SUCCESS fixture in the repository today, so this issue would introduce a new driver-neutral test artifact area rather than refactor existing files.
• Existing e2e tests already exercise sandbox image selection through openshell sandbox create --from ..., so the proposed workload images can be consumed by the same mechanism that Define GPU validation tests for GPU-enabled drivers #1472 should use.
• Current Docker build and publish helpers under tasks/docker.toml and tasks/scripts/docker-*.sh are focused on gateway/supervisor images. The implementation should either add explicit GPU workload image tasks/workflows or document the manual build/publish commands clearly enough that Define GPU validation tests for GPU-enabled drivers #1472 can reference immutable tags or digests.
• The proposed smoke-pass and smoke-fail fixtures are useful even though they do not prove GPU functionality: they validate default-command execution, image compatibility, success-marker assertions, and failure diagnostics without depending on CUDA runtime availability.
• The cuda-basic image should stay a contract/spec initially unless the PR also supplies a real deterministic CUDA workload and publish path. That keeps test(e2e): define GPU validation image artifacts #1476 from blocking on CUDA image infrastructure that may need separate review.

Recommendation

Route this as the first buildable step before #1472. Suggested labels: feature request, topic:testing, test:e2e, test:e2e-gpu, and area:build. Do not add state:agent-ready during triage; that remains the human approval step before implementation.

[elezar]

🏗️ build-plan

Implementation Plan

Issue type: feat
Complexity: Medium
Confidence: High — design agreed locally; CUDA static-link validation remains the main implementation detail

Summary

Define GPU workload image artifacts in this repository first so #1472 has concrete images to consume, while preserving an immediate follow-up migration path to NVIDIA/OpenShell-Community. The PR will add smoke pass/fail images, a working cuda-basic image using CUDA 12.8 samples, a local Docker/Podman build task, an env manifest for built image refs, and documentation for local validation and future publishing.

Scope

• e2e/gpu/README.md: document the workload image contract, success/failure markers, /usr/local/bin/openshell-gpu-workload, direct container-engine validation, latest.env, and the planned OpenShell-Community migration.
• e2e/gpu/images/smoke-pass/: add a source directory for gpu-workload-smoke-pass, based on ghcr.io/nvidia/openshell-community/sandboxes/base:latest, whose workload prints OPENSHELL_GPU_WORKLOAD_SUCCESS and exits 0.
• e2e/gpu/images/smoke-fail/: add a source directory for gpu-workload-smoke-fail, based on the same community base image, whose workload prints OPENSHELL_GPU_WORKLOAD_FAILURE, emits a stable diagnostic, and exits non-zero.
• e2e/gpu/images/cuda-basic/: add a source directory for gpu-workload-cuda-basic. It builds CUDA 12.8 deviceQuery and vectorAdd samples from NVIDIA/cuda-samples tag v12.8, copies the binaries into the community base final image, and runs both through /usr/local/bin/openshell-gpu-workload.
• tasks/test.toml and a helper script under tasks/scripts/: add mise run e2e:gpu:images:build. Build all image directories containing a Dockerfile by default. Use OPENSHELL_GPU_WORKLOAD_IMAGES=smoke-pass,smoke-fail to select source directories.
• e2e/gpu/images/.build/latest.env: generate a shell-safe env file with the image refs that were just built. Do not add latest.json in this first implementation.

Implementation Decisions

• Image directories use short source names: smoke-pass, smoke-fail, and cuda-basic.
• Image tags use prefixed image names: gpu-workload-smoke-pass, gpu-workload-smoke-fail, and gpu-workload-cuda-basic.
• Local tags use localhost/openshell/<image>:<tag>. The default tag is the source commit short SHA, with -dirty appended for dirty local-tree builds. OPENSHELL_GPU_WORKLOAD_IMAGE_TAG=<tag> can override this for manual workflows.
• The build task uses the existing tasks/scripts/container-engine.sh helper and does not introduce another engine selector. Callers may set existing CONTAINER_ENGINE=docker|podman; unset follows the helper's current auto-detection. Consistency with the Rust e2e helper is tracked separately in fix(e2e): align container engine selection across helpers #1481.
• The first implementation builds from local e2e/gpu/images only. Add Git-ref source support when the image definitions migrate to OpenShell-Community.
• Add only the build task in this PR; do not add a clean task yet.
• cuda-basic uses a CUDA 12.8 base builder image plus explicit minimal build dependencies such as build-essential, cmake, curl, g++, and cuda-nvcc-12-8. Do not use a CUDA devel image by default.
• Prefer static CUDA runtime linkage for copied sample binaries. Do not add CUDA runtime library copying unless static linkage proves impossible.
• Do not vendor GPU driver libraries such as libcuda.so.1; those must come from host GPU runtime/CDI injection. CUDA compat libraries are the only acceptable future exception and should not be needed for CUDA 12.8.

Implementation Steps

1. Add the e2e/gpu/images/ layout with SPDX headers, per-image READMEs, workload scripts, and Dockerfiles.
2. Implement smoke-pass and smoke-fail on top of ghcr.io/nvidia/openshell-community/sandboxes/base:latest.
3. Implement cuda-basic as a multi-stage image: CUDA 12.8 base builder, pinned CUDA samples v12.8, final OpenShell community base image, wrapper script that runs deviceQuery then vectorAdd.
4. Tag images deterministically as localhost/openshell/<image>:<short-sha> or localhost/openshell/<image>:<short-sha>-dirty, unless OPENSHELL_GPU_WORKLOAD_IMAGE_TAG is set.
5. Generate e2e/gpu/images/.build/latest.env with OPENSHELL_E2E_GPU_SMOKE_PASS_IMAGE, OPENSHELL_E2E_GPU_SMOKE_FAIL_IMAGE, OPENSHELL_E2E_GPU_CUDA_WORKLOAD_IMAGE, tag/source metadata, and the selected container engine.
6. Document direct Docker/Podman validation commands, including CDI GPU validation for cuda-basic.
7. Document future publishing under OpenShell-Community, with tests consuming immutable tags or digests.

Test Plan

• Unit tests: N/A — this adds image definitions, scripts, tasks, and documentation rather than Rust/Python library logic.
• Integration checks: Run mise run e2e:gpu:images:build and verify that all selected images build and latest.env identifies the resulting image refs.
• Direct container validation: Run smoke images with Docker or Podman and verify marker output and exit codes. On a CUDA-capable host with CDI configured, run cuda-basic directly with --device nvidia.com/gpu=all and verify deviceQuery, vectorAdd, and OPENSHELL_GPU_WORKLOAD_SUCCESS.
• Repository checks: Run mise run pre-commit.
• Deferred: Full OpenShell sandbox-side GPU execution remains in Define GPU validation tests for GPU-enabled drivers #1472.

Risks & Open Questions

• cuda-basic final stage uses the OpenShell community base image, not a CUDA runtime image. The implementation must verify that the copied sample binaries run with static CUDA runtime linkage.
• Existing OpenShell sandbox create does not run OCI image CMD; the contract must expose /usr/local/bin/openshell-gpu-workload for OpenShell e2e tests while still setting CMD for direct container-engine validation.
• The image definitions intentionally start in this repo for fast iteration, but should migrate to OpenShell-Community immediately after the contract proves useful.
• LSM compatibility risk is low: the images and build tasks do not depend on host process identity, /proc/<pid>/exe, or cross-label process inspection.

Documentation Impact

• Add internal e2e documentation under e2e/gpu/.
• Add per-image READMEs for local build/run behavior and CUDA samples licensing notes.
• No published user docs under docs/ are expected for this internal e2e artifact contract.

---

Revision 2 — recorded implementation decisions from local design review
Revision 1 — initial plan