WIP: feat(recipes): add A100 EKS training Kubeflow overlay chain

[yuanchen8911]

Summary

Add A100 EKS overlays (issue #1002): the EKS Ubuntu Kubeflow training leaf plus its ancestor chain and the cross-cutting deployment-phase floor. Modeled on the H100/H200 EKS training overlays.

Motivation / Context

a100 is a declared accelerator in pkg/recipe/criteria.go but has zero overlays in recipes/overlays/, so aicr recipe --accelerator a100 --service eks ... cannot resolve. Companion to the A100 OKE (#1294) and AKS (#1295) PRs; this slice covers EKS.

Fixes: N/A (incremental — part of #1002)
Related: #1002, #1294, #1295, #1306, #969, #1256

A100 overlay series — tracked in #1002: #1294 (OKE) · #1295 (AKS) · #1305 (EKS) ← this PR · #1306 (GKE)

Coordination: a100-any.yaml (the cross-service A100 deployment floor) is byte-identical across #1294, #1295, and this PR. Only one needs to introduce it — whichever lands first, the others drop the duplicate on rebase.

Type of Change

• New feature (non-breaking change that adds functionality)

Component(s) Affected

• Recipe engine / data (pkg/recipe)

Implementation Notes

New overlays (reuse existing eks/eks-training parents and values-eks-training.yaml — no new component values files):

• a100-any — deployment-phase floor: 4 standard checks + Deployment.gpu-operator.version >= v24.6.0 (H100/H200-generation baseline; A100 operator-supported since v22.9).
• a100-eks-training — base: eks-training; K8s >= 1.30. A100 has no NVLink ComputeDomain requirement, so it keeps the EKS training baseline rather than H100/H200's 1.32.4. gpu-operator cdi + gdrcopy, nfd topologyUpdater. Conformance mirrors the H100/H200 EKS training set.
• a100-eks-ubuntu-training — + os-ubuntu mixin.
• a100-eks-ubuntu-training-kubeflow — + platform-kubeflow (Kubeflow Trainer for distributed TrainJob).

Key decisions documented in-file:

• Nodewright tuning uses the nvidia-tuned generic profile (tuning-generic.yaml, accelerator: generic), mirroring rtx-pro-6000-eks. nvidia-setup ships baked-in (service, accelerator) configs only for eks-h100 / eks-gb200; unlike H200 (same Hopper HGX platform, reuses the h100 profile), A100 is Ampere with no matching profile. The generic profile is the supported baseline for such targets: governor=performance, iommu=pt, BBR, without the HGX-specific IB/hugepage/CPU-isolation settings the h100/gb200 profiles bake in. Note this is a safe baseline, not HGX-optimal — p4d/p4de A100 multi-node training would benefit from IB/NVLink tuning the generic profile omits; a dedicated A100 profile is a possible follow-up.
• Performance gating intentionally omitted. The H100/H200 EKS sibling pins nccl-all-reduce-bw >= 300, calibrated on 8-GPU Hopper NVLink nodes with EFA — neither fabric-class aware (nccl-all-reduce-bw training gate is a fixed absolute fabric-specific busbw value applied to SKU-agnostic recipes → false-fails EKS/H100 small SKUs #1256) nor valid for A100. An A100-on-EKS NCCL baseline is deferred to a follow-up.

Testing

go test ./pkg/recipe/...                 # PASS (incl. TestOverlayValidationPhaseFloor auto-discovery)
yamllint recipes/overlays/a100-*.yaml    # clean
# End-to-end resolution of the new leaf:
aicr recipe --service eks --accelerator a100 --os ubuntu --intent training --platform kubeflow
#   -> components=14 overlays=8; K8s '>= 1.30'; Deployment.gpu-operator.version '>= v24.6.0';
#      kubeflow-trainer present; nodewright-customizations renders tuning-generic (accelerator=generic);
#      gpu-operator inherits values-eks-training.yaml

Full make qualify not required: this touches only YAML overlay files (zero .go changes), so the Go lint/test/e2e gates cannot regress from it. The embedded overlays are exercised by go test ./pkg/recipe/... (passes) and yamllint (clean). No docs/ page enumerates individual overlay leaves, so no doc update is needed.

Risk Assessment

• Low — Additive overlays only; no existing recipe or Go path changes. Easy to revert.

Rollout notes: Additive. Other A100 EKS leaves (plain training, inference, dynamo) and remaining services tracked under #1002.

Checklist

• Tests pass locally (go test ./pkg/recipe/...)
• Linter passes (yamllint clean; no .go files changed)
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality (covered by existing auto-discovery TestOverlayValidationPhaseFloor)
• I updated docs if user-facing behavior changed (N/A — no leaf enumeration in docs)
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S)

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR introduces four new recipe overlay manifests for A100 accelerator workloads in the AICR repository. A wildcard a100-any overlay establishes baseline deployment validation checks and a gpu-operator version constraint (>= v24.6.0) for all A100 recipes. An a100-eks-training overlay inherits from the eks-training base, targets EKS workloads with A100 accelerator and training intent, enforces Kubernetes >= 1.30, configures gpu-operator Helm settings (enables cdi and gdrcopy), and specifies intent-layer conformance validation checks. Two platform specializations follow: a100-eks-ubuntu-training applies Ubuntu OS constraints via the os-ubuntu mixin, and a100-eks-ubuntu-training-kubeflow adds Kubeflow platform support via os-ubuntu and platform-kubeflow mixins. All variants enforce Kubernetes >= 1.30.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Suggested labels

size/M

Suggested reviewers

• xdu31
• mchmarny
• njhensley

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title clearly describes the main change: adding A100 EKS training Kubeflow overlay configuration files to enable recipe resolution.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description provides detailed context: it adds A100 EKS overlays to enable aicr recipe --accelerator a100 --service eks resolution, specifies four new YAML files with rationale, documents key design decisions (generic tuning profile, omitted performance gating), lists testing performed, and explicitly ties changes to the file summaries in the changeset.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

Recipe evidence check

Affected leaf overlays: 3

Recipe	Pointer	Verify	Digest match
a100-eks-training	⚠️ missing	—	—
a100-eks-ubuntu-training-kubeflow	⚠️ missing	—	—
a100-eks-ubuntu-training	⚠️ missing	—	—

How to refresh evidence

Run on a cluster matching the recipe's criteria:

aicr snapshot -o snapshot.yaml
aicr validate \
  -r recipes/overlays/<slug>.yaml \
  -s snapshot.yaml \
  --emit-attestation ./out \
  --push ghcr.io/<your-fork>/aicr-evidence
cp ./out/pointer.yaml recipes/evidence/<slug>.yaml

This gate is warning-only and never blocks merge. See ADR-007 for the trust model.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@recipes/overlays/a100-eks-ubuntu-training.yaml`:
- Around line 36-38: Remove the redundant K8s.server.version constraint from the
a100-eks-ubuntu-training overlay: delete the duplicated constraints entry that
sets "K8s.server.version >= 1.30" in the a100-eks-ubuntu-training specialization
and rely on the version floor inherited from the ancestor
a100-eks-training.yaml; only keep an explicit constraint here if this
specialization needs a stricter (higher) minimum than the ancestor.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 8c71c1c5-ab8f-479b-a9cc-e7093dd44fff

📥 Commits

Reviewing files that changed from the base of the PR and between 51ef12d and 6b4a4bd.

📒 Files selected for processing (4)

• recipes/overlays/a100-any.yaml
• recipes/overlays/a100-eks-training.yaml
• recipes/overlays/a100-eks-ubuntu-training-kubeflow.yaml
• recipes/overlays/a100-eks-ubuntu-training.yaml

[coderabbitai]

♻️ Duplicate comments (2)

recipes/overlays/a100-eks-ubuntu-training-kubeflow.yaml (1)

38-40: 🧹 Nitpick | 🔵 Trivial | ⚡ Quick win

Redundant constraint inherited from ancestor overlay.

The K8s.server.version >= 1.30 constraint is already defined in the ancestor a100-eks-training overlay (line 34-35) and inherited through the overlay chain (a100-eks-training → a100-eks-ubuntu-training → this file). Repeating it creates a maintenance burden—if the version floor is raised, multiple files must be updated. Remove this redundant constraint block and rely on inheritance unless this specialization requires a stricter (higher) minimum version.

♻️ Proposed cleanup

  mixins:
    - os-ubuntu
    - platform-kubeflow

- # A100 + EKS specific constraints (not covered by mixin)
- constraints:
-   - name: K8s.server.version
-     value: ">= 1.30"
-
  componentRefs: []

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@recipes/overlays/a100-eks-ubuntu-training-kubeflow.yaml` around lines 38 -
40, Remove the redundant K8s.server.version constraint block from the
overlays/a100-eks-ubuntu-training-kubeflow overlay: locate the constraints:
section and delete the entry with name: K8s.server.version and value: ">= 1.30"
(it’s inherited from the a100-eks-training overlay); if this overlay truly needs
a stricter minimum, replace the value instead of duplicating the same
constraint.

recipes/overlays/a100-eks-ubuntu-training.yaml (1)

36-38: 🧹 Nitpick | 🔵 Trivial | ⚡ Quick win

Redundant constraint inherited from base overlay.

The K8s.server.version >= 1.30 constraint is already defined in the base a100-eks-training overlay (line 34-35). Due to overlay inheritance, this constraint is automatically inherited by all descendants. Repeating it here creates a maintenance burden—if the version floor is raised, multiple files must be updated. Remove this redundant constraint block and rely on inheritance unless this specialization requires a stricter (higher) minimum version.

♻️ Proposed cleanup

  mixins:
    - os-ubuntu

- # A100 + EKS specific constraints (not covered by mixin)
- constraints:
-   - name: K8s.server.version
-     value: ">= 1.30"
-
  componentRefs: []

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@recipes/overlays/a100-eks-ubuntu-training.yaml` around lines 36 - 38, Remove
the redundant constraints block that defines "K8s.server.version >= 1.30" from
the a100-eks-ubuntu-training overlay (the "- name: K8s.server.version / value:
\">= 1.30\"" entry) so the overlay inherits the setting from the base
a100-eks-training overlay; only reintroduce a constraints entry here if you
intend to enforce a stricter minimum version, and after removal run the overlay
validation to confirm no other duplicates remain.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In `@recipes/overlays/a100-eks-ubuntu-training-kubeflow.yaml`:
- Around line 38-40: Remove the redundant K8s.server.version constraint block
from the overlays/a100-eks-ubuntu-training-kubeflow overlay: locate the
constraints: section and delete the entry with name: K8s.server.version and
value: ">= 1.30" (it’s inherited from the a100-eks-training overlay); if this
overlay truly needs a stricter minimum, replace the value instead of duplicating
the same constraint.

In `@recipes/overlays/a100-eks-ubuntu-training.yaml`:
- Around line 36-38: Remove the redundant constraints block that defines
"K8s.server.version >= 1.30" from the a100-eks-ubuntu-training overlay (the "-
name: K8s.server.version / value: \">= 1.30\"" entry) so the overlay inherits
the setting from the base a100-eks-training overlay; only reintroduce a
constraints entry here if you intend to enforce a stricter minimum version, and
after removal run the overlay validation to confirm no other duplicates remain.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 289c0246-c2c5-4d06-a957-ba6cf54c2d6f

📥 Commits

Reviewing files that changed from the base of the PR and between 6b4a4bd and 331ae98.

📒 Files selected for processing (4)

• recipes/overlays/a100-any.yaml
• recipes/overlays/a100-eks-training.yaml
• recipes/overlays/a100-eks-ubuntu-training-kubeflow.yaml
• recipes/overlays/a100-eks-ubuntu-training.yaml