Unify ARO and ROSA Helm charts under single charts

.cursor/skills/deploy-existing-openshift/SKILL.md

@@ -13,7 +13,52 @@ Before deploying, verify these automatically (do NOT ask the user unless somethi
 2. **Cluster access** — run `oc whoami` directly on the host (not inside the container). If it fails, ask the user to log in.
 3. **AI serving namespace** — always use `private-assistant-ai-serving` (the default). Do not ask.
 4. **DevSpace namespace** — ask the user for a suffix. The namespace will be `private-assistant-<suffix>` (e.g. if the user says "itay", the namespace is `private-assistant-itay`).
-5. **RHCL (AI Gateway)** — existing OpenShift does not install the RHCL *operator* via make. Confirm `oc get crd authpolicies.kuadrant.io`. The ai-serving chart creates a `Kuadrant` CR in `kuadrant-system` when `aiGateway.kuadrant.create=true`. IDE traffic defaults to `pca-ai-gateway` with per-DevSpaces API keys. llm-d Gateway is annotated `opendatahub.io/managed=false` so ODH does not attach conflicting AuthPolicies.
+5. **RHCL (AI Gateway)** — existing OpenShift does not install the RHCL *operator* via make. Confirm `oc get crd authpolicies.kuadrant.io`. The ai-serving chart creates a `Kuadrant` CR in `kuadrant-system` when `aiGateway.kuadrant.create=true` (the default for existing OCP). IDE traffic defaults to `pca-ai-gateway` with per-DevSpaces API keys. llm-d Gateway is annotated `opendatahub.io/managed=false` so ODH does not attach conflicting AuthPolicies.
+
+   **If `kuadrant-system` is already owned by another Helm release** (shared cluster), the install will fail with an ownership conflict. Detect with:
+   ```bash
+   oc get namespace kuadrant-system -o jsonpath='{.metadata.annotations.meta\.helm\.sh/release-name}' 2>/dev/null
+   ```
+   If it prints a different release name, add `HELM_ARGS='--set aiGateway.kuadrant.create=false'` to reuse the existing Kuadrant instance:
+   ```bash
+   make ai-serving-deploy-existing-openshift HELM_ARGS='--set aiGateway.kuadrant.create=false'
+   ```
+
+## ARO-specific overrides
+
+On ARO (Azure), the chart defaults use AWS storage/hardware. Add these to `HELM_ARGS` for any ARO deployment:
+
+```bash
+HELM_ARGS="--set storage.storageClass=managed-csi \
+  --set hardware.instanceTypeLabel=nvidia.com/gpu.product \
+  --set hardware.gpuProduct=NVIDIA-H100-NVL \
+  --set hardware.cpu.request=8 --set hardware.cpu.limit=16 \
+  --set hardware.memory.request=80Gi --set hardware.memory.limit=120Gi \
+  --set model.id=Qwen/Qwen3.6-35B-A3B-FP8 \
+  --set model.name=qwen36-vllm \
+  --set vllm.useCustomRuntime=true \
+  --set vllm.image=vllm/vllm-openai:v0.19.0 \
+  --set vllm.maxModelLen=262144 \
+  --set vllm.toolCallParser=qwen3_xml \
+  --set epp.enabled=false \
+  --set aiGateway.kuadrant.create=false"
+```
+
+Also enable TrustyAI in the DSC before deploying with guardrails (ARO only — RHOAI doesn't enable TrustyAI by default):
+```bash
+oc patch datasciencecluster default-dsc --type=merge \
+  -p '{"spec":{"components":{"trustyai":{"managementState":"Managed"}}}}'
+# Wait ~1 min for GuardrailsOrchestrator CRD to appear
+oc get crd guardrailsorchestrators.trustyai.opendatahub.io
+```
+
+GPU node management (ARO, `aro-pca-aue` cluster):
+```bash
+# Scale up before testing (H100 node costs money)
+oc scale machineset aro-pca-aue-cq6r2-gpu-h100 -n openshift-machine-api --replicas=1
+# Scale down when done — CRITICAL
+oc scale machineset aro-pca-aue-cq6r2-gpu-h100 -n openshift-machine-api --replicas=0
+```
 
 ## Deployment Steps
 
@@ -125,3 +170,14 @@ make devspace-undeploy-existing-openshift DEV_NAMESPACE=<DEV_NS>
 # Remove the AI serving stack (removes namespace)
 make ai-serving-undeploy-existing-openshift
 ```
+
+**If the namespace gets stuck terminating**, it is almost always a `GuardrailsOrchestrator` finalizer (`trustyai.opendatahub.io/gorch-finalizer`) blocking deletion. Check and clear it:
+```bash
+# Confirm the blocker
+oc get namespace <NS> -o jsonpath='{.status.conditions[?(@.type=="NamespaceContentRemaining")].message}'
+
+# Remove the finalizer so the namespace can complete termination
+oc patch $(oc get guardrailsorchestrators -n <NS> -o name) \
+  -n <NS> --type=merge -p '{"metadata":{"finalizers":[]}}'
+```
+This is safe — the TrustyAI operator has already been notified of deletion; patching the finalizer just unblocks the namespace controller.

AGENTS.md

@@ -32,17 +32,17 @@ ArgoCD syncs these Helm charts in order (via `pca-app-of-apps`):
 | `pca-platform-config` | 2 | Namespaces, HF token, DSC/DSCI, NFD, NVIDIA ClusterPolicy, CheCluster, OAuth HTPasswd, Maas gateway, LWS CR; optional `pca-guardrails` / `pca-mcp` | AI ns (default `ai-serving`); optional per-dev namespaces |
 | `pca-ai-serving` | 3 | PVC, HardwareProfile, LLMInferenceService (llm-d/vLLM), llm-d gateway + HTTPRoute, RHCL AI Gateway (`pca-ai-gateway`) + AuthPolicy; `pca-observability` (Grafana; optional Langfuse/OTel) | AI ns |
 | `pca-devspaces` | 4 | DevWorkspace, Roo/Continue/Cline ConfigMaps, per-ns API keys, RBAC; global DevSpaces ConfigMaps | Per-dev ns; globals in `openshift-devspaces` |
+| `pca-benchmarks` | 5 | GuideLLM sweep Job (one-shot; **disabled by default — opt-in via cloud values files**) | AI ns |
 
 ## Where each target deploys
 
-| Target | Charts |
-|--------|--------|
-| **ROSA / ARO** | All five via ArgoCD |
-| **Existing OpenShift** | `pca-platform-config`, `pca-ai-serving`, `pca-devspaces` only (Helm) |
+| Target | Charts | Cloud-specific config |
+|--------|--------|-----------------------|
+| **ROSA** | All six via ArgoCD (`charts/`) | `values-rosa.yaml` per chart |
+| **ARO** | All six via ArgoCD (`charts/`) | `values-aro.yaml` per chart |
+| **Existing OpenShift** | `pca-platform-config`, `pca-ai-serving`, `pca-devspaces` only (Helm) | `deploy_existing_openshift/values-*.yaml` |
 
-Existing OpenShift uses the charts under `PCA_Deployment_ROSA/charts` with overrides in `deploy_existing_openshift/`.
-
-ARO charts lag behind ROSA for most features; RHCL gateway pieces were mirrored for parity where practical. Full ARO migration to ROSA charts is still planned.
+Both ROSA and ARO use the same unified `charts/` directory at the repo root. Cloud-specific values (hardware, storage class, model, enabled features) are in `values-aro.yaml` / `values-rosa.yaml` within each chart. Terraform injects `gitops.cloud: aro|rosa` to select the right overlay via ArgoCD `valueFiles`. **`gitops.cloud` must be explicitly set — an empty or missing value causes a hard Helm render error in `pca-app-of-apps`.**
 
 ## ROSA / ARO — full from-scratch
 
@@ -53,7 +53,7 @@ Terraform provisions the cluster; GitOps (`pca-app-of-apps`) syncs the five char
 
 ## Existing OpenShift — Helm-only
 
-No infrastructure provisioning. Deploys onto a cluster that already has RHOAI, GPU operator, and DevSpaces installed. Uses ROSA charts via `make ai-serving-deploy-existing-openshift` (once) and `make devspace-deploy-existing-openshift` (one or more developers).
+No infrastructure provisioning. Deploys onto a cluster that already has RHOAI, GPU operator, and DevSpaces installed. Uses the unified `charts/` via `make ai-serving-deploy-existing-openshift` (once) and `make devspace-deploy-existing-openshift` (one or more developers).
 
 Deploy AI serving first, then each DevSpace into its own `DEV_NAMESPACE` (or the AI ns for a single-dev setup). The first DevSpace release owns the global ConfigMaps in `openshift-devspaces`; later ones must pass `HELM_ARGS='--set devspacesGlobalConfig.enabled=false'` or Helm conflicts.
 
@@ -75,21 +75,22 @@ Package lives in `tests/cluster-smoke/` (see its README). Optional Langfuse / OT
 ## Directory Structure
 
 ```
-PCA_Deployment_ROSA/          # Full ROSA (AWS) deployment — source of truth for charts
-├── terraform/                # Cluster provisioning (VPC, ROSA, GPU node pool)
-└── charts/
-    ├── pca-app-of-apps/      # Root ArgoCD AppProject + child Applications
-    ├── pca-operators/        # Operator Subscriptions (RHOAI, GPU, DevSpaces, NFD, RHCL, …)
-    ├── pca-platform-config/  # Namespace, RBAC, secrets, DSC (+ optional guardrails, pca-mcp)
-    ├── pca-ai-serving/       # LLMInferenceService, llm-d + pca-ai-gateway, pca-observability
-    │   └── charts/pca-observability/  # Grafana + optional Langfuse/OTel Collector
-    └── pca-devspaces/        # Per-developer DevWorkspaces + Roo/Continue/Cline + API keys
-
-PCA_Deployment_ARO/           # Full ARO (Azure) — charts lag; will migrate to ROSA charts
-├── terraform/
-└── charts/
-
-deploy_existing_openshift/    # Helm value overrides (reuses ROSA charts)
+charts/                       # Unified Helm charts — single set for ROSA and ARO
+├── pca-app-of-apps/          # Root ArgoCD AppProject + child Applications
+├── pca-operators/            # Operator Subscriptions (RHOAI, GPU, DevSpaces, NFD, RHCL, …)
+├── pca-platform-config/      # Namespace, RBAC, secrets, DSC (+ optional guardrails, pca-mcp)
+├── pca-ai-serving/           # LLMInferenceService, llm-d + pca-ai-gateway; optional EPP, TLS job
+│   └── charts/pca-observability/  # Grafana + optional Langfuse/OTel Collector
+├── pca-devspaces/            # Per-developer DevWorkspaces + Roo/Continue/Cline + API keys
+└── pca-benchmarks/           # GuideLLM sweep (optional; enabled in values-aro.yaml)
+
+PCA_Deployment_ROSA/          # ROSA (AWS) — Terraform only
+└── terraform/                # Cluster provisioning (VPC, ROSA HCP, GPU node pool)
+
+PCA_Deployment_ARO/           # ARO (Azure) — Terraform only
+└── terraform/                # Cluster provisioning (VNet, ARO cluster, GPU MachineSet)
+
+deploy_existing_openshift/    # Helm value overrides for existing OpenShift (no Terraform)
 ├── README.md                 # Deploy steps + parameters (incl. RHCL prerequisite)
 ├── values-platform-config.yaml
 ├── values-ai-serving.yaml

Makefile

@@ -8,7 +8,7 @@ PROJECT_DIR := $(shell pwd)
 AI_NAMESPACE ?= private-assistant-ai-serving
 HF_TOKEN ?= $(HUGGINGFACE_TOKEN)
 MCP_ENABLED ?= false
-CHARTS_DIR := PCA_Deployment_ROSA/charts
+CHARTS_DIR := charts
 SCRIPTS_DIR := PCA_Deployment_ROSA/scripts
 DEPLOY_VALUES_DIR := deploy_existing_openshift