feat: default model to Qwen3.6-35B-A3B + GPU Operator v26.3 + H100

PCA_Deployment_ROSA/README.md

@@ -44,7 +44,7 @@ The end result is a ROSA HCP cluster running:
 │  Terraform (Phase 1 — Run once)                                  │
 │  ├── AWS VPC, Subnets, NAT Gateway                               │
 │  ├── ROSA HCP Cluster (STS/OIDC)                                 │
-│  ├── Machine Pools (workers, GPU L40S, opt. Inferentia)          │
+│  ├── Machine Pools (workers, GPU H100, opt. Inferentia)          │
 │  ├── HTPasswd IDP + Developer Users                              │
 │  └── OpenShift GitOps Operator + App-of-Apps bootstrap           │
 ├──────────────────────────────────────────────────────────────────┤
@@ -104,7 +104,7 @@ You need the following tools installed on your local machine before starting.
 | 6 | `kubeseal` | latest (optional) | Sealed Secrets for production |
 
 **Accounts required:**
-- **AWS account** with permissions to create VPC, IAM roles, and EC2 instances (including `g6e.2xlarge` GPU instances)
+- **AWS account** with permissions to create VPC, IAM roles, and EC2 instances (including `p5.4xlarge` GPU instances)
 - **Red Hat account** with ROSA entitlement (https://console.redhat.com)
 - **HuggingFace account** with an API token for model downloads (https://huggingface.co/settings/tokens)
 
@@ -298,7 +298,7 @@ default_worker_max_replicas  = 6
 
 # GPU pool
 gpu_pool_enabled  = true
-gpu_instance_type = "g6e.2xlarge"
+gpu_instance_type = "p5.4xlarge"
 gpu_pool_replicas = 1
 
 # Inferentia pool (set to true if you have inf2 quota)
@@ -650,9 +650,9 @@ echo "Console: $(terraform output -raw cluster_console_url)"
 ### Wave 3 — AI Serving (`pca-ai-serving`)
 
 - **PVC**: 100Gi `model-cache` on `gp3-csi` (persists model weights across restarts)
-- **ServingRuntime + InferenceService** (`qwen3-coder`): `Qwen/Qwen3.6-35B-A3B-FP8` (custom vLLM runtime; API name matches `model.id`) with vLLM args:
+- **ServingRuntime + InferenceService** (`qwen3-coder`): `Qwen/Qwen3.6-35B-A3B-FP8` with custom vLLM v0.19.0 runtime:
   - `--tool-call-parser qwen3_xml --reasoning-parser qwen3`
-  - `--max-model-len 32768 --gpu-memory-utilization 0.90`
+  - `--max-model-len 262144 --gpu-memory-utilization 0.90`
   - `--enable-prefix-caching --kv-cache-dtype fp8`
   - EPP scorer weights: queue=2, kv-cache=2, prefix-cache=3
 - **llm-d Gateway + HTTPRoute**: cluster-internal llm-d Gateway with EPP routing
@@ -680,7 +680,7 @@ Each extension connects to the self-hosted model endpoint. Here is exactly what
 | Setting | Value |
 |---------|-------|
 | **Model Endpoint (Base URL)** | `https://llm-d-gateway-data-science-gateway-class.ai-serving.svc.cluster.local/v1` |
-| **Model ID** | `Qwen/Qwen3.6-35B-A3B-FP8` |
+| **Model ID** | `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8` (served name alias for `Qwen/Qwen3.6-35B-A3B-FP8`) |
 | **API Key** | `EMPTY` (no auth required — cluster-internal) |
 
 | Extension | Auto-Configured? | How | What the User Sees |
@@ -706,7 +706,7 @@ Each extension connects to the self-hosted model endpoint. Here is exactly what
 | `default_worker_min_replicas` | `3` | Min workers (autoscaling) |
 | `default_worker_max_replicas` | `6` | Max workers (autoscaling) |
 | `gpu_pool_enabled` | `true` | Create GPU machine pool |
-| `gpu_instance_type` | `g6e.2xlarge` | GPU instance type (NVIDIA L40S) |
+| `gpu_instance_type` | `p5.4xlarge` | GPU instance type (NVIDIA H100) |
 | `gpu_pool_replicas` | `1` | Number of GPU nodes |
 | `inferentia_pool_enabled` | `false` | Create Inferentia machine pool |
 | `inferentia_instance_type` | `inf2.24xlarge` | Inferentia instance type |
@@ -716,11 +716,11 @@ Each extension connects to the self-hosted model endpoint. Here is exactly what
 
 | vLLM Argument | Value | Purpose |
 |---------------|-------|---------|
-| `--max-model-len` | `32768` | Context window (tokens) |
+| `--max-model-len` | `262144` | Context window (tokens) |
 | `--gpu-memory-utilization` | `0.90` | GPU memory fraction |
 | `--enable-prefix-caching` | — | KV-cache reuse across requests |
 | `--enable-auto-tool-choice` | — | Tool/function calling support |
-| `--tool-call-parser` | `qwen3_coder` | Tool call format parser |
+| `--tool-call-parser` | `qwen3_xml` | Tool call format parser |
 | `--reasoning-parser` | `qwen3` | Reasoning extraction |
 | `--kv-cache-dtype` | `fp8` | Memory-efficient KV-cache |
 
@@ -736,9 +736,37 @@ cd terraform
 terraform apply -var="gpu_pool_replicas=2"
 
 # Via ROSA CLI (immediate, not persisted in Terraform state)
-rosa edit machinepool gpu-l40s --cluster=rosa-pca --replicas=2
+rosa edit machinepool gpu-h100 --cluster=rosa-pca --replicas=2
 ```
 
+### Multi-GPU alternative (g6e.12xlarge, 4x L40S)
+
+For teams that prefer L40S GPUs or cannot get H100 quota, the model runs on
+`g6e.12xlarge` (4x NVIDIA L40S, 192 GB total GPU memory) with tensor parallelism.
+
+**Terraform overrides** (`terraform.tfvars`):
+
+```hcl
+gpu_instance_type = "g6e.12xlarge"
+gpu_pool_name     = "gpu-l40s-4x"
+```
+
+**Helm overrides** (add to `charts/pca-ai-serving/values-rosa.yaml`):
+
+```yaml
+hardware:
+  gpuProduct: "g6e.12xlarge"
+  gpuCount: 4
+  cpu:
+    request: "8"
+    limit: "16"
+  memory:
+    request: 80Gi
+    limit: 120Gi
+```
+
+vLLM automatically uses `--tensor-parallel-size=4` (derived from `hardware.gpuCount`).
+
 ### Scaling model replicas
 
 Edit `charts/pca-ai-serving/values-rosa.yaml` → update `llmInferenceService.replicas`, commit and push. ArgoCD syncs automatically.
@@ -817,7 +845,7 @@ Confirm when prompted. This deletes the cluster, machine pools, GitOps bootstrap
 ### Terraform apply fails on ROSA cluster creation
 
 - Verify your OCM token is valid: `rosa login --token="${RHCS_TOKEN}" && rosa whoami`
-- Check AWS quotas: GPU instances (`g6e.2xlarge`) require a service limit increase in many accounts
+- Check AWS quotas: GPU instances (`p5.4xlarge`) require a service limit increase in many accounts
 - Ensure the AWS account has ROSA enabled: `rosa verify quota --region=us-east-2`
 
 ### ArgoCD application stuck in "OutOfSync" or "Progressing"

PCA_Deployment_ROSA/terraform/main.tf

@@ -166,11 +166,11 @@ resource "rhcs_cluster_wait" "wait" {
 # Machine Pools
 # ════════════════════════════════════════════════
 
-# GPU Machine Pool (NVIDIA L40S — g6e.2xlarge)
+# GPU Machine Pool (NVIDIA H100 — p5.4xlarge)
 resource "rhcs_hcp_machine_pool" "gpu" {
   count   = var.gpu_pool_enabled ? 1 : 0
   cluster = rhcs_cluster_rosa_hcp.cluster.id
-  name    = "gpu-l40s"
+  name    = var.gpu_pool_name
 
   subnet_id = local.private_subnet_ids[0]
   auto_repair = true

PCA_Deployment_ROSA/terraform/terraform.tfvars.example

@@ -26,9 +26,9 @@ default_worker_autoscaling   = true
 default_worker_min_replicas  = 3
 default_worker_max_replicas  = 6
 
-# GPU pool (NVIDIA L40S)
+# GPU pool (NVIDIA H100)
 gpu_pool_enabled  = true
-gpu_instance_type = "g6e.2xlarge"
+gpu_instance_type = "p5.4xlarge"
 gpu_pool_replicas = 1
 
 # Inferentia pool (disabled by default)

PCA_Deployment_ROSA/terraform/variables.tf

@@ -138,7 +138,7 @@ variable "default_worker_max_replicas" {
 }
 
 # ──────────────────────────────────────────────
-# GPU Machine Pool (NVIDIA L40S)
+# GPU Machine Pool (NVIDIA H100)
 # ──────────────────────────────────────────────
 variable "gpu_pool_enabled" {
   description = "Enable GPU machine pool"
@@ -149,7 +149,13 @@ variable "gpu_pool_enabled" {
 variable "gpu_instance_type" {
   description = "Instance type for GPU nodes"
   type        = string
-  default     = "g6e.2xlarge"
+  default     = "p5.4xlarge"
+}
+
+variable "gpu_pool_name" {
+  description = "Machine pool name for GPU nodes (changing this recreates the pool)"
+  type        = string
+  default     = "gpu-h100"
 }
 
 variable "gpu_pool_replicas" {

charts/pca-ai-serving/charts/pca-observability/README.md

@@ -75,7 +75,7 @@ Optional values overrides (`grafana.adminPassword`, `langfuse.credentials.salt`,
 
 ## GPU cost PLACEHOLDER
 
-`cost.gpuHourlyUsd: 1.86` is an **illustrative** L40S on-demand figure from the sizing doc — **not** billing truth.
+`cost.gpuHourlyUsd: 4.50` is an **illustrative** H100 on-demand figure from the sizing doc — **not** billing truth.
 
 - `cost.gpuHourlyUsdIsPlaceholder: true` (default)
 - Panel titles include **PLACEHOLDER — override per cluster**

charts/pca-ai-serving/charts/pca-observability/templates/dashboards/board-c.yaml

@@ -83,7 +83,7 @@ data:
         {
           "type": "stat",
           "title": "PLACEHOLDER — GPU $/hr context (override per cluster)",
-          "description": "Illustrative L40S on-demand from sizing doc — NOT billing truth. Set cost.gpuHourlyUsd / cost.gpuHourlyUsdIsPlaceholder in values.",
+          "description": "Illustrative H100 on-demand from sizing doc — NOT billing truth. Set cost.gpuHourlyUsd / cost.gpuHourlyUsdIsPlaceholder in values.",
           "gridPos": { "h": 4, "w": 12, "x": 0, "y": 16 },
           "options": { "reduceOptions": { "calcs": ["lastNotNull"] } },
           "targets": [

charts/pca-ai-serving/charts/pca-observability/templates/dashboards/board-d.yaml

@@ -58,7 +58,7 @@ data:
         {
           "type": "stat",
           "title": "PLACEHOLDER — GPU $/hr (not billing truth)",
-          "description": "Illustrative L40S on-demand from sizing doc. Set cost.gpuHourlyUsdIsPlaceholder: false when overridden with real rates.",
+          "description": "Illustrative H100 on-demand from sizing doc. Set cost.gpuHourlyUsdIsPlaceholder: false when overridden with real rates.",
           "gridPos": { "h": 4, "w": 8, "x": 0, "y": 12 },
           "targets": [
             {

charts/pca-ai-serving/charts/pca-observability/values.yaml

@@ -150,8 +150,8 @@ prometheus:
   # accessMode=namespace so Grafana can query those metrics without cluster RBAC.
   gpuMetricsNamespace: nvidia-gpu-operator
 cost:
-  # PLACEHOLDER — illustrative L40S on-demand from sizing doc; not billing truth
-  gpuHourlyUsd: 1.86
+  # PLACEHOLDER — illustrative H100 on-demand from sizing doc; not billing truth
+  gpuHourlyUsd: 4.50
   gpuHourlyUsdIsPlaceholder: true
 persistence:
   storageClass: ""

charts/pca-ai-serving/templates/inferenceservice.yaml

@@ -1,7 +1,6 @@
 {{- if .Values.vllm.useCustomRuntime }}
 # InferenceService using custom ServingRuntime — paired with servingruntime.yaml.
-# Used when LLMInferenceService + RHOAI bundled vLLM cannot serve the target model.
-# Set vllm.useCustomRuntime: true and vllm.image in values-aro.yaml.
+# Default for all clouds. Set vllm.useCustomRuntime: false to fall back to LLMInferenceService.
 apiVersion: serving.kserve.io/v1beta1
 kind: InferenceService
 metadata:

charts/pca-ai-serving/templates/servingruntime.yaml

@@ -1,7 +1,6 @@
 {{- if .Values.vllm.useCustomRuntime }}
-# Custom ServingRuntime for upstream vLLM — used when RHOAI bundled vLLM does not
-# support the target model architecture (e.g. Qwen3.6 needs vLLM >= 0.18).
-# Chart default: useCustomRuntime: true with vllm.image (ARO/ROSA/existing OpenShift).
+# Custom ServingRuntime for upstream vLLM — default for all clouds.
+# Set vllm.useCustomRuntime: false to fall back to LLMInferenceService with RHOAI bundled vLLM.
 apiVersion: serving.kserve.io/v1alpha1
 kind: ServingRuntime
 metadata:

charts/pca-ai-serving/values-aro.yaml

@@ -1,37 +1,12 @@
+# ARO-specific overrides — model/vLLM/hardware defaults inherit from base values.yaml.
 model:
-  id: "Qwen/Qwen3.6-35B-A3B-FP8"
-  # servedName left empty (inherits "") so API name equals model.id
-  name: "qwen3-coder" # service/pod label name (aligned with ROSA)
+  name: "qwen3-coder"
   poolName: "qwen3-coder-inference-pool"
   workloadServiceSuffix: "predictor"
   workloadServicePort: 80
-vllm:
-  useCustomRuntime: true
-  image: "vllm/vllm-openai:v0.19.0"
-  maxModelLen: 262144
-  toolCallParser: "qwen3_xml"
-  reasoningParser: "qwen3"
-  extraEnv:
-    # NOTE: VLLM_ENABLE_CUDA_COMPATIBILITY and LD_LIBRARY_PATH compat shims are NOT needed
-    # on driver 580+ (CUDA 13.0) — they caused Error 803 and were deliberately removed.
-    # Add them back only if the GPU node has driver 550 (CUDA 12.4) or older.
-    HF_HOME: "/model-cache"
-    HF_HUB_OFFLINE: "0"
-    TRITON_CACHE_DIR: "/model-cache/triton-cache"
-    XDG_CACHE_HOME: "/model-cache/xdg-cache"
-    HOME: "/tmp"
-    DG_JIT_CACHE_DIR: "/model-cache/deep-gemm"
-    VLLM_CACHE_ROOT: "/model-cache/vllm-cache"
 hardware:
   gpuProduct: "NVIDIA-H100-NVL"
   instanceTypeLabel: "nvidia.com/gpu.product"
-  gpuCount: 1
-  cpu:
-    request: "8"
-    limit: "16"
-  memory:
-    request: 80Gi
-    limit: 120Gi
 storage:
   storageClass: "managed-csi"
   modelCacheSize: 100Gi

charts/pca-ai-serving/values-rosa.yaml

@@ -1,23 +1,8 @@
-model:
-  id: "Qwen/Qwen3.6-35B-A3B-FP8"
-  name: "qwen3-coder"
-  poolName: "qwen3-coder-inference-pool"
-vllm:
-  maxModelLen: 32768
-  reasoningParser: "qwen3"
+# ROSA-specific overrides — model/vLLM/hardware inherit from base values.yaml.
 hardware:
-  gpuProduct: "g6e.2xlarge"
   instanceTypeLabel: "node.kubernetes.io/instance-type"
-  gpuCount: 1
-  cpu:
-    request: "2"
-    limit: "4"
-  memory:
-    request: 16Gi
-    limit: 48Gi
 storage:
   storageClass: "gp3-csi"
-  modelCacheSize: 100Gi
 observability:
   enabled: true
 pca-observability:

charts/pca-ai-serving/values.yaml

@@ -2,15 +2,15 @@ namespace: ai-serving
 clusterResources: true
 model:
   id: "Qwen/Qwen3.6-35B-A3B-FP8"
-  servedName: "" # empty = use model.id as the vLLM --served-model-name
+  servedName: "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8" # backward-compat API alias — IDEs / smoke tests still send this name
   name: "qwen3-coder"
   poolName: "qwen3-coder-inference-pool"
   # InferenceService (useCustomRuntime) creates <name>-predictor:80.
   # LLMInferenceService path uses kserve-workload-svc:8000 — keep these paired with useCustomRuntime.
   workloadServiceSuffix: "predictor"
   workloadServicePort: 80
 vllm:
-  maxModelLen: 32768
+  maxModelLen: 262144
   toolCallParser: "qwen3_xml"
   reasoningParser: "qwen3"
   image: "vllm/vllm-openai:v0.19.0"
@@ -23,9 +23,8 @@ vllm:
     HOME: "/tmp"
     DG_JIT_CACHE_DIR: "/model-cache/deep-gemm"
     VLLM_CACHE_ROOT: "/model-cache/vllm-cache"
-  # Set true when RHOAI bundled vLLM doesn't support the model architecture.
-  # Renders ServingRuntime + InferenceService instead of LLMInferenceService.
-  # Requires vllm.image to be set. Qwen3.6 needs vLLM >= 0.18.
+  # Renders ServingRuntime + InferenceService with the custom vllm.image.
+  # Set false to fall back to LLMInferenceService with RHOAI bundled vLLM.
   useCustomRuntime: true
   # Set false to disable Qwen3 thinking mode (adds --chat-template-kwargs).
   # Use false when the model's thinking blocks exceed test max_tokens budgets.
@@ -35,15 +34,15 @@ vllm:
   # Optional vLLM --gpu-memory-utilization (empty = vLLM default).
   gpuMemoryUtilization: ""
 hardware:
-  gpuProduct: "g6e.2xlarge"
+  gpuProduct: "p5.4xlarge"
   instanceTypeLabel: "node.kubernetes.io/instance-type"
   gpuCount: 1
   cpu:
-    request: "2"
-    limit: "4"
+    request: "8"
+    limit: "16"
   memory:
-    request: 16Gi
-    limit: 48Gi
+    request: 80Gi
+    limit: 120Gi
 storage:
   storageClass: "gp3-csi"
   modelCacheSize: 100Gi
@@ -107,8 +106,8 @@ pca-observability:
     # ROSA full provision: cluster. Existing OpenShift override: namespace.
     accessMode: cluster
   cost:
-    # PLACEHOLDER — illustrative L40S on-demand from sizing doc; not billing truth
-    gpuHourlyUsd: 1.86
+    # PLACEHOLDER — illustrative H100 on-demand from sizing doc; not billing truth
+    gpuHourlyUsd: 4.50
     gpuHourlyUsdIsPlaceholder: true
   persistence:
     storageClass: ""

charts/pca-operators/templates/subscriptions.yaml

@@ -58,7 +58,7 @@ metadata:
   annotations:
     argocd.argoproj.io/sync-wave: "1"
 spec:
-  channel: {{ (.Values.operators).nvidiaGpu.channel | default "v25.3" }}
+  channel: {{ (.Values.operators).nvidiaGpu.channel | default "v26.3" }}
   installPlanApproval: Automatic
   name: gpu-operator-certified
   source: certified-operators

charts/pca-operators/values.yaml

@@ -4,7 +4,7 @@ operators:
   rhoai:
     channel: stable-3.4
   nvidiaGpu:
-    channel: v25.3
+    channel: v26.3
   nfd:
     enabled: true
   serviceMesh:

deploy_existing_openshift/README.md

@@ -156,7 +156,7 @@ oc get secret pca-grafana-admin -n $AI_NAMESPACE -o jsonpath='{.data.admin-passw
 oc get secret pca-langfuse-credentials -n $AI_NAMESPACE -o jsonpath='{.data.init-user-password}' | base64 -d; echo
 ```
 
-**GPU $/hr PLACEHOLDER:** `cost.gpuHourlyUsd: 1.86` is illustrative L40S on-demand — **not** billing truth. Override per cluster and set `cost.gpuHourlyUsdIsPlaceholder: false`.
+**GPU $/hr PLACEHOLDER:** `cost.gpuHourlyUsd: 4.50` is illustrative H100 on-demand — **not** billing truth. Override per cluster and set `cost.gpuHourlyUsdIsPlaceholder: false`.
 
 **Attribution:** Roo + Continue + Cline send `X-PCA-User` / `X-PCA-DevSpace` / optional `X-PCA-Team` (from `devspaces[].team`). Full prompt/completion bodies go to Langfuse when `ioCapture=full`. See `charts/pca-ai-serving/charts/pca-observability/README.md`.
 

deploy_existing_openshift/values-ai-serving.yaml

@@ -4,6 +4,8 @@ clusterResources: false
 pca-observability:
   prometheus:
     accessMode: namespace
+# Context-window cap for existing-OpenShift where GPU may have less VRAM than H100.
+# Base default is 262144 (full Qwen3.6 native context). Remove if GPU has >= 80 GB VRAM.
 vllm:
   maxModelLen: 49152
 # Self-signed cert for llm-d / RHCL gateways (LLMInferenceService no longer creates kserve-self-signed-certs).