diff --git a/PCA_Deployment_ROSA/README.md b/PCA_Deployment_ROSA/README.md index 18f52e2..88a74cb 100644 --- a/PCA_Deployment_ROSA/README.md +++ b/PCA_Deployment_ROSA/README.md @@ -44,7 +44,7 @@ The end result is a ROSA HCP cluster running: │ Terraform (Phase 1 — Run once) │ │ ├── AWS VPC, Subnets, NAT Gateway │ │ ├── ROSA HCP Cluster (STS/OIDC) │ -│ ├── Machine Pools (workers, GPU L40S, opt. Inferentia) │ +│ ├── Machine Pools (workers, GPU H100, opt. Inferentia) │ │ ├── HTPasswd IDP + Developer Users │ │ └── OpenShift GitOps Operator + App-of-Apps bootstrap │ ├──────────────────────────────────────────────────────────────────┤ @@ -104,7 +104,7 @@ You need the following tools installed on your local machine before starting. | 6 | `kubeseal` | latest (optional) | Sealed Secrets for production | **Accounts required:** -- **AWS account** with permissions to create VPC, IAM roles, and EC2 instances (including `g6e.2xlarge` GPU instances) +- **AWS account** with permissions to create VPC, IAM roles, and EC2 instances (including `p5.4xlarge` GPU instances) - **Red Hat account** with ROSA entitlement (https://console.redhat.com) - **HuggingFace account** with an API token for model downloads (https://huggingface.co/settings/tokens) @@ -298,7 +298,7 @@ default_worker_max_replicas = 6 # GPU pool gpu_pool_enabled = true -gpu_instance_type = "g6e.2xlarge" +gpu_instance_type = "p5.4xlarge" gpu_pool_replicas = 1 # Inferentia pool (set to true if you have inf2 quota) @@ -650,9 +650,9 @@ echo "Console: $(terraform output -raw cluster_console_url)" ### Wave 3 — AI Serving (`pca-ai-serving`) - **PVC**: 100Gi `model-cache` on `gp3-csi` (persists model weights across restarts) -- **ServingRuntime + InferenceService** (`qwen3-coder`): `Qwen/Qwen3.6-35B-A3B-FP8` (custom vLLM runtime; API name matches `model.id`) with vLLM args: +- **ServingRuntime + InferenceService** (`qwen3-coder`): `Qwen/Qwen3.6-35B-A3B-FP8` with custom vLLM v0.19.0 runtime: - `--tool-call-parser qwen3_xml --reasoning-parser qwen3` - - `--max-model-len 32768 --gpu-memory-utilization 0.90` + - `--max-model-len 262144 --gpu-memory-utilization 0.90` - `--enable-prefix-caching --kv-cache-dtype fp8` - EPP scorer weights: queue=2, kv-cache=2, prefix-cache=3 - **llm-d Gateway + HTTPRoute**: cluster-internal llm-d Gateway with EPP routing @@ -680,7 +680,7 @@ Each extension connects to the self-hosted model endpoint. Here is exactly what | Setting | Value | |---------|-------| | **Model Endpoint (Base URL)** | `https://llm-d-gateway-data-science-gateway-class.ai-serving.svc.cluster.local/v1` | -| **Model ID** | `Qwen/Qwen3.6-35B-A3B-FP8` | +| **Model ID** | `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8` (served name alias for `Qwen/Qwen3.6-35B-A3B-FP8`) | | **API Key** | `EMPTY` (no auth required — cluster-internal) | | Extension | Auto-Configured? | How | What the User Sees | @@ -706,7 +706,7 @@ Each extension connects to the self-hosted model endpoint. Here is exactly what | `default_worker_min_replicas` | `3` | Min workers (autoscaling) | | `default_worker_max_replicas` | `6` | Max workers (autoscaling) | | `gpu_pool_enabled` | `true` | Create GPU machine pool | -| `gpu_instance_type` | `g6e.2xlarge` | GPU instance type (NVIDIA L40S) | +| `gpu_instance_type` | `p5.4xlarge` | GPU instance type (NVIDIA H100) | | `gpu_pool_replicas` | `1` | Number of GPU nodes | | `inferentia_pool_enabled` | `false` | Create Inferentia machine pool | | `inferentia_instance_type` | `inf2.24xlarge` | Inferentia instance type | @@ -716,11 +716,11 @@ Each extension connects to the self-hosted model endpoint. Here is exactly what | vLLM Argument | Value | Purpose | |---------------|-------|---------| -| `--max-model-len` | `32768` | Context window (tokens) | +| `--max-model-len` | `262144` | Context window (tokens) | | `--gpu-memory-utilization` | `0.90` | GPU memory fraction | | `--enable-prefix-caching` | — | KV-cache reuse across requests | | `--enable-auto-tool-choice` | — | Tool/function calling support | -| `--tool-call-parser` | `qwen3_coder` | Tool call format parser | +| `--tool-call-parser` | `qwen3_xml` | Tool call format parser | | `--reasoning-parser` | `qwen3` | Reasoning extraction | | `--kv-cache-dtype` | `fp8` | Memory-efficient KV-cache | @@ -736,9 +736,37 @@ cd terraform terraform apply -var="gpu_pool_replicas=2" # Via ROSA CLI (immediate, not persisted in Terraform state) -rosa edit machinepool gpu-l40s --cluster=rosa-pca --replicas=2 +rosa edit machinepool gpu-h100 --cluster=rosa-pca --replicas=2 ``` +### Multi-GPU alternative (g6e.12xlarge, 4x L40S) + +For teams that prefer L40S GPUs or cannot get H100 quota, the model runs on +`g6e.12xlarge` (4x NVIDIA L40S, 192 GB total GPU memory) with tensor parallelism. + +**Terraform overrides** (`terraform.tfvars`): + +```hcl +gpu_instance_type = "g6e.12xlarge" +gpu_pool_name = "gpu-l40s-4x" +``` + +**Helm overrides** (add to `charts/pca-ai-serving/values-rosa.yaml`): + +```yaml +hardware: + gpuProduct: "g6e.12xlarge" + gpuCount: 4 + cpu: + request: "8" + limit: "16" + memory: + request: 80Gi + limit: 120Gi +``` + +vLLM automatically uses `--tensor-parallel-size=4` (derived from `hardware.gpuCount`). + ### Scaling model replicas Edit `charts/pca-ai-serving/values-rosa.yaml` → update `llmInferenceService.replicas`, commit and push. ArgoCD syncs automatically. @@ -817,7 +845,7 @@ Confirm when prompted. This deletes the cluster, machine pools, GitOps bootstrap ### Terraform apply fails on ROSA cluster creation - Verify your OCM token is valid: `rosa login --token="${RHCS_TOKEN}" && rosa whoami` -- Check AWS quotas: GPU instances (`g6e.2xlarge`) require a service limit increase in many accounts +- Check AWS quotas: GPU instances (`p5.4xlarge`) require a service limit increase in many accounts - Ensure the AWS account has ROSA enabled: `rosa verify quota --region=us-east-2` ### ArgoCD application stuck in "OutOfSync" or "Progressing" diff --git a/PCA_Deployment_ROSA/terraform/main.tf b/PCA_Deployment_ROSA/terraform/main.tf index 6ce206e..0b45bfe 100644 --- a/PCA_Deployment_ROSA/terraform/main.tf +++ b/PCA_Deployment_ROSA/terraform/main.tf @@ -166,11 +166,11 @@ resource "rhcs_cluster_wait" "wait" { # Machine Pools # ════════════════════════════════════════════════ -# GPU Machine Pool (NVIDIA L40S — g6e.2xlarge) +# GPU Machine Pool (NVIDIA H100 — p5.4xlarge) resource "rhcs_hcp_machine_pool" "gpu" { count = var.gpu_pool_enabled ? 1 : 0 cluster = rhcs_cluster_rosa_hcp.cluster.id - name = "gpu-l40s" + name = var.gpu_pool_name subnet_id = local.private_subnet_ids[0] auto_repair = true diff --git a/PCA_Deployment_ROSA/terraform/terraform.tfvars.example b/PCA_Deployment_ROSA/terraform/terraform.tfvars.example index 842ec9c..1f31fbf 100644 --- a/PCA_Deployment_ROSA/terraform/terraform.tfvars.example +++ b/PCA_Deployment_ROSA/terraform/terraform.tfvars.example @@ -26,9 +26,9 @@ default_worker_autoscaling = true default_worker_min_replicas = 3 default_worker_max_replicas = 6 -# GPU pool (NVIDIA L40S) +# GPU pool (NVIDIA H100) gpu_pool_enabled = true -gpu_instance_type = "g6e.2xlarge" +gpu_instance_type = "p5.4xlarge" gpu_pool_replicas = 1 # Inferentia pool (disabled by default) diff --git a/PCA_Deployment_ROSA/terraform/variables.tf b/PCA_Deployment_ROSA/terraform/variables.tf index cc19871..3625352 100644 --- a/PCA_Deployment_ROSA/terraform/variables.tf +++ b/PCA_Deployment_ROSA/terraform/variables.tf @@ -138,7 +138,7 @@ variable "default_worker_max_replicas" { } # ────────────────────────────────────────────── -# GPU Machine Pool (NVIDIA L40S) +# GPU Machine Pool (NVIDIA H100) # ────────────────────────────────────────────── variable "gpu_pool_enabled" { description = "Enable GPU machine pool" @@ -149,7 +149,13 @@ variable "gpu_pool_enabled" { variable "gpu_instance_type" { description = "Instance type for GPU nodes" type = string - default = "g6e.2xlarge" + default = "p5.4xlarge" +} + +variable "gpu_pool_name" { + description = "Machine pool name for GPU nodes (changing this recreates the pool)" + type = string + default = "gpu-h100" } variable "gpu_pool_replicas" { diff --git a/charts/pca-ai-serving/charts/pca-observability/README.md b/charts/pca-ai-serving/charts/pca-observability/README.md index 255f962..fedc924 100644 --- a/charts/pca-ai-serving/charts/pca-observability/README.md +++ b/charts/pca-ai-serving/charts/pca-observability/README.md @@ -75,7 +75,7 @@ Optional values overrides (`grafana.adminPassword`, `langfuse.credentials.salt`, ## GPU cost PLACEHOLDER -`cost.gpuHourlyUsd: 1.86` is an **illustrative** L40S on-demand figure from the sizing doc — **not** billing truth. +`cost.gpuHourlyUsd: 4.50` is an **illustrative** H100 on-demand figure from the sizing doc — **not** billing truth. - `cost.gpuHourlyUsdIsPlaceholder: true` (default) - Panel titles include **PLACEHOLDER — override per cluster** diff --git a/charts/pca-ai-serving/charts/pca-observability/templates/dashboards/board-c.yaml b/charts/pca-ai-serving/charts/pca-observability/templates/dashboards/board-c.yaml index e64e1e3..dc30eb4 100644 --- a/charts/pca-ai-serving/charts/pca-observability/templates/dashboards/board-c.yaml +++ b/charts/pca-ai-serving/charts/pca-observability/templates/dashboards/board-c.yaml @@ -83,7 +83,7 @@ data: { "type": "stat", "title": "PLACEHOLDER — GPU $/hr context (override per cluster)", - "description": "Illustrative L40S on-demand from sizing doc — NOT billing truth. Set cost.gpuHourlyUsd / cost.gpuHourlyUsdIsPlaceholder in values.", + "description": "Illustrative H100 on-demand from sizing doc — NOT billing truth. Set cost.gpuHourlyUsd / cost.gpuHourlyUsdIsPlaceholder in values.", "gridPos": { "h": 4, "w": 12, "x": 0, "y": 16 }, "options": { "reduceOptions": { "calcs": ["lastNotNull"] } }, "targets": [ diff --git a/charts/pca-ai-serving/charts/pca-observability/templates/dashboards/board-d.yaml b/charts/pca-ai-serving/charts/pca-observability/templates/dashboards/board-d.yaml index c3a900f..e997134 100644 --- a/charts/pca-ai-serving/charts/pca-observability/templates/dashboards/board-d.yaml +++ b/charts/pca-ai-serving/charts/pca-observability/templates/dashboards/board-d.yaml @@ -58,7 +58,7 @@ data: { "type": "stat", "title": "PLACEHOLDER — GPU $/hr (not billing truth)", - "description": "Illustrative L40S on-demand from sizing doc. Set cost.gpuHourlyUsdIsPlaceholder: false when overridden with real rates.", + "description": "Illustrative H100 on-demand from sizing doc. Set cost.gpuHourlyUsdIsPlaceholder: false when overridden with real rates.", "gridPos": { "h": 4, "w": 8, "x": 0, "y": 12 }, "targets": [ { diff --git a/charts/pca-ai-serving/charts/pca-observability/values.yaml b/charts/pca-ai-serving/charts/pca-observability/values.yaml index fda6ebb..e7852a9 100644 --- a/charts/pca-ai-serving/charts/pca-observability/values.yaml +++ b/charts/pca-ai-serving/charts/pca-observability/values.yaml @@ -150,8 +150,8 @@ prometheus: # accessMode=namespace so Grafana can query those metrics without cluster RBAC. gpuMetricsNamespace: nvidia-gpu-operator cost: - # PLACEHOLDER — illustrative L40S on-demand from sizing doc; not billing truth - gpuHourlyUsd: 1.86 + # PLACEHOLDER — illustrative H100 on-demand from sizing doc; not billing truth + gpuHourlyUsd: 4.50 gpuHourlyUsdIsPlaceholder: true persistence: storageClass: "" diff --git a/charts/pca-ai-serving/templates/inferenceservice.yaml b/charts/pca-ai-serving/templates/inferenceservice.yaml index 7ded750..65ca369 100644 --- a/charts/pca-ai-serving/templates/inferenceservice.yaml +++ b/charts/pca-ai-serving/templates/inferenceservice.yaml @@ -1,7 +1,6 @@ {{- if .Values.vllm.useCustomRuntime }} # InferenceService using custom ServingRuntime — paired with servingruntime.yaml. -# Used when LLMInferenceService + RHOAI bundled vLLM cannot serve the target model. -# Set vllm.useCustomRuntime: true and vllm.image in values-aro.yaml. +# Default for all clouds. Set vllm.useCustomRuntime: false to fall back to LLMInferenceService. apiVersion: serving.kserve.io/v1beta1 kind: InferenceService metadata: diff --git a/charts/pca-ai-serving/templates/servingruntime.yaml b/charts/pca-ai-serving/templates/servingruntime.yaml index 709eab6..1c5f84b 100644 --- a/charts/pca-ai-serving/templates/servingruntime.yaml +++ b/charts/pca-ai-serving/templates/servingruntime.yaml @@ -1,7 +1,6 @@ {{- if .Values.vllm.useCustomRuntime }} -# Custom ServingRuntime for upstream vLLM — used when RHOAI bundled vLLM does not -# support the target model architecture (e.g. Qwen3.6 needs vLLM >= 0.18). -# Chart default: useCustomRuntime: true with vllm.image (ARO/ROSA/existing OpenShift). +# Custom ServingRuntime for upstream vLLM — default for all clouds. +# Set vllm.useCustomRuntime: false to fall back to LLMInferenceService with RHOAI bundled vLLM. apiVersion: serving.kserve.io/v1alpha1 kind: ServingRuntime metadata: diff --git a/charts/pca-ai-serving/values-aro.yaml b/charts/pca-ai-serving/values-aro.yaml index 6e8d640..c337e64 100644 --- a/charts/pca-ai-serving/values-aro.yaml +++ b/charts/pca-ai-serving/values-aro.yaml @@ -1,37 +1,12 @@ +# ARO-specific overrides — model/vLLM/hardware defaults inherit from base values.yaml. model: - id: "Qwen/Qwen3.6-35B-A3B-FP8" - # servedName left empty (inherits "") so API name equals model.id - name: "qwen3-coder" # service/pod label name (aligned with ROSA) + name: "qwen3-coder" poolName: "qwen3-coder-inference-pool" workloadServiceSuffix: "predictor" workloadServicePort: 80 -vllm: - useCustomRuntime: true - image: "vllm/vllm-openai:v0.19.0" - maxModelLen: 262144 - toolCallParser: "qwen3_xml" - reasoningParser: "qwen3" - extraEnv: - # NOTE: VLLM_ENABLE_CUDA_COMPATIBILITY and LD_LIBRARY_PATH compat shims are NOT needed - # on driver 580+ (CUDA 13.0) — they caused Error 803 and were deliberately removed. - # Add them back only if the GPU node has driver 550 (CUDA 12.4) or older. - HF_HOME: "/model-cache" - HF_HUB_OFFLINE: "0" - TRITON_CACHE_DIR: "/model-cache/triton-cache" - XDG_CACHE_HOME: "/model-cache/xdg-cache" - HOME: "/tmp" - DG_JIT_CACHE_DIR: "/model-cache/deep-gemm" - VLLM_CACHE_ROOT: "/model-cache/vllm-cache" hardware: gpuProduct: "NVIDIA-H100-NVL" instanceTypeLabel: "nvidia.com/gpu.product" - gpuCount: 1 - cpu: - request: "8" - limit: "16" - memory: - request: 80Gi - limit: 120Gi storage: storageClass: "managed-csi" modelCacheSize: 100Gi diff --git a/charts/pca-ai-serving/values-rosa.yaml b/charts/pca-ai-serving/values-rosa.yaml index d427bc9..de84c99 100644 --- a/charts/pca-ai-serving/values-rosa.yaml +++ b/charts/pca-ai-serving/values-rosa.yaml @@ -1,23 +1,8 @@ -model: - id: "Qwen/Qwen3.6-35B-A3B-FP8" - name: "qwen3-coder" - poolName: "qwen3-coder-inference-pool" -vllm: - maxModelLen: 32768 - reasoningParser: "qwen3" +# ROSA-specific overrides — model/vLLM/hardware inherit from base values.yaml. hardware: - gpuProduct: "g6e.2xlarge" instanceTypeLabel: "node.kubernetes.io/instance-type" - gpuCount: 1 - cpu: - request: "2" - limit: "4" - memory: - request: 16Gi - limit: 48Gi storage: storageClass: "gp3-csi" - modelCacheSize: 100Gi observability: enabled: true pca-observability: diff --git a/charts/pca-ai-serving/values.yaml b/charts/pca-ai-serving/values.yaml index 083fe74..574ec46 100644 --- a/charts/pca-ai-serving/values.yaml +++ b/charts/pca-ai-serving/values.yaml @@ -2,7 +2,7 @@ namespace: ai-serving clusterResources: true model: id: "Qwen/Qwen3.6-35B-A3B-FP8" - servedName: "" # empty = use model.id as the vLLM --served-model-name + servedName: "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8" # backward-compat API alias — IDEs / smoke tests still send this name name: "qwen3-coder" poolName: "qwen3-coder-inference-pool" # InferenceService (useCustomRuntime) creates -predictor:80. @@ -10,7 +10,7 @@ model: workloadServiceSuffix: "predictor" workloadServicePort: 80 vllm: - maxModelLen: 32768 + maxModelLen: 262144 toolCallParser: "qwen3_xml" reasoningParser: "qwen3" image: "vllm/vllm-openai:v0.19.0" @@ -23,9 +23,8 @@ vllm: HOME: "/tmp" DG_JIT_CACHE_DIR: "/model-cache/deep-gemm" VLLM_CACHE_ROOT: "/model-cache/vllm-cache" - # Set true when RHOAI bundled vLLM doesn't support the model architecture. - # Renders ServingRuntime + InferenceService instead of LLMInferenceService. - # Requires vllm.image to be set. Qwen3.6 needs vLLM >= 0.18. + # Renders ServingRuntime + InferenceService with the custom vllm.image. + # Set false to fall back to LLMInferenceService with RHOAI bundled vLLM. useCustomRuntime: true # Set false to disable Qwen3 thinking mode (adds --chat-template-kwargs). # Use false when the model's thinking blocks exceed test max_tokens budgets. @@ -35,15 +34,15 @@ vllm: # Optional vLLM --gpu-memory-utilization (empty = vLLM default). gpuMemoryUtilization: "" hardware: - gpuProduct: "g6e.2xlarge" + gpuProduct: "p5.4xlarge" instanceTypeLabel: "node.kubernetes.io/instance-type" gpuCount: 1 cpu: - request: "2" - limit: "4" + request: "8" + limit: "16" memory: - request: 16Gi - limit: 48Gi + request: 80Gi + limit: 120Gi storage: storageClass: "gp3-csi" modelCacheSize: 100Gi @@ -107,8 +106,8 @@ pca-observability: # ROSA full provision: cluster. Existing OpenShift override: namespace. accessMode: cluster cost: - # PLACEHOLDER — illustrative L40S on-demand from sizing doc; not billing truth - gpuHourlyUsd: 1.86 + # PLACEHOLDER — illustrative H100 on-demand from sizing doc; not billing truth + gpuHourlyUsd: 4.50 gpuHourlyUsdIsPlaceholder: true persistence: storageClass: "" diff --git a/charts/pca-operators/templates/subscriptions.yaml b/charts/pca-operators/templates/subscriptions.yaml index 9f77790..c234ce3 100644 --- a/charts/pca-operators/templates/subscriptions.yaml +++ b/charts/pca-operators/templates/subscriptions.yaml @@ -58,7 +58,7 @@ metadata: annotations: argocd.argoproj.io/sync-wave: "1" spec: - channel: {{ (.Values.operators).nvidiaGpu.channel | default "v25.3" }} + channel: {{ (.Values.operators).nvidiaGpu.channel | default "v26.3" }} installPlanApproval: Automatic name: gpu-operator-certified source: certified-operators diff --git a/charts/pca-operators/values.yaml b/charts/pca-operators/values.yaml index c80c5b3..65899a0 100644 --- a/charts/pca-operators/values.yaml +++ b/charts/pca-operators/values.yaml @@ -4,7 +4,7 @@ operators: rhoai: channel: stable-3.4 nvidiaGpu: - channel: v25.3 + channel: v26.3 nfd: enabled: true serviceMesh: diff --git a/deploy_existing_openshift/README.md b/deploy_existing_openshift/README.md index f758cf3..a2dfd1c 100644 --- a/deploy_existing_openshift/README.md +++ b/deploy_existing_openshift/README.md @@ -156,7 +156,7 @@ oc get secret pca-grafana-admin -n $AI_NAMESPACE -o jsonpath='{.data.admin-passw oc get secret pca-langfuse-credentials -n $AI_NAMESPACE -o jsonpath='{.data.init-user-password}' | base64 -d; echo ``` -**GPU $/hr PLACEHOLDER:** `cost.gpuHourlyUsd: 1.86` is illustrative L40S on-demand — **not** billing truth. Override per cluster and set `cost.gpuHourlyUsdIsPlaceholder: false`. +**GPU $/hr PLACEHOLDER:** `cost.gpuHourlyUsd: 4.50` is illustrative H100 on-demand — **not** billing truth. Override per cluster and set `cost.gpuHourlyUsdIsPlaceholder: false`. **Attribution:** Roo + Continue + Cline send `X-PCA-User` / `X-PCA-DevSpace` / optional `X-PCA-Team` (from `devspaces[].team`). Full prompt/completion bodies go to Langfuse when `ioCapture=full`. See `charts/pca-ai-serving/charts/pca-observability/README.md`. diff --git a/deploy_existing_openshift/values-ai-serving.yaml b/deploy_existing_openshift/values-ai-serving.yaml index b840db7..478f792 100644 --- a/deploy_existing_openshift/values-ai-serving.yaml +++ b/deploy_existing_openshift/values-ai-serving.yaml @@ -4,6 +4,8 @@ clusterResources: false pca-observability: prometheus: accessMode: namespace +# Context-window cap for existing-OpenShift where GPU may have less VRAM than H100. +# Base default is 262144 (full Qwen3.6 native context). Remove if GPU has >= 80 GB VRAM. vllm: maxModelLen: 49152 # Self-signed cert for llm-d / RHCL gateways (LLMInferenceService no longer creates kserve-self-signed-certs).