New skill - rh-qs-deploy-test

[Hadar301]

Skill proposal:

name: rh-qs-deploy-test
description: Deploy a quickstart to a live OpenShift cluster, iteratively fix issues, validate all components, and report required repo changes. Run as a subagent from the main pipeline between rh-qs-deploy and rh-qs-document.

rh-qs-deploy-test

Purpose

Verify that a quickstart's deployment artifacts actually work on a real OpenShift cluster. Deploy the Helm chart, diagnose and fix failures, validate end-to-end connectivity, and produce a structured report for the calling agent. The report captures every required repo change, the working helm install command, and cluster prerequisites — so the main agent can feed fixes back into rh-qs-deploy and accurate instructions into rh-qs-document.

When to use

• After rh-qs-deploy produces Helm chart, Containerfiles, and values.yaml
• User asks to "test the deployment", "deploy and verify", or "validate on cluster"
• Before rh-qs-document — to ensure documented deploy steps are real

Execution model

This skill runs as a subagent in a worktree. The calling agent spawns it with:

Agent({
  subagent_type: "general-purpose",
  isolation: "worktree",
  prompt: "<see invocation template below>"
})

The subagent works in an isolated copy of the repo, makes whatever temporary edits are needed to get the deployment working (Helm templates, values.yaml, Containerfiles), and returns a structured report. The worktree is discarded — no changes land on the main branch. The main agent reads the report and decides which changes to apply upstream.

Invocation template

The calling agent must provide these parameters in the prompt:

Deploy and validate this quickstart on OpenShift.

Quickstart repo: <absolute path to repo root>
Helm chart path: <relative path, e.g. deploy/helm/my-quickstart>
Namespace: <target namespace>
Cluster: <cluster API URL or "current context">
Container registry: <e.g. quay.io/myorg>
Secrets:
  - HF_TOKEN=<value>          (if models need it)
  - <any other secrets>

Read the skill at core/skills/deployment/rh-qs-deploy-test/SKILL.md and follow its workflow.
Return ONLY the structured report defined in the skill — no other commentary.

Prerequisites

• oc CLI authenticated to the target cluster
• helm 3.x installed
• podman installed (for image builds)
• Container registry login active (podman login)
• Target namespace exists (oc new-project or oc project)
• Helm chart passes helm lint (from rh-qs-deploy)

Workflow

1. Discover the quickstart

Read the repo to build a mental model of what needs to deploy. Do not assume any specific quickstart structure — discover it.

Scan:

• Chart.yaml — list all dependencies (llm-service, llama-stack, pgvector, minio, etc.)
• values.yaml — find all image references, model configurations, enabled/disabled flags
• templates/ — list all Deployments, Services, Routes, ConfigMaps
• packages/ — find all Containerfile or Dockerfile entries (these are custom images that need building)
• compose.yml — cross-reference services for port and env var expectations

Produce a component inventory:

Component	Source	Image	Needs Build	Needs GPU
api	packages/api/Containerfile	quay.io/org/slug-api	yes	no
ui	packages/ui/Containerfile	quay.io/org/slug-ui	yes	no
llm-service	ai-architecture-charts	vllm/vllm-openai	no	yes
pgvector	ai-architecture-charts	pgvector/pgvector	no	no
...	...	...	...	...

2. Pre-flight the cluster

Gather cluster state before deploying anything.

oc project <namespace>
oc get nodes -o custom-columns='NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory,GPU:.status.capacity.nvidia\.com/gpu'
oc get resourcequota -n <namespace>
oc get limitrange -n <namespace>

For GPU workloads, check taints on GPU nodes:

oc get nodes -l nvidia.com/gpu.present=true -o json | jq '.items[].spec.taints'

Record:

• Available GPUs vs required GPUs
• GPU node taints that need tolerations
• Any namespace quotas that would block deployment
• Existing resources in the namespace (avoid collisions)

3. Build and push custom images

For each Containerfile found in step 1:

podman build --platform linux/amd64 \
  -t <registry>/<image-name>:latest \
  -f <path-to-containerfile> <build-context>
podman push <registry>/<image-name>:latest

Common build issues to handle:

• Missing lockfiles (pnpm-lock.yaml) — drop --frozen-lockfile or generate one
• pnpm ERR_PNPM_IGNORED_BUILDS — add --ignore-scripts
• TypeScript errors — fix tsconfig.json (noEmit: true, add Vite client types)
• Permission errors in UBI images — UBI runs as non-root; ensure no writes to read-only paths

Update values.yaml image references to point to the pushed images. Set pullPolicy: Always.

Record every Containerfile change as a required repo change.

4. Adapt Helm values to cluster

GPU tolerations: If step 2 found taints on GPU nodes, add tolerations to every model in values.yaml:

tolerations:
  - key: <taint-key>
    effect: NoSchedule
    operator: Exists

Secrets: Set tokens and credentials via --set flags on the helm command — never write secrets to files.

Port mappings: Cross-reference container ports with Service targetPorts. Common mismatch: UBI nginx images listen on 8080, not 3000/5173.

Record every values.yaml or template change as a required repo change.

5. Install

helm install <release-name> <chart-path> \
  --namespace <namespace> \
  --set <secret-overrides> \
  <any other --set flags from step 4>

6. Fix loop

Iterate until all core components are Running/Ready or max 5 iterations reached.

Each iteration:

oc get pods -n <namespace>

For each non-Running pod, diagnose and fix:

Diagnosis tree

Pod Status	Check	Common Cause	Fix
Pending	oc describe pod → Events	GPU taint not tolerated	Add toleration to values.yaml model config
Pending	Events: Insufficient resource	Not enough GPUs/CPU/memory	Scale down other workloads or reduce replicas
ImagePullBackOff	Events: unauthorized or not found	Image not pushed or registry private	Build and push image (step 3), or add imagePullSecret
CrashLoopBackOff	oc logs → PermissionError	Container writes to read-only path	Add emptyDir volume mount for that path
CrashLoopBackOff	oc logs → "failed switching to user"	Container tries to switch user (gosu/su-exec)	Override command/entrypoint to skip user switch
CrashLoopBackOff	oc logs → config/validation error	Bad config in ConfigMap	Fix the ConfigMap template (missing fields, wrong syntax)
CrashLoopBackOff	oc logs → port already in use or bind error	Port mismatch	Fix containerPort in Deployment and targetPort in Service
Error	oc logs → startup failure	Application error	Read logs, fix the root cause
Init:0/N	oc logs -c <init> → waiting for service	Dependency not deployed (e.g. DSPA)	Document as cluster prerequisite

After each fix:

helm upgrade <release-name> <chart-path> \
  --namespace <namespace> \
  --set <same overrides>

If a Deployment doesn't pick up ConfigMap changes, restart it:

oc rollout restart deployment/<name> -n <namespace>

GPU rollout constraint: When updating GPU workloads, delete old pods before new ones schedule — there are no spare GPUs for rolling updates:

# Scale old ReplicaSet to 0, or delete old pod directly
oc delete pod <old-pod> -n <namespace>

Record every fix as a required repo change, noting the file, what to change, and why.

7. Validate end-to-end

Once all core pods are Running, validate connectivity at each layer.

Method: oc exec into a running pod (typically the API pod) and test from inside the cluster. This avoids route/TLS issues and tests real service-to-service connectivity.

Layer-by-layer validation

a. Infrastructure services (pgvector, minio, llamastack):

oc exec <api-pod> -- python3 -c "
import httpx, asyncio
async def test():
    async with httpx.AsyncClient(timeout=10) as c:
        r = await c.get('http://<service>:<port>/health')
        print(f'{r.status_code}')
asyncio.run(test())
"

b. Model backends (vLLM / llm-service):

# List models
oc exec <api-pod> -- curl -s http://<vllm-service>:<port>/v1/models

# Test inference
oc exec <api-pod> -- curl -s http://<vllm-service>:<port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"<model-name>","messages":[{"role":"user","content":"hi"}],"max_tokens":5}'

If vLLM returns "model does not exist", the --served-model-name arg doesn't match what the router/app sends. Record this as a required repo change.

c. Application API:

# Health check
oc exec <api-pod> -- curl -s http://localhost:<port>/healthz

# Functional test (app-specific — adapt based on what routes exist)
oc exec <api-pod> -- curl -s http://localhost:<port>/api/v1/<endpoint> \
  -X POST -H "Content-Type: application/json" -d '<test payload>'

d. External routes:

ROUTE=$(oc get route <route-name> -n <namespace> -o jsonpath='{.spec.host}')
curl -sk https://${ROUTE}/healthz

Record any connectivity failures and their fixes.

8. Produce the report

Return only this structured report. No other commentary.

# Deploy Test Report: <quickstart-name>

## Status: <PASSED | PARTIAL | FAILED>

## Component Status

| Component | Pod | Status | Route URL |
|-----------|-----|--------|-----------|
| api | <pod-name> | Running 1/1 | https://... |
| ui | <pod-name> | Running 1/1 | https://... |
| llm-service | <pod-name> | Running 2/2 | (internal) |
| pgvector | <pod-name> | Running 1/1 | (internal) |
| ... | ... | ... | ... |

## Helm Install Command

The exact command that successfully deployed the quickstart:

```bash
helm install <release> <chart-path> \
  --namespace <namespace> \
  --set key=val \
  ...

Required Repo Changes

Changes that must be applied to the repo for the deployment to work on OpenShift without manual intervention.

1.

What:
Why: <root cause — e.g. "UBI nginx listens on 8080, not 3000">

- old content
+ new content

2.

...

Cluster Prerequisites

Conditions the target cluster must meet that are outside the quickstart's control.

Prerequisite	Details
GPU count	N GPUs required (N available on test cluster)
GPU taints	Nodes tainted with <key> — tolerations added in values.yaml
RHOAI operator	Required for
DSPA	Required for ingestion pipeline (not deployed by this chart)
...	...

Validation Results

Test	Result	Notes
Direct vLLM inference	PASS/FAIL	responded
API health check	PASS/FAIL
End-to-end chat/query	PASS/FAIL	Details
External route access	PASS/FAIL
...	...	...

Iterations

Number of helm upgrade cycles: N

Unresolved Issues

<List anything still broken and why, or "None">

## Scope boundaries

### Do
- Make temporary edits in the worktree to get the deployment working
- Test every service layer from inside the cluster
- Capture every fix as a structured "required repo change" with diffs
- Report cluster prerequisites that should be documented
- Delete/recreate pods when GPU rollout is blocked
- Bound the fix loop to 5 iterations — report what's still broken after that

### Do NOT
- Commit or push any changes — the worktree is disposable
- Write secrets to files — use `--set` only
- Fix application bugs — report them as unresolved issues
- Assume any specific quickstart structure — discover everything from Chart.yaml, values.yaml, and templates/
- Deploy infrastructure the chart doesn't manage (DSPA, operators) — report as prerequisites
- Continue iterating past 5 upgrade cycles — report remaining issues and stop

## Relationship to other skills

- **Input from:** `rh-qs-deploy` (Helm chart, Containerfiles, values.yaml exist and pass `helm lint`)
- **Output to:** Main agent, which feeds the report into:
  - `rh-qs-deploy` — to apply required repo changes
  - `rh-qs-document` — to write accurate deploy instructions, prerequisites, and troubleshooting