feat: add model-deploy ASE evaluation submission

submissions/model-deploy/evals/evals.json

@@ -0,0 +1,17 @@
+{
+  "skill_name": "model-deploy",
+  "evals": [
+    {
+      "id": "model-deploy-eval",
+      "name": "AI Model Deployment Methodology",
+      "prompt": "You need to deploy a Llama 3 model on OpenShift AI. The cluster has NVIDIA GPUs available.\n\nDescribe your model deployment methodology: what serving runtimes are available, how you select the right one, how you validate GPU availability, how you create the InferenceService, and how you monitor the deployment rollout.",
+      "expected_output": "A deployment methodology using KServe InferenceService with runtime selection (vLLM, NVIDIA NIM, Caikit+TGIS), GPU pre-flight checks (node availability, taints, tolerations), hardware profile lookup, and rollout monitoring with pod status checks.",
+      "assertions": [
+        "The output describes selecting between serving runtimes: vLLM, NVIDIA NIM, and Caikit+TGIS, explaining when each is appropriate.",
+        "The output describes GPU pre-flight validation: checking node availability, GPU taints, and configuring tolerations on the InferenceService.",
+        "The output describes creating a KServe InferenceService custom resource as the deployment mechanism.",
+        "The output describes monitoring the deployment rollout by checking predictor pod status and InferenceService readiness conditions."
+      ]
+    }
+  ]
+}

submissions/model-deploy/metadata.yaml

@@ -0,0 +1,5 @@
+name: model-deploy
+description: "Deploy AI/ML models on OpenShift AI using KServe with vLLM, NVIDIA NIM,"
+persona: rh-ai-engineer
+version: "1.0.0"
+eval_engine: ase

submissions/model-deploy/skills/references/known-model-profiles.md

@@ -0,0 +1,83 @@
+---
+title: Known Model Hardware Profiles
+category: references
+tags: [models, gpu, hardware, profiles, llama, granite, mixtral, mistral]
+semantic_keywords: [model GPU requirements, hardware profiles, vLLM configuration, model deployment specs]
+use_cases: [model-deploy, debug-inference, ai-observability]
+last_updated: 2026-03-10
+---
+
+# Known Model Hardware Profiles
+
+Hardware profiles for commonly deployed models on OpenShift AI. This file serves as a fast-path cache for `/model-deploy` — when a model is listed here, the agent uses these specs directly. When a model is not listed, the agent falls back to the live doc lookup protocol defined in [live-doc-lookup.md](live-doc-lookup.md).
+
+**Important**: These are recommended minimums. Actual requirements may vary based on quantization, sequence length, and batch size. Validate against live documentation for production deployments.
+
+## Model Source Conventions
+
+Each model lists a recommended `storageUri` with its authentication requirements:
+- **`hf://`** — HuggingFace Hub. Public models require no authentication. **Preferred default for public open-source models.**
+- **`oci://`** — OCI container registry. Requires image pull secrets with appropriate entitlements. `registry.redhat.io/rhelai1/*` images require RHEL AI subscription entitlements (not included in standard OpenShift pull secrets).
+- **`s3://`** — S3-compatible storage. Requires storage credentials configured in the namespace.
+
+When the user does not specify a model source, use the `hf://` URI listed in the profile below.
+
+## Llama 3.x (Meta)
+
+| Variant | Parameters | GPUs | GPU Type | VRAM | Key vLLM Args |
+|---------|-----------|------|----------|------|---------------|
+| Llama 3.1 8B | 8B | 1 | A10G/L4/A100/H100 | 16GB | `--max-model-len=4096` |
+| Llama 3.1 70B | 70B | 4 | A100 80GB | 320GB | `--max-model-len=4096 --tensor-parallel-size=4` |
+| Llama 3.1 70B | 70B | 2 | H100 80GB | 160GB | `--max-model-len=4096 --tensor-parallel-size=2` |
+| Llama 3.1 405B | 405B | 8 | A100 80GB / H100 | 640GB | `--max-model-len=4096 --tensor-parallel-size=8` |
+
+- **Recommended storageUri**: `hf://meta-llama/Llama-3.1-8B-Instruct` (public, no auth — requires HuggingFace license acceptance)
+- **OCI alternative**: `oci://registry.redhat.io/rhelai1/llama-3-1-8b-instruct` (requires RHEL AI entitlements)
+- Tool calling: `--tool-call-parser hermes --chat-template` (for Llama 3.1+ instruct variants)
+- Quantization: AWQ, GPTQ, FP8 variants reduce GPU requirements significantly
+
+## Granite 3.x (IBM/Red Hat)
+
+| Variant | Parameters | GPUs | GPU Type | VRAM | Key vLLM Args |
+|---------|-----------|------|----------|------|---------------|
+| Granite 3.1 2B | 2B | 1 | Any GPU | 8GB | `--max-model-len=4096` |
+| Granite 3.1 8B | 8B | 1 | A10G/L4/A100 | 16GB | `--max-model-len=4096` |
+
+- **Recommended storageUri (2B)**: `hf://ibm-granite/granite-3.1-2b-instruct` (public, no auth)
+- **Recommended storageUri (8B)**: `hf://ibm-granite/granite-3.1-8b-instruct` (public, no auth)
+- **OCI alternative**: `oci://registry.redhat.io/rhelai1/granite-3-1-2b-instruct` (requires RHEL AI entitlements)
+- Tool calling: `--tool-call-parser granite --chat-template`
+- Red Hat-supported model family on RHOAI
+
+## Mixtral (Mistral AI)
+
+| Variant | Parameters | GPUs | GPU Type | VRAM | Key vLLM Args |
+|---------|-----------|------|----------|------|---------------|
+| Mixtral 8x7B | 46.7B (MoE) | 2 | A100 80GB | 160GB | `--tensor-parallel-size=2` |
+| Mixtral 8x22B | 141B (MoE) | 4 | A100 80GB | 320GB | `--tensor-parallel-size=4` |
+
+- **Recommended storageUri (8x7B)**: `hf://mistralai/Mixtral-8x7B-Instruct-v0.1` (public, no auth)
+- Mixture-of-Experts architecture: only ~13B/45B parameters active per token
+
+## Mistral (Mistral AI)
+
+| Variant | Parameters | GPUs | GPU Type | VRAM | Key vLLM Args |
+|---------|-----------|------|----------|------|---------------|
+| Mistral 7B | 7B | 1 | A10G/L4/A100 | 16GB | `--max-model-len=8192` |
+| Mistral Large (123B) | 123B | 4 | A100 80GB | 320GB | `--tensor-parallel-size=4` |
+
+- **Recommended storageUri (7B)**: `hf://mistralai/Mistral-7B-Instruct-v0.3` (public, no auth)
+
+## When a Model Is Not Listed
+
+If the requested model is not in this file, the agent MUST use the live doc lookup protocol:
+
+1. Read [live-doc-lookup.md](live-doc-lookup.md) for the lookup procedure
+2. Fetch hardware specs from the appropriate source
+3. Report findings to the user before proceeding with deployment
+
+Common cases requiring live lookup:
+- Newly released models (after this file's last update)
+- Domain-specific fine-tuned models
+- Models with custom quantization
+- NIM-specific optimized profiles

submissions/model-deploy/skills/references/model-deploy.md

@@ -0,0 +1,160 @@
+# /model-deploy Examples
+
+Example interactions demonstrating the model-deploy skill workflow.
+
+## Example 1: Deploy Llama 3.1 8B with vLLM
+
+```
+User: "Deploy Llama 3.1 8B on my cluster"
+
+Agent: [Invokes /model-deploy skill]
+       [Reads supported-runtimes.md]
+
+I consulted supported-runtimes.md to understand runtime capabilities.
+
+Step 1 of 10: Gathering deployment information.
+
+| Setting | Value | Source |
+|---------|-------|--------|
+| Model | Llama 3.1 8B Instruct | user input |
+| Runtime | TBD | auto-detect |
+| Namespace | ? | needs input |
+| Model Source | ? | needs input |
+| Deployment Mode | Serverless | default |
+
+What namespace should I deploy to, and where are the model weights stored?
+(e.g., S3 bucket, OCI registry, PVC)
+
+User: "namespace: ai-models, model is on S3 at s3://my-bucket/llama-3.1-8b"
+
+Agent: Step 2 of 10: Determining runtime.
+
+I consulted supported-runtimes.md to understand runtime capabilities.
+
+Based on Llama 3.1 8B (open-source HuggingFace model):
+
+Recommended runtime: vLLM
+Reason: Default runtime for open-source LLMs. Broad compatibility,
+        OpenAI-compatible API, no additional setup required.
+
+Proceed with vLLM? (yes / no / suggest alternative)
+
+User: "yes"
+
+Agent: Step 3 of 10: Looking up hardware profile.
+
+I consulted known-model-profiles.md to find hardware requirements for Llama 3.1 8B.
+
+| Requirement | Value |
+|-------------|-------|
+| GPUs | 1x A10G/L4/A100/H100 |
+| VRAM | 16GB |
+| Key Args | --max-model-len=4096 --tool-call-parser hermes |
+
+Step 4 of 10: GPU pre-flight check skipped (ai-observability MCP not configured).
+Ensure your cluster has at least 1 GPU with 16GB VRAM.
+
+Step 5 of 10: Skipped (NIM platform check not needed for vLLM).
+
+Step 6-7 of 10: Generating InferenceService YAML.
+
+apiVersion: serving.kserve.io/v1beta1
+kind: InferenceService
+metadata:
+  name: llama-3-1-8b
+  namespace: ai-models
+  annotations:
+    serving.kserve.io/deploymentMode: Serverless
+spec:
+  predictor:
+    model:
+      modelFormat:
+        name: vLLM
+      runtime: vllm-runtime
+      storageUri: s3://my-bucket/llama-3.1-8b
+      resources:
+        limits:
+          nvidia.com/gpu: "1"
+        requests:
+          cpu: "4"
+          memory: "16Gi"
+    args:
+      - --max-model-len=4096
+      - --tool-call-parser=hermes
+
+| Setting | Value |
+|---------|-------|
+| Model | llama-3-1-8b |
+| Runtime | vllm-runtime |
+| GPUs | 1 |
+| Namespace | ai-models |
+| Mode | Serverless |
+
+Proceed with creating this InferenceService? (yes/no/modify)
+
+User: "yes"
+
+Agent: Step 8 of 10: Creating InferenceService...
+       [Creates via resources_create_or_update]
+
+       InferenceService created. Monitoring rollout...
+
+Step 9 of 10: Monitoring rollout.
+
+| Stage | Status |
+|-------|--------|
+| Pod Scheduled | Complete |
+| Image Pulled | Complete |
+| Container Started | Complete |
+| Model Loaded | Complete |
+| Ready | Complete |
+
+Step 10 of 10: Deployment complete!
+
+Model Deployment Successful
+
+Model: llama-3-1-8b
+Runtime: vLLM
+Namespace: ai-models
+GPUs: 1
+
+Inference Endpoint:
+  URL: https://llama-3-1-8b-ai-models.apps.cluster.example.com
+  API: OpenAI-compatible REST
+
+Quick Test:
+  curl -X POST https://llama-3-1-8b-ai-models.apps.cluster.example.com/v1/chat/completions \
+    -H "Content-Type: application/json" \
+    -d '{"model": "llama-3-1-8b", "messages": [{"role": "user", "content": "Hello"}]}'
+
+Next Steps:
+  Monitor performance: "/ai-observability llama-3-1-8b"
+```
+
+## Example 2: Deploy a Model with NIM (NIM Not Set Up)
+
+```
+User: "Deploy Llama 3.1 70B with NIM"
+
+Agent: [Invokes /model-deploy skill]
+       ...
+       Step 2: User requested NIM runtime.
+
+       Step 5: Checking NIM platform...
+
+NIM platform is not set up in namespace ai-models.
+
+The NIM runtime requires a one-time platform setup. Would you like to:
+1. Run /nim-setup now to configure the NIM platform
+2. Switch to vLLM runtime instead (no setup required)
+3. Abort deployment
+
+User: "1"
+
+Agent: [Invokes /nim-setup skill — walks through NGC credential setup]
+       ...
+       NIM Platform Setup Complete.
+
+       Resuming model deployment...
+       [Continues from Step 6 with NIM runtime]
+```

submissions/model-deploy/skills/references/nim-setup.md

@@ -0,0 +1,115 @@
+# /nim-setup Examples
+
+Example interactions demonstrating the nim-setup skill workflow.
+
+## Example 1: First-Time NIM Setup
+
+```
+User: "Set up NIM on my cluster"
+
+Agent: [Invokes /nim-setup skill]
+       [Reads supported-runtimes.md]
+
+I consulted supported-runtimes.md to understand NIM platform requirements.
+
+Let me verify the prerequisites for NIM setup.
+
+Step 1 of 7: Checking GPU Operator and NFD Operator...
+
+GPU Operator: gpu-operator-certified.v24.6.2 (Succeeded)
+NFD Operator: nfd.v4.19.0 (Succeeded)
+
+Both operators are installed and healthy.
+
+Step 2 of 7: I need your NGC credentials.
+
+To set up NIM, I need your NVIDIA NGC API key.
+You can generate one at: https://ngc.nvidia.com/setup/api-key
+
+Please provide:
+1. Your NGC API key
+2. The target namespace for NIM resources
+
+User: "Here's my key: [user provides NGC API key], namespace: ai-models"
+
+Agent: Step 3 of 7: Creating NGC image pull secret...
+
+I'll create a docker-registry secret for pulling NIM images from nvcr.io:
+
+  apiVersion: v1
+  kind: Secret
+  metadata:
+    name: ngc-image-pull-secret
+    namespace: ai-models
+  type: kubernetes.io/dockerconfigjson
+  data:
+    .dockerconfigjson: [base64-encoded config for nvcr.io - REDACTED]
+
+Should I create this image pull secret in namespace `ai-models`? (yes/no)
+
+User: "yes"
+
+Agent: [Creates secret via resources_create_or_update]
+       Image pull secret created successfully.
+
+Step 4 of 7: Creating NGC API key secret...
+
+  apiVersion: v1
+  kind: Secret
+  metadata:
+    name: ngc-api-key
+    namespace: ai-models
+  type: Opaque
+  stringData:
+    NGC_API_KEY: "[REDACTED]"
+
+Should I create this API key secret in namespace `ai-models`? (yes/no)
+
+User: "yes"
+
+Agent: [Creates secret via resources_create_or_update]
+       API key secret created successfully.
+
+Step 5 of 7: Creating NIM Account CR...
+
+  apiVersion: nim.opendatahub.io/v1
+  kind: Account
+  metadata:
+    name: nim-account
+    namespace: ai-models
+  spec:
+    apiKeySecret:
+      name: ngc-api-key
+    imagePullSecret:
+      name: ngc-image-pull-secret
+
+Should I create this NIM Account CR in namespace `ai-models`? (yes/no)
+
+User: "yes"
+
+Agent: [Creates Account CR via resources_create_or_update]
+       Account CR created successfully.
+
+Step 6 of 7: Would you like to customize the NIM model catalog? (yes/no, default: no)
+
+User: "no"
+
+Agent: Step 7 of 7: Validating NIM platform readiness...
+
+NIM Platform Setup Complete
+
+Account CR: `nim-account` (namespace: ai-models)
+  Status: Ready
+
+NGC Credentials:
+  Image Pull Secret: ngc-image-pull-secret [created]
+  API Key Secret: ngc-api-key [created]
+
+NIM ServingRuntimes Available:
+  - nim-llama-3-1-8b-instruct
+  - nim-llama-3-1-70b-instruct
+  - nim-mistral-7b-instruct
+
+Next Steps:
+  To deploy a model with NIM: "/model-deploy llama-3.1-8b --runtime nim"
+```