feat: add mcp-aap-validator ASE evaluation submission

submissions/mcp-aap-validator/evals/evals.json

@@ -0,0 +1,17 @@
+{
+  "skill_name": "mcp-aap-validator",
+  "evals": [
+    {
+      "id": "mcp-aap-validator-eval",
+      "name": "AAP MCP Validation Methodology",
+      "prompt": "You are a Red Hat SRE about to run a remediation workflow that uses Ansible Automation Platform. First you need to verify AAP connectivity.\n\nDescribe your AAP MCP validation methodology: what servers you test, what tool calls you use for validation, what the common failure modes are, and how you present the results.",
+      "expected_output": "A validation methodology that tests both AAP MCP servers (job-management and inventory-management), explains that AAP_MCP_SERVER must point to the MCP gateway endpoint (not the UI URL), and presents per-server PASSED/FAILED outcomes in a structured table.",
+      "assertions": [
+        "The output describes testing both AAP MCP servers: job-management (job_templates_list) and inventory-management (inventories_list).",
+        "The output explains that AAP_MCP_SERVER must point to the MCP gateway endpoint, not the main AAP UI URL, and that 404 errors indicate a wrong URL.",
+        "The output describes common failure modes including expired tokens (401), insufficient RBAC permissions (403), and wrong URL (404).",
+        "The output presents validation results in a structured table with per-server PASSED/FAILED/PARTIAL outcomes."
+      ]
+    }
+  ]
+}

submissions/mcp-aap-validator/metadata.yaml

@@ -0,0 +1,5 @@
+name: mcp-aap-validator
+description: "Validate AAP (Ansible Automation Platform) MCP server connectivity."
+persona: rh-sre
+version: "1.0.0"
+eval_engine: ase

submissions/mcp-aap-validator/skills/SKILL.md

@@ -0,0 +1,75 @@
+---
+name: mcp-aap-validator
+description: 'Validate AAP (Ansible Automation Platform) MCP server connectivity.
+  Use when the user asks to "validate AAP MCP", "check AAP connection", or when other
+  skills need to verify AAP MCP availability before job management or inventory operations.
+
+  '
+license: Apache-2.0
+---
+
+# MCP AAP Validator
+
+Validates connectivity to AAP MCP servers by running lightweight tool calls.
+
+## When to Use This Skill
+
+Use when validating AAP MCP before job template operations, troubleshooting connection issues, or when other skills (e.g. playbook-executor) need to verify availability. Do NOT use for creating templates—use job-template-creator.
+
+## Workflow
+
+1. **Test connectivity**: Call these tools to verify each server responds:
+   - `job_templates_list` (page_size: 10) from aap-mcp-job-management
+   - `inventories_list` (page_size: 10) from aap-mcp-inventory-management
+2. **If any fails**: Provide a comprehensive message with possible root causes (see below).
+3. **Report**: Output a table with validated servers and outcome (emojis).
+
+## Failure Message (Root Causes)
+
+When a tool call fails, include:
+
+```
+❌ AAP MCP connection failed
+
+**Possible root causes:**
+- **Credentials**: AAP_MCP_SERVER or AAP_API_TOKEN not set or invalid
+- **401 Unauthorized**: Token expired or invalid → regenerate in AAP Web UI
+- **403 Forbidden**: Token lacks RBAC permissions (need Job Templates, Inventories)
+- **404 Not Found**: Wrong AAP_MCP_SERVER URL (must point to MCP gateway, not main AAP UI)
+- **Connection timeout**: Server unreachable, firewall, or network issue
+- **SSL/TLS error**: Certificate verification problem
+
+**Troubleshooting:**
+1. Verify env vars: AAP_MCP_SERVER, AAP_API_TOKEN (never echo values)
+2. Get token: AAP Web UI → Users → [Your User] → Tokens → Create
+3. Ensure AAP_MCP_SERVER points to MCP gateway endpoint
+4. Restart host after config changes
+```
+
+## Report Format
+
+Always end with a table:
+
+| Server | Outcome |
+|--------|---------|
+| aap-mcp-job-management | ✅ PASSED |
+| aap-mcp-inventory-management | ✅ PASSED |
+
+Use ✅ for success, ❌ for failure, ⚠️ for partial (e.g. one server OK, one failed).
+
+## Dependencies
+
+### Required MCP Servers
+- `aap-mcp-job-management` - AAP job template and execution
+- `aap-mcp-inventory-management` - AAP inventory management
+
+### Required MCP Tools
+- `job_templates_list` (from aap-mcp-job-management) - Connectivity test
+- `inventories_list` (from aap-mcp-inventory-management) - Connectivity test
+
+### Related Skills
+- `/playbook-executor` - Requires AAP MCP validation before execution
+- `/job-template-creator` - Requires AAP MCP validation before template operations
+
+### Reference Documentation
+- [AAP Integration Test Guide](../../docs/testing/aap-integration-test-guide.md) - AAP MCP setup, env vars, connectivity verification