[Feature] Deterministic Quality Validation (`capa validate`)

[Minitour]

Description

CAPA installs skills from multiple sources (GitLab, GitHub, registries, inline). There is no built-in mechanism to check whether an installed skill conforms to the SKILL.md specification or meets minimum quality thresholds.

A skill whose description is too short to trigger reliably, or whose frontmatter is malformed, installs identically to a well-formed one. Anthropic's skill creator guide documents undertriggering as a known failure mode caused by poor description quality.

Motivation

• Teams have no way to evaluate skill quality before installing
• Poorly described skills cause undertriggering — the agent fails to invoke the skill when it should
• Skills with broken references or oversized content waste context window tokens
• There's no quality gate for CI pipelines

Proposal

Introduce capa validate as a deterministic, fully offline quality checker for skills.

Scoring rubric (8 checks, 0–100)

Check	Points	Rationale
YAML frontmatter parseable	25	Fatal if missing — no further checks run
name field present and valid	10	Required for manifest and registry
description present	10	Primary triggering mechanism for agents
Description >= 30 words	10	Specificity threshold to prevent undertriggering
SKILL.md <= 500 lines	15	Token budget constraint — oversized skills waste context window
Allowed subdirectories only	10	Enforces scripts/, references/, assets/ only
Referenced resources exist	15	No broken relative links
Resources in correct subdirs	5	Scripts in scripts/, docs in references/

CLI interface

capa validate                         # validate all installed skills
capa validate skills/web-researcher   # validate a specific skill
capa validate --format json           # machine-readable output
capa validate --min-score 60          # exit 1 if any skill scores below threshold

Output format (compiler-style diagnostics)

Validating: web-researcher (gitlab:myorg/skills@web-researcher)

  pass    YAML frontmatter valid
  pass    name field present: "web-researcher"
  error   line 4: description too short (12 words, recommended: 30+)
  pass    SKILL.md line count OK (87 lines)
  warning Unknown subdirectory "helpers" — only scripts/, references/, assets/ allowed
  pass    All referenced resources exist

  Score: 45/100
  Validated against: skill-format v1.0

Design principles

• Warnings never block installation — a 12/100 skill can still be installed
• Scoring is deterministic — no LLM, no network calls, same input always produces same score
• CAPA does not own the spec — scoring is derived from Anthropic's published SKILL.md creator guide; if the spec changes, the scorer is updated
• Scores are informational — they help publishers improve quality, they don't gatekeep

Integration with capa audit

If policy enforcement (#policy-issue) lands first, add an optional policy field:

# capa-policy.yaml
validation:
  min_score: 60                   # reject skills below this score
  require_description: true       # require description >= 30 words

Web UI integration

Show validation scores in the CAPA web UI next to each installed skill, with visual indicators (green/yellow/red) and expandable diagnostics.

Acceptance Criteria

• capa validate runs all 8 checks locally with zero external dependencies
• Output includes compiler-style diagnostics with severity, line numbers, and messages
• --format json produces structured output for CI integration
• --min-score N flag exits non-zero if any skill scores below the threshold
• Scores are visible in the web UI
• Validation never blocks installation (informational only, unless policy enforcement is configured)