Community corpus scan foundation

[jdutton]

Type

Research / evidence-producing. This issue does not ship rule changes or detector code. It ships a working scanner at corpus scale plus a triage document that becomes the evidence base for a separate rule-calibration PR.

Background

Phase 1 of the community corpus work shipped in 0.1.34: vat corpus scan runs vat audit and (optionally) vat skill review over a YAML-seeded list of ~11 plugins. That is the smallest viable instrument. This issue turns it into a real feedback loop.

Two-bucket discipline is non-negotiable:

Bucket	Reporting posture
Official/reviewed (Anthropic claude-plugins-official, knowledge-work-plugins, VAT, vibe-validate)	Named findings OK — we are learning from exemplary work
Community (everything else)	Aggregate-only, NEVER named. No public shaming.

Goal

Make vat corpus scan actually usable at corpus scale (50–200 artifacts) instead of the 11-plugin smoke test we have today, and produce the first real triage document with concrete rule-calibration proposals backed by evidence.

Tasks

• Expand corpus/seed.yaml to cover the full official/reviewed bucket: claude-plugins-official (~147), knowledge-work-plugins (~41), VAT, vibe-validate. Tag each entry with bucket: official|community, confidence: first-party|curated|listed, maturity: production|experimental|example.
• Community-bucket enumeration. Add a dynamic discovery path (e.g., GitHub topic search for claude-skills / anthropic-skills, npm scope search). Output for community entries must be structurally aggregate-only — make it impossible to commit named community findings by accident (enforce in the report writer, not just by convention).
• Report schema v2. Extend the existing report with meta-signals:
  • surfacesFound, skillsDiscovered, linkedFilesTraversed
  • scanDirectoryDiscoveryMisses (cases where direct audit finds skills that recursive scan missed)
  • per-surface counts (agents, commands, hooks, mcpServers) as observation-only
• Failed-load handling. Separate unloadable category for broken manifests / clone failures / network errors. Counts but does not pollute rule-firing stats.
• Rate-limiting + retries for git clones; respect GitHub's anonymous rate limits.
• Reproducibility. Write run outputs to docs/corpus/runs/<date>-<sha>/ with the resolved seed, VAT version, and the full report. Official-bucket findings are named; community-bucket findings are aggregate.
• First real run. Execute against the full official/reviewed corpus plus a community sample. Produce a triage document classifying each code's firings as true positive / false positive / ambiguous. Do NOT ship rule changes in this issue — that is a separate PR after triage lands.

Non-goals

• No rule additions or severity changes (those follow corpus evidence in a separate PR)
• No compat-badge rendering (separate effort, gated on triage output)
• No compat-verdict calibration (sister issue Empirical Chat | Cowork | Code compat research #100 covers that empirically)

Acceptance criteria

• bun run vat corpus scan corpus/seed.yaml --out tmp/run/ completes against ~150–250 artifacts in reasonable wall time (target: under 10 min on a laptop with warm git cache)
• A triage doc lands at docs/corpus/runs/<date>-<sha>/triage.md with per-code counts and at least 5 concrete rule-calibration proposals ranked by evidence weight
• Zero named community-bucket findings in any committed artifact

Reading order

1. docs/validation-codes.md — current rule set entering this work
2. docs/validation-rule-design.md — the rule-addition bar
3. packages/cli/src/commands/corpus/ — what is already built

Discipline reminders

• Evidence before rules
• Humble defaults (info / warning; error only with demonstrated harm)
• Configurability-first (every new rule must be suppressible via validation.severity / validation.allow)
• Non-judgment language — describe patterns, not authors
• [vat: ...] attribution prefix on every surface-level finding

Sister issue

#100 — empirical Chat/Cowork/Code compat-testing research — produces a complementary evidence base. Its findings should land before this issue's triage doc so the triage can incorporate compat-verdict reality alongside rule-firing reality.