Enhance onboarding with diversity skills and ML audit features

openclaw.plugin.json

@@ -9,6 +9,11 @@
   },
   "skills": [
     "skills/algorithm-selection",
+    "skills/metric-audit",
+    "skills/error-analysis",
+    "skills/repro-check",
+    "skills/experiment-tracker",
+    "skills/ablation-planner",
     "skills/baseline-runner",
     "skills/dataset-validate",
     "skills/idea-generation",

skills/ablation-planner/SKILL.md

@@ -0,0 +1,89 @@
+---
+name: ablation-planner
+description: "Use this when the project needs a focused ablation plan that ties method claims and model components to specific validation experiments."
+metadata:
+  {
+    "openclaw":
+      {
+        "emoji": "🧩",
+      },
+  }
+---
+
+# Ablation Planner
+
+**Don't ask permission. Just do it.**
+
+Use this skill before formal experiments when the method has identifiable components and the project needs a deliberate ablation plan instead of ad hoc toggles.
+
+Outputs go to the workspace root.
+
+## Use This When
+
+- `plan_res.md` exists
+- the method has clear components or design choices
+- the project will likely need ablations to defend its main claims
+
+## Do Not Use This When
+
+- the project is only reproducing a baseline with no new component
+- the model design is still too unstable to define meaningful ablations
+
+## Required Inputs
+
+- `plan_res.md`
+- `selection_res.md` when available
+- `paper/claim_inventory.md` when available
+
+If `plan_res.md` is missing, stop and say: `Run /research-plan first to define the implementation path.`
+
+## Required Output
+
+- `ablation_plan.md`
+
+## Workflow
+
+### Step 1: Read Claims and Components
+
+Read:
+
+- `plan_res.md`
+- `selection_res.md` when present
+- `paper/claim_inventory.md` when present
+
+Extract:
+
+- what the method claims
+- which components might explain the gains
+- which claims require direct validation
+
+### Step 2: Map Claims to Ablations
+
+For each main claim, define at least one ablation that tests whether the claimed component actually matters.
+
+### Step 3: Prioritize a Small Decisive Plan
+
+Keep 2-4 ablations only, with:
+
+- rationale
+- change to make
+- expected direction
+- cost or difficulty
+
+### Step 4: Write `ablation_plan.md`
+
+Use `references/ablation-plan-template.md`.
+
+The plan must include:
+
+- claim
+- component
+- experimental change
+- expected outcome
+- success criterion
+
+## Rules
+
+1. Every ablation must answer a claim, not just disable a module.
+2. Prefer a small decisive plan over a long weak list.
+3. Mark very expensive ablations as deferred instead of pretending they do not exist.

skills/ablation-planner/references/ablation-plan-template.md

@@ -0,0 +1,16 @@
+# Ablation Plan
+
+## Scope
+- Source files:
+- Main method:
+
+## Planned Ablations
+| Claim | Component | Experimental Change | Expected Outcome | Success Criterion | Priority |
+|-------|-----------|---------------------|------------------|-------------------|----------|
+| ... | ... | ... | ... | ... | high / medium / low |
+
+## Deferred Ablations
+1. ...
+
+## Next Step
+- ...

skills/error-analysis/SKILL.md

@@ -0,0 +1,94 @@
+---
+name: error-analysis
+description: "Use this when experiment results exist but the project still lacks a structured analysis of failure cases, bad buckets, and likely causes."
+metadata:
+  {
+    "openclaw":
+      {
+        "emoji": "🔎",
+      },
+  }
+---
+
+# Error Analysis
+
+**Don't ask permission. Just do it.**
+
+Use this skill after experiments when the team needs to understand where the model fails, not just how often.
+
+Outputs go to the workspace root.
+
+## Use This When
+
+- `experiment_res.md` exists
+- predictions, logs, or representative outputs are available
+- the project needs targeted follow-up experiments instead of only aggregate scores
+
+## Do Not Use This When
+
+- there are no experiment results yet
+- there are no outputs or examples to inspect
+
+## Required Inputs
+
+- `experiment_res.md`
+- prediction logs, generated outputs, or sample artifacts when available
+- `paper/claim_inventory.md` when available
+
+If `experiment_res.md` is missing, stop and say: `Run /research-experiment first to generate experiment results.`
+
+## Required Output
+
+- `error_analysis.md`
+
+## Workflow
+
+### Step 1: Read the Current Result Story
+
+Read:
+
+- `experiment_res.md`
+- prediction or output artifacts when present
+- `paper/claim_inventory.md` when present
+
+Extract:
+
+- headline wins
+- weak spots
+- suspicious gaps between aggregate metrics and actual behavior
+
+### Step 2: Build Failure Buckets
+
+Group failures by the most informative structure available, for example:
+
+- class or label
+- length or difficulty
+- source or domain
+- prompt type
+- retrieval quality
+- latency or resource regime
+
+### Step 3: Find Actionable Causes
+
+For each major bucket, identify:
+
+- what the failure looks like
+- what likely causes it
+- what follow-up experiment could test that cause
+
+### Step 4: Write `error_analysis.md`
+
+Use `references/error-analysis-template.md`.
+
+The report must include:
+
+- failure buckets
+- representative examples
+- likely causes
+- recommended next experiments
+
+## Rules
+
+1. Do not repeat aggregate metrics as if they were analysis.
+2. Every failure bucket must point to concrete evidence.
+3. Prefer actionable causes over broad speculation.

skills/error-analysis/references/error-analysis-template.md

@@ -0,0 +1,16 @@
+# Error Analysis
+
+## Scope
+- Source files:
+- Main task:
+
+## Failure Buckets
+| Bucket | Evidence | Likely Cause | Severity | Suggested Follow-up |
+|--------|----------|--------------|----------|---------------------|
+| ... | ... | ... | ... | ... |
+
+## Representative Cases
+1. ...
+
+## Recommended Next Experiments
+1. ...

skills/experiment-tracker/SKILL.md

@@ -0,0 +1,82 @@
+---
+name: experiment-tracker
+description: "Use this when a project needs a structured registry of experiment runs, configs, seeds, and outcomes across implementation and experiment stages."
+metadata:
+  {
+    "openclaw":
+      {
+        "emoji": "🗃️",
+      },
+  }
+---
+
+# Experiment Tracker
+
+**Don't ask permission. Just do it.**
+
+Use this skill to keep runs traceable when a project starts accumulating multiple configurations or repeated experiments.
+
+Outputs go to the workspace root.
+
+## Use This When
+
+- the project has started executing training or evaluation runs
+- more than one run or configuration is likely
+
+## Do Not Use This When
+
+- the project has not started execution
+- there are no run artifacts or logs to register
+
+## Required Inputs
+
+- `plan_res.md`
+- `ml_res.md` and/or `experiment_res.md`
+- config files, run logs, and execution commands when available
+
+If both `ml_res.md` and `experiment_res.md` are missing, stop and say: `Run /research-implement or /research-experiment first to create experiment artifacts.`
+
+## Required Output
+
+- `experiment_registry.md`
+
+## Workflow
+
+### Step 1: Collect Current Runs
+
+Read current execution artifacts and gather:
+
+- run identifier
+- config or config delta
+- seed
+- headline result
+- notes
+
+### Step 2: Normalize the Registry
+
+Write each run in one standard format so headline runs and exploratory runs can be compared safely.
+
+### Step 3: Separate Canonical Runs
+
+Mark:
+
+- headline runs
+- exploratory runs
+- failed or incomplete runs
+
+### Step 4: Write `experiment_registry.md`
+
+Use `references/experiment-registry-template.md`.
+
+The registry must include:
+
+- run table
+- best run
+- failed runs
+- missing metadata
+
+## Rules
+
+1. Every run should include config, seed, and result summary when available.
+2. Do not mix failed exploratory runs into headline claims.
+3. Missing metadata must be called out explicitly.

skills/experiment-tracker/references/experiment-registry-template.md

@@ -0,0 +1,15 @@
+# Experiment Registry
+
+## Run Table
+| Run ID | Purpose | Config / Delta | Seed | Headline Result | Status | Notes |
+|--------|---------|----------------|------|-----------------|--------|-------|
+| ... | ... | ... | ... | ... | headline / exploratory / failed | ... |
+
+## Best Current Run
+- ...
+
+## Failed or Incomplete Runs
+1. ...
+
+## Missing Metadata
+1. ...