diff --git a/openclaw.plugin.json b/openclaw.plugin.json index 14e51aa..890e11c 100644 --- a/openclaw.plugin.json +++ b/openclaw.plugin.json @@ -9,6 +9,11 @@ }, "skills": [ "skills/algorithm-selection", + "skills/metric-audit", + "skills/error-analysis", + "skills/repro-check", + "skills/experiment-tracker", + "skills/ablation-planner", "skills/baseline-runner", "skills/dataset-validate", "skills/idea-generation", diff --git a/skills/ablation-planner/SKILL.md b/skills/ablation-planner/SKILL.md new file mode 100644 index 0000000..07d915f --- /dev/null +++ b/skills/ablation-planner/SKILL.md @@ -0,0 +1,89 @@ +--- +name: ablation-planner +description: "Use this when the project needs a focused ablation plan that ties method claims and model components to specific validation experiments." +metadata: + { + "openclaw": + { + "emoji": "🧩", + }, + } +--- + +# Ablation Planner + +**Don't ask permission. Just do it.** + +Use this skill before formal experiments when the method has identifiable components and the project needs a deliberate ablation plan instead of ad hoc toggles. + +Outputs go to the workspace root. + +## Use This When + +- `plan_res.md` exists +- the method has clear components or design choices +- the project will likely need ablations to defend its main claims + +## Do Not Use This When + +- the project is only reproducing a baseline with no new component +- the model design is still too unstable to define meaningful ablations + +## Required Inputs + +- `plan_res.md` +- `selection_res.md` when available +- `paper/claim_inventory.md` when available + +If `plan_res.md` is missing, stop and say: `Run /research-plan first to define the implementation path.` + +## Required Output + +- `ablation_plan.md` + +## Workflow + +### Step 1: Read Claims and Components + +Read: + +- `plan_res.md` +- `selection_res.md` when present +- `paper/claim_inventory.md` when present + +Extract: + +- what the method claims +- which components might explain the gains +- which claims require direct validation + +### Step 2: Map Claims to Ablations + +For each main claim, define at least one ablation that tests whether the claimed component actually matters. + +### Step 3: Prioritize a Small Decisive Plan + +Keep 2-4 ablations only, with: + +- rationale +- change to make +- expected direction +- cost or difficulty + +### Step 4: Write `ablation_plan.md` + +Use `references/ablation-plan-template.md`. + +The plan must include: + +- claim +- component +- experimental change +- expected outcome +- success criterion + +## Rules + +1. Every ablation must answer a claim, not just disable a module. +2. Prefer a small decisive plan over a long weak list. +3. Mark very expensive ablations as deferred instead of pretending they do not exist. diff --git a/skills/ablation-planner/references/ablation-plan-template.md b/skills/ablation-planner/references/ablation-plan-template.md new file mode 100644 index 0000000..7123a05 --- /dev/null +++ b/skills/ablation-planner/references/ablation-plan-template.md @@ -0,0 +1,16 @@ +# Ablation Plan + +## Scope +- Source files: +- Main method: + +## Planned Ablations +| Claim | Component | Experimental Change | Expected Outcome | Success Criterion | Priority | +|-------|-----------|---------------------|------------------|-------------------|----------| +| ... | ... | ... | ... | ... | high / medium / low | + +## Deferred Ablations +1. ... + +## Next Step +- ... diff --git a/skills/error-analysis/SKILL.md b/skills/error-analysis/SKILL.md new file mode 100644 index 0000000..7d18d20 --- /dev/null +++ b/skills/error-analysis/SKILL.md @@ -0,0 +1,94 @@ +--- +name: error-analysis +description: "Use this when experiment results exist but the project still lacks a structured analysis of failure cases, bad buckets, and likely causes." +metadata: + { + "openclaw": + { + "emoji": "🔎", + }, + } +--- + +# Error Analysis + +**Don't ask permission. Just do it.** + +Use this skill after experiments when the team needs to understand where the model fails, not just how often. + +Outputs go to the workspace root. + +## Use This When + +- `experiment_res.md` exists +- predictions, logs, or representative outputs are available +- the project needs targeted follow-up experiments instead of only aggregate scores + +## Do Not Use This When + +- there are no experiment results yet +- there are no outputs or examples to inspect + +## Required Inputs + +- `experiment_res.md` +- prediction logs, generated outputs, or sample artifacts when available +- `paper/claim_inventory.md` when available + +If `experiment_res.md` is missing, stop and say: `Run /research-experiment first to generate experiment results.` + +## Required Output + +- `error_analysis.md` + +## Workflow + +### Step 1: Read the Current Result Story + +Read: + +- `experiment_res.md` +- prediction or output artifacts when present +- `paper/claim_inventory.md` when present + +Extract: + +- headline wins +- weak spots +- suspicious gaps between aggregate metrics and actual behavior + +### Step 2: Build Failure Buckets + +Group failures by the most informative structure available, for example: + +- class or label +- length or difficulty +- source or domain +- prompt type +- retrieval quality +- latency or resource regime + +### Step 3: Find Actionable Causes + +For each major bucket, identify: + +- what the failure looks like +- what likely causes it +- what follow-up experiment could test that cause + +### Step 4: Write `error_analysis.md` + +Use `references/error-analysis-template.md`. + +The report must include: + +- failure buckets +- representative examples +- likely causes +- recommended next experiments + +## Rules + +1. Do not repeat aggregate metrics as if they were analysis. +2. Every failure bucket must point to concrete evidence. +3. Prefer actionable causes over broad speculation. diff --git a/skills/error-analysis/references/error-analysis-template.md b/skills/error-analysis/references/error-analysis-template.md new file mode 100644 index 0000000..151d4ae --- /dev/null +++ b/skills/error-analysis/references/error-analysis-template.md @@ -0,0 +1,16 @@ +# Error Analysis + +## Scope +- Source files: +- Main task: + +## Failure Buckets +| Bucket | Evidence | Likely Cause | Severity | Suggested Follow-up | +|--------|----------|--------------|----------|---------------------| +| ... | ... | ... | ... | ... | + +## Representative Cases +1. ... + +## Recommended Next Experiments +1. ... diff --git a/skills/experiment-tracker/SKILL.md b/skills/experiment-tracker/SKILL.md new file mode 100644 index 0000000..43952d1 --- /dev/null +++ b/skills/experiment-tracker/SKILL.md @@ -0,0 +1,82 @@ +--- +name: experiment-tracker +description: "Use this when a project needs a structured registry of experiment runs, configs, seeds, and outcomes across implementation and experiment stages." +metadata: + { + "openclaw": + { + "emoji": "🗃️", + }, + } +--- + +# Experiment Tracker + +**Don't ask permission. Just do it.** + +Use this skill to keep runs traceable when a project starts accumulating multiple configurations or repeated experiments. + +Outputs go to the workspace root. + +## Use This When + +- the project has started executing training or evaluation runs +- more than one run or configuration is likely + +## Do Not Use This When + +- the project has not started execution +- there are no run artifacts or logs to register + +## Required Inputs + +- `plan_res.md` +- `ml_res.md` and/or `experiment_res.md` +- config files, run logs, and execution commands when available + +If both `ml_res.md` and `experiment_res.md` are missing, stop and say: `Run /research-implement or /research-experiment first to create experiment artifacts.` + +## Required Output + +- `experiment_registry.md` + +## Workflow + +### Step 1: Collect Current Runs + +Read current execution artifacts and gather: + +- run identifier +- config or config delta +- seed +- headline result +- notes + +### Step 2: Normalize the Registry + +Write each run in one standard format so headline runs and exploratory runs can be compared safely. + +### Step 3: Separate Canonical Runs + +Mark: + +- headline runs +- exploratory runs +- failed or incomplete runs + +### Step 4: Write `experiment_registry.md` + +Use `references/experiment-registry-template.md`. + +The registry must include: + +- run table +- best run +- failed runs +- missing metadata + +## Rules + +1. Every run should include config, seed, and result summary when available. +2. Do not mix failed exploratory runs into headline claims. +3. Missing metadata must be called out explicitly. diff --git a/skills/experiment-tracker/references/experiment-registry-template.md b/skills/experiment-tracker/references/experiment-registry-template.md new file mode 100644 index 0000000..88789ca --- /dev/null +++ b/skills/experiment-tracker/references/experiment-registry-template.md @@ -0,0 +1,15 @@ +# Experiment Registry + +## Run Table +| Run ID | Purpose | Config / Delta | Seed | Headline Result | Status | Notes | +|--------|---------|----------------|------|-----------------|--------|-------| +| ... | ... | ... | ... | ... | headline / exploratory / failed | ... | + +## Best Current Run +- ... + +## Failed or Incomplete Runs +1. ... + +## Missing Metadata +1. ... diff --git a/skills/metric-audit/SKILL.md b/skills/metric-audit/SKILL.md new file mode 100644 index 0000000..b288add --- /dev/null +++ b/skills/metric-audit/SKILL.md @@ -0,0 +1,98 @@ +--- +name: metric-audit +description: "Use this when a project needs a dedicated review of metrics, baselines, guardrails, and evidence boundaries before or after experiments." +metadata: + { + "openclaw": + { + "emoji": "📏", + }, + } +--- + +# Metric Audit + +**Don't ask permission. Just do it.** + +Use this skill when a project already has an evaluation plan or experiment results, but the metric contract is still unclear, mixed, or hard to defend. + +Outputs go to the workspace root. + +## Use This When + +- `plan_res.md` exists +- or `experiment_res.md` already exists +- the project has headline numbers that need a consistent protocol and baseline story + +## Do Not Use This When + +- the project has not defined evaluation targets yet +- there are no metrics, guards, or baselines to inspect + +## Required Inputs + +- `plan_res.md` +- `experiment_res.md` when available +- `baseline_res.md` when available +- `paper/claim_inventory.md` when available + +If `plan_res.md` is missing, stop and say: `Run /research-plan first to define the evaluation protocol.` + +## Required Output + +- `metric_audit.md` + +## Workflow + +### Step 1: Read the Metric Contract + +Read: + +- `plan_res.md` +- `experiment_res.md` when present +- `baseline_res.md` when present +- `paper/claim_inventory.md` when present + +Extract: + +- headline metrics +- units +- averaging rules +- baselines +- thresholds or guardrails +- evidence layer (`simulator`, `local_runtime`, or `full_runtime`) + +### Step 2: Audit Consistency + +Check: + +- whether the same metric name means the same thing everywhere +- whether units and averaging are explicit +- whether each headline metric has a named baseline +- whether guardrails are present when claims depend on them + +### Step 3: Audit Evidence Boundaries + +Check: + +- whether simulator or proxy evidence is written as runtime evidence +- whether multiple runtime layers are mixed in one conclusion without being named + +### Step 4: Write `metric_audit.md` + +Use `references/metric-audit-template.md`. + +The report must include: + +- metric inventory +- consistency findings +- missing baseline or protocol notes +- evidence-boundary findings +- verdict: `PASS`, `NEEDS_REVISION`, or `BLOCKED` +- exact next step + +## Rules + +1. Never accept a headline metric without a baseline or protocol note. +2. Never merge different runtime layers into one unqualified claim. +3. Treat ambiguous units or averaging rules as blocking until clarified. diff --git a/skills/metric-audit/references/metric-audit-template.md b/skills/metric-audit/references/metric-audit-template.md new file mode 100644 index 0000000..fb47e30 --- /dev/null +++ b/skills/metric-audit/references/metric-audit-template.md @@ -0,0 +1,19 @@ +# Metric Audit + +## Scope +- Files reviewed: +- Headline metrics: + +## Metric Inventory +| Metric | Unit | Averaging | Baseline | Guardrail | Evidence Layer | Status | +|--------|------|-----------|----------|-----------|----------------|--------| +| ... | ... | ... | ... | ... | ... | PASS / NEEDS_REVISION | + +## Findings +1. ... + +## Verdict +PASS / NEEDS_REVISION / BLOCKED + +## Next Step +- ... diff --git a/skills/repro-check/SKILL.md b/skills/repro-check/SKILL.md new file mode 100644 index 0000000..3b10fce --- /dev/null +++ b/skills/repro-check/SKILL.md @@ -0,0 +1,99 @@ +--- +name: repro-check +description: "Use this when a project needs a reproducibility check across code, environment, commands, seeds, and artifact paths before sharing results." +metadata: + { + "openclaw": + { + "emoji": "♻️", + "requires": { "bins": ["python3", "uv"] }, + }, + } +--- + +# Repro Check + +**Don't ask permission. Just do it.** + +Use this skill before sharing results or making strong claims about repeatability. + +Outputs go to the workspace root. + +## Use This When + +- `ml_res.md` or `experiment_res.md` exists +- the team needs confidence that a clean rerun is possible + +## Do Not Use This When + +- implementation has not been executed yet +- there is no runnable code or command surface to inspect + +## Required Inputs + +- `plan_res.md` +- `ml_res.md` and/or `experiment_res.md` +- `project/` +- run commands, configs, and environment notes when available + +If both `ml_res.md` and `experiment_res.md` are missing, stop and say: `Run /research-implement or /research-experiment first to generate executable results.` + +## Required Output + +- `repro_check.md` + +## Workflow + +### Step 1: Read the Execution Contract + +Read: + +- `plan_res.md` +- `ml_res.md` +- `experiment_res.md` +- runnable scripts and config files under `project/` + +Extract: + +- entry command +- config source +- seed handling +- environment assumptions +- data paths +- output paths + +### Step 2: Audit Reproducibility Surface + +Check: + +- whether a clean rerun command is obvious +- whether the environment is declared enough to recreate +- whether seeds are named +- whether data paths are explicit +- whether outputs map cleanly to the claimed results + +### Step 3: Decide Reproduction Status + +Return one of: + +- `PASS` +- `PARTIAL` +- `BLOCKED` + +### Step 4: Write `repro_check.md` + +Use `references/repro-check-template.md`. + +The report must include: + +- rerun command +- missing prerequisites +- exact blockers +- reproducibility verdict +- next step + +## Rules + +1. Missing seed or missing command is never a full pass. +2. Separate "can rerun" from "will match exactly". +3. Treat hidden manual steps as blocking. diff --git a/skills/repro-check/references/repro-check-template.md b/skills/repro-check/references/repro-check-template.md new file mode 100644 index 0000000..04008a6 --- /dev/null +++ b/skills/repro-check/references/repro-check-template.md @@ -0,0 +1,24 @@ +# Repro Check + +## Scope +- Files reviewed: +- Target result: + +## Reproduction Surface +| Item | Present | Notes | +|------|---------|-------| +| Entry command | yes / no | ... | +| Config source | yes / no | ... | +| Seed | yes / no | ... | +| Data path | yes / no | ... | +| Environment notes | yes / no | ... | +| Output path | yes / no | ... | + +## Verdict +PASS / PARTIAL / BLOCKED + +## Blockers +1. ... + +## Next Step +- ... diff --git a/skills/research-experiment/SKILL.md b/skills/research-experiment/SKILL.md index 65b50a0..7f730a6 100644 --- a/skills/research-experiment/SKILL.md +++ b/skills/research-experiment/SKILL.md @@ -186,3 +186,9 @@ python3 run.py --experiment {exp_name} 5. **补充实验迭代必须做 2 轮（Novix Exp Analyzer 机制）** — 第 1 轮针对初始结果，第 2 轮针对补充实验结果 6. 补充实验不改核心算法，只改实验配置/参数/可视化代码 7. Every headline metric must include a baseline, and every main conclusion must point back to real outputs or figure files + +## Follow-on Skills + +- Use `/experiment-tracker` when multiple runs or config branches need one clean registry. +- Use `/error-analysis` after this skill when aggregate metrics exist but failure buckets are still unclear. +- Use `/metric-audit` after this skill when headline numbers, baselines, or evidence boundaries need a dedicated consistency review. diff --git a/skills/research-plan/SKILL.md b/skills/research-plan/SKILL.md index 6229364..2bd2560 100644 --- a/skills/research-plan/SKILL.md +++ b/skills/research-plan/SKILL.md @@ -129,3 +129,8 @@ metadata: 2. 不要写"通用"计划 — 每个参数都要有具体值或合理估计 3. 如果参考仓库存在，组件表必须包含参考代码路径 4. plan_res.md 的完成标志：四个部分都存在且非空 + +## Follow-on Skills + +- Use `/metric-audit` after this skill when metric wording, baselines, guardrails, or evidence layers still need a dedicated review. +- Use `/ablation-planner` after this skill when the project needs a focused component-level validation plan before full experiments. diff --git a/skills/research-review/SKILL.md b/skills/research-review/SKILL.md index 93cf773..a873a6d 100644 --- a/skills/research-review/SKILL.md +++ b/skills/research-review/SKILL.md @@ -319,6 +319,10 @@ metadata: --- +## Follow-on Skills + +- Use `/repro-check` after a PASS verdict when the team needs a dedicated reproducibility review before sharing results or writing strong claims. + ## Rules ### 审查标准 diff --git a/src/templates/bootstrap.ts b/src/templates/bootstrap.ts index f8d8f10..ae05ba7 100644 --- a/src/templates/bootstrap.ts +++ b/src/templates/bootstrap.ts @@ -25,10 +25,15 @@ You were just created as the research agent for project "${projectName}". Comple - existing papers/ -> /research-survey - existing survey_res.md, but route still undecided -> /algorithm-selection - existing survey_res.md -> /research-plan + - existing plan_res.md, and metric definitions or evidence layers still look unclear -> /metric-audit - existing plan_res.md, but data quality still needs review -> /dataset-validate + - existing plan_res.md, and the project needs a decisive component-level validation plan -> /ablation-planner - existing plan_res.md, and matched baseline evidence is needed -> /baseline-runner + - multiple runs or configs already exist, but the registry is still messy -> /experiment-tracker - existing implementation code -> /research-review + - existing review PASS, and reproducibility needs to be checked before sharing -> /repro-check - existing review PASS -> /research-experiment + - existing experiment_res.md, but failure modes are still unclear -> /error-analysis - existing experiment_res.md plus figures -> /write-paper - existing paper/draft.md -> /artifact-review - existing figures with inconsistent captions or style -> /figure-standardize @@ -40,9 +45,14 @@ You were just created as the research agent for project "${projectName}". Comple - I only have a research direction and no materials yet -> finish setup, then run /metabolism - I already have a batch of papers -> run /research-survey - I finished the survey but have not chosen a model route -> run /algorithm-selection +- I have a plan but the metric and evidence contract still feels unclear -> run /metric-audit - I already have a plan and want to validate the data first -> run /dataset-validate +- I already have a plan and need a small, decisive ablation set -> run /ablation-planner - I already have a plan and want honest baseline numbers first -> run /baseline-runner +- I already have several runs and want one clean registry -> run /experiment-tracker - I already have implementation code -> run /research-review +- I already passed review and want to check reproducibility before sharing -> run /repro-check +- I already have experiment results but need structured failure analysis -> run /error-analysis - I already have experiment results and figures -> run /write-paper - I already have a draft that needs review -> run /artifact-review @@ -173,11 +183,16 @@ Do not modify output files once written unless the user explicitly asks you to. | /research-survey | knowledge/, survey_res.md | | /algorithm-selection | selection_res.md | | /research-plan | plan_res.md | +| /metric-audit | metric_audit.md | | /dataset-validate | data_validation.md | +| /ablation-planner | ablation_plan.md | | /baseline-runner | baseline_res.md, experiments/baselines/ | | /research-implement | experiments/ | | /research-review | experiments/review/ | +| /repro-check | repro_check.md | | /research-experiment | experiments/results/, experiment_res.md | +| /experiment-tracker | experiment_registry.md | +| /error-analysis | error_analysis.md | | /idea-generation | ideas/ | | /write-review-paper | review/ | | /write-paper | paper/claim_inventory.md, paper/figures_manifest.md, paper/draft.md, paper/manuscript.tex, paper/build/manuscript.pdf | @@ -190,12 +205,27 @@ Do not modify output files once written unless the user explicitly asks you to. - Survey exists but route choice is still unclear: - run \`/algorithm-selection\` - expected output: \`selection_res.md\` +- Plan exists and metric wording, baselines, or evidence layers need review: + - run \`/metric-audit\` + - expected output: \`metric_audit.md\` - Plan exists and the project needs a dedicated data-quality check: - run \`/dataset-validate\` - expected output: \`data_validation.md\` +- Plan exists and the project needs a concise component-level ablation set: + - run \`/ablation-planner\` + - expected output: \`ablation_plan.md\` - Plan exists and the project needs honest comparison numbers: - run \`/baseline-runner\` - expected outputs: \`baseline_res.md\`, optional baseline artifacts under \`experiments/baselines/\` +- Several runs already exist and the project needs a clean ledger: + - run \`/experiment-tracker\` + - expected output: \`experiment_registry.md\` +- Review already passed and the team needs reproducibility confidence before sharing: + - run \`/repro-check\` + - expected output: \`repro_check.md\` +- Experiment results exist but failure buckets are still unclear: + - run \`/error-analysis\` + - expected output: \`error_analysis.md\` ## Writing and Release Entry Points