Background
Users want agent_judge to load a Skill dedicated to judging. The current configuration supports natural-language criteria, model selection, pass threshold, and timeout, but there is no way to install a Skill for the judge agent itself.
From a quick code read:
config.JudgeConfig currently has fields such as type, model, criteria, pass_threshold, and timeout_seconds, but no judge Skill reference.
defaultEvaluator.setupCaseEnvironment() installs evalCfg.Skills for the main run agent before case execution.
agent_judge later creates/resolves a judge agent in resolveJudgeAgent(), then judge.NewJudge() wraps it and AgentJudge.Evaluate() calls judgeAgent.Run() directly.
- There is no install step that calls
judgeAgent.InstallSkill(...) for a judge-specific Skill.
User Need
Some eval authors want to package judge behavior as a Skill instead of only inline YAML criteria. This is reasonable when judging requires:
- domain-specific evaluation guidance that is too large or reusable for
judge.criteria;
- strict output/rubric instructions shared across many evals;
- specialized comparison logic, style guides, or policy checks;
- a stable judge prompt that can evolve independently from individual case YAML files.
This should be separate from the Skill under test. The runner agent may need the evaluated Skill installed, while the judge agent may need a different Skill that teaches it how to grade.
Possible Design
Add a judge-level Skill configuration, for example:
judge:
type: agent_judge
model: anthropic/claude-sonnet-4-6
skill:
source: local_path
path: evals/fixtures/judge-skill
criteria:
- "The answer is correct according to the judge skill rubric"
Or support multiple judge Skills if that better matches existing skills semantics:
judge:
type: agent_judge
model: anthropic/claude-sonnet-4-6
skills:
- source: local_path
path: evals/fixtures/judge-skill
Expected behavior:
- Only
agent_judge uses the judge Skill configuration.
- The judge Skill is installed into the judge agent before
AgentJudge.Evaluate() calls Run.
- The judge Skill does not replace or leak into the main run agent's evaluated Skill installation.
- Existing
judge.criteria behavior remains backward compatible and can still be used alongside a judge Skill.
- Relative paths should resolve consistently with existing local Skill/MCP fixture conventions.
Acceptance Criteria
JudgeConfig supports a documented judge Skill reference for type: agent_judge.
- The evaluator installs the configured judge Skill(s) into the judge agent before invoking the judge run.
- The main run agent and judge agent keep their Skill installations isolated.
- Unit tests cover config loading, validation, and evaluator install behavior.
- An integration/e2e-style fixture demonstrates an
agent_judge case using a judge-specific Skill.
- Documentation is updated in
docs/guide/writing-evals.md, docs/zh/guide/writing-evals.md, and the skill-upper reference/template docs.
Background
Users want
agent_judgeto load a Skill dedicated to judging. The current configuration supports natural-language criteria, model selection, pass threshold, and timeout, but there is no way to install a Skill for the judge agent itself.From a quick code read:
config.JudgeConfigcurrently has fields such astype,model,criteria,pass_threshold, andtimeout_seconds, but no judge Skill reference.defaultEvaluator.setupCaseEnvironment()installsevalCfg.Skillsfor the main run agent before case execution.agent_judgelater creates/resolves a judge agent inresolveJudgeAgent(), thenjudge.NewJudge()wraps it andAgentJudge.Evaluate()callsjudgeAgent.Run()directly.judgeAgent.InstallSkill(...)for a judge-specific Skill.User Need
Some eval authors want to package judge behavior as a Skill instead of only inline YAML criteria. This is reasonable when judging requires:
judge.criteria;This should be separate from the Skill under test. The runner agent may need the evaluated Skill installed, while the judge agent may need a different Skill that teaches it how to grade.
Possible Design
Add a judge-level Skill configuration, for example:
Or support multiple judge Skills if that better matches existing
skillssemantics:Expected behavior:
agent_judgeuses the judge Skill configuration.AgentJudge.Evaluate()callsRun.judge.criteriabehavior remains backward compatible and can still be used alongside a judge Skill.Acceptance Criteria
JudgeConfigsupports a documented judge Skill reference fortype: agent_judge.agent_judgecase using a judge-specific Skill.docs/guide/writing-evals.md,docs/zh/guide/writing-evals.md, and the skill-upper reference/template docs.