Skip to content

feat(judge): allow agent_judge to install a judge-specific skill #134

Description

@roark47

Background

Users want agent_judge to load a Skill dedicated to judging. The current configuration supports natural-language criteria, model selection, pass threshold, and timeout, but there is no way to install a Skill for the judge agent itself.

From a quick code read:

  • config.JudgeConfig currently has fields such as type, model, criteria, pass_threshold, and timeout_seconds, but no judge Skill reference.
  • defaultEvaluator.setupCaseEnvironment() installs evalCfg.Skills for the main run agent before case execution.
  • agent_judge later creates/resolves a judge agent in resolveJudgeAgent(), then judge.NewJudge() wraps it and AgentJudge.Evaluate() calls judgeAgent.Run() directly.
  • There is no install step that calls judgeAgent.InstallSkill(...) for a judge-specific Skill.

User Need

Some eval authors want to package judge behavior as a Skill instead of only inline YAML criteria. This is reasonable when judging requires:

  • domain-specific evaluation guidance that is too large or reusable for judge.criteria;
  • strict output/rubric instructions shared across many evals;
  • specialized comparison logic, style guides, or policy checks;
  • a stable judge prompt that can evolve independently from individual case YAML files.

This should be separate from the Skill under test. The runner agent may need the evaluated Skill installed, while the judge agent may need a different Skill that teaches it how to grade.

Possible Design

Add a judge-level Skill configuration, for example:

judge:
  type: agent_judge
  model: anthropic/claude-sonnet-4-6
  skill:
    source: local_path
    path: evals/fixtures/judge-skill
  criteria:
    - "The answer is correct according to the judge skill rubric"

Or support multiple judge Skills if that better matches existing skills semantics:

judge:
  type: agent_judge
  model: anthropic/claude-sonnet-4-6
  skills:
    - source: local_path
      path: evals/fixtures/judge-skill

Expected behavior:

  • Only agent_judge uses the judge Skill configuration.
  • The judge Skill is installed into the judge agent before AgentJudge.Evaluate() calls Run.
  • The judge Skill does not replace or leak into the main run agent's evaluated Skill installation.
  • Existing judge.criteria behavior remains backward compatible and can still be used alongside a judge Skill.
  • Relative paths should resolve consistently with existing local Skill/MCP fixture conventions.

Acceptance Criteria

  • JudgeConfig supports a documented judge Skill reference for type: agent_judge.
  • The evaluator installs the configured judge Skill(s) into the judge agent before invoking the judge run.
  • The main run agent and judge agent keep their Skill installations isolated.
  • Unit tests cover config loading, validation, and evaluator install behavior.
  • An integration/e2e-style fixture demonstrates an agent_judge case using a judge-specific Skill.
  • Documentation is updated in docs/guide/writing-evals.md, docs/zh/guide/writing-evals.md, and the skill-upper reference/template docs.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Fields

No fields configured for Feature.

Projects

Status
Backlog

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions