Skip to content

Judge: support ensemble voting for higher confidence verdicts #4

Description

@riccardomerenda

Problem

Each ambiguous prompt gets a single judge call. A single LLM judgment can be noisy — one bad call can flip a prompt from "equivalent" to "candidate_worse" and propagate into the batch decision.

Proposal

Add an optional judge_ensemble config:

evaluation:
  judge_strategy: tiered
  judge_ensemble_calls: 3        # default: 1 (current behavior)
  judge_ensemble_threshold: 0.66 # majority required to accept verdict

When judge_ensemble_calls > 1:

  • Run N independent judge calls per prompt
  • Take majority verdict
  • Average confidence scores
  • Flag prompts with split verdicts for human review in the report

Impact

  • Reduces single-call noise at the cost of N× judge spend
  • Opt-in — default stays at 1 call, no behavior change

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions