Problem
Each ambiguous prompt gets a single judge call. A single LLM judgment can be noisy — one bad call can flip a prompt from "equivalent" to "candidate_worse" and propagate into the batch decision.
Proposal
Add an optional judge_ensemble config:
evaluation:
judge_strategy: tiered
judge_ensemble_calls: 3 # default: 1 (current behavior)
judge_ensemble_threshold: 0.66 # majority required to accept verdict
When judge_ensemble_calls > 1:
- Run N independent judge calls per prompt
- Take majority verdict
- Average confidence scores
- Flag prompts with split verdicts for human review in the report
Impact
- Reduces single-call noise at the cost of N× judge spend
- Opt-in — default stays at 1 call, no behavior change
Problem
Each ambiguous prompt gets a single judge call. A single LLM judgment can be noisy — one bad call can flip a prompt from "equivalent" to "candidate_worse" and propagate into the batch decision.
Proposal
Add an optional
judge_ensembleconfig:When
judge_ensemble_calls > 1:Impact