Problem
Archetype inference (refusal_regression, hallucination_risk, format_drift, etc.) currently relies on regex matching and keyword detection in the judge's freeform rationale. This is fragile — if the judge model changes phrasing style, archetypes silently degrade to the
semantic_regression fallback.
Proposal
Include the archetype as a required field in the judge's structured output schema. Instead of inferring the failure type after the fact, ask the judge to classify it directly:
{
"verdict": "candidate_worse",
"confidence": 0.85,
"archetype": "refusal_regression",
"rationale": "..."
}
Fallback to the current heuristic inference if the judge omits or returns an unknown archetype.
Impact
- More reliable failure classification
- Decouples archetype accuracy from judge model's prose style
- Enables future archetype-specific decision logic
Problem
Archetype inference (
refusal_regression,hallucination_risk,format_drift, etc.) currently relies on regex matching and keyword detection in the judge's freeform rationale. This is fragile — if the judge model changes phrasing style, archetypes silently degrade to thesemantic_regressionfallback.Proposal
Include the archetype as a required field in the judge's structured output schema. Instead of inferring the failure type after the fact, ask the judge to classify it directly:
{ "verdict": "candidate_worse", "confidence": 0.85, "archetype": "refusal_regression", "rationale": "..." } Fallback to the current heuristic inference if the judge omits or returns an unknown archetype. Impact - More reliable failure classification - Decouples archetype accuracy from judge model's prose style - Enables future archetype-specific decision logic