Docs: settle the vera judge command plan - #193
Open
luca-belli wants to merge 1 commit into
Open
Conversation
Record the decisions needed before `vera judge` can be built, so the implementation is a mechanical mirror of #190 rather than a design exercise. Name the evaluation output segment for the target rather than the rubric (`evaluations/<target>/`), matching `c_<chatbot>`. Only the path segment changes: `judging.rubrics` keeps its name under AD-20, which exists to prevent exactly this kind of schema break, and `--rubric` already takes a target name. Clarify in Phase 4 that the segment names the rubric-providing target, which `--rubric` allows to differ from the run's own target. Carve out `judge.py` from Phase 1's script deletion. `vera judge` ships without `--resume` because the resume contract is deferred, so `judge.py` is retained solely as the resume entry point until `vera resume` exists, and is deleted the moment it does. Scoped narrowly on purpose: a legacy script kept alive for one reason otherwise becomes permanent. State that `--conversations` takes exactly one folder, preserving today's `judge.py --folder` behavior, while keeping the flag and config field list-shaped per AD-20's reasoning. Judging several folders is already expressible by judging each and combining with `vera pool`. Defer `vera judge --target all` to Phase 4, with a Done-when bar rather than a floating note. The resolution is clean today -- N separate length-1 runs -- but until the `<target>/` segment exists those runs are distinguishable only by timestamp, so the output cannot be attributed to a rubric. `--target all` keeps its full meaning for `generate`. Also record that Phase 1 judging keeps the existing `<gen_run>/evaluations/j_*` layout, renamed in Phase 3. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Settles the decisions needed before
vera judgecan be built, so thatimplementation is a mechanical mirror of #190 rather than a design exercise.
Docs only — no code. Stacked on #191 because the legacy-scripts paragraph this
extends was introduced there.
Decisions recorded
Evaluation output is named for the target, not the rubric —
evaluations/<target>/, matchingc_<chatbot>. Only the path segment changes:judging.rubricskeeps its name under AD-20, which exists to prevent exactlythis kind of schema break, and
--rubricalready takes a target name. Phase 4notes that the segment names the rubric-providing target, which
--rubricallows to differ from the run's own target.
judge.pyis carved out of Phase 1's script deletion.vera judgeshipswithout
--resumebecause the resume contract is deferred, sojudge.pyisretained solely as the resume entry point until
vera resumeexists, thendeleted. Scoped narrowly on purpose: a legacy script kept alive for one reason
otherwise becomes permanent.
--conversationstakes exactly one folder, preserving today'sjudge.py --folderbehavior, while keeping the flag and config fieldlist-shaped per AD-20's reasoning. Judging several folders is already
expressible — judge each, then combine with
vera pool.vera judge --target allis deferred to Phase 4, with a Done-when barrather than a floating note. The resolution is clean today (N separate length-1
runs); the blocker is that until the
<target>/segment exists, those runs aredistinguishable only by timestamp, so output cannot be attributed to a rubric.
--target allkeeps its full meaning forgenerate.Phase 1 judging keeps the existing
<gen_run>/evaluations/j_*layout,renamed in Phase 3.
Not changed, deliberately
judging.rubrics— renaming it is the schema break AD-20 prevents--rubric— already a target-name selector, symmetric with--personasConsequences for the implementation
--targetXOR--rubricfor judge mirrors--targetXOR--personasforgenerate exactly: same exclusivity rule, same
allsemantics, different fieldsets.
ResolvedTargetalready validates and resolves all five fields, so judgeneeds no new manifest code — and the resolution helper should be parameterized
by which fields the target supplies rather than copied per command.
With single-conversation mode dropped, resume excluded, one output rule, and
set_debugmoving to the CLI, the judge domain-prep step is no larger than #190.🤖 Generated with Claude Code