Skip to content

Docs: settle the vera judge command plan - #193

Open
luca-belli wants to merge 1 commit into
feat/vera-generatefrom
docs/judge-command-plan
Open

Docs: settle the vera judge command plan#193
luca-belli wants to merge 1 commit into
feat/vera-generatefrom
docs/judge-command-plan

Conversation

@luca-belli

Copy link
Copy Markdown
Collaborator

Summary

Settles the decisions needed before vera judge can be built, so that
implementation is a mechanical mirror of #190 rather than a design exercise.

Docs only — no code. Stacked on #191 because the legacy-scripts paragraph this
extends was introduced there.

Decisions recorded

Evaluation output is named for the target, not the rubric
evaluations/<target>/, matching c_<chatbot>. Only the path segment changes:
judging.rubrics keeps its name under AD-20, which exists to prevent exactly
this kind of schema break, and --rubric already takes a target name. Phase 4
notes that the segment names the rubric-providing target, which --rubric
allows to differ from the run's own target.

judge.py is carved out of Phase 1's script deletion. vera judge ships
without --resume because the resume contract is deferred, so judge.py is
retained solely as the resume entry point until vera resume exists, then
deleted. Scoped narrowly on purpose: a legacy script kept alive for one reason
otherwise becomes permanent.

--conversations takes exactly one folder, preserving today's
judge.py --folder behavior, while keeping the flag and config field
list-shaped per AD-20's reasoning. Judging several folders is already
expressible — judge each, then combine with vera pool.

vera judge --target all is deferred to Phase 4, with a Done-when bar
rather than a floating note. The resolution is clean today (N separate length-1
runs); the blocker is that until the <target>/ segment exists, those runs are
distinguishable only by timestamp, so output cannot be attributed to a rubric.
--target all keeps its full meaning for generate.

Phase 1 judging keeps the existing <gen_run>/evaluations/j_* layout,
renamed in Phase 3.

Not changed, deliberately

  • judging.rubrics — renaming it is the schema break AD-20 prevents
  • --rubric — already a target-name selector, symmetric with --personas
  • The spine's adopted rules, beyond the same path-segment rename for consistency

Consequences for the implementation

--target XOR --rubric for judge mirrors --target XOR --personas for
generate exactly: same exclusivity rule, same all semantics, different field
sets. ResolvedTarget already validates and resolves all five fields, so judge
needs no new manifest code — and the resolution helper should be parameterized
by which fields the target supplies rather than copied per command.

With single-conversation mode dropped, resume excluded, one output rule, and
set_debug moving to the CLI, the judge domain-prep step is no larger than #190.

🤖 Generated with Claude Code

Record the decisions needed before `vera judge` can be built, so the
implementation is a mechanical mirror of #190 rather than a design exercise.

Name the evaluation output segment for the target rather than the rubric
(`evaluations/<target>/`), matching `c_<chatbot>`. Only the path segment
changes: `judging.rubrics` keeps its name under AD-20, which exists to
prevent exactly this kind of schema break, and `--rubric` already takes a
target name. Clarify in Phase 4 that the segment names the rubric-providing
target, which `--rubric` allows to differ from the run's own target.

Carve out `judge.py` from Phase 1's script deletion. `vera judge` ships
without `--resume` because the resume contract is deferred, so `judge.py`
is retained solely as the resume entry point until `vera resume` exists,
and is deleted the moment it does. Scoped narrowly on purpose: a legacy
script kept alive for one reason otherwise becomes permanent.

State that `--conversations` takes exactly one folder, preserving today's
`judge.py --folder` behavior, while keeping the flag and config field
list-shaped per AD-20's reasoning. Judging several folders is already
expressible by judging each and combining with `vera pool`.

Defer `vera judge --target all` to Phase 4, with a Done-when bar rather
than a floating note. The resolution is clean today -- N separate length-1
runs -- but until the `<target>/` segment exists those runs are
distinguishable only by timestamp, so the output cannot be attributed to a
rubric. `--target all` keeps its full meaning for `generate`.

Also record that Phase 1 judging keeps the existing
`<gen_run>/evaluations/j_*` layout, renamed in Phase 3.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant