feat(cli): add unified vera.py CLI orchestrator wired to domain services - #184
Closed
luca-belli wants to merge 22 commits into
Closed
feat(cli): add unified vera.py CLI orchestrator wired to domain services#184luca-belli wants to merge 22 commits into
luca-belli wants to merge 22 commits into
Conversation
Isolate Human-validated Conversations
remove auto disable default thinking
RubricConfig.load_bundle() inlined manifest JSON parsing/validation, which only judge.py could reach. Per the updated Phase 0 scope in docs/architecture.md, generate.py needs to read the same manifest's personas list, so the reading/validation logic moves to utils/ (leaf layer) instead -- generate/ and judge/ must never import each other. load_bundle() now delegates to utils.rubric_manifest.load_manifest(); behavior is unchanged, covered by the existing load_bundle tests plus new direct tests for the extracted module.
load_manifest_personas() returned personas entries verbatim instead of
resolving them relative to the manifest's own folder, contradicting
docs/architecture.md's stated rule for manifest paths (the same rule
rubric_file/etc. already follow via RubricConfig.load()). Update
data/rubric_manifest.json's personas entry to the correct
manifest-relative form ("personas.tsv", not "data/personas.tsv") now
that resolution actually happens, and cover both relative and
absolute-path entries with tests.
Caught while reorganizing data/ into per-rubric subfolders on a
downstream branch -- fixing it here instead, since this PR hasn't
merged yet and is where the bug was introduced.
Same shape as load_manifest_personas(): another optional field a rubric bundle manifest can carry, read by utils/ so both generate/ and judge/ can reach it without importing each other. Unused until a later PR wires a schema-specific persona context template into generation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
data/rubric_manifest.json now points at data/persona_context_template.txt via the new persona_context_template_file field (accessor added in the prior commit), matching how the personas field was wired into this same manifest ahead of generate.py's --rubric-manifest flag landing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
judge.py --rubrics already loads a rubric bundle manifest's rubric half; generate.py had no way to load the same manifest's personas half, leaving a manifest's rubric+personas attachment unverified on the generation side (docs/architecture.md's updated Phase 0 scope). Add --rubric-manifest to generate.py: resolves the manifest's personas list via utils.rubric_manifest.load_manifest_personas() and threads it through ConversationRunner as persona_prompt_path (a new, previously-hardcoded-to-data/personas.tsv parameter) instead of the fixed default. Mirrors judge.py's own "first entry wins, warn on extras" handling for manifests listing more than one persona file. Also wires --rubric-manifest through run_pipeline.py's generation step, alongside its existing --rubrics (judging) flag.
The shared data/persona_prompt_template.txt hardcoded the SI backstory
block, so a second rubric schema (e.g. PHQ9) couldn't reuse it with its
own persona fields. Split it into a shared template (stylistic/behavioral
instructions, with a {persona_context} placeholder) and a schema-specific
context template (data/persona_context_template.txt, selected via the
manifest's persona_context_template_file field added in the prior PR).
generate_conversations/utils.py's load_prompts_from_csv() now requires a
persona_context_template_path, formats it against the TSV row first, and
validates upfront that the context template's placeholders all exist as
CSV columns (previously a missing column silently skipped the row with a
printed warning). generate.py resolves this path from --rubric-manifest
the same way it already resolves persona_prompt_path, defaulting to
data/persona_context_template.txt when no manifest is given.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Everything currently in data/ (rubric.tsv, rubric_prompt_beginning.txt, question_prompt.txt, rubric_manifest.json, personas.tsv) is specific to the SI rubric -- once a second rubric type (e.g. PHQ9) exists, data/ needs a subfolder per rubric rather than one flat, implicitly-SI set of files. Move all five into data/SI/ and update every hardcoded default/reference across generate.py, judge.py, run_pipeline.py, judge/score.py, judge/score_utils.py, scripts/, README.md, docs, and tests.
--rubrics/--rubric-manifest require a full manifest path; there's no way to select a rubric by bare name (e.g. typing "SI" anywhere on the command line) until --target lands on the future vera.py CLI. Also documents generate.py --rubric-manifest, which wasn't in the README at all.
New top-level vera.py implements the CLI surface from docs/architecture.md and docs/vera-cli-use-cases.md: generate, judge, score, pool, pipeline, and resume subcommands, sharing a single FLAG_SPECS registry so -c/-u/-j/--config etc. are defined exactly once and reused across subcommands. utils/config_schema.py centralizes the config.json shape (ModelSpec, GenerationConfig, JudgingConfig, RunConfig, RubricBundleManifest) that both CLI shorthand and --config/VERA_RUN_CONFIG resolve into, per the Phase 1 migration plan. Business-logic wiring into the existing generate/judge engines is left for a follow-up change; each subcommand validates its inputs, resolves and prints the canonical RunConfig, and stops.
…ra --target generation
luca-belli
marked this pull request as draft
August 3, 2026 04:39
luca-belli
changed the base branch from
main
to
refactor/move-rubric-data-into-si-folder
August 5, 2026 03:08
Collaborator
Author
|
Temporarily closing while correcting the stacked branch ancestry after the validation pipeline rebased it onto main. This draft will be reopened on top of #180. |
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This stacked PR adds the Phase 1 unified
vera.pycommand surface on top of #180. It wiresgenerate,judge,score,pool, andpipelineto parser-independent application functions;resumeis reserved and fails explicitly until its state/checksum contract is implemented.Architecture decisions
feat/VERA_2.0is the source of truth for the CLI architecture.--sampleis the sole CLI flag permitted alongside--config.vera.pycalls domain functions directly. It does not import or dynamically invoke the legacygenerate.pyorjudge.pyCLIs.--targetresolves persona files from a manifest for generation and retains the manifest for judging.--target; when absent, generation uses its normal default.Implementation
Validation
1003 passed, 8 deselected.--targetpath.Deferred work
The architecture requires deleting the legacy root CLIs by the end of Phase 1. This PR removes them from the new CLI dependency graph but leaves physical deletion for the remaining migration work.