From 8caff01b4fe96c2ec3af50a6fc745d4e573d806a Mon Sep 17 00:00:00 2001 From: Luca Belli <129434630+sator-labs@users.noreply.github.com> Date: Thu, 6 Aug 2026 16:41:24 -0700 Subject: [PATCH 1/6] docs: define unified CLI architecture contract --- docs/ARCHITECTURE-SPINE.md | 18 ++-- docs/architecture.md | 111 +++++++++++++++++++++++-- docs/design/vera-cli-runtime-wiring.md | 36 ++++++++ docs/vera-cli-use-cases.md | 35 ++++++-- 4 files changed, 179 insertions(+), 21 deletions(-) create mode 100644 docs/design/vera-cli-runtime-wiring.md diff --git a/docs/ARCHITECTURE-SPINE.md b/docs/ARCHITECTURE-SPINE.md index a163fe148..704ef14e1 100644 --- a/docs/ARCHITECTURE-SPINE.md +++ b/docs/ARCHITECTURE-SPINE.md @@ -7,7 +7,7 @@ paradigm: 'Layered architecture (strict top-down dependency direction) with per- scope: 'VERA-MH target architecture: pipeline CLI, generate/judge/score split, llm_clients/, workers/, storage/, utils/, and the config/naming/enforcement contract that binds them' status: final created: '2026-07-21' -updated: '2026-07-27' +updated: '2026-08-06' binds: [] sources: - docs/architecture.md @@ -29,7 +29,7 @@ Layer → package mapping: | Layer | Packages | | --- | --- | -| CLI orchestrator | `vera.py` | +| CLI | `vera.py`, `vera_cli/` | | Domain (mutually isolated) | `generate/`, `judge/`, `score/` | | Workers | `workers/` | | Infrastructure | `llm_clients/`, `storage/` | @@ -39,7 +39,8 @@ Layer → package mapping: ```mermaid graph TD - CLI["vera.py (CLI, thin)"] --> GEN["generate/"] + ENTRY["vera.py (sole executable, thin)"] --> CLI["vera_cli/"] + CLI --> GEN["generate/"] CLI --> JUDGE["judge/"] CLI --> SCORE["score/"] GEN --> WORKERS["workers/"] @@ -61,9 +62,9 @@ No edge runs `generate/` ↔ `judge/` ↔ `score/`, and none runs from `workers/ ### AD-1 — Single, thin CLI entrypoint -- **Binds:** `vera.py`, `generate/`, `judge/`, `score/` +- **Binds:** `vera.py`, `vera_cli/`, `generate/`, `judge/`, `score/` - **Prevents:** business logic creeping into the CLI entrypoint; a fat orchestrator re-implementing what domain runners already do; a second root-level entrypoint script reappearing after migration. -- **Rule:** [ADOPTED] `vera.py` is the only root-level orchestrator; domain packages are libraries, not invoked directly as scripts, and no root-level Python entry-point scripts exist alongside it. `vera.py` only parses arguments and delegates — pipeline step sequencing delegates to domain runners/handlers, never to inline logic in `vera.py` itself. +- **Rule:** [ADOPTED] `vera.py` is the only root-level executable; domain packages are libraries, not invoked directly as scripts, and no root-level Python entry-point scripts exist alongside it. `vera.py` only loads arguments and dispatches. CLI support lives in `vera_cli/`: `arguments.py` owns flag definitions, CLI defaults, and complete input resolution; `commands.py` owns thin orchestration adapters that call parser-independent domain functions directly. Domain packages never import `vera_cli/`, and neither `vera.py` nor `vera_cli/` contains domain behavior. ### AD-2 — Domain package isolation and workers inversion of control @@ -184,7 +185,7 @@ No edge runs `generate/` ↔ `judge/` ↔ `score/`, and none runs from `workers/ - **Binds:** `vera.py`, `utils/config_schema.py` - **Prevents:** a merge/override mechanism between two config sources that would make the effective config ambiguous or order-dependent. -- **Rule:** [ADOPTED] For a given run, a piece of information (model selection/repeats, sampling knobs, persona/rubric lists, etc.) is supplied via `--config` JSON or via CLI flags, never both. Supplying the same information through both is rejected/errors — there is no silent merge. `--config` always resolves internally to the same canonical flag-set the CLI would have produced, and that resolved form is printed at run start. **`--sample ` is the one deliberate, named exception:** it MAY be combined with `--config`, since it's a debug-only smoke-test override (UC4) — it never sets information `config.json` itself carries, it only caps how much of the already-resolved persona/rubric/judge lists get used for this invocation, and it's never persisted into the run's own `config.json` artifact (AD-18) or any resumed state. No other flag gets this treatment; a future flag needs its own named exception here, not an implicit ride on `--sample`'s. +- **Rule:** [ADOPTED] For a given run, a piece of information (model selection/repeats, sampling knobs, persona/rubric lists, etc.) is supplied via `--config` JSON or via CLI flags, never both. Supplying the same information through both is rejected/errors — there is no silent merge. Both forms resolve to the same canonical `RunConfig` before print, persistence, or dispatch. CLI-only presentation/execution controls such as `--debug` and `--print` do not supply run configuration and may accompany either form. **`--sample ` is the one deliberate behavior-altering exception:** it MAY be combined with `--config`, since it's a debug-only smoke-test override (UC4) — it never sets information `config.json` itself carries, it only caps how much of the already-resolved persona/rubric/judge lists get used for this invocation, and it's never persisted into the run's own `config.json` artifact (AD-18) or any resumed state. No other run-scoping flag gets this treatment; a future flag needs its own named exception here, not an implicit ride on `--sample`'s. ### AD-18 — config.json and state.json are two distinct artifacts @@ -208,7 +209,7 @@ No edge runs `generate/` ↔ `judge/` ↔ `score/`, and none runs from `workers/ - **Binds:** rubric bundle manifest files, `utils/config_schema.py`, `judge/` - **Prevents:** judge-model defaults or other per-run execution knobs leaking into the manifest, which would make the same rubric bundle behave differently across runs without a visible reason. -- **Rule:** [ADOPTED] A rubric bundle manifest describes what a rubric **is** (`rubric_file`, `rubric_prompt_beginning_file`, `question_prompt_file`, an informational `personas` list) — static content that changes rarely. `config.json`'s `judging.rubrics[].models` describes how to **run** it for a given invocation — judge models, repeats, per-rubric overrides — and changes every run. Judge-model defaults never belong in the manifest. When a config omits judge-model selection entirely, the fallback default MUST be defined in exactly one place — `utils/config_schema.py`, the protected stable interface for config shape (AD-15) — never in the manifest, and never re-defined inside `judge/rubric_config.py` (AD-22) or any other loader, which may only read the default, not define its own copy. This is a MUST, not descriptive prose: a default landing anywhere ungoverned would produce exactly the run-changing-behavior-with-no-visible-reason effect this AD exists to prevent, without escalating through AD-15's design-doc gate. **The manifest's `personas` field stays informational-only for every invocation shape except one:** `vera pipeline --target ` is an additive, opt-in shorthand that resolves `` to one rubric bundle manifest and expands to setting BOTH `generation.personas` and `judging.rubrics` from it in a single shot — the manifest's `personas` field becomes the actual, authoritative generation input only for this shorthand. Any other invocation shape (`--rubric` plus independently-specified generation personas, or `--config` with both blocks set explicitly) keeps `generation`/`judging` fully orthogonal exactly as AD-19 requires — `--target` is the one deliberate, named exception, not a general weakening of AD-19. **`--target` and `config.json` mirror each other, including the exception's boundary:** a top-level `target` field in the input `config.json` performs the identical expansion `--target ` does on the CLI. Setting `target` alongside explicit `generation.personas` or `judging.rubrics` in the same input config is rejected outright — never silently merged or overridden — mirroring AD-17's either/or rule between CLI flags and `--config`. The run's own immutable `config.json` artifact (AD-18) always stores the fully-expanded form; `target` is resolved away before that artifact is written, the same way `-u`/`-j` shorthand already resolves into concrete model entries, so `vera resume` never has to re-resolve a manifest that might have changed on disk since the run started. +- **Rule:** [ADOPTED] A rubric bundle manifest describes what a rubric **is** (`rubric_file`, `rubric_prompt_beginning_file`, `question_prompt_file`, optional `personas`, and optional `persona_context_template_file`) — static content that changes rarely. `config.json`'s `judging.rubrics[].models` describes how to **run** it for a given invocation — judge models, repeats, per-rubric overrides — and changes every run. Judge-model defaults never belong in the manifest. When a config omits judge-model selection entirely, the fallback default MUST be defined in exactly one place — `utils/config_schema.py`, the protected stable interface for config shape (AD-15) — never in the manifest, and never re-defined inside `judge/rubric_config.py` (AD-22) or any other loader, which may only read the default, not define its own copy. This is a MUST, not descriptive prose: a default landing anywhere ungoverned would produce exactly the run-changing-behavior-with-no-visible-reason effect this AD exists to prevent, without escalating through AD-15's design-doc gate. The manifest's generation fields stay informational-only unless the caller explicitly selects it with `--target` or top-level input-config `target`. Both `vera generate --target ` and `vera pipeline --target ` expand the manifest into generation personas, the persona context template, and the judging rubric in canonical `RunConfig`; generate dispatches only the generation values. The manifest's personas and context template are contextually required whenever the target is used for generation, and absence is an error with no SI fallback. Setting `target` alongside explicit `generation.personas` or `judging.rubrics` is rejected outright — never silently merged or overridden. Target expansion completes before print, persistence, or dispatch; the canonical and persisted `RunConfig` contains applicable concrete paths and no `target`, so resume never re-resolves a manifest that may have changed. Invocation shapes that independently select personas and rubrics keep generation and judging fully orthogonal exactly as AD-19 requires. ### AD-22 — Reusable rubric-loading logic lives in a library helper, not a doomed script @@ -314,6 +315,9 @@ Package tree: ```text vera.py # CLI orchestrator, thin +vera_cli/ + arguments.py # flags, CLI defaults, config/target resolution + commands.py # thin adapters to domain functions generate/ conversation_simulator.py # pure core runner.py # owns I/O, delegates to workers/ diff --git a/docs/architecture.md b/docs/architecture.md index d4bac1dcf..1253c84f1 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -51,8 +51,12 @@ Deep dives: [judge.md](./judge.md) (question flow and rubric navigation), [struc ## Layer model ```text -CLI orchestrator (vera.py) — thin, no business logic - ↓ delegates to +CLI layer +├── vera.py — sole executable; loads arguments and dispatches +└── vera_cli/ + ├── arguments.py — flags, defaults, and config resolution + └── commands.py — thin command adapters + ↓ calls Domain packages (generate/, judge/, score/) ↓ register handlers with Workers (workers/) — queue protocol, worker pool, job dispatch @@ -64,6 +68,10 @@ Shared utilities (utils/) — leaf layer **Import rules:** +- `vera.py` delegates CLI parsing, resolution, and command adaptation to + `vera_cli/`; it contains no business logic. +- `vera_cli/` may import domain packages and `utils/`. Domain packages never + import `vera_cli/`. - Domain packages (`generate/`, `judge/`, `score/`) do not import each other. - `workers/` does not import domain packages — domain registers handlers upward (inversion of control), never the reverse. - `llm_clients/` and `storage/` do not import domain packages or `workers/`. @@ -84,7 +92,39 @@ Shared utilities (utils/) — leaf layer ## CLI surface -Exactly **one** root-level orchestrator: **`vera.py`**. Subcommands parse arguments and delegate to domain runners; they contain no business logic. Full flag/config reference: [vera-cli-use-cases.md](./vera-cli-use-cases.md). +Exactly **one** root-level executable: **`vera.py`**. It loads the CLI arguments, +requests a fully resolved invocation from `vera_cli/`, dispatches the selected +command, and renders CLI errors. Full flag/config reference: +[vera-cli-use-cases.md](./vera-cli-use-cases.md). + +### CLI runtime boundary + +The CLI layer has three responsibilities: + +- `vera.py` is the thin executable and contains no command-specific business + logic. +- `vera_cli/arguments.py` defines flags and their CLI defaults, reads JSON from a + file, stdin, or `VERA_RUN_CONFIG`, enforces input exclusivity, resolves paths + and targets, and produces the complete canonical configuration before print, + persistence, or dispatch. +- `vera_cli/commands.py` contains only orchestration adapters. It receives + resolved values and calls parser-independent domain functions directly; it + does not resolve configuration or invoke legacy CLI entry points. + +`utils/config_schema.py` owns schema validation and canonical serialization. It +does not parse CLI arguments, read config or manifest files, resolve paths, or +define CLI behavior defaults. + +Domain entry points accept resolved domain values rather than `argparse` +namespaces, input config files, or rubric manifests. Generation exposes its +application function directly from its runner; it has no service wrapper and no +behavioral parameter defaults. Pooling likewise delegates to the function owned +by the scoring domain rather than to a script entry point. + +Legacy root scripts may remain temporarily while their replacement feature is +migrated, but they are compatibility adapters only. They are not architectural +dependencies and are deleted after the corresponding `vera.py` command is +available. | Subcommand | Delegates to | Purpose | |------------|--------------|---------| @@ -97,12 +137,39 @@ Exactly **one** root-level orchestrator: **`vera.py`**. Subcommands parse argume **Deferred resume contract:** `vera resume` is part of the target CLI. The constraints already adopted in `ARCHITECTURE-SPINE.md` — `state.json` as the single mutable artifact `vera resume` writes with a single-writer rule (AD-18), resume's exemption from the run-collision check (AD-24), and its path-first stage contracts (AD-23) — remain stable and are not reopened by this note. What's still unspecified is the surrounding execution machinery: complete run hierarchy, state ownership beyond the single-writer rule, task identity, retry/idempotency semantics, and partial-write recovery behavior. Those must be specified in a dedicated design document and adopted as a stable contract in a later migration phase before resume implementation is considered complete. -Config and CLI flags are strictly either/or, never combined for the same run — see [vera-cli-use-cases.md](./vera-cli-use-cases.md#config-mechanism). `-c` selects the chatbot under test; `-u`/`-j` shorthand selects models/repeats for the user/judge side respectively; bespoke sampling knobs are config-only. `--rubric` selects the rubric-bundle manifest (see [Rubric bundle manifest](#rubric-bundle-manifest) below) — always a list, though only a length-1 list is supported until Phase 4. `-c` is required for `generate`/`pipeline` whenever `--config` isn't used — there is no default chatbot. +### Input resolution + +Config and run-defining CLI flags are strictly either/or, never combined for the +same run — see [vera-cli-use-cases.md](./vera-cli-use-cases.md#config-mechanism). +Debug and presentation controls such as `--sample`, `--debug`, and `--print` are +invocation-only; they are not serialized into `RunConfig`. `-c` selects the +chatbot under test; `-u`/`-j` shorthand selects models/repeats for the user/judge +side respectively; bespoke sampling knobs are config-only. `--rubric` selects +the rubric-bundle manifest (see [Rubric bundle manifest](#rubric-bundle-manifest) +below) — always a list, though only a length-1 list is supported until Phase 4. +`-c` is required for `generate`/`pipeline` whenever `--config` isn't used — there +is no default chatbot. + +Generation behavior defaults are defined only at the CLI flag boundary. A +config-driven run provides the corresponding generation fields explicitly; it +does not inherit or merge CLI defaults. Both input forms resolve to a complete +`RunConfig`, and the generation runner receives every parameter explicitly and +defines no behavioral defaults of its own. + +Consequently, config-driven generation explicitly provides `turns`, `output`, +`max_concurrent`, `max_total_words`, `persona_speaks_first`, `sessions`, and +`persona_context_template`, using `null` where the schema allows no limit or no +session list. + +Standalone judging follows the same rule: `JudgingConfig.conversations` mirrors +`--conversations`. A standalone `judge` config must provide it; a `pipeline` +config may omit it because the generation stage supplies the resolved +conversation paths directly. ```bash uv run python vera.py pipeline --config run.json -uv run python vera.py generate -c sonnet -u gpt:1 -uv run python vera.py judge -j claude:1 --rubric data/si_rubric.json --conversations output/c_sonnet//conversations/ +uv run python vera.py generate -c sonnet -u gpt:1 --target SI +uv run python vera.py judge -j claude:1 --rubric data/SI/rubric_manifest.json --conversations output/c_sonnet//conversations/ uv run python vera.py score -r output/.../results.csv uv run python vera.py pool --evaluations path/to/evaluations/... path/to/evaluations/... uv run python vera.py resume --config output/c_sonnet//config.json @@ -119,10 +186,16 @@ A rubric is a self-describing bundle, not a bare `.tsv` path with assumed siblin "rubric_file": "rubric.tsv", "rubric_prompt_beginning_file": "rubric_prompt_beginning.txt", "question_prompt_file": "question_prompt.txt", - "personas": ["data/personas.tsv"] + "personas": ["data/personas.tsv"], + "persona_context_template_file": "persona_context_template.txt" } ``` +`personas` and `persona_context_template_file` remain optional for judge-only +manifests. Both are contextually required when the manifest is selected for +generation through `--target`; generation never substitutes SI files when either +is absent. + Paths are relative to the manifest's own folder — distinct from `config.json`, whose paths resolve relative to `$ROOT` (the directory containing `vera.py`), never to the manifest, the config file's own location, or the CLI's working directory (see [Config mechanism](./vera-cli-use-cases.md#config-mechanism)). `personas` is **informational only** — it documents which personas this rubric is intended/validated for, for humans and tooling to discover; it does not make generation consume it automatically. Generation still chooses personas independently (the `generation`/`judging` orthogonality invariant holds). This manifest shape is exactly what a `judging.rubrics[]` config entry looks like once Phase 3 formalizes the schema — the format isn't thrown away when the CLI is replaced, it's the design. **Example — two path fields, two different anchors:** @@ -145,12 +218,28 @@ Same shape of field, same-looking relative string, two different rules — hence **Separation of concerns, since these two now overlap in subject matter:** the manifest describes what a rubric **is** — its content, files, and intended personas — and changes rarely. `config.json`'s `judging.rubrics[].models` describes how to **run** it for a given invocation — which judge models, repeats, per-rubric overrides — and changes every run. Judge-model defaults belong in `config.json`, never in the manifest; the manifest never carries execution knobs. -**`--target ` shorthand:** the manifest's `personas` field stays informational-only for every invocation shape *except two, both deliberate and explicit opt-ins, never automatic*. `vera pipeline --target ` (Phase 1) resolves `` to one rubric bundle manifest and expands to setting **both** `generation.personas` and `judging.rubrics` from it in a single shot. `generate.py --rubric-manifest ` (Phase 0's stopgap, on the legacy script, ahead of `vera.py` existing) does the same thing for the personas half only, on the same manifest shape. Every other invocation (`--rubric` plus independently-specified generation personas, or `--config` with both blocks set explicitly) keeps `generation`/`judging` fully orthogonal — these are two named exceptions, not a general weakening of that invariant. +**`--target ` shorthand:** the manifest's `personas` field stays +informational-only unless the caller explicitly selects the manifest as a target. +`vera generate --target ` and `vera pipeline --target ` both expand +the manifest into generation personas, the persona context template, and the +judging rubric in the canonical `RunConfig`; `generate` dispatches only the +generation values. A top-level `target` in an input config mirrors this +expansion. Setting it alongside explicit +`generation.personas` or `judging.rubrics` is an error, never a merge or override. + +Target expansion is complete before `--print`, persistence, or command dispatch. +The canonical `RunConfig` contains concrete persona, context-template, and rubric +paths rather than `target`, so a persisted run never needs to re-resolve a +manifest that may later change. If generation selects a target whose manifest +omits personas or `persona_context_template_file`, resolution fails explicitly; +there is no fallback to SI data. Every invocation that independently specifies +personas and rubrics keeps generation and judging orthogonal. ## Package responsibilities | Package / path | Owns | Key modules | |----------------|------|-------------| +| `vera_cli/` | CLI flags/defaults, input resolution, thin command adapters | `arguments.py`, `commands.py` | | `generate/` | Simulation, turns, batch runner (pure core; handler owns I/O) | `conversation_simulator.py`, `runner.py` | | `judge/` | Rubric navigation, LLM judge, improvement reporting (pure core; handler owns I/O) | `question_navigator.py`, `llm_judge.py`, `scripts/summarize_results.py` | | `score/` | Aggregation, visualization, pooling — split out of `judge/` | `score.py`, `score_viz.py`, `pool.py` | @@ -202,8 +291,12 @@ Agents and contributors must comply. Import boundaries are documented in the [La ### MUST -- **Single CLI:** orchestration lives in `vera.py` only. +- **Single CLI:** `vera.py` is the sole executable; CLI support code lives in + `vera_cli/`, and domain behavior remains in domain packages. - **Subcommands:** `generate`, `judge`, `score`, `pool`, `pipeline`, `resume` (add or remove only via [ESCALATE](#escalate-stop-and-ask)). +- **Resolved boundary:** flags, config, paths, and targets resolve to canonical + values before print, persistence, or dispatch. Domain functions never parse + CLI/config inputs or define CLI behavior defaults. - **Generation:** conversation simulation logic stays in `generate/`; the simulator core is pure (no filesystem, no logging) — the handler owns all I/O. - **Judging:** rubric navigation and LLM-judge logic stay in `judge/`, also pure-core-plus-handler. Judge never auto-scores — `vera score`/`vera pool` are separate subcommands. **Rubric navigation logic lives in code, never in the prompt:** which question is asked next given an answer is determined entirely by `QuestionNavigator` walking `question_flow_data` parsed from the rubric TSV — the judge LLM answers/judges the current question only, and is never asked to decide or influence what comes next. - **Scoring:** aggregation, visualization, and pooling stay in `score/`, never re-absorbed into `judge/`. diff --git a/docs/design/vera-cli-runtime-wiring.md b/docs/design/vera-cli-runtime-wiring.md new file mode 100644 index 000000000..635e18948 --- /dev/null +++ b/docs/design/vera-cli-runtime-wiring.md @@ -0,0 +1,36 @@ +# VERA CLI runtime wiring decision record + +Status: Accepted +Date: 2026-08-06 + +## Context + +Wiring the unified CLI requires changes to `utils/config_schema.py`, a stable +interface. Standalone judging also exposed an ambiguity: it needs conversation +paths, while AD-17 forbids combining JSON config with run-defining CLI flags. + +## Decision + +The unified CLI runtime and configuration contracts are canonical in: + +- [Architecture: CLI runtime boundary](../architecture.md#cli-runtime-boundary) +- [Architecture: input resolution](../architecture.md#input-resolution) +- [Architecture: rubric bundle manifest](../architecture.md#rubric-bundle-manifest) +- [CLI/config use cases](../vera-cli-use-cases.md) + +This record captures the rationale for the corresponding stable-interface +changes without duplicating those contracts here. The implementation resolves +every input into canonical values before dispatch and calls parser-independent +domain functions. + +## Consequences + +- Existing pipeline configs may omit `judging.conversations`, because generation + supplies those paths. Standalone judge configs must now include it and may no + longer combine `--config` with `--conversations`. +- Generation configs that relied on Python or CLI defaults must add the required + generation behavior fields explicitly. CLI invocations retain defaults at the + flag-definition boundary. +- Legacy root scripts remain usable only during their feature-by-feature + replacement. They are not dependencies of the unified CLI and are removed in a + later cleanup PR. diff --git a/docs/vera-cli-use-cases.md b/docs/vera-cli-use-cases.md index da9dd133c..e96a0725d 100644 --- a/docs/vera-cli-use-cases.md +++ b/docs/vera-cli-use-cases.md @@ -24,8 +24,8 @@ CLI shorthand and the input `config.json` are deliberately mirrored, flag-for-fi | Subcommand | CLI shorthand — minimum required | Input `config.json` — minimum required fields | |---|---|---| | `generate` | `-c ` + `-u ` (≥1) + (`--personas ` **or** `--target `) | `generation.chatbot`, `generation.user` (≥1), and (`generation.personas` (≥1) **or** top-level `target`) | -| `judge` | `-j ` (≥1) + `--conversations ` + `--rubric ` | `judging.models` (≥1), `judging.rubrics` (≥1) — plus whatever conversations path the run is scoped to | -| `pipeline` | everything `generate` needs **and** everything `judge` needs — or just `-c`, `-u`, `-j`, `--target ` (`--target` covers persona+rubric in one shot, pipeline-only) | both `generation` and `judging` blocks fully populated (their individual minimums above), or `-c`/`-u`/`-j`-equivalent fields plus a top-level `target` | +| `judge` | `-j ` (≥1) + `--conversations ` + `--rubric ` | `judging.models` (≥1), `judging.rubrics` (≥1), and `judging.conversations` (≥1) | +| `pipeline` | everything `generate` needs **and** everything `judge` needs — or just `-c`, `-u`, `-j`, `--target ` (`--target` covers persona+rubric in one shot) | both `generation` and `judging` blocks fully populated, or `-c`/`-u`/`-j`-equivalent fields plus a top-level `target`; `judging.conversations` is omitted because generation supplies it | | `score` | the results path (`-r `) — nothing else | n/a — `score` reads an existing `results.csv`, not a run config | | `pool` | `--evaluations ` (≥1) | n/a | | `resume` | `--config ` — that's the entire invocation | n/a (it *is* the config being resumed) | @@ -63,6 +63,13 @@ vera generate -c sonnet -u gpt:1 sonnet:2 --personas data/personas.tsv ``` Bare-minimum required flags for Option B: `-c ` (the chatbot under test — no default), `-u ` (at least one user-side model), and either `--personas ` or `--target ` (personas have no silently-assumed default file either). `--rubric`/`--target` are additionally required if this invocation will also be judged. +Generation behavior is also controlled at this input boundary. CLI invocations +default to `--turns 3`, `--output output`, unlimited concurrency, no total-word +cap, persona-first ordering, one unnamed session, and +`--persona-context-template data/SI/persona_context_template.txt` when explicit +persona files are selected. `--target` obtains the context template from its +manifest instead. The generation runner itself has no parameter defaults. + Or, using `--target` to pull personas from a rubric bundle manifest instead of naming them explicitly: ``` vera generate -c sonnet -u gpt:1 --target SI @@ -79,8 +86,8 @@ Personas come from one or more persona files, each containing multiple personas; Judge one **or more** existing transcript folders against one or more rubrics. Each rubric has a default judge-LLM set, overridable per rubric. Multiple judges per rubric are supported (for judge-agreement analysis). Judging is decoupled from generation — point it at whatever conversation folders you need, with no enforced coupling to originating personas. ``` -vera judge --conversations output/c_sonnet//conversations/ --config run.json -vera judge -j claude:1 gpt:2 --conversations output/c_sonnet//conversations/ --rubric data/si_rubric.json +vera judge --config run.json +vera judge -j claude:1 gpt:2 --conversations output/c_sonnet//conversations/ --rubric data/SI/rubric_manifest.json ``` No `-c` here: judging is decoupled from chatbot selection by design (see the orthogonality invariant above) — the chatbot is already implicit in whichever `--conversations` folder is passed in. @@ -147,9 +154,17 @@ Top-level `generation` and `judging` blocks are **completely orthogonal** — mo {"name": "claude-sonnet-2026xxxx", "repeats": 1, "temperature": 0.7}, {"name": "gpt-5", "repeats": 2} ], - "personas": ["data/personas_a.json", "data/personas_b.json"] + "personas": ["data/personas_a.json", "data/personas_b.json"], + "turns": 3, + "output": "output", + "max_concurrent": null, + "max_total_words": null, + "persona_speaks_first": true, + "sessions": null, + "persona_context_template": "data/SI/persona_context_template.txt" }, "judging": { + "conversations": ["output/c_sonnet/example/conversations"], "models": [ {"name": "claude-sonnet-2026xxxx", "repeats": 1} ], @@ -161,6 +176,16 @@ Top-level `generation` and `judging` blocks are **completely orthogonal** — mo } ``` +`judging.conversations` is required for standalone `vera judge` configs. A +`pipeline` config omits it because the generation stage supplies the resolved +conversation paths without combining config input with a CLI flag. + +These generation behavior fields are required in config mode, including +explicit `null` where no limit or session list is intended. Config mode never +inherits the CLI defaults. When top-level `target` supplies the persona bundle, +`persona_context_template` is `null` and the selected manifest must define +`persona_context_template_file`. + `generation.chatbot` is the chatbot under test — same shape as one entry in `generation.user`, but a single object, not a list (only one chatbot per run; see use case 1). It is distinct from `generation.user`, which is the user-side (`u`) LLM list — the two share a field shape but are never conflated: naming one `chatbot` and the other `user` (rather than both `models`) makes which is which unambiguous at the field-name level, not just from prose. Each list entry's `name` is always a **specific model identifier** (e.g. `claude-sonnet-2026xxxx`), using the provider's own naming — never a bare provider name like `"openai"`. Bespoke sampling knobs (temperature, top_p, max_tokens) are config-only, never expressible via `-u`/`-j` shorthand; a model named only via the shorthand gets the provider's environment-sourced defaults. Provider connection details (endpoint, API version, region) stay env-sourced only, never overridable here. From 29d861131608355283cededfcf811c4c97b8b586 Mon Sep 17 00:00:00 2001 From: Luca Belli <129434630+sator-labs@users.noreply.github.com> Date: Fri, 7 Aug 2026 09:04:08 -0700 Subject: [PATCH 2/6] docs: define manifests as complete targets --- docs/ARCHITECTURE-SPINE.md | 32 +++---- docs/architecture.md | 111 ++++++++++++++++--------- docs/design/vera-cli-runtime-wiring.md | 5 +- docs/vera-cli-use-cases.md | 81 +++++++++++++----- 4 files changed, 154 insertions(+), 75 deletions(-) diff --git a/docs/ARCHITECTURE-SPINE.md b/docs/ARCHITECTURE-SPINE.md index 704ef14e1..2cff3644c 100644 --- a/docs/ARCHITECTURE-SPINE.md +++ b/docs/ARCHITECTURE-SPINE.md @@ -7,7 +7,7 @@ paradigm: 'Layered architecture (strict top-down dependency direction) with per- scope: 'VERA-MH target architecture: pipeline CLI, generate/judge/score split, llm_clients/, workers/, storage/, utils/, and the config/naming/enforcement contract that binds them' status: final created: '2026-07-21' -updated: '2026-08-06' +updated: '2026-08-07' binds: [] sources: - docs/architecture.md @@ -205,17 +205,17 @@ No edge runs `generate/` ↔ `judge/` ↔ `score/`, and none runs from `workers/ - **Rule:** [ADOPTED] `judging.rubrics` (and its CLI precursor, `--rubrics`) is list-shaped starting at the very first phase that implements it, even while only a length-1 list is supported/validated until multi-rubric support ships. This closes a real schema-break risk: locking a scalar shape early would force a breaking config change later. - **Prevents:** a later multi-rubric phase requiring a breaking schema change for every config already written against a scalar `rubric` field. -### AD-21 — Rubric bundle manifest vs config.json separation of concerns +### AD-21 — Target manifest vs config.json separation of concerns -- **Binds:** rubric bundle manifest files, `utils/config_schema.py`, `judge/` -- **Prevents:** judge-model defaults or other per-run execution knobs leaking into the manifest, which would make the same rubric bundle behave differently across runs without a visible reason. -- **Rule:** [ADOPTED] A rubric bundle manifest describes what a rubric **is** (`rubric_file`, `rubric_prompt_beginning_file`, `question_prompt_file`, optional `personas`, and optional `persona_context_template_file`) — static content that changes rarely. `config.json`'s `judging.rubrics[].models` describes how to **run** it for a given invocation — judge models, repeats, per-rubric overrides — and changes every run. Judge-model defaults never belong in the manifest. When a config omits judge-model selection entirely, the fallback default MUST be defined in exactly one place — `utils/config_schema.py`, the protected stable interface for config shape (AD-15) — never in the manifest, and never re-defined inside `judge/rubric_config.py` (AD-22) or any other loader, which may only read the default, not define its own copy. This is a MUST, not descriptive prose: a default landing anywhere ungoverned would produce exactly the run-changing-behavior-with-no-visible-reason effect this AD exists to prevent, without escalating through AD-15's design-doc gate. The manifest's generation fields stay informational-only unless the caller explicitly selects it with `--target` or top-level input-config `target`. Both `vera generate --target ` and `vera pipeline --target ` expand the manifest into generation personas, the persona context template, and the judging rubric in canonical `RunConfig`; generate dispatches only the generation values. The manifest's personas and context template are contextually required whenever the target is used for generation, and absence is an error with no SI fallback. Setting `target` alongside explicit `generation.personas` or `judging.rubrics` is rejected outright — never silently merged or overridden. Target expansion completes before print, persistence, or dispatch; the canonical and persisted `RunConfig` contains applicable concrete paths and no `target`, so resume never re-resolves a manifest that may have changed. Invocation shapes that independently select personas and rubrics keep generation and judging fully orthogonal exactly as AD-19 requires. +- **Binds:** target manifests, `vera_cli/`, `utils/config_schema.py`, `generate/`, `judge/` +- **Prevents:** partial manifests that work for one command but fail for another; per-run execution knobs leaking into reusable target definitions; selecting a target when the caller intended to control generation and judging independently. +- **Rule:** [ADOPTED] A target is a complete evaluation bundle defined by `data//manifest.json`. Every manifest requires `rubric_file`, `rubric_prompt_beginning_file`, `question_prompt_file`, `personas`, and `persona_context_template_file`; paths resolve relative to the manifest. `--target ` and top-level config `target` consume the complete bundle. `--personas` plus `--persona-context-template` and `--rubric ` remain explicit alternatives that consume only the selected generation or judging components, and a pipeline may use both explicit forms instead of `--target`. Target selection is mutually exclusive with explicit persona/rubric selection. `--target all` produces one canonical invocation per discovered target and never merges different targets' personas, prompts, or rubrics. Target expansion completes before print, persistence, or dispatch; canonical and persisted `RunConfig` contains concrete paths and no `target`. Model defaults and all other run behavior belong in CLI flag definitions or `config.json`, never in the manifest. -### AD-22 — Reusable rubric-loading logic lives in a library helper, not a doomed script +### AD-22 — Target-manifest resolution belongs to the CLI boundary -- **Binds:** `judge/` -- **Prevents:** logic that later CLI phases need being thrown away with the CLI script it was first written inside. -- **Rule:** [ADOPTED] The rubric-bundle-manifest loading logic is implemented as a library-layer helper (e.g. `judge/rubric_config.py`), not inline in a CLI script's `main()` that is slated for deletion. Later phases call the same helper rather than reimplementing rubric loading. +- **Binds:** `vera_cli/arguments.py`, `generate/`, `judge/` +- **Prevents:** either domain owning a cross-domain target definition; generation and judging reimplementing manifest resolution differently; legacy script parsers becoming architectural dependencies. +- **Rule:** [ADOPTED] `vera_cli/arguments.py` loads and validates target manifests and expands them to concrete generation and judging inputs. Domain functions receive only their resolved values and never a target manifest. Explicit `--rubric` consumes the rubric and judging-prompt portion of a complete target manifest; explicit `--personas` consumes caller-provided persona files and its separately resolved persona-context template. Legacy helpers may adapt these inputs during migration but are not dependencies of the unified CLI. ### AD-23 — Output layout is nested-only, with path-first stage contracts @@ -231,13 +231,13 @@ No edge runs `generate/` ↔ `judge/` ↔ `score/`, and none runs from `workers/ ### AD-25 — Rubric/persona content lives outside code proper, never requiring a code change -- **Binds:** `data/`, the rubric bundle manifest (AD-21), persona files, all packages that consume them +- **Binds:** `data/`, target manifests (AD-21), persona and rubric files, all packages that consume them - **Prevents:** an engineer embedding rubric dimensions, question flows, or persona definitions directly in Python (e.g. as constants, dataclass defaults, or inline dicts) rather than as data files — which would silently reintroduce a code-change requirement for a non-developer-facing workflow, and would fragment "where does rubric/persona content live" across code and data depending on who built which part. - **Rule:** [ADOPTED] Rubric and persona content (dimensions, question flows, prompt text, persona definitions) MUST live in `data/` (or another location outside any Python package), never embedded in code, because VERA-MH must be usable by non-developers who add or edit rubrics and personas without touching Python. `data/` is a supporting path outside the import graph specifically for this reason — no domain package may hardcode rubric/persona content as an alternative to reading it from `data/`. ### AD-26 — Rubric navigation logic lives in code, never in the prompt -- **Binds:** `judge/`, rubric bundle manifests (AD-21), `data/` (AD-25) +- **Binds:** `judge/`, target manifests (AD-21), `data/` (AD-25) - **Prevents:** an engineer embedding flow-control hints ("if the answer is X, the next relevant topic is Y") inside prompt text and asking the LLM to decide what happens next — non-deterministic, untestable, and a divergence risk between two engineers who might otherwise put navigation logic in different places (one in code, one in a prompt) for the same rubric. - **Rule:** [ADOPTED] Which question is asked next given an answer (the rubric's `GOTO`/`END`/`ASSIGN_END`/conditional-jump directives, per `judge.md`) is determined entirely by code — the rubric's question-flow data (parsed from its TSV into `question_flow_data`) navigated deterministically by `QuestionNavigator`. The judge LLM's role is strictly to answer/judge the current question; it is never asked to decide or influence which question comes next. Rubric content in `data/` (AD-25) may describe *what* the flow is (the TSV's navigation column), but the *logic that walks it* is code, not something inferred from prompt text. @@ -247,11 +247,11 @@ No edge runs `generate/` ↔ `judge/` ↔ `score/`, and none runs from `workers/ - **Prevents:** the chatbot under test being inferred from context or defaulted silently; `generation.user` (the user/persona-side LLM list) being mistaken for chatbot selection, or vice versa, since both are LLM-selection fields in the same `generation` block; a bare `models` field leaving it ambiguous which of the two roles it names. - **Rule:** [ADOPTED] The chatbot under test is selected via `-c [:repeats]` on the CLI or `generation.chatbot` in `config.json` — a single object, not a list (AD-19's single-chatbot-per-invocation scope), and required whenever `--config` is not used. There is no default chatbot. `generation.chatbot` is distinct from `generation.user`, which selects the user-side (`u`) LLM(s); the two are never conflated or merged into one field, and neither is named the generic `models` precisely because `generation` has two competing LLM roles — each gets its own entity-specific field name instead. `vera judge` never takes `-c` — judging is decoupled from chatbot selection by design (AD-5), and the chatbot is already implicit in whichever `--conversations` folder is passed in. -### AD-28 — Config path fields resolve to $ROOT; manifest path fields resolve to themselves +### AD-28 — Config path fields resolve to $ROOT; target-manifest paths resolve to themselves -- **Binds:** `utils/config_schema.py`, rubric bundle manifests (AD-21) -- **Prevents:** a config's meaning depending on the working directory the CLI happened to be invoked from, or on where the config file itself was saved — the exact ambiguity that made an earlier CWD-relative design fragile; a reader assuming both JSON files (`config.json` and the rubric bundle manifest) share one path-resolution rule just because they look structurally similar. -- **Rule:** [ADOPTED] Path fields inside `config.json` (e.g. `generation.personas`) resolve relative to `$ROOT` — the directory containing `vera.py` — regardless of the CLI's working directory, the config file's own location, or how the config arrives (`--config `, `--config -`, or `VERA_RUN_CONFIG`); this is a single rule across all three input forms, not a rule-plus-fallback. Rubric bundle manifest path fields resolve relative to the manifest's own folder instead (AD-21), a deliberately different anchor that keeps a manifest folder portable across checkouts — `config.json` doesn't need that property, since it is checkout-specific by nature. +- **Binds:** `utils/config_schema.py`, target manifests (AD-21) +- **Prevents:** a config's meaning depending on the working directory the CLI happened to be invoked from, or on where the config file itself was saved; a reader assuming `config.json` and a target manifest share one path-resolution rule because both are JSON. +- **Rule:** [ADOPTED] Path fields inside `config.json` (e.g. `generation.personas`) resolve relative to `$ROOT` — the directory containing `vera.py` — regardless of the CLI's working directory, the config file's own location, or how the config arrives (`--config `, `--config -`, or `VERA_RUN_CONFIG`). Target-manifest path fields resolve relative to the manifest's own folder instead (AD-21), keeping the complete target directory portable across checkouts. ## Consistency Conventions @@ -325,7 +325,7 @@ judge/ question_navigator.py # pure core llm_judge.py # pure core runner.py # owns I/O, delegates to workers/ - rubric_config.py # reusable rubric-bundle-manifest loader (AD-22) + rubric_config.py # rubric-domain configuration after CLI resolution score/ score.py score_viz.py diff --git a/docs/architecture.md b/docs/architecture.md index 1253c84f1..a60a172d9 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -22,7 +22,7 @@ They never import each other. A full workflow runs generation, judging, score, a All user-facing operations go through **`vera.py`** subcommands. Domain packages are libraries; they are not invoked directly as scripts. ```text -data/personas.tsv ──► generate ──► c_//conversations/*.json +data//manifest.json ──► generate ──► c_//conversations/*.json │ ▼ judge ──► c_//evaluations//j_*/results.csv @@ -38,6 +38,7 @@ data/personas.tsv ──► generate ──► c_//conversations/* | Concept | Location | Notes | |---------|----------|-------| +| Target | `data//manifest.json` | Complete evaluation bundle: rubric, personas, and the prompts needed by generation and judging. | | Persona | one or more persona files | Simulated user; drives the user-side (`u`) LLM. Duplicate persona names across files are possible — disambiguated by file + name. | | Chatbot | `-c`/`generation.chatbot` | Provider/agent LLM under test (the `c` in `c_/`); selected the same way as the user-side (`u`) and judge-side (`j`) models, never inferred from context | | Transcript | `c_//conversations/*.json` | Turn-by-turn chat log; filename encodes persona file + name + chatbot model | @@ -116,7 +117,7 @@ does not parse CLI arguments, read config or manifest files, resolve paths, or define CLI behavior defaults. Domain entry points accept resolved domain values rather than `argparse` -namespaces, input config files, or rubric manifests. Generation exposes its +namespaces, input config files, or target manifests. Generation exposes its application function directly from its runner; it has no service wrapper and no behavioral parameter defaults. Pooling likewise delegates to the function owned by the scoring domain rather than to a script entry point. @@ -144,11 +145,12 @@ same run — see [vera-cli-use-cases.md](./vera-cli-use-cases.md#config-mechanis Debug and presentation controls such as `--sample`, `--debug`, and `--print` are invocation-only; they are not serialized into `RunConfig`. `-c` selects the chatbot under test; `-u`/`-j` shorthand selects models/repeats for the user/judge -side respectively; bespoke sampling knobs are config-only. `--rubric` selects -the rubric-bundle manifest (see [Rubric bundle manifest](#rubric-bundle-manifest) -below) — always a list, though only a length-1 list is supported until Phase 4. -`-c` is required for `generate`/`pipeline` whenever `--config` isn't used — there -is no default chatbot. +side respectively; bespoke sampling knobs are config-only. `--target` selects a +complete target, while `--personas` and `--rubric` explicitly select only the +generation or judging inputs from their respective sources. `--rubric` remains +list-shaped, though only a length-1 list is supported until Phase 4. `-c` is +required for `generate`/`pipeline` whenever `--config` isn't used — there is no +default chatbot. Generation behavior defaults are defined only at the CLI flag boundary. A config-driven run provides the corresponding generation fields explicitly; it @@ -169,7 +171,8 @@ conversation paths directly. ```bash uv run python vera.py pipeline --config run.json uv run python vera.py generate -c sonnet -u gpt:1 --target SI -uv run python vera.py judge -j claude:1 --rubric data/SI/rubric_manifest.json --conversations output/c_sonnet//conversations/ +uv run python vera.py generate -c sonnet -u gpt:1 --personas data/custom/personas.tsv --persona-context-template data/custom/persona_prompt.txt +uv run python vera.py judge -j claude:1 --rubric data/SI/manifest.json --conversations output/c_sonnet//conversations/ uv run python vera.py score -r output/.../results.csv uv run python vera.py pool --evaluations path/to/evaluations/... path/to/evaluations/... uv run python vera.py resume --config output/c_sonnet//config.json @@ -177,63 +180,90 @@ uv run python vera.py resume --config output/c_sonnet//config.json `vera judge` never takes `-c`: judging is decoupled from chatbot selection by design (the `generation`/`judging` orthogonality invariant, below) — the chatbot is already implicit in whichever `--conversations` folder is passed in. -### Rubric bundle manifest +### Target manifest -A rubric is a self-describing bundle, not a bare `.tsv` path with assumed sibling filenames. `--rubric`/`judging.rubrics[]` entries point at a manifest file: +A **target** is the complete, reusable combination of a rubric, personas, and +the prompts required to generate and judge conversations. Every target is +defined by a file named `manifest.json`; “manifest” is the representation and +“target” is the domain concept. + +The manifest is complete rather than tailored to one command: ```json { "rubric_file": "rubric.tsv", "rubric_prompt_beginning_file": "rubric_prompt_beginning.txt", "question_prompt_file": "question_prompt.txt", - "personas": ["data/personas.tsv"], + "personas": ["personas.tsv"], "persona_context_template_file": "persona_context_template.txt" } ``` -`personas` and `persona_context_template_file` remain optional for judge-only -manifests. Both are contextually required when the manifest is selected for -generation through `--target`; generation never substitutes SI files when either -is absent. +All five fields are required. A command may consume only part of the target, but +that does not make the remaining fields optional: `generate` uses the personas +and persona prompt, while `judge` uses the rubric and judging prompts. -Paths are relative to the manifest's own folder — distinct from `config.json`, whose paths resolve relative to `$ROOT` (the directory containing `vera.py`), never to the manifest, the config file's own location, or the CLI's working directory (see [Config mechanism](./vera-cli-use-cases.md#config-mechanism)). `personas` is **informational only** — it documents which personas this rubric is intended/validated for, for humans and tooling to discover; it does not make generation consume it automatically. Generation still chooses personas independently (the `generation`/`judging` orthogonality invariant holds). This manifest shape is exactly what a `judging.rubrics[]` config entry looks like once Phase 3 formalizes the schema — the format isn't thrown away when the CLI is replaced, it's the design. +Paths are relative to the manifest's own folder — distinct from `config.json`, +whose paths resolve relative to `$ROOT` (the directory containing `vera.py`), +never to the manifest, the config file's own location, or the CLI's working +directory (see [Config mechanism](./vera-cli-use-cases.md#config-mechanism)). **Example — two path fields, two different anchors:** ```text -project-root/ ← $ROOT (contains vera.py) +project-root/ ← $ROOT (contains vera.py) ├── vera.py ├── configs/ -│ └── run.json ← config.json +│ └── run.json ← config.json └── data/ ├── personas_a.json - └── si_rubric_bundle.json ← manifest - (personas: ["personas/si_personas.tsv"]) + └── SI/ + ├── manifest.json ← target manifest + ├── personas.tsv + ├── persona_context_template.txt + ├── rubric.tsv + ├── rubric_prompt_beginning.txt + └── question_prompt.txt ``` - `configs/run.json`'s `generation.personas: ["data/personas_a.json"]` always means `project-root/data/personas_a.json` — resolved against `$ROOT`, no matter where you run `vera.py` from or where `run.json` itself lives. -- `data/si_rubric_bundle.json`'s `personas: ["personas/si_personas.tsv"]` always means `data/personas/si_personas.tsv` — resolved against the manifest's own folder (`data/`), so the whole `data/` folder stays portable if copied into a different checkout, independent of `$ROOT`. +- `data/SI/manifest.json`'s `personas: ["personas.tsv"]` always means `data/SI/personas.tsv` — resolved against the manifest's own folder, so the whole target directory stays portable if copied into a different checkout, independent of `$ROOT`. Same shape of field, same-looking relative string, two different rules — hence calling both out explicitly here rather than leaving it implicit. -**Separation of concerns, since these two now overlap in subject matter:** the manifest describes what a rubric **is** — its content, files, and intended personas — and changes rarely. `config.json`'s `judging.rubrics[].models` describes how to **run** it for a given invocation — which judge models, repeats, per-rubric overrides — and changes every run. Judge-model defaults belong in `config.json`, never in the manifest; the manifest never carries execution knobs. +**Separation of concerns:** the target manifest describes the static evaluation +bundle and changes rarely. `config.json` describes how to run it — models, +repeats, and per-run overrides — and changes every run. Model defaults and other +execution knobs belong in CLI flag definitions or `config.json`, never in the +manifest. + +**Whole-target and explicit-component selection:** `--target ` +loads the whole manifest. `vera generate --target ...` consumes its persona and +persona-prompt fields; `vera judge --target ...` consumes its rubric and +judging-prompt fields; `vera pipeline --target ...` consumes both. + +Advanced callers may keep generation and judging independent. `--personas` +selects persona files explicitly (with `--persona-context-template` controlling +their prompt), and `--rubric ` explicitly selects only the +rubric and judging-prompt portion of a target manifest. A pipeline may use +`--personas` and `--rubric` together instead of `--target`. These explicit forms +do not pull in the target's other components. + +A top-level `target` in an input config mirrors whole-target selection. Setting +it alongside explicit `generation.personas` or `judging.rubrics` is an error, +never a merge or override. -**`--target ` shorthand:** the manifest's `personas` field stays -informational-only unless the caller explicitly selects the manifest as a target. -`vera generate --target ` and `vera pipeline --target ` both expand -the manifest into generation personas, the persona context template, and the -judging rubric in the canonical `RunConfig`; `generate` dispatches only the -generation values. A top-level `target` in an input config mirrors this -expansion. Setting it alongside explicit -`generation.personas` or `judging.rubrics` is an error, never a merge or override. +`--target all` enumerates targets; it does not merge their personas, prompts, or +rubrics. The resolver produces one complete canonical invocation per target so +each target retains its own persona and judging prompts. Target expansion is complete before `--print`, persistence, or command dispatch. -The canonical `RunConfig` contains concrete persona, context-template, and rubric -paths rather than `target`, so a persisted run never needs to re-resolve a -manifest that may later change. If generation selects a target whose manifest -omits personas or `persona_context_template_file`, resolution fails explicitly; -there is no fallback to SI data. Every invocation that independently specifies -personas and rubrics keeps generation and judging orthogonal. +The canonical `RunConfig` contains concrete persona, context-template, rubric, +and judging-prompt paths rather than `target`, so a persisted run never needs to +re-resolve a manifest that may later change. An incomplete manifest is not a +valid target, and there is no fallback to SI data. Every invocation that +explicitly specifies personas and rubrics keeps generation and judging +orthogonal. ## Package responsibilities @@ -297,6 +327,9 @@ Agents and contributors must comply. Import boundaries are documented in the [La - **Resolved boundary:** flags, config, paths, and targets resolve to canonical values before print, persistence, or dispatch. Domain functions never parse CLI/config inputs or define CLI behavior defaults. +- **Targets:** every `manifest.json` defines one complete target (rubric, + personas, and prompts). `--target` selects the complete bundle; + `--personas` and `--rubric` remain explicit component-level alternatives. - **Generation:** conversation simulation logic stays in `generate/`; the simulator core is pure (no filesystem, no logging) — the handler owns all I/O. - **Judging:** rubric navigation and LLM-judge logic stay in `judge/`, also pure-core-plus-handler. Judge never auto-scores — `vera score`/`vera pool` are separate subcommands. **Rubric navigation logic lives in code, never in the prompt:** which question is asked next given an answer is determined entirely by `QuestionNavigator` walking `question_flow_data` parsed from the rubric TSV — the judge LLM answers/judges the current question only, and is never asked to decide or influence what comes next. - **Scoring:** aggregation, visualization, and pooling stay in `score/`, never re-absorbed into `judge/`. @@ -389,15 +422,15 @@ uv run pytest -m "not live" | Phase | Goal | Key changes | Done when | |-------|------|--------------|-----------| -| **0 — De-risk multi-rubric** | Prove out the rubric-bundle-manifest design cheaply on **both** sides of the pipeline — generation and judging — with code that survives into Phase 1 rather than being thrown away. A rubric bundle manifest is meant to attach a rubric *and* the personas it's validated for as one unit; proving out only the judging half here would leave that attachment unverified until Phase 1 | Add a small, reusable helper (e.g. `judge/rubric_config.py::RubricConfig.load_bundle()` or similar — a library function, not inline in `judge.py`'s `main()`) that reads a **rubric bundle manifest** (see [Rubric bundle manifest](#rubric-bundle-manifest)) and loads the `RubricConfig` it describes. Wire `judge.py`'s existing `--rubrics` flag (`nargs="+"`, already list-shaped) to call this helper with `args.rubrics[0]`; passing more than one manifest prints a warning and uses only the first (list-from-day-one, length-1-for-now — the first step, not the final limitation). **Generation side, same phase:** `generate.py` has no persona-file selection flag today — personas load from a fixed internal default regardless of what's being judged, which is exactly the gap that leaves a manifest's rubric and its intended personas unattached in practice. Add a `--rubric-manifest ` flag to `generate.py` that reads the same manifest (via the same helper, just pulling `personas` instead of the rubric fields) and uses it as the persona file list for that run — `generate.py`'s first way to select personas by file at all, and the stopgap, single-script version of what `--target` later does through `vera.py`. This is opt-in (omitting the flag keeps today's fixed-default behavior), so `generation`/`judging` orthogonality still holds for every invocation that doesn't ask for it. Because the helper lives in the library layer both `judge.runner` and `generate.py` already delegate to, Phase 1's `vera judge`/`vera generate --target` reuse it directly instead of reimplementing manifest loading from scratch. No score split, no naming scheme, no `config.json`, no stable interfaces touched | `pytest -m "not live"` green; `--rubrics ` actually loads that rubric bundle instead of the hardcoded `data/` one; `generate.py --rubric-manifest ` actually loads that manifest's `personas` list instead of the fixed internal default; a fixture manifest (with a non-empty `personas` list) exists in `tests/fixtures/` to make both testable | +| **0 — De-risk multi-rubric** | Prove the target-manifest format on both generation and judging before the unified CLI replaces legacy scripts | Treat every manifest as one complete target containing a rubric, personas, and both prompt sets. Legacy judge code may consume only its rubric fields and legacy generation code only its persona fields, but both read the same complete manifest. Preserve the existing explicit persona/rubric paths so Phase 1 can offer both whole-target and component-level selection | `pytest -m "not live"` green; one complete fixture target drives both legacy generation and judging; incomplete manifests fail validation; explicit persona/rubric selection remains covered | | **S — Storage abstraction** *(orthogonal — see note above)* | Decouple "what path/key to use" from "how to persist bytes," so a future non-local backend (S3, etc.) is a new implementation, not a rewrite | New `storage/` package: `StorageBackend` (ABC) with `write(key, bytes)` / `read(key)` / `exists(key)`, plus `LocalFilesystemStorage` as the default implementation — mirrors the `LLMInterface`/`QueueProtocol` idiom (interface and implementations live together in one concern-scoped package). The backend knows nothing about run semantics; `utils/naming.py` still builds keys/paths, `storage/` just persists what it's given. All domain/`workers/` code that currently touches the filesystem directly switches to calling through `StorageBackend` instead | `pytest -m "not live"` green; no domain or `workers/` code calls `open()`/`pathlib` file-write directly for run artifacts — everything routes through `StorageBackend`; `LocalFilesystemStorage` is behaviorally identical to today's direct-filesystem writes | | **O — Adopt OpenSpec** *(orthogonal — see note above)* | Turn "consider an OpenSpec change if the team adopts that workflow" from a maybe into an actual, exercised requirement | Populate `openspec/changes/` with a real OpenSpec change document the next time a large multi-file feature lands (the existing ESCALATE trigger: new judge dimensions, pipeline CLI changes). Currently `openspec/` is empty scaffolding — this phase is "done" only once a real change has actually gone through it, not just once the config exists | A qualifying multi-file feature has shipped with a real OpenSpec change document under `openspec/changes/`, and the ESCALATE section's language is updated from "if the team adopts" to a firm MUST for future qualifying changes | -| **1 — New CLI + config** | `vera.py` fully replaces the top-level scripts | Not cosmetic: `vera.py` subcommands (`generate`, `judge`, `score`, `pool`, `pipeline`, `resume`) become the only entry points. `vera judge` exposes rubric selection via `--rubric` (reusing Phase 0's bundle-manifest helper directly) — rubric selection is not lost during the CLI replacement. `-u`/`-j`/`--sample` shorthand and `--config` ship now, using an **informal** `config.json` shape (mirrors the flags, `generation`/`judging` orthogonal blocks, mutual-exclusivity-with-CLI rule, `judging.rubrics` already a list) — not yet locked as a stable interface. The resolved-form-printed-at-start traceability behavior ships now too (stdout only). Internals underneath still call the existing `generate`/`judge` code as-is — no score split, old `p_*`/`j_*` output layout retained until Phase 2/3. `generate.py`/`judge.py`/`run_pipeline.py` are **deleted entirely at the end of this phase** — no deprecation-stub period. **Known risk, accepted explicitly:** configs written against this informal shape may need updating once Phase 3 formalizes `utils/config_schema.py` as a stable interface | `pytest -m "not live"` green; `vera.py` is the only documented entry point, and `generate.py`/`judge.py`/`run_pipeline.py` no longer exist in the repo; `-u`/`-j`/`--sample`/`--config`/`--rubric` all functional against the existing internals; a **structural** parity/regression test suite (same call sequence, arguments, and output structure — see the generation-testing note below) proves the new CLI wires into the same underlying engine calls the deleted scripts made. Comparatively low risk: Phase 1 only replaces the front end, the generation/judging engine itself (`generate/`, `judge/` internals) is untouched here — the higher-risk moment is Phase 5, where the engine itself changes | +| **1 — New CLI + config** | `vera.py` fully replaces the top-level scripts | Add whole-target selection through `--target` and top-level config `target`. Preserve explicit component selection through `--personas` plus `--persona-context-template`, and `--rubric`, so callers can keep generation and judging independent. Resolve target manifests before print or dispatch, then call existing domain behavior directly. Ship `-u`/`-j`/`--sample` and the informal config shape; delete `generate.py`/`judge.py`/`run_pipeline.py` at the end of the phase | `pytest -m "not live"` green; `vera.py` is the only documented entry point; target and explicit-component paths have structural parity tests; `--config`, `--target`, `--personas`, `--rubric`, `-u`, `-j`, and `--sample` are functional; legacy root scripts are gone | | **2 — Scoring split** | Extract `score/` | `score.py`/`score_viz.py`/`pool.py` move out of `judge/` into `score/`; pure move, no new behavior. `vera.py` (the only entry point since Phase 1) gets its imports updated directly — no shim needed, since the legacy root scripts no longer exist. A minimal import-linter contract is added covering only the `judge/` ⊥ `score/` boundary this phase creates | `pytest -m "not live"` green; the `judge/` ⊥ `score/` import-linter contract passes; no code imports `judge.score`/`judge.pool` | | **3 — Traceability & naming** | Harden Phase 1's config shape; add persistence; swap in the new naming scheme | `utils/config_schema.py` formalizes Phase 1's informal `config.json` shape into a stable interface (design doc required for future changes) — `judging.rubrics` stays a list (continuing the list-from-day-one approach already used since Phase 0/1), still length-1-only in practice until Phase 4. Adds the persisted artifacts: `config.json` written to disk + `state.json` + `.sha256` sidecar; `utils/naming.py` (the naming/layout module, already tracked today implementing the legacy `p_`/`a_` scheme) is **rewritten** for the `c_`/`u_`/`j_` scheme, retiring the `p_*`/`j_*` layout Phase 1 kept. Existing `output/` run folders under the old layout are left alone — no migration script, only new runs use the new layout. **Acknowledged compatibility break:** anything outside `vera.py` that parses the old `p_*`/`j_*` pattern directly (`spring_scripts/`, `distribute_files.py`, `score_comparison.py`, notebooks, human-review tooling) breaks the moment new runs use `c_*`/`u_*`/`j_*` instead — accepted, since the vast majority of real usage goes through the CLI, not direct path-parsing. **This break is about auto-discovery, not about reading old data at all:** `vera judge --conversations ` and `vera score -r ` keep working against existing old-layout output, since both take an explicit path and read files by their own format, never by re-deriving meaning from the parent folder's naming pattern. What genuinely doesn't carry over is `vera resume` on an old run — `config.json`/`.sha256`/`state.json` didn't exist under the old layout, so there's nothing for `resume` to read regardless of naming scheme. Import-linter contract extended to cover `utils/` as a leaf | `pytest -m "not live"` green; a run's `config.json` round-trips through `vera resume`; `utils/` leaf-layer import-linter contract passes | | **4 — Multi-rubric support** | Support multiple rubrics per run | Built on Phase 3's `config.json`/naming: `judging.rubrics[]` now supports length > 1 (the list shape has existed since Phase 0 — this phase lifts the length-1 restriction, it doesn't introduce the list); per-rubric judge-model overrides; per-rubric `evaluations//` folder separation | `pytest -m "not live"` green; a config with 2+ rubrics produces separated `evaluations//` output for each; **and** a pre-existing length-1 `judging.rubrics` config from Phase 1-3 still produces identical behavior — backward compatibility with the single-rubric case is verified, not just the new multi-rubric case | -**Adding a new rubric only requires Phase 0-4, not Phase 5.** Everything a new rubric needs — the bundle-manifest format, `--rubric`/`--target` selection, and (once Phase 4 lands) running it alongside other rubrics in the same config — is available as soon as Phase 4 is done. Phase 5 replaces the generation/judging *engine* underneath (`workers/` unification, concurrency handling); it doesn't change what a rubric is or how one gets added. A new rubric can ship on Phase 4 alone and doesn't need to wait for Phase 5 to land first. +**Adding a new target or rubric only requires Phase 0-4, not Phase 5.** A complete target is added as `data//manifest.json`; an expert may still select its rubric component explicitly with `--rubric`. Phase 5 replaces the generation/judging *engine* underneath and does not change target or rubric definition. | **5 — Substantial refactoring** | Everything else | `workers/` unification (both `generate/runner.py` and `judge/runner.py` already hand-roll the same asyncio-queue worker-pool pattern independently — concrete evidence for this step); `llm_clients/` plugin-registry formalization; import-linter contract completed (all remaining boundaries) + grimp; quality-gate tightening (pyright blocking, coverage 30%→60%); root-clutter cleanup (see below). **This is the phase that actually rewrites the generation/judging engine** (both runners move off their own hand-rolled asyncio queues onto the shared `workers/` pool) — see the engine-testing note below, this needs more care than any structural CLI change. **Check whether `scripts/pool_vera_scores.py` is still needed** at this point — if so, fold its logic into `score/pool.py` and remove or thin the script; if `vera pool` has already fully superseded it by now, just delete it. **Provider concurrency limits, not yet addressed anywhere in this doc:** the shared `workers/` pool can fan out many parallel jobs (e.g. `-u gpt:5 sonnet:5`); `llm_clients/` already has per-call retry/backoff but nothing caps per-provider *concurrency*, so real parallelism at this phase could blow through a provider's rate limit for the first time. Needs a concurrency cap per provider before this phase is considered done, not just noted as a known gap | `pytest -m "not live"` green; pyright blocking in CI; coverage ≥ 60%; full import-linter contract (all layer boundaries from the [Layer model](#layer-model) section) passes; no root-level Python entry points except `vera.py`; all four testing tiers described below (structural parity, live-LLM smoke test, semantic similarity check, manual spot-check), specifically for the `workers/` migration; a per-provider concurrency cap exists and is exercised by at least one test that fans out more jobs than the cap allows | **Rollback:** a phase that ships broken is reverted via its own PR(s) — there is no dual-path/feature-flag expectation carrying old and new behavior side by side during a phase's rollout. diff --git a/docs/design/vera-cli-runtime-wiring.md b/docs/design/vera-cli-runtime-wiring.md index 635e18948..e91de34b2 100644 --- a/docs/design/vera-cli-runtime-wiring.md +++ b/docs/design/vera-cli-runtime-wiring.md @@ -15,7 +15,7 @@ The unified CLI runtime and configuration contracts are canonical in: - [Architecture: CLI runtime boundary](../architecture.md#cli-runtime-boundary) - [Architecture: input resolution](../architecture.md#input-resolution) -- [Architecture: rubric bundle manifest](../architecture.md#rubric-bundle-manifest) +- [Architecture: target manifest](../architecture.md#target-manifest) - [CLI/config use cases](../vera-cli-use-cases.md) This record captures the rationale for the corresponding stable-interface @@ -31,6 +31,9 @@ domain functions. - Generation configs that relied on Python or CLI defaults must add the required generation behavior fields explicitly. CLI invocations retain defaults at the flag-definition boundary. +- Target manifests are complete bundles of rubric, personas, and prompts. + `--target` selects the bundle, while explicit `--personas` and `--rubric` + remain available for callers who want to select components independently. - Legacy root scripts remain usable only during their feature-by-feature replacement. They are not dependencies of the unified CLI and are removed in a later cleanup PR. diff --git a/docs/vera-cli-use-cases.md b/docs/vera-cli-use-cases.md index e96a0725d..e9a585809 100644 --- a/docs/vera-cli-use-cases.md +++ b/docs/vera-cli-use-cases.md @@ -17,20 +17,35 @@ Three entities, each with a single-letter prefix used throughout the CLI, config **These letters are `vera.py`-only and are not the same flags as today's scripts.** `generate.py`/`judge.py` already use `-c`/`-r` for unrelated things (`-c` is `--max-concurrent` in `generate.py` and `--conversation` in `judge.py`; `-r` is `--runs` in `generate.py` and `--rubrics` in `judge.py`). `vera.py` intentionally repurposes them for the `u`/`c`/`j` vocabulary above. There is no coexistence window: Phase 1 of the migration (see [architecture.md#migration-from-current-layout](./architecture.md#migration-from-current-layout)) deletes `generate.py`/`judge.py`/`run_pipeline.py` entirely in the same change that ships `vera.py`, so the old and new meanings of `-c`/`-r` never need to be told apart at runtime. +A **target** is separate from the u/c/j entities: it is a reusable evaluation +bundle containing a rubric, personas, and all generation/judging prompts. A +target is represented by `data//manifest.json`. + ## Minimum required arguments CLI shorthand and the input `config.json` are deliberately mirrored, flag-for-field — the same information is required either way, just spelled differently: | Subcommand | CLI shorthand — minimum required | Input `config.json` — minimum required fields | |---|---|---| -| `generate` | `-c ` + `-u ` (≥1) + (`--personas ` **or** `--target `) | `generation.chatbot`, `generation.user` (≥1), and (`generation.personas` (≥1) **or** top-level `target`) | -| `judge` | `-j ` (≥1) + `--conversations ` + `--rubric ` | `judging.models` (≥1), `judging.rubrics` (≥1), and `judging.conversations` (≥1) | +| `generate` | `-c ` + `-u ` (≥1) + (`--target ` **or** explicit `--personas ` + `--persona-context-template `) | `generation.chatbot`, `generation.user` (≥1), and (top-level `target` **or** explicit `generation.personas` + `generation.persona_context_template`) | +| `judge` | `-j ` (≥1) + `--conversations ` + (`--target ` **or** explicit `--rubric `) | `judging.models` (≥1), `judging.conversations` (≥1), and (top-level `target` **or** explicit `judging.rubrics` (≥1)) | | `pipeline` | everything `generate` needs **and** everything `judge` needs — or just `-c`, `-u`, `-j`, `--target ` (`--target` covers persona+rubric in one shot) | both `generation` and `judging` blocks fully populated, or `-c`/`-u`/`-j`-equivalent fields plus a top-level `target`; `judging.conversations` is omitted because generation supplies it | | `score` | the results path (`-r `) — nothing else | n/a — `score` reads an existing `results.csv`, not a run config | | `pool` | `--evaluations ` (≥1) | n/a | | `resume` | `--config ` — that's the entire invocation | n/a (it *is* the config being resumed) | -**`target` mirrors `--target` exactly, including its mutual-exclusivity rule:** a top-level `target` field in the input `config.json` expands to `generation.personas` + `judging.rubrics` from the resolved rubric bundle manifest, the same expansion `--target ` performs on the CLI. **Setting `target` alongside explicit `generation.personas` or `judging.rubrics` in the same input config is an error** — rejected outright, not silently merged or overridden — mirroring the existing either/or rule between CLI flags and `--config`. The run's own immutable `config.json` artifact (written to `output/.../config.json`, the one `vera resume` reads) always stores the fully-expanded form — `target` is resolved away before that artifact is written, the same way `-u`/`-j` shorthand is already resolved into concrete model entries today, so `vera resume` never has to re-resolve a manifest that might have changed on disk since the run started. +**`target` mirrors `--target` exactly, including its mutual-exclusivity rule:** +a top-level `target` field in the input `config.json` resolves one complete target +manifest and expands its personas, persona prompt, rubric, and judging prompts. +Setting `target` alongside explicit `generation.personas` or +`judging.rubrics` is an error — rejected outright, not silently merged or +overridden. The run's own immutable `config.json` always stores the fully +expanded form, so `vera resume` never re-resolves a manifest that might have +changed. + +Whole-target selection is optional. Callers who want independent control may +use `--personas` plus `--persona-context-template` for generation and `--rubric` +for judging; a pipeline can use both explicit forms instead of `--target`. ## Use case 1 — End-to-end test of one LLM @@ -42,9 +57,16 @@ vera pipeline --config run.json **Resolved:** stays single-chatbot-per-invocation. Comparing chatbots is always an external loop over single-chatbot pipeline runs, consistent with use case 2. Native multi-chatbot support (one combined comparison report) is not built now — flagged as a possible future addition if a real need emerges, not ruled out permanently. -**`--target ` shorthand:** `vera pipeline --target SI` resolves `SI` to one rubric bundle manifest (see [Rubric bundle manifest](./architecture.md#rubric-bundle-manifest)) and sets *both* the generation personas and the judging rubric from it in one shot — for the common case of "run the canonical test for X." This is the one deliberate exception to personas/rubrics being chosen independently; every other invocation (`--rubric` plus separately-specified personas, or explicit `--config` blocks) keeps generation and judging fully orthogonal. `--target` never selects the chatbot — `-c`/`generation.chatbot` is required regardless of whether `--target` is used. +**`--target ` shorthand:** `vera pipeline --target SI` resolves +`data/SI/manifest.json` and selects its personas, persona prompt, rubric, and +judging prompts in one shot — the common case of “run the canonical test for +SI.” It never selects the chatbot. Explicit `--personas` and `--rubric` remain +available when the caller deliberately wants different components. -**No implicit "run everything":** if neither `--target` nor `--rubric`/`judging.rubrics` is given, the CLI errors rather than defaulting to some or all rubrics. To deliberately run every known evaluator, use `--target all`, which resolves every rubric bundle manifest — an explicit opt-in, not a default. +**No implicit "run everything":** if neither `--target` nor explicit +`--rubric`/`judging.rubrics` is given where judging is requested, the CLI errors. +`--target all` deliberately resolves every discovered target manifest as a +separate invocation. It never merges personas or prompts from different targets. ## Use case 2 — Batch generate across personas @@ -59,18 +81,25 @@ vera generate --config run.json **Option B — CLI shorthand:** ``` -vera generate -c sonnet -u gpt:1 sonnet:2 --personas data/personas.tsv +vera generate -c sonnet -u gpt:1 sonnet:2 \ + --personas data/personas.tsv \ + --persona-context-template data/custom/persona_prompt.txt ``` -Bare-minimum required flags for Option B: `-c ` (the chatbot under test — no default), `-u ` (at least one user-side model), and either `--personas ` or `--target ` (personas have no silently-assumed default file either). `--rubric`/`--target` are additionally required if this invocation will also be judged. +Bare-minimum required flags for Option B: `-c ` (the chatbot under +test — no default), `-u ` (at least one user-side model), and +either a complete `--target` or explicit `--personas` plus +`--persona-context-template`. `--personas` is a supported first-class path, not +only a compatibility fallback. Generation behavior is also controlled at this input boundary. CLI invocations default to `--turns 3`, `--output output`, unlimited concurrency, no total-word -cap, persona-first ordering, one unnamed session, and -`--persona-context-template data/SI/persona_context_template.txt` when explicit -persona files are selected. `--target` obtains the context template from its -manifest instead. The generation runner itself has no parameter defaults. +cap, persona-first ordering, and one unnamed session. Explicit persona files use +the context template supplied through `--persona-context-template`; `--target` +obtains it from the target manifest. The generation runner itself has no +parameter defaults. -Or, using `--target` to pull personas from a rubric bundle manifest instead of naming them explicitly: +Or, using `--target` to select the complete target manifest instead of naming +the generation inputs explicitly: ``` vera generate -c sonnet -u gpt:1 --target SI ``` @@ -83,18 +112,26 @@ Personas come from one or more persona files, each containing multiple personas; ## Use case 3 — Judge existing conversations -Judge one **or more** existing transcript folders against one or more rubrics. Each rubric has a default judge-LLM set, overridable per rubric. Multiple judges per rubric are supported (for judge-agreement analysis). Judging is decoupled from generation — point it at whatever conversation folders you need, with no enforced coupling to originating personas. +Judge one **or more** existing transcript folders using either a complete target +or an explicitly selected rubric. Each rubric has a default judge-LLM set, +overridable per rubric. Multiple judges per rubric are supported. Judging remains +decoupled from generation. ``` vera judge --config run.json -vera judge -j claude:1 gpt:2 --conversations output/c_sonnet//conversations/ --rubric data/SI/rubric_manifest.json +vera judge -j claude:1 --conversations output/c_sonnet//conversations/ --target SI +vera judge -j claude:1 gpt:2 --conversations output/c_sonnet//conversations/ --rubric data/SI/manifest.json ``` No `-c` here: judging is decoupled from chatbot selection by design (see the orthogonality invariant above) — the chatbot is already implicit in whichever `--conversations` folder is passed in. `-j : ...` mirrors `-u`'s syntax for the judge side. `repeats` here means re-running the same transcript through the same judge model N times, to measure judge consistency/variance. -`--rubric`/`judging.rubrics[]` entries point at a [rubric bundle manifest](./architecture.md#rubric-bundle-manifest) (canonical definition), not a bare `.tsv` path. +`--target` consumes the rubric and judging prompts from the selected complete +[target manifest](./architecture.md#target-manifest). Explicit `--rubric` does +the same for only the rubric portion and does not select that target's personas +or persona prompt. It accepts a target name or manifest path, never a bare TSV +with implicitly assumed sibling prompts. **Resolved (multi-folder judge output):** judge keeps results independent per folder; the `score/` layer aggregates across folders when needed, not judge itself. There are also in-between options — e.g. kept separate, but the score layer aggregates them. @@ -140,7 +177,7 @@ Reads the immutable `config.json` (verifying its `.sha256` sidecar first) plus t - **CLI flags and `--config` are strictly either/or, never combined for the same run.** A given piece of information (model selection/repeats, sampling knobs, persona/rubric lists) is supplied via one or the other, never both — the implementation rejects the combination rather than silently merging. **`--sample ` (see [Use case 4](#use-case-4--smoke-test)) is the one deliberate exception:** it MAY be passed alongside `--config`, since it never carries information `config.json` defines -- it only caps how much of the already-resolved lists get used for that invocation, purely for debugging, and it's never written into the run's own `config.json` artifact. - Internally, `--config` always resolves to the same canonical flag-set the CLI would produce, so there is exactly one resolved form regardless of input path. The tool prints this resolved form at run start for terminal/CI-log visibility (it does not write to the shell's own history — an opt-in `--print` flag emits the resolved flag-string with no execution, for a caller who wants to `eval` it into their own shell explicitly). - JSON, not YAML — robust when passed as a one-line env var or stdin payload with no escaping ambiguity. -- **Path fields inside `config.json` (`generation.personas`, etc.) resolve relative to `$ROOT`** — the directory containing `vera.py` — never relative to the current working directory the CLI was invoked from, and never relative to `config.json`'s own location. This is a single rule regardless of how the config arrives (`--config `, `--config -`, or `VERA_RUN_CONFIG`), so a config's meaning never depends on where your shell happens to be or where you saved the file. This is distinct from the [rubric bundle manifest](./architecture.md#rubric-bundle-manifest), which deliberately resolves relative to *itself* instead, so a manifest folder stays portable across checkouts — `config.json` doesn't need that property, since it's checkout-specific by nature. +- **Path fields inside `config.json` (`generation.personas`, etc.) resolve relative to `$ROOT`** — the directory containing `vera.py` — never relative to the current working directory or the config file. A [target manifest](./architecture.md#target-manifest) deliberately resolves its fields relative to its own directory so the complete target remains portable. ### `config.json` shape @@ -180,11 +217,17 @@ Top-level `generation` and `judging` blocks are **completely orthogonal** — mo `pipeline` config omits it because the generation stage supplies the resolved conversation paths without combining config input with a CLI flag. +In an input config, `judging.rubrics` is the explicit component-selection path: +each entry identifies a target name or manifest and consumes only its rubric and +judging prompts. In the persisted canonical config, those entries contain the +resolved concrete paths rather than a manifest reference. + These generation behavior fields are required in config mode, including explicit `null` where no limit or session list is intended. Config mode never -inherits the CLI defaults. When top-level `target` supplies the persona bundle, -`persona_context_template` is `null` and the selected manifest must define -`persona_context_template_file`. +inherits the CLI defaults. Top-level `target` expands a complete manifest into +the concrete generation and judging fields before the canonical config is +printed or persisted. Explicit `generation.personas` and `judging.rubrics` +remain supported when no top-level target is set. `generation.chatbot` is the chatbot under test — same shape as one entry in `generation.user`, but a single object, not a list (only one chatbot per run; see use case 1). It is distinct from `generation.user`, which is the user-side (`u`) LLM list — the two share a field shape but are never conflated: naming one `chatbot` and the other `user` (rather than both `models`) makes which is which unambiguous at the field-name level, not just from prose. From 65fe3b9085f667398c71d48491fb3e084dd4c59c Mon Sep 17 00:00:00 2001 From: Luca Belli <129434630+sator-labs@users.noreply.github.com> Date: Fri, 7 Aug 2026 09:41:41 -0700 Subject: [PATCH 3/6] docs: resolve explicit inputs from targets --- docs/ARCHITECTURE-SPINE.md | 4 +-- docs/architecture.md | 29 +++++++++++--------- docs/design/vera-cli-runtime-wiring.md | 3 +- docs/vera-cli-use-cases.md | 38 +++++++++++++------------- 4 files changed, 39 insertions(+), 35 deletions(-) diff --git a/docs/ARCHITECTURE-SPINE.md b/docs/ARCHITECTURE-SPINE.md index 2cff3644c..604aef077 100644 --- a/docs/ARCHITECTURE-SPINE.md +++ b/docs/ARCHITECTURE-SPINE.md @@ -209,13 +209,13 @@ No edge runs `generate/` ↔ `judge/` ↔ `score/`, and none runs from `workers/ - **Binds:** target manifests, `vera_cli/`, `utils/config_schema.py`, `generate/`, `judge/` - **Prevents:** partial manifests that work for one command but fail for another; per-run execution knobs leaking into reusable target definitions; selecting a target when the caller intended to control generation and judging independently. -- **Rule:** [ADOPTED] A target is a complete evaluation bundle defined by `data//manifest.json`. Every manifest requires `rubric_file`, `rubric_prompt_beginning_file`, `question_prompt_file`, `personas`, and `persona_context_template_file`; paths resolve relative to the manifest. `--target ` and top-level config `target` consume the complete bundle. `--personas` plus `--persona-context-template` and `--rubric ` remain explicit alternatives that consume only the selected generation or judging components, and a pipeline may use both explicit forms instead of `--target`. Target selection is mutually exclusive with explicit persona/rubric selection. `--target all` produces one canonical invocation per discovered target and never merges different targets' personas, prompts, or rubrics. Target expansion completes before print, persistence, or dispatch; canonical and persisted `RunConfig` contains concrete paths and no `target`. Model defaults and all other run behavior belong in CLI flag definitions or `config.json`, never in the manifest. +- **Rule:** [ADOPTED] A target is a complete evaluation bundle defined by `data//manifest.json`. Every manifest requires `rubric_file`, `rubric_prompt_beginning_file`, `question_prompt_file`, `personas`, and `persona_context_template_file`; paths resolve relative to the manifest. `--target ` and top-level config `target` consume the complete bundle. `--personas ` and `--rubric ` remain explicit alternatives: the former consumes the target's personas and persona prompt, while the latter consumes its rubric and judging prompts. A pipeline may combine components from different targets, such as `--personas HFO --rubric SI`. Whole-target selection is mutually exclusive with explicit component selection. `--target all` produces one canonical invocation per discovered target and never merges different targets' personas, prompts, or rubrics. Target expansion completes before print, persistence, or dispatch; canonical and persisted `RunConfig` contains concrete paths and no target selectors. Model defaults and all other run behavior belong in CLI flag definitions or `config.json`, never in the manifest. ### AD-22 — Target-manifest resolution belongs to the CLI boundary - **Binds:** `vera_cli/arguments.py`, `generate/`, `judge/` - **Prevents:** either domain owning a cross-domain target definition; generation and judging reimplementing manifest resolution differently; legacy script parsers becoming architectural dependencies. -- **Rule:** [ADOPTED] `vera_cli/arguments.py` loads and validates target manifests and expands them to concrete generation and judging inputs. Domain functions receive only their resolved values and never a target manifest. Explicit `--rubric` consumes the rubric and judging-prompt portion of a complete target manifest; explicit `--personas` consumes caller-provided persona files and its separately resolved persona-context template. Legacy helpers may adapt these inputs during migration but are not dependencies of the unified CLI. +- **Rule:** [ADOPTED] `vera_cli/arguments.py` loads and validates target manifests and expands them to concrete generation and judging inputs. Domain functions receive only their resolved values and never a target manifest. Explicit `--rubric ` consumes the rubric and judging-prompt portion; explicit `--personas ` consumes the personas and persona-prompt portion. Legacy helpers may adapt these inputs during migration but are not dependencies of the unified CLI. ### AD-23 — Output layout is nested-only, with path-first stage contracts diff --git a/docs/architecture.md b/docs/architecture.md index a60a172d9..2f089b145 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -147,10 +147,11 @@ invocation-only; they are not serialized into `RunConfig`. `-c` selects the chatbot under test; `-u`/`-j` shorthand selects models/repeats for the user/judge side respectively; bespoke sampling knobs are config-only. `--target` selects a complete target, while `--personas` and `--rubric` explicitly select only the -generation or judging inputs from their respective sources. `--rubric` remains -list-shaped, though only a length-1 list is supported until Phase 4. `-c` is -required for `generate`/`pipeline` whenever `--config` isn't used — there is no -default chatbot. +persona or rubric component of a named target. Each component includes its +associated prompt from that target's manifest. `--rubric` remains list-shaped, +though only a length-1 list is supported until Phase 4. `-c` is required for +`generate`/`pipeline` whenever `--config` isn't used — there is no default +chatbot. Generation behavior defaults are defined only at the CLI flag boundary. A config-driven run provides the corresponding generation fields explicitly; it @@ -171,8 +172,8 @@ conversation paths directly. ```bash uv run python vera.py pipeline --config run.json uv run python vera.py generate -c sonnet -u gpt:1 --target SI -uv run python vera.py generate -c sonnet -u gpt:1 --personas data/custom/personas.tsv --persona-context-template data/custom/persona_prompt.txt -uv run python vera.py judge -j claude:1 --rubric data/SI/manifest.json --conversations output/c_sonnet//conversations/ +uv run python vera.py generate -c sonnet -u gpt:1 --personas SI +uv run python vera.py judge -j claude:1 --rubric SI --conversations output/c_sonnet//conversations/ uv run python vera.py score -r output/.../results.csv uv run python vera.py pool --evaluations path/to/evaluations/... path/to/evaluations/... uv run python vera.py resume --config output/c_sonnet//config.json @@ -243,11 +244,12 @@ persona-prompt fields; `vera judge --target ...` consumes its rubric and judging-prompt fields; `vera pipeline --target ...` consumes both. Advanced callers may keep generation and judging independent. `--personas` -selects persona files explicitly (with `--persona-context-template` controlling -their prompt), and `--rubric ` explicitly selects only the -rubric and judging-prompt portion of a target manifest. A pipeline may use -`--personas` and `--rubric` together instead of `--target`. These explicit forms -do not pull in the target's other components. +`` selects only the personas and persona prompt from that +target, while `--rubric ` selects only its rubric and +judging prompts. For example, `--personas HFO --rubric SI` deliberately combines +the persona side of HFO with the rubric side of SI. A pipeline may use both +explicit flags instead of `--target`; neither flag pulls in the target's other +component. A top-level `target` in an input config mirrors whole-target selection. Setting it alongside explicit `generation.personas` or `judging.rubrics` is an error, @@ -329,7 +331,8 @@ Agents and contributors must comply. Import boundaries are documented in the [La CLI/config inputs or define CLI behavior defaults. - **Targets:** every `manifest.json` defines one complete target (rubric, personas, and prompts). `--target` selects the complete bundle; - `--personas` and `--rubric` remain explicit component-level alternatives. + `--personas ` and `--rubric ` remain explicit component-level + alternatives and include the selected component's prompts. - **Generation:** conversation simulation logic stays in `generate/`; the simulator core is pure (no filesystem, no logging) — the handler owns all I/O. - **Judging:** rubric navigation and LLM-judge logic stay in `judge/`, also pure-core-plus-handler. Judge never auto-scores — `vera score`/`vera pool` are separate subcommands. **Rubric navigation logic lives in code, never in the prompt:** which question is asked next given an answer is determined entirely by `QuestionNavigator` walking `question_flow_data` parsed from the rubric TSV — the judge LLM answers/judges the current question only, and is never asked to decide or influence what comes next. - **Scoring:** aggregation, visualization, and pooling stay in `score/`, never re-absorbed into `judge/`. @@ -425,7 +428,7 @@ uv run pytest -m "not live" | **0 — De-risk multi-rubric** | Prove the target-manifest format on both generation and judging before the unified CLI replaces legacy scripts | Treat every manifest as one complete target containing a rubric, personas, and both prompt sets. Legacy judge code may consume only its rubric fields and legacy generation code only its persona fields, but both read the same complete manifest. Preserve the existing explicit persona/rubric paths so Phase 1 can offer both whole-target and component-level selection | `pytest -m "not live"` green; one complete fixture target drives both legacy generation and judging; incomplete manifests fail validation; explicit persona/rubric selection remains covered | | **S — Storage abstraction** *(orthogonal — see note above)* | Decouple "what path/key to use" from "how to persist bytes," so a future non-local backend (S3, etc.) is a new implementation, not a rewrite | New `storage/` package: `StorageBackend` (ABC) with `write(key, bytes)` / `read(key)` / `exists(key)`, plus `LocalFilesystemStorage` as the default implementation — mirrors the `LLMInterface`/`QueueProtocol` idiom (interface and implementations live together in one concern-scoped package). The backend knows nothing about run semantics; `utils/naming.py` still builds keys/paths, `storage/` just persists what it's given. All domain/`workers/` code that currently touches the filesystem directly switches to calling through `StorageBackend` instead | `pytest -m "not live"` green; no domain or `workers/` code calls `open()`/`pathlib` file-write directly for run artifacts — everything routes through `StorageBackend`; `LocalFilesystemStorage` is behaviorally identical to today's direct-filesystem writes | | **O — Adopt OpenSpec** *(orthogonal — see note above)* | Turn "consider an OpenSpec change if the team adopts that workflow" from a maybe into an actual, exercised requirement | Populate `openspec/changes/` with a real OpenSpec change document the next time a large multi-file feature lands (the existing ESCALATE trigger: new judge dimensions, pipeline CLI changes). Currently `openspec/` is empty scaffolding — this phase is "done" only once a real change has actually gone through it, not just once the config exists | A qualifying multi-file feature has shipped with a real OpenSpec change document under `openspec/changes/`, and the ESCALATE section's language is updated from "if the team adopts" to a firm MUST for future qualifying changes | -| **1 — New CLI + config** | `vera.py` fully replaces the top-level scripts | Add whole-target selection through `--target` and top-level config `target`. Preserve explicit component selection through `--personas` plus `--persona-context-template`, and `--rubric`, so callers can keep generation and judging independent. Resolve target manifests before print or dispatch, then call existing domain behavior directly. Ship `-u`/`-j`/`--sample` and the informal config shape; delete `generate.py`/`judge.py`/`run_pipeline.py` at the end of the phase | `pytest -m "not live"` green; `vera.py` is the only documented entry point; target and explicit-component paths have structural parity tests; `--config`, `--target`, `--personas`, `--rubric`, `-u`, `-j`, and `--sample` are functional; legacy root scripts are gone | +| **1 — New CLI + config** | `vera.py` fully replaces the top-level scripts | Add whole-target selection through `--target` and top-level config `target`. Preserve explicit component selection through `--personas ` and `--rubric `, so callers can combine the persona side of one target with the rubric side of another. Resolve target manifests before print or dispatch, then call existing domain behavior directly. Ship `-u`/`-j`/`--sample` and the informal config shape; delete `generate.py`/`judge.py`/`run_pipeline.py` at the end of the phase | `pytest -m "not live"` green; `vera.py` is the only documented entry point; target and explicit-component paths have structural parity tests; `--config`, `--target`, `--personas`, `--rubric`, `-u`, `-j`, and `--sample` are functional; legacy root scripts are gone | | **2 — Scoring split** | Extract `score/` | `score.py`/`score_viz.py`/`pool.py` move out of `judge/` into `score/`; pure move, no new behavior. `vera.py` (the only entry point since Phase 1) gets its imports updated directly — no shim needed, since the legacy root scripts no longer exist. A minimal import-linter contract is added covering only the `judge/` ⊥ `score/` boundary this phase creates | `pytest -m "not live"` green; the `judge/` ⊥ `score/` import-linter contract passes; no code imports `judge.score`/`judge.pool` | | **3 — Traceability & naming** | Harden Phase 1's config shape; add persistence; swap in the new naming scheme | `utils/config_schema.py` formalizes Phase 1's informal `config.json` shape into a stable interface (design doc required for future changes) — `judging.rubrics` stays a list (continuing the list-from-day-one approach already used since Phase 0/1), still length-1-only in practice until Phase 4. Adds the persisted artifacts: `config.json` written to disk + `state.json` + `.sha256` sidecar; `utils/naming.py` (the naming/layout module, already tracked today implementing the legacy `p_`/`a_` scheme) is **rewritten** for the `c_`/`u_`/`j_` scheme, retiring the `p_*`/`j_*` layout Phase 1 kept. Existing `output/` run folders under the old layout are left alone — no migration script, only new runs use the new layout. **Acknowledged compatibility break:** anything outside `vera.py` that parses the old `p_*`/`j_*` pattern directly (`spring_scripts/`, `distribute_files.py`, `score_comparison.py`, notebooks, human-review tooling) breaks the moment new runs use `c_*`/`u_*`/`j_*` instead — accepted, since the vast majority of real usage goes through the CLI, not direct path-parsing. **This break is about auto-discovery, not about reading old data at all:** `vera judge --conversations ` and `vera score -r ` keep working against existing old-layout output, since both take an explicit path and read files by their own format, never by re-deriving meaning from the parent folder's naming pattern. What genuinely doesn't carry over is `vera resume` on an old run — `config.json`/`.sha256`/`state.json` didn't exist under the old layout, so there's nothing for `resume` to read regardless of naming scheme. Import-linter contract extended to cover `utils/` as a leaf | `pytest -m "not live"` green; a run's `config.json` round-trips through `vera resume`; `utils/` leaf-layer import-linter contract passes | | **4 — Multi-rubric support** | Support multiple rubrics per run | Built on Phase 3's `config.json`/naming: `judging.rubrics[]` now supports length > 1 (the list shape has existed since Phase 0 — this phase lifts the length-1 restriction, it doesn't introduce the list); per-rubric judge-model overrides; per-rubric `evaluations//` folder separation | `pytest -m "not live"` green; a config with 2+ rubrics produces separated `evaluations//` output for each; **and** a pre-existing length-1 `judging.rubrics` config from Phase 1-3 still produces identical behavior — backward compatibility with the single-rubric case is verified, not just the new multi-rubric case | diff --git a/docs/design/vera-cli-runtime-wiring.md b/docs/design/vera-cli-runtime-wiring.md index e91de34b2..c6a4b3b9f 100644 --- a/docs/design/vera-cli-runtime-wiring.md +++ b/docs/design/vera-cli-runtime-wiring.md @@ -33,7 +33,8 @@ domain functions. flag-definition boundary. - Target manifests are complete bundles of rubric, personas, and prompts. `--target` selects the bundle, while explicit `--personas` and `--rubric` - remain available for callers who want to select components independently. + select the named target's persona or rubric component, including that + component's associated prompts. - Legacy root scripts remain usable only during their feature-by-feature replacement. They are not dependencies of the unified CLI and are removed in a later cleanup PR. diff --git a/docs/vera-cli-use-cases.md b/docs/vera-cli-use-cases.md index e9a585809..f97631e74 100644 --- a/docs/vera-cli-use-cases.md +++ b/docs/vera-cli-use-cases.md @@ -23,11 +23,13 @@ target is represented by `data//manifest.json`. ## Minimum required arguments -CLI shorthand and the input `config.json` are deliberately mirrored, flag-for-field — the same information is required either way, just spelled differently: +CLI shorthand and input `config.json` resolve to the same canonical form. CLI +component selectors use target names for convenience; explicit config fields +may state the resulting concrete paths directly. | Subcommand | CLI shorthand — minimum required | Input `config.json` — minimum required fields | |---|---|---| -| `generate` | `-c ` + `-u ` (≥1) + (`--target ` **or** explicit `--personas ` + `--persona-context-template `) | `generation.chatbot`, `generation.user` (≥1), and (top-level `target` **or** explicit `generation.personas` + `generation.persona_context_template`) | +| `generate` | `-c ` + `-u ` (≥1) + (`--target ` **or** explicit `--personas `) | `generation.chatbot`, `generation.user` (≥1), and (top-level `target` **or** explicit `generation.personas` + `generation.persona_context_template`) | | `judge` | `-j ` (≥1) + `--conversations ` + (`--target ` **or** explicit `--rubric `) | `judging.models` (≥1), `judging.conversations` (≥1), and (top-level `target` **or** explicit `judging.rubrics` (≥1)) | | `pipeline` | everything `generate` needs **and** everything `judge` needs — or just `-c`, `-u`, `-j`, `--target ` (`--target` covers persona+rubric in one shot) | both `generation` and `judging` blocks fully populated, or `-c`/`-u`/`-j`-equivalent fields plus a top-level `target`; `judging.conversations` is omitted because generation supplies it | | `score` | the results path (`-r `) — nothing else | n/a — `score` reads an existing `results.csv`, not a run config | @@ -44,8 +46,8 @@ expanded form, so `vera resume` never re-resolves a manifest that might have changed. Whole-target selection is optional. Callers who want independent control may -use `--personas` plus `--persona-context-template` for generation and `--rubric` -for judging; a pipeline can use both explicit forms instead of `--target`. +use `--personas HFO` for the persona side and `--rubric SI` for the rubric side; +a pipeline can use both explicit forms instead of `--target`. ## Use case 1 — End-to-end test of one LLM @@ -81,22 +83,20 @@ vera generate --config run.json **Option B — CLI shorthand:** ``` -vera generate -c sonnet -u gpt:1 sonnet:2 \ - --personas data/personas.tsv \ - --persona-context-template data/custom/persona_prompt.txt +vera generate -c sonnet -u gpt:1 sonnet:2 --personas SI ``` Bare-minimum required flags for Option B: `-c ` (the chatbot under test — no default), `-u ` (at least one user-side model), and -either a complete `--target` or explicit `--personas` plus -`--persona-context-template`. `--personas` is a supported first-class path, not -only a compatibility fallback. +either a complete `--target` or explicit `--personas `. `--personas` is +a supported first-class path, not only a compatibility fallback; it selects the +persona files and persona prompt from that target's manifest. Generation behavior is also controlled at this input boundary. CLI invocations default to `--turns 3`, `--output output`, unlimited concurrency, no total-word -cap, persona-first ordering, and one unnamed session. Explicit persona files use -the context template supplied through `--persona-context-template`; `--target` -obtains it from the target manifest. The generation runner itself has no -parameter defaults. +cap, persona-first ordering, and one unnamed session. The explicit persona +component includes the files and context template resolved from the target named +by `--personas`; `--target` resolves the same fields while also selecting the +rubric side. The generation runner itself has no parameter defaults. Or, using `--target` to select the complete target manifest instead of naming the generation inputs explicitly: @@ -120,7 +120,7 @@ decoupled from generation. ``` vera judge --config run.json vera judge -j claude:1 --conversations output/c_sonnet//conversations/ --target SI -vera judge -j claude:1 gpt:2 --conversations output/c_sonnet//conversations/ --rubric data/SI/manifest.json +vera judge -j claude:1 gpt:2 --conversations output/c_sonnet//conversations/ --rubric SI ``` No `-c` here: judging is decoupled from chatbot selection by design (see the orthogonality invariant above) — the chatbot is already implicit in whichever `--conversations` folder is passed in. @@ -128,10 +128,10 @@ No `-c` here: judging is decoupled from chatbot selection by design (see the ort `-j : ...` mirrors `-u`'s syntax for the judge side. `repeats` here means re-running the same transcript through the same judge model N times, to measure judge consistency/variance. `--target` consumes the rubric and judging prompts from the selected complete -[target manifest](./architecture.md#target-manifest). Explicit `--rubric` does -the same for only the rubric portion and does not select that target's personas -or persona prompt. It accepts a target name or manifest path, never a bare TSV -with implicitly assumed sibling prompts. +[target manifest](./architecture.md#target-manifest). Explicit `--rubric SI` +consumes only SI's rubric and judging prompts and does not select SI's personas +or persona prompt. Both flags accept a target name or manifest path, never a bare +TSV with implicitly assumed sibling prompts. **Resolved (multi-folder judge output):** judge keeps results independent per folder; the `score/` layer aggregates across folders when needed, not judge itself. There are also in-between options — e.g. kept separate, but the score layer aggregates them. From 49607caab82164d7bb7dd2548eef9359734aff80 Mon Sep 17 00:00:00 2001 From: Luca Belli <129434630+sator-labs@users.noreply.github.com> Date: Fri, 7 Aug 2026 15:37:20 -0700 Subject: [PATCH 4/6] docs: clarify transitional generation boundary --- docs/ARCHITECTURE-SPINE.md | 13 ++++---- docs/architecture.md | 41 +++++++++++++++----------- docs/design/vera-cli-runtime-wiring.md | 8 +++-- 3 files changed, 36 insertions(+), 26 deletions(-) diff --git a/docs/ARCHITECTURE-SPINE.md b/docs/ARCHITECTURE-SPINE.md index 604aef077..740c56734 100644 --- a/docs/ARCHITECTURE-SPINE.md +++ b/docs/ARCHITECTURE-SPINE.md @@ -64,7 +64,7 @@ No edge runs `generate/` ↔ `judge/` ↔ `score/`, and none runs from `workers/ - **Binds:** `vera.py`, `vera_cli/`, `generate/`, `judge/`, `score/` - **Prevents:** business logic creeping into the CLI entrypoint; a fat orchestrator re-implementing what domain runners already do; a second root-level entrypoint script reappearing after migration. -- **Rule:** [ADOPTED] `vera.py` is the only root-level executable; domain packages are libraries, not invoked directly as scripts, and no root-level Python entry-point scripts exist alongside it. `vera.py` only loads arguments and dispatches. CLI support lives in `vera_cli/`: `arguments.py` owns flag definitions, CLI defaults, and complete input resolution; `commands.py` owns thin orchestration adapters that call parser-independent domain functions directly. Domain packages never import `vera_cli/`, and neither `vera.py` nor `vera_cli/` contains domain behavior. +- **Rule:** [ADOPTED] `vera.py` is the only root-level executable in the target layout; domain packages are libraries, not invoked directly as scripts. `vera.py` only loads arguments and dispatches. CLI support lives in small, responsibility-focused `vera_cli/` modules: top-level parser, per-command flags/defaults, canonical config resolution, and thin command adapters. During migration an adapter may import a reusable function from a legacy root module, but never its CLI parser or a subprocess. Removing `generate.py` and introducing the permanent `generate/` package happen atomically. Domain packages never import `vera_cli/`, and neither `vera.py` nor `vera_cli/` contains domain behavior. ### AD-2 — Domain package isolation and workers inversion of control @@ -213,9 +213,9 @@ No edge runs `generate/` ↔ `judge/` ↔ `score/`, and none runs from `workers/ ### AD-22 — Target-manifest resolution belongs to the CLI boundary -- **Binds:** `vera_cli/arguments.py`, `generate/`, `judge/` +- **Binds:** `vera_cli/`, `generate/`, `judge/` - **Prevents:** either domain owning a cross-domain target definition; generation and judging reimplementing manifest resolution differently; legacy script parsers becoming architectural dependencies. -- **Rule:** [ADOPTED] `vera_cli/arguments.py` loads and validates target manifests and expands them to concrete generation and judging inputs. Domain functions receive only their resolved values and never a target manifest. Explicit `--rubric ` consumes the rubric and judging-prompt portion; explicit `--personas ` consumes the personas and persona-prompt portion. Legacy helpers may adapt these inputs during migration but are not dependencies of the unified CLI. +- **Rule:** [ADOPTED] `vera_cli/targets.py` loads and validates target manifests; each command's `*_config.py` resolver expands the relevant components into canonical inputs. Domain functions receive only their resolved values and never a target manifest. Explicit `--rubric ` consumes the rubric and judging-prompt portion; explicit `--personas ` consumes the personas and persona-prompt portion. Legacy helpers may adapt these inputs during migration but are not dependencies of the unified CLI. ### AD-23 — Output layout is nested-only, with path-first stage contracts @@ -316,8 +316,11 @@ Package tree: ```text vera.py # CLI orchestrator, thin vera_cli/ - arguments.py # flags, CLI defaults, config/target resolution - commands.py # thin adapters to domain functions + arguments.py # top-level parser + config.py, targets.py # shared config and manifest helpers + *_arguments.py # per-command flags and CLI defaults + *_config.py # per-command canonical resolution + *_command.py # thin adapters to domain functions generate/ conversation_simulator.py # pure core runner.py # owns I/O, delegates to workers/ diff --git a/docs/architecture.md b/docs/architecture.md index 2f089b145..4d9736164 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -55,8 +55,10 @@ Deep dives: [judge.md](./judge.md) (question flow and rubric navigation), [struc CLI layer ├── vera.py — sole executable; loads arguments and dispatches └── vera_cli/ - ├── arguments.py — flags, defaults, and config resolution - └── commands.py — thin command adapters + ├── arguments.py — top-level parser + ├── *_arguments.py — per-command flags and CLI defaults + ├── *_config.py — per-command canonical resolution + └── *_command.py — thin command adapters ↓ calls Domain packages (generate/, judge/, score/) ↓ register handlers with @@ -104,32 +106,35 @@ The CLI layer has three responsibilities: - `vera.py` is the thin executable and contains no command-specific business logic. -- `vera_cli/arguments.py` defines flags and their CLI defaults, reads JSON from a - file, stdin, or `VERA_RUN_CONFIG`, enforces input exclusivity, resolves paths - and targets, and produces the complete canonical configuration before print, - persistence, or dispatch. -- `vera_cli/commands.py` contains only orchestration adapters. It receives - resolved values and calls parser-independent domain functions directly; it - does not resolve configuration or invoke legacy CLI entry points. +- `vera_cli/arguments.py` builds the top-level parser. Small per-command + `*_arguments.py` modules define that command's flags and CLI defaults. +- Shared config input and target-manifest helpers stay in focused modules; + per-command `*_config.py` modules enforce input exclusivity and produce the + complete canonical configuration before print, persistence, or dispatch. +- Per-command `*_command.py` modules contain only orchestration adapters. They + receive resolved values and call importable Python functions directly; they + never invoke another CLI parser or subprocess. `utils/config_schema.py` owns schema validation and canonical serialization. It does not parse CLI arguments, read config or manifest files, resolve paths, or define CLI behavior defaults. Domain entry points accept resolved domain values rather than `argparse` -namespaces, input config files, or target manifests. Generation exposes its -application function directly from its runner; it has no service wrapper and no -behavioral parameter defaults. Pooling likewise delegates to the function owned -by the scoring domain rather than to a script entry point. +namespaces, input config files, or target manifests. They define no CLI behavior +defaults. Pooling likewise delegates to the function owned by the scoring domain +rather than to a script entry point. Legacy root scripts may remain temporarily while their replacement feature is -migrated, but they are compatibility adapters only. They are not architectural -dependencies and are deleted after the corresponding `vera.py` command is -available. +migrated. During that transition, `vera_cli` may import the reusable function +from a root script, but never its argument parser. For generation, the temporary +flow is `vera_cli.generate_command` → `generate.main`. Removing `generate.py` +and moving that function plus the existing `generate_conversations/` code into +the permanent `generate/` package is one atomic later change, so a root +`generate.py` module and a top-level `generate/` package never coexist. | Subcommand | Delegates to | Purpose | |------------|--------------|---------| -| `vera generate` | `generate.runner` | Simulate conversations → `c_//conversations/` | +| `vera generate` | generation application function (temporarily `generate.main`) | Simulate conversations → `c_//conversations/` | | `vera judge` | `judge.runner` | Evaluate transcripts → `evaluations//j_*` | | `vera score` | `score.score` | Aggregate `results.csv` → scores and visualizations | | `vera pool` | `score.pool` | Concatenate multiple evaluation folders into one pooled result | @@ -271,7 +276,7 @@ orthogonal. | Package / path | Owns | Key modules | |----------------|------|-------------| -| `vera_cli/` | CLI flags/defaults, input resolution, thin command adapters | `arguments.py`, `commands.py` | +| `vera_cli/` | CLI flags/defaults, input resolution, thin command adapters | `arguments.py`, `config.py`, `targets.py`, per-command modules | | `generate/` | Simulation, turns, batch runner (pure core; handler owns I/O) | `conversation_simulator.py`, `runner.py` | | `judge/` | Rubric navigation, LLM judge, improvement reporting (pure core; handler owns I/O) | `question_navigator.py`, `llm_judge.py`, `scripts/summarize_results.py` | | `score/` | Aggregation, visualization, pooling — split out of `judge/` | `score.py`, `score_viz.py`, `pool.py` | diff --git a/docs/design/vera-cli-runtime-wiring.md b/docs/design/vera-cli-runtime-wiring.md index c6a4b3b9f..ac67b272e 100644 --- a/docs/design/vera-cli-runtime-wiring.md +++ b/docs/design/vera-cli-runtime-wiring.md @@ -35,6 +35,8 @@ domain functions. `--target` selects the bundle, while explicit `--personas` and `--rubric` select the named target's persona or rubric component, including that component's associated prompts. -- Legacy root scripts remain usable only during their feature-by-feature - replacement. They are not dependencies of the unified CLI and are removed in a - later cleanup PR. +- Legacy root parsers remain usable only during their feature-by-feature + replacement. A unified command may temporarily call an importable function + from the corresponding root module; it must not invoke that module's parser or + run it as a subprocess. Generation uses `generate.main` until the root module + is atomically replaced by the permanent `generate/` package. From aa1611ca3b19a4abe341e52adc2b2b1a995ab7c2 Mon Sep 17 00:00:00 2001 From: Luca Belli <129434630+sator-labs@users.noreply.github.com> Date: Tue, 11 Aug 2026 15:09:36 -0700 Subject: [PATCH 5/6] docs: simplify unified CLI module structure --- docs/ARCHITECTURE-SPINE.md | 9 +++------ docs/architecture.md | 37 +++++++++++++++++-------------------- 2 files changed, 20 insertions(+), 26 deletions(-) diff --git a/docs/ARCHITECTURE-SPINE.md b/docs/ARCHITECTURE-SPINE.md index 740c56734..0d466ad06 100644 --- a/docs/ARCHITECTURE-SPINE.md +++ b/docs/ARCHITECTURE-SPINE.md @@ -215,7 +215,7 @@ No edge runs `generate/` ↔ `judge/` ↔ `score/`, and none runs from `workers/ - **Binds:** `vera_cli/`, `generate/`, `judge/` - **Prevents:** either domain owning a cross-domain target definition; generation and judging reimplementing manifest resolution differently; legacy script parsers becoming architectural dependencies. -- **Rule:** [ADOPTED] `vera_cli/targets.py` loads and validates target manifests; each command's `*_config.py` resolver expands the relevant components into canonical inputs. Domain functions receive only their resolved values and never a target manifest. Explicit `--rubric ` consumes the rubric and judging-prompt portion; explicit `--personas ` consumes the personas and persona-prompt portion. Legacy helpers may adapt these inputs during migration but are not dependencies of the unified CLI. +- **Rule:** [ADOPTED] `vera_cli/targets.py` loads and validates target manifests; each `vera_cli/.py` adapter expands the relevant components into canonical inputs. Domain functions receive only their resolved values and never a target manifest. Explicit `--rubric ` consumes the rubric and judging-prompt portion; explicit `--personas ` consumes the personas and persona-prompt portion. Legacy helpers may adapt these inputs during migration but are not dependencies of the unified CLI. ### AD-23 — Output layout is nested-only, with path-first stage contracts @@ -314,13 +314,10 @@ output/ Package tree: ```text -vera.py # CLI orchestrator, thin +vera.py # root parser, command registration, dispatch vera_cli/ - arguments.py # top-level parser config.py, targets.py # shared config and manifest helpers - *_arguments.py # per-command flags and CLI defaults - *_config.py # per-command canonical resolution - *_command.py # thin adapters to domain functions + .py # flags, defaults, resolution, thin adapter generate/ conversation_simulator.py # pure core runner.py # owns I/O, delegates to workers/ diff --git a/docs/architecture.md b/docs/architecture.md index 4d9736164..013fdd141 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -53,12 +53,11 @@ Deep dives: [judge.md](./judge.md) (question flow and rubric navigation), [struc ```text CLI layer -├── vera.py — sole executable; loads arguments and dispatches +├── vera.py — sole executable; builds the root parser and dispatches └── vera_cli/ - ├── arguments.py — top-level parser - ├── *_arguments.py — per-command flags and CLI defaults - ├── *_config.py — per-command canonical resolution - └── *_command.py — thin command adapters + ├── .py — flags, defaults, resolution, and thin adapter + ├── config.py — shared config input helpers + └── targets.py — shared target-manifest resolution ↓ calls Domain packages (generate/, judge/, score/) ↓ register handlers with @@ -71,8 +70,8 @@ Shared utilities (utils/) — leaf layer **Import rules:** -- `vera.py` delegates CLI parsing, resolution, and command adaptation to - `vera_cli/`; it contains no business logic. +- `vera.py` owns only the root parser, explicit subcommand registration, and + dispatch. It contains no command-specific flags, defaults, or business logic. - `vera_cli/` may import domain packages and `utils/`. Domain packages never import `vera_cli/`. - Domain packages (`generate/`, `judge/`, `score/`) do not import each other. @@ -102,18 +101,16 @@ command, and renders CLI errors. Full flag/config reference: ### CLI runtime boundary -The CLI layer has three responsibilities: +The CLI layer has two levels of responsibility: -- `vera.py` is the thin executable and contains no command-specific business - logic. -- `vera_cli/arguments.py` builds the top-level parser. Small per-command - `*_arguments.py` modules define that command's flags and CLI defaults. -- Shared config input and target-manifest helpers stay in focused modules; - per-command `*_config.py` modules enforce input exclusivity and produce the - complete canonical configuration before print, persistence, or dispatch. -- Per-command `*_command.py` modules contain only orchestration adapters. They - receive resolved values and call importable Python functions directly; they - never invoke another CLI parser or subprocess. +- `vera.py` builds the root parser, explicitly registers each supported + subcommand, parses once, and dispatches. It contains no command-specific + flags, defaults, resolution, or business logic. +- One `vera_cli/.py` adapter per subcommand keeps that command's flags, + CLI defaults, canonical resolution, and thin call to the domain function + together. Shared config input and target-manifest mechanics stay in + `vera_cli/config.py` and `vera_cli/targets.py`. Command adapters never invoke + another CLI parser or subprocess. `utils/config_schema.py` owns schema validation and canonical serialization. It does not parse CLI arguments, read config or manifest files, resolve paths, or @@ -127,7 +124,7 @@ rather than to a script entry point. Legacy root scripts may remain temporarily while their replacement feature is migrated. During that transition, `vera_cli` may import the reusable function from a root script, but never its argument parser. For generation, the temporary -flow is `vera_cli.generate_command` → `generate.main`. Removing `generate.py` +flow is `vera_cli.generate` → `generate.main`. Removing `generate.py` and moving that function plus the existing `generate_conversations/` code into the permanent `generate/` package is one atomic later change, so a root `generate.py` module and a top-level `generate/` package never coexist. @@ -276,7 +273,7 @@ orthogonal. | Package / path | Owns | Key modules | |----------------|------|-------------| -| `vera_cli/` | CLI flags/defaults, input resolution, thin command adapters | `arguments.py`, `config.py`, `targets.py`, per-command modules | +| `vera_cli/` | One cohesive adapter per command plus shared config/target helpers | `generate.py`, `judge.py`, `config.py`, `targets.py` | | `generate/` | Simulation, turns, batch runner (pure core; handler owns I/O) | `conversation_simulator.py`, `runner.py` | | `judge/` | Rubric navigation, LLM judge, improvement reporting (pure core; handler owns I/O) | `question_navigator.py`, `llm_judge.py`, `scripts/summarize_results.py` | | `score/` | Aggregation, visualization, pooling — split out of `judge/` | `score.py`, `score_viz.py`, `pool.py` | From bbfb4e18e5b2559b72770eb49e160d0eb7dc8d22 Mon Sep 17 00:00:00 2001 From: Luca Belli <129434630+sator-labs@users.noreply.github.com> Date: Wed, 12 Aug 2026 14:42:32 -0700 Subject: [PATCH 6/6] docs: resolve unified CLI review feedback --- docs/ARCHITECTURE-SPINE.md | 6 +-- docs/architecture.md | 34 ++++++++++------ docs/design/vera-cli-runtime-wiring.md | 2 + docs/vera-cli-use-cases.md | 56 +++++++++++++++++++++++--- 4 files changed, 76 insertions(+), 22 deletions(-) diff --git a/docs/ARCHITECTURE-SPINE.md b/docs/ARCHITECTURE-SPINE.md index 0d466ad06..6c481ba19 100644 --- a/docs/ARCHITECTURE-SPINE.md +++ b/docs/ARCHITECTURE-SPINE.md @@ -185,13 +185,13 @@ No edge runs `generate/` ↔ `judge/` ↔ `score/`, and none runs from `workers/ - **Binds:** `vera.py`, `utils/config_schema.py` - **Prevents:** a merge/override mechanism between two config sources that would make the effective config ambiguous or order-dependent. -- **Rule:** [ADOPTED] For a given run, a piece of information (model selection/repeats, sampling knobs, persona/rubric lists, etc.) is supplied via `--config` JSON or via CLI flags, never both. Supplying the same information through both is rejected/errors — there is no silent merge. Both forms resolve to the same canonical `RunConfig` before print, persistence, or dispatch. CLI-only presentation/execution controls such as `--debug` and `--print` do not supply run configuration and may accompany either form. **`--sample ` is the one deliberate behavior-altering exception:** it MAY be combined with `--config`, since it's a debug-only smoke-test override (UC4) — it never sets information `config.json` itself carries, it only caps how much of the already-resolved persona/rubric/judge lists get used for this invocation, and it's never persisted into the run's own `config.json` artifact (AD-18) or any resumed state. No other run-scoping flag gets this treatment; a future flag needs its own named exception here, not an implicit ride on `--sample`'s. +- **Rule:** [ADOPTED] For a given run, a piece of information (model selection/repeats, sampling knobs, persona/rubric lists, etc.) is supplied via `--config` JSON or via CLI flags, never both. Supplying the same information through both is rejected/errors — there is no silent merge. Both forms resolve to the same canonical `RunConfig` before print, persistence, or dispatch. CLI-only controls such as `--debug`, `--sample`, and `--print` may accompany either form. `--debug` and `--sample ` are recorded as invocation metadata in the run's persisted `config.json`, because they describe how the run actually executed; a resumed sampled run retains that sampled scope. `--print` is not persisted because it exits without creating a run. **`--sample ` is the one deliberate behavior-altering exception:** it caps how much of the already-resolved persona/rubric/judge lists this invocation uses without replacing those canonical lists. No other run-scoping flag gets this treatment; a future flag needs its own named exception here, not an implicit ride on `--sample`'s. ### AD-18 — config.json and state.json are two distinct artifacts - **Binds:** `utils/config_schema.py`, all subcommands that write run output - **Prevents:** one evolving "manifest" file trying to be both the immutable record of what was requested and the mutable record of what happened — which is exactly the resume-correctness bug this split avoids; two independent code paths racing to mutate `state.json` with no defined lock or merge order; the same logical config hashing differently across two implementations and silently defeating the idempotency signal the hash exists to provide. -- **Rule:** [ADOPTED] Every run writes `config.json` (immutable copy of the resolved config, written once at run start, never modified afterward) plus a `config.json.sha256` sidecar (hash lives outside the file it hashes, avoiding self-referential canonicalization). `config.json.sha256` is computed over a canonical serialization of `config.json`'s content — sorted keys, fixed separators, no incidental whitespace — and that content excludes any wall-clock/generation timestamp field, so identical semantic configs (same models, rubrics, personas, knobs) hash identically regardless of when or how many times they are produced. **This is computed exactly once, by exactly one function, and that single value is what both the run-id folder name's `` component (Consistency Conventions, Naming) and the `config.json.sha256` sidecar's content contain — never two independent computations that could drift apart.** The filename `config.json` itself never carries the hash (rejected: would put the same value in a third place with no added integrity benefit, since a corrupted file wouldn't automatically stop matching its own filename — verification still requires hashing the actual bytes, which is what the sidecar is for). `state.json` is a separate, mutable file tracking run progress (completed items, errors, output paths so far) — it is the only file `vera resume` writes to. `state.json` records both the requested and the actual-resolved model identifier. `state.json` has exactly one writer per run: the domain-package runner (`generate/runner.py` / `judge/runner.py`) that owns I/O per AD-3/AD-4. `workers/`'s `JobContext` observes job start/end for logging (AD-13) but never writes `state.json` itself. +- **Rule:** [ADOPTED] Every run writes `config.json` (immutable copy of the resolved config and its invocation metadata, written once at run start, never modified afterward) plus a `config.json.sha256` sidecar (hash lives outside the file it hashes, avoiding self-referential canonicalization). `config.json.sha256` is computed over a canonical serialization of `config.json`'s content — sorted keys, fixed separators, no incidental whitespace — and that content excludes any wall-clock/generation timestamp field, so identical semantic configs and invocation controls (same models, rubrics, personas, knobs, `debug`, and `sample`) hash identically regardless of when or how many times they are produced. **This is computed exactly once, by exactly one function, and that single value is what both the run-id folder name's `` component (Consistency Conventions, Naming) and the `config.json.sha256` sidecar's content contain — never two independent computations that could drift apart.** The filename `config.json` itself never carries the hash (rejected: would put the same value in a third place with no added integrity benefit, since a corrupted file wouldn't automatically stop matching its own filename — verification still requires hashing the actual bytes, which is what the sidecar is for). `state.json` is a separate, mutable file tracking run progress (completed items, errors, output paths so far) — it is the only file `vera resume` writes to. `state.json` records both the requested and the actual-resolved model identifier. `state.json` has exactly one writer per run: the domain-package runner (`generate/runner.py` / `judge/runner.py`) that owns I/O per AD-3/AD-4. `workers/`'s `JobContext` observes job start/end for logging (AD-13) but never writes `state.json` itself. ### AD-19 — generation and judging config blocks are orthogonal @@ -259,7 +259,7 @@ No edge runs `generate/` ↔ `judge/` ↔ `score/`, and none runs from `workers/ | --- | --- | | Naming (entities, files, folders) | Three-entity vocabulary `u`/`c`/`j` (user/persona LLM, chatbot under test, judge) applies at both folder and file level. Run-id = `__` when nested under a per-model parent (model already given by the parent folder), or `___` when flat (nothing else names the model). `` is a generated human-memorable tag (e.g. a word-pair generator) purely so a person can recognize a run without quoting a sha — it carries no identity of its own, is never a substitute for the sha, and never needs to encode which model was used since the surrounding path already does that job. Conversation filename = `u___c_.json`. Generation groups persistently per chatbot (`c_/` accumulates every run against that model); judging stays flat per run (`j____/`, no persistent per-judge-model parent) — an intentional asymmetry, not an inconsistency. Standalone judging (no parent pipeline run to nest under) groups under `evaluations///`, sibling to the `c_*` directories — a persistent, per-config container, exactly mirroring how `c_/` accumulates every run against that model rather than being itself a run-root. The actual run-root inside it is still a freshly-generated `j____/` folder, so re-invoking with an identical config never collides — AD-24 needs no exemption for it, the same as it needs none for `c_/`. The sha in the container name is for discoverability (grouping every run of one exact config together for a human or tool to find), never for deduplication or idempotency — re-running the same config is a new, distinct run, same as generation. | | Data & formats (config shape, hashing, model identifiers) | `config.json` is JSON only, never YAML (robust for stdin/env-var transport with no escaping ambiguity). `generation.user` and `judging.models` are each a **list** of `{name, repeats, }` objects, not an object keyed by model name — so the same model can appear twice with different knobs in one run; `generation` names its two LLM-list fields by entity (`chatbot`, `user`) rather than a shared `models`, since `judging` has only one LLM role and keeps the generic name unambiguously. Every list entry's `name` is always a specific model identifier in the provider's own naming (e.g. `claude-sonnet-2026xxxx`), never a bare provider name. Bespoke sampling knobs (temperature, top_p, max_tokens) are config-only, never expressible via `-u`/`-j` shorthand. Provider connection details (endpoint, API version, region) stay env-sourced only. `config.json.sha256` lives as a sidecar file, never as a field inside `config.json` itself. | -| State & cross-cutting (config/CLI exclusivity, logging, resume) | `--config` and CLI flags are strictly either/or (AD-17) — except `--sample `, a debug-only smoke-test override that MAY combine with `--config`; the resolved form is printed at run start (stdout only) for traceability, with an opt-in `--print` flag to emit it without executing. `vera resume` is the only path that writes to `state.json`, and it first verifies `config.json` against its `.sha256` sidecar before reading either file. Logging is wrapper/context-owned only (AD-13) — never inline in a pure core. Folder-already-exists on run start errors out, no overwrite (AD-24). | +| State & cross-cutting (config/CLI exclusivity, logging, resume) | `--config` and run-defining CLI flags are strictly either/or (AD-17). `--debug`, `--sample`, and `--print` MAY accompany config input; executed runs persist `debug` and `sample` as invocation metadata, while `--print` creates no run. The resolved form is printed at run start for traceability. `vera resume` is the only path that writes to `state.json`, and it first verifies `config.json` against its `.sha256` sidecar before reading either file. Logging is wrapper/context-owned only (AD-13) — never inline in a pure core. Folder-already-exists on run start errors out, no overwrite (AD-24). | ## Stack diff --git a/docs/architecture.md b/docs/architecture.md index 013fdd141..fccd0e9f1 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -144,16 +144,18 @@ the permanent `generate/` package is one atomic later change, so a root Config and run-defining CLI flags are strictly either/or, never combined for the same run — see [vera-cli-use-cases.md](./vera-cli-use-cases.md#config-mechanism). -Debug and presentation controls such as `--sample`, `--debug`, and `--print` are -invocation-only; they are not serialized into `RunConfig`. `-c` selects the -chatbot under test; `-u`/`-j` shorthand selects models/repeats for the user/judge -side respectively; bespoke sampling knobs are config-only. `--target` selects a -complete target, while `--personas` and `--rubric` explicitly select only the -persona or rubric component of a named target. Each component includes its -associated prompt from that target's manifest. `--rubric` remains list-shaped, -though only a length-1 list is supported until Phase 4. `-c` is required for -`generate`/`pipeline` whenever `--config` isn't used — there is no default -chatbot. +Debug and presentation controls such as `--sample`, `--debug`, and `--print` +may accompany either input form. Executed runs record `sample` and `debug` as +invocation metadata in their immutable `config.json`, so it describes how the +run actually executed; `--print` creates no run and is not persisted. `-c` +selects the chatbot under test; `-u`/`-j` shorthand selects models/repeats for +the user/judge side respectively; bespoke sampling knobs are config-only. +`--target` selects a complete target, while `--personas` and `--rubric` +explicitly select only the persona or rubric component of a named target. Each +component includes its associated prompt from that target's manifest. +`--rubric` remains list-shaped, though only a length-1 list is supported until +Phase 4. `-c` is required for `generate`/`pipeline` whenever `--config` isn't +used — there is no default chatbot. Generation behavior defaults are defined only at the CLI flag boundary. A config-driven run provides the corresponding generation fields explicitly; it @@ -219,7 +221,6 @@ project-root/ ← $ROOT (contains vera.py) ├── configs/ │ └── run.json ← config.json └── data/ - ├── personas_a.json └── SI/ ├── manifest.json ← target manifest ├── personas.tsv @@ -229,7 +230,7 @@ project-root/ ← $ROOT (contains vera.py) └── question_prompt.txt ``` -- `configs/run.json`'s `generation.personas: ["data/personas_a.json"]` always means `project-root/data/personas_a.json` — resolved against `$ROOT`, no matter where you run `vera.py` from or where `run.json` itself lives. +- `configs/run.json`'s `generation.personas: ["data/SI/personas.tsv"]` always means `project-root/data/SI/personas.tsv` — resolved against `$ROOT`, no matter where you run `vera.py` from or where `run.json` itself lives. - `data/SI/manifest.json`'s `personas: ["personas.tsv"]` always means `data/SI/personas.tsv` — resolved against the manifest's own folder, so the whole target directory stays portable if copied into a different checkout, independent of `$ROOT`. Same shape of field, same-looking relative string, two different rules — hence calling both out explicitly here rather than leaving it implicit. @@ -245,6 +246,13 @@ loads the whole manifest. `vera generate --target ...` consumes its persona and persona-prompt fields; `vera judge --target ...` consumes its rubric and judging-prompt fields; `vera pipeline --target ...` consumes both. +For standalone judging, `--target SI` and `--rubric SI` therefore resolve to +the same rubric and judging prompts. The difference is wording, not runtime +behavior: `--target` says “select SI's complete target” even though judge uses +only its judging component, while `--rubric` explicitly names that component. +Both forms remain available so a caller can express either intent consistently +with `generate` and `pipeline`. + Advanced callers may keep generation and judging independent. `--personas` `` selects only the personas and persona prompt from that target, while `--rubric ` selects only its rubric and @@ -338,7 +346,7 @@ Agents and contributors must comply. Import boundaries are documented in the [La - **Generation:** conversation simulation logic stays in `generate/`; the simulator core is pure (no filesystem, no logging) — the handler owns all I/O. - **Judging:** rubric navigation and LLM-judge logic stay in `judge/`, also pure-core-plus-handler. Judge never auto-scores — `vera score`/`vera pool` are separate subcommands. **Rubric navigation logic lives in code, never in the prompt:** which question is asked next given an answer is determined entirely by `QuestionNavigator` walking `question_flow_data` parsed from the rubric TSV — the judge LLM answers/judges the current question only, and is never asked to decide or influence what comes next. - **Scoring:** aggregation, visualization, and pooling stay in `score/`, never re-absorbed into `judge/`. -- **Config vs CLI:** `--config` and CLI flags are strictly either/or for a given run — never combined, except `--sample ` (see [Use case 4](./vera-cli-use-cases.md#use-case-4--smoke-test)), a debug-only smoke-test override that never carries information `config.json` itself defines and is never persisted into the run's own `config.json`. `generation` and `judging` blocks in `config.json` are completely orthogonal; model selection for one must never influence the other. +- **Config vs CLI:** `--config` and run-defining CLI flags are strictly either/or for a given run. `--sample `, `--debug`, and `--print` may accompany either form; executed runs record `sample` and `debug` as invocation metadata in their immutable `config.json`, while `--print` creates no run. `generation` and `judging` blocks in `config.json` are completely orthogonal; model selection for one must never influence the other. - **Naming/layout:** all folder/file naming logic (the `c_`/`u_`/`j_` scheme) lives in one `utils/` module — never duplicated across handlers. - **Traceability:** every run writes an immutable `config.json` (+ `.sha256` sidecar) and a separate, mutable `state.json`. `state.json` records both the requested and actual-resolved model identifier. - **LLM providers:** new providers implement [llm_clients/llm_interface.py](../llm_clients/llm_interface.py) and register in [llm_clients/llm_factory.py](../llm_clients/llm_factory.py). Every model-list entry's `name` in config is always a specific model identifier, never a bare provider name. diff --git a/docs/design/vera-cli-runtime-wiring.md b/docs/design/vera-cli-runtime-wiring.md index ac67b272e..8724cc140 100644 --- a/docs/design/vera-cli-runtime-wiring.md +++ b/docs/design/vera-cli-runtime-wiring.md @@ -31,6 +31,8 @@ domain functions. - Generation configs that relied on Python or CLI defaults must add the required generation behavior fields explicitly. CLI invocations retain defaults at the flag-definition boundary. +- Executed runs persist the effective `debug` and `sample` values as invocation + metadata in their immutable `config.json`; `--print` creates no run artifact. - Target manifests are complete bundles of rubric, personas, and prompts. `--target` selects the bundle, while explicit `--personas` and `--rubric` select the named target's persona or rubric component, including that diff --git a/docs/vera-cli-use-cases.md b/docs/vera-cli-use-cases.md index f97631e74..b6d19d0b3 100644 --- a/docs/vera-cli-use-cases.md +++ b/docs/vera-cli-use-cases.md @@ -92,7 +92,7 @@ a supported first-class path, not only a compatibility fallback; it selects the persona files and persona prompt from that target's manifest. Generation behavior is also controlled at this input boundary. CLI invocations -default to `--turns 3`, `--output output`, unlimited concurrency, no total-word +default to `--turns 30`, `--output output`, unlimited concurrency, no total-word cap, persona-first ordering, and one unnamed session. The explicit persona component includes the files and context template resolved from the target named by `--personas`; `--target` resolves the same fields while also selecting the @@ -133,6 +133,17 @@ consumes only SI's rubric and judging prompts and does not select SI's personas or persona prompt. Both flags accept a target name or manifest path, never a bare TSV with implicitly assumed sibling prompts. +For standalone `judge`, these are two ways to word the same effective request: + +```text +--target SI → select SI, from which judge consumes the rubric and judge prompts +--rubric SI → explicitly select SI's rubric and judge prompts +``` + +They resolve to identical judging inputs. The distinction becomes meaningful +for `pipeline`, where `--target` also supplies the generation personas and +persona prompt, while `--rubric` supplies only the judging component. + **Resolved (multi-folder judge output):** judge keeps results independent per folder; the `score/` layer aggregates across folders when needed, not judge itself. There are also in-between options — e.g. kept separate, but the score layer aggregates them. **Resolved (multi-rubric output layout):** generated conversations for a run all live together in one unified folder regardless of which rubrics will later judge them — generation has no knowledge of rubrics. Judging output IS separated per rubric, since different rubrics produce different, non-comparable scores. (The unified-conversations choice is soft, not a hard invariant — may change if a concrete need for per-rubric conversation grouping emerges.) @@ -145,9 +156,17 @@ Run the full pipeline (or generate/judge alone) against a small sample of person vera pipeline --config run.json --sample 2 ``` -`--sample N` overrides the config's full persona (and rubric/judge, where relevant) list at run time. This avoids hand-maintaining a separate small-scale config just for smoke testing. +`--sample N` caps the config's full persona (and rubric/judge, where relevant) +list at run time. This avoids hand-maintaining a separate small-scale config +just for smoke testing. -**`--sample` is the sole, named exception to the CLI/`--config` either-or rule** (AD-17 in [ARCHITECTURE-SPINE.md](./ARCHITECTURE-SPINE.md)): it's a debug-only cap on how much of the config's already-resolved lists get used, never a way to set information `config.json` itself carries, and it's never written into the run's own `config.json` artifact -- `vera resume` on a sampled run still resolves against the full original config. No other flag gets this treatment. +**`--sample` is the sole behavior-altering exception to the CLI/`--config` +either-or rule** (AD-17 in +[ARCHITECTURE-SPINE.md](./ARCHITECTURE-SPINE.md)): it caps how much of the +already-resolved lists get used rather than replacing those lists. The executed +value is recorded in the run's persisted `config.json` invocation metadata, so +`vera resume` retains the same sampled scope. `--debug` is recorded alongside +it; `--print` creates no run and therefore has nothing to persist. ## Use case 5 — Pool @@ -174,25 +193,50 @@ Reads the immutable `config.json` (verifying its `.sha256` sidecar first) plus t - `--config ` — JSON file, for local use. - `--config -` — read JSON from stdin. - `VERA_RUN_CONFIG` env var — inline JSON content, for remote/CI dispatch where uploading or mounting a file isn't convenient. -- **CLI flags and `--config` are strictly either/or, never combined for the same run.** A given piece of information (model selection/repeats, sampling knobs, persona/rubric lists) is supplied via one or the other, never both — the implementation rejects the combination rather than silently merging. **`--sample ` (see [Use case 4](#use-case-4--smoke-test)) is the one deliberate exception:** it MAY be passed alongside `--config`, since it never carries information `config.json` defines -- it only caps how much of the already-resolved lists get used for that invocation, purely for debugging, and it's never written into the run's own `config.json` artifact. +- **Run-defining CLI flags and `--config` are strictly either/or, never combined for the same run.** A given piece of information (model selection/repeats, sampling knobs, persona/rubric lists) is supplied via one or the other, never both — the implementation rejects the combination rather than silently merging. Invocation controls `--sample`, `--debug`, and `--print` MAY accompany config input. Executed runs persist `sample` and `debug` under `invocation` in their immutable `config.json`; `--print` exits without creating a run. - Internally, `--config` always resolves to the same canonical flag-set the CLI would produce, so there is exactly one resolved form regardless of input path. The tool prints this resolved form at run start for terminal/CI-log visibility (it does not write to the shell's own history — an opt-in `--print` flag emits the resolved flag-string with no execution, for a caller who wants to `eval` it into their own shell explicitly). - JSON, not YAML — robust when passed as a one-line env var or stdin payload with no escaping ambiguity. - **Path fields inside `config.json` (`generation.personas`, etc.) resolve relative to `$ROOT`** — the directory containing `vera.py` — never relative to the current working directory or the config file. A [target manifest](./architecture.md#target-manifest) deliberately resolves its fields relative to its own directory so the complete target remains portable. +For example, given `configs/run.json` and `data/SI/manifest.json`: + +```text +configs/run.json: "personas": ["data/SI/personas.tsv"] + └─ resolves from $ROOT + +data/SI/manifest.json: "personas": ["personas.tsv"] + └─ resolves from data/SI/ + +Both resolve to: $ROOT/data/SI/personas.tsv +``` + +The different anchors let a run config remain checkout-relative while a target +directory remains portable as a unit. + ### `config.json` shape +The persisted run artifact includes generated `invocation` metadata in addition +to the resolved generation and judging inputs. A fresh input config does not +need to author this block: the CLI records the effective `--debug` and +`--sample` values when it creates the run. `vera resume` reads those persisted +values and retains the same invocation scope. + Top-level `generation` and `judging` blocks are **completely orthogonal** — model selection for one must never influence or be influenced by the other. Model-list fields (a list, not an object keyed by name, so the same model can appear twice with different knobs) are named per entity rather than a bare `models`, since `generation` has two LLM roles (`chatbot`, `user`) competing for that name; `judging` keeps `models` since only one LLM role exists there: ```json { + "invocation": { + "debug": false, + "sample": null + }, "generation": { "chatbot": {"name": "claude-sonnet-2026xxxx", "repeats": 1}, "user": [ {"name": "claude-sonnet-2026xxxx", "repeats": 1, "temperature": 0.7}, {"name": "gpt-5", "repeats": 2} ], - "personas": ["data/personas_a.json", "data/personas_b.json"], - "turns": 3, + "personas": ["data/SI/personas.tsv"], + "turns": 30, "output": "output", "max_concurrent": null, "max_total_words": null,