Docs: settle the vera judge command plan

docs/ARCHITECTURE-SPINE.md

@@ -221,7 +221,7 @@ No edge runs `generate/` ↔ `judge/` ↔ `score/`, and none runs from `workers/
 
 - **Binds:** `generate/`, `judge/`, `score/`, `utils/naming.py`
 - **Prevents:** stage code depending on a flat-layout path shape that no longer exists; resume logic re-deriving state from scratch instead of reading it.
-- **Rule:** [ADOPTED] The flat output layout is removed for *new writes*; only the nested layout (`output/c_<chatbot>/<run>/conversations/`, `.../evaluations/<rubric>/j_<run>/...`) is valid for a run created going forward. This does not remove the ability to *read* existing flat-layout data: `vera judge --conversations <folder>` and `vera score -r <results.csv>` take an explicit path and read files by their own format, never by re-deriving meaning from the parent folder's naming pattern, so both keep working against old-layout output. `vera resume` is the one path that genuinely can't operate on an old run, since `config.json`/`.sha256`/`state.json` didn't exist under the old layout — there is nothing for it to read, independent of naming scheme. Stage handoffs are path-first contracts, recording models/personas/artifact paths/timestamps, usable as input for resume or a new run.
+- **Rule:** [ADOPTED] The flat output layout is removed for *new writes*; only the nested layout (`output/c_<chatbot>/<run>/conversations/`, `.../evaluations/<target>/j_<run>/...`) is valid for a run created going forward. This does not remove the ability to *read* existing flat-layout data: `vera judge --conversations <folder>` and `vera score -r <results.csv>` take an explicit path and read files by their own format, never by re-deriving meaning from the parent folder's naming pattern, so both keep working against old-layout output. `vera resume` is the one path that genuinely can't operate on an old run, since `config.json`/`.sha256`/`state.json` didn't exist under the old layout — there is nothing for it to read, independent of naming scheme. Stage handoffs are path-first contracts, recording models/personas/artifact paths/timestamps, usable as input for resume or a new run.
 
 ### AD-24 — Existing run-output collisions error out
 

docs/architecture.md

@@ -25,7 +25,7 @@ All user-facing operations go through **`vera.py`** subcommands. Domain packages
 data/<target>/manifest.json ──► generate ──► c_<chatbot>/<run>/conversations/*.json
                                         │
                                         ▼
-                                   judge ──► c_<chatbot>/<run>/evaluations/<rubric>/j_*/results.csv
+                                   judge ──► c_<chatbot>/<run>/evaluations/<target>/j_*/results.csv
                                         │
                                         ▼
                                    score ──► .../j_*/scores/
@@ -148,7 +148,7 @@ models identically.
 | Subcommand | Delegates to | Purpose |
 |------------|--------------|---------|
 | `vera generate` | generation application function (temporarily `generate.run_for_user_models`) | Simulate conversations → `c_<chatbot>/<run>/conversations/` |
-| `vera judge` | `judge.runner` | Evaluate transcripts → `evaluations/<rubric>/j_*` |
+| `vera judge` | `judge.runner` | Evaluate transcripts → `evaluations/<target>/j_*` |
 | `vera score` | `score.score` | Aggregate `results.csv` → scores and visualizations |
 | `vera pool` | `score.pool` | Concatenate multiple evaluation folders into one pooled result |
 | `vera pipeline` | orchestration layer | Full workflow for one chatbot; passes paths between steps |
@@ -446,10 +446,10 @@ uv run pytest -m "not live"
 | **0 — De-risk multi-rubric** | Prove the target-manifest format on both generation and judging before the unified CLI replaces legacy scripts | Treat every manifest as one complete target containing a rubric, personas, and both prompt sets. Legacy judge code may consume only its rubric fields and legacy generation code only its persona fields, but both read the same complete manifest. Preserve the existing explicit persona/rubric paths so Phase 1 can offer both whole-target and component-level selection | `pytest -m "not live"` green; one complete fixture target drives both legacy generation and judging; incomplete manifests fail validation; explicit persona/rubric selection remains covered |
 | **S — Storage abstraction** *(orthogonal — see note above)* | Decouple "what path/key to use" from "how to persist bytes," so a future non-local backend (S3, etc.) is a new implementation, not a rewrite | New `storage/` package: `StorageBackend` (ABC) with `write(key, bytes)` / `read(key)` / `exists(key)`, plus `LocalFilesystemStorage` as the default implementation — mirrors the `LLMInterface`/`QueueProtocol` idiom (interface and implementations live together in one concern-scoped package). The backend knows nothing about run semantics; `utils/naming.py` still builds keys/paths, `storage/` just persists what it's given. All domain/`workers/` code that currently touches the filesystem directly switches to calling through `StorageBackend` instead | `pytest -m "not live"` green; no domain or `workers/` code calls `open()`/`pathlib` file-write directly for run artifacts — everything routes through `StorageBackend`; `LocalFilesystemStorage` is behaviorally identical to today's direct-filesystem writes |
 | **O — Adopt OpenSpec** *(orthogonal — see note above)* | Turn "consider an OpenSpec change if the team adopts that workflow" from a maybe into an actual, exercised requirement | Populate `openspec/changes/` with a real OpenSpec change document the next time a large multi-file feature lands (the existing ESCALATE trigger: new judge dimensions, pipeline CLI changes). Currently `openspec/` is empty scaffolding — this phase is "done" only once a real change has actually gone through it, not just once the config exists | A qualifying multi-file feature has shipped with a real OpenSpec change document under `openspec/changes/`, and the ESCALATE section's language is updated from "if the team adopts" to a firm MUST for future qualifying changes |
-| **1 — New CLI + config** | `vera.py` fully replaces the top-level scripts | Add whole-target selection through `--target` and top-level config `target`. Preserve explicit component selection through `--personas <target>` and `--rubric <target>`, so callers can combine the persona side of one target with the rubric side of another. Resolve target manifests before print or dispatch, then call existing domain behavior directly. Ship `-u`/`-j`/`--sample` and the informal config shape; delete `generate.py`/`judge.py`/`run_pipeline.py` at the end of the phase | `pytest -m "not live"` green; `vera.py` is the only documented entry point; target and explicit-component paths have structural parity tests; `--config`, `--target`, `--personas`, `--rubric`, `-u`, `-j`, and `--sample` are functional; legacy root scripts are gone |
+| **1 — New CLI + config** | `vera.py` fully replaces the top-level scripts | Add whole-target selection through `--target` and top-level config `target`. Preserve explicit component selection through `--personas <target>` and `--rubric <target>`, so callers can combine the persona side of one target with the rubric side of another. Resolve target manifests before print or dispatch, then call existing domain behavior directly. Ship `-u`/`-j`/`--sample` and the informal config shape; delete `generate.py`/`run_pipeline.py` at the end of the phase. **`judge.py` is the one exception and outlives this phase:** `vera judge` ships without `--resume`, because the resume contract is deferred (see the Deferred resume contract note above), so `judge.py` is retained *solely* as the resume entry point until `vera resume` exists. It is not a general escape hatch — no new work targets it, and it is deleted the moment `vera resume` lands. Judging output also keeps the existing `<gen_run>/evaluations/j_*` layout in this phase; Phase 3 renames it and Phase 4 adds the `<target>/` segment | `pytest -m "not live"` green; `vera.py` is the only documented entry point for everything except resume; target and explicit-component paths have structural parity tests; `--config`, `--target`, `--personas`, `--rubric`, `-u`, `-j`, and `--sample` are functional; `generate.py` and `run_pipeline.py` are gone |
 | **2 — Scoring split** | Extract `score/` | `score.py`/`score_viz.py`/`pool.py` move out of `judge/` into `score/`; pure move, no new behavior. `vera.py` (the only entry point since Phase 1) gets its imports updated directly — no shim needed, since the legacy root scripts no longer exist. A minimal import-linter contract is added covering only the `judge/` ⊥ `score/` boundary this phase creates | `pytest -m "not live"` green; the `judge/` ⊥ `score/` import-linter contract passes; no code imports `judge.score`/`judge.pool` |
 | **3 — Traceability & naming** | Harden Phase 1's config shape; add persistence; swap in the new naming scheme | `utils/config_schema.py` formalizes Phase 1's informal `config.json` shape into a stable interface (design doc required for future changes) — `judging.rubrics` stays a list (continuing the list-from-day-one approach already used since Phase 0/1), still length-1-only in practice until Phase 4. Adds the persisted artifacts: `config.json` written to disk + `state.json` + `.sha256` sidecar; `utils/naming.py` (the naming/layout module, already tracked today implementing the legacy `p_`/`a_` scheme) is **rewritten** for the `c_`/`u_`/`j_` scheme, retiring the `p_*`/`j_*` layout Phase 1 kept. Existing `output/` run folders under the old layout are left alone — no migration script, only new runs use the new layout. **Acknowledged compatibility break:** anything outside `vera.py` that parses the old `p_*`/`j_*` pattern directly (`spring_scripts/`, `distribute_files.py`, `score_comparison.py`, notebooks, human-review tooling) breaks the moment new runs use `c_*`/`u_*`/`j_*` instead — accepted, since the vast majority of real usage goes through the CLI, not direct path-parsing. **This break is about auto-discovery, not about reading old data at all:** `vera judge --conversations <old p_* folder>` and `vera score -r <old results.csv>` keep working against existing old-layout output, since both take an explicit path and read files by their own format, never by re-deriving meaning from the parent folder's naming pattern. What genuinely doesn't carry over is `vera resume` on an old run — `config.json`/`.sha256`/`state.json` didn't exist under the old layout, so there's nothing for `resume` to read regardless of naming scheme. Import-linter contract extended to cover `utils/` as a leaf | `pytest -m "not live"` green; a run's `config.json` round-trips through `vera resume`; `utils/` leaf-layer import-linter contract passes |
-| **4 — Multi-rubric support** | Support multiple rubrics per run | Built on Phase 3's `config.json`/naming: `judging.rubrics[]` now supports length > 1 (the list shape has existed since Phase 0 — this phase lifts the length-1 restriction, it doesn't introduce the list); per-rubric judge-model overrides; per-rubric `evaluations/<rubric>/` folder separation | `pytest -m "not live"` green; a config with 2+ rubrics produces separated `evaluations/<rubric>/` output for each; **and** a pre-existing length-1 `judging.rubrics` config from Phase 1-3 still produces identical behavior — backward compatibility with the single-rubric case is verified, not just the new multi-rubric case |
+| **4 — Multi-rubric support** | Support multiple rubrics per run | Built on Phase 3's `config.json`/naming: `judging.rubrics[]` now supports length > 1 (the list shape has existed since Phase 0 — this phase lifts the length-1 restriction, it doesn't introduce the list); per-rubric judge-model overrides; per-rubric `evaluations/<target>/` folder separation, one folder per rubric-providing target (the segment is named for the target the rubric came from, not for the rubric file, so it matches `c_<chatbot>`; with `--rubric` that may differ from the run's own target). Also enables `vera judge --target all`, deferred from Phase 1 because judging output could not be attributed to a rubric before this segment existed | `pytest -m "not live"` green; a config with 2+ rubrics produces separated `evaluations/<target>/` output for each; **and** a pre-existing length-1 `judging.rubrics` config from Phase 1-3 still produces identical behavior — backward compatibility with the single-rubric case is verified, not just the new multi-rubric case; `vera judge --target all` no longer errors and its output is attributable per target |
 
 **Adding a new target or rubric only requires Phase 0-4, not Phase 5.** A complete target is added as `data/<target>/manifest.json`; an expert may still select its rubric component explicitly with `--rubric`. Phase 5 replaces the generation/judging *engine* underneath and does not change target or rubric definition.
 | **5 — Substantial refactoring** | Everything else | `workers/` unification (both `generate/runner.py` and `judge/runner.py` already hand-roll the same asyncio-queue worker-pool pattern independently — concrete evidence for this step); `llm_clients/` plugin-registry formalization; import-linter contract completed (all remaining boundaries) + grimp; quality-gate tightening (pyright blocking, coverage 30%→60%); root-clutter cleanup (see below). **This is the phase that actually rewrites the generation/judging engine** (both runners move off their own hand-rolled asyncio queues onto the shared `workers/` pool) — see the engine-testing note below, this needs more care than any structural CLI change. **Check whether `scripts/pool_vera_scores.py` is still needed** at this point — if so, fold its logic into `score/pool.py` and remove or thin the script; if `vera pool` has already fully superseded it by now, just delete it. **Provider concurrency limits, not yet addressed anywhere in this doc:** the shared `workers/` pool can fan out many parallel jobs (e.g. `-u gpt:5 sonnet:5`); `llm_clients/` already has per-call retry/backoff but nothing caps per-provider *concurrency*, so real parallelism at this phase could blow through a provider's rate limit for the first time. Needs a concurrency cap per provider before this phase is considered done, not just noted as a known gap | `pytest -m "not live"` green; pyright blocking in CI; coverage ≥ 60%; full import-linter contract (all layer boundaries from the [Layer model](#layer-model) section) passes; no root-level Python entry points except `vera.py`; all four testing tiers described below (structural parity, live-LLM smoke test, semantic similarity check, manual spot-check), specifically for the `workers/` migration; a per-provider concurrency cap exists and is exercised by at least one test that fans out more jobs than the cap allows |

docs/vera-cli-use-cases.md

@@ -21,6 +21,11 @@ points in the process of being removed: each is deleted once its replacement
 `vera.py` command ships, and they exist only to keep unmigrated workflows
 running in the meantime. Do not build on them.
 
+`judge.py` is the one that lingers. `vera judge` ships without `--resume` because
+the resume contract is deferred, so `judge.py` stays as the *only* way to resume
+an evaluation until `vera resume` exists — and for nothing else. It is deleted
+when `vera resume` lands.
+
 They already use `-c`/`-r` for unrelated things (`-c` is `--max-concurrent` in
 `generate.py` and `--conversation` in `judge.py`; `-r` is `--runs` in
 `generate.py` and `--rubrics` in `judge.py`). `vera.py` intentionally repurposes
@@ -41,7 +46,7 @@ may state the resulting concrete paths directly.
 | Subcommand | CLI shorthand — minimum required | Input `config.json` — minimum required fields |
 |---|---|---|
 | `generate` | `-c <chatbot>` + `-u <model:repeats...>` (≥1) + (`--target <name-or-path>` **or** explicit `--personas <name-or-manifest-path>`) | `generation.chatbot`, `generation.user` (≥1), and (top-level `target` **or** explicit `generation.personas` + `generation.persona_context_template`) |
-| `judge` | `-j <model:repeats...>` (≥1) + `--conversations <folder...>` + (`--target <name-or-path>` **or** explicit `--rubric <name-or-manifest-path>`) | `judging.models` (≥1), `judging.conversations` (≥1), and (top-level `target` **or** explicit `judging.rubrics` (≥1)) |
+| `judge` | `-j <model:repeats...>` (≥1) + `--conversations <folder>` (exactly 1 — see below) + (`--target <name-or-path>` **or** explicit `--rubric <name-or-manifest-path>`) | `judging.models` (≥1), `judging.conversations` (list, length 1), and (top-level `target` **or** explicit `judging.rubrics` (≥1)) |
 | `pipeline` | everything `generate` needs **and** everything `judge` needs — or just `-c`, `-u`, `-j`, `--target <name>` (`--target` covers persona+rubric in one shot) | both `generation` and `judging` blocks fully populated, or `-c`/`-u`/`-j`-equivalent fields plus a top-level `target`; `judging.conversations` is omitted because generation supplies it |
 | `score` | the results path (`-r <results.csv>`) — nothing else | n/a — `score` reads an existing `results.csv`, not a run config |
 | `pool` | `--evaluations <folder...>` (≥1) | n/a |
@@ -81,6 +86,30 @@ available when the caller deliberately wants different components.
 `--target all` deliberately resolves every discovered target manifest as a
 separate invocation. It never merges personas or prompts from different targets.
 
+**`--conversations` takes exactly one folder.** This preserves today's
+`judge.py --folder` behavior, which accepts a single conversation run folder
+(nested `p_*__/conversations/`, or a legacy flat folder). The flag name and the
+`judging.conversations` config field are nonetheless **list-shaped from day one**,
+following the same reasoning as [AD-20](./ARCHITECTURE-SPINE.md#ad-20--judgingrubrics-is-a-list-from-day-one)
+for `judging.rubrics`: only a length-1 list is accepted or validated, but locking
+a scalar shape now would force a breaking config change if multi-folder judging
+ever ships. More than one folder is a clear error, not a silent truncation. Note
+that judging several folders is already expressible today — judge each, then
+combine with `vera pool`.
+
+**`vera judge --target all` is deferred, not disallowed.** It errors for now and
+is scheduled for Phase 4; `--target all` keeps its full meaning for `generate`
+throughout.
+
+The blocker is output attribution, not semantics. Judging every target means
+evaluating the same conversations under N rubrics, which resolves cleanly to N
+separate length-1 runs — but until Phase 4 adds the `evaluations/<target>/`
+segment, all N land in `<gen_run>/evaluations/j_*` distinguishable only by
+timestamp, because the judge run folder encodes judge model and time, not the
+rubric that produced it. Erroring until the path can attribute the result is
+preferable to writing output nobody can later tell apart. Phase 4 lifts this
+alongside multi-rubric support, and the error message should say so.
+
 ## Use case 2 — Batch generate across personas
 
 One chatbot under test, generated against multiple personas, each carrying its own user-side LLM.