Feat: add unified vera judge command

CHANGELOG.md

@@ -4,6 +4,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 ## [Unreleased]
 
+### Breaking / migration
+
+- **`vera judge` requires an explicit output location for flat conversation folders** — Legacy [`judge.py`](judge.py) falls back to writing evaluations into `evaluations/` *relative to the working directory* when `--folder` points at a flat folder of `.txt` transcripts rather than a generation run. `vera judge` does not carry that fallback: it errors and asks for `-o/--output`. The default for a generation run is unchanged and still lands beside the transcripts, at `<conversation run>/evaluations/`. Reading old flat-layout conversations continues to work — pass `-o` to say where the results go. The fallback was dropped because it detached evaluations from the conversations that produced them, and because the same relative path means different directories depending on the input form (CLI paths resolve against the working directory, config paths against the repository root). Legacy `judge.py` keeps the old behavior until it is removed.
+
 ## [v1.2.0](https://github.com/SpringCare/VERA-MH/releases/tag/v1.2.0) \- 2026-07-16
 
 ### Breaking / migration

docs/architecture.md

@@ -446,7 +446,7 @@ uv run pytest -m "not live"
 | **0 — De-risk multi-rubric** | Prove the target-manifest format on both generation and judging before the unified CLI replaces legacy scripts | Treat every manifest as one complete target containing a rubric, personas, and both prompt sets. Legacy judge code may consume only its rubric fields and legacy generation code only its persona fields, but both read the same complete manifest. Preserve the existing explicit persona/rubric paths so Phase 1 can offer both whole-target and component-level selection | `pytest -m "not live"` green; one complete fixture target drives both legacy generation and judging; incomplete manifests fail validation; explicit persona/rubric selection remains covered |
 | **S — Storage abstraction** *(orthogonal — see note above)* | Decouple "what path/key to use" from "how to persist bytes," so a future non-local backend (S3, etc.) is a new implementation, not a rewrite | New `storage/` package: `StorageBackend` (ABC) with `write(key, bytes)` / `read(key)` / `exists(key)`, plus `LocalFilesystemStorage` as the default implementation — mirrors the `LLMInterface`/`QueueProtocol` idiom (interface and implementations live together in one concern-scoped package). The backend knows nothing about run semantics; `utils/naming.py` still builds keys/paths, `storage/` just persists what it's given. All domain/`workers/` code that currently touches the filesystem directly switches to calling through `StorageBackend` instead | `pytest -m "not live"` green; no domain or `workers/` code calls `open()`/`pathlib` file-write directly for run artifacts — everything routes through `StorageBackend`; `LocalFilesystemStorage` is behaviorally identical to today's direct-filesystem writes |
 | **O — Adopt OpenSpec** *(orthogonal — see note above)* | Turn "consider an OpenSpec change if the team adopts that workflow" from a maybe into an actual, exercised requirement | Populate `openspec/changes/` with a real OpenSpec change document the next time a large multi-file feature lands (the existing ESCALATE trigger: new judge dimensions, pipeline CLI changes). Currently `openspec/` is empty scaffolding — this phase is "done" only once a real change has actually gone through it, not just once the config exists | A qualifying multi-file feature has shipped with a real OpenSpec change document under `openspec/changes/`, and the ESCALATE section's language is updated from "if the team adopts" to a firm MUST for future qualifying changes |
-| **1 — New CLI + config** | `vera.py` fully replaces the top-level scripts | Add whole-target selection through `--target` and top-level config `target`. Preserve explicit component selection through `--personas <target>` and `--rubric <target>`, so callers can combine the persona side of one target with the rubric side of another. Resolve target manifests before print or dispatch, then call existing domain behavior directly. Ship `-u`/`-j`/`--sample` and the informal config shape; delete `generate.py`/`run_pipeline.py` at the end of the phase. **`judge.py` is the one exception and outlives this phase:** `vera judge` ships without `--resume`, because the resume contract is deferred (see the Deferred resume contract note above), so `judge.py` is retained *solely* as the resume entry point until `vera resume` exists. It is not a general escape hatch — no new work targets it, and it is deleted the moment `vera resume` lands. Judging output also keeps the existing `<gen_run>/evaluations/j_*` layout in this phase; Phase 3 renames it and Phase 4 adds the `<target>/` segment | `pytest -m "not live"` green; `vera.py` is the only documented entry point for everything except resume; target and explicit-component paths have structural parity tests; `--config`, `--target`, `--personas`, `--rubric`, `-u`, `-j`, and `--sample` are functional; `generate.py` and `run_pipeline.py` are gone |
+| **1 — New CLI + config** | `vera.py` fully replaces the top-level scripts | Add whole-target selection through `--target` and top-level config `target`. Preserve explicit component selection through `--personas <target>` and `--rubric <target>`, so callers can combine the persona side of one target with the rubric side of another. Resolve target manifests before print or dispatch, then call existing domain behavior directly. Ship `-u`/`-j`/`--sample` and the informal config shape; delete `generate.py`/`run_pipeline.py` at the end of the phase. **`judge.py` is the one exception and outlives this phase:** `vera judge` ships without `--resume`, because the resume contract is deferred (see the Deferred resume contract note above), so `judge.py` is retained *solely* as the resume entry point until `vera resume` exists. It is not a general escape hatch — no new work targets it, and it is deleted the moment `vera resume` lands. Judging output also keeps the existing `<gen_run>/evaluations/j_*` layout in this phase; Phase 3 renames it and Phase 4 adds the `<target>/` segment. **Acknowledged compatibility break:** `vera judge` drops legacy `judge.py`'s fallback of writing evaluations to `evaluations/` relative to the working directory when the input is a flat transcript folder rather than a generation run — it errors and requires `-o/--output` instead. Accepted because that fallback detached evaluations from the conversations that produced them, and because a bare relative path resolves against the working directory on the CLI but against the repository root in a config. Reading old flat-layout conversations still works with an explicit `-o`, satisfying the read-old-data guarantee above | `pytest -m "not live"` green; `vera.py` is the only documented entry point for everything except resume; target and explicit-component paths have structural parity tests; `--config`, `--target`, `--personas`, `--rubric`, `-u`, `-j`, and `--sample` are functional; `generate.py` and `run_pipeline.py` are gone |
 | **2 — Scoring split** | Extract `score/` | `score.py`/`score_viz.py`/`pool.py` move out of `judge/` into `score/`; pure move, no new behavior. `vera.py` (the only entry point since Phase 1) gets its imports updated directly — no shim needed, since the legacy root scripts no longer exist. A minimal import-linter contract is added covering only the `judge/` ⊥ `score/` boundary this phase creates | `pytest -m "not live"` green; the `judge/` ⊥ `score/` import-linter contract passes; no code imports `judge.score`/`judge.pool` |
 | **3 — Traceability & naming** | Harden Phase 1's config shape; add persistence; swap in the new naming scheme | `utils/config_schema.py` formalizes Phase 1's informal `config.json` shape into a stable interface (design doc required for future changes) — `judging.rubrics` stays a list (continuing the list-from-day-one approach already used since Phase 0/1), still length-1-only in practice until Phase 4. Adds the persisted artifacts: `config.json` written to disk + `state.json` + `.sha256` sidecar; `utils/naming.py` (the naming/layout module, already tracked today implementing the legacy `p_`/`a_` scheme) is **rewritten** for the `c_`/`u_`/`j_` scheme, retiring the `p_*`/`j_*` layout Phase 1 kept. Existing `output/` run folders under the old layout are left alone — no migration script, only new runs use the new layout. **Acknowledged compatibility break:** anything outside `vera.py` that parses the old `p_*`/`j_*` pattern directly (`spring_scripts/`, `distribute_files.py`, `score_comparison.py`, notebooks, human-review tooling) breaks the moment new runs use `c_*`/`u_*`/`j_*` instead — accepted, since the vast majority of real usage goes through the CLI, not direct path-parsing. **This break is about auto-discovery, not about reading old data at all:** `vera judge --conversations <old p_* folder>` and `vera score -r <old results.csv>` keep working against existing old-layout output, since both take an explicit path and read files by their own format, never by re-deriving meaning from the parent folder's naming pattern. What genuinely doesn't carry over is `vera resume` on an old run — `config.json`/`.sha256`/`state.json` didn't exist under the old layout, so there's nothing for `resume` to read regardless of naming scheme. Import-linter contract extended to cover `utils/` as a leaf | `pytest -m "not live"` green; a run's `config.json` round-trips through `vera resume`; `utils/` leaf-layer import-linter contract passes |
 | **4 — Multi-rubric support** | Support multiple rubrics per run | Built on Phase 3's `config.json`/naming: `judging.rubrics[]` now supports length > 1 (the list shape has existed since Phase 0 — this phase lifts the length-1 restriction, it doesn't introduce the list); per-rubric judge-model overrides; per-rubric `evaluations/<target>/` folder separation, one folder per rubric-providing target (the segment is named for the target the rubric came from, not for the rubric file, so it matches `c_<chatbot>`; with `--rubric` that may differ from the run's own target). Also enables `vera judge --target all`, deferred from Phase 1 because judging output could not be attributed to a rubric before this segment existed | `pytest -m "not live"` green; a config with 2+ rubrics produces separated `evaluations/<target>/` output for each; **and** a pre-existing length-1 `judging.rubrics` config from Phase 1-3 still produces identical behavior — backward compatibility with the single-rubric case is verified, not just the new multi-rubric case; `vera judge --target all` no longer errors and its output is attributable per target |