From 94c06819391890fa2ecd3975b2f438f709d4f591 Mon Sep 17 00:00:00 2001 From: Jiri Puc Date: Thu, 9 Jul 2026 23:58:24 +0200 Subject: [PATCH 1/7] Implementation evals: 53 eval-banana harness_judge design-conformance checks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit plan/implementation_evals/ — acceptance checks that judge the final implementation against the binding design (D1-D61 + requirements): one YAML per check, self-contained instructions citing the binding sources, conjunctive conditions, binary verdicts. Areas: E0 documents (8), E1 blocks/chunks (2), E2 claims (9), E3 relations/observations (5), entity resolution/review (3), K plane (4), projections (7), retrieval (6), ops (6), code/boundary (3). All 53 validated against the eval-banana check schema (unique ids, harness_judge, schema v1). questions.md §5: register two inconsistencies found while authoring — the requirements temporal-split paragraph predates D43/D49 (#30), and the requirements E3 bullet omits observations (#31). Co-Authored-By: Claude Fable 5 --- plan/implementation_evals/README.md | 71 +++++++++++++++++++ .../boundary_library_scope_d60.yaml | 21 ++++++ .../boundary_provider_ports_d61.yaml | 27 +++++++ .../code_config_via_pydantic_settings.yaml | 24 +++++++ .../e0_blockizer_owns_identity.yaml | 25 +++++++ .../e0_content_addressed_reuse.yaml | 27 +++++++ .../e0_converter_router_versioned.yaml | 21 ++++++ .../e0_lineages_versions_modes.yaml | 31 ++++++++ .../e0_origin_stamped_at_ingest.yaml | 19 +++++ .../e0_postgres_holds_no_bodies.yaml | 21 ++++++ .../e0_raw_immutable_id_addressed.yaml | 24 +++++++ .../e0_structure_contract_unconditional.yaml | 24 +++++++ .../e1_chunks_whole_blocks_no_overlap.yaml | 25 +++++++ .../e1_multigranularity_retrieval.yaml | 21 ++++++ .../e2_asserted_validity_immutable.yaml | 24 +++++++ .../e2_claims_immutable_append_only.yaml | 24 +++++++ .../e2_context_bundle_two_calls.yaml | 26 +++++++ .../e2_grounding_layered_dual_field.yaml | 28 ++++++++ .../e2_no_pre_extraction_gate.yaml | 25 +++++++ .../e2_selection_ledger_replay.yaml | 21 ++++++ .../e2_selection_recall_envelope.yaml | 23 ++++++ .../e2_stance_kept_as_observation.yaml | 27 +++++++ .../e2_testimony_currency_counting.yaml | 33 +++++++++ .../e3_blocking_cheap_first_cascade.yaml | 22 ++++++ .../e3_observations_untyped_adjudicated.yaml | 30 ++++++++ .../e3_predicate_registry_governed.yaml | 27 +++++++ .../e3_relations_evidence_collapse.yaml | 21 ++++++ ...e3_supersession_fact_level_bitemporal.yaml | 22 ++++++ .../eval_checks/er_merges_reversible.yaml | 24 +++++++ .../er_resolution_cascade_t0_t4.yaml | 25 +++++++ .../eval_checks/er_review_queue_cli.yaml | 21 ++++++ .../eval_checks/k_citations_binding.yaml | 23 ++++++ .../k_compiled_vs_authored_ownership.yaml | 24 +++++++ .../eval_checks/k_one_mechanism_n_scopes.yaml | 27 +++++++ .../eval_checks/k_planner_writer_driver.yaml | 28 ++++++++ .../ops_deletion_cascade_grains.yaml | 25 +++++++ .../ops_execution_classes_bound.yaml | 25 +++++++ .../eval_checks/ops_idempotency_dlq.yaml | 21 ++++++ .../ops_metering_budgets_enforced.yaml | 21 ++++++ .../ops_producer_checker_families.yaml | 21 ++++++ .../ops_versioned_replay_from_storage.yaml | 21 ++++++ .../p1_lance_estate_rebuildable.yaml | 21 ++++++ .../eval_checks/p2_asof_and_communities.yaml | 22 ++++++ .../eval_checks/p2_no_vectors_in_graph.yaml | 21 ++++++ .../p2_projection_contract_views.yaml | 30 ++++++++ .../p2_rebuild_first_snapshots.yaml | 23 ++++++ .../eval_checks/p3_corpus_fs_projection.yaml | 24 +++++++ .../p_projections_hold_no_authority.yaml | 22 ++++++ .../ret_claims_never_answer_now.yaml | 23 ++++++ .../eval_checks/ret_envelope_contract.yaml | 29 ++++++++ .../ret_filesystem_first_mounts_skill.yaml | 26 +++++++ .../eval_checks/ret_hydration_reverifies.yaml | 25 +++++++ .../ret_primitives_recipes_registry.yaml | 27 +++++++ .../ret_trust_boundary_no_content_auth.yaml | 22 ++++++ questions.md | 11 +++ 55 files changed, 1366 insertions(+) create mode 100644 plan/implementation_evals/README.md create mode 100644 plan/implementation_evals/eval_checks/boundary_library_scope_d60.yaml create mode 100644 plan/implementation_evals/eval_checks/boundary_provider_ports_d61.yaml create mode 100644 plan/implementation_evals/eval_checks/code_config_via_pydantic_settings.yaml create mode 100644 plan/implementation_evals/eval_checks/e0_blockizer_owns_identity.yaml create mode 100644 plan/implementation_evals/eval_checks/e0_content_addressed_reuse.yaml create mode 100644 plan/implementation_evals/eval_checks/e0_converter_router_versioned.yaml create mode 100644 plan/implementation_evals/eval_checks/e0_lineages_versions_modes.yaml create mode 100644 plan/implementation_evals/eval_checks/e0_origin_stamped_at_ingest.yaml create mode 100644 plan/implementation_evals/eval_checks/e0_postgres_holds_no_bodies.yaml create mode 100644 plan/implementation_evals/eval_checks/e0_raw_immutable_id_addressed.yaml create mode 100644 plan/implementation_evals/eval_checks/e0_structure_contract_unconditional.yaml create mode 100644 plan/implementation_evals/eval_checks/e1_chunks_whole_blocks_no_overlap.yaml create mode 100644 plan/implementation_evals/eval_checks/e1_multigranularity_retrieval.yaml create mode 100644 plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml create mode 100644 plan/implementation_evals/eval_checks/e2_claims_immutable_append_only.yaml create mode 100644 plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml create mode 100644 plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml create mode 100644 plan/implementation_evals/eval_checks/e2_no_pre_extraction_gate.yaml create mode 100644 plan/implementation_evals/eval_checks/e2_selection_ledger_replay.yaml create mode 100644 plan/implementation_evals/eval_checks/e2_selection_recall_envelope.yaml create mode 100644 plan/implementation_evals/eval_checks/e2_stance_kept_as_observation.yaml create mode 100644 plan/implementation_evals/eval_checks/e2_testimony_currency_counting.yaml create mode 100644 plan/implementation_evals/eval_checks/e3_blocking_cheap_first_cascade.yaml create mode 100644 plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml create mode 100644 plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml create mode 100644 plan/implementation_evals/eval_checks/e3_relations_evidence_collapse.yaml create mode 100644 plan/implementation_evals/eval_checks/e3_supersession_fact_level_bitemporal.yaml create mode 100644 plan/implementation_evals/eval_checks/er_merges_reversible.yaml create mode 100644 plan/implementation_evals/eval_checks/er_resolution_cascade_t0_t4.yaml create mode 100644 plan/implementation_evals/eval_checks/er_review_queue_cli.yaml create mode 100644 plan/implementation_evals/eval_checks/k_citations_binding.yaml create mode 100644 plan/implementation_evals/eval_checks/k_compiled_vs_authored_ownership.yaml create mode 100644 plan/implementation_evals/eval_checks/k_one_mechanism_n_scopes.yaml create mode 100644 plan/implementation_evals/eval_checks/k_planner_writer_driver.yaml create mode 100644 plan/implementation_evals/eval_checks/ops_deletion_cascade_grains.yaml create mode 100644 plan/implementation_evals/eval_checks/ops_execution_classes_bound.yaml create mode 100644 plan/implementation_evals/eval_checks/ops_idempotency_dlq.yaml create mode 100644 plan/implementation_evals/eval_checks/ops_metering_budgets_enforced.yaml create mode 100644 plan/implementation_evals/eval_checks/ops_producer_checker_families.yaml create mode 100644 plan/implementation_evals/eval_checks/ops_versioned_replay_from_storage.yaml create mode 100644 plan/implementation_evals/eval_checks/p1_lance_estate_rebuildable.yaml create mode 100644 plan/implementation_evals/eval_checks/p2_asof_and_communities.yaml create mode 100644 plan/implementation_evals/eval_checks/p2_no_vectors_in_graph.yaml create mode 100644 plan/implementation_evals/eval_checks/p2_projection_contract_views.yaml create mode 100644 plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml create mode 100644 plan/implementation_evals/eval_checks/p3_corpus_fs_projection.yaml create mode 100644 plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml create mode 100644 plan/implementation_evals/eval_checks/ret_claims_never_answer_now.yaml create mode 100644 plan/implementation_evals/eval_checks/ret_envelope_contract.yaml create mode 100644 plan/implementation_evals/eval_checks/ret_filesystem_first_mounts_skill.yaml create mode 100644 plan/implementation_evals/eval_checks/ret_hydration_reverifies.yaml create mode 100644 plan/implementation_evals/eval_checks/ret_primitives_recipes_registry.yaml create mode 100644 plan/implementation_evals/eval_checks/ret_trust_boundary_no_content_auth.yaml diff --git a/plan/implementation_evals/README.md b/plan/implementation_evals/README.md new file mode 100644 index 0000000..4b6a2f9 --- /dev/null +++ b/plan/implementation_evals/README.md @@ -0,0 +1,71 @@ +# Implementation evals — design-conformance acceptance checks + +An exhaustive set of [eval-banana](https://github.com/writeitai/eval-banana) `harness_judge` +checks that verify the **final implementation honors the binding design** — the decision log +(D1–D61), the requirements, and the design docs. Each check points an LLM judge at the binding +sources and at the code, states the invariant as concrete conditions, and demands a binary +verdict. + +**What these are:** acceptance checks for design conformance — "does the code do what the +design binds it to do?" **What these are not:** quality metrics. Extraction precision, +resolution P/R curves, retrieval recall@k live in the D22 golden-set eval harness (a separate, +data-driven asset); these checks verify the *machinery and its invariants exist and hold*. + +## Running + +eval-banana auto-discovers the `eval_checks/` directory here. From the repo root: + +```bash +uvx eval-banana run --harness-agent claude # or codex / gemini / opencode … +``` + +Two conventions when running: + +- **Judge family (D53).** The checks judge code largely produced by one model family; run the + judge harness on a **different family** than the primary implementation producer, per the + producer/checker separation decision. +- **Absence is failure.** These are acceptance checks for the *complete* system: a check whose + subsystem is not yet implemented scores 0 by design. Until the roadmap phases land, a partial + score is expected — the set is the finish line, not a progress bar. Filter by `tags` to run + the subset matching a delivered phase. + +## Conventions in the checks + +- Every check is `type: harness_judge`, one YAML file per check, self-contained (the judge sees + only `description` + `instructions` — never this README), and cites its binding sources + (decision numbers + design docs) so the judge reads the ground truth first. +- Conditions are conjunctive: score 1 only if **every** listed condition demonstrably holds, + with file paths cited in the reason. +- `tags` carry the area and the decision numbers, for filtered runs + (e.g. everything tagged `D43`, or all of `e2`). + +## Inventory (53 checks) + +| Area (prefix) | Checks | Binding sources covered | +|---|---|---| +| `e0_*` — documents/files | 8 | D36–D39, D42, D51 (raw mount), D54–D57 | +| `e1_*` — blocks/chunks | 2 | D57, D58, D25 | +| `e2_*` — claims/extraction | 9 | D2, D3, D19, D25, D31–D35, D41, D54, D59 | +| `e3_*` — relations/observations | 5 | D2–D5, D15, D18, D43 | +| `er_*` — entity resolution/review | 3 | D17, D20, D21, D24 | +| `k_*` — knowledge plane | 4 | D45–D47, D54 | +| `p_*`, `p1_*`, `p2_*`, `p3_*` — projections | 7 | D6–D11, D40, D44, D55 | +| `ret_*` — retrieval | 6 | D9, D41, D48–D51 | +| `ops_*` — orchestration/cross-cutting | 6 | D7, D12, D33, D52, D53, D55, requirements | +| `code_*`, `boundary_*` — conventions/scope | 3 | D60, D61, requirements §Code | + +Not every decision needs its own check: withdrawn decisions (D26–D30), naming/plane decisions +(D13, D14), and pure-analysis decisions are covered implicitly or are not implementation- +observable. Where one check verifies several decisions, the tags say so. + +## Inconsistencies found while authoring + +Authoring these against the full corpus surfaced inconsistencies, registered in +`questions.md` §5 (the repo's fix register): the requirements' temporal-split paragraph +predates D43 (observations also carry adjudicated validity), and the requirements' E3 bullet +omits observations entirely. See `questions.md` items 30–31. + +## Review + +`codex_review.md` in this directory records an independent Codex review of the set (accuracy +against the designs, coverage gaps, judge-ability). diff --git a/plan/implementation_evals/eval_checks/boundary_library_scope_d60.yaml b/plan/implementation_evals/eval_checks/boundary_library_scope_d60.yaml new file mode 100644 index 0000000..43493c8 --- /dev/null +++ b/plan/implementation_evals/eval_checks/boundary_library_scope_d60.yaml @@ -0,0 +1,21 @@ +schema_version: 1 +id: boundary_library_scope_d60 +type: harness_judge +description: The library boundary holds — no web UI, no multi-tenant control plane, no SSO/billing; agent surfaces are the complete consumption story; correctness never gated. +tags: [boundary, D60] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D60 and CLAUDE.md Rule 3, then survey the repository's shipped surfaces. + These are acceptance checks for the FINAL system. + + Verify ALL of the following: + 1. No web UI ships in this repository (no frontend app, no server-rendered dashboards); the + consumption surfaces are API, CLI, MCP server, and mounted filesystems. + 2. No multi-tenant machinery exists: no orgs/users/roles tables, no SSO/SAML, no billing — + one deployment is one trust domain. + 3. No correctness-determining mechanism (extraction, resolution, supersession, grounding, + evals, budgets, DLQ, deletion) is stubbed out, feature-flagged off, or delegated to an + external/commercial service — everything the design calls binding is implemented here. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/boundary_provider_ports_d61.yaml b/plan/implementation_evals/eval_checks/boundary_provider_ports_d61.yaml new file mode 100644 index 0000000..eb91d90 --- /dev/null +++ b/plan/implementation_evals/eval_checks/boundary_provider_ports_d61.yaml @@ -0,0 +1,27 @@ +schema_version: 1 +id: boundary_provider_ports_d61 +type: harness_judge +description: The deployment substrate is reached only through the provider ports, each with a self-host and a reference adapter; the engine itself is not abstracted; one deployment runs self-hosted. +tags: [boundary, D61] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D61 and requirements_v3.md §Fixed choices & the reference deployment, then + inspect the adapter layer and self-host stack under src/ and the deploy profiles. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Ports exist as explicit interfaces for: object store, task queue/scheduler, mount + publication, K git remote, model/embedding providers, telemetry export, and the auth + perimeter — and substrate access goes through them (no direct GCS/Cloud Tasks calls + scattered in domain code). + 2. Each port has both adapters: self-host (S3-compatible store, Postgres-backed queue, local + directory mounts, generic git remote, BYO keys) and the reference set (GCS, Cloud Tasks, + gcsfuse, hosted git). + 3. The engine is NOT abstracted: Postgres, LanceDB, LadybugDB, and the E/K/P data model are + used directly — no "database-agnostic" indirection layers wrap them. + 4. A runnable self-host profile exists (docker-compose or equivalent) that brings up one + complete deployment on the self-host adapters, with every correctness capability + included. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/code_config_via_pydantic_settings.yaml b/plan/implementation_evals/eval_checks/code_config_via_pydantic_settings.yaml new file mode 100644 index 0000000..a38088e --- /dev/null +++ b/plan/implementation_evals/eval_checks/code_config_via_pydantic_settings.yaml @@ -0,0 +1,24 @@ +schema_version: 1 +id: code_config_via_pydantic_settings +type: harness_judge +description: All configuration enters through pydantic-settings; secrets are SecretStr/SecretBytes unwrapped only at point of use; no ad-hoc environment access. +tags: [code, requirements] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding convention. + Read requirements_v3.md §Code, then inspect the settings module(s) and grep the source for + environment access under src/. These are acceptance checks for the FINAL system: if the + subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Environment variables are read exclusively via pydantic-settings BaseSettings classes — + no os.environ / os.getenv anywhere in src/ (the ruff TID251 ban must still be configured + in pyproject.toml), and no evasions (subprocess env tricks, importing environ under an + alias, reading /proc/self/environ). + 2. Every secret-bearing setting (DB URLs with passwords, API keys, tokens) is typed + SecretStr/SecretBytes, and .get_secret_value() appears only at the call sites that hand + the value to a client/driver — never stored unwrapped on another object, never logged. + 3. Settings objects are constructed at composition boundaries (entry points/workers), not + scattered ad hoc through domain logic. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_blockizer_owns_identity.yaml b/plan/implementation_evals/eval_checks/e0_blockizer_owns_identity.yaml new file mode 100644 index 0000000..2e5cb6e --- /dev/null +++ b/plan/implementation_evals/eval_checks/e0_blockizer_owns_identity.yaml @@ -0,0 +1,25 @@ +schema_version: 1 +id: e0_blockizer_owns_identity +type: harness_judge +description: One deterministic, versioned blockizer derives the block sequence; sections snap to the block grid; blocks are never derived from sections. +tags: [e0, e1, D57] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D57 and plan/designs/e1_chunks_design.md §2–§3, then locate the blockizer + under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. + + Verify ALL of the following: + 1. A single deterministic blockizer (no LLM calls, no randomness) runs downstream of every + converter route, versioned as blockizer_version, emitting blocks.json with ordinal, type, + char span into document.md, best-effort page/bbox provenance, and block_hash. + 2. document.md is treated as the immutable, content-hash-addressed coordinate system: claim + spans, blocks, sections, and chunks all reference it by offset. + 3. PageIndex/structurer section spans are normalized onto the block grid by a deterministic + snap rule (backward-snap, partition enforcement, nesting validation, degrade-to-parent) — + and structuring can never hard-fail a document. + 4. Direction invariant: nothing derives blocks from sections — LLM output never touches the + identity layer. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_content_addressed_reuse.yaml b/plan/implementation_evals/eval_checks/e0_content_addressed_reuse.yaml new file mode 100644 index 0000000..2c12f1a --- /dev/null +++ b/plan/implementation_evals/eval_checks/e0_content_addressed_reuse.yaml @@ -0,0 +1,27 @@ +schema_version: 1 +id: e0_content_addressed_reuse +type: harness_judge +description: Re-ingesting an edited document costs proportional to the edit — extraction/embedding work is keyed by content, with no LLM output in any reuse key. +tags: [e0, e2, D56] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D56 and plan/designs/evidence_lifecycle_design.md §reuse, then inspect the + E2 idempotency keys and the version-diff logic under src/. These are acceptance checks for + the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. E2 extraction idempotency keys on an extraction_input_hash built from stable components + only: the chunk's own block hashes + neighbor-chunk block hashes + stable header facts + + extractor version + structurer version. NO LLM output (section path text, summaries, E1 + prefix) participates in the key. + 2. An unchanged chunk reuses its claims — re-attached to the new version's chunk row via the + chunk_claims map (written on both fresh extraction and reuse; exact, never inferred from + content-hash joins); a chunk whose neighbors changed re-extracts. + 3. Embeddings key on (chunk content hash, embedding version); conversion artifacts on + (content object, converter version). + 4. The efficiency ladder exists with cheapest exits first: connector-metadata no-op → + content-object no-op → conversion reuse → chunk-grain extraction reuse → delta-only + downstream triggers. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_converter_router_versioned.yaml b/plan/implementation_evals/eval_checks/e0_converter_router_versioned.yaml new file mode 100644 index 0000000..cd37db1 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e0_converter_router_versioned.yaml @@ -0,0 +1,21 @@ +schema_version: 1 +id: e0_converter_router_versioned +type: harness_judge +description: Conversion is a pluggable, versioned router; converters emit document.md + page map + media[], never blocks. +tags: [e0, D38, D57] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D38 and D57 (the refined converter contract) and + plan/designs/e0_files_design.md §3, then locate the conversion module under src/. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. A converter interface exists with a router selecting a converter by input type from + per-deployment configuration (not hardcoded if/else on one format). + 2. The converter contract returns document.md + a page map + media[] — converters do NOT + emit blocks (block derivation is the blockizer's job, downstream of every route). + 3. Every conversion is stamped with converter_name/converter_version on the document version + row, and a converter/routing change triggers re-conversion keyed by version (D7 discipline). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_lineages_versions_modes.yaml b/plan/implementation_evals/eval_checks/e0_lineages_versions_modes.yaml new file mode 100644 index 0000000..3375954 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e0_lineages_versions_modes.yaml @@ -0,0 +1,31 @@ +schema_version: 1 +id: e0_lineages_versions_modes +type: harness_judge +description: Documents are lineages with connector-native identity, append-only immutable versions, deduplicated content objects, and snapshot/living semantics including retract-on-removal. +tags: [e0, D55, D54] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D55 and plan/designs/evidence_lifecycle_design.md, then inspect the + documents/document_versions/content_objects schema and the watched-source sync logic under + src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. + + Verify ALL of the following: + 1. The logical document is a lineage with stable doc_id identified by (source_kind, + source_ref); renames/moves update metadata on the same lineage; a new ref is a new lineage. + 2. Versions are append-only rows (one per observed snapshot) referencing content objects + deduplicated by content_hash (bytes stored/converted once, even across lineages); + unchanged revision/etag or unchanged bytes are no-ops; rapid edits are debounced. + 3. Each lineage carries versioning_mode: snapshot (every version independent dated testimony, + the fail-safe default) or living (current version is the standing statement). + 4. In living mode, removal of a fact's sole current support closes the fact per shape + (relations/effective-state observations: valid_until capped at source_modified_at; + measurements: invalidated_at — never a valid-time cap, per D43's no-cap rule), recorded as + retracted_source_removal — loud, attributed, reversible; retraction checks run only after + the sync cycle completes (an intra-cycle move is a support swap, never retract+reassert). + 5. Deletion is uniform (a version / a lineage / observed-deleted-at-source): contribution + removed, claims retained as history with currency ended, solely-supported facts closed per + shape, recorded; hard-forget is a separate grain. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_origin_stamped_at_ingest.yaml b/plan/implementation_evals/eval_checks/e0_origin_stamped_at_ingest.yaml new file mode 100644 index 0000000..27dfd50 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e0_origin_stamped_at_ingest.yaml @@ -0,0 +1,19 @@ +schema_version: 1 +id: e0_origin_stamped_at_ingest +type: harness_judge +description: Every input carries an immutable origin stamp (external vs self/system-generated) set at ingest. +tags: [e0, D42] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D42, then inspect the ingest worker and documents schema under src/. These + are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Every ingested document records an origin field distinguishing at minimum external from + self/system-generated, set at E0 ingest time. + 2. The field is immutable after ingest — no code path updates it. + 3. Nothing downstream silently drops it (it survives on the lineage/version rows so + confidence math can later discount self-generated echoes). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_postgres_holds_no_bodies.yaml b/plan/implementation_evals/eval_checks/e0_postgres_holds_no_bodies.yaml new file mode 100644 index 0000000..c215e50 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e0_postgres_holds_no_bodies.yaml @@ -0,0 +1,21 @@ +schema_version: 1 +id: e0_postgres_holds_no_bodies +type: harness_judge +description: Postgres stores only compact metadata and the queryable section index — never document bodies or Markdown text. +tags: [e0, D37] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D37 and plan/designs/e0_files_design.md §2, then inspect the schema + (migrations/models) under src/. These are acceptance checks for the FINAL system: if the + subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. No Postgres table/column stores document body text or converted Markdown (documents and + related tables hold identity, versions, state, artifact URIs, hashes, costs, section + titles/paths/roles/spans/summaries only). + 2. Bodies (document.md, blocks.json, pageindex.json) are stored as object-store artifacts and + referenced from Postgres by URI. + 3. Blocks (D57) are NOT Postgres rows — a sidecar artifact plus derived keys only. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_raw_immutable_id_addressed.yaml b/plan/implementation_evals/eval_checks/e0_raw_immutable_id_addressed.yaml new file mode 100644 index 0000000..c6cc954 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e0_raw_immutable_id_addressed.yaml @@ -0,0 +1,24 @@ +schema_version: 1 +id: e0_raw_immutable_id_addressed +type: harness_judge +description: Raw bytes are stored immutably in a dedicated raw store, ID-addressed, off the navigation path with audited access. +tags: [e0, D36, D37, D51] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D36/D37 (+ the "Refined by D51" note) and plan/designs/e0_files_design.md + §2, then locate the implementation under src/ (storage layout, ingest worker, mount config). + These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Original file bytes are written to a dedicated raw store, keyed by ID + content hash + (never title-addressed paths), and nothing in the codebase ever mutates or overwrites a + stored raw object (write-once). + 2. Converted artifacts (document.md, sidecars) live in a separate artifacts store, not the + raw store. + 3. Raw originals are reachable only via explicit pointers (P3 stubs / document frontmatter), + never listed on the navigation path — and raw reads go through a path with data-access + audit logging (D51). + 4. content_hash = sha256 of raw bytes is computed at ingest and used as the idempotency key. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_structure_contract_unconditional.yaml b/plan/implementation_evals/eval_checks/e0_structure_contract_unconditional.yaml new file mode 100644 index 0000000..d96793e --- /dev/null +++ b/plan/implementation_evals/eval_checks/e0_structure_contract_unconditional.yaml @@ -0,0 +1,24 @@ +schema_version: 1 +id: e0_structure_contract_unconditional +type: harness_judge +description: Every document gets a section structure (synthetic root fallback), with roles, summaries-as-context, and a placement hint. +tags: [e0, D39] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D39 and plan/designs/e0_files_design.md §4, then locate the structurer and + the document_sections schema under src/. These are acceptance checks for the FINAL system: + if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. The output contract is unconditional: every document ends up with document_sections rows — + short/simple documents get a single synthetic root section (full span, role=body) rather + than no structure. + 2. Sections carry a role from a defined enum (body, references, boilerplate, nav, etc.), used + downstream by E2 Selection at proposition grain — never as a pre-extraction skip. + 3. Per-section summaries are stored as context (feeding prefixes/navigation), never as facts; + the structurer run is versioned (structurer_version) and replayed from storage on rebuild. + 4. A placement hint (a proposed corpus-filesystem path) is produced and stored as advisory + input for P3 — not a commitment. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e1_chunks_whole_blocks_no_overlap.yaml b/plan/implementation_evals/eval_checks/e1_chunks_whole_blocks_no_overlap.yaml new file mode 100644 index 0000000..7251519 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e1_chunks_whole_blocks_no_overlap.yaml @@ -0,0 +1,25 @@ +schema_version: 1 +id: e1_chunks_whole_blocks_no_overlap +type: harness_judge +description: Chunks are non-overlapping runs of whole blocks within one section, packed by semchunk with anchor-stabilized boundaries. +tags: [e1, D58, D57] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D58 and plan/designs/e1_chunks_design.md §4, then locate the chunker under + src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. + + Verify ALL of the following: + 1. A chunk is an ordered run of WHOLE blocks within one section — chunk boundaries always + coincide with block boundaries, and a chunk never crosses a section boundary. + 2. There is NO overlap between chunks anywhere (no sliding windows, no overlap parameters); + cross-boundary context comes from the E2 bundle's neighbor chunks instead. + 3. Packing uses semchunk to a configured token budget with anchor-stabilized boundaries + (packing restarts at content-defined anchor blocks so an early edit perturbs packing only + to the next anchor). + 4. Edge rules exist: an oversized atomic block (e.g. a table) becomes its own oversized + chunk; a pathological giant paragraph falls back to deterministic sentence splitting. + 5. chunk_content_hash = hash of the ordered block hashes. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e1_multigranularity_retrieval.yaml b/plan/implementation_evals/eval_checks/e1_multigranularity_retrieval.yaml new file mode 100644 index 0000000..8a55143 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e1_multigranularity_retrieval.yaml @@ -0,0 +1,21 @@ +schema_version: 1 +id: e1_multigranularity_retrieval +type: harness_judge +description: Claims are the needle index, chunks the passage index; low-value roles are filtered retrieval-side, never skipped at extraction. +tags: [e1, p1, D58, D25] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D58 (embedding granularity) and D25, then inspect the P1 index writers and + default search recipes under src/. These are acceptance checks for the FINAL system: if the + subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Every accepted, decontextualized claim is embedded into P1 (claims are the fine-grain + "needle" channel); chunks are embedded as the passage channel. + 2. P1 chunk rows carry a section-role scalar, and DEFAULT search recipes filter out + references/nav/boilerplate/legal chunks retrieval-side. + 3. Crucially (D25): that filtering happens at RETRIEVAL time over what was indexed — nothing + skips extraction or indexing by role upstream (no pre-extraction value gate reintroduced). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml b/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml new file mode 100644 index 0000000..600b8a8 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml @@ -0,0 +1,24 @@ +schema_version: 1 +id: e2_asserted_validity_immutable +type: harness_judge +description: Claims carry an immutable source-asserted validity interval (from/until/precision/kind), grounded by window-membership, never revisable, never a belief authority. +tags: [e2, D41] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D41, then inspect the claim schema and the temporal extraction under src/. + These are acceptance checks for the FINAL system: if the subsystem is absent or partial, + score 0. + + Verify ALL of the following: + 1. Claims carry claim_valid_from/claim_valid_until + a precision + a kind + (proposition-validity vs event-time vs measurement-period), emitted in the same E2 call. + 2. The interval is grounded: the date substring must verbatim-exist in the bundle + (window-membership, D32) — ungrounded dates are rejected. + 3. The columns are immutable: no UPDATE path, no invalidated_at/status/superseded_by on them; + a contradicting source yields a NEW claim with its own interval. + 4. The relation/observation adjudicator may consult claim_valid_* as evidence input with a + monotonicity guard (a late retrospective cannot move an adjudicated window), and nothing + reads claim validity back as current belief. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_claims_immutable_append_only.yaml b/plan/implementation_evals/eval_checks/e2_claims_immutable_append_only.yaml new file mode 100644 index 0000000..26f7a38 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e2_claims_immutable_append_only.yaml @@ -0,0 +1,24 @@ +schema_version: 1 +id: e2_claims_immutable_append_only +type: harness_judge +description: Claims are immutable, append-only evidence — no update path, no supersession fields, never invalidated. +tags: [e2, D2, D3, D54] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D2/D3 (and D54's "currency is bookkeeping, never validity"), then inspect + the claims schema/models and every code path that touches claim rows under src/. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Claim rows are never UPDATEd after insert in any semantic field — no code path rewrites + claim_text, spans, asserted validity, or timestamps; claims have no superseded_by / + invalidated_at / status-with-supersession-semantics columns. + 2. Supersession and contradiction adjudication write to relations/observations only — no + claim is ever "closed" when a fact is superseded. + 3. The one permitted mutable adjunct is D54 testimony currency: a cached currency flag backed + by an append-only, reason-coded transitions ledger — and currency is bookkeeping (search + defaults, counting), never validity: transaction-time reconstructions still see old + generations. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml b/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml new file mode 100644 index 0000000..6c3ce95 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml @@ -0,0 +1,26 @@ +schema_version: 1 +id: e2_context_bundle_two_calls +type: harness_judge +description: E2 extraction runs over a context bundle (never a bare chunk) as two calls — Selection, then fused decontextualize+decompose+ground. +tags: [e2, D31, D19] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D31/D19 and plan/designs/e2_e3_claims_relations_design.md §3, then locate + the extractor under src/. These are acceptance checks for the FINAL system: if the subsystem + is absent or partial, score 0. + + Verify ALL of the following: + 1. The extraction call receives a bundle: document header, section path/role (+ summary), the + chunk's context prefix (when the E1 prefix stage exists), same-section neighbor chunks, + and known-entity hints — never a bare chunk. + 2. The call architecture is two calls: Selection (verifiability KEEP/REWRITE/DROP) as its own + call, then a fused disambiguation/decontextualization + decomposition + grounding call. A + one-call collapse is only acceptable if the repo documents a measured ablation justifying + it; the literal per-sentence multi-call loop must not be the implementation. + 3. Coreference is satisfied in-call (D19): no dedicated coref model or pre-pass exists, and + the output contract forbids dangling pronouns/references in accepted claims. + 4. Extraction batching (if present) batches a section's contiguous chunks per call while + keeping claims anchored per-chunk and idempotency keys per-chunk (D58). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml b/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml new file mode 100644 index 0000000..ce8ffa2 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml @@ -0,0 +1,28 @@ +schema_version: 1 +id: e2_grounding_layered_dual_field +type: harness_judge +description: Claim grounding is dual-field (standalone claim_text + verbatim source_span with offsets + added_context[]) with layered acceptance — anchor, window-membership, entailment, sampled audit. +tags: [e2, D32] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D32 and plan/designs/e2_e3_claims_relations_design.md §3.3, then inspect + the claim schema and grounding validation under src/. These are acceptance checks for the + FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Every claim stores BOTH a standalone claim_text and a verbatim source_span with character + offsets into document.md (the D57 coordinate system), plus an added_context list naming + the bundle source of every added substring. + 2. Acceptance layers, cheapest first: (a) deterministic anchor — source_span verified in code + as a real slice of the source (never trusted from the model); (b) deterministic + window-membership — every declared added substring verbatim-exists in its declared bundle + element (fabrications rejected); (c) an in-call entailment self-verdict including the + attribution rule ("X said Y" entails "X said Y", never bare "Y"); (d) a SAMPLED + independent entailment audit — never per-claim, and run on a different model family (D53). + 3. A decontextualized claim whose text is not a verbatim substring is ACCEPTED when its + anchor + declared additions check out (the old verbatim-substring hard reject must not + exist); a claim with an unverifiable added substring is REJECTED. + 4. Grounding uses the bundle only — no external/world knowledge is permitted or injected. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_no_pre_extraction_gate.yaml b/plan/implementation_evals/eval_checks/e2_no_pre_extraction_gate.yaml new file mode 100644 index 0000000..c0d7e04 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e2_no_pre_extraction_gate.yaml @@ -0,0 +1,25 @@ +schema_version: 1 +id: e2_no_pre_extraction_gate +type: harness_judge +description: There is no pre-extraction value/salience gate — every chunked document is fully extracted; junk control is in-call Selection plus redundancy collapse. +tags: [e2, D25, D34] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D25/D34 and plan/designs/e2_e3_claims_relations_design.md §4, then inspect + the pipeline between chunking and extraction under src/. These are acceptance checks for the + FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. No stage between E1 and E2 decides whether a chunk/section/document is "worth" extracting + — no salience classifier, no novelty ANN check, no deferred-extraction queue, no + gate_decisions-style state. + 2. Every document that survives chunking is fully extracted; the section role signal is fed + INTO the E2 call (Selection drops at proposition grain) rather than used as a binary skip. + 3. The only permitted deterministic short-circuits are idempotency/reuse keys (content-hash + no-ops, D12/D56) — content-based, never value-based. + 4. If a deterministic role-based section filter exists (the documented D25 add-back), it must + be justified in the repo by a measured break-even — absent that evidence, its presence is + a violation. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_selection_ledger_replay.yaml b/plan/implementation_evals/eval_checks/e2_selection_ledger_replay.yaml new file mode 100644 index 0000000..ff78451 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e2_selection_ledger_replay.yaml @@ -0,0 +1,21 @@ +schema_version: 1 +id: e2_selection_ledger_replay +type: harness_judge +description: Every Selection drop and decontextualization edit is append-only, versioned ledger state; rebuilds replay from storage and never re-call the model. +tags: [e2, D33, D7] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D33 (and D7's replay discipline), then inspect the claim_extraction_decisions + ledger and the rebuild paths under src/. These are acceptance checks for the FINAL system: + if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. An append-only, version-stamped ledger records every Selection outcome (drops with reason + class, rewrites, kept_flagged) and every decontextualization edit. + 2. The ledger is never updated or deleted from — corrections append. + 3. Rebuild paths read stored claims + decisions and never re-invoke the LLM (nondeterministic + rungs are replay-from-storage); the extraction worker is idempotent on its content-based + input hash + extractor version. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_selection_recall_envelope.yaml b/plan/implementation_evals/eval_checks/e2_selection_recall_envelope.yaml new file mode 100644 index 0000000..45de0c2 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e2_selection_recall_envelope.yaml @@ -0,0 +1,23 @@ +schema_version: 1 +id: e2_selection_recall_envelope +type: harness_judge +description: Selection is recall-conservative — KEEP bias, never-drop classes, kept_flagged outcome, per-fact canary CI. +tags: [e2, D35, D22] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D35, then inspect the Selection prompt/rubric, outcome enum, and eval + harness under src/. These are acceptance checks for the FINAL system: if the subsystem is + absent or partial, score 0. + + Verify ALL of the following: + 1. The Selection rubric instructs conservative KEEP when unsure, and mixed sentences are + REWRITTEN to their verifiable span rather than dropped. + 2. Never-drop lexical classes are protected regardless of phrasing: quantities, dates, + named-entity+predicate statements, change-of-state language. + 3. A low-confidence kept_flagged outcome exists (mark-for-review, not delete) — Selection + never hard-deletes a borderline candidate silently. + 4. The eval harness plants rare-fact canaries and fails CI if Selection drops one; drop-rate + tuning is per-fact, not corpus-average. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_stance_kept_as_observation.yaml b/plan/implementation_evals/eval_checks/e2_stance_kept_as_observation.yaml new file mode 100644 index 0000000..645b6b5 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e2_stance_kept_as_observation.yaml @@ -0,0 +1,27 @@ +schema_version: 1 +id: e2_stance_kept_as_observation +type: harness_judge +description: Attributed stance is kept and normalized to an observation on the holder; stance content never becomes a world-fact; unattributed opinion still drops. +tags: [e2, e3, D59, D43] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D59, then inspect the Selection rubric and E3 normalization under src/. + These are acceptance checks for the FINAL system: if the subsystem is absent or partial, + score 0. + + Verify ALL of the following: + 1. Selection keeps stances attributed to a resolvable holder ("X said/believes/prefers/ + opposes Y", including the document author's own voice via the bundle header) as + verifiable propositions about X. + 2. E3 normalizes a kept stance into an observation anchored on the HOLDER (ordinary D43 + machinery: untyped, bi-temporal); a changed mind is ordinary supersession (old stance's + window caps at the new stance's asserted time); conflicting same-time reports coexist via + contradiction_group. + 3. The guard: no relation or observation about Y itself is ever derived from "X believes Y" — + only the stance-about-X. + 4. Unattributed opinion, advice, hypotheticals, and generic truisms still drop; a stance + whose holder cannot be decontextualized to a resolvable entity drops (ledger reason + opinion, meaning unattributed-only). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_testimony_currency_counting.yaml b/plan/implementation_evals/eval_checks/e2_testimony_currency_counting.yaml new file mode 100644 index 0000000..746c982 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e2_testimony_currency_counting.yaml @@ -0,0 +1,33 @@ +schema_version: 1 +id: e2_testimony_currency_counting +type: harness_judge +description: evidence_count means distinct current-testimony lineages per stance — invariant under re-extraction, version churn, and repetition; zero-support handling splits by cause. +tags: [e2, e3, D54] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D54 and plan/designs/evidence_lifecycle_design.md, then inspect the + currency ledger, the count maintenance, and the zero-support paths under src/. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Claims carry testimony currency: current iff they belong to their lineage's current + extraction basis; re-extraction flips a superseded generation non-current wholesale by + coordinates (no content matching); living-mode version supersession flips claims whose + chunks left the current version; snapshot mode flips nothing. + 2. Currency changes are recorded in an append-only, reason-coded transitions ledger plus a + cached flag — and never touch validity (no adjudication, no invalidated_at, claims stay + immutable). + 3. evidence_count/contradict_count on relations AND observations are maintained as: distinct + document lineages with current-testimony support, per stance — never claim rows, extractor + generations, source versions, or poll cycles. + 4. Zero-current-support splits by cause: source/curator-driven loss closes solely-supported + facts per shape (states: valid_until cap; measurements: invalidated_at) recorded as + retracted_source_removal with no flag; processing-driven loss (new extractor generation + fails to re-derive from an UNCHANGED file) is flagged support_withdrawn for review — the + flag's only trigger — and flagged facts are excluded from K3 eligibility and marked in the + retrieval envelope. + 5. P1's default claim channel indexes current testimony only; an audit channel/opt-in sees + all generations. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e3_blocking_cheap_first_cascade.yaml b/plan/implementation_evals/eval_checks/e3_blocking_cheap_first_cascade.yaml new file mode 100644 index 0000000..7d32166 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e3_blocking_cheap_first_cascade.yaml @@ -0,0 +1,22 @@ +schema_version: 1 +id: e3_blocking_cheap_first_cascade +type: harness_judge +description: Supersession candidates come from (entity, predicate) blocking with a cheap-first escalation cascade — LLM spend scales with ambiguity, not volume. +tags: [e3, D4] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D4, then inspect the supersession worker under src/. These are acceptance + checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Candidate conflicts are found by blocking on (entity_id, predicate) over the relations + table — never an O(N) corpus-wide vector scan per write. + 2. A novelty gate routes clear ADD/NOOP cases past any LLM (zero-candidate, exact/near-exact + duplicate, clear-novelty exits). + 3. The residue escalates cheapest-first (deterministic → embedding similarity → small model → + frontier model), and the escalation order is visible in code, not just prose. + 4. Observation supersession follows the same cascade blocked on the resolved entity (D43), + sharing the engine rather than duplicating it. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml b/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml new file mode 100644 index 0000000..fb78795 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml @@ -0,0 +1,30 @@ +schema_version: 1 +id: e3_observations_untyped_adjudicated +type: harness_judge +description: Observations are untyped, entity-anchored, bi-temporal facts — no attribute vocabulary or value columns; supersession by exhaustive entity blocking with the no-cap rule and fail-safe coexist. +tags: [e3, D43] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D43 and plan/designs/observations_design.md, then inspect the observations + schema and the add-observation worker under src/. These are acceptance checks for the FINAL + system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Observations are anchored to a resolved entity with an NL statement carrying the value and + any period; there is NO governed attribute key, NO value_domain/unit_dimension/cardinality, + NO structured value or period column, and NO typed exclusion constraint (only FK-to-entity + + temporal sanity checks). + 2. Blocking fetches ALL of the entity's live observations by exact key (exhaustive; indexed); + semantic similarity only RANKS candidates for hub entities — it never gates membership. + 3. The adjudicator outcomes are evidence / supersede / contradict-coexist / new, each gated on + a positive same-thing match judged semantically; the no-cap rule holds: only a changing + effective state is ever capped; a measurement/fixed-period figure is never capped on + valid-time — a conflicting same-period figure coexists with a shared contradiction_group. + 4. The fail-safe contract is in code: supersede is permitted only above an explicit margin + against a positively-matched prior AND writes an observation_adjudications reason row; any + incomplete comparison falls to coexist/new — the failure mode is a duplicate row, never a + silent overwrite. + 5. Observations never project to the P2 graph; they are embedded in P1 for retrieval. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml b/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml new file mode 100644 index 0000000..1fb995c --- /dev/null +++ b/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml @@ -0,0 +1,27 @@ +schema_version: 1 +id: e3_predicate_registry_governed +type: harness_judge +description: The predicate vocabulary is a governed registry with an other: escape and promotion review; domain/range constraints are enforced; time is never a predicate or node. +tags: [e3, D5, D18, D15] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D5/D15/D18 and plan/designs/registries_design.md, then inspect the + predicate registry and extraction constraints under src/. These are acceptance checks for + the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. A Postgres predicate registry exists (name, description, synonyms, status, subject/object + type constraints); extraction is constrained to it with an other: escape; a + review path promotes/maps frequent other: values. + 2. Domain/range constraints are mechanically enforced at normalization (an edge-type-map + style check that rejects type-invalid triples). + 3. The seed core ships as registry CONTENT (schema.org-aligned types + predicates with + parent anchoring; user extensions declare a core parent — extend-never-fork), not as DDL + or hardcoded enums. + 4. Time is never represented as a predicate, a Date node, or a relation object — temporal + data lives in the bi-temporal columns and claim asserted-validity only. + 5. Extraction prompts render from the registry (types/predicates/descriptions/examples), so + ontology changes are row edits, not prompt engineering. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e3_relations_evidence_collapse.yaml b/plan/implementation_evals/eval_checks/e3_relations_evidence_collapse.yaml new file mode 100644 index 0000000..eb11a37 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e3_relations_evidence_collapse.yaml @@ -0,0 +1,21 @@ +schema_version: 1 +id: e3_relations_evidence_collapse +type: harness_judge +description: Claims and relations are distinct, joined many-to-many with stance; corpus redundancy collapses into one relation with evidence rows. +tags: [e3, D2] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D2 and plan/analysis/concepts.md, then inspect the relations and + relation_evidence schema and the E3 normalization under src/. These are acceptance checks + for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Relations are (subject_entity, predicate, object_entity) rows whose identity is the fact; + claims join via relation_evidence(relation_id, claim_id, stance supports|contradicts) — + never claim_id stamped on graph edges, never 1:1 claim=relation assumptions. + 2. A claim can yield 0..n relations; the same fact asserted again links evidence to the + EXISTING relation rather than inserting a duplicate relation row. + 3. Duplicate evidence is idempotent (evidence-once per claim/fact pair). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e3_supersession_fact_level_bitemporal.yaml b/plan/implementation_evals/eval_checks/e3_supersession_fact_level_bitemporal.yaml new file mode 100644 index 0000000..9f43195 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e3_supersession_fact_level_bitemporal.yaml @@ -0,0 +1,22 @@ +schema_version: 1 +id: e3_supersession_fact_level_bitemporal +type: harness_judge +description: Supersession closes validity windows on relations (and observations) — never on claims; both time axes are queryable. +tags: [e3, D3, D43] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D3 (and D43), then inspect the supersession write paths and the temporal + columns under src/. These are acceptance checks for the FINAL system: if the subsystem is + absent or partial, score 0. + + Verify ALL of the following: + 1. Relations and observations carry the bi-temporal quartet: valid_from/valid_until + (world time) and ingested_at/invalidated_at (belief time); supersession caps valid_until — + it never deletes rows and never touches claims. + 2. invalidated_at means "we no longer believe this was ever right", distinct from a closed + validity window ("it ended"); a superseded row stays active in that sense. + 3. Both time-travel questions are answerable in the query layer: what was true at T + (valid-time) and what did we believe at T (transaction-time). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/er_merges_reversible.yaml b/plan/implementation_evals/eval_checks/er_merges_reversible.yaml new file mode 100644 index 0000000..a6b3a7e --- /dev/null +++ b/plan/implementation_evals/eval_checks/er_merges_reversible.yaml @@ -0,0 +1,24 @@ +schema_version: 1 +id: er_merges_reversible +type: harness_judge +description: Entity merges are reversible, append-only adjudications — decisions ledger, pre-merge snapshots, redirect chains; merges are no-ops for the graph because P2 rebuilds. +tags: [registries, D21] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D21, then inspect the merge/cluster machinery under src/. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. resolution_decisions is append-only with superseded_by chains; merge_events record a + pre-merge membership snapshot; entities carry a merged_into redirect (a redirect, not a + rewrite — relations are NOT re-pointed in Postgres). + 2. Un-merge is implementable from stored state alone (snapshot + decisions), and + negative/exclusion edges can be recorded. + 3. Clustering is connected-components-to-gather with a component-size guard, then HAC + distance-cut inside blobs — never bare transitive closure; incremental assignment + re-clusters only the local neighborhood. + 4. A generic-identifier guard exists (an alias that suddenly links many entities is + down-weighted and re-evaluated). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/er_resolution_cascade_t0_t4.yaml b/plan/implementation_evals/eval_checks/er_resolution_cascade_t0_t4.yaml new file mode 100644 index 0000000..f94d1c5 --- /dev/null +++ b/plan/implementation_evals/eval_checks/er_resolution_cascade_t0_t4.yaml @@ -0,0 +1,25 @@ +schema_version: 1 +id: er_resolution_cascade_t0_t4 +type: harness_judge +description: Entity resolution is the T0–T4 cascade — exact, fuzzy blocking (recall-first), phonetic, embedding, LLM — with per-type versioned thresholds; cheap tiers escalate, never auto-reject. +tags: [registries, D17] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D17 and plan/designs/registries_design.md, then inspect the resolver under + src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. + + Verify ALL of the following: + 1. The cascade runs T0 exact match on the canonical name form → T1 fuzzy blocking (pg_trgm, + recall-first low floor — candidate generation, never a decision) → T2 phonetic + (Daitch-Mokotoff, not Soundex) → T3 embedding similarity on the residue → T4 LLM + adjudication on the ambiguous band, with human review for high blast-radius. + 2. Accept/reject bands are per-type, versioned configuration stamped with resolver_version — + not hardcoded global constants. + 3. Cheap tiers escalate near-misses rather than auto-rejecting (blocking sets a recall + ceiling; no tier below T4 makes a final negative decision on a near-miss). + 4. No third-party external authority (Wikidata, ORCID, etc.) is called anywhere on the + resolution path, and no external_ids table exists (D20). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/er_review_queue_cli.yaml b/plan/implementation_evals/eval_checks/er_review_queue_cli.yaml new file mode 100644 index 0000000..f9eab54 --- /dev/null +++ b/plan/implementation_evals/eval_checks/er_review_queue_cli.yaml @@ -0,0 +1,21 @@ +schema_version: 1 +id: er_review_queue_cli +type: harness_judge +description: Human review is a thin Postgres-backed CLI cluster queue — impact-band routed, hub merges never auto-accepted, verdicts reversible and provenance-stamped; no web UI in the library. +tags: [registries, D24, D60] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D24 (and D60), then inspect the review tooling under src/. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. A CLI review queue over Postgres exists, reviewing CLUSTERS (not pairwise), routing only + the expected_impact = blast_radius x (1 - confidence) middle band to humans. + 2. High-degree hub merges never auto-accept regardless of confidence. + 3. Every review action appends a reversible, provenance-stamped record consistent with D21 + (redirect-preserving; nothing destructive). + 4. No web review UI ships in this repository (D60: the web UI is the cloud product's + surface; the library's review surface is the CLI). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/k_citations_binding.yaml b/plan/implementation_evals/eval_checks/k_citations_binding.yaml new file mode 100644 index 0000000..6c20a6e --- /dev/null +++ b/plan/implementation_evals/eval_checks/k_citations_binding.yaml @@ -0,0 +1,23 @@ +schema_version: 1 +id: k_citations_binding +type: harness_judge +description: Every K artifact records the evidence it rests on — citations are a binding writer output; uncited candidates are counted. +tags: [k, D45, D46] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D45/D46 and plan/designs/k_layers_design.md, then inspect the writer + output contract and knowledge_artifact_evidence under src/. These are acceptance checks for + the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. A writer's output is rejected/invalid without citations; citations are persisted in a + knowledge_artifact_evidence table linking artifacts to the evidence IDs they rest on. + 2. Uncited candidate evidence (routed to the page but not cited) is counted and surfaced — + not silently ignored. + 3. Authored pages declare their citations (and optional watch rules) in frontmatter that the + driver syncs to Postgres. + 4. Deletion reach and staleness for K are computed from these citation records (SQL over + recorded inputs), demonstrating the citations are load-bearing, not decorative. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/k_compiled_vs_authored_ownership.yaml b/plan/implementation_evals/eval_checks/k_compiled_vs_authored_ownership.yaml new file mode 100644 index 0000000..48fa949 --- /dev/null +++ b/plan/implementation_evals/eval_checks/k_compiled_vs_authored_ownership.yaml @@ -0,0 +1,24 @@ +schema_version: 1 +id: k_compiled_vs_authored_ownership +type: harness_judge +description: Two page kinds with an ownership contract — compiled bodies machine-owned with curation sidecars and quarantined human edits; authored pages never auto-rewritten, only flagged. +tags: [k, D46] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D46 and plan/designs/k_layers_design.md, then inspect the page-kind + handling under src/. These are acceptance checks for the FINAL system: if the subsystem is + absent or partial, score 0. + + Verify ALL of the following: + 1. Every K artifact is typed compiled or authored. + 2. Compiled pages: body machine-owned, regenerated when stale; human input enters only via a + per-page curation sidecar (pins/exclusions/corrections/guidance) that regeneration must + honor and can never destroy. + 3. A direct human edit to a compiled body is DETECTED via content_hash mismatch and + QUARANTINED into a proposed sidecar entry — never silently overwritten by the next + compile, never silently absorbed. + 4. Authored pages are never auto-regenerated; when evidence they cite changes they receive a + review FLAG (not a rewrite) — the system never rewrites human words, even to forget. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/k_one_mechanism_n_scopes.yaml b/plan/implementation_evals/eval_checks/k_one_mechanism_n_scopes.yaml new file mode 100644 index 0000000..2a4ead5 --- /dev/null +++ b/plan/implementation_evals/eval_checks/k_one_mechanism_n_scopes.yaml @@ -0,0 +1,27 @@ +schema_version: 1 +id: k_one_mechanism_n_scopes +type: harness_judge +description: One compile mechanism, N scopes — tiers are configuration; K3 is the stricter-config belief tier; the framework contract itself is not configurable. +tags: [k, D47] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D47 and plan/designs/k_layers_design.md §2, then inspect the scope/tier + configuration under src/. These are acceptance checks for the FINAL system: if the subsystem + is absent or partial, score 0. + + Verify ALL of the following: + 1. There is exactly ONE compile pipeline; K1/K2/K3 exist as configuration (scopes = git + subtree + registry rows + rules), not as separate machinery or code paths. + 2. Each scope has a shared model page that is a declared compile input of every page in the + scope. + 3. The belief tier (K3) is stricter configuration: rules select only settled evidence + (evidence_count >= N, no live contradiction_group, D54-flagged facts excluded), updates + are evidence-gated (never timer-driven), and every belief cites supporting AND + contradicting evidence. + 4. A deployment can reshape/rename/drop/invent scopes and tiers via configuration — but the + framework contract is NOT configurable: page kinds + ownership, binding citations, the + single automated committer, and trigger-surface acyclicity are enforced regardless of + configuration. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/k_planner_writer_driver.yaml b/plan/implementation_evals/eval_checks/k_planner_writer_driver.yaml new file mode 100644 index 0000000..b23ce33 --- /dev/null +++ b/plan/implementation_evals/eval_checks/k_planner_writer_driver.yaml @@ -0,0 +1,28 @@ +schema_version: 1 +id: k_planner_writer_driver +type: harness_judge +description: Plane K is compiled by planner/writer/driver — mechanical zero-LLM routing, dependency-ordered scheduling, one automated committer, no concurrent sessions. +tags: [k, D45] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D45 and plan/designs/k_layers_design.md, then inspect the K compile system + under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. + + Verify ALL of the following: + 1. Three roles exist and are separated: an LLM planner owning structure (pages + routing + rules, recorded as append-only knowledge_plan_decisions); LLM writers owning content (one + writer per page per cycle); a DETERMINISTIC driver owning staleness computation, + dependency-ordered scheduling (scope model page first, children before parents, root index + last), output validation, and commits. + 2. Routing rules are mechanical: a closed kind set (entity, entity_subtree, predicate_beat, + community, doc_set, scope_interests, manual) evaluated by SQL over E-plane keys via an + inverted key index — NO LLM call decides routing at evidence-arrival time. + 3. The driver is the repo's ONLY automated committer; there is no in-session merge-conflict + retry and no hot-file rolling-window machinery (removed by D45). + 4. Staleness is computed, never guessed: a page is stale iff its recorded inputs_hash no + longer matches — and inputs_hash keys on FACT STATE, never raw claim IDs (D54's + stale-storm guard); claim-grain citations key on (lineage, chunk_content_hash). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ops_deletion_cascade_grains.yaml b/plan/implementation_evals/eval_checks/ops_deletion_cascade_grains.yaml new file mode 100644 index 0000000..844c275 --- /dev/null +++ b/plan/implementation_evals/eval_checks/ops_deletion_cascade_grains.yaml @@ -0,0 +1,25 @@ +schema_version: 1 +id: ops_deletion_cascade_grains +type: harness_judge +description: Deletion has grains (version / lineage / hard-forget) and cascades through every derived layer — GCS objects, spine rows, projections, K reach — with facts closed per shape, recorded and reversible. +tags: [ops, D55, D46, requirements] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read requirements_v3.md §Knowledge lifecycle (deletion cascade), decisions.md D55/D46, and + plan/designs/e0_files_design.md §2, then inspect the deletion workers under src/. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Three grains exist: delete a version (currency ends, lineage continues), delete a lineage + (full cascade), hard-forget (indistinguishable-from-never-existed semantics). + 2. Lineage deletion removes raw + artifact objects and spine rows, cascades chunks -> claims + -> evidence links; solely-supported facts are closed per shape (states capped, + measurements invalidated), recorded and reversible — never silently vanished. + 3. Projections converge by rebuild (the document simply is not materialized next cycle), and + K is reached mechanically via citations: compiled pages recompile without the removed + evidence; authored pages get a review flag (never a rewrite). + 4. A file observed deleted at its source is treated as lineage deletion, stamped with the + observing sync cycle. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ops_execution_classes_bound.yaml b/plan/implementation_evals/eval_checks/ops_execution_classes_bound.yaml new file mode 100644 index 0000000..f9a84a6 --- /dev/null +++ b/plan/implementation_evals/eval_checks/ops_execution_classes_bound.yaml @@ -0,0 +1,25 @@ +schema_version: 1 +id: ops_execution_classes_bound +type: harness_judge +description: Every worker declares an execution class; agent harnesses exist only on plane K and review/audit seats; every LLM worker has an append-only transcript. +tags: [ops, D52, D33] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D52 and plan/analysis/workers.md + plan/designs/orchestration_design.md, + then inspect the worker implementations under src/. These are acceptance checks for the + FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Every worker is classified deterministic / programmatic-LLM / agent-harness (the schema's + pipeline enums carry a value for every worker). + 2. NO agent-harness worker exists on any per-document, per-claim, or query path — harnesses + appear only as K planner/writers and review/audit seats, each with a declared write + surface it cannot exceed. + 3. Every worker that makes an LLM call writes an append-only transcript/ledger + (extraction decisions, adjudications, compilations) — no LLM call without a durable + record. + 4. Programmatic-LLM workers sit inside cheap-first cascades (spend scales with ambiguity, + never volume). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ops_idempotency_dlq.yaml b/plan/implementation_evals/eval_checks/ops_idempotency_dlq.yaml new file mode 100644 index 0000000..dda0ba6 --- /dev/null +++ b/plan/implementation_evals/eval_checks/ops_idempotency_dlq.yaml @@ -0,0 +1,21 @@ +schema_version: 1 +id: ops_idempotency_dlq +type: harness_judge +description: Every worker is idempotent (content hash + processing version); failures never disappear — bounded retries then dead-letter into Postgres with recorded status. +tags: [ops, D12, D56] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D12 (and D56's finer keys) and plan/designs/orchestration_design.md, then + inspect the workers and queue handling under src/. These are acceptance checks for the FINAL + system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Every worker is re-runnable and keyed on a content-derived hash + its own processing + version — re-delivery or re-run of an unchanged input is a no-op. + 2. Retries are bounded (per the design: max 2) and exhausted work dead-letters into Postgres + with recorded status and enough context to inspect and replay — failures never vanish. + 3. The per-document E chain is task-queue driven; aggregate layers (K, P) are debounced or + scheduled, never triggered per document. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ops_metering_budgets_enforced.yaml b/plan/implementation_evals/eval_checks/ops_metering_budgets_enforced.yaml new file mode 100644 index 0000000..09a82cc --- /dev/null +++ b/plan/implementation_evals/eval_checks/ops_metering_budgets_enforced.yaml @@ -0,0 +1,21 @@ +schema_version: 1 +id: ops_metering_budgets_enforced +type: harness_judge +description: Per-layer cost metering exists and budgets are enforced, not advisory — exhaustion blocks/parks work rather than logging a warning. +tags: [ops, D52, requirements] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read requirements_v3.md §Operational properties (cost discipline) and decisions.md D52, then + inspect the cost ledgers and budget checks under src/. These are acceptance checks for the + FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Every LLM/embedding/OCR call records cost/tokens/model/version into a per-layer ledger in + Postgres. + 2. Budgets are enforced in code: a worker performs a budget preflight, and exhaustion + blocks or parks the work (a durable, resumable state) — it does not merely log. + 3. Spend is attributable per layer/stage/deployment (the metering supports the design's + dashboards and the cheap-first verification). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ops_producer_checker_families.yaml b/plan/implementation_evals/eval_checks/ops_producer_checker_families.yaml new file mode 100644 index 0000000..2beb60a --- /dev/null +++ b/plan/implementation_evals/eval_checks/ops_producer_checker_families.yaml @@ -0,0 +1,21 @@ +schema_version: 1 +id: ops_producer_checker_families +type: harness_judge +description: Every checking seat runs on a different model family than the producer it checks, auditable via recorded model assignments. +tags: [ops, D53] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D53, then inspect the model assignment configuration under src/. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Checking seats (the sampled grounding judge, contradiction/citation-faithfulness evals, + the review agent, the K reflection pass) are configured on a different model FAMILY than + the producer they check. + 2. Model assignments are recorded (pipeline_component_versions naming producer and checker + models) so the separation is auditable. + 3. Running a checker in the producer's family requires a recorded exception — it cannot + happen as a quiet config default. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ops_versioned_replay_from_storage.yaml b/plan/implementation_evals/eval_checks/ops_versioned_replay_from_storage.yaml new file mode 100644 index 0000000..0616798 --- /dev/null +++ b/plan/implementation_evals/eval_checks/ops_versioned_replay_from_storage.yaml @@ -0,0 +1,21 @@ +schema_version: 1 +id: ops_versioned_replay_from_storage +type: harness_judge +description: Every model-derived artifact records prompt/model/embedding versions, and every nondeterministic stage replays from storage on rebuild — never re-derives. +tags: [ops, D7, D33, requirements] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read requirements_v3.md §Operational properties (versioned processing) and decisions.md + D7/D33, then inspect version stamping and rebuild paths under src/. These are acceptance + checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Every derived artifact carries its producing versions (converter, structurer, blockizer, + extractor prompt+model, embedding model, resolver, compile prompt) as persisted columns. + 2. Rebuild/replay paths for every nondeterministic stage read stored outputs — no rebuild + path re-invokes an LLM/structurer/OCR to regenerate what storage already holds. + 3. A version bump reprocesses exactly the affected artifacts (version-filtered batch), and + downstream invalidation keys include upstream versions. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/p1_lance_estate_rebuildable.yaml b/plan/implementation_evals/eval_checks/p1_lance_estate_rebuildable.yaml new file mode 100644 index 0000000..924be94 --- /dev/null +++ b/plan/implementation_evals/eval_checks/p1_lance_estate_rebuildable.yaml @@ -0,0 +1,21 @@ +schema_version: 1 +id: p1_lance_estate_rebuildable +type: harness_judge +description: P1 is one Lance estate (chunks, claims, relation labels, observation labels) written inline by E workers but fully rebuildable by batch; every embedding stamped with its version. +tags: [p1, D8, D9, D7] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D8/D9 and overall_design.md §4-5, then inspect the P1 writers and rebuild + path under src/. These are acceptance checks for the FINAL system: if the subsystem is + absent or partial, score 0. + + Verify ALL of the following: + 1. Lance tables exist for chunks, claims, relation fact-labels (and observation labels), with + the scalar columns the recipes filter on. + 2. E-plane workers write P1 inline as artifacts land, but a batch rebuild path from Postgres + exists and is exercised (tests/drills) — P1 carries no authority. + 3. Every embedded row is stamped with its embedding model/version, and re-embedding by + version filter is a supported batch operation (the embedding-migration path). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/p2_asof_and_communities.yaml b/plan/implementation_evals/eval_checks/p2_asof_and_communities.yaml new file mode 100644 index 0000000..414b1b5 --- /dev/null +++ b/plan/implementation_evals/eval_checks/p2_asof_and_communities.yaml @@ -0,0 +1,22 @@ +schema_version: 1 +id: p2_asof_and_communities +type: harness_judge +description: As-of traversal is inline path-predicate filtering (never MATCH over PROJECT_GRAPH); community detection runs externally and writes back to Postgres. +tags: [p2, D10, D11, D44] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D10/D11/D44, then inspect the graph query layer and the community pass + under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. + + Verify ALL of the following: + 1. Temporal as-of graph queries filter edges inline in path predicates over the four temporal + columns (or use a materialized persistent graph for heavy analytics) — no code attempts + MATCH-traversal over PROJECT_GRAPH (which feeds GDS algorithms only). + 2. Community detection (Louvain/Leiden-class) runs as an external pass (igraph/graspologic) + over the same export that feeds the rebuild — not inside LadybugDB. + 3. Community assignments and centrality write back to POSTGRES (the graph stays a + projection); graph-derived metrics are computed post-load, never reprojected as truth. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/p2_no_vectors_in_graph.yaml b/plan/implementation_evals/eval_checks/p2_no_vectors_in_graph.yaml new file mode 100644 index 0000000..1a15685 --- /dev/null +++ b/plan/implementation_evals/eval_checks/p2_no_vectors_in_graph.yaml @@ -0,0 +1,21 @@ +schema_version: 1 +id: p2_no_vectors_in_graph +type: harness_judge +description: The graph holds no embeddings and no FTS — vectors live in Lance; relation fact-labels are embedded in a Lance table with scalar columns. +tags: [p2, p1, D8] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D8, then inspect the P2 snapshot contents and the Lance table definitions + under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. + + Verify ALL of the following: + 1. No vector index, no embedding column, and no FTS index exists in the LadybugDB snapshot. + 2. Each relation has a canonical fact label embedded in a Lance relations table keyed by + relation_id, with scalar columns (subject_id, predicate, object_id, validity window, + evidence_count) for filtered hybrid search. + 3. The retrieval flow is Lance-entry -> ID-keyed expansion in the graph (top-k relation_ids + then BFS/paths), not in-graph semantic search. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/p2_projection_contract_views.yaml b/plan/implementation_evals/eval_checks/p2_projection_contract_views.yaml new file mode 100644 index 0000000..5ae357e --- /dev/null +++ b/plan/implementation_evals/eval_checks/p2_projection_contract_views.yaml @@ -0,0 +1,30 @@ +schema_version: 1 +id: p2_projection_contract_views +type: harness_judge +description: The Postgres->LadybugDB boundary is the v_graph_* views with merge-redirect survivor resolution, keep-retracted windows, preserved parallel edges; observations and claims never project. +tags: [p2, D44] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D44 and plan/designs/postgres_schema_design.md §10.A, then inspect the + projection views and the graph build under src/. These are acceptance checks for the FINAL + system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. The projection is defined by read-only Postgres views (v_graph_entities, v_graph_documents, + v_graph_relates, v_graph_mentioned_in, v_graph_crossref, v_graph_is_document, and a shared + v_graph_survivor); mechanical casts (timestamptz -> naive UTC, enums -> text, dropped + columns) live in the views — the single auditable boundary. + 2. Merge-redirect is honored: relation endpoints are recursively resolved through + entities.merged_into to the surviving entity, cycle-safe, with a pre-snapshot validation + gate that aborts on cycles or dangling endpoints (a naive WHERE status='active' projection + is a violation). + 3. Retracted edges are kept within a retention window (transaction-time as-of), with + node/edge retention aligned; parallel edges (distinct relation_id) are preserved — no + blind DISTINCT collapse. + 4. The graph model is one Entity node + one Document node + a generic RELATES table with + predicate as a property (+ structural MENTIONED_IN / DOC_CROSSREF / IS_DOCUMENT) — not + per-type node tables or per-predicate rel tables; entity IDs stay UUID. + 5. Observations and claims never project into the graph. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml b/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml new file mode 100644 index 0000000..0bfa9fc --- /dev/null +++ b/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml @@ -0,0 +1,23 @@ +schema_version: 1 +id: p2_rebuild_first_snapshots +type: harness_judge +description: P2 is rebuild-first — full rebuild, validate, publish an immutable snapshot, pointer-swap; readers are read-only and hot-swap; no incremental event application. +tags: [p2, D7] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D7 and plan/designs/p2_graph_design.md, then inspect the P2 worker and + reader under src/. These are acceptance checks for the FINAL system: if the subsystem is + absent or partial, score 0. + + Verify ALL of the following: + 1. The graph worker periodically rebuilds the ENTIRE graph from a Postgres export (Parquet + hop as the committed transport), validates it, and publishes an immutable versioned + snapshot with a write-then-pointer-swap. + 2. Readers download the latest snapshot, open READ_ONLY, and hot-swap on updates — never a + writable shared graph. + 3. There is NO incremental event application to a live graph (a deliberate non-goal) — + entity merges are handled by the rebuild, not by re-pointing edges in the graph. + 4. P3 follows the same discipline: full snapshot build + atomic latest-pointer swap (D40). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/p3_corpus_fs_projection.yaml b/plan/implementation_evals/eval_checks/p3_corpus_fs_projection.yaml new file mode 100644 index 0000000..776721b --- /dev/null +++ b/plan/implementation_evals/eval_checks/p3_corpus_fs_projection.yaml @@ -0,0 +1,24 @@ +schema_version: 1 +id: p3_corpus_fs_projection +type: harness_judge +description: P3 is a rebuildable corpus-filesystem projection — generated stubs + _index.md/llms.txt, built from Postgres + artifacts; K pages cross-linked, never a structural input; paths anchor on lineages. +tags: [p3, D40, D55] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D40 (and D55's stability contract) and plan/designs/e0_files_design.md §6 + plus plan/analysis/p3_agent_navigation.md, then inspect the P3 builder under src/. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. The tree is materialized into versioned snapshots with an atomic latest-pointer swap; + mounts read latest read-only. + 2. Leaves are GENERATED stub files with frontmatter (doc_id, artifact_uri, content_hash, + pointers) — never symlinks; every level carries a generated _index.md and llms.txt. + 3. The tree structure is built from Postgres (placement hints + entities/relations) + the + GCS artifacts; K pages are cross-linked by reference but are NOT a structural input (P3 + stays rebuildable from the E spine alone). + 4. P3 paths and K citations anchor on document LINEAGES (stable across versions/renames), so + an edited document keeps its path. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml b/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml new file mode 100644 index 0000000..c709b91 --- /dev/null +++ b/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml @@ -0,0 +1,22 @@ +schema_version: 1 +id: p_projections_hold_no_authority +type: harness_judge +description: Validity/invalidation state has exactly one home (Postgres); P1/P2/P3 hold derived, rebuildable copies that are never independently mutated. +tags: [p, D6] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D6, then inspect every write path into Lance, LadybugDB, and the corpus + filesystem under src/. These are acceptance checks for the FINAL system: if the subsystem is + absent or partial, score 0. + + Verify ALL of the following: + 1. No code path mutates validity/invalidation/contradiction state in a projection store — + Lance/Ladybug/P3 receive filtered copies from Postgres and are never the site of a + decision. + 2. Every projection is deletable and rebuildable from Postgres (+ artifacts) by a tested + batch path; nothing in a projection is source-of-truth. + 3. The graph writer is dumb and deterministic — no adjudication, no supersession logic, no + LLM calls inside projection builders. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_claims_never_answer_now.yaml b/plan/implementation_evals/eval_checks/ret_claims_never_answer_now.yaml new file mode 100644 index 0000000..864c52e --- /dev/null +++ b/plan/implementation_evals/eval_checks/ret_claims_never_answer_now.yaml @@ -0,0 +1,23 @@ +schema_version: 1 +id: ret_claims_never_answer_now +type: harness_judge +description: The claim/fact temporal split is enforced in the API surface — claim queries answer "what was asserted"; current-truth routes only through validity-filtered relations and observations. +tags: [retrieval, D41, D49, requirements] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read requirements_v3.md §Retrieval (the temporal-split paragraph), decisions.md D41/D49, + then inspect recipe definitions and naming under src/. These are acceptance checks for the + FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Claim-grain recipes (e.g. claims_as_of) are typed evidence-grain and structurally CANNOT + be composed into a current-fact answer (the registry linter rejects answer_intent = + current_facts over claim primitives). + 2. "Is this true now / was this true at T as current belief" routes exclusively through + validity-filtered relations AND observations (both fact layers, D43). + 3. Recipe/API naming makes the distinction explicit to an agent consumer (asserted vs + believed), matching the requirement that an agent must never read a claim's validity as + the system's current belief. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_envelope_contract.yaml b/plan/implementation_evals/eval_checks/ret_envelope_contract.yaml new file mode 100644 index 0000000..420ab26 --- /dev/null +++ b/plan/implementation_evals/eval_checks/ret_envelope_contract.yaml @@ -0,0 +1,29 @@ +schema_version: 1 +id: ret_envelope_contract +type: harness_judge +description: Every retrieval response is an envelope — declared grain, contradiction co-members never silently absent, per-source freshness stamps with believed_at horizons, explicit truncation, typed negatives. +tags: [retrieval, D49] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D49 and plan/designs/retrieval_design.md, then inspect the envelope types + and their enforcement under src/. These are acceptance checks for the FINAL system: if the + subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Responses carry a declared grain (fact/evidence/compiled/composite) enforced at + composition: current-fact answers assemble only from validity-filtered + relations/observations; claims can never answer "is it true now"; composite answers are + parts[], each strictly single-grain. + 2. Contradiction co-members are inline up to a guaranteed cap, beyond which the block always + carries group_id + returned/total + a continuation — code cannot return a one-sided + answer for a contradicted fact. + 3. Per-source freshness stamps ship in the envelope (PG live; P1 write lag; P2 snapshot + timestamp + believed_at horizon with a typed boundary fallback; K compiled_at + staleness + + open-flag count). + 4. Truncation is always marked with a continuation (no silent top-k); the applied temporal + parameters and identity regime (current aliases vs identity_as_of) are echoed. + 5. A typed negative taxonomy exists: unknown_entity / known_empty / boundary — and there is + NO denied kind; hard-forgotten content is indistinguishable from never-existed. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_filesystem_first_mounts_skill.yaml b/plan/implementation_evals/eval_checks/ret_filesystem_first_mounts_skill.yaml new file mode 100644 index 0000000..fd4c56c --- /dev/null +++ b/plan/implementation_evals/eval_checks/ret_filesystem_first_mounts_skill.yaml @@ -0,0 +1,26 @@ +schema_version: 1 +id: ret_filesystem_first_mounts_skill +type: harness_judge +description: Consumption is filesystem-first — four read-only mounts (raw off-path, audit-logged, mime-routed), full mount/API parity, and a versioned consumption skill ships with the system. +tags: [retrieval, D51] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D51 and plan/designs/retrieval_design.md, then inspect the mount + publication and the shipped skill under src/. These are acceptance checks for the FINAL + system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Four read-only surfaces are mountable: P3, E0 artifacts (Markdown + structure + derived + media), E0 raw originals, and the K repo checkout. + 2. Raw is OFF the navigation path — reached only via explicit pointers from stubs/frontmatter + — with mandatory data-access audit logging and mime-routed storage classes + (agent-readable media standard/nearline; audit-only originals archive). + 3. Full mount/API parity: everything readable through mounts is also reachable via API/CLI + (including byte fetches by artifact handle) for environments that cannot mount. + 4. A consumption skill ships with the system — versioned, partially rendered per deployment + (scopes, mounts, enabled recipes) — teaching planes, grains, contradiction/freshness + semantics, the mount layout, and the precedence rule (filesystem for what a filesystem + can do; API for search/graph/as-of/hydration). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_hydration_reverifies.yaml b/plan/implementation_evals/eval_checks/ret_hydration_reverifies.yaml new file mode 100644 index 0000000..434296a --- /dev/null +++ b/plan/implementation_evals/eval_checks/ret_hydration_reverifies.yaml @@ -0,0 +1,25 @@ +schema_version: 1 +id: ret_hydration_reverifies +type: harness_judge +description: Projections propose, the spine disposes — every query-engine result hydrates by-ID against live Postgres; drops are counted; compound results revalidate as units. +tags: [retrieval, D48] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D48 and plan/designs/retrieval_design.md, then inspect the query engine + under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. + + Verify ALL of the following: + 1. Every API/CLI/MCP result passes through by-ID hydration against live Postgres before + reaching the caller — P1/P2 hits only NOMINATE candidates; validity, invalidation, and + contradiction membership are re-read from the spine. + 2. Candidates the spine no longer holds live are dropped, and the drop count is reported in + the response envelope. + 3. Compound results (e.g. graph paths) revalidate as units: one invalidated edge drops the + whole path — it is never returned with a hole or silently re-routed. + 4. The two exempt surfaces are exactly the designed ones: mounted reads (snapshot semantics + + freshness metadata) and K prose (compiled-grain with staleness state) — nothing else skips + hydration. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_primitives_recipes_registry.yaml b/plan/implementation_evals/eval_checks/ret_primitives_recipes_registry.yaml new file mode 100644 index 0000000..8cd651b --- /dev/null +++ b/plan/implementation_evals/eval_checks/ret_primitives_recipes_registry.yaml @@ -0,0 +1,27 @@ +schema_version: 1 +id: ret_primitives_recipes_registry +type: harness_judge +description: The query machine is typed zero-LLM primitives + recipes as registry rows (output_grain/answer_intent enforced); recipes add convenience, never capability; no NL->plan compiler. +tags: [retrieval, D50, D9] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D50 and plan/designs/retrieval_design.md, then inspect the primitives and + recipe registry under src/. These are acceptance checks for the FINAL system: if the + subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Primitives are typed, orthogonal, side-effect-free, zero-LLM operations (resolve with + non-LLM tiers only — no T4 on the hot path, ranked candidates never a silent guess; + lookup; search; graph; fuse as an explicit RRF operator; rerank; hydrate; transcript; + delta; pages_about; enumerated aggregate; streaming scan on a separate resource pool). + 2. Recipes are REGISTRY ROWS (name, description, typed parameters, typed primitive chain, + output_grain and answer_intent enums, version) — not code; a linter/validation enforces + grain semantics mechanically on the enums at registration; MCP tools render from the + registry. + 3. Each recipe replays as its primitive chain with an empty diff (testable: recipes add + convenience, never capability). + 4. There is no natural-language-to-query-plan compiler anywhere on the query path, and no + LLM call in the core search path (D9). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_trust_boundary_no_content_auth.yaml b/plan/implementation_evals/eval_checks/ret_trust_boundary_no_content_auth.yaml new file mode 100644 index 0000000..7a0e502 --- /dev/null +++ b/plan/implementation_evals/eval_checks/ret_trust_boundary_no_content_auth.yaml @@ -0,0 +1,22 @@ +schema_version: 1 +id: ret_trust_boundary_no_content_auth +type: harness_judge +description: One trust domain per deployment — no content-level authorization or per-user scoping anywhere inside the library; perimeter auth only, via a swappable seam. +tags: [retrieval, D50, D60, D61] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D50 (trust model) and D60/D61, then inspect the API/mount surfaces under + src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. + + Verify ALL of the following: + 1. No query path, mount builder, or projection filters content by user/role/tenant — a + deployment is one trust domain; isolation is deployment separation, never in-library + content filtering. + 2. Perimeter authentication exists as a swappable seam (API-key middleware in the library; + replaceable per D61's auth-perimeter port) — and nothing deeper. + 3. There is no orgs/users/RBAC/SSO machinery in this repository (D60: that is the cloud + product). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/questions.md b/questions.md index f7dfa46..08564a1 100644 --- a/questions.md +++ b/questions.md @@ -172,6 +172,17 @@ Keep this current: when something here is decided, move it to a decision and pru 29. **E3 claim→predicate mapping is thin.** §5 delegates internals to the registries but doesn't specify how the governed predicate is *chosen* from claim text or where domain/range is enforced in the flow. +30. **Requirements temporal-split paragraph predates D43/D49.** `requirements_v3.md` §Retrieval says + "All supersession / current-fact validity **lives only on relations**" and cites D3/D6/D41 — but + observations carry the same adjudicated validity windows and supersession (D43), and D49's grain + discipline assembles current-fact answers from validity-filtered **relations and observations**. + Update the paragraph (and its citation list) to name both fact layers. (Found while authoring + `plan/implementation_evals/`.) +31. **Requirements E3 bullet omits observations.** `requirements_v3.md` §Plane E describes E3 as + relations only ("distinct facts `(subject, predicate, object)` … the unit of supersession and + contradiction") — no mention of observations (D43) or stance observations (D59), though the + README's E3 row and the designs treat them as a canonical fact layer. Update the bullet. (Found + while authoring `plan/implementation_evals/`.) ## Resolved since the last version of this file (moved to decisions) From a56472bb32f0823036479230e991a1bb7f44a854 Mon Sep 17 00:00:00 2001 From: Jiri Puc Date: Fri, 10 Jul 2026 00:12:47 +0200 Subject: [PATCH 2/7] =?UTF-8?q?Implementation=20evals:=20apply=20Codex=20r?= =?UTF-8?q?eview=20=E2=80=94=204=20factual=20fixes,=20splits,=20+16=20chec?= =?UTF-8?q?ks=20(69=20total)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codex (gpt-5.5) reviewed the initial 53-check set (codex_review.md, committed alongside). Applied in full: - Factual must-fixes: deletion check now matches normal-delete retention semantics (claims/evidence kept as audit history; only hard-forget scrubs); execution-classes check names E2 extraction as the deliberate volume-proportional exception (D25/D31); Selection ledger check matches schema (plain keeps not ledgered); bitemporal check carries the D43 no-cap rule for measurements. - Judgeability: split the omnibus checks (boundary D60 -> 3, retrieval envelope D49 -> 5, lineage/living D55 -> 2); reworded checks to demand concrete evidence (shared hydrator boundary, import-linter contracts, startup/CI family validator, targeted reuse tests). - Coverage: +9 checks — D1/D46 source-of-truth split, D16 scope views, D22 golden sets, D23 registry scale, D36 sub-worker chain, code tooling/migrations, D62 delivery artifacts + enforced architecture, D63 embedding port config. - Mechanics: missing decision tags added, cite paths fixed, README re-inventoried (69 checks, scope D1-D63). Co-Authored-By: Claude Fable 5 --- plan/implementation_evals/README.md | 54 ++-- plan/implementation_evals/codex_review.md | 238 ++++++++++++++++++ .../boundary_correctness_in_library_d60.yaml | 18 ++ ...undary_extension_points_no_bypass_d60.yaml | 22 ++ .../boundary_library_scope_d60.yaml | 21 -- .../boundary_no_human_control_plane_d60.yaml | 18 ++ .../code_architecture_enforced_d62.yaml | 22 ++ .../code_tooling_and_migrations.yaml | 22 ++ ...elivery_artifacts_and_task_shells_d62.yaml | 29 +++ .../e0_content_addressed_reuse.yaml | 28 ++- .../e0_lineage_version_schema.yaml | 25 ++ .../e0_lineages_versions_modes.yaml | 31 --- .../e0_living_mode_retraction.yaml | 26 ++ .../e0_origin_stamped_at_ingest.yaml | 5 +- .../e0_subworker_chain_crossrefs_d36.yaml | 22 ++ .../e1_multigranularity_retrieval.yaml | 14 +- .../e2_asserted_validity_immutable.yaml | 2 +- .../e2_context_bundle_two_calls.yaml | 7 +- .../e2_grounding_layered_dual_field.yaml | 2 +- .../e2_selection_ledger_replay.yaml | 15 +- .../e3_blocking_cheap_first_cascade.yaml | 2 +- .../e3_observations_untyped_adjudicated.yaml | 17 +- ...e3_supersession_fact_level_bitemporal.yaml | 25 +- .../embedding_model_port_config_d63.yaml | 26 ++ .../er_resolution_cascade_t0_t4.yaml | 2 +- .../eval_checks/eval_golden_sets_d22.yaml | 26 ++ .../eval_checks/k_citations_binding.yaml | 5 +- .../eval_checks/k_one_mechanism_n_scopes.yaml | 2 +- .../ops_deletion_cascade_grains.yaml | 34 +-- .../ops_execution_classes_bound.yaml | 9 +- .../eval_checks/ops_idempotency_dlq.yaml | 24 +- .../ops_producer_checker_families.yaml | 11 +- .../p1_lance_estate_rebuildable.yaml | 2 +- .../p2_rebuild_first_snapshots.yaml | 2 +- .../p_projections_hold_no_authority.yaml | 22 +- .../registry_scale_partitioning_d23.yaml | 24 ++ .../eval_checks/ret_envelope_contract.yaml | 29 --- ...t_envelope_contradiction_completeness.yaml | 19 ++ .../ret_envelope_freshness_stamps.yaml | 22 ++ .../ret_envelope_grain_discipline.yaml | 21 ++ .../ret_envelope_negative_taxonomy.yaml | 22 ++ .../ret_envelope_truncation_markers.yaml | 19 ++ .../eval_checks/ret_hydration_reverifies.yaml | 15 +- .../scope_views_share_graph_d16.yaml | 23 ++ .../source_of_truth_split_d1_d46.yaml | 22 ++ 45 files changed, 838 insertions(+), 208 deletions(-) create mode 100644 plan/implementation_evals/codex_review.md create mode 100644 plan/implementation_evals/eval_checks/boundary_correctness_in_library_d60.yaml create mode 100644 plan/implementation_evals/eval_checks/boundary_extension_points_no_bypass_d60.yaml delete mode 100644 plan/implementation_evals/eval_checks/boundary_library_scope_d60.yaml create mode 100644 plan/implementation_evals/eval_checks/boundary_no_human_control_plane_d60.yaml create mode 100644 plan/implementation_evals/eval_checks/code_architecture_enforced_d62.yaml create mode 100644 plan/implementation_evals/eval_checks/code_tooling_and_migrations.yaml create mode 100644 plan/implementation_evals/eval_checks/delivery_artifacts_and_task_shells_d62.yaml create mode 100644 plan/implementation_evals/eval_checks/e0_lineage_version_schema.yaml delete mode 100644 plan/implementation_evals/eval_checks/e0_lineages_versions_modes.yaml create mode 100644 plan/implementation_evals/eval_checks/e0_living_mode_retraction.yaml create mode 100644 plan/implementation_evals/eval_checks/e0_subworker_chain_crossrefs_d36.yaml create mode 100644 plan/implementation_evals/eval_checks/embedding_model_port_config_d63.yaml create mode 100644 plan/implementation_evals/eval_checks/eval_golden_sets_d22.yaml create mode 100644 plan/implementation_evals/eval_checks/registry_scale_partitioning_d23.yaml delete mode 100644 plan/implementation_evals/eval_checks/ret_envelope_contract.yaml create mode 100644 plan/implementation_evals/eval_checks/ret_envelope_contradiction_completeness.yaml create mode 100644 plan/implementation_evals/eval_checks/ret_envelope_freshness_stamps.yaml create mode 100644 plan/implementation_evals/eval_checks/ret_envelope_grain_discipline.yaml create mode 100644 plan/implementation_evals/eval_checks/ret_envelope_negative_taxonomy.yaml create mode 100644 plan/implementation_evals/eval_checks/ret_envelope_truncation_markers.yaml create mode 100644 plan/implementation_evals/eval_checks/scope_views_share_graph_d16.yaml create mode 100644 plan/implementation_evals/eval_checks/source_of_truth_split_d1_d46.yaml diff --git a/plan/implementation_evals/README.md b/plan/implementation_evals/README.md index 4b6a2f9..7709330 100644 --- a/plan/implementation_evals/README.md +++ b/plan/implementation_evals/README.md @@ -2,14 +2,15 @@ An exhaustive set of [eval-banana](https://github.com/writeitai/eval-banana) `harness_judge` checks that verify the **final implementation honors the binding design** — the decision log -(D1–D61), the requirements, and the design docs. Each check points an LLM judge at the binding -sources and at the code, states the invariant as concrete conditions, and demands a binary -verdict. +(**D1–D63**), the requirements, and the design docs. Each check points an LLM judge at the +binding sources and at the code, states the invariant as concrete conditions, and demands a +binary verdict. **What these are:** acceptance checks for design conformance — "does the code do what the design binds it to do?" **What these are not:** quality metrics. Extraction precision, resolution P/R curves, retrieval recall@k live in the D22 golden-set eval harness (a separate, -data-driven asset); these checks verify the *machinery and its invariants exist and hold*. +data-driven asset — itself covered here by `eval_golden_sets_d22`); these checks verify the +*machinery and its invariants exist and hold*. ## Running @@ -38,34 +39,41 @@ Two conventions when running: with file paths cited in the reason. - `tags` carry the area and the decision numbers, for filtered runs (e.g. everything tagged `D43`, or all of `e2`). +- Where an invariant is a *deliberate exception* (e.g. E2 extraction is volume-proportional by + D25 while adjudication is ambiguity-scaled by D4), the check says so explicitly, so a judge + cannot fail a correct implementation for obeying the design. -## Inventory (53 checks) +## Inventory (69 checks) -| Area (prefix) | Checks | Binding sources covered | +| Area | Checks | Binding sources covered | |---|---|---| -| `e0_*` — documents/files | 8 | D36–D39, D42, D51 (raw mount), D54–D57 | -| `e1_*` — blocks/chunks | 2 | D57, D58, D25 | -| `e2_*` — claims/extraction | 9 | D2, D3, D19, D25, D31–D35, D41, D54, D59 | -| `e3_*` — relations/observations | 5 | D2–D5, D15, D18, D43 | -| `er_*` — entity resolution/review | 3 | D17, D20, D21, D24 | -| `k_*` — knowledge plane | 4 | D45–D47, D54 | -| `p_*`, `p1_*`, `p2_*`, `p3_*` — projections | 7 | D6–D11, D40, D44, D55 | -| `ret_*` — retrieval | 6 | D9, D41, D48–D51 | -| `ops_*` — orchestration/cross-cutting | 6 | D7, D12, D33, D52, D53, D55, requirements | -| `code_*`, `boundary_*` — conventions/scope | 3 | D60, D61, requirements §Code | +| E0 documents/files (`e0_*`) | 10 | D36–D39, D42, D51, D54–D57 | +| E1 blocks/chunks (`e1_*`) | 2 | D57, D58 | +| E2 claims/extraction (`e2_*`) | 9 | D2, D3, D19, D25, D31–D35, D41, D54, D59, D63 | +| E3 relations/observations (`e3_*`) | 5 | D2–D5, D15, D18, D43, D55 | +| Registries/ER + eval (`er_*`, `registry_*`, `eval_*`) | 5 | D17, D20–D24 | +| K plane (`k_*`) | 4 | D45–D47, D54 | +| Projections (`p_*`, `p1_*`, `p2_*`, `p3_*`, `embedding_*`) | 8 | D6–D11, D40, D44, D55, D61, D63 | +| Retrieval (`ret_*`) | 10 | D9, D41, D48–D51 | +| Ops/cross-cutting (`ops_*`, `source_of_truth_*`, `scope_views_*`) | 8 | D1, D7, D12, D16, D33, D46, D52–D56, D62 | +| Code & boundary (`code_*`, `boundary_*`, `delivery_*`) | 8 | D60–D62, requirements §Code | Not every decision needs its own check: withdrawn decisions (D26–D30), naming/plane decisions (D13, D14), and pure-analysis decisions are covered implicitly or are not implementation- observable. Where one check verifies several decisions, the tags say so. +## Review + +`codex_review.md` records an independent Codex (gpt-5.5) review of the initial 53-check set. +**The current set incorporates that review**: its four factual must-fixes (deletion retention +semantics, the E2 volume-proportional exception, ledger scope, the observations no-cap rule), +the judgeability rewrites and splits (the boundary and envelope omnibus checks, lineage/living +split), the mechanics fixes (tags, cite paths), and nine added checks closing its coverage gaps +(D1/D46, D16, D22, D23, D36, code tooling, D62 ×2, D63). + ## Inconsistencies found while authoring Authoring these against the full corpus surfaced inconsistencies, registered in `questions.md` §5 (the repo's fix register): the requirements' temporal-split paragraph -predates D43 (observations also carry adjudicated validity), and the requirements' E3 bullet -omits observations entirely. See `questions.md` items 30–31. - -## Review - -`codex_review.md` in this directory records an independent Codex review of the set (accuracy -against the designs, coverage gaps, judge-ability). +predates D43/D49 (observations also carry adjudicated validity), and the requirements' E3 +bullet omits observations entirely. See `questions.md` items 30–31. diff --git a/plan/implementation_evals/codex_review.md b/plan/implementation_evals/codex_review.md new file mode 100644 index 0000000..46ffeed --- /dev/null +++ b/plan/implementation_evals/codex_review.md @@ -0,0 +1,238 @@ +# Codex Review of `plan/implementation_evals` + +Verdict: the suite is directionally strong and covers most of the final-system invariants, but it is not yet safe as an acceptance gate. Several checks over-claim binding decisions, especially around deletion, extraction ledgers, execution classes, and relation/observation temporal semantics. Those must be fixed before these are handed to binary LLM judges, because a correct implementation could fail for obeying the actual design. + +## Must Fix + +1. **Fix `ops_deletion_cascade_grains`: it contradicts the deletion design.** The check says lineage deletion "cascades chunks -> claims -> evidence links"; the binding schema says normal delete keeps chunks, claims, and evidence links as audit history. `postgres_schema_design.md` section 13.1: "`claims` are NOT deleted on a normal delete"; `relation_evidence` / `observation_evidence` rows are "retained as historical links." `evidence_lifecycle_design.md` section 8 likewise says claims are "retained as history" and only hard-forget scrubs content. +2. **Fix `ops_execution_classes_bound`: condition 4 is false for E2 extraction.** D52 describes execution classes, but D31/D25 bind E2 extraction as a two-call programmatic LLM over every chunked document. That spend is volume-proportional by design. Do not require every programmatic-LLM worker to sit inside an ambiguity-only cheap-first cascade. +3. **Fix `e2_selection_ledger_replay`: it overstates what the ledger records.** D33 says "Every Selection drop ... and every decontextualization edit" is ledgered. The schema explicitly says plain keeps are not recorded; `claim_extraction_decisions` records `selection_drop`, `selection_keep_flagged`, and `decontext_edit`. +4. **Fix `e3_supersession_fact_level_bitemporal`: it overstates valid-time capping for observations.** D43's no-cap rule says measurement/fixed-period observations are never capped on valid-time; conflicting same-period figures coexist, and source-removal belief withdrawal uses `invalidated_at`. +5. **Add checks for D22 and D23.** The suite does not adequately cover the golden-set/eval plan or the registry scale/indexing rules, both of which are binding and implementation-observable. +6. **Split the biggest omnibus checks before using binary scoring.** Several checks combine schema, worker behavior, runtime config, tests, and global absence claims. A single root defect would double-fail multiple checks, and a judge will struggle to prove all paths statically. + +## Factual Accuracy + +### Incorrect or Over-Claimed Checks + +**`ops_deletion_cascade_grains.yaml`** + +Condition 2 is wrong. It asserts lineage deletion removes "spine rows" and cascades `chunks -> claims -> evidence links`. The binding design says normal deletion purges raw/artifact objects and removes the document's current contribution, but retains audit rows: + +- `postgres_schema_design.md` section 13.1: deleting a lineage soft-tombstones `documents` and `document_versions`; `chunks` rows are retained; "`claims` are NOT deleted on a normal delete"; `relation_evidence` / `observation_evidence` are retained as historical links. +- `evidence_lifecycle_design.md` section 8: "Claims are retained as history ... their currency ends." + +Suggested replacement: + +```text +Normal deletion at version or lineage grain purges raw/artifact bytes and tombstones the lineage/version rows, ends testimony currency for affected claims, recomputes counts, and closes solely-supported facts per shape. Chunks, claims, evidence links, relations, and entities are retained as audit history unless the operation is hard-forget; hard-forget separately scrubs source-bearing payloads so forgotten content is indistinguishable from never-existed. +``` + +**`ops_execution_classes_bound.yaml`** + +Condition 4 says all programmatic-LLM workers sit inside cheap-first cascades and spend scales with ambiguity, never volume. That is not true for the E2 extractor. D31 binds a two-call extraction over the context bundle; D25 says every document that survives chunking is fully extracted. `workers.md` calls `extract_claims` "the volume cost center." + +Suggested replacement: + +```text +Programmatic-LLM workers are fixed-shape, schema-constrained, transcripted, and budgeted. Resolution/adjudication workers use cheap-first cascades so LLM spend scales with ambiguity; E2 extraction is the deliberate volume-proportional exception and must remain a fixed two-call extractor, not an agent harness. +``` + +**`e2_selection_ledger_replay.yaml`** + +Condition 1 says the ledger records "every Selection outcome." D33 only binds drops and decontextualization edits. `postgres_schema_design.md` section 8 narrows this further: the ledger records drops, low-confidence keeps, and decontextualization edits; "Plain keeps are NOT recorded (they ARE the claims row)." + +Suggested replacement: + +```text +An append-only, version-stamped ledger records every Selection drop, every low-confidence kept_flagged outcome, and every decontextualization edit. Plain keeps are represented by the inserted claim row and need not have separate ledger rows. +``` + +**`e3_supersession_fact_level_bitemporal.yaml`** + +Condition 1 says supersession caps `valid_until` on "relations and observations." For observations this is only true for changing effective states. D43 states the no-cap rule: a measurement/fixed-period figure is never capped on valid-time; same-period conflicts coexist in a `contradiction_group`. D55/D54 source-removal handling invalidates measurement observations rather than capping their valid-time. + +Suggested replacement: + +```text +Relations and effective-state observations carry bi-temporal windows and supersession caps valid_until without touching claims. Measurement/fixed-period observations obey D43's no-cap rule: same-period conflicts coexist, and source-removal belief withdrawal uses invalidated_at rather than ending the measured period. +``` + +**`k_citations_binding.yaml`** + +Mostly correct, but condition 1 says `knowledge_artifact_evidence` links artifacts to "evidence IDs." The actual schema allows exactly one of `claim_id`, `relation_id`, or `doc_id`. This is probably intended, but the wording should not imply a single generic evidence table or claim-only evidence. Reword to "claim, relation, or document IDs." + +**`e0_origin_stamped_at_ingest.yaml`** + +Condition 3 says the origin "survives on the lineage/version rows." D42 binds an immutable E0 origin stamp; the schema stores `origin` on `documents` (the lineage), not necessarily on every `document_versions` row. Reword to avoid requiring duplication: + +```text +The origin is stored on the durable document lineage/input metadata and remains reachable from all versions and downstream provenance. +``` + +**`ops_idempotency_dlq.yaml`** + +Condition 1 says every worker is keyed on a content-derived hash plus processing version. That fits per-document and chunk workers, but aggregate workers use recorded input/snapshot hashes or processing-state target/version keys. D12's spirit is idempotent, versioned rerunnability; D45 makes K `inputs_hash` the aggregate equivalent. Reword to allow content-derived keys where applicable and manifest/snapshot/input hashes for aggregate jobs. + +### Mostly Accurate Checks + +The following checks accurately reflect the cited decisions, with no factual issue I would block on: + +- E0/E1: `e0_blockizer_owns_identity`, `e0_content_addressed_reuse`, `e0_converter_router_versioned`, `e0_lineages_versions_modes`, `e0_postgres_holds_no_bodies`, `e0_raw_immutable_id_addressed`, `e0_structure_contract_unconditional`, `e1_chunks_whole_blocks_no_overlap`, `e1_multigranularity_retrieval`. +- E2/E3, apart from the noted ledger and temporal wording: `e2_asserted_validity_immutable`, `e2_claims_immutable_append_only`, `e2_context_bundle_two_calls`, `e2_grounding_layered_dual_field`, `e2_no_pre_extraction_gate`, `e2_selection_recall_envelope`, `e2_stance_kept_as_observation`, `e2_testimony_currency_counting`, `e3_blocking_cheap_first_cascade`, `e3_observations_untyped_adjudicated`, `e3_predicate_registry_governed`, `e3_relations_evidence_collapse`. +- Registries/K/P/retrieval/boundary, with judgeability caveats below: `er_*`, `k_*`, `p*`, `ret_*`, `boundary_*`, `code_config_via_pydantic_settings`. + +## Coverage Gaps + +The suite is broad, but several binding decisions have no direct check and deserve one. + +**D1 / D46: split source of truth** + +Suggested id: `source_of_truth_split_d1_d46` + +Assert: + +- Postgres is authoritative for E0-E3 and deterministic/projection control state. +- The K git repo is a source of truth for human-authored pages and curation sidecars, with its own backup/export path. +- Compiled K pages are semantically regenerable from spine evidence plus recorded compile inputs. +- Postgres stores K provenance/control rows, not K bodies as the authority. + +**D16 / D50: one graph, many lenses** + +Suggested id: `scope_views_share_graph_d16` + +Assert: + +- K2 scopes never create separate entity spaces or separate canonical graphs. +- Scope-specific graph views are registry/configured filters over the shared graph/export. +- Materialized filtered snapshots, if present, are performance/scope-view projections only, not authorities and not content-authorization boundaries. +- Different trust boundaries require separate deployments, not in-library per-scope filtering. + +**D22: golden sets and evaluation plan** + +Suggested id: `eval_golden_sets_d22` + +Assert: + +- ER has a human-adjudicated golden eval set separate from any training set, with hard positives/negatives, per-type strata, Wilson CIs, per-tier metrics, and canary reruns per `resolver_version`. +- Measurement labels are human-adjudicated; the cascade/LLM may propose pairs but cannot label its own eval truth. +- Retrieval evals track recall@k per recipe version and rerank tuning. +- Selection and contradiction/grounding evals are recorded in the eval-run/canary machinery. + +**D23: registry scale and indexing** + +Suggested id: `registry_scale_partitioning_d23` + +Assert: + +- The large append-only registry/evidence tables are partitioned per the current schema design and avoid write-amplifying indexes on hot tables. +- `entities` / `aliases` are not partitioned as hot blocking targets. +- Alias blocking has trigram and Daitch-Mokotoff indexes; T3 embeddings live in Lance; HNSW is never in OLTP Postgres. +- Relations have the required btree composites for `(subject_entity_id, predicate[, object])`. +- Representative load-test hooks exist before partition/index choices are locked. + +**D36: E0 sub-worker chain and crossrefs** + +Suggested id: `e0_subworker_chain_crossrefs_d36` + +Assert: + +- E0 remains one product layer implemented as separately idempotent, separately observable sub-workers: ingest, convert, structure, crossref. +- Each sub-worker has its own version/idempotency boundary so a config/version bump reruns only the affected sub-worker and downstream dependents. +- Crossref extraction records citations/document links as E0 metadata; PageIndex post-processing is not promoted to a top-level E layer. + +**Requirements Code beyond settings** + +Suggested id: `code_tooling_and_migrations` + +Assert: + +- Pyright, pytest, Ruff, and Alembic are configured and wired into CI/test commands. +- Schema changes are represented as Alembic migrations rather than ad hoc DDL. +- The code uses typed Python structures/enums/Literals where the requirements call for typed contracts. + +## Judgeability + +Several checks ask a static LLM judge to prove global runtime behavior. They should either be split or require concrete evidence artifacts such as central interfaces, tests, lints, or migration constraints. + +**`boundary_library_scope_d60`** + +Condition 3 is too broad: "No correctness-determining mechanism ... is stubbed out, feature-flagged off, or delegated" asks the judge to audit the entire system in one condition. Split into: + +- `boundary_no_human_control_plane_d60`: no web UI, no tenancy/SSO/billing. +- `boundary_correctness_in_library_d60`: named correctness modules exist in-repo and are not cloud-only. +- `boundary_extension_points_cannot_bypass_invariants_d60`: ingestion/review/control-plane extension points still call the invariant-preserving pipeline. + +**`e0_lineages_versions_modes`** + +This check combines lineage identity, content-object reuse, debounce, snapshot/living semantics, living retraction, deletion, and sync-cycle barriers. Split into: + +- `e0_lineage_version_schema_d55` +- `e0_living_mode_retraction_d55_d54` +- `ops_deletion_grains_d55` + +**`e0_content_addressed_reuse`** + +"Cost proportional to the edit" and "efficiency ladder exists" are hard to infer from code alone. Require targeted tests: unchanged content no-op, one-block edit reuses unchanged chunk claims, neighbor changes force re-extraction, and no LLM-derived field appears in `extraction_input_hash`. + +**`e2_context_bundle_two_calls`** + +"No dangling pronouns/references in accepted claims" is a quality guarantee, not statically provable. Reword condition 3 to judge implementation mechanics: + +```text +The extraction schema/prompt requires decontextualized claims with resolved referents; deterministic validation rejects unresolved pronouns/references where detectable; no dedicated coref model or pre-pass exists. +``` + +**`e3_observations_untyped_adjudicated`** + +Condition 4 asks the judge to decide whether fail-safe coexist is "in code" across all incomplete comparisons. Make it concrete: require an adjudication outcome enum, margin/threshold config, persisted `observation_adjudications` reason rows, and tests for ambiguous comparisons falling to coexist/new. + +**`ret_envelope_contract`** + +This is five acceptance suites in one: grain discipline, contradiction completeness, freshness stamps, truncation/identity echo, and negative taxonomy. Split into separate checks. A single incomplete envelope type would currently fail a very large check and obscure the defect. + +**`ret_hydration_reverifies`** + +"Every API/CLI/MCP result" is only judgeable if all surfaces route through one query service/hydrator. Add a condition requiring a shared hydrator boundary plus contract tests for API, CLI, and MCP paths. + +**`p_projections_hold_no_authority`** + +Global absence of projection-side decisions is hard to prove by reading all writes. Make it more judgeable with architectural constraints: projection builders may import read repositories and projection writers only; they may not import adjudicators, LLM clients, or mutation repositories for validity state. + +**`ops_producer_checker_families`** + +Runtime model-family separation is config-dependent. Require a validator that fails startup/CI if checker and producer model families match without an explicit recorded exception. + +## Overlaps and Conflicts + +**Direct conflict** + +- `ops_deletion_cascade_grains` conflicts with `e2_claims_immutable_append_only` and `e2_testimony_currency_counting`. The former says deletion cascades through claims/evidence links; the latter checks correctly preserve claims as immutable evidence with currency ended. +- `ops_execution_classes_bound` condition 4 conflicts with `e2_context_bundle_two_calls` and D25/D31 because E2 extraction is volume-proportional, not ambiguity-only. + +**Double-fail overlap** + +- `boundary_library_scope_d60`, `ret_trust_boundary_no_content_auth`, and `er_review_queue_cli` all check "no web UI / no orgs/users/RBAC/SSO." Keep the boundary check high-level and let the retrieval/review checks focus on their local surfaces. +- `ret_claims_never_answer_now` overlaps with `ret_envelope_contract` condition 1. This is acceptable if kept as separate API naming/linter coverage, but do not let both fail from the same missing enum check unless that double weight is intended. +- `e2_no_pre_extraction_gate` overlaps with `e1_multigranularity_retrieval` condition 3. One should own "no pre-extraction skip"; the other should own "role scalar exists and default recipes filter retrieval-side." +- `p_projections_hold_no_authority` overlaps with `p1_lance_estate_rebuildable`, `p2_rebuild_first_snapshots`, `p2_projection_contract_views`, and `p3_corpus_fs_projection`. That is useful as a plane-level guard, but expect double failures from one missing rebuild path. +- `e2_selection_ledger_replay` overlaps with `ops_versioned_replay_from_storage`; keep the former E2-specific and make the latter a cross-stage version/replay spot check. + +## Mechanics + +- File count is correct: 53 YAML files, 53 checks. IDs match filenames and there are no duplicate ids. +- README inventory counts are correct by prefix. +- `p1_lance_estate_rebuildable.yaml` cites `overall_design.md section 4-5`; the actual path is `plan/designs/overall_design.md`. +- The README says the suite covers the decision log "(D1-D61)", but `decisions.md` now includes D62 and D63. If this suite is intentionally scoped to D1-D61, say so explicitly in the README. If not, add D62/D63 checks or update the inventory. +- Tags are mostly useful, but several conditions cite decisions missing from tags: + - `er_resolution_cascade_t0_t4` condition 4 cites D20 but tags omit `D20`. + - `e2_grounding_layered_dual_field` condition 2 cites D53 but tags omit `D53`. + - `e3_blocking_cheap_first_cascade` condition 4 cites D43 but tags omit `D43`. + - `e2_asserted_validity_immutable` condition 2 cites D32 but tags omit `D32`. + - `p2_rebuild_first_snapshots` condition 4 cites D40 but tags omit `D40`. + - `k_one_mechanism_n_scopes` condition 3 cites D54 but tags omit `D54`. +- Some section references are loose rather than broken: `e0_content_addressed_reuse` says `evidence_lifecycle_design.md section reuse`, but the heading is "Content-addressed reuse"; prefer exact section numbers/titles. + +## Bottom Line + +After the must-fix items, this will be a credible design-conformance suite. The most important corrections are to make deletion match the actual normal-delete vs hard-forget split, stop requiring every programmatic LLM worker to be an ambiguity-only cascade, and add D22/D23 coverage. Then split the largest checks enough that a binary judge can produce reliable, actionable failures. diff --git a/plan/implementation_evals/eval_checks/boundary_correctness_in_library_d60.yaml b/plan/implementation_evals/eval_checks/boundary_correctness_in_library_d60.yaml new file mode 100644 index 0000000..3c60d0b --- /dev/null +++ b/plan/implementation_evals/eval_checks/boundary_correctness_in_library_d60.yaml @@ -0,0 +1,18 @@ +schema_version: 1 +id: boundary_correctness_in_library_d60 +type: harness_judge +description: Every correctness-determining mechanism the design binds is implemented in this repository — none stubbed, feature-flagged off, or delegated to an external/commercial service. +tags: [boundary, D60] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D60, then verify each named correctness module exists in-repo and is + functional (not a stub raising NotImplemented, not disabled by default, not calling out to a + proprietary cloud service to do the work). + + Check each of these named mechanisms exists in this repository: claim extraction with + Selection; entity resolution (T0-T4); relation/observation supersession + contradiction + adjudication; grounding validation; the eval harness + canaries; budget enforcement; the + DLQ; the deletion cascade; the review CLI. + + Score 1 only if EVERY named mechanism is present and functional in-repo — cite file paths + in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/boundary_extension_points_no_bypass_d60.yaml b/plan/implementation_evals/eval_checks/boundary_extension_points_no_bypass_d60.yaml new file mode 100644 index 0000000..006dbe9 --- /dev/null +++ b/plan/implementation_evals/eval_checks/boundary_extension_points_no_bypass_d60.yaml @@ -0,0 +1,22 @@ +schema_version: 1 +id: boundary_extension_points_no_bypass_d60 +type: harness_judge +description: Extension points cannot bypass invariants — ingestion always writes through E0; review always appends reversible verdicts; no consumer hook reaches around the pipeline. +tags: [boundary, D60, D62] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D60 (constraint 2) and D62 (client surface), then inspect every extension + point (connector interface, ingest API, review sink, provider ports) under src/. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Every ingestion path — connectors, the client push API, bulk import — writes through E0 + (lineage identity, origin stamp, content hashing); no API allows inserting claims, + relations, observations, or K pages directly. + 2. Review integrations can only append reversible, provenance-stamped verdicts (the D24 + shape) — no hook allows destructive merges or direct registry rewrites. + 3. Provider-port adapters (D61) receive substrate operations only — no port interface + exposes a way to mutate spine semantics (validity, adjudication outcomes, citations). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/boundary_library_scope_d60.yaml b/plan/implementation_evals/eval_checks/boundary_library_scope_d60.yaml deleted file mode 100644 index 43493c8..0000000 --- a/plan/implementation_evals/eval_checks/boundary_library_scope_d60.yaml +++ /dev/null @@ -1,21 +0,0 @@ -schema_version: 1 -id: boundary_library_scope_d60 -type: harness_judge -description: The library boundary holds — no web UI, no multi-tenant control plane, no SSO/billing; agent surfaces are the complete consumption story; correctness never gated. -tags: [boundary, D60] -instructions: | - You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D60 and CLAUDE.md Rule 3, then survey the repository's shipped surfaces. - These are acceptance checks for the FINAL system. - - Verify ALL of the following: - 1. No web UI ships in this repository (no frontend app, no server-rendered dashboards); the - consumption surfaces are API, CLI, MCP server, and mounted filesystems. - 2. No multi-tenant machinery exists: no orgs/users/roles tables, no SSO/SAML, no billing — - one deployment is one trust domain. - 3. No correctness-determining mechanism (extraction, resolution, supersession, grounding, - evals, budgets, DLQ, deletion) is stubbed out, feature-flagged off, or delegated to an - external/commercial service — everything the design calls binding is implemented here. - - Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. - Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/boundary_no_human_control_plane_d60.yaml b/plan/implementation_evals/eval_checks/boundary_no_human_control_plane_d60.yaml new file mode 100644 index 0000000..acd1406 --- /dev/null +++ b/plan/implementation_evals/eval_checks/boundary_no_human_control_plane_d60.yaml @@ -0,0 +1,18 @@ +schema_version: 1 +id: boundary_no_human_control_plane_d60 +type: harness_judge +description: No web UI and no multi-tenant control plane ship in this repository — agent surfaces (API/CLI/MCP/mounts) are the complete consumption story. +tags: [boundary, D60] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D60 and CLAUDE.md Rule 3, then survey the repository's shipped surfaces. + These are acceptance checks for the FINAL system. + + Verify ALL of the following: + 1. No web UI ships (no frontend app, no server-rendered dashboards); consumption surfaces + are API, CLI, MCP server, and mounted filesystems. + 2. No multi-tenant machinery exists: no orgs/users/roles tables, no SSO/SAML, no billing — + one deployment is one trust domain. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/code_architecture_enforced_d62.yaml b/plan/implementation_evals/eval_checks/code_architecture_enforced_d62.yaml new file mode 100644 index 0000000..deb3e99 --- /dev/null +++ b/plan/implementation_evals/eval_checks/code_architecture_enforced_d62.yaml @@ -0,0 +1,22 @@ +schema_version: 1 +id: code_architecture_enforced_d62 +type: harness_judge +description: The hexagonal architecture is mechanically enforced — pure infra-free core, SQL only in spine/, vendor SDKs only in adapters/, import-linter contracts failing CI, explicit composition roots. +tags: [code, D62] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D62 (the enforced code architecture), then inspect the package layout and + the import-linter configuration under the repo. These are acceptance checks for the FINAL + system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. The layout follows the bound shape: model/core/spine/ports/adapters/llm/workers/surfaces/ + eval/profiles (names may evolve slightly, but the layering must match: pure domain core; + SQL confined to the spine layer; vendor SDKs confined to adapters). + 2. core is pure and infra-free — no SQL, no vendor SDK imports, no I/O clients. + 3. import-linter (or equivalent) contracts encode the allowed arrows and FAIL CI on illegal + imports — architecture erosion is loud, not gradual. + 4. Profiles are explicit composition roots wiring ports to adapters — no DI framework. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/code_tooling_and_migrations.yaml b/plan/implementation_evals/eval_checks/code_tooling_and_migrations.yaml new file mode 100644 index 0000000..6f83a83 --- /dev/null +++ b/plan/implementation_evals/eval_checks/code_tooling_and_migrations.yaml @@ -0,0 +1,22 @@ +schema_version: 1 +id: code_tooling_and_migrations +type: harness_judge +description: The code conventions hold — pyright/pytest/ruff wired into CI, all schema changes as Alembic migrations, typed contracts throughout. +tags: [code, requirements] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding convention. + Read requirements_v3.md §Code, then inspect the tooling configuration, CI workflows, and + migrations under the repo. These are acceptance checks for the FINAL system: if the + subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Pyright, pytest, and ruff are configured (pyproject.toml) and enforced in CI on every PR; + the pyright configuration has not been quietly weakened (per pyproject's + relax-with-receipts rule, any disabled check carries a written reason). + 2. Every schema change is an Alembic migration — no ad-hoc DDL executed outside the + migration chain; migrations replay cleanly from empty. + 3. Domain contracts are typed (Pydantic models, enums, Literal/TypedDict where the design + names closed sets — adjudication outcomes, execution classes, grains, negative kinds). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/delivery_artifacts_and_task_shells_d62.yaml b/plan/implementation_evals/eval_checks/delivery_artifacts_and_task_shells_d62.yaml new file mode 100644 index 0000000..51fd37e --- /dev/null +++ b/plan/implementation_evals/eval_checks/delivery_artifacts_and_task_shells_d62.yaml @@ -0,0 +1,29 @@ +schema_version: 1 +id: delivery_artifacts_and_task_shells_d62 +type: harness_judge +description: The delivery contract holds — PyPI client package with extras, GHCR images + CI-tested compose profile, client surface limits, and delivery-only task execution with two shells. +tags: [boundary, ops, D62] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D62, then inspect the packaging, compose profile, client surface, and task + execution under the repo. These are acceptance checks for the FINAL system: if the subsystem + is absent or partial, score 0. + + Verify ALL of the following: + 1. The package is positioned as the client: base install = typed SDK + CLI + MCP server; + server/connectors/K machinery live behind extras ([server], [connectors-*], [k]). + 2. Container images are published (GHCR) and a CI-TESTED docker-compose self-host profile + exists (Postgres + MinIO + api + worker); a quickstart brings up a working deployment + (the ten-minute quickstart is a release gate). + 3. The client surface is exactly: query (SDK/CLI/MCP), lineage-aware ingest (source_kind/ + source_ref/source_modified_at/versioning_mode on push; writes through E0), connector + MANAGEMENT never execution (connectors run deployment-side), and the review/admin CLI. + 4. Task execution is one model with two delivery shells: processing_state rows are the sole + authority; the self-host shell wakes on LISTEN/NOTIFY and claims with SKIP LOCKED + (enqueue transactional with the caller's state writes); the reference shell is Cloud + Tasks push; a janitor sweep re-announces lost deliveries on both. + 5. Export/import rides rebuild-first: portable state = Postgres dump + raw/artifacts buckets + + the K repo; projections rebuild on import. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_content_addressed_reuse.yaml b/plan/implementation_evals/eval_checks/e0_content_addressed_reuse.yaml index 2c12f1a..8c92cb1 100644 --- a/plan/implementation_evals/eval_checks/e0_content_addressed_reuse.yaml +++ b/plan/implementation_evals/eval_checks/e0_content_addressed_reuse.yaml @@ -1,27 +1,31 @@ schema_version: 1 id: e0_content_addressed_reuse type: harness_judge -description: Re-ingesting an edited document costs proportional to the edit — extraction/embedding work is keyed by content, with no LLM output in any reuse key. +description: Re-ingesting an edited document costs proportional to the edit — reuse keys contain no LLM output, and targeted tests prove the reuse behavior. tags: [e0, e2, D56] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D56 and plan/designs/evidence_lifecycle_design.md §reuse, then inspect the - E2 idempotency keys and the version-diff logic under src/. These are acceptance checks for - the FINAL system: if the subsystem is absent or partial, score 0. + Read decisions.md D56 and plan/designs/evidence_lifecycle_design.md ("Content-addressed + reuse"), then inspect the E2 idempotency keys, the version-diff logic, and the tests under + src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. Verify ALL of the following: 1. E2 extraction idempotency keys on an extraction_input_hash built from stable components only: the chunk's own block hashes + neighbor-chunk block hashes + stable header facts + extractor version + structurer version. NO LLM output (section path text, summaries, E1 - prefix) participates in the key. - 2. An unchanged chunk reuses its claims — re-attached to the new version's chunk row via the - chunk_claims map (written on both fresh extraction and reuse; exact, never inferred from - content-hash joins); a chunk whose neighbors changed re-extracts. + prefix) participates in the key — LLM-derived context is carried forward for unchanged + regions instead. + 2. The chunk_claims map records claim occurrences on both fresh extraction and reuse (one + immutable claim attached to every version-chunk that carried it; exact, never inferred + from content-hash joins). 3. Embeddings key on (chunk content hash, embedding version); conversion artifacts on - (content object, converter version). - 4. The efficiency ladder exists with cheapest exits first: connector-metadata no-op → - content-object no-op → conversion reuse → chunk-grain extraction reuse → delta-only - downstream triggers. + (content object, converter version); reuse alignment is a block-hash sequence diff with + anchor-stabilized chunk boundaries. + 4. Targeted TESTS exist and pass, demonstrating: an unchanged re-ingest is a no-op; a + one-block edit reuses the unchanged chunks' claims and re-extracts only affected chunks; + a neighbor-chunk change forces re-extraction of the dependent chunk; and no LLM-derived + field appears in the reuse key. Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_lineage_version_schema.yaml b/plan/implementation_evals/eval_checks/e0_lineage_version_schema.yaml new file mode 100644 index 0000000..e98b07a --- /dev/null +++ b/plan/implementation_evals/eval_checks/e0_lineage_version_schema.yaml @@ -0,0 +1,25 @@ +schema_version: 1 +id: e0_lineage_version_schema +type: harness_judge +description: Documents are lineages with connector-native identity; versions are append-only rows over deduplicated content objects; ingestion debounces and no-ops on unchanged content. +tags: [e0, D55] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D55 and plan/designs/evidence_lifecycle_design.md, then inspect the + documents/document_versions/content_objects schema and the sync logic under src/. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. The logical document is a lineage with a stable doc_id identified by (source_kind, + source_ref); renames/moves are metadata updates on the same lineage; a new ref is a new + lineage. + 2. Versions are append-only rows (one per observed snapshot, carrying conversion/structure + provenance and source_modified_at feeding derived claims' asserted time) referencing + content objects deduplicated by content_hash — bytes stored and converted once, even + across lineages. + 3. Each lineage carries a versioning_mode: snapshot (fail-safe default) or living. + 4. Watched-source ingestion debounces rapid edits (a stability window), and unchanged + revision/etag or unchanged bytes are no-ops. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_lineages_versions_modes.yaml b/plan/implementation_evals/eval_checks/e0_lineages_versions_modes.yaml deleted file mode 100644 index 3375954..0000000 --- a/plan/implementation_evals/eval_checks/e0_lineages_versions_modes.yaml +++ /dev/null @@ -1,31 +0,0 @@ -schema_version: 1 -id: e0_lineages_versions_modes -type: harness_judge -description: Documents are lineages with connector-native identity, append-only immutable versions, deduplicated content objects, and snapshot/living semantics including retract-on-removal. -tags: [e0, D55, D54] -instructions: | - You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D55 and plan/designs/evidence_lifecycle_design.md, then inspect the - documents/document_versions/content_objects schema and the watched-source sync logic under - src/. These are acceptance checks for the FINAL system: if the subsystem is absent or - partial, score 0. - - Verify ALL of the following: - 1. The logical document is a lineage with stable doc_id identified by (source_kind, - source_ref); renames/moves update metadata on the same lineage; a new ref is a new lineage. - 2. Versions are append-only rows (one per observed snapshot) referencing content objects - deduplicated by content_hash (bytes stored/converted once, even across lineages); - unchanged revision/etag or unchanged bytes are no-ops; rapid edits are debounced. - 3. Each lineage carries versioning_mode: snapshot (every version independent dated testimony, - the fail-safe default) or living (current version is the standing statement). - 4. In living mode, removal of a fact's sole current support closes the fact per shape - (relations/effective-state observations: valid_until capped at source_modified_at; - measurements: invalidated_at — never a valid-time cap, per D43's no-cap rule), recorded as - retracted_source_removal — loud, attributed, reversible; retraction checks run only after - the sync cycle completes (an intra-cycle move is a support swap, never retract+reassert). - 5. Deletion is uniform (a version / a lineage / observed-deleted-at-source): contribution - removed, claims retained as history with currency ended, solely-supported facts closed per - shape, recorded; hard-forget is a separate grain. - - Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. - Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_living_mode_retraction.yaml b/plan/implementation_evals/eval_checks/e0_living_mode_retraction.yaml new file mode 100644 index 0000000..aa5cf20 --- /dev/null +++ b/plan/implementation_evals/eval_checks/e0_living_mode_retraction.yaml @@ -0,0 +1,26 @@ +schema_version: 1 +id: e0_living_mode_retraction +type: harness_judge +description: In living mode, removal retracts — sole-support loss closes facts per shape, recorded as retracted_source_removal, loud and reversible, evaluated only after the sync cycle completes. +tags: [e0, e3, D55, D54, D43] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D55 (removal semantics) and D54/D43, then inspect the version-supersession + and retraction logic under src/. These are acceptance checks for the FINAL system: if the + subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. In living mode, when content removal leaves a fact with zero current support, the fact is + adjudicated CLOSED per shape: relations and effective-state observations get valid_until + capped at the version's source_modified_at; measurement/fixed-period observations get + invalidated_at (never a valid-time cap — D43 no-cap). + 2. The closure is recorded as retracted_source_removal — attributed, reversible, and NOT + routed through a review flag (removal retracts; there is no removal_semantics: review + softener). + 3. With other current support remaining, the effect is a decrement only. + 4. Retraction checks evaluate only AFTER the connector's sync cycle completes, so an + intra-cycle section move resolves as a support swap — never retract-then-reassert. + 5. In snapshot mode, version succession never retracts anything. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_origin_stamped_at_ingest.yaml b/plan/implementation_evals/eval_checks/e0_origin_stamped_at_ingest.yaml index 27dfd50..1e22ea3 100644 --- a/plan/implementation_evals/eval_checks/e0_origin_stamped_at_ingest.yaml +++ b/plan/implementation_evals/eval_checks/e0_origin_stamped_at_ingest.yaml @@ -12,8 +12,9 @@ instructions: | 1. Every ingested document records an origin field distinguishing at minimum external from self/system-generated, set at E0 ingest time. 2. The field is immutable after ingest — no code path updates it. - 3. Nothing downstream silently drops it (it survives on the lineage/version rows so - confidence math can later discount self-generated echoes). + 3. The origin is stored on the durable document lineage/input metadata and remains reachable + from all versions and downstream provenance (so confidence math can later discount + self-generated echoes) — duplication onto every version row is not required. Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_subworker_chain_crossrefs_d36.yaml b/plan/implementation_evals/eval_checks/e0_subworker_chain_crossrefs_d36.yaml new file mode 100644 index 0000000..0fc264b --- /dev/null +++ b/plan/implementation_evals/eval_checks/e0_subworker_chain_crossrefs_d36.yaml @@ -0,0 +1,22 @@ +schema_version: 1 +id: e0_subworker_chain_crossrefs_d36 +type: harness_judge +description: E0 is one product layer implemented as separately idempotent, separately observable sub-workers (ingest, convert, structure, crossref) with per-sub-worker version boundaries. +tags: [e0, D36] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D36 and plan/designs/e0_files_design.md §1, then inspect the E0 workers + under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. + + Verify ALL of the following: + 1. E0 runs as a chain of distinct sub-workers — ingest, convert, structure, crossref — each + separately idempotent (keyed on content identity + its OWN version) and separately + observable (own processing state / status). + 2. A version bump of one sub-worker re-runs only that sub-worker and its downstream + dependents — never the whole chain. + 3. Crossref extraction records citations/document links as E0 metadata; structuring is not + promoted to a top-level E layer (E1–E3 keep their meanings). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e1_multigranularity_retrieval.yaml b/plan/implementation_evals/eval_checks/e1_multigranularity_retrieval.yaml index 8a55143..8c4ed98 100644 --- a/plan/implementation_evals/eval_checks/e1_multigranularity_retrieval.yaml +++ b/plan/implementation_evals/eval_checks/e1_multigranularity_retrieval.yaml @@ -1,21 +1,21 @@ schema_version: 1 id: e1_multigranularity_retrieval type: harness_judge -description: Claims are the needle index, chunks the passage index; low-value roles are filtered retrieval-side, never skipped at extraction. -tags: [e1, p1, D58, D25] +description: Claims are the needle index, chunks the passage index; default recipes filter low-value roles retrieval-side over what was indexed. +tags: [e1, p1, D58] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D58 (embedding granularity) and D25, then inspect the P1 index writers and - default search recipes under src/. These are acceptance checks for the FINAL system: if the + Read decisions.md D58 (embedding granularity), then inspect the P1 index writers and default + search recipes under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: 1. Every accepted, decontextualized claim is embedded into P1 (claims are the fine-grain "needle" channel); chunks are embedded as the passage channel. 2. P1 chunk rows carry a section-role scalar, and DEFAULT search recipes filter out - references/nav/boilerplate/legal chunks retrieval-side. - 3. Crucially (D25): that filtering happens at RETRIEVAL time over what was indexed — nothing - skips extraction or indexing by role upstream (no pre-extraction value gate reintroduced). + references/nav/boilerplate/legal chunks retrieval-side — filtering over what was indexed + (everything is still indexed; the no-pre-extraction-gate invariant is owned by + e2_no_pre_extraction_gate). Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml b/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml index 600b8a8..82e84db 100644 --- a/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml +++ b/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml @@ -2,7 +2,7 @@ schema_version: 1 id: e2_asserted_validity_immutable type: harness_judge description: Claims carry an immutable source-asserted validity interval (from/until/precision/kind), grounded by window-membership, never revisable, never a belief authority. -tags: [e2, D41] +tags: [e2, D41, D32] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. Read decisions.md D41, then inspect the claim schema and the temporal extraction under src/. diff --git a/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml b/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml index 6c3ce95..f740c2c 100644 --- a/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml +++ b/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml @@ -11,14 +11,15 @@ instructions: | Verify ALL of the following: 1. The extraction call receives a bundle: document header, section path/role (+ summary), the - chunk's context prefix (when the E1 prefix stage exists), same-section neighbor chunks, + chunk's context prefix (the E1 prefix stage exists per D63), same-section neighbor chunks, and known-entity hints — never a bare chunk. 2. The call architecture is two calls: Selection (verifiability KEEP/REWRITE/DROP) as its own call, then a fused disambiguation/decontextualization + decomposition + grounding call. A one-call collapse is only acceptable if the repo documents a measured ablation justifying it; the literal per-sentence multi-call loop must not be the implementation. - 3. Coreference is satisfied in-call (D19): no dedicated coref model or pre-pass exists, and - the output contract forbids dangling pronouns/references in accepted claims. + 3. Coreference is handled in-call (D19): no dedicated coref model or pre-pass exists; the + extraction schema/prompt requires decontextualized claims with resolved referents; and + deterministic validation rejects unresolved pronouns/references where detectable. 4. Extraction batching (if present) batches a section's contiguous chunks per call while keeping claims anchored per-chunk and idempotency keys per-chunk (D58). diff --git a/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml b/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml index ce8ffa2..21fc12b 100644 --- a/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml +++ b/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml @@ -2,7 +2,7 @@ schema_version: 1 id: e2_grounding_layered_dual_field type: harness_judge description: Claim grounding is dual-field (standalone claim_text + verbatim source_span with offsets + added_context[]) with layered acceptance — anchor, window-membership, entailment, sampled audit. -tags: [e2, D32] +tags: [e2, D32, D53] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. Read decisions.md D32 and plan/designs/e2_e3_claims_relations_design.md §3.3, then inspect diff --git a/plan/implementation_evals/eval_checks/e2_selection_ledger_replay.yaml b/plan/implementation_evals/eval_checks/e2_selection_ledger_replay.yaml index ff78451..a0e0e5c 100644 --- a/plan/implementation_evals/eval_checks/e2_selection_ledger_replay.yaml +++ b/plan/implementation_evals/eval_checks/e2_selection_ledger_replay.yaml @@ -1,17 +1,20 @@ schema_version: 1 id: e2_selection_ledger_replay type: harness_judge -description: Every Selection drop and decontextualization edit is append-only, versioned ledger state; rebuilds replay from storage and never re-call the model. +description: Selection drops, kept_flagged outcomes, and decontextualization edits are append-only versioned ledger state; rebuilds replay from storage and never re-call the model. tags: [e2, D33, D7] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D33 (and D7's replay discipline), then inspect the claim_extraction_decisions - ledger and the rebuild paths under src/. These are acceptance checks for the FINAL system: - if the subsystem is absent or partial, score 0. + Read decisions.md D33 (and D7's replay discipline) and plan/designs/postgres_schema_design.md + §8, then inspect the claim_extraction_decisions ledger and the rebuild paths under src/. + These are acceptance checks for the FINAL system: if the subsystem is absent or partial, + score 0. Verify ALL of the following: - 1. An append-only, version-stamped ledger records every Selection outcome (drops with reason - class, rewrites, kept_flagged) and every decontextualization edit. + 1. An append-only, version-stamped ledger records every Selection DROP (with reason class), + every low-confidence kept_flagged outcome, and every decontextualization edit. Plain + keeps are represented by the inserted claim row itself and are NOT required to have + ledger rows (per the schema design). 2. The ledger is never updated or deleted from — corrections append. 3. Rebuild paths read stored claims + decisions and never re-invoke the LLM (nondeterministic rungs are replay-from-storage); the extraction worker is idempotent on its content-based diff --git a/plan/implementation_evals/eval_checks/e3_blocking_cheap_first_cascade.yaml b/plan/implementation_evals/eval_checks/e3_blocking_cheap_first_cascade.yaml index 7d32166..1091271 100644 --- a/plan/implementation_evals/eval_checks/e3_blocking_cheap_first_cascade.yaml +++ b/plan/implementation_evals/eval_checks/e3_blocking_cheap_first_cascade.yaml @@ -2,7 +2,7 @@ schema_version: 1 id: e3_blocking_cheap_first_cascade type: harness_judge description: Supersession candidates come from (entity, predicate) blocking with a cheap-first escalation cascade — LLM spend scales with ambiguity, not volume. -tags: [e3, D4] +tags: [e3, D4, D43] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. Read decisions.md D4, then inspect the supersession worker under src/. These are acceptance diff --git a/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml b/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml index fb78795..56f5950 100644 --- a/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml +++ b/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml @@ -16,14 +16,15 @@ instructions: | + temporal sanity checks). 2. Blocking fetches ALL of the entity's live observations by exact key (exhaustive; indexed); semantic similarity only RANKS candidates for hub entities — it never gates membership. - 3. The adjudicator outcomes are evidence / supersede / contradict-coexist / new, each gated on - a positive same-thing match judged semantically; the no-cap rule holds: only a changing - effective state is ever capped; a measurement/fixed-period figure is never capped on - valid-time — a conflicting same-period figure coexists with a shared contradiction_group. - 4. The fail-safe contract is in code: supersede is permitted only above an explicit margin - against a positively-matched prior AND writes an observation_adjudications reason row; any - incomplete comparison falls to coexist/new — the failure mode is a duplicate row, never a - silent overwrite. + 3. The adjudicator outcomes are a typed enum (evidence / supersede / contradict-coexist / + new), each gated on a positive same-thing match judged semantically; the no-cap rule + holds: only a changing effective state is ever capped; a measurement/fixed-period figure + is never capped on valid-time — a conflicting same-period figure coexists with a shared + contradiction_group. + 4. The fail-safe contract is concrete in code: an explicit margin/threshold configuration + gates supersede; every supersede/contradict writes an observation_adjudications reason + row; and tests exist demonstrating that ambiguous or incomplete comparisons fall to + coexist/new (the failure mode is a duplicate row, never a silent overwrite). 5. Observations never project to the P2 graph; they are embedded in P1 for retrieval. Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. diff --git a/plan/implementation_evals/eval_checks/e3_supersession_fact_level_bitemporal.yaml b/plan/implementation_evals/eval_checks/e3_supersession_fact_level_bitemporal.yaml index 9f43195..73e60ff 100644 --- a/plan/implementation_evals/eval_checks/e3_supersession_fact_level_bitemporal.yaml +++ b/plan/implementation_evals/eval_checks/e3_supersession_fact_level_bitemporal.yaml @@ -1,21 +1,26 @@ schema_version: 1 id: e3_supersession_fact_level_bitemporal type: harness_judge -description: Supersession closes validity windows on relations (and observations) — never on claims; both time axes are queryable. -tags: [e3, D3, D43] +description: Supersession closes validity windows on relations and effective-state observations — never on claims; measurements obey the no-cap rule; both time axes are queryable. +tags: [e3, D3, D43, D55] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D3 (and D43), then inspect the supersession write paths and the temporal - columns under src/. These are acceptance checks for the FINAL system: if the subsystem is - absent or partial, score 0. + Read decisions.md D3 and D43 (the no-cap rule) and D55 (source-removal shapes), then inspect + the supersession write paths and temporal columns under src/. These are acceptance checks + for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: - 1. Relations and observations carry the bi-temporal quartet: valid_from/valid_until - (world time) and ingested_at/invalidated_at (belief time); supersession caps valid_until — - it never deletes rows and never touches claims. - 2. invalidated_at means "we no longer believe this was ever right", distinct from a closed + 1. Relations and observations carry the bi-temporal quartet: valid_from/valid_until (world + time) and ingested_at/invalidated_at (belief time); supersession caps valid_until on + relations and on CHANGING EFFECTIVE-STATE observations — it never deletes rows and never + touches claims. + 2. Measurement/fixed-period observations obey D43's no-cap rule: they are NEVER capped on + valid-time; a conflicting same-period figure coexists in a shared contradiction_group; + when belief in a measurement is withdrawn (e.g. source removal, D55) the mechanism is + invalidated_at, never a valid-time cap. + 3. invalidated_at means "we no longer believe this was ever right", distinct from a closed validity window ("it ended"); a superseded row stays active in that sense. - 3. Both time-travel questions are answerable in the query layer: what was true at T + 4. Both time-travel questions are answerable in the query layer: what was true at T (valid-time) and what did we believe at T (transaction-time). Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. diff --git a/plan/implementation_evals/eval_checks/embedding_model_port_config_d63.yaml b/plan/implementation_evals/eval_checks/embedding_model_port_config_d63.yaml new file mode 100644 index 0000000..3113916 --- /dev/null +++ b/plan/implementation_evals/eval_checks/embedding_model_port_config_d63.yaml @@ -0,0 +1,26 @@ +schema_version: 1 +id: embedding_model_port_config_d63 +type: harness_judge +description: The embedding model is provider-port configuration, never architecture — versioned per artifact, swappable by version-scoped re-embed; the E1 context-prefix stage exists (conventional default). +tags: [p1, e1, D63, D61] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D63 (and D61's embedder port), then inspect the embedder port, its + adapters, and the E1 pipeline under src/. These are acceptance checks for the FINAL system: + if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. The embedding model is per-deployment configuration of the embedder port — no embedding + model name is hardcoded in domain code; the shipped default configuration is + qwen3-embedding-8b via an OpenRouter (OpenAI-compatible) adapter, with a self-hosted + open-weights adapter as the second documented adapter of the same port. + 2. Every embedded artifact carries an embedding version resolving to + pipeline_component_versions (model, dimension, params). + 3. Changing the model is a version-scoped re-embed batch (a supported operation), not a code + change. + 4. The E1 context-prefix stage exists as designed (the conventional-embedder branch), its + output stored and carried forward for unchanged regions (D56) — and prefixes never enter + reuse keys. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/er_resolution_cascade_t0_t4.yaml b/plan/implementation_evals/eval_checks/er_resolution_cascade_t0_t4.yaml index f94d1c5..55bdbb5 100644 --- a/plan/implementation_evals/eval_checks/er_resolution_cascade_t0_t4.yaml +++ b/plan/implementation_evals/eval_checks/er_resolution_cascade_t0_t4.yaml @@ -2,7 +2,7 @@ schema_version: 1 id: er_resolution_cascade_t0_t4 type: harness_judge description: Entity resolution is the T0–T4 cascade — exact, fuzzy blocking (recall-first), phonetic, embedding, LLM — with per-type versioned thresholds; cheap tiers escalate, never auto-reject. -tags: [registries, D17] +tags: [registries, D17, D20] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. Read decisions.md D17 and plan/designs/registries_design.md, then inspect the resolver under diff --git a/plan/implementation_evals/eval_checks/eval_golden_sets_d22.yaml b/plan/implementation_evals/eval_checks/eval_golden_sets_d22.yaml new file mode 100644 index 0000000..4a7b8d9 --- /dev/null +++ b/plan/implementation_evals/eval_checks/eval_golden_sets_d22.yaml @@ -0,0 +1,26 @@ +schema_version: 1 +id: eval_golden_sets_d22 +type: harness_judge +description: The golden-set eval plan is implemented — human-adjudicated ER eval set separate from any training set, per-tier metrics with Wilson CIs, canary reruns per resolver version, retrieval recall@k per recipe. +tags: [eval, D22, D35] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D22 (and D35's canaries) and plan/designs/registries_design.md (eval + section), then inspect the eval harness under src/. These are acceptance checks for the + FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. An ER golden EVAL set exists as a first-class asset: human-adjudicated labeled pairs, + hard positives AND hard negatives, stratified per entity type — kept strictly separate + from any training data (the circularity rule: the cascade/LLM may propose pairs, but + measurement labels are human-adjudicated). + 2. The harness computes per-tier precision/recall with Wilson confidence intervals, and a + canary regression suite re-runs per resolver_version bump. + 3. Retrieval evals measure recall@k per recipe (per version), supporting rerank-weight + tuning; Selection/grounding/contradiction evals record results in the same eval-run + machinery. + 4. Eval runs are recorded (versioned results), so threshold changes are justified by + measurements, not vibes. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/k_citations_binding.yaml b/plan/implementation_evals/eval_checks/k_citations_binding.yaml index 6c20a6e..94750ee 100644 --- a/plan/implementation_evals/eval_checks/k_citations_binding.yaml +++ b/plan/implementation_evals/eval_checks/k_citations_binding.yaml @@ -10,8 +10,9 @@ instructions: | the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: - 1. A writer's output is rejected/invalid without citations; citations are persisted in a - knowledge_artifact_evidence table linking artifacts to the evidence IDs they rest on. + 1. A writer's output is rejected/invalid without citations; citations are persisted in + knowledge_artifact_evidence linking artifacts to the claim, relation, or document IDs + they rest on (the schema allows exactly one reference kind per row). 2. Uncited candidate evidence (routed to the page but not cited) is counted and surfaced — not silently ignored. 3. Authored pages declare their citations (and optional watch rules) in frontmatter that the diff --git a/plan/implementation_evals/eval_checks/k_one_mechanism_n_scopes.yaml b/plan/implementation_evals/eval_checks/k_one_mechanism_n_scopes.yaml index 2a4ead5..878db53 100644 --- a/plan/implementation_evals/eval_checks/k_one_mechanism_n_scopes.yaml +++ b/plan/implementation_evals/eval_checks/k_one_mechanism_n_scopes.yaml @@ -2,7 +2,7 @@ schema_version: 1 id: k_one_mechanism_n_scopes type: harness_judge description: One compile mechanism, N scopes — tiers are configuration; K3 is the stricter-config belief tier; the framework contract itself is not configurable. -tags: [k, D47] +tags: [k, D47, D54] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. Read decisions.md D47 and plan/designs/k_layers_design.md §2, then inspect the scope/tier diff --git a/plan/implementation_evals/eval_checks/ops_deletion_cascade_grains.yaml b/plan/implementation_evals/eval_checks/ops_deletion_cascade_grains.yaml index 844c275..46a8edd 100644 --- a/plan/implementation_evals/eval_checks/ops_deletion_cascade_grains.yaml +++ b/plan/implementation_evals/eval_checks/ops_deletion_cascade_grains.yaml @@ -1,24 +1,30 @@ schema_version: 1 id: ops_deletion_cascade_grains type: harness_judge -description: Deletion has grains (version / lineage / hard-forget) and cascades through every derived layer — GCS objects, spine rows, projections, K reach — with facts closed per shape, recorded and reversible. -tags: [ops, D55, D46, requirements] +description: Deletion has grains (version / lineage / hard-forget); normal delete purges bytes and ends currency while RETAINING audit history; only hard-forget scrubs content. +tags: [ops, D55, D46, D54, requirements] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read requirements_v3.md §Knowledge lifecycle (deletion cascade), decisions.md D55/D46, and - plan/designs/e0_files_design.md §2, then inspect the deletion workers under src/. These are - acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + Read decisions.md D55/D54/D46, plan/designs/evidence_lifecycle_design.md §8, and + plan/designs/postgres_schema_design.md §13.1, then inspect the deletion workers under src/. + These are acceptance checks for the FINAL system: if the subsystem is absent or partial, + score 0. Verify ALL of the following: - 1. Three grains exist: delete a version (currency ends, lineage continues), delete a lineage - (full cascade), hard-forget (indistinguishable-from-never-existed semantics). - 2. Lineage deletion removes raw + artifact objects and spine rows, cascades chunks -> claims - -> evidence links; solely-supported facts are closed per shape (states capped, - measurements invalidated), recorded and reversible — never silently vanished. - 3. Projections converge by rebuild (the document simply is not materialized next cycle), and - K is reached mechanically via citations: compiled pages recompile without the removed - evidence; authored pages get a review flag (never a rewrite). - 4. A file observed deleted at its source is treated as lineage deletion, stamped with the + 1. Three grains exist: delete a version (currency ends; lineage continues), delete a lineage + (the full cascade), and hard-forget (indistinguishable-from-never-existed semantics). + 2. Normal deletion (version or lineage grain) purges raw/artifact bytes and tombstones the + lineage/version rows, ends testimony currency for affected claims, recomputes counts, and + closes solely-supported facts per shape (states: valid_until cap; measurements: + invalidated_at) — recorded and reversible, never silent. Chunks, claims, evidence links, + relations, and entities are RETAINED as audit history on a normal delete (claims are NOT + deleted; evidence rows remain historical links). + 3. Hard-forget is the separate grain that scrubs source-bearing payloads across versions so + forgotten content is indistinguishable from never-existed. + 4. Projections converge by rebuild (the deleted document is simply not materialized next + cycle), and K is reached mechanically via citations: compiled pages recompile without the + removed evidence; authored pages get a review flag (never a rewrite). + 5. A file observed deleted at its source is treated as lineage deletion, stamped with the observing sync cycle. Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. diff --git a/plan/implementation_evals/eval_checks/ops_execution_classes_bound.yaml b/plan/implementation_evals/eval_checks/ops_execution_classes_bound.yaml index f9a84a6..6a01afb 100644 --- a/plan/implementation_evals/eval_checks/ops_execution_classes_bound.yaml +++ b/plan/implementation_evals/eval_checks/ops_execution_classes_bound.yaml @@ -2,7 +2,7 @@ schema_version: 1 id: ops_execution_classes_bound type: harness_judge description: Every worker declares an execution class; agent harnesses exist only on plane K and review/audit seats; every LLM worker has an append-only transcript. -tags: [ops, D52, D33] +tags: [ops, D52, D33, D25] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. Read decisions.md D52 and plan/analysis/workers.md + plan/designs/orchestration_design.md, @@ -18,8 +18,11 @@ instructions: | 3. Every worker that makes an LLM call writes an append-only transcript/ledger (extraction decisions, adjudications, compilations) — no LLM call without a durable record. - 4. Programmatic-LLM workers sit inside cheap-first cascades (spend scales with ambiguity, - never volume). + 4. Programmatic-LLM workers are fixed-shape, schema-constrained, transcripted, and budgeted. + Resolution/adjudication workers sit inside cheap-first cascades (spend scales with + ambiguity); E2 extraction is the DELIBERATE volume-proportional exception (D25/D31: every + chunked document is fully extracted by the fixed two-call extractor) — do not fail it for + scaling with volume, and do not accept it becoming an agent harness. Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ops_idempotency_dlq.yaml b/plan/implementation_evals/eval_checks/ops_idempotency_dlq.yaml index dda0ba6..2fac3e8 100644 --- a/plan/implementation_evals/eval_checks/ops_idempotency_dlq.yaml +++ b/plan/implementation_evals/eval_checks/ops_idempotency_dlq.yaml @@ -1,20 +1,26 @@ schema_version: 1 id: ops_idempotency_dlq type: harness_judge -description: Every worker is idempotent (content hash + processing version); failures never disappear — bounded retries then dead-letter into Postgres with recorded status. -tags: [ops, D12, D56] +description: Every worker is idempotent on the key fitting its grain; work is processing_state rows (the sole authority) with bounded retries, dead-letter, and a janitor sweep. +tags: [ops, D12, D56, D62] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D12 (and D56's finer keys) and plan/designs/orchestration_design.md, then - inspect the workers and queue handling under src/. These are acceptance checks for the FINAL - system: if the subsystem is absent or partial, score 0. + Read decisions.md D12, D56, and D62 (task execution) and + plan/designs/orchestration_design.md, then inspect the workers and queue handling under + src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. Verify ALL of the following: - 1. Every worker is re-runnable and keyed on a content-derived hash + its own processing - version — re-delivery or re-run of an unchanged input is a no-op. - 2. Retries are bounded (per the design: max 2) and exhausted work dead-letters into Postgres + 1. Every worker is re-runnable and idempotent on the key fitting its grain: per-document/ + chunk workers on a content-derived hash + their own processing version (D12/D56); + aggregate workers (K compiles, projection rebuilds) on recorded input/snapshot hashes + (e.g. inputs_hash, D45). Re-delivery or re-run of an unchanged input is a no-op. + 2. Work items are processing_state rows in Postgres — the SOLE authority (D62); queue + deliveries are delivery-only signals; handlers are registered per stage and + shell-agnostic; a janitor sweep re-announces lost deliveries. + 3. Retries are bounded (per the design: max 2) and exhausted work dead-letters into Postgres with recorded status and enough context to inspect and replay — failures never vanish. - 3. The per-document E chain is task-queue driven; aggregate layers (K, P) are debounced or + 4. The per-document E chain is task-driven; aggregate layers (K, P) are debounced or scheduled, never triggered per document. Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. diff --git a/plan/implementation_evals/eval_checks/ops_producer_checker_families.yaml b/plan/implementation_evals/eval_checks/ops_producer_checker_families.yaml index 2beb60a..c30acdc 100644 --- a/plan/implementation_evals/eval_checks/ops_producer_checker_families.yaml +++ b/plan/implementation_evals/eval_checks/ops_producer_checker_families.yaml @@ -1,7 +1,7 @@ schema_version: 1 id: ops_producer_checker_families type: harness_judge -description: Every checking seat runs on a different model family than the producer it checks, auditable via recorded model assignments. +description: Every checking seat runs on a different model family than the producer it checks — validated mechanically at startup/CI, auditable via recorded model assignments. tags: [ops, D53] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. @@ -12,10 +12,11 @@ instructions: | 1. Checking seats (the sampled grounding judge, contradiction/citation-faithfulness evals, the review agent, the K reflection pass) are configured on a different model FAMILY than the producer they check. - 2. Model assignments are recorded (pipeline_component_versions naming producer and checker - models) so the separation is auditable. - 3. Running a checker in the producer's family requires a recorded exception — it cannot - happen as a quiet config default. + 2. A mechanical validator (config validation at startup and/or a CI check) FAILS when a + checker's model family matches its producer's, unless an explicit recorded exception is + present — the separation cannot silently drift. + 3. Model assignments are recorded (pipeline_component_versions naming producer and checker + models) so the separation is auditable after the fact. Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/p1_lance_estate_rebuildable.yaml b/plan/implementation_evals/eval_checks/p1_lance_estate_rebuildable.yaml index 924be94..ec379c0 100644 --- a/plan/implementation_evals/eval_checks/p1_lance_estate_rebuildable.yaml +++ b/plan/implementation_evals/eval_checks/p1_lance_estate_rebuildable.yaml @@ -5,7 +5,7 @@ description: P1 is one Lance estate (chunks, claims, relation labels, observatio tags: [p1, D8, D9, D7] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D8/D9 and overall_design.md §4-5, then inspect the P1 writers and rebuild + Read decisions.md D8/D9 and plan/designs/overall_design.md §4–§5, then inspect the P1 writers and rebuild path under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml b/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml index 0bfa9fc..36463de 100644 --- a/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml +++ b/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml @@ -2,7 +2,7 @@ schema_version: 1 id: p2_rebuild_first_snapshots type: harness_judge description: P2 is rebuild-first — full rebuild, validate, publish an immutable snapshot, pointer-swap; readers are read-only and hot-swap; no incremental event application. -tags: [p2, D7] +tags: [p2, D7, D40] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. Read decisions.md D7 and plan/designs/p2_graph_design.md, then inspect the P2 worker and diff --git a/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml b/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml index c709b91..51474a8 100644 --- a/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml +++ b/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml @@ -1,22 +1,26 @@ schema_version: 1 id: p_projections_hold_no_authority type: harness_judge -description: Validity/invalidation state has exactly one home (Postgres); P1/P2/P3 hold derived, rebuildable copies that are never independently mutated. -tags: [p, D6] +description: Validity state has exactly one home (Postgres); projection builders are architecturally barred from decision-making — enforced import boundaries, dumb deterministic writers. +tags: [p, D6, D62] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D6, then inspect every write path into Lance, LadybugDB, and the corpus - filesystem under src/. These are acceptance checks for the FINAL system: if the subsystem is - absent or partial, score 0. + Read decisions.md D6 (and D62's enforced architecture), then inspect the projection builders + and the import-linter contracts under src/. These are acceptance checks for the FINAL + system: if the subsystem is absent or partial, score 0. Verify ALL of the following: - 1. No code path mutates validity/invalidation/contradiction state in a projection store — + 1. Projection builders (P1/P2/P3 writers) import only read-side repositories and projection + writers — the architecture contracts (D62 import-linter) bar them from importing + adjudicators, LLM clients, or any repository that mutates validity/contradiction state; + CI fails on illegal imports. + 2. No code path mutates validity/invalidation/contradiction state in a projection store — Lance/Ladybug/P3 receive filtered copies from Postgres and are never the site of a decision. - 2. Every projection is deletable and rebuildable from Postgres (+ artifacts) by a tested + 3. Every projection is deletable and rebuildable from Postgres (+ artifacts) by a tested batch path; nothing in a projection is source-of-truth. - 3. The graph writer is dumb and deterministic — no adjudication, no supersession logic, no - LLM calls inside projection builders. + 4. The graph writer is deterministic — no adjudication, no supersession logic, no LLM calls + inside projection builders. Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/registry_scale_partitioning_d23.yaml b/plan/implementation_evals/eval_checks/registry_scale_partitioning_d23.yaml new file mode 100644 index 0000000..4bb2472 --- /dev/null +++ b/plan/implementation_evals/eval_checks/registry_scale_partitioning_d23.yaml @@ -0,0 +1,24 @@ +schema_version: 1 +id: registry_scale_partitioning_d23 +type: harness_judge +description: The registry scale rules hold — big append-only tables partitioned, hot tables btree-only, blocking indexes on aliases, no HNSW in OLTP Postgres. +tags: [registries, D23, D17] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D23 and plan/designs/postgres_schema_design.md, then inspect the + migrations under src/. These are acceptance checks for the FINAL system: if the subsystem is + absent or partial, score 0. + + Verify ALL of the following: + 1. The large append-only tables (mentions, resolution_decisions, relation_evidence — per the + current schema design) are partitioned (range by ingest period or the schema's chosen + scheme), and the hot append-only tables avoid write-amplifying index types (btree-only + there). + 2. entities and aliases (the blocking targets) are NOT partitioned. + 3. Alias blocking indexes exist: GIN gin_trgm_ops and a Daitch-Mokotoff expression index on + the normalized name; relations carry the btree composite on + (subject_entity_id, predicate[, object]). + 4. Embedding similarity (T3) runs in Lance — no pgvector/HNSW index in OLTP Postgres. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_envelope_contract.yaml b/plan/implementation_evals/eval_checks/ret_envelope_contract.yaml deleted file mode 100644 index 420ab26..0000000 --- a/plan/implementation_evals/eval_checks/ret_envelope_contract.yaml +++ /dev/null @@ -1,29 +0,0 @@ -schema_version: 1 -id: ret_envelope_contract -type: harness_judge -description: Every retrieval response is an envelope — declared grain, contradiction co-members never silently absent, per-source freshness stamps with believed_at horizons, explicit truncation, typed negatives. -tags: [retrieval, D49] -instructions: | - You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D49 and plan/designs/retrieval_design.md, then inspect the envelope types - and their enforcement under src/. These are acceptance checks for the FINAL system: if the - subsystem is absent or partial, score 0. - - Verify ALL of the following: - 1. Responses carry a declared grain (fact/evidence/compiled/composite) enforced at - composition: current-fact answers assemble only from validity-filtered - relations/observations; claims can never answer "is it true now"; composite answers are - parts[], each strictly single-grain. - 2. Contradiction co-members are inline up to a guaranteed cap, beyond which the block always - carries group_id + returned/total + a continuation — code cannot return a one-sided - answer for a contradicted fact. - 3. Per-source freshness stamps ship in the envelope (PG live; P1 write lag; P2 snapshot - timestamp + believed_at horizon with a typed boundary fallback; K compiled_at + staleness - + open-flag count). - 4. Truncation is always marked with a continuation (no silent top-k); the applied temporal - parameters and identity regime (current aliases vs identity_as_of) are echoed. - 5. A typed negative taxonomy exists: unknown_entity / known_empty / boundary — and there is - NO denied kind; hard-forgotten content is indistinguishable from never-existed. - - Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. - Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_envelope_contradiction_completeness.yaml b/plan/implementation_evals/eval_checks/ret_envelope_contradiction_completeness.yaml new file mode 100644 index 0000000..bfdbf33 --- /dev/null +++ b/plan/implementation_evals/eval_checks/ret_envelope_contradiction_completeness.yaml @@ -0,0 +1,19 @@ +schema_version: 1 +id: ret_envelope_contradiction_completeness +type: harness_judge +description: Contradiction co-members are never silently absent — inline up to a cap, then group_id + returned/total + continuation; one-sided answers are a contract violation. +tags: [retrieval, D49] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D49, then inspect the envelope contradiction handling under src/. These + are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. When a returned fact belongs to a contradiction group, co-members ship inline up to a + guaranteed cap; beyond the cap the block always carries group_id + returned/total counts + + a continuation handle. + 2. The code path makes a one-sided answer impossible for contradicted facts (a contract + test demonstrates it) — never a ranking choice that happens to drop the other side. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_envelope_freshness_stamps.yaml b/plan/implementation_evals/eval_checks/ret_envelope_freshness_stamps.yaml new file mode 100644 index 0000000..68e2273 --- /dev/null +++ b/plan/implementation_evals/eval_checks/ret_envelope_freshness_stamps.yaml @@ -0,0 +1,22 @@ +schema_version: 1 +id: ret_envelope_freshness_stamps +type: harness_judge +description: Per-source freshness stamps ship in every envelope — PG live, P1 write lag, P2 snapshot age with believed_at horizon + typed boundary fallback, K compiled_at + staleness + flag count. +tags: [retrieval, D49, D48] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D49 (freshness stamps) and D48, then inspect the envelope freshness + section under src/. These are acceptance checks for the FINAL system: if the subsystem is + absent or partial, score 0. + + Verify ALL of the following: + 1. Envelopes stamp each contributing source: Postgres (live), P1 (write lag), P2 (snapshot + timestamp), K (compiled_at + staleness state + open-flag count). + 2. Each channel's believed_at horizon is carried; a transaction-time query outside the hot + snapshot's retention window returns a typed boundary naming the fallback (PG traversal or + an archived snapshot) rather than a silent wrong answer. + 3. The applied temporal parameters (valid_at / believed_at) and the identity regime (current + aliases vs identity_as_of) are echoed in composition-ready form. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_envelope_grain_discipline.yaml b/plan/implementation_evals/eval_checks/ret_envelope_grain_discipline.yaml new file mode 100644 index 0000000..a3e55e5 --- /dev/null +++ b/plan/implementation_evals/eval_checks/ret_envelope_grain_discipline.yaml @@ -0,0 +1,21 @@ +schema_version: 1 +id: ret_envelope_grain_discipline +type: harness_judge +description: Every response declares its grain (fact/evidence/compiled/composite), enforced at composition — claims can never answer "is it true now"; composite parts are single-grain. +tags: [retrieval, D49, D41] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D49 and plan/designs/retrieval_design.md, then inspect the envelope types + and grain enforcement under src/. These are acceptance checks for the FINAL system: if the + subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Every primitive and recipe declares a grain (fact / evidence / compiled / composite), and + the type system or registry validation enforces it at composition time. + 2. Current-fact answers can only be assembled from validity-filtered relations/observations; + a claim-grain source composed into a current-fact answer is rejected mechanically. + 3. A composite answer is parts[], each part strictly single-grain — mixed answers never + dilute the discipline. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_envelope_negative_taxonomy.yaml b/plan/implementation_evals/eval_checks/ret_envelope_negative_taxonomy.yaml new file mode 100644 index 0000000..3cefa64 --- /dev/null +++ b/plan/implementation_evals/eval_checks/ret_envelope_negative_taxonomy.yaml @@ -0,0 +1,22 @@ +schema_version: 1 +id: ret_envelope_negative_taxonomy +type: harness_judge +description: A typed negative taxonomy — unknown_entity / known_empty / boundary; no denied kind; hard-forgotten content is indistinguishable from never-existed. +tags: [retrieval, D49, D50] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D49 (typed negatives) and D50 (trust model), then inspect the negative + result types under src/. These are acceptance checks for the FINAL system: if the subsystem + is absent or partial, score 0. + + Verify ALL of the following: + 1. Empty/failed lookups return a typed negative: unknown_entity (nothing resolves), + known_empty (resolved, genuinely nothing there), or boundary (a named limitation with a + workaround, e.g. the D43 cross-entity numeric-scan boundary). + 2. There is NO denied kind (content-level authorization is out of library scope). + 3. Hard-forgotten content answers exactly like never-existed content — same negative kind, + same shape, no distinguishing metadata; a contract test (or CI gate wired to the deletion + cascade) demonstrates the indistinguishability. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_envelope_truncation_markers.yaml b/plan/implementation_evals/eval_checks/ret_envelope_truncation_markers.yaml new file mode 100644 index 0000000..4eb35f9 --- /dev/null +++ b/plan/implementation_evals/eval_checks/ret_envelope_truncation_markers.yaml @@ -0,0 +1,19 @@ +schema_version: 1 +id: ret_envelope_truncation_markers +type: harness_judge +description: No silent caps anywhere — truncation is always marked with a continuation; hub answers are ranked pages, never a quiet top-k or a timeout. +tags: [retrieval, D49] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D49, then inspect result assembly and pagination under src/. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Every place a result set is bounded (top-k, page size, budget) emits an explicit + truncation marker with a continuation handle — grep for result-limiting code and confirm + none returns a bounded list unmarked. + 2. Hub-entity answers page deterministically (ranked pages with continuations), and no query + path resolves a too-big answer by silently timing out or capping. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_hydration_reverifies.yaml b/plan/implementation_evals/eval_checks/ret_hydration_reverifies.yaml index 434296a..f4b56c8 100644 --- a/plan/implementation_evals/eval_checks/ret_hydration_reverifies.yaml +++ b/plan/implementation_evals/eval_checks/ret_hydration_reverifies.yaml @@ -1,7 +1,7 @@ schema_version: 1 id: ret_hydration_reverifies type: harness_judge -description: Projections propose, the spine disposes — every query-engine result hydrates by-ID against live Postgres; drops are counted; compound results revalidate as units. +description: Projections propose, the spine disposes — all query surfaces route through one shared hydrator against live Postgres; drops are counted; compound results revalidate as units. tags: [retrieval, D48] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. @@ -10,13 +10,14 @@ instructions: | partial, score 0. Verify ALL of the following: - 1. Every API/CLI/MCP result passes through by-ID hydration against live Postgres before - reaching the caller — P1/P2 hits only NOMINATE candidates; validity, invalidation, and - contradiction membership are re-read from the spine. - 2. Candidates the spine no longer holds live are dropped, and the drop count is reported in - the response envelope. + 1. There is a SINGLE shared query service/hydrator boundary that API, CLI, and MCP all route + through (verify the three surfaces share it — contract tests covering all three paths + count as evidence); P1/P2 hits only NOMINATE candidates. + 2. Hydration re-reads validity windows, invalidation state, and contradiction membership by + ID from live Postgres before results reach a caller; candidates the spine no longer holds + live are dropped and the drop count is reported in the response envelope. 3. Compound results (e.g. graph paths) revalidate as units: one invalidated edge drops the - whole path — it is never returned with a hole or silently re-routed. + whole path — never returned with a hole or silently re-routed. 4. The two exempt surfaces are exactly the designed ones: mounted reads (snapshot semantics + freshness metadata) and K prose (compiled-grain with staleness state) — nothing else skips hydration. diff --git a/plan/implementation_evals/eval_checks/scope_views_share_graph_d16.yaml b/plan/implementation_evals/eval_checks/scope_views_share_graph_d16.yaml new file mode 100644 index 0000000..1d9430d --- /dev/null +++ b/plan/implementation_evals/eval_checks/scope_views_share_graph_d16.yaml @@ -0,0 +1,23 @@ +schema_version: 1 +id: scope_views_share_graph_d16 +type: harness_judge +description: One graph, many lenses — scopes never get their own entity space or graph; scope views are registry-declared filters; different trust boundaries mean separate deployments. +tags: [cross, D16, D50] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D16 (and D50's trust-model refinement), then inspect scope handling under + src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. + + Verify ALL of the following: + 1. K2 scopes never create a separate entity space, a separate registry, or a separate + canonical graph — one deployment has one entity space and one P2 graph. + 2. Scope-specific graph views are registry/config-declared filters (predicate/type lists) + over the shared graph or its export — materialized filtered snapshots, if implemented, + are emitted by the same rebuild from the same truth, as performance/scope-view + projections only. + 3. Nothing uses scope views as an access-control mechanism — different trust boundaries are + served by separate deployments, never by in-library per-scope filtering (D50). + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/source_of_truth_split_d1_d46.yaml b/plan/implementation_evals/eval_checks/source_of_truth_split_d1_d46.yaml new file mode 100644 index 0000000..0277123 --- /dev/null +++ b/plan/implementation_evals/eval_checks/source_of_truth_split_d1_d46.yaml @@ -0,0 +1,22 @@ +schema_version: 1 +id: source_of_truth_split_d1_d46 +type: harness_judge +description: The split source of truth holds — Postgres authoritative for the E spine and control state; the K git repo authoritative for authored content; compiled pages semantically regenerable. +tags: [cross, D1, D46] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read decisions.md D1 and D46, then inspect the store boundaries under src/. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Postgres is authoritative for E0–E3 and for deterministic/projection control state; + nothing treats a projection or the K repo as authority for evidence or validity. + 2. The K git repo is a source of truth for human-authored pages and curation sidecars, with + its own backup/export path; Postgres holds K provenance/control rows (plans, citations, + compilations), never K bodies as the authority. + 3. Compiled K pages are semantically regenerable: the recorded compile inputs (routing keys, + citations, inputs_hash, prompt/model versions) are sufficient to recompile a page from + the spine without consulting the old page body. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. From 878926d44200a3af18670ab5ec758390b4a569fb Mon Sep 17 00:00:00 2001 From: Jiri Puc Date: Fri, 10 Jul 2026 00:32:03 +0200 Subject: [PATCH 3/7] Evals follow-up: questions #30/#31 marked resolved (PR #43); pick up D64 (seed core) in the predicate check + README range Co-Authored-By: Claude Fable 5 --- plan/implementation_evals/README.md | 2 +- .../e3_predicate_registry_governed.yaml | 4 ++-- questions.md | 17 +++++++---------- 3 files changed, 10 insertions(+), 13 deletions(-) diff --git a/plan/implementation_evals/README.md b/plan/implementation_evals/README.md index 7709330..2c9bd5d 100644 --- a/plan/implementation_evals/README.md +++ b/plan/implementation_evals/README.md @@ -2,7 +2,7 @@ An exhaustive set of [eval-banana](https://github.com/writeitai/eval-banana) `harness_judge` checks that verify the **final implementation honors the binding design** — the decision log -(**D1–D63**), the requirements, and the design docs. Each check points an LLM judge at the +(**D1–D64**), the requirements, and the design docs. Each check points an LLM judge at the binding sources and at the code, states the invariant as concrete conditions, and demands a binary verdict. diff --git a/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml b/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml index 1fb995c..608cdb2 100644 --- a/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml +++ b/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml @@ -2,10 +2,10 @@ schema_version: 1 id: e3_predicate_registry_governed type: harness_judge description: The predicate vocabulary is a governed registry with an other: escape and promotion review; domain/range constraints are enforced; time is never a predicate or node. -tags: [e3, D5, D18, D15] +tags: [e3, D5, D18, D15, D64] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D5/D15/D18 and plan/designs/registries_design.md, then inspect the + Read decisions.md D5/D15/D18/D64 (the current seed core) and plan/designs/registries_design.md, then inspect the predicate registry and extraction constraints under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/questions.md b/questions.md index 08564a1..205a75c 100644 --- a/questions.md +++ b/questions.md @@ -172,16 +172,13 @@ Keep this current: when something here is decided, move it to a decision and pru 29. **E3 claim→predicate mapping is thin.** §5 delegates internals to the registries but doesn't specify how the governed predicate is *chosen* from claim text or where domain/range is enforced in the flow. -30. **Requirements temporal-split paragraph predates D43/D49.** `requirements_v3.md` §Retrieval says - "All supersession / current-fact validity **lives only on relations**" and cites D3/D6/D41 — but - observations carry the same adjudicated validity windows and supersession (D43), and D49's grain - discipline assembles current-fact answers from validity-filtered **relations and observations**. - Update the paragraph (and its citation list) to name both fact layers. (Found while authoring - `plan/implementation_evals/`.) -31. **Requirements E3 bullet omits observations.** `requirements_v3.md` §Plane E describes E3 as - relations only ("distinct facts `(subject, predicate, object)` … the unit of supersession and - contradiction") — no mention of observations (D43) or stance observations (D59), though the - README's E3 row and the designs treat them as a canonical fact layer. Update the bullet. (Found +30. ~~**Requirements temporal-split paragraph predates D43/D49.**~~ **RESOLVED (PR #43).** + `requirements_v3.md` §Retrieval now reads "current-fact validity lives on the fact layers — + relations and observations" (the claim/fact split; citations extended with D43/D49). (Found + while authoring `plan/implementation_evals/`.) +31. ~~**Requirements E3 bullet omits observations.**~~ **RESOLVED (PR #43).** The E3 bullet now + names both fact layers (relations + observations, incl. D59 stance observations), both + bi-temporal units of supersession/contradiction, with the graph-projection distinction. (Found while authoring `plan/implementation_evals/`.) ## Resolved since the last version of this file (moved to decisions) From 19c02c4474cfdcb82d2130f0d1da8c20d018fb5e Mon Sep 17 00:00:00 2001 From: Jiri Puc Date: Sat, 11 Jul 2026 16:31:57 +0200 Subject: [PATCH 4/7] =?UTF-8?q?Evals:=20source=20precedence=20=E2=80=94=20?= =?UTF-8?q?designs=20control,=20decisions=20index=20(author=20review)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every check now carries an explicit source-precedence rule in its instructions (judges never see the README): the design docs are the binding statement of the current system; decision entries are read with their refinement/withdrawal annotations; on any apparent disagreement the judge treats the designs (incl. postgres_schema_design.md) as controlling and reports the discrepancy rather than failing a design-conformant implementation. The 27 checks that cited only decisions gain explicit design-doc citations. README records the convention ("decisions for coverage, designs for truth"). Rationale: the decision log is append-only testimony refined by later entries; authoring checks from raw decision prose risks failing a correct implementation on superseded text — the exact hazard Codex's review caught once (the deletion check, corrected from postgres_schema_design §13.1). Co-Authored-By: Claude Fable 5 --- .../eval_checks/boundary_correctness_in_library_d60.yaml | 9 ++++++++- .../boundary_extension_points_no_bypass_d60.yaml | 9 ++++++++- .../eval_checks/boundary_no_human_control_plane_d60.yaml | 7 +++++++ .../eval_checks/boundary_provider_ports_d61.yaml | 7 +++++++ .../eval_checks/code_architecture_enforced_d62.yaml | 9 ++++++++- .../eval_checks/code_config_via_pydantic_settings.yaml | 7 +++++++ .../eval_checks/code_tooling_and_migrations.yaml | 7 +++++++ .../delivery_artifacts_and_task_shells_d62.yaml | 9 ++++++++- .../eval_checks/e0_blockizer_owns_identity.yaml | 7 +++++++ .../eval_checks/e0_content_addressed_reuse.yaml | 7 +++++++ .../eval_checks/e0_converter_router_versioned.yaml | 7 +++++++ .../eval_checks/e0_lineage_version_schema.yaml | 7 +++++++ .../eval_checks/e0_living_mode_retraction.yaml | 9 ++++++++- .../eval_checks/e0_origin_stamped_at_ingest.yaml | 9 ++++++++- .../eval_checks/e0_postgres_holds_no_bodies.yaml | 7 +++++++ .../eval_checks/e0_raw_immutable_id_addressed.yaml | 7 +++++++ .../eval_checks/e0_structure_contract_unconditional.yaml | 7 +++++++ .../eval_checks/e0_subworker_chain_crossrefs_d36.yaml | 7 +++++++ .../eval_checks/e1_chunks_whole_blocks_no_overlap.yaml | 7 +++++++ .../eval_checks/e1_multigranularity_retrieval.yaml | 9 ++++++++- .../eval_checks/e2_asserted_validity_immutable.yaml | 9 ++++++++- .../eval_checks/e2_claims_immutable_append_only.yaml | 9 ++++++++- .../eval_checks/e2_context_bundle_two_calls.yaml | 7 +++++++ .../eval_checks/e2_grounding_layered_dual_field.yaml | 7 +++++++ .../eval_checks/e2_no_pre_extraction_gate.yaml | 7 +++++++ .../eval_checks/e2_selection_ledger_replay.yaml | 7 +++++++ .../eval_checks/e2_selection_recall_envelope.yaml | 9 ++++++++- .../eval_checks/e2_stance_kept_as_observation.yaml | 9 ++++++++- .../eval_checks/e2_testimony_currency_counting.yaml | 7 +++++++ .../eval_checks/e3_blocking_cheap_first_cascade.yaml | 9 ++++++++- .../eval_checks/e3_observations_untyped_adjudicated.yaml | 7 +++++++ .../eval_checks/e3_predicate_registry_governed.yaml | 7 +++++++ .../eval_checks/e3_relations_evidence_collapse.yaml | 7 +++++++ .../e3_supersession_fact_level_bitemporal.yaml | 9 ++++++++- .../eval_checks/embedding_model_port_config_d63.yaml | 9 ++++++++- .../eval_checks/er_merges_reversible.yaml | 9 ++++++++- .../eval_checks/er_resolution_cascade_t0_t4.yaml | 7 +++++++ .../eval_checks/er_review_queue_cli.yaml | 9 ++++++++- .../eval_checks/eval_golden_sets_d22.yaml | 7 +++++++ .../eval_checks/k_citations_binding.yaml | 7 +++++++ .../eval_checks/k_compiled_vs_authored_ownership.yaml | 7 +++++++ .../eval_checks/k_one_mechanism_n_scopes.yaml | 7 +++++++ .../eval_checks/k_planner_writer_driver.yaml | 7 +++++++ .../eval_checks/ops_deletion_cascade_grains.yaml | 7 +++++++ .../eval_checks/ops_execution_classes_bound.yaml | 7 +++++++ .../eval_checks/ops_idempotency_dlq.yaml | 7 +++++++ .../eval_checks/ops_metering_budgets_enforced.yaml | 7 +++++++ .../eval_checks/ops_producer_checker_families.yaml | 9 ++++++++- .../eval_checks/ops_versioned_replay_from_storage.yaml | 7 +++++++ .../eval_checks/p1_lance_estate_rebuildable.yaml | 7 +++++++ .../eval_checks/p2_asof_and_communities.yaml | 9 ++++++++- .../eval_checks/p2_no_vectors_in_graph.yaml | 9 ++++++++- .../eval_checks/p2_projection_contract_views.yaml | 7 +++++++ .../eval_checks/p2_rebuild_first_snapshots.yaml | 7 +++++++ .../eval_checks/p3_corpus_fs_projection.yaml | 7 +++++++ .../eval_checks/p_projections_hold_no_authority.yaml | 9 ++++++++- .../eval_checks/registry_scale_partitioning_d23.yaml | 7 +++++++ .../eval_checks/ret_claims_never_answer_now.yaml | 7 +++++++ .../ret_envelope_contradiction_completeness.yaml | 9 ++++++++- .../eval_checks/ret_envelope_freshness_stamps.yaml | 9 ++++++++- .../eval_checks/ret_envelope_grain_discipline.yaml | 7 +++++++ .../eval_checks/ret_envelope_negative_taxonomy.yaml | 9 ++++++++- .../eval_checks/ret_envelope_truncation_markers.yaml | 9 ++++++++- .../eval_checks/ret_filesystem_first_mounts_skill.yaml | 7 +++++++ .../eval_checks/ret_hydration_reverifies.yaml | 7 +++++++ .../eval_checks/ret_primitives_recipes_registry.yaml | 7 +++++++ .../eval_checks/ret_trust_boundary_no_content_auth.yaml | 9 ++++++++- .../eval_checks/scope_views_share_graph_d16.yaml | 9 ++++++++- .../eval_checks/source_of_truth_split_d1_d46.yaml | 9 ++++++++- 69 files changed, 510 insertions(+), 27 deletions(-) diff --git a/plan/implementation_evals/eval_checks/boundary_correctness_in_library_d60.yaml b/plan/implementation_evals/eval_checks/boundary_correctness_in_library_d60.yaml index 3c60d0b..d6a02b0 100644 --- a/plan/implementation_evals/eval_checks/boundary_correctness_in_library_d60.yaml +++ b/plan/implementation_evals/eval_checks/boundary_correctness_in_library_d60.yaml @@ -5,7 +5,7 @@ description: Every correctness-determining mechanism the design binds is impleme tags: [boundary, D60] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D60, then verify each named correctness module exists in-repo and is + Read decisions.md D60 and plan/requirements/requirements_v3.md §Vision, then verify each named correctness module exists in-repo and is functional (not a stub raising NotImplemented, not disabled by default, not calling out to a proprietary cloud service to do the work). @@ -14,5 +14,12 @@ instructions: | adjudication; grounding validation; the eval harness + canaries; budget enforcement; the DLQ; the deletion cascade; the review CLI. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY named mechanism is present and functional in-repo — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/boundary_extension_points_no_bypass_d60.yaml b/plan/implementation_evals/eval_checks/boundary_extension_points_no_bypass_d60.yaml index 006dbe9..98382e7 100644 --- a/plan/implementation_evals/eval_checks/boundary_extension_points_no_bypass_d60.yaml +++ b/plan/implementation_evals/eval_checks/boundary_extension_points_no_bypass_d60.yaml @@ -5,7 +5,7 @@ description: Extension points cannot bypass invariants — ingestion always writ tags: [boundary, D60, D62] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D60 (constraint 2) and D62 (client surface), then inspect every extension + Read decisions.md D60 (constraint 2) and D62 (client surface) and plan/designs/packaging_distribution_design.md (the client surface), then inspect every extension point (connector interface, ingest API, review sink, provider ports) under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -18,5 +18,12 @@ instructions: | 3. Provider-port adapters (D61) receive substrate operations only — no port interface exposes a way to mutate spine semantics (validity, adjudication outcomes, citations). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/boundary_no_human_control_plane_d60.yaml b/plan/implementation_evals/eval_checks/boundary_no_human_control_plane_d60.yaml index acd1406..3ad292f 100644 --- a/plan/implementation_evals/eval_checks/boundary_no_human_control_plane_d60.yaml +++ b/plan/implementation_evals/eval_checks/boundary_no_human_control_plane_d60.yaml @@ -14,5 +14,12 @@ instructions: | 2. No multi-tenant machinery exists: no orgs/users/roles tables, no SSO/SAML, no billing — one deployment is one trust domain. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/boundary_provider_ports_d61.yaml b/plan/implementation_evals/eval_checks/boundary_provider_ports_d61.yaml index eb91d90..508a5a7 100644 --- a/plan/implementation_evals/eval_checks/boundary_provider_ports_d61.yaml +++ b/plan/implementation_evals/eval_checks/boundary_provider_ports_d61.yaml @@ -23,5 +23,12 @@ instructions: | complete deployment on the self-host adapters, with every correctness capability included. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/code_architecture_enforced_d62.yaml b/plan/implementation_evals/eval_checks/code_architecture_enforced_d62.yaml index deb3e99..a0a02f4 100644 --- a/plan/implementation_evals/eval_checks/code_architecture_enforced_d62.yaml +++ b/plan/implementation_evals/eval_checks/code_architecture_enforced_d62.yaml @@ -5,7 +5,7 @@ description: The hexagonal architecture is mechanically enforced — pure infra- tags: [code, D62] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D62 (the enforced code architecture), then inspect the package layout and + Read decisions.md D62 (the enforced code architecture) and plan/designs/packaging_distribution_design.md (the enforced layout), then inspect the package layout and the import-linter configuration under the repo. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -18,5 +18,12 @@ instructions: | imports — architecture erosion is loud, not gradual. 4. Profiles are explicit composition roots wiring ports to adapters — no DI framework. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/code_config_via_pydantic_settings.yaml b/plan/implementation_evals/eval_checks/code_config_via_pydantic_settings.yaml index a38088e..7997a54 100644 --- a/plan/implementation_evals/eval_checks/code_config_via_pydantic_settings.yaml +++ b/plan/implementation_evals/eval_checks/code_config_via_pydantic_settings.yaml @@ -20,5 +20,12 @@ instructions: | 3. Settings objects are constructed at composition boundaries (entry points/workers), not scattered ad hoc through domain logic. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/code_tooling_and_migrations.yaml b/plan/implementation_evals/eval_checks/code_tooling_and_migrations.yaml index 6f83a83..e1c1ad5 100644 --- a/plan/implementation_evals/eval_checks/code_tooling_and_migrations.yaml +++ b/plan/implementation_evals/eval_checks/code_tooling_and_migrations.yaml @@ -18,5 +18,12 @@ instructions: | 3. Domain contracts are typed (Pydantic models, enums, Literal/TypedDict where the design names closed sets — adjudication outcomes, execution classes, grains, negative kinds). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/delivery_artifacts_and_task_shells_d62.yaml b/plan/implementation_evals/eval_checks/delivery_artifacts_and_task_shells_d62.yaml index 51fd37e..9f0087c 100644 --- a/plan/implementation_evals/eval_checks/delivery_artifacts_and_task_shells_d62.yaml +++ b/plan/implementation_evals/eval_checks/delivery_artifacts_and_task_shells_d62.yaml @@ -5,7 +5,7 @@ description: The delivery contract holds — PyPI client package with extras, GH tags: [boundary, ops, D62] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D62, then inspect the packaging, compose profile, client surface, and task + Read decisions.md D62 and plan/designs/packaging_distribution_design.md, then inspect the packaging, compose profile, client surface, and task execution under the repo. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -25,5 +25,12 @@ instructions: | 5. Export/import rides rebuild-first: portable state = Postgres dump + raw/artifacts buckets + the K repo; projections rebuild on import. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_blockizer_owns_identity.yaml b/plan/implementation_evals/eval_checks/e0_blockizer_owns_identity.yaml index 2e5cb6e..62785d1 100644 --- a/plan/implementation_evals/eval_checks/e0_blockizer_owns_identity.yaml +++ b/plan/implementation_evals/eval_checks/e0_blockizer_owns_identity.yaml @@ -21,5 +21,12 @@ instructions: | 4. Direction invariant: nothing derives blocks from sections — LLM output never touches the identity layer. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_content_addressed_reuse.yaml b/plan/implementation_evals/eval_checks/e0_content_addressed_reuse.yaml index 8c92cb1..6da9b3e 100644 --- a/plan/implementation_evals/eval_checks/e0_content_addressed_reuse.yaml +++ b/plan/implementation_evals/eval_checks/e0_content_addressed_reuse.yaml @@ -27,5 +27,12 @@ instructions: | a neighbor-chunk change forces re-extraction of the dependent chunk; and no LLM-derived field appears in the reuse key. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_converter_router_versioned.yaml b/plan/implementation_evals/eval_checks/e0_converter_router_versioned.yaml index cd37db1..14267f1 100644 --- a/plan/implementation_evals/eval_checks/e0_converter_router_versioned.yaml +++ b/plan/implementation_evals/eval_checks/e0_converter_router_versioned.yaml @@ -17,5 +17,12 @@ instructions: | 3. Every conversion is stamped with converter_name/converter_version on the document version row, and a converter/routing change triggers re-conversion keyed by version (D7 discipline). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_lineage_version_schema.yaml b/plan/implementation_evals/eval_checks/e0_lineage_version_schema.yaml index e98b07a..f3a2a9d 100644 --- a/plan/implementation_evals/eval_checks/e0_lineage_version_schema.yaml +++ b/plan/implementation_evals/eval_checks/e0_lineage_version_schema.yaml @@ -21,5 +21,12 @@ instructions: | 4. Watched-source ingestion debounces rapid edits (a stability window), and unchanged revision/etag or unchanged bytes are no-ops. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_living_mode_retraction.yaml b/plan/implementation_evals/eval_checks/e0_living_mode_retraction.yaml index aa5cf20..166b4e1 100644 --- a/plan/implementation_evals/eval_checks/e0_living_mode_retraction.yaml +++ b/plan/implementation_evals/eval_checks/e0_living_mode_retraction.yaml @@ -5,7 +5,7 @@ description: In living mode, removal retracts — sole-support loss closes facts tags: [e0, e3, D55, D54, D43] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D55 (removal semantics) and D54/D43, then inspect the version-supersession + Read decisions.md D55 (removal semantics) and D54/D43 and plan/designs/evidence_lifecycle_design.md, then inspect the version-supersession and retraction logic under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -22,5 +22,12 @@ instructions: | intra-cycle section move resolves as a support swap — never retract-then-reassert. 5. In snapshot mode, version succession never retracts anything. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_origin_stamped_at_ingest.yaml b/plan/implementation_evals/eval_checks/e0_origin_stamped_at_ingest.yaml index 1e22ea3..bd171b1 100644 --- a/plan/implementation_evals/eval_checks/e0_origin_stamped_at_ingest.yaml +++ b/plan/implementation_evals/eval_checks/e0_origin_stamped_at_ingest.yaml @@ -5,7 +5,7 @@ description: Every input carries an immutable origin stamp (external vs self/sys tags: [e0, D42] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D42, then inspect the ingest worker and documents schema under src/. These + Read decisions.md D42 and plan/designs/postgres_schema_design.md (documents), then inspect the ingest worker and documents schema under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: @@ -16,5 +16,12 @@ instructions: | from all versions and downstream provenance (so confidence math can later discount self-generated echoes) — duplication onto every version row is not required. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_postgres_holds_no_bodies.yaml b/plan/implementation_evals/eval_checks/e0_postgres_holds_no_bodies.yaml index c215e50..92348dd 100644 --- a/plan/implementation_evals/eval_checks/e0_postgres_holds_no_bodies.yaml +++ b/plan/implementation_evals/eval_checks/e0_postgres_holds_no_bodies.yaml @@ -17,5 +17,12 @@ instructions: | referenced from Postgres by URI. 3. Blocks (D57) are NOT Postgres rows — a sidecar artifact plus derived keys only. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_raw_immutable_id_addressed.yaml b/plan/implementation_evals/eval_checks/e0_raw_immutable_id_addressed.yaml index c6cc954..88a0cd2 100644 --- a/plan/implementation_evals/eval_checks/e0_raw_immutable_id_addressed.yaml +++ b/plan/implementation_evals/eval_checks/e0_raw_immutable_id_addressed.yaml @@ -20,5 +20,12 @@ instructions: | audit logging (D51). 4. content_hash = sha256 of raw bytes is computed at ingest and used as the idempotency key. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_structure_contract_unconditional.yaml b/plan/implementation_evals/eval_checks/e0_structure_contract_unconditional.yaml index d96793e..92ee953 100644 --- a/plan/implementation_evals/eval_checks/e0_structure_contract_unconditional.yaml +++ b/plan/implementation_evals/eval_checks/e0_structure_contract_unconditional.yaml @@ -20,5 +20,12 @@ instructions: | 4. A placement hint (a proposed corpus-filesystem path) is produced and stored as advisory input for P3 — not a commitment. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e0_subworker_chain_crossrefs_d36.yaml b/plan/implementation_evals/eval_checks/e0_subworker_chain_crossrefs_d36.yaml index 0fc264b..019d3fc 100644 --- a/plan/implementation_evals/eval_checks/e0_subworker_chain_crossrefs_d36.yaml +++ b/plan/implementation_evals/eval_checks/e0_subworker_chain_crossrefs_d36.yaml @@ -18,5 +18,12 @@ instructions: | 3. Crossref extraction records citations/document links as E0 metadata; structuring is not promoted to a top-level E layer (E1–E3 keep their meanings). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e1_chunks_whole_blocks_no_overlap.yaml b/plan/implementation_evals/eval_checks/e1_chunks_whole_blocks_no_overlap.yaml index 7251519..90f451f 100644 --- a/plan/implementation_evals/eval_checks/e1_chunks_whole_blocks_no_overlap.yaml +++ b/plan/implementation_evals/eval_checks/e1_chunks_whole_blocks_no_overlap.yaml @@ -21,5 +21,12 @@ instructions: | chunk; a pathological giant paragraph falls back to deterministic sentence splitting. 5. chunk_content_hash = hash of the ordered block hashes. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e1_multigranularity_retrieval.yaml b/plan/implementation_evals/eval_checks/e1_multigranularity_retrieval.yaml index 8c4ed98..08734f4 100644 --- a/plan/implementation_evals/eval_checks/e1_multigranularity_retrieval.yaml +++ b/plan/implementation_evals/eval_checks/e1_multigranularity_retrieval.yaml @@ -5,7 +5,7 @@ description: Claims are the needle index, chunks the passage index; default reci tags: [e1, p1, D58] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D58 (embedding granularity), then inspect the P1 index writers and default + Read decisions.md D58 (embedding granularity) and plan/designs/e1_chunks_design.md, then inspect the P1 index writers and default search recipes under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -17,5 +17,12 @@ instructions: | (everything is still indexed; the no-pre-extraction-gate invariant is owned by e2_no_pre_extraction_gate). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml b/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml index 82e84db..b180224 100644 --- a/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml +++ b/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml @@ -5,7 +5,7 @@ description: Claims carry an immutable source-asserted validity interval (from/u tags: [e2, D41, D32] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D41, then inspect the claim schema and the temporal extraction under src/. + Read decisions.md D41 and plan/designs/e2_e3_claims_relations_design.md §7, then inspect the claim schema and the temporal extraction under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -20,5 +20,12 @@ instructions: | monotonicity guard (a late retrospective cannot move an adjudicated window), and nothing reads claim validity back as current belief. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_claims_immutable_append_only.yaml b/plan/implementation_evals/eval_checks/e2_claims_immutable_append_only.yaml index 26f7a38..34c597e 100644 --- a/plan/implementation_evals/eval_checks/e2_claims_immutable_append_only.yaml +++ b/plan/implementation_evals/eval_checks/e2_claims_immutable_append_only.yaml @@ -5,7 +5,7 @@ description: Claims are immutable, append-only evidence — no update path, no s tags: [e2, D2, D3, D54] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D2/D3 (and D54's "currency is bookkeeping, never validity"), then inspect + Read decisions.md D2/D3 (and D54's "currency is bookkeeping, never validity") and plan/designs/e2_e3_claims_relations_design.md, then inspect the claims schema/models and every code path that touches claim rows under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -20,5 +20,12 @@ instructions: | defaults, counting), never validity: transaction-time reconstructions still see old generations. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml b/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml index f740c2c..50d4216 100644 --- a/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml +++ b/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml @@ -23,5 +23,12 @@ instructions: | 4. Extraction batching (if present) batches a section's contiguous chunks per call while keeping claims anchored per-chunk and idempotency keys per-chunk (D58). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml b/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml index 21fc12b..1ade67d 100644 --- a/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml +++ b/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml @@ -24,5 +24,12 @@ instructions: | exist); a claim with an unverifiable added substring is REJECTED. 4. Grounding uses the bundle only — no external/world knowledge is permitted or injected. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_no_pre_extraction_gate.yaml b/plan/implementation_evals/eval_checks/e2_no_pre_extraction_gate.yaml index c0d7e04..d28252f 100644 --- a/plan/implementation_evals/eval_checks/e2_no_pre_extraction_gate.yaml +++ b/plan/implementation_evals/eval_checks/e2_no_pre_extraction_gate.yaml @@ -21,5 +21,12 @@ instructions: | be justified in the repo by a measured break-even — absent that evidence, its presence is a violation. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_selection_ledger_replay.yaml b/plan/implementation_evals/eval_checks/e2_selection_ledger_replay.yaml index a0e0e5c..fbb3620 100644 --- a/plan/implementation_evals/eval_checks/e2_selection_ledger_replay.yaml +++ b/plan/implementation_evals/eval_checks/e2_selection_ledger_replay.yaml @@ -20,5 +20,12 @@ instructions: | rungs are replay-from-storage); the extraction worker is idempotent on its content-based input hash + extractor version. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_selection_recall_envelope.yaml b/plan/implementation_evals/eval_checks/e2_selection_recall_envelope.yaml index 45de0c2..9196b3f 100644 --- a/plan/implementation_evals/eval_checks/e2_selection_recall_envelope.yaml +++ b/plan/implementation_evals/eval_checks/e2_selection_recall_envelope.yaml @@ -5,7 +5,7 @@ description: Selection is recall-conservative — KEEP bias, never-drop classes, tags: [e2, D35, D22] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D35, then inspect the Selection prompt/rubric, outcome enum, and eval + Read decisions.md D35 and plan/designs/e2_e3_claims_relations_design.md §3.4, then inspect the Selection prompt/rubric, outcome enum, and eval harness under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -19,5 +19,12 @@ instructions: | 4. The eval harness plants rare-fact canaries and fails CI if Selection drops one; drop-rate tuning is per-fact, not corpus-average. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_stance_kept_as_observation.yaml b/plan/implementation_evals/eval_checks/e2_stance_kept_as_observation.yaml index 645b6b5..e6f2911 100644 --- a/plan/implementation_evals/eval_checks/e2_stance_kept_as_observation.yaml +++ b/plan/implementation_evals/eval_checks/e2_stance_kept_as_observation.yaml @@ -5,7 +5,7 @@ description: Attributed stance is kept and normalized to an observation on the h tags: [e2, e3, D59, D43] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D59, then inspect the Selection rubric and E3 normalization under src/. + Read decisions.md D59 and plan/designs/observations_design.md, then inspect the Selection rubric and E3 normalization under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -23,5 +23,12 @@ instructions: | whose holder cannot be decontextualized to a resolvable entity drops (ledger reason opinion, meaning unattributed-only). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e2_testimony_currency_counting.yaml b/plan/implementation_evals/eval_checks/e2_testimony_currency_counting.yaml index 746c982..71c6b36 100644 --- a/plan/implementation_evals/eval_checks/e2_testimony_currency_counting.yaml +++ b/plan/implementation_evals/eval_checks/e2_testimony_currency_counting.yaml @@ -29,5 +29,12 @@ instructions: | 5. P1's default claim channel indexes current testimony only; an audit channel/opt-in sees all generations. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e3_blocking_cheap_first_cascade.yaml b/plan/implementation_evals/eval_checks/e3_blocking_cheap_first_cascade.yaml index 1091271..3574b80 100644 --- a/plan/implementation_evals/eval_checks/e3_blocking_cheap_first_cascade.yaml +++ b/plan/implementation_evals/eval_checks/e3_blocking_cheap_first_cascade.yaml @@ -5,7 +5,7 @@ description: Supersession candidates come from (entity, predicate) blocking with tags: [e3, D4, D43] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D4, then inspect the supersession worker under src/. These are acceptance + Read decisions.md D4 and plan/designs/e2_e3_claims_relations_design.md §5, then inspect the supersession worker under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: @@ -18,5 +18,12 @@ instructions: | 4. Observation supersession follows the same cascade blocked on the resolved entity (D43), sharing the engine rather than duplicating it. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml b/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml index 56f5950..d08f945 100644 --- a/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml +++ b/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml @@ -27,5 +27,12 @@ instructions: | coexist/new (the failure mode is a duplicate row, never a silent overwrite). 5. Observations never project to the P2 graph; they are embedded in P1 for retrieval. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml b/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml index 608cdb2..926b630 100644 --- a/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml +++ b/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml @@ -23,5 +23,12 @@ instructions: | 5. Extraction prompts render from the registry (types/predicates/descriptions/examples), so ontology changes are row edits, not prompt engineering. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e3_relations_evidence_collapse.yaml b/plan/implementation_evals/eval_checks/e3_relations_evidence_collapse.yaml index eb11a37..65c290b 100644 --- a/plan/implementation_evals/eval_checks/e3_relations_evidence_collapse.yaml +++ b/plan/implementation_evals/eval_checks/e3_relations_evidence_collapse.yaml @@ -17,5 +17,12 @@ instructions: | EXISTING relation rather than inserting a duplicate relation row. 3. Duplicate evidence is idempotent (evidence-once per claim/fact pair). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/e3_supersession_fact_level_bitemporal.yaml b/plan/implementation_evals/eval_checks/e3_supersession_fact_level_bitemporal.yaml index 73e60ff..d8a68e5 100644 --- a/plan/implementation_evals/eval_checks/e3_supersession_fact_level_bitemporal.yaml +++ b/plan/implementation_evals/eval_checks/e3_supersession_fact_level_bitemporal.yaml @@ -5,7 +5,7 @@ description: Supersession closes validity windows on relations and effective-sta tags: [e3, D3, D43, D55] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D3 and D43 (the no-cap rule) and D55 (source-removal shapes), then inspect + Read decisions.md D3 and D43 (the no-cap rule) and D55 (source-removal shapes) and plan/designs/postgres_schema_design.md §9, then inspect the supersession write paths and temporal columns under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -23,5 +23,12 @@ instructions: | 4. Both time-travel questions are answerable in the query layer: what was true at T (valid-time) and what did we believe at T (transaction-time). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/embedding_model_port_config_d63.yaml b/plan/implementation_evals/eval_checks/embedding_model_port_config_d63.yaml index 3113916..482dafa 100644 --- a/plan/implementation_evals/eval_checks/embedding_model_port_config_d63.yaml +++ b/plan/implementation_evals/eval_checks/embedding_model_port_config_d63.yaml @@ -5,7 +5,7 @@ description: The embedding model is provider-port configuration, never architect tags: [p1, e1, D63, D61] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D63 (and D61's embedder port), then inspect the embedder port, its + Read decisions.md D63 (and D61's embedder port) and plan/designs/e1_chunks_design.md §5, then inspect the embedder port, its adapters, and the E1 pipeline under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -22,5 +22,12 @@ instructions: | output stored and carried forward for unchanged regions (D56) — and prefixes never enter reuse keys. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/er_merges_reversible.yaml b/plan/implementation_evals/eval_checks/er_merges_reversible.yaml index a6b3a7e..c86e292 100644 --- a/plan/implementation_evals/eval_checks/er_merges_reversible.yaml +++ b/plan/implementation_evals/eval_checks/er_merges_reversible.yaml @@ -5,7 +5,7 @@ description: Entity merges are reversible, append-only adjudications — decisio tags: [registries, D21] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D21, then inspect the merge/cluster machinery under src/. These are + Read decisions.md D21 and plan/designs/registries_design.md, then inspect the merge/cluster machinery under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: @@ -20,5 +20,12 @@ instructions: | 4. A generic-identifier guard exists (an alias that suddenly links many entities is down-weighted and re-evaluated). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/er_resolution_cascade_t0_t4.yaml b/plan/implementation_evals/eval_checks/er_resolution_cascade_t0_t4.yaml index 55bdbb5..9dbc5ed 100644 --- a/plan/implementation_evals/eval_checks/er_resolution_cascade_t0_t4.yaml +++ b/plan/implementation_evals/eval_checks/er_resolution_cascade_t0_t4.yaml @@ -21,5 +21,12 @@ instructions: | 4. No third-party external authority (Wikidata, ORCID, etc.) is called anywhere on the resolution path, and no external_ids table exists (D20). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/er_review_queue_cli.yaml b/plan/implementation_evals/eval_checks/er_review_queue_cli.yaml index f9eab54..953a764 100644 --- a/plan/implementation_evals/eval_checks/er_review_queue_cli.yaml +++ b/plan/implementation_evals/eval_checks/er_review_queue_cli.yaml @@ -5,7 +5,7 @@ description: Human review is a thin Postgres-backed CLI cluster queue — impact tags: [registries, D24, D60] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D24 (and D60), then inspect the review tooling under src/. These are + Read decisions.md D24 (and D60) and plan/designs/registries_design.md, then inspect the review tooling under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: @@ -17,5 +17,12 @@ instructions: | 4. No web review UI ships in this repository (D60: the web UI is the cloud product's surface; the library's review surface is the CLI). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/eval_golden_sets_d22.yaml b/plan/implementation_evals/eval_checks/eval_golden_sets_d22.yaml index 4a7b8d9..a65962d 100644 --- a/plan/implementation_evals/eval_checks/eval_golden_sets_d22.yaml +++ b/plan/implementation_evals/eval_checks/eval_golden_sets_d22.yaml @@ -22,5 +22,12 @@ instructions: | 4. Eval runs are recorded (versioned results), so threshold changes are justified by measurements, not vibes. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/k_citations_binding.yaml b/plan/implementation_evals/eval_checks/k_citations_binding.yaml index 94750ee..ff3a755 100644 --- a/plan/implementation_evals/eval_checks/k_citations_binding.yaml +++ b/plan/implementation_evals/eval_checks/k_citations_binding.yaml @@ -20,5 +20,12 @@ instructions: | 4. Deletion reach and staleness for K are computed from these citation records (SQL over recorded inputs), demonstrating the citations are load-bearing, not decorative. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/k_compiled_vs_authored_ownership.yaml b/plan/implementation_evals/eval_checks/k_compiled_vs_authored_ownership.yaml index 48fa949..97ac9f6 100644 --- a/plan/implementation_evals/eval_checks/k_compiled_vs_authored_ownership.yaml +++ b/plan/implementation_evals/eval_checks/k_compiled_vs_authored_ownership.yaml @@ -20,5 +20,12 @@ instructions: | 4. Authored pages are never auto-regenerated; when evidence they cite changes they receive a review FLAG (not a rewrite) — the system never rewrites human words, even to forget. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/k_one_mechanism_n_scopes.yaml b/plan/implementation_evals/eval_checks/k_one_mechanism_n_scopes.yaml index 878db53..32ddcf9 100644 --- a/plan/implementation_evals/eval_checks/k_one_mechanism_n_scopes.yaml +++ b/plan/implementation_evals/eval_checks/k_one_mechanism_n_scopes.yaml @@ -23,5 +23,12 @@ instructions: | single automated committer, and trigger-surface acyclicity are enforced regardless of configuration. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/k_planner_writer_driver.yaml b/plan/implementation_evals/eval_checks/k_planner_writer_driver.yaml index b23ce33..3810aa0 100644 --- a/plan/implementation_evals/eval_checks/k_planner_writer_driver.yaml +++ b/plan/implementation_evals/eval_checks/k_planner_writer_driver.yaml @@ -24,5 +24,12 @@ instructions: | longer matches — and inputs_hash keys on FACT STATE, never raw claim IDs (D54's stale-storm guard); claim-grain citations key on (lineage, chunk_content_hash). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ops_deletion_cascade_grains.yaml b/plan/implementation_evals/eval_checks/ops_deletion_cascade_grains.yaml index 46a8edd..8166c5d 100644 --- a/plan/implementation_evals/eval_checks/ops_deletion_cascade_grains.yaml +++ b/plan/implementation_evals/eval_checks/ops_deletion_cascade_grains.yaml @@ -27,5 +27,12 @@ instructions: | 5. A file observed deleted at its source is treated as lineage deletion, stamped with the observing sync cycle. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ops_execution_classes_bound.yaml b/plan/implementation_evals/eval_checks/ops_execution_classes_bound.yaml index 6a01afb..3f6a984 100644 --- a/plan/implementation_evals/eval_checks/ops_execution_classes_bound.yaml +++ b/plan/implementation_evals/eval_checks/ops_execution_classes_bound.yaml @@ -24,5 +24,12 @@ instructions: | chunked document is fully extracted by the fixed two-call extractor) — do not fail it for scaling with volume, and do not accept it becoming an agent harness. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ops_idempotency_dlq.yaml b/plan/implementation_evals/eval_checks/ops_idempotency_dlq.yaml index 2fac3e8..602403a 100644 --- a/plan/implementation_evals/eval_checks/ops_idempotency_dlq.yaml +++ b/plan/implementation_evals/eval_checks/ops_idempotency_dlq.yaml @@ -23,5 +23,12 @@ instructions: | 4. The per-document E chain is task-driven; aggregate layers (K, P) are debounced or scheduled, never triggered per document. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ops_metering_budgets_enforced.yaml b/plan/implementation_evals/eval_checks/ops_metering_budgets_enforced.yaml index 09a82cc..b32ba20 100644 --- a/plan/implementation_evals/eval_checks/ops_metering_budgets_enforced.yaml +++ b/plan/implementation_evals/eval_checks/ops_metering_budgets_enforced.yaml @@ -17,5 +17,12 @@ instructions: | 3. Spend is attributable per layer/stage/deployment (the metering supports the design's dashboards and the cheap-first verification). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ops_producer_checker_families.yaml b/plan/implementation_evals/eval_checks/ops_producer_checker_families.yaml index c30acdc..a0934c0 100644 --- a/plan/implementation_evals/eval_checks/ops_producer_checker_families.yaml +++ b/plan/implementation_evals/eval_checks/ops_producer_checker_families.yaml @@ -5,7 +5,7 @@ description: Every checking seat runs on a different model family than the produ tags: [ops, D53] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D53, then inspect the model assignment configuration under src/. These are + Read decisions.md D53 and plan/designs/orchestration_design.md + plan/designs/k_layers_design.md §7, then inspect the model assignment configuration under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: @@ -18,5 +18,12 @@ instructions: | 3. Model assignments are recorded (pipeline_component_versions naming producer and checker models) so the separation is auditable after the fact. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ops_versioned_replay_from_storage.yaml b/plan/implementation_evals/eval_checks/ops_versioned_replay_from_storage.yaml index 0616798..f34fbed 100644 --- a/plan/implementation_evals/eval_checks/ops_versioned_replay_from_storage.yaml +++ b/plan/implementation_evals/eval_checks/ops_versioned_replay_from_storage.yaml @@ -17,5 +17,12 @@ instructions: | 3. A version bump reprocesses exactly the affected artifacts (version-filtered batch), and downstream invalidation keys include upstream versions. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/p1_lance_estate_rebuildable.yaml b/plan/implementation_evals/eval_checks/p1_lance_estate_rebuildable.yaml index ec379c0..1e5c529 100644 --- a/plan/implementation_evals/eval_checks/p1_lance_estate_rebuildable.yaml +++ b/plan/implementation_evals/eval_checks/p1_lance_estate_rebuildable.yaml @@ -17,5 +17,12 @@ instructions: | 3. Every embedded row is stamped with its embedding model/version, and re-embedding by version filter is a supported batch operation (the embedding-migration path). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/p2_asof_and_communities.yaml b/plan/implementation_evals/eval_checks/p2_asof_and_communities.yaml index 414b1b5..d02d647 100644 --- a/plan/implementation_evals/eval_checks/p2_asof_and_communities.yaml +++ b/plan/implementation_evals/eval_checks/p2_asof_and_communities.yaml @@ -5,7 +5,7 @@ description: As-of traversal is inline path-predicate filtering (never MATCH ove tags: [p2, D10, D11, D44] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D10/D11/D44, then inspect the graph query layer and the community pass + Read decisions.md D10/D11/D44 and plan/designs/p2_graph_design.md, then inspect the graph query layer and the community pass under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -18,5 +18,12 @@ instructions: | 3. Community assignments and centrality write back to POSTGRES (the graph stays a projection); graph-derived metrics are computed post-load, never reprojected as truth. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/p2_no_vectors_in_graph.yaml b/plan/implementation_evals/eval_checks/p2_no_vectors_in_graph.yaml index 1a15685..66c0ef7 100644 --- a/plan/implementation_evals/eval_checks/p2_no_vectors_in_graph.yaml +++ b/plan/implementation_evals/eval_checks/p2_no_vectors_in_graph.yaml @@ -5,7 +5,7 @@ description: The graph holds no embeddings and no FTS — vectors live in Lance; tags: [p2, p1, D8] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D8, then inspect the P2 snapshot contents and the Lance table definitions + Read decisions.md D8 and plan/designs/p2_graph_design.md, then inspect the P2 snapshot contents and the Lance table definitions under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -17,5 +17,12 @@ instructions: | 3. The retrieval flow is Lance-entry -> ID-keyed expansion in the graph (top-k relation_ids then BFS/paths), not in-graph semantic search. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/p2_projection_contract_views.yaml b/plan/implementation_evals/eval_checks/p2_projection_contract_views.yaml index 5ae357e..6a1a8df 100644 --- a/plan/implementation_evals/eval_checks/p2_projection_contract_views.yaml +++ b/plan/implementation_evals/eval_checks/p2_projection_contract_views.yaml @@ -26,5 +26,12 @@ instructions: | per-type node tables or per-predicate rel tables; entity IDs stay UUID. 5. Observations and claims never project into the graph. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml b/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml index 36463de..cca8f39 100644 --- a/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml +++ b/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml @@ -19,5 +19,12 @@ instructions: | entity merges are handled by the rebuild, not by re-pointing edges in the graph. 4. P3 follows the same discipline: full snapshot build + atomic latest-pointer swap (D40). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/p3_corpus_fs_projection.yaml b/plan/implementation_evals/eval_checks/p3_corpus_fs_projection.yaml index 776721b..e3b9f94 100644 --- a/plan/implementation_evals/eval_checks/p3_corpus_fs_projection.yaml +++ b/plan/implementation_evals/eval_checks/p3_corpus_fs_projection.yaml @@ -20,5 +20,12 @@ instructions: | 4. P3 paths and K citations anchor on document LINEAGES (stable across versions/renames), so an edited document keeps its path. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml b/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml index 51474a8..4fcda83 100644 --- a/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml +++ b/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml @@ -5,7 +5,7 @@ description: Validity state has exactly one home (Postgres); projection builders tags: [p, D6, D62] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D6 (and D62's enforced architecture), then inspect the projection builders + Read decisions.md D6 (and D62's enforced architecture) and plan/designs/overall_design.md §2, then inspect the projection builders and the import-linter contracts under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -22,5 +22,12 @@ instructions: | 4. The graph writer is deterministic — no adjudication, no supersession logic, no LLM calls inside projection builders. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/registry_scale_partitioning_d23.yaml b/plan/implementation_evals/eval_checks/registry_scale_partitioning_d23.yaml index 4bb2472..699783a 100644 --- a/plan/implementation_evals/eval_checks/registry_scale_partitioning_d23.yaml +++ b/plan/implementation_evals/eval_checks/registry_scale_partitioning_d23.yaml @@ -20,5 +20,12 @@ instructions: | (subject_entity_id, predicate[, object]). 4. Embedding similarity (T3) runs in Lance — no pgvector/HNSW index in OLTP Postgres. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_claims_never_answer_now.yaml b/plan/implementation_evals/eval_checks/ret_claims_never_answer_now.yaml index 864c52e..ba1309d 100644 --- a/plan/implementation_evals/eval_checks/ret_claims_never_answer_now.yaml +++ b/plan/implementation_evals/eval_checks/ret_claims_never_answer_now.yaml @@ -19,5 +19,12 @@ instructions: | believed), matching the requirement that an agent must never read a claim's validity as the system's current belief. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_envelope_contradiction_completeness.yaml b/plan/implementation_evals/eval_checks/ret_envelope_contradiction_completeness.yaml index bfdbf33..6267037 100644 --- a/plan/implementation_evals/eval_checks/ret_envelope_contradiction_completeness.yaml +++ b/plan/implementation_evals/eval_checks/ret_envelope_contradiction_completeness.yaml @@ -1,7 +1,7 @@ schema_version: 1 id: ret_envelope_contradiction_completeness type: harness_judge -description: Contradiction co-members are never silently absent — inline up to a cap, then group_id + returned/total + continuation; one-sided answers are a contract violation. +description: Contradiction co-members are never silently absent — inline up to a cap and plan/designs/retrieval_design.md, then group_id + returned/total + continuation; one-sided answers are a contract violation. tags: [retrieval, D49] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. @@ -15,5 +15,12 @@ instructions: | 2. The code path makes a one-sided answer impossible for contradicted facts (a contract test demonstrates it) — never a ranking choice that happens to drop the other side. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_envelope_freshness_stamps.yaml b/plan/implementation_evals/eval_checks/ret_envelope_freshness_stamps.yaml index 68e2273..1256ae5 100644 --- a/plan/implementation_evals/eval_checks/ret_envelope_freshness_stamps.yaml +++ b/plan/implementation_evals/eval_checks/ret_envelope_freshness_stamps.yaml @@ -5,7 +5,7 @@ description: Per-source freshness stamps ship in every envelope — PG live, P1 tags: [retrieval, D49, D48] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D49 (freshness stamps) and D48, then inspect the envelope freshness + Read decisions.md D49 (freshness stamps) and D48 and plan/designs/retrieval_design.md, then inspect the envelope freshness section under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -18,5 +18,12 @@ instructions: | 3. The applied temporal parameters (valid_at / believed_at) and the identity regime (current aliases vs identity_as_of) are echoed in composition-ready form. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_envelope_grain_discipline.yaml b/plan/implementation_evals/eval_checks/ret_envelope_grain_discipline.yaml index a3e55e5..10b6d80 100644 --- a/plan/implementation_evals/eval_checks/ret_envelope_grain_discipline.yaml +++ b/plan/implementation_evals/eval_checks/ret_envelope_grain_discipline.yaml @@ -17,5 +17,12 @@ instructions: | 3. A composite answer is parts[], each part strictly single-grain — mixed answers never dilute the discipline. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_envelope_negative_taxonomy.yaml b/plan/implementation_evals/eval_checks/ret_envelope_negative_taxonomy.yaml index 3cefa64..9de88e6 100644 --- a/plan/implementation_evals/eval_checks/ret_envelope_negative_taxonomy.yaml +++ b/plan/implementation_evals/eval_checks/ret_envelope_negative_taxonomy.yaml @@ -5,7 +5,7 @@ description: A typed negative taxonomy — unknown_entity / known_empty / bounda tags: [retrieval, D49, D50] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D49 (typed negatives) and D50 (trust model), then inspect the negative + Read decisions.md D49 (typed negatives) and D50 (trust model) and plan/designs/retrieval_design.md, then inspect the negative result types under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -18,5 +18,12 @@ instructions: | same shape, no distinguishing metadata; a contract test (or CI gate wired to the deletion cascade) demonstrates the indistinguishability. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_envelope_truncation_markers.yaml b/plan/implementation_evals/eval_checks/ret_envelope_truncation_markers.yaml index 4eb35f9..799bfc4 100644 --- a/plan/implementation_evals/eval_checks/ret_envelope_truncation_markers.yaml +++ b/plan/implementation_evals/eval_checks/ret_envelope_truncation_markers.yaml @@ -5,7 +5,7 @@ description: No silent caps anywhere — truncation is always marked with a cont tags: [retrieval, D49] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D49, then inspect result assembly and pagination under src/. These are + Read decisions.md D49 and plan/designs/retrieval_design.md, then inspect result assembly and pagination under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: @@ -15,5 +15,12 @@ instructions: | 2. Hub-entity answers page deterministically (ranked pages with continuations), and no query path resolves a too-big answer by silently timing out or capping. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_filesystem_first_mounts_skill.yaml b/plan/implementation_evals/eval_checks/ret_filesystem_first_mounts_skill.yaml index fd4c56c..f1cadaa 100644 --- a/plan/implementation_evals/eval_checks/ret_filesystem_first_mounts_skill.yaml +++ b/plan/implementation_evals/eval_checks/ret_filesystem_first_mounts_skill.yaml @@ -22,5 +22,12 @@ instructions: | semantics, the mount layout, and the precedence rule (filesystem for what a filesystem can do; API for search/graph/as-of/hydration). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_hydration_reverifies.yaml b/plan/implementation_evals/eval_checks/ret_hydration_reverifies.yaml index f4b56c8..b59d63e 100644 --- a/plan/implementation_evals/eval_checks/ret_hydration_reverifies.yaml +++ b/plan/implementation_evals/eval_checks/ret_hydration_reverifies.yaml @@ -22,5 +22,12 @@ instructions: | freshness metadata) and K prose (compiled-grain with staleness state) — nothing else skips hydration. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_primitives_recipes_registry.yaml b/plan/implementation_evals/eval_checks/ret_primitives_recipes_registry.yaml index 8cd651b..e65ba7b 100644 --- a/plan/implementation_evals/eval_checks/ret_primitives_recipes_registry.yaml +++ b/plan/implementation_evals/eval_checks/ret_primitives_recipes_registry.yaml @@ -23,5 +23,12 @@ instructions: | 4. There is no natural-language-to-query-plan compiler anywhere on the query path, and no LLM call in the core search path (D9). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ret_trust_boundary_no_content_auth.yaml b/plan/implementation_evals/eval_checks/ret_trust_boundary_no_content_auth.yaml index 7a0e502..9f60007 100644 --- a/plan/implementation_evals/eval_checks/ret_trust_boundary_no_content_auth.yaml +++ b/plan/implementation_evals/eval_checks/ret_trust_boundary_no_content_auth.yaml @@ -5,7 +5,7 @@ description: One trust domain per deployment — no content-level authorization tags: [retrieval, D50, D60, D61] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D50 (trust model) and D60/D61, then inspect the API/mount surfaces under + Read decisions.md D50 (trust model) and D60/D61 and plan/designs/retrieval_design.md §9, then inspect the API/mount surfaces under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -18,5 +18,12 @@ instructions: | 3. There is no orgs/users/RBAC/SSO machinery in this repository (D60: that is the cloud product). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/scope_views_share_graph_d16.yaml b/plan/implementation_evals/eval_checks/scope_views_share_graph_d16.yaml index 1d9430d..4886055 100644 --- a/plan/implementation_evals/eval_checks/scope_views_share_graph_d16.yaml +++ b/plan/implementation_evals/eval_checks/scope_views_share_graph_d16.yaml @@ -5,7 +5,7 @@ description: One graph, many lenses — scopes never get their own entity space tags: [cross, D16, D50] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D16 (and D50's trust-model refinement), then inspect scope handling under + Read decisions.md D16 (and D50's trust-model refinement) and plan/designs/registries_design.md §1, then inspect scope handling under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -19,5 +19,12 @@ instructions: | 3. Nothing uses scope views as an access-control mechanism — different trust boundaries are served by separate deployments, never by in-library per-scope filtering (D50). + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/source_of_truth_split_d1_d46.yaml b/plan/implementation_evals/eval_checks/source_of_truth_split_d1_d46.yaml index 0277123..df36348 100644 --- a/plan/implementation_evals/eval_checks/source_of_truth_split_d1_d46.yaml +++ b/plan/implementation_evals/eval_checks/source_of_truth_split_d1_d46.yaml @@ -5,7 +5,7 @@ description: The split source of truth holds — Postgres authoritative for the tags: [cross, D1, D46] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D1 and D46, then inspect the store boundaries under src/. These are + Read decisions.md D1 and D46 and plan/designs/overall_design.md §2, then inspect the store boundaries under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: @@ -18,5 +18,12 @@ instructions: | citations, inputs_hash, prompt/model versions) are sufficient to recompile a page from the spine without consulting the old page body. + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. Otherwise score 0. From c19702e4c6b57fbfd798509343d5616a096c5d51 Mon Sep 17 00:00:00 2001 From: Jiri Puc Date: Sat, 11 Jul 2026 16:32:33 +0200 Subject: [PATCH 5/7] Evals README: record the source-precedence convention (decisions for coverage, designs for truth) Co-Authored-By: Claude Fable 5 --- plan/implementation_evals/README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/plan/implementation_evals/README.md b/plan/implementation_evals/README.md index 2c9bd5d..557332c 100644 --- a/plan/implementation_evals/README.md +++ b/plan/implementation_evals/README.md @@ -42,6 +42,16 @@ Two conventions when running: - Where an invariant is a *deliberate exception* (e.g. E2 extraction is volume-proportional by D25 while adjudication is ambiguity-scaled by D4), the check says so explicitly, so a judge cannot fail a correct implementation for obeying the design. +- **Decisions for coverage, designs for truth.** The decision log is an append-only record + whose entries are refined and withdrawn by later ones — it serves as the *coverage index* + (which binding decisions have checks), while the design docs are the binding statement of + the *current* system. Every check therefore carries an explicit **source-precedence rule** + in its instructions: if cited sources appear to disagree, the design docs (including + `postgres_schema_design.md`) are controlling, decision entries are read with their + refinement/withdrawal annotations, and the judge reports the discrepancy instead of failing + an implementation that follows the current design. (This is the repo's own claims-vs-facts + epistemology applied to its documentation: the log is testimony; the designs are the + adjudicated current belief.) ## Inventory (69 checks) From 9297f7dae5928cc276e8115c2f30c80267dfaa91 Mon Sep 17 00:00:00 2001 From: Jiri Puc Date: Sun, 12 Jul 2026 10:53:36 +0200 Subject: [PATCH 6/7] =?UTF-8?q?Evals:=20read=20design-first=20=E2=80=94=20?= =?UTF-8?q?every=20check=20leads=20with=20the=20binding=20design=20doc?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Author review round 2: the precedence rule alone left the checks decision-first in reading order (64/69 opened with "Read decisions.md Dxx and "). Flipped all of them: the Read instruction now leads with the design doc / requirements section as "the binding source", with decisions.md cited second for rationale and refinement history. Also repaired ret_envelope_contradiction_completeness, whose description had been corrupted by the earlier cite-insertion (its description contained ", then", which the insertion script matched instead of the Read sentence). Full sweep: 69 checks valid, none decision-first, no polluted descriptions, every check cites a design/requirements source and carries the precedence rule. Co-Authored-By: Claude Fable 5 --- plan/implementation_evals/README.md | 19 +++++++++++-------- .../boundary_correctness_in_library_d60.yaml | 3 ++- ...undary_extension_points_no_bypass_d60.yaml | 3 ++- .../boundary_no_human_control_plane_d60.yaml | 3 ++- .../boundary_provider_ports_d61.yaml | 3 ++- .../code_architecture_enforced_d62.yaml | 3 ++- ...elivery_artifacts_and_task_shells_d62.yaml | 3 ++- .../e0_blockizer_owns_identity.yaml | 3 ++- .../e0_content_addressed_reuse.yaml | 4 ++-- .../e0_converter_router_versioned.yaml | 4 ++-- .../e0_lineage_version_schema.yaml | 3 ++- .../e0_living_mode_retraction.yaml | 3 ++- .../e0_origin_stamped_at_ingest.yaml | 3 ++- .../e0_postgres_holds_no_bodies.yaml | 3 ++- .../e0_raw_immutable_id_addressed.yaml | 4 ++-- .../e0_structure_contract_unconditional.yaml | 3 ++- .../e0_subworker_chain_crossrefs_d36.yaml | 3 ++- .../e1_chunks_whole_blocks_no_overlap.yaml | 3 ++- .../e1_multigranularity_retrieval.yaml | 3 ++- .../e2_asserted_validity_immutable.yaml | 3 ++- .../e2_claims_immutable_append_only.yaml | 3 ++- .../e2_context_bundle_two_calls.yaml | 3 ++- .../e2_grounding_layered_dual_field.yaml | 3 ++- .../e2_no_pre_extraction_gate.yaml | 3 ++- .../e2_selection_ledger_replay.yaml | 4 ++-- .../e2_selection_recall_envelope.yaml | 3 ++- .../e2_stance_kept_as_observation.yaml | 3 ++- .../e2_testimony_currency_counting.yaml | 3 ++- .../e3_blocking_cheap_first_cascade.yaml | 3 ++- .../e3_observations_untyped_adjudicated.yaml | 3 ++- .../e3_predicate_registry_governed.yaml | 3 ++- .../e3_relations_evidence_collapse.yaml | 3 ++- ...e3_supersession_fact_level_bitemporal.yaml | 3 ++- .../embedding_model_port_config_d63.yaml | 3 ++- .../eval_checks/er_merges_reversible.yaml | 3 ++- .../er_resolution_cascade_t0_t4.yaml | 3 ++- .../eval_checks/er_review_queue_cli.yaml | 3 ++- .../eval_checks/eval_golden_sets_d22.yaml | 4 ++-- .../eval_checks/k_citations_binding.yaml | 3 ++- .../k_compiled_vs_authored_ownership.yaml | 3 ++- .../eval_checks/k_one_mechanism_n_scopes.yaml | 3 ++- .../eval_checks/k_planner_writer_driver.yaml | 3 ++- .../ops_deletion_cascade_grains.yaml | 4 ++-- .../ops_execution_classes_bound.yaml | 4 ++-- .../eval_checks/ops_idempotency_dlq.yaml | 4 ++-- .../ops_producer_checker_families.yaml | 3 ++- .../p1_lance_estate_rebuildable.yaml | 3 ++- .../eval_checks/p2_asof_and_communities.yaml | 3 ++- .../eval_checks/p2_no_vectors_in_graph.yaml | 3 ++- .../p2_projection_contract_views.yaml | 3 ++- .../p2_rebuild_first_snapshots.yaml | 3 ++- .../eval_checks/p3_corpus_fs_projection.yaml | 4 ++-- .../p_projections_hold_no_authority.yaml | 3 ++- .../registry_scale_partitioning_d23.yaml | 3 ++- ...t_envelope_contradiction_completeness.yaml | 8 +++++--- .../ret_envelope_freshness_stamps.yaml | 3 ++- .../ret_envelope_grain_discipline.yaml | 3 ++- .../ret_envelope_negative_taxonomy.yaml | 3 ++- .../ret_envelope_truncation_markers.yaml | 3 ++- .../ret_filesystem_first_mounts_skill.yaml | 3 ++- .../eval_checks/ret_hydration_reverifies.yaml | 3 ++- .../ret_primitives_recipes_registry.yaml | 3 ++- .../ret_trust_boundary_no_content_auth.yaml | 3 ++- .../scope_views_share_graph_d16.yaml | 3 ++- .../source_of_truth_split_d1_d46.yaml | 3 ++- 65 files changed, 142 insertions(+), 83 deletions(-) diff --git a/plan/implementation_evals/README.md b/plan/implementation_evals/README.md index 557332c..0223f9d 100644 --- a/plan/implementation_evals/README.md +++ b/plan/implementation_evals/README.md @@ -44,14 +44,17 @@ Two conventions when running: cannot fail a correct implementation for obeying the design. - **Decisions for coverage, designs for truth.** The decision log is an append-only record whose entries are refined and withdrawn by later ones — it serves as the *coverage index* - (which binding decisions have checks), while the design docs are the binding statement of - the *current* system. Every check therefore carries an explicit **source-precedence rule** - in its instructions: if cited sources appear to disagree, the design docs (including - `postgres_schema_design.md`) are controlling, decision entries are read with their - refinement/withdrawal annotations, and the judge reports the discrepancy instead of failing - an implementation that follows the current design. (This is the repo's own claims-vs-facts - epistemology applied to its documentation: the log is testimony; the designs are the - adjudicated current belief.) + (which binding decisions have checks; the D-numbers in tags and conditions are the repo's + *names* for invariants, used by the design docs themselves), while the design docs are the + binding statement of the *current* system. Every check therefore **reads design-first** + (its Read instruction leads with the design doc or requirements section as "the binding + source"; the decision entries are cited for rationale and refinement history) **and carries + an explicit source-precedence rule** in its instructions: if cited sources appear to + disagree, the design docs (including `postgres_schema_design.md`) are controlling, decision + entries are read with their refinement/withdrawal annotations, and the judge reports the + discrepancy instead of failing an implementation that follows the current design. (This is + the repo's own claims-vs-facts epistemology applied to its documentation: the log is + testimony; the designs are the adjudicated current belief.) ## Inventory (69 checks) diff --git a/plan/implementation_evals/eval_checks/boundary_correctness_in_library_d60.yaml b/plan/implementation_evals/eval_checks/boundary_correctness_in_library_d60.yaml index d6a02b0..d24e998 100644 --- a/plan/implementation_evals/eval_checks/boundary_correctness_in_library_d60.yaml +++ b/plan/implementation_evals/eval_checks/boundary_correctness_in_library_d60.yaml @@ -5,7 +5,8 @@ description: Every correctness-determining mechanism the design binds is impleme tags: [boundary, D60] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D60 and plan/requirements/requirements_v3.md §Vision, then verify each named correctness module exists in-repo and is + Read plan/requirements/requirements_v3.md §Vision — the binding source — and decisions.md D60 for rationale and + refinement history, then verify each named correctness module exists in-repo and is functional (not a stub raising NotImplemented, not disabled by default, not calling out to a proprietary cloud service to do the work). diff --git a/plan/implementation_evals/eval_checks/boundary_extension_points_no_bypass_d60.yaml b/plan/implementation_evals/eval_checks/boundary_extension_points_no_bypass_d60.yaml index 98382e7..330b20a 100644 --- a/plan/implementation_evals/eval_checks/boundary_extension_points_no_bypass_d60.yaml +++ b/plan/implementation_evals/eval_checks/boundary_extension_points_no_bypass_d60.yaml @@ -5,7 +5,8 @@ description: Extension points cannot bypass invariants — ingestion always writ tags: [boundary, D60, D62] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D60 (constraint 2) and D62 (client surface) and plan/designs/packaging_distribution_design.md (the client surface), then inspect every extension + Read plan/designs/packaging_distribution_design.md (the client surface) — the binding source — and decisions.md D60 (constraint 2) and D62 (client surface) for rationale and + refinement history, then inspect every extension point (connector interface, ingest API, review sink, provider ports) under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/boundary_no_human_control_plane_d60.yaml b/plan/implementation_evals/eval_checks/boundary_no_human_control_plane_d60.yaml index 3ad292f..1ef082f 100644 --- a/plan/implementation_evals/eval_checks/boundary_no_human_control_plane_d60.yaml +++ b/plan/implementation_evals/eval_checks/boundary_no_human_control_plane_d60.yaml @@ -5,7 +5,8 @@ description: No web UI and no multi-tenant control plane ship in this repository tags: [boundary, D60] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D60 and CLAUDE.md Rule 3, then survey the repository's shipped surfaces. + Read CLAUDE.md Rule 3 — the binding source — and decisions.md D60 for rationale and + refinement history, then survey the repository's shipped surfaces. These are acceptance checks for the FINAL system. Verify ALL of the following: diff --git a/plan/implementation_evals/eval_checks/boundary_provider_ports_d61.yaml b/plan/implementation_evals/eval_checks/boundary_provider_ports_d61.yaml index 508a5a7..d5c1669 100644 --- a/plan/implementation_evals/eval_checks/boundary_provider_ports_d61.yaml +++ b/plan/implementation_evals/eval_checks/boundary_provider_ports_d61.yaml @@ -5,7 +5,8 @@ description: The deployment substrate is reached only through the provider ports tags: [boundary, D61] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D61 and requirements_v3.md §Fixed choices & the reference deployment, then + Read requirements_v3.md §Fixed choices & the reference deployment — the binding source — and decisions.md D61 for rationale and + refinement history, then inspect the adapter layer and self-host stack under src/ and the deploy profiles. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/code_architecture_enforced_d62.yaml b/plan/implementation_evals/eval_checks/code_architecture_enforced_d62.yaml index a0a02f4..604a3aa 100644 --- a/plan/implementation_evals/eval_checks/code_architecture_enforced_d62.yaml +++ b/plan/implementation_evals/eval_checks/code_architecture_enforced_d62.yaml @@ -5,7 +5,8 @@ description: The hexagonal architecture is mechanically enforced — pure infra- tags: [code, D62] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D62 (the enforced code architecture) and plan/designs/packaging_distribution_design.md (the enforced layout), then inspect the package layout and + Read plan/designs/packaging_distribution_design.md (the enforced layout) — the binding source — and decisions.md D62 (the enforced code architecture) for rationale and + refinement history, then inspect the package layout and the import-linter configuration under the repo. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/delivery_artifacts_and_task_shells_d62.yaml b/plan/implementation_evals/eval_checks/delivery_artifacts_and_task_shells_d62.yaml index 9f0087c..1dcc506 100644 --- a/plan/implementation_evals/eval_checks/delivery_artifacts_and_task_shells_d62.yaml +++ b/plan/implementation_evals/eval_checks/delivery_artifacts_and_task_shells_d62.yaml @@ -5,7 +5,8 @@ description: The delivery contract holds — PyPI client package with extras, GH tags: [boundary, ops, D62] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D62 and plan/designs/packaging_distribution_design.md, then inspect the packaging, compose profile, client surface, and task + Read plan/designs/packaging_distribution_design.md — the binding source — and decisions.md D62 for rationale and + refinement history, then inspect the packaging, compose profile, client surface, and task execution under the repo. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e0_blockizer_owns_identity.yaml b/plan/implementation_evals/eval_checks/e0_blockizer_owns_identity.yaml index 62785d1..f92b633 100644 --- a/plan/implementation_evals/eval_checks/e0_blockizer_owns_identity.yaml +++ b/plan/implementation_evals/eval_checks/e0_blockizer_owns_identity.yaml @@ -5,7 +5,8 @@ description: One deterministic, versioned blockizer derives the block sequence; tags: [e0, e1, D57] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D57 and plan/designs/e1_chunks_design.md §2–§3, then locate the blockizer + Read plan/designs/e1_chunks_design.md §2–§3 — the binding source — and decisions.md D57 for rationale and + refinement history, then locate the blockizer under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e0_content_addressed_reuse.yaml b/plan/implementation_evals/eval_checks/e0_content_addressed_reuse.yaml index 6da9b3e..2dbef70 100644 --- a/plan/implementation_evals/eval_checks/e0_content_addressed_reuse.yaml +++ b/plan/implementation_evals/eval_checks/e0_content_addressed_reuse.yaml @@ -5,8 +5,8 @@ description: Re-ingesting an edited document costs proportional to the edit — tags: [e0, e2, D56] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D56 and plan/designs/evidence_lifecycle_design.md ("Content-addressed - reuse"), then inspect the E2 idempotency keys, the version-diff logic, and the tests under + Read plan/designs/evidence_lifecycle_design.md ("Content-addressed reuse") — the binding source — and decisions.md D56 for rationale and + refinement history, then inspect the E2 idempotency keys, the version-diff logic, and the tests under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e0_converter_router_versioned.yaml b/plan/implementation_evals/eval_checks/e0_converter_router_versioned.yaml index 14267f1..3398143 100644 --- a/plan/implementation_evals/eval_checks/e0_converter_router_versioned.yaml +++ b/plan/implementation_evals/eval_checks/e0_converter_router_versioned.yaml @@ -5,8 +5,8 @@ description: Conversion is a pluggable, versioned router; converters emit docume tags: [e0, D38, D57] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D38 and D57 (the refined converter contract) and - plan/designs/e0_files_design.md §3, then locate the conversion module under src/. These are + Read plan/designs/e0_files_design.md §3 — the binding source — and decisions.md D38 and D57 (the refined converter contract) for rationale and + refinement history, then locate the conversion module under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: diff --git a/plan/implementation_evals/eval_checks/e0_lineage_version_schema.yaml b/plan/implementation_evals/eval_checks/e0_lineage_version_schema.yaml index f3a2a9d..ffadb59 100644 --- a/plan/implementation_evals/eval_checks/e0_lineage_version_schema.yaml +++ b/plan/implementation_evals/eval_checks/e0_lineage_version_schema.yaml @@ -5,7 +5,8 @@ description: Documents are lineages with connector-native identity; versions are tags: [e0, D55] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D55 and plan/designs/evidence_lifecycle_design.md, then inspect the + Read plan/designs/evidence_lifecycle_design.md — the binding source — and decisions.md D55 for rationale and + refinement history, then inspect the documents/document_versions/content_objects schema and the sync logic under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e0_living_mode_retraction.yaml b/plan/implementation_evals/eval_checks/e0_living_mode_retraction.yaml index 166b4e1..ee36ae2 100644 --- a/plan/implementation_evals/eval_checks/e0_living_mode_retraction.yaml +++ b/plan/implementation_evals/eval_checks/e0_living_mode_retraction.yaml @@ -5,7 +5,8 @@ description: In living mode, removal retracts — sole-support loss closes facts tags: [e0, e3, D55, D54, D43] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D55 (removal semantics) and D54/D43 and plan/designs/evidence_lifecycle_design.md, then inspect the version-supersession + Read plan/designs/evidence_lifecycle_design.md — the binding source — and decisions.md D55 (removal semantics) and D54/D43 for rationale and + refinement history, then inspect the version-supersession and retraction logic under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e0_origin_stamped_at_ingest.yaml b/plan/implementation_evals/eval_checks/e0_origin_stamped_at_ingest.yaml index bd171b1..b6df30c 100644 --- a/plan/implementation_evals/eval_checks/e0_origin_stamped_at_ingest.yaml +++ b/plan/implementation_evals/eval_checks/e0_origin_stamped_at_ingest.yaml @@ -5,7 +5,8 @@ description: Every input carries an immutable origin stamp (external vs self/sys tags: [e0, D42] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D42 and plan/designs/postgres_schema_design.md (documents), then inspect the ingest worker and documents schema under src/. These + Read plan/designs/postgres_schema_design.md (documents) — the binding source — and decisions.md D42 for rationale and + refinement history, then inspect the ingest worker and documents schema under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: diff --git a/plan/implementation_evals/eval_checks/e0_postgres_holds_no_bodies.yaml b/plan/implementation_evals/eval_checks/e0_postgres_holds_no_bodies.yaml index 92348dd..54ad712 100644 --- a/plan/implementation_evals/eval_checks/e0_postgres_holds_no_bodies.yaml +++ b/plan/implementation_evals/eval_checks/e0_postgres_holds_no_bodies.yaml @@ -5,7 +5,8 @@ description: Postgres stores only compact metadata and the queryable section ind tags: [e0, D37] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D37 and plan/designs/e0_files_design.md §2, then inspect the schema + Read plan/designs/e0_files_design.md §2 — the binding source — and decisions.md D37 for rationale and + refinement history, then inspect the schema (migrations/models) under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e0_raw_immutable_id_addressed.yaml b/plan/implementation_evals/eval_checks/e0_raw_immutable_id_addressed.yaml index 88a0cd2..c53afe0 100644 --- a/plan/implementation_evals/eval_checks/e0_raw_immutable_id_addressed.yaml +++ b/plan/implementation_evals/eval_checks/e0_raw_immutable_id_addressed.yaml @@ -5,8 +5,8 @@ description: Raw bytes are stored immutably in a dedicated raw store, ID-address tags: [e0, D36, D37, D51] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D36/D37 (+ the "Refined by D51" note) and plan/designs/e0_files_design.md - §2, then locate the implementation under src/ (storage layout, ingest worker, mount config). + Read plan/designs/e0_files_design.md §2 — the binding source — and decisions.md D36/D37 (+ the "Refined by D51" note) for rationale and + refinement history, then locate the implementation under src/ (storage layout, ingest worker, mount config). These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: diff --git a/plan/implementation_evals/eval_checks/e0_structure_contract_unconditional.yaml b/plan/implementation_evals/eval_checks/e0_structure_contract_unconditional.yaml index 92ee953..f3a414d 100644 --- a/plan/implementation_evals/eval_checks/e0_structure_contract_unconditional.yaml +++ b/plan/implementation_evals/eval_checks/e0_structure_contract_unconditional.yaml @@ -5,7 +5,8 @@ description: Every document gets a section structure (synthetic root fallback), tags: [e0, D39] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D39 and plan/designs/e0_files_design.md §4, then locate the structurer and + Read plan/designs/e0_files_design.md §4 — the binding source — and decisions.md D39 for rationale and + refinement history, then locate the structurer and the document_sections schema under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e0_subworker_chain_crossrefs_d36.yaml b/plan/implementation_evals/eval_checks/e0_subworker_chain_crossrefs_d36.yaml index 019d3fc..963d7a9 100644 --- a/plan/implementation_evals/eval_checks/e0_subworker_chain_crossrefs_d36.yaml +++ b/plan/implementation_evals/eval_checks/e0_subworker_chain_crossrefs_d36.yaml @@ -5,7 +5,8 @@ description: E0 is one product layer implemented as separately idempotent, separ tags: [e0, D36] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D36 and plan/designs/e0_files_design.md §1, then inspect the E0 workers + Read plan/designs/e0_files_design.md §1 — the binding source — and decisions.md D36 for rationale and + refinement history, then inspect the E0 workers under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e1_chunks_whole_blocks_no_overlap.yaml b/plan/implementation_evals/eval_checks/e1_chunks_whole_blocks_no_overlap.yaml index 90f451f..88afea3 100644 --- a/plan/implementation_evals/eval_checks/e1_chunks_whole_blocks_no_overlap.yaml +++ b/plan/implementation_evals/eval_checks/e1_chunks_whole_blocks_no_overlap.yaml @@ -5,7 +5,8 @@ description: Chunks are non-overlapping runs of whole blocks within one section, tags: [e1, D58, D57] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D58 and plan/designs/e1_chunks_design.md §4, then locate the chunker under + Read plan/designs/e1_chunks_design.md §4 — the binding source — and decisions.md D58 for rationale and + refinement history, then locate the chunker under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e1_multigranularity_retrieval.yaml b/plan/implementation_evals/eval_checks/e1_multigranularity_retrieval.yaml index 08734f4..055b205 100644 --- a/plan/implementation_evals/eval_checks/e1_multigranularity_retrieval.yaml +++ b/plan/implementation_evals/eval_checks/e1_multigranularity_retrieval.yaml @@ -5,7 +5,8 @@ description: Claims are the needle index, chunks the passage index; default reci tags: [e1, p1, D58] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D58 (embedding granularity) and plan/designs/e1_chunks_design.md, then inspect the P1 index writers and default + Read plan/designs/e1_chunks_design.md — the binding source — and decisions.md D58 (embedding granularity) for rationale and + refinement history, then inspect the P1 index writers and default search recipes under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml b/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml index b180224..ebe195b 100644 --- a/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml +++ b/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml @@ -5,7 +5,8 @@ description: Claims carry an immutable source-asserted validity interval (from/u tags: [e2, D41, D32] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D41 and plan/designs/e2_e3_claims_relations_design.md §7, then inspect the claim schema and the temporal extraction under src/. + Read plan/designs/e2_e3_claims_relations_design.md §7 — the binding source — and decisions.md D41 for rationale and + refinement history, then inspect the claim schema and the temporal extraction under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e2_claims_immutable_append_only.yaml b/plan/implementation_evals/eval_checks/e2_claims_immutable_append_only.yaml index 34c597e..777fa12 100644 --- a/plan/implementation_evals/eval_checks/e2_claims_immutable_append_only.yaml +++ b/plan/implementation_evals/eval_checks/e2_claims_immutable_append_only.yaml @@ -5,7 +5,8 @@ description: Claims are immutable, append-only evidence — no update path, no s tags: [e2, D2, D3, D54] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D2/D3 (and D54's "currency is bookkeeping, never validity") and plan/designs/e2_e3_claims_relations_design.md, then inspect + Read plan/designs/e2_e3_claims_relations_design.md — the binding source — and decisions.md D2/D3 (and D54's "currency is bookkeeping, never validity") for rationale and + refinement history, then inspect the claims schema/models and every code path that touches claim rows under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml b/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml index 50d4216..5aa2d08 100644 --- a/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml +++ b/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml @@ -5,7 +5,8 @@ description: E2 extraction runs over a context bundle (never a bare chunk) as tw tags: [e2, D31, D19] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D31/D19 and plan/designs/e2_e3_claims_relations_design.md §3, then locate + Read plan/designs/e2_e3_claims_relations_design.md §3 — the binding source — and decisions.md D31/D19 for rationale and + refinement history, then locate the extractor under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml b/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml index 1ade67d..33a53ac 100644 --- a/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml +++ b/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml @@ -5,7 +5,8 @@ description: Claim grounding is dual-field (standalone claim_text + verbatim sou tags: [e2, D32, D53] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D32 and plan/designs/e2_e3_claims_relations_design.md §3.3, then inspect + Read plan/designs/e2_e3_claims_relations_design.md §3.3 — the binding source — and decisions.md D32 for rationale and + refinement history, then inspect the claim schema and grounding validation under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e2_no_pre_extraction_gate.yaml b/plan/implementation_evals/eval_checks/e2_no_pre_extraction_gate.yaml index d28252f..8411a79 100644 --- a/plan/implementation_evals/eval_checks/e2_no_pre_extraction_gate.yaml +++ b/plan/implementation_evals/eval_checks/e2_no_pre_extraction_gate.yaml @@ -5,7 +5,8 @@ description: There is no pre-extraction value/salience gate — every chunked do tags: [e2, D25, D34] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D25/D34 and plan/designs/e2_e3_claims_relations_design.md §4, then inspect + Read plan/designs/e2_e3_claims_relations_design.md §4 — the binding source — and decisions.md D25/D34 for rationale and + refinement history, then inspect the pipeline between chunking and extraction under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e2_selection_ledger_replay.yaml b/plan/implementation_evals/eval_checks/e2_selection_ledger_replay.yaml index fbb3620..a8df0c2 100644 --- a/plan/implementation_evals/eval_checks/e2_selection_ledger_replay.yaml +++ b/plan/implementation_evals/eval_checks/e2_selection_ledger_replay.yaml @@ -5,8 +5,8 @@ description: Selection drops, kept_flagged outcomes, and decontextualization edi tags: [e2, D33, D7] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D33 (and D7's replay discipline) and plan/designs/postgres_schema_design.md - §8, then inspect the claim_extraction_decisions ledger and the rebuild paths under src/. + Read plan/designs/postgres_schema_design.md §8 — the binding source — and decisions.md D33 (and D7's replay discipline) for rationale and + refinement history, then inspect the claim_extraction_decisions ledger and the rebuild paths under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e2_selection_recall_envelope.yaml b/plan/implementation_evals/eval_checks/e2_selection_recall_envelope.yaml index 9196b3f..cd019e1 100644 --- a/plan/implementation_evals/eval_checks/e2_selection_recall_envelope.yaml +++ b/plan/implementation_evals/eval_checks/e2_selection_recall_envelope.yaml @@ -5,7 +5,8 @@ description: Selection is recall-conservative — KEEP bias, never-drop classes, tags: [e2, D35, D22] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D35 and plan/designs/e2_e3_claims_relations_design.md §3.4, then inspect the Selection prompt/rubric, outcome enum, and eval + Read plan/designs/e2_e3_claims_relations_design.md §3.4 — the binding source — and decisions.md D35 for rationale and + refinement history, then inspect the Selection prompt/rubric, outcome enum, and eval harness under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e2_stance_kept_as_observation.yaml b/plan/implementation_evals/eval_checks/e2_stance_kept_as_observation.yaml index e6f2911..0001642 100644 --- a/plan/implementation_evals/eval_checks/e2_stance_kept_as_observation.yaml +++ b/plan/implementation_evals/eval_checks/e2_stance_kept_as_observation.yaml @@ -5,7 +5,8 @@ description: Attributed stance is kept and normalized to an observation on the h tags: [e2, e3, D59, D43] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D59 and plan/designs/observations_design.md, then inspect the Selection rubric and E3 normalization under src/. + Read plan/designs/observations_design.md — the binding source — and decisions.md D59 for rationale and + refinement history, then inspect the Selection rubric and E3 normalization under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e2_testimony_currency_counting.yaml b/plan/implementation_evals/eval_checks/e2_testimony_currency_counting.yaml index 71c6b36..3a14fe2 100644 --- a/plan/implementation_evals/eval_checks/e2_testimony_currency_counting.yaml +++ b/plan/implementation_evals/eval_checks/e2_testimony_currency_counting.yaml @@ -5,7 +5,8 @@ description: evidence_count means distinct current-testimony lineages per stance tags: [e2, e3, D54] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D54 and plan/designs/evidence_lifecycle_design.md, then inspect the + Read plan/designs/evidence_lifecycle_design.md — the binding source — and decisions.md D54 for rationale and + refinement history, then inspect the currency ledger, the count maintenance, and the zero-support paths under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e3_blocking_cheap_first_cascade.yaml b/plan/implementation_evals/eval_checks/e3_blocking_cheap_first_cascade.yaml index 3574b80..c2b74c8 100644 --- a/plan/implementation_evals/eval_checks/e3_blocking_cheap_first_cascade.yaml +++ b/plan/implementation_evals/eval_checks/e3_blocking_cheap_first_cascade.yaml @@ -5,7 +5,8 @@ description: Supersession candidates come from (entity, predicate) blocking with tags: [e3, D4, D43] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D4 and plan/designs/e2_e3_claims_relations_design.md §5, then inspect the supersession worker under src/. These are acceptance + Read plan/designs/e2_e3_claims_relations_design.md §5 — the binding source — and decisions.md D4 for rationale and + refinement history, then inspect the supersession worker under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: diff --git a/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml b/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml index d08f945..27dec71 100644 --- a/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml +++ b/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml @@ -5,7 +5,8 @@ description: Observations are untyped, entity-anchored, bi-temporal facts — no tags: [e3, D43] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D43 and plan/designs/observations_design.md, then inspect the observations + Read plan/designs/observations_design.md — the binding source — and decisions.md D43 for rationale and + refinement history, then inspect the observations schema and the add-observation worker under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml b/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml index 926b630..7977cdc 100644 --- a/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml +++ b/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml @@ -5,7 +5,8 @@ description: The predicate vocabulary is a governed registry with an other:LadybugDB boundary is the v_graph_* views with merge- tags: [p2, D44] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D44 and plan/designs/postgres_schema_design.md §10.A, then inspect the + Read plan/designs/postgres_schema_design.md §10.A — the binding source — and decisions.md D44 for rationale and + refinement history, then inspect the projection views and the graph build under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml b/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml index cca8f39..f9089cd 100644 --- a/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml +++ b/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml @@ -5,7 +5,8 @@ description: P2 is rebuild-first — full rebuild, validate, publish an immutabl tags: [p2, D7, D40] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D7 and plan/designs/p2_graph_design.md, then inspect the P2 worker and + Read plan/designs/p2_graph_design.md — the binding source — and decisions.md D7 for rationale and + refinement history, then inspect the P2 worker and reader under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/p3_corpus_fs_projection.yaml b/plan/implementation_evals/eval_checks/p3_corpus_fs_projection.yaml index e3b9f94..61ec988 100644 --- a/plan/implementation_evals/eval_checks/p3_corpus_fs_projection.yaml +++ b/plan/implementation_evals/eval_checks/p3_corpus_fs_projection.yaml @@ -5,8 +5,8 @@ description: P3 is a rebuildable corpus-filesystem projection — generated stub tags: [p3, D40, D55] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D40 (and D55's stability contract) and plan/designs/e0_files_design.md §6 - plus plan/analysis/p3_agent_navigation.md, then inspect the P3 builder under src/. These are + Read plan/designs/e0_files_design.md §6 plus plan/analysis/p3_agent_navigation.md — the binding source — and decisions.md D40 (and D55's stability contract) for rationale and + refinement history, then inspect the P3 builder under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: diff --git a/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml b/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml index 4fcda83..5dcbac9 100644 --- a/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml +++ b/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml @@ -5,7 +5,8 @@ description: Validity state has exactly one home (Postgres); projection builders tags: [p, D6, D62] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D6 (and D62's enforced architecture) and plan/designs/overall_design.md §2, then inspect the projection builders + Read plan/designs/overall_design.md §2 — the binding source — and decisions.md D6 (and D62's enforced architecture) for rationale and + refinement history, then inspect the projection builders and the import-linter contracts under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/registry_scale_partitioning_d23.yaml b/plan/implementation_evals/eval_checks/registry_scale_partitioning_d23.yaml index 699783a..8757d32 100644 --- a/plan/implementation_evals/eval_checks/registry_scale_partitioning_d23.yaml +++ b/plan/implementation_evals/eval_checks/registry_scale_partitioning_d23.yaml @@ -5,7 +5,8 @@ description: The registry scale rules hold — big append-only tables partitione tags: [registries, D23, D17] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D23 and plan/designs/postgres_schema_design.md, then inspect the + Read plan/designs/postgres_schema_design.md — the binding source — and decisions.md D23 for rationale and + refinement history, then inspect the migrations under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/ret_envelope_contradiction_completeness.yaml b/plan/implementation_evals/eval_checks/ret_envelope_contradiction_completeness.yaml index 6267037..2725227 100644 --- a/plan/implementation_evals/eval_checks/ret_envelope_contradiction_completeness.yaml +++ b/plan/implementation_evals/eval_checks/ret_envelope_contradiction_completeness.yaml @@ -1,12 +1,14 @@ schema_version: 1 id: ret_envelope_contradiction_completeness type: harness_judge -description: Contradiction co-members are never silently absent — inline up to a cap and plan/designs/retrieval_design.md, then group_id + returned/total + continuation; one-sided answers are a contract violation. +description: Contradiction co-members are never silently absent — inline up to a cap, beyond it group_id + returned/total + continuation; one-sided answers are a contract violation. tags: [retrieval, D49] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D49, then inspect the envelope contradiction handling under src/. These - are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + Read plan/designs/retrieval_design.md — the binding source — and decisions.md D49 for + rationale and refinement history, then inspect the envelope contradiction handling under + src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. Verify ALL of the following: 1. When a returned fact belongs to a contradiction group, co-members ship inline up to a diff --git a/plan/implementation_evals/eval_checks/ret_envelope_freshness_stamps.yaml b/plan/implementation_evals/eval_checks/ret_envelope_freshness_stamps.yaml index 1256ae5..5002850 100644 --- a/plan/implementation_evals/eval_checks/ret_envelope_freshness_stamps.yaml +++ b/plan/implementation_evals/eval_checks/ret_envelope_freshness_stamps.yaml @@ -5,7 +5,8 @@ description: Per-source freshness stamps ship in every envelope — PG live, P1 tags: [retrieval, D49, D48] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D49 (freshness stamps) and D48 and plan/designs/retrieval_design.md, then inspect the envelope freshness + Read plan/designs/retrieval_design.md — the binding source — and decisions.md D49 (freshness stamps) and D48 for rationale and + refinement history, then inspect the envelope freshness section under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/ret_envelope_grain_discipline.yaml b/plan/implementation_evals/eval_checks/ret_envelope_grain_discipline.yaml index 10b6d80..4165a74 100644 --- a/plan/implementation_evals/eval_checks/ret_envelope_grain_discipline.yaml +++ b/plan/implementation_evals/eval_checks/ret_envelope_grain_discipline.yaml @@ -5,7 +5,8 @@ description: Every response declares its grain (fact/evidence/compiled/composite tags: [retrieval, D49, D41] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D49 and plan/designs/retrieval_design.md, then inspect the envelope types + Read plan/designs/retrieval_design.md — the binding source — and decisions.md D49 for rationale and + refinement history, then inspect the envelope types and grain enforcement under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/ret_envelope_negative_taxonomy.yaml b/plan/implementation_evals/eval_checks/ret_envelope_negative_taxonomy.yaml index 9de88e6..3297def 100644 --- a/plan/implementation_evals/eval_checks/ret_envelope_negative_taxonomy.yaml +++ b/plan/implementation_evals/eval_checks/ret_envelope_negative_taxonomy.yaml @@ -5,7 +5,8 @@ description: A typed negative taxonomy — unknown_entity / known_empty / bounda tags: [retrieval, D49, D50] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D49 (typed negatives) and D50 (trust model) and plan/designs/retrieval_design.md, then inspect the negative + Read plan/designs/retrieval_design.md — the binding source — and decisions.md D49 (typed negatives) and D50 (trust model) for rationale and + refinement history, then inspect the negative result types under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/ret_envelope_truncation_markers.yaml b/plan/implementation_evals/eval_checks/ret_envelope_truncation_markers.yaml index 799bfc4..0a6add3 100644 --- a/plan/implementation_evals/eval_checks/ret_envelope_truncation_markers.yaml +++ b/plan/implementation_evals/eval_checks/ret_envelope_truncation_markers.yaml @@ -5,7 +5,8 @@ description: No silent caps anywhere — truncation is always marked with a cont tags: [retrieval, D49] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D49 and plan/designs/retrieval_design.md, then inspect result assembly and pagination under src/. These are + Read plan/designs/retrieval_design.md — the binding source — and decisions.md D49 for rationale and + refinement history, then inspect result assembly and pagination under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: diff --git a/plan/implementation_evals/eval_checks/ret_filesystem_first_mounts_skill.yaml b/plan/implementation_evals/eval_checks/ret_filesystem_first_mounts_skill.yaml index f1cadaa..568a03f 100644 --- a/plan/implementation_evals/eval_checks/ret_filesystem_first_mounts_skill.yaml +++ b/plan/implementation_evals/eval_checks/ret_filesystem_first_mounts_skill.yaml @@ -5,7 +5,8 @@ description: Consumption is filesystem-first — four read-only mounts (raw off- tags: [retrieval, D51] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D51 and plan/designs/retrieval_design.md, then inspect the mount + Read plan/designs/retrieval_design.md — the binding source — and decisions.md D51 for rationale and + refinement history, then inspect the mount publication and the shipped skill under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/ret_hydration_reverifies.yaml b/plan/implementation_evals/eval_checks/ret_hydration_reverifies.yaml index b59d63e..698b0c2 100644 --- a/plan/implementation_evals/eval_checks/ret_hydration_reverifies.yaml +++ b/plan/implementation_evals/eval_checks/ret_hydration_reverifies.yaml @@ -5,7 +5,8 @@ description: Projections propose, the spine disposes — all query surfaces rout tags: [retrieval, D48] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D48 and plan/designs/retrieval_design.md, then inspect the query engine + Read plan/designs/retrieval_design.md — the binding source — and decisions.md D48 for rationale and + refinement history, then inspect the query engine under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/ret_primitives_recipes_registry.yaml b/plan/implementation_evals/eval_checks/ret_primitives_recipes_registry.yaml index e65ba7b..04d9c6c 100644 --- a/plan/implementation_evals/eval_checks/ret_primitives_recipes_registry.yaml +++ b/plan/implementation_evals/eval_checks/ret_primitives_recipes_registry.yaml @@ -5,7 +5,8 @@ description: The query machine is typed zero-LLM primitives + recipes as registr tags: [retrieval, D50, D9] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D50 and plan/designs/retrieval_design.md, then inspect the primitives and + Read plan/designs/retrieval_design.md — the binding source — and decisions.md D50 for rationale and + refinement history, then inspect the primitives and recipe registry under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/ret_trust_boundary_no_content_auth.yaml b/plan/implementation_evals/eval_checks/ret_trust_boundary_no_content_auth.yaml index 9f60007..71c94db 100644 --- a/plan/implementation_evals/eval_checks/ret_trust_boundary_no_content_auth.yaml +++ b/plan/implementation_evals/eval_checks/ret_trust_boundary_no_content_auth.yaml @@ -5,7 +5,8 @@ description: One trust domain per deployment — no content-level authorization tags: [retrieval, D50, D60, D61] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D50 (trust model) and D60/D61 and plan/designs/retrieval_design.md §9, then inspect the API/mount surfaces under + Read plan/designs/retrieval_design.md §9 — the binding source — and decisions.md D50 (trust model) and D60/D61 for rationale and + refinement history, then inspect the API/mount surfaces under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/scope_views_share_graph_d16.yaml b/plan/implementation_evals/eval_checks/scope_views_share_graph_d16.yaml index 4886055..bb24b5e 100644 --- a/plan/implementation_evals/eval_checks/scope_views_share_graph_d16.yaml +++ b/plan/implementation_evals/eval_checks/scope_views_share_graph_d16.yaml @@ -5,7 +5,8 @@ description: One graph, many lenses — scopes never get their own entity space tags: [cross, D16, D50] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D16 (and D50's trust-model refinement) and plan/designs/registries_design.md §1, then inspect scope handling under + Read plan/designs/registries_design.md §1 — the binding source — and decisions.md D16 (and D50's trust-model refinement) for rationale and + refinement history, then inspect scope handling under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/source_of_truth_split_d1_d46.yaml b/plan/implementation_evals/eval_checks/source_of_truth_split_d1_d46.yaml index df36348..063a514 100644 --- a/plan/implementation_evals/eval_checks/source_of_truth_split_d1_d46.yaml +++ b/plan/implementation_evals/eval_checks/source_of_truth_split_d1_d46.yaml @@ -5,7 +5,8 @@ description: The split source of truth holds — Postgres authoritative for the tags: [cross, D1, D46] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read decisions.md D1 and D46 and plan/designs/overall_design.md §2, then inspect the store boundaries under src/. These are + Read plan/designs/overall_design.md §2 — the binding source — and decisions.md D1 and D46 for rationale and + refinement history, then inspect the store boundaries under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: From 0179606474ce562ac2f1e79587388993f896b7be Mon Sep 17 00:00:00 2001 From: Jiri Puc Date: Sun, 12 Jul 2026 11:41:50 +0200 Subject: [PATCH 7/7] =?UTF-8?q?Evals:=20apply=20Codex=20round-2=20review?= =?UTF-8?q?=20(gpt-5.6-sol)=20=E2=80=94=2014=20must-fixes,=20+7=20checks?= =?UTF-8?q?=20(76=20total)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The round-2 review (codex_review_2.md, committed alongside) judged the suite against the design docs as ground truth. All load-bearing quotes were verified against the designs before applying. Applied in full: - Surgery repairs: six mis-pathed requirements citations fixed (plan/requirements/ prefix) + binding-source phrasing added to the unflipped Read sentences; four non-design "binding source" designations rebound (CLAUDE.md, concepts.md, workers.md, the P3 analysis are now context, not binding). - Over-binding corrections (checks could fail conforming code): provider calls through D61 ports are conformant; direct-env access allowed via the requirement's reasoned per-line exception; ingest bumps legitimately cascade downstream; effective_period added to the claim_valid_kind enum; borderline per-claim grounding escalation is designed; observations use the shared adjudication_outcome enum; re-occurring relations in non-overlapping windows get new rows; reviewer may be a designated agent; writers run parallel across disjoint pages ("editing shared files" restored); model page binds writer invocations, not every page; version-vs-lineage tombstone branches spelled out; execution classes need an inspectable mapping (no invented enum); retry bound is a tunable per-stage default; producer/checker never-share per orchestration §8 (D53's recorded-exception prose loses under precedence); projection purity judged as outcome, not an invented import graph. - D64 made scoreable (16-predicate seed incl. uses/reports_to signatures); D23 expanded to the full partitioned estate + the ungated load-test hook (a round-1 application miss). - +7 coverage checks: K trigger surface/acyclicity, K eval canaries + citation faithfulness, K writer runtime two-band, K hard-forget git history, orchestration topology, S58 skill acceptance, continuous ER health. - Tags reconciled with scored conditions; README recounted (76), column relabeled "primary binding sources", both reviews recorded. Co-Authored-By: Claude Fable 5 --- plan/implementation_evals/README.md | 75 ++-- plan/implementation_evals/codex_review_2.md | 366 ++++++++++++++++++ .../boundary_correctness_in_library_d60.yaml | 26 +- ...undary_extension_points_no_bypass_d60.yaml | 2 +- .../boundary_no_human_control_plane_d60.yaml | 7 +- .../boundary_provider_ports_d61.yaml | 3 +- .../code_config_via_pydantic_settings.yaml | 16 +- .../code_tooling_and_migrations.yaml | 12 +- .../e0_converter_router_versioned.yaml | 2 +- .../e0_postgres_holds_no_bodies.yaml | 2 +- .../e0_subworker_chain_crossrefs_d36.yaml | 12 +- .../e1_chunks_whole_blocks_no_overlap.yaml | 2 +- .../e2_asserted_validity_immutable.yaml | 26 +- .../e2_context_bundle_two_calls.yaml | 2 +- .../e2_grounding_layered_dual_field.yaml | 18 +- .../e2_no_pre_extraction_gate.yaml | 2 +- .../e2_selection_recall_envelope.yaml | 2 +- .../e3_observations_untyped_adjudicated.yaml | 21 +- .../e3_predicate_registry_governed.yaml | 19 +- .../e3_relations_evidence_collapse.yaml | 25 +- ...e3_supersession_fact_level_bitemporal.yaml | 4 +- .../embedding_model_port_config_d63.yaml | 2 +- .../er_continuous_health_metrics.yaml | 30 ++ .../eval_checks/er_review_queue_cli.yaml | 18 +- .../k_eval_canaries_faithfulness.yaml | 31 ++ .../k_hard_forget_git_history.yaml | 32 ++ .../eval_checks/k_one_mechanism_n_scopes.yaml | 13 +- .../eval_checks/k_planner_writer_driver.yaml | 26 +- .../k_trigger_surface_acyclic.yaml | 32 ++ .../k_writer_runtime_two_band.yaml | 32 ++ .../ops_deletion_cascade_grains.yaml | 27 +- .../ops_execution_classes_bound.yaml | 22 +- .../eval_checks/ops_idempotency_dlq.yaml | 16 +- .../ops_metering_budgets_enforced.yaml | 4 +- .../ops_orchestration_topology.yaml | 32 ++ .../ops_producer_checker_families.yaml | 18 +- .../ops_versioned_replay_from_storage.yaml | 4 +- .../p1_lance_estate_rebuildable.yaml | 7 +- .../p2_rebuild_first_snapshots.yaml | 3 +- .../eval_checks/p3_corpus_fs_projection.yaml | 8 +- .../p_projections_hold_no_authority.yaml | 26 +- .../registry_scale_partitioning_d23.yaml | 27 +- .../ret_claims_never_answer_now.yaml | 7 +- .../ret_envelope_grain_discipline.yaml | 2 +- .../ret_envelope_negative_taxonomy.yaml | 2 +- .../eval_checks/ret_skill_s58_acceptance.yaml | 30 ++ .../scope_views_share_graph_d16.yaml | 4 +- 47 files changed, 877 insertions(+), 222 deletions(-) create mode 100644 plan/implementation_evals/codex_review_2.md create mode 100644 plan/implementation_evals/eval_checks/er_continuous_health_metrics.yaml create mode 100644 plan/implementation_evals/eval_checks/k_eval_canaries_faithfulness.yaml create mode 100644 plan/implementation_evals/eval_checks/k_hard_forget_git_history.yaml create mode 100644 plan/implementation_evals/eval_checks/k_trigger_surface_acyclic.yaml create mode 100644 plan/implementation_evals/eval_checks/k_writer_runtime_two_band.yaml create mode 100644 plan/implementation_evals/eval_checks/ops_orchestration_topology.yaml create mode 100644 plan/implementation_evals/eval_checks/ret_skill_s58_acceptance.yaml diff --git a/plan/implementation_evals/README.md b/plan/implementation_evals/README.md index 0223f9d..35f3a3a 100644 --- a/plan/implementation_evals/README.md +++ b/plan/implementation_evals/README.md @@ -1,16 +1,17 @@ # Implementation evals — design-conformance acceptance checks An exhaustive set of [eval-banana](https://github.com/writeitai/eval-banana) `harness_judge` -checks that verify the **final implementation honors the binding design** — the decision log -(**D1–D64**), the requirements, and the design docs. Each check points an LLM judge at the -binding sources and at the code, states the invariant as concrete conditions, and demands a -binary verdict. +checks that verify the **final implementation honors the binding design** — the design docs +under `plan/designs/`, the requirements, and the decision log (**D1–D64**) as the coverage +index. Each check points an LLM judge at the binding sources and at the code, states the +invariant as concrete conditions, and demands a binary verdict. **What these are:** acceptance checks for design conformance — "does the code do what the design binds it to do?" **What these are not:** quality metrics. Extraction precision, resolution P/R curves, retrieval recall@k live in the D22 golden-set eval harness (a separate, -data-driven asset — itself covered here by `eval_golden_sets_d22`); these checks verify the -*machinery and its invariants exist and hold*. +data-driven asset — itself covered here by `eval_golden_sets_d22` and +`er_continuous_health_metrics`); these checks verify the *machinery and its invariants exist +and hold*. ## Running @@ -34,11 +35,12 @@ Two conventions when running: - Every check is `type: harness_judge`, one YAML file per check, self-contained (the judge sees only `description` + `instructions` — never this README), and cites its binding sources - (decision numbers + design docs) so the judge reads the ground truth first. + so the judge reads the ground truth first. - Conditions are conjunctive: score 1 only if **every** listed condition demonstrably holds, with file paths cited in the reason. -- `tags` carry the area and the decision numbers, for filtered runs - (e.g. everything tagged `D43`, or all of `e2`). +- `tags` carry the area and the **primary decision numbers** a check's scored conditions + invoke, for filtered runs (e.g. everything tagged `D43`, or all of `e2`). Tags are kept + consistent with the conditions, but incidental cross-references are not exhaustively tagged. - Where an invariant is a *deliberate exception* (e.g. E2 extraction is volume-proportional by D25 while adjudication is ambiguity-scaled by D4), the check says so explicitly, so a judge cannot fail a correct implementation for obeying the design. @@ -56,37 +58,52 @@ Two conventions when running: the repo's own claims-vs-facts epistemology applied to its documentation: the log is testimony; the designs are the adjudicated current belief.) -## Inventory (69 checks) +## Inventory (76 checks) -| Area | Checks | Binding sources covered | +Counts by area; the sources column names the **primary** binding decisions (not every +cross-reference). + +| Area | Checks | Primary binding sources | |---|---|---| | E0 documents/files (`e0_*`) | 10 | D36–D39, D42, D51, D54–D57 | | E1 blocks/chunks (`e1_*`) | 2 | D57, D58 | -| E2 claims/extraction (`e2_*`) | 9 | D2, D3, D19, D25, D31–D35, D41, D54, D59, D63 | -| E3 relations/observations (`e3_*`) | 5 | D2–D5, D15, D18, D43, D55 | -| Registries/ER + eval (`er_*`, `registry_*`, `eval_*`) | 5 | D17, D20–D24 | -| K plane (`k_*`) | 4 | D45–D47, D54 | +| E2 claims/extraction (`e2_*`) | 9 | D2, D3, D19, D25, D31–D35, D41, D54, D59 | +| E3 relations/observations (`e3_*`) | 5 | D2–D5, D15, D18, D43, D55, D64 | +| Registries/ER + eval (`er_*`, `registry_*`, `eval_*`) | 6 | D17, D20–D24 | +| K plane (`k_*`) | 8 | D45–D47, D22, D52, D54, D55 | | Projections (`p_*`, `p1_*`, `p2_*`, `p3_*`, `embedding_*`) | 8 | D6–D11, D40, D44, D55, D61, D63 | -| Retrieval (`ret_*`) | 10 | D9, D41, D48–D51 | -| Ops/cross-cutting (`ops_*`, `source_of_truth_*`, `scope_views_*`) | 8 | D1, D7, D12, D16, D33, D46, D52–D56, D62 | +| Retrieval (`ret_*`) | 11 | D9, D41, D43, D48–D51 | +| Ops/cross-cutting (`ops_*`, `source_of_truth_*`, `scope_views_*`) | 9 | D1, D7, D12, D16, D33, D46, D52–D56, D62 | | Code & boundary (`code_*`, `boundary_*`, `delivery_*`) | 8 | D60–D62, requirements §Code | Not every decision needs its own check: withdrawn decisions (D26–D30), naming/plane decisions (D13, D14), and pure-analysis decisions are covered implicitly or are not implementation- observable. Where one check verifies several decisions, the tags say so. -## Review - -`codex_review.md` records an independent Codex (gpt-5.5) review of the initial 53-check set. -**The current set incorporates that review**: its four factual must-fixes (deletion retention -semantics, the E2 volume-proportional exception, ledger scope, the observations no-cap rule), -the judgeability rewrites and splits (the boundary and envelope omnibus checks, lineage/living -split), the mechanics fixes (tags, cite paths), and nine added checks closing its coverage gaps -(D1/D46, D16, D22, D23, D36, code tooling, D62 ×2, D63). +## Reviews + +Two independent Codex reviews, both preserved and both applied: + +- **`codex_review.md`** (round 1, gpt-5.5) reviewed the initial 53-check set: four factual + must-fixes (deletion retention, the E2 volume-proportional exception, ledger scope, the + no-cap rule), omnibus splits, nine added checks, tag/cite mechanics. +- **`codex_review_2.md`** (round 2, gpt-5.6-sol, design-docs-as-ground-truth) reviewed the + 69-check set after the design-first flip: repaired the mis-pathed/unflipped Read sentences + and the non-design "binding source" designations; corrected checks that over-bound decision + prose or round-1 suggestions beyond what the designs bind (provider calls through ports, + writer parallelism across disjoint pages, the borderline per-claim grounding escalation, + reviewer agents, the shared `adjudication_outcome` enum, non-overlapping-window relation + rows, tunable retry bounds, the producer/checker never-share rule, outcome-based projection + purity); made D64 scoreable (the 16-predicate seed incl. `uses`/`reports_to`); expanded D23 + to the full partitioned estate + the ungated load-test hook; and added seven checks closing + its coverage gaps (K trigger surface/eval/writer-runtime/hard-forget-history, orchestration + topology, the S58 skill acceptance test, continuous ER health). ## Inconsistencies found while authoring -Authoring these against the full corpus surfaced inconsistencies, registered in -`questions.md` §5 (the repo's fix register): the requirements' temporal-split paragraph -predates D43/D49 (observations also carry adjudicated validity), and the requirements' E3 -bullet omits observations entirely. See `questions.md` items 30–31. +Authoring these against the full corpus surfaced inconsistencies, registered (and since fixed) +via `questions.md` §5 items 30–31 → PR #43. Round 2 additionally surfaced two design-corpus +discrepancies the checks now handle via the precedence rule: the retry bound (overall/ +requirements say "max 2", the schema binds it as a tunable per-stage default) and the +producer/checker exception (D53's prose allows a recorded exception; `orchestration_design.md` +§8 binds never-share — the design controls). diff --git a/plan/implementation_evals/codex_review_2.md b/plan/implementation_evals/codex_review_2.md new file mode 100644 index 0000000..8301ae5 --- /dev/null +++ b/plan/implementation_evals/codex_review_2.md @@ -0,0 +1,366 @@ +# Round 2 review of `plan/implementation_evals` + +**Verdict:** not yet safe as a binary design-conformance acceptance gate. The suite is structurally healthy (69 valid YAML files, unique filename-matching IDs, one uniform precedence paragraph, and accurate area counts), and all four Round 1 factual corrections are now present. However, the design-first flip is incomplete: six `Read` instructions name a nonexistent requirements path, five never identify a binding source, and four designate non-design material as binding. More importantly, several checks can reject a conforming implementation or accept a non-conforming one: the K concurrency description contradicts the binding K design; the D64-tagged check never requires the 16-predicate seed; D23 coverage is incomplete; several conditions follow decision-log wording or reviewer suggestions beyond what the designs bind; and some conditions require the wrong enum or architecture shape. + +## MUST-FIX + +1. **Repair the ten defective `Read` instructions identified under Surgery scars.** Six contain the nonexistent path `requirements_v3.md`; five of those also lack the promised “binding source” wording. Four instructions call `CLAUDE.md` or `plan/analysis/*` material binding even though the suite's own precedence rule says `plan/designs/*` controls. +2. **Fix `boundary_correctness_in_library_d60`.** “Not calling out to a proprietary cloud service to do the work” is broad enough to prohibit the configured model/OCR provider calls that D61 and `packaging_distribution_design.md` §§4–5 explicitly allow through ports. The invariant is that correctness machinery cannot be cloud-product-only or commercially gated, not that it cannot call a configured provider. +3. **Fix the two requirements checks that over-bind code policy.** `code_config_via_pydantic_settings` forbids every `os.environ`/`os.getenv` occurrence even though `requirements_v3.md` §Code explicitly permits a reasoned per-line exception. `code_tooling_and_migrations` requires CI-on-every-PR and a “relax-with-receipts” policy that §Code does not bind. +4. **Fix `e0_subworker_chain_crossrefs_d36` condition 2.** “Only that sub-worker and its downstream dependents — never the whole chain” is self-contradictory for an `ingest` version bump, whose downstream dependents are the rest of the chain. `e0_files_design.md` §1 binds per-sub-worker idempotency and dependency-scoped reruns, not an absolute no-whole-chain result. +5. **Fix `e2_asserted_validity_immutable`.** Condition 1 omits the binding `effective_period` member of `claim_valid_kind` (`postgres_schema_design.md` §§1 and 8). Condition 4 requires a “monotonicity guard” found in decision prose but not in any binding design section; under the suite's stated precedence it cannot be a scored condition unless the design is amended first. +6. **Fix `e2_grounding_layered_dual_field` condition 2.** It says the independent audit is “never per-claim,” while `e2_e3_claims_relations_design.md` §3.3 explicitly permits per-claim judging for the borderline band. Require sampling as the default and allow the designed borderline escalation. +7. **Fix the E3 checks.** `e3_observations_untyped_adjudicated` demands an enum shaped as `evidence / supersede / contradict-coexist / new`, but the binding shared `adjudication_outcome` enum is `add | noop | supersede | contradict | same_as_merge_proposal | retracted_source_removal` (`postgres_schema_design.md` §§1, 9.A). `e3_relations_evidence_collapse` says every repeat of “the same fact” must attach to an existing row, omitting the compatible-validity qualification: §9 deliberately permits a new row for a recurring `(subject, predicate, object)` fact in a non-overlapping validity window. +8. **Fix `er_review_queue_cli` for the current reviewer contract.** Its description and condition 1 say the middle band routes to humans. `k_layers_design.md` §7 permits a human **or designated reviewer agent** and says the latter is normal in agent-operated deployments. The check should require the Postgres queue plus CLI/agent-consumer boundary and the same reversible verdict contract, not a human-only runtime. +9. **Fix the K checks.** `k_planner_writer_driver` says “no concurrent sessions,” contradicting `k_layers_design.md` §6, which explicitly runs writers in parallel across disjoint pages; the forbidden case is concurrent sessions editing shared files. `k_one_mechanism_n_scopes` condition 2 says the shared model page is a compile input of “every page,” but §7 binds it as an input of every **writer**; authored pages have no compile input and fact-sheet-only pages skip the writer. +10. **Fix the ops checks.** `ops_execution_classes_bound` falsely says schema pipeline enums carry an execution-class value for every worker; the binding schema has stage/component enums, not an execution-class enum or complete worker inventory. `ops_idempotency_dlq` hard-codes “max 2” despite `postgres_schema_design.md` §2 calling two a tunable starting value. `ops_producer_checker_families` permits a recorded same-family exception, while the controlling `orchestration_design.md` §8 says checker seats “never share a family” with their producer; the check currently follows the looser decision-log prose instead of the design. +11. **Fix `p_projections_hold_no_authority` condition 1.** The designs bind dumb deterministic projection writers and one validity home (`overall_design.md` §2; `p2_graph_design.md` §1), but D62's package-level import contracts do not bind the invented finer rule that every projection builder may import only read repositories/projection writers or that CI must encode those particular submodule arrows. Score the architectural outcome, or first bind that exact import graph in `packaging_distribution_design.md` §4. +12. **Make D23 coverage match the binding schema and restore the omitted Round 1 load-test condition.** `registry_scale_partitioning_d23` names only three large tables, missing the other partitioned DDL in schema §§7–9.A, and it omits the representative ungated-volume load-test hook explicitly called for by `registries_design.md` §§9 and 11 and by the Round 1 review. +13. **Make D64 scoreable.** `e3_predicate_registry_governed` cites/tags D64 but never requires exactly the current 16-predicate seed or names `uses` and `reports_to`. A 14-predicate pre-D64 implementation can score 1. Bind the exact seed from `registries_design.md` §4 (or at minimum assert 16 plus those two promoted rows and their signatures). +14. **Close the remaining binding coverage gaps:** K watches/subscriptions/dispatch and trigger acyclicity (`k_layers_design.md` §5), K eval acceptance machinery (§7), the S58 cold-consumer skill test (`retrieval_design.md` §§8 and 11), hard-forget erasure of cited K git history/backups (`k_layers_design.md` §10), and the orchestration lane/queue/backfill and cross-cloud batching rules (`orchestration_design.md` §§2–5). + +## 1. Surgery scars + +### 1.1 Description proofreading + +No description contains an obvious regex splice, duplicated path fragment, truncated clause, or garbled `, then` surgery artifact. The descriptions parse as complete sentences. + +One description is grammatically intact but factually damaged in effect: + +- `k_planner_writer_driver.yaml`: + + > `Plane K is compiled by planner/writer/driver — mechanical zero-LLM routing, dependency-ordered scheduling, one automated committer, no concurrent sessions.` + + The last phrase drops the binding qualifier “editing shared files.” `k_layers_design.md` §6 says writers run in parallel across disjoint pages; `requirements_v3.md` §Fixed choices says “no concurrent sessions editing shared files.” + +### 1.2 Broken or unflipped `Read` paths + +The following six instructions cite `requirements_v3.md` at repository root. That file does not exist; the actual path is `plan/requirements/requirements_v3.md`. + +- `boundary_provider_ports_d61.yaml`: + + > `Read requirements_v3.md §Fixed choices & the reference deployment — the binding source — and decisions.md D61 for rationale and refinement history, then inspect the adapter layer and self-host stack under src/ and the deploy profiles.` + +- `code_config_via_pydantic_settings.yaml`: + + > `Read requirements_v3.md §Code, then inspect the settings module(s) and grep the source for environment access under src/.` + +- `code_tooling_and_migrations.yaml`: + + > `Read requirements_v3.md §Code, then inspect the tooling configuration, CI workflows, and migrations under the repo.` + +- `ops_metering_budgets_enforced.yaml`: + + > `Read requirements_v3.md §Operational properties (cost discipline) and decisions.md D52, then inspect the cost ledgers and budget checks under src/.` + +- `ops_versioned_replay_from_storage.yaml`: + + > `Read requirements_v3.md §Operational properties (versioned processing) and decisions.md D7/D33, then inspect version stamping and rebuild paths under src/.` + +- `ret_claims_never_answer_now.yaml`: + + > `Read requirements_v3.md §Retrieval (the temporal-split paragraph), decisions.md D41/D49, then inspect recipe definitions and naming under src/.` + +The latter five also do not say which source is binding and do not use the advertised design/requirements-first + decision-rationale form. The two code checks have no decision citation, which is fine, but should still say that the correctly pathed requirements section is the binding source. The other three should identify the requirements/design source as binding and the decision entries as rationale/history. + +### 1.3 Wrong material designated as “the binding source” + +These are not grammar corruption, but they are source-precedence corruption and conflict with the common paragraph in the same file: + +- `boundary_no_human_control_plane_d60.yaml`: + + > `Read CLAUDE.md Rule 3 — the binding source — and decisions.md D60 for rationale and refinement history, then survey the repository's shipped surfaces.` + + `CLAUDE.md` is repository guidance, not a binding design. Use `packaging_distribution_design.md` §§1–2 and 5 plus the relevant requirements boundary. + +- `e3_relations_evidence_collapse.yaml`: + + > `Read plan/analysis/concepts.md — the binding source — and decisions.md D2 for rationale and refinement history, then inspect the relations and relation_evidence schema and the E3 normalization under src/.` + + `plan/analysis/concepts.md` is analysis. The binding sources are `postgres_schema_design.md` §9 and `e2_e3_claims_relations_design.md` §5. + +- `ops_execution_classes_bound.yaml`: + + > `Read plan/analysis/workers.md + plan/designs/orchestration_design.md — the binding source — and decisions.md D52 for rationale and refinement history, then inspect the worker implementations under src/.` + + `workers.md` is the inventory referenced by the design, but it is not co-equal binding ground truth. Lead with `orchestration_design.md` §8 as binding; read the inventory second as implementation mapping. + +- `p3_corpus_fs_projection.yaml`: + + > `Read plan/designs/e0_files_design.md §6 plus plan/analysis/p3_agent_navigation.md — the binding source — and decisions.md D40 (and D55's stability contract) for rationale and refinement history, then inspect the P3 builder under src/.` + + The binding source is `e0_files_design.md` §6. The navigation analysis may be supporting context, not part of the phrase “the binding source.” + +### 1.4 Uniform precedence paragraph + +All 69 checks contain the source-precedence paragraph, and all 69 copies are byte-identical after YAML parsing. That mechanic is sound. Its protection is weakened, however, when the preceding `Read` sentence itself calls a non-design file binding; the judge receives two incompatible source instructions in one prompt. + +## 2. Factual accuracy against the designs + +### 2.1 Round 1 must-fixes: correctly present + +All four Round 1 factual repairs are materially present: + +1. **Deletion retention:** `ops_deletion_cascade_grains` condition 2 now says normal deletion retains chunks, claims, evidence links, relations, and entities as audit history, matching `postgres_schema_design.md` §13.1 and `evidence_lifecycle_design.md` §8. Minor wording still needs correction: “version or lineage ... tombstones the lineage/version rows” can imply that deleting one version tombstones the lineage; §13.1 says a version delete tombstones only that version and the lineage continues. +2. **E2 volume exception:** `ops_execution_classes_bound` condition 4 explicitly identifies E2 extraction as the deliberate volume-proportional fixed two-call exception, matching `orchestration_design.md` §3 and `e2_e3_claims_relations_design.md` §§1, 3, and 4. +3. **Selection ledger scope:** `e2_selection_ledger_replay` condition 1 requires drops, `kept_flagged`, and decontextualization edits while excluding plain keeps, matching `postgres_schema_design.md` §8 exactly. +4. **D43 no-cap:** `e3_supersession_fact_level_bitemporal`, `e3_observations_untyped_adjudicated`, `e0_living_mode_retraction`, and the deletion/count checks all preserve the measurement/fixed-period no-cap rule, matching `observations_design.md` §3 and schema §9.A. + +### 2.2 Conditions that misstate or overstate the binding design + +#### `boundary_correctness_in_library_d60.yaml` + +The instruction defines functional as: + +> `not calling out to a proprietary cloud service to do the work` + +That is not the D60 boundary. `packaging_distribution_design.md` §§4–5 and `requirements_v3.md` §Fixed choices explicitly support model/embedding provider adapters; E0 also permits routed OCR providers. D60 forbids making correctness conditional on the separate commercial **memory cloud**, not provider inference behind a declared port. Narrow the condition to “not implemented only by, or gated on, an out-of-repo commercial control plane,” while allowing configured provider calls. + +#### `code_config_via_pydantic_settings.yaml` + +Condition 1 requires: + +> `no os.environ / os.getenv anywhere in src/` + +`requirements_v3.md` §Code says direct access is banned **but explicitly allows a per-line ignore with a reason**. The check must accept that documented exception mechanism or the requirement must be tightened first. The rest of the check (typed settings, `SecretStr`/`SecretBytes`, point-of-use unwrap) matches §Code. + +#### `code_tooling_and_migrations.yaml` + +Condition 1 adds two requirements absent from its binding source: + +> `enforced in CI on every PR` + +and + +> `per pyproject's relax-with-receipts rule, any disabled check carries a written reason` + +`requirements_v3.md` §Code binds Pyright, pytest, Alembic, typing, and Ruff's TID251 enforcement indirectly; it does not bind an every-PR CI trigger or the named waiver policy. These may be good repository policies, but a harness judge cannot score them as design conformance from the cited source. + +#### `e0_subworker_chain_crossrefs_d36.yaml` + +Condition 2 ends: + +> `never the whole chain` + +An ingest-component bump can legitimately invalidate every downstream E0 component. `e0_files_design.md` §1 binds “that sub-worker and downstream dependents,” which is the useful invariant. Replace the absolute tail with “never unaffected upstream or sibling work merely because one downstream component changed.” + +#### `e2_asserted_validity_immutable.yaml` + +- Condition 1's parenthetical lists `proposition-validity vs event-time vs measurement-period` but omits `effective_period`, which is in the binding enum in `postgres_schema_design.md` §§1 and 8. +- Condition 4 requires `a monotonicity guard (a late retrospective cannot move an adjudicated window)`. That wording exists in D41, but no binding design section specifies the guard. The design says claim validity is immutable evidence and fact validity remains on relations; the scoreable condition should stop there unless the guard is promoted into a design. + +#### `e2_grounding_layered_dual_field.yaml` + +Condition 2 says the independent audit is: + +> `never per-claim` + +`e2_e3_claims_relations_design.md` §3.3 says the normal audit is sampled/offline, **and** “only a borderline band ever escalates to a per-claim judge.” A correct borderline escalation would currently fail. + +#### `e3_observations_untyped_adjudicated.yaml` + +Condition 3 requires: + +> `a typed enum (evidence / supersede / contradict-coexist / new)` + +The binding DDL uses the shared `adjudication_outcome` enum: `add`, `noop`, `supersede`, `contradict`, `same_as_merge_proposal`, and `retracted_source_removal` (`postgres_schema_design.md` §§1 and 9.A). The prose concepts map roughly (`noop` is evidence collapse; `add` is new), but a binary judge may require the literal four-value shape and reject the correct schema. Refer to the binding enum and separately test the observation worker's allowed semantic paths. + +#### `e3_relations_evidence_collapse.yaml` + +Condition 2 says: + +> `the same fact asserted again links evidence to the EXISTING relation rather than inserting a duplicate relation row` + +This needs “with compatible/overlapping validity.” `postgres_schema_design.md` §9 defines relation identity as `(subject, predicate, object) + validity interval` and deliberately permits recurring facts in non-overlapping windows. The current condition can reject the correct new-row behavior for “Alice worked at Acme, left, then later rejoined.” Conditions 1 and 3 otherwise correctly enforce the M:N evidence layer and evidence-once key. + +#### `er_review_queue_cli.yaml` + +The description says “Human review,” and condition 1 routes the middle band “to humans.” `registries_design.md` §8 uses human language, but the later, binding `k_layers_design.md` §7 generalizes the accountable reviewer to a human **or designated reviewer agent**. The current check is not design-complete and its single-source Read instruction hides the refinement. + +#### `k_planner_writer_driver.yaml` + +The description's `no concurrent sessions` contradicts `k_layers_design.md` §6: + +> `Writers run in parallel across disjoint pages` + +What is prohibited is concurrent sessions editing shared files or committing independently. The driver remains the sole automated committer and dependency order still applies. + +Condition 1 also says “one writer per page per cycle” without limiting that to stale compiled prose pages. Authored pages are not writer targets, unchanged pages do not compile, and fact-sheet-only pages skip writers (`k_layers_design.md` §§4–6). + +#### `k_one_mechanism_n_scopes.yaml` + +Condition 2 requires the shared model page as a compile input of `every page in the scope`. `k_layers_design.md` §7 says it is a declared input of every **writer**. Authored pages have no compile input/`inputs_hash`, and fact-sheet-only pages skip the writer. Require it for every applicable compiled writer invocation. + +#### `ops_deletion_cascade_grains.yaml` + +The Round 1 retention correction is good, but this phrase is imprecise: + +> `Normal deletion (version or lineage grain) purges raw/artifact bytes and tombstones the lineage/version rows` + +Per `postgres_schema_design.md` §13.1, version deletion tombstones only that version; lineage deletion tombstones the lineage and every version. Spell out the two branches so a judge does not demand lineage tombstoning for version deletion. + +#### `ops_execution_classes_bound.yaml` + +Condition 1 appends: + +> `(the schema's pipeline enums carry a value for every worker)` + +No binding schema enum records execution class, and `pipeline_component`/`pipeline_stage` do not enumerate every worker in the inventory (for example deletion and hard-forget are not component values). `orchestration_design.md` §8 binds the classification, not this storage representation. Require a complete inspectable inventory/config mapping, or add an execution-class field to the schema design first. + +#### `ops_idempotency_dlq.yaml` + +Condition 3 says: + +> `Retries are bounded (per the design: max 2)` + +There is a design-corpus discrepancy: `overall_design.md` §4 and `requirements_v3.md` §Fixed choices say max 2, while the more specific binding schema §2 says `max_attempts = 2` is a **tunable per-stage starting point, not a committed constant**. The check should not make a correct configured non-2 value fail. Require bounded, persisted per-stage retry policy with shipped default 2, and report the source discrepancy under the standard precedence rule. + +#### `ops_producer_checker_families.yaml` + +Condition 2 permits same-family checking: + +> `unless an explicit recorded exception is present` + +That follows D53's decision prose, but the controlling `orchestration_design.md` §8 says checking seats “never share a family” with the producer. Under this suite's explicit precedence, a recorded exception cannot score 1. Remove the exception or reconcile the design first. + +#### `p_projections_hold_no_authority.yaml` + +Condition 1 invents a precise submodule/import contract: + +> `Projection builders (P1/P2/P3 writers) import only read-side repositories and projection writers` + +The designs bind the outcome: projections never decide or mutate truth, P2's writer is dumb/deterministic, and package-level dependency arrows are CI-enforced. `packaging_distribution_design.md` §4 does not bind read/write repository subpackages or forbid projection workers from importing all core modules via the stated import-linter graph. This can fail a conforming architecture for using a different internal module split. + +#### `registry_scale_partitioning_d23.yaml` + +Condition 1 narrows “the large append-only tables” to: + +> `(mentions, resolution_decisions, relation_evidence — per the current schema design)` + +That parenthetical is false for the current schema. Binding partitioned DDL also exists for `claims`, `claim_extraction_decisions`, `chunks`, `chunk_claims`, `testimony_currency_events`, and `observation_evidence` (`postgres_schema_design.md` §§7–9.A and 12). The check also omits `registries_design.md` §9/§11's required load-test at ungated D25 volume. This is both factual incompleteness and a Round 1 application miss. + +#### `e3_predicate_registry_governed.yaml` + +Condition 3 says only: + +> `The seed core ships as registry CONTENT (schema.org-aligned types + predicates with parent anchoring...)` + +Nothing requires 16 predicates or the two D64 promotions. A pre-D64 14-predicate seed passes every listed condition. `registries_design.md` §4 binds the authoritative 16 and the exact signatures `uses: Person|Organization → Product` and `reports_to: Person → Person`. + +### 2.3 Checks with no material factual defect found + +Subject to the citation and coverage issues below, the remaining conditions track the designs closely. In particular, the E0 storage/conversion/structure checks, E1 block/chunk mechanics, no-pre-gate rule, Selection ledger and recall envelope, testimony counting, D43 no-cap behavior, ER cascade/reversibility, projection view contract, retrieval hydration/envelope/grain/negative mechanics, provider ports, delivery shells, and source-of-truth split are substantively aligned. + +## 3. Citation correctness + +### 3.1 Wrong or insufficient binding citations + +| Check | Current binding citation | Problem | Correct binding target | +|---|---|---|---| +| `boundary_correctness_in_library_d60` | requirements §Vision | Too vague to bind the nine named mechanisms or the provider boundary. | `packaging_distribution_design.md` introduction/§§1–5 plus the named subsystem designs; this check is still an omnibus. | +| `boundary_no_human_control_plane_d60` | `CLAUDE.md` Rule 3 | Not a design document. | `packaging_distribution_design.md` §§1–2, 5 and `retrieval_design.md` §§7, 9. | +| `e2_asserted_validity_immutable` | E2/E3 design §7 | §7 is a summary and does not bind the full enum/schema; it also cannot prove the monotonicity condition. | `postgres_schema_design.md` §§1, 8 plus E2/E3 §§3.3, 7. | +| `e3_predicate_registry_governed` | whole `registries_design.md` | D64 is score-critical and the exact seed is in one place. | `registries_design.md` §4 (especially “Seed core” and D64 graduations), plus §7 for promotion. | +| `e3_relations_evidence_collapse` | `plan/analysis/concepts.md` | Analysis is incorrectly called binding. | `postgres_schema_design.md` §9 and `e2_e3_claims_relations_design.md` §5. | +| `e3_supersession_fact_level_bitemporal` | schema §9 | §9 binds relations, not observation no-cap/retraction semantics. | schema §§9 **and 9.A**, plus `observations_design.md` §3. | +| `ops_execution_classes_bound` | analysis workers + orchestration, jointly binding | The analysis inventory is not binding and cannot override the design. | `orchestration_design.md` §8 first; inventory second only for mapping. | +| `p1_lance_estate_rebuildable` | `overall_design.md` §§4–5 | Those sections do not bind all four Lance targets, especially observation labels. | E1 §5; E2/E3 §5; observations §§1, 5; P2 §6; overall §§4–5 for orchestration/rebuild. | +| `p2_rebuild_first_snapshots` | `p2_graph_design.md` | Condition 4 separately scores P3 snapshot behavior not bound by that doc. | Add `e0_files_design.md` §6 for condition 4. | +| `p3_corpus_fs_projection` | E0 §6 + P3 analysis jointly binding | Analysis is not binding. | `e0_files_design.md` §6; lifecycle §2 for lineage stability. | +| `registry_scale_partitioning_d23` | whole schema design | Too broad and misses the load-test source. | `postgres_schema_design.md` §§0, 4, 7–9.A, 12 and `registries_design.md` §§9, 11. | +| `ret_claims_never_answer_now` | malformed requirements path | Correct requirement, wrong path, and it omits the design that implements the bar. | `plan/requirements/requirements_v3.md` §Retrieval plus `retrieval_design.md` §§4 and 6. | +| `scope_views_share_graph_d16` | `registries_design.md` §1 | §1 binds deployment/entity-space separation, but not the scope-view mechanism. | `registries_design.md` §4 (“Scopes share one graph...”) plus `retrieval_design.md` §9. | + +The malformed requirements paths in §1.2 are citation failures even where their conditions are otherwise correct. + +### 3.2 Broad but usable citations + +Several files cite an entire design even though a precise section exists (`er_merges_reversible` → registries §6; `er_resolution_cascade_t0_t4` → §§3 and 5; `er_review_queue_cli` → §8; `eval_golden_sets_d22` → §10; the K checks → their named §§3–8). These are not wrong-document failures, but tightening them would reduce judge drift and make the source precedence operational rather than ceremonial. + +## 4. Coverage against D1–D64 and binding requirements + +### 4.1 Decision coverage status + +- D1–D12 are covered, with the D7/D12 citation/tag caveats below. +- D13–D14 are engine/naming choices and are either excluded by the suite's stated policy or covered incidentally by the fixed-engine/provider checks. +- D15–D25 are represented; D22 and D23 are only partially complete for the reasons below. +- D26–D30 are correctly excluded as withdrawn. +- D31–D63 all have at least one direct check or a scoreable condition, though several conditions need the factual corrections above. +- **D64 is not actually covered.** It is tagged and cited but not asserted. + +### 4.2 Remaining uncovered or under-covered binding behavior + +1. **D64 seed content:** require the authoritative 16-predicate core and the `uses`/`reports_to` promotions/signatures (`registries_design.md` §4). +2. **D23 full physical design and load test:** cover all currently partitioned tables and the ungated-volume load-test hook. The current check covers the original three named tables and blocking indexes, not the current DDL estate. +3. **K trigger surface (D45/D46):** no check covers page watches, subscription-owned routing rules, debounced delta-carrying dispatch, idempotent subscribers, the four authored-page notification channels, declaration lint, or the one-way E → compiled → authored acyclicity invariant (`k_layers_design.md` §5). +4. **K evaluation (D22 pattern applied by D45):** no check requires planted writer-completeness canaries, sampled citation-faithfulness audits, and evidence-change→recompile staleness latency (`k_layers_design.md` §7). `eval_golden_sets_d22` covers ER, retrieval, Selection, grounding, and contradiction only. +5. **K two-band/runtime boundary:** no check requires deterministic fact-sheet rendering, fact-sheet-only pages, stock-harness writer sandbox/no-internet/read-only memory, archived writer transcripts, or driver-only output acceptance (`k_layers_design.md` §§5 and 7). Some pieces are touched by citations and execution classes, but the binding shape is not accepted end-to-end. +6. **Hard-forget in K:** `ops_deletion_cascade_grains` covers normal K refresh and generic source-payload scrubbing, but not the required git-history rewrite/squash and backup treatment for compiled pages that ever cited forgotten evidence, nor authored-page redaction duty (`k_layers_design.md` §10). +7. **Orchestration topology:** no check requires one queue per deployment/stage/lane, steady-state vs backfill separation, version-filter backfill seeding, lane-specific budgets, E3 `(document, entity)` batching, front-loaded reads/batched writes, or pgBouncer in the representative load test (`orchestration_design.md` §§2–5). The current ops checks cover idempotency/DLQ, budgets, classes, and family separation, not this binding runtime design. +8. **D51 skill acceptance:** `ret_filesystem_first_mounts_skill` checks that the skill exists and names its curriculum, but not the binding S58 cold-harness acceptance test required by `retrieval_design.md` §§8 and 11. +9. **D22 continuous ER health:** `eval_golden_sets_d22` covers golden pairs and versioned metrics but not the continuous cluster-size, singleton, unresolved-mention, merge-acceptance, and alias-growth health metrics in `registries_design.md` §10. + +These are implementation-observable and should not be dismissed as pure rationale or naming decisions. + +## 5. Mechanics and README accuracy + +### 5.1 File/YAML mechanics + +- Exactly **69** `.yaml` files exist. +- All 69 parse as YAML. +- All have `schema_version: 1` and `type: harness_judge`. +- All 69 IDs are unique. +- Every ID exactly matches its filename stem. +- All 69 contain the same explicit source-precedence paragraph. +- The README area counts are exact: E0 10, E1 2, E2 9, E3 5, registries/ER/eval 5, K 4, projections 8, retrieval 10, ops 8, code/boundary 8. + +### 5.2 Tag consistency + +Tags are syntactically well-formed, but they are not fully consistent with the README promise that D-tags support decision-filtered runs. + +Six D-tags have no literal citation anywhere in their file: + +- `e1_chunks_whole_blocks_no_overlap`: `D57` +- `e2_selection_recall_envelope`: `D22` +- `ops_execution_classes_bound`: `D33` +- `p1_lance_estate_rebuildable`: `D7` +- `registry_scale_partitioning_d23`: `D17` +- `ret_envelope_grain_discipline`: `D41` + +Conversely, these conditions explicitly invoke material decisions that are absent from their tags: + +- `boundary_extension_points_no_bypass_d60`: D24, D61 +- `e0_converter_router_versioned`: D7 +- `e0_postgres_holds_no_bodies`: D57 +- `e2_context_bundle_two_calls`: D58, D63 +- `e2_grounding_layered_dual_field`: D57 +- `e2_no_pre_extraction_gate`: D12, D56 +- `embedding_model_port_config_d63`: D56 +- `er_review_queue_cli`: D21 +- `k_planner_writer_driver`: D54 +- `ops_execution_classes_bound`: D31 +- `ops_idempotency_dlq`: D45 +- `ret_claims_never_answer_now`: D43 +- `ret_envelope_negative_taxonomy`: D43 + +Not every cross-reference needs a tag, but these are asserted in scored conditions, so filtering by those decisions currently misses checks that can fail on them. Either make tags exhaustive for scoreable D-invariants or narrow the README claim to “primary decision tags.” + +### 5.3 README claims + +Accurate: + +- 69-check inventory and every per-area count. +- One YAML per check, conjunctive scoring, absence-is-failure, and the common precedence paragraph. +- The four Round 1 factual fixes are present. +- The nine named Round 1 additions exist. + +Inaccurate or overstated: + +1. **“Every check therefore reads design-first ... as ‘the binding source’.”** False for the five unflipped requirements-first sentences and contradicted by the four non-design binding designations. +2. **“The current set incorporates [the Round 1] review.”** Not in full: the D23 check omitted the Round 1 review's representative load-test hook, and its table scope did not follow the current schema. +3. **“Inventory ... Binding sources covered.”** The counts are right, but the source lists are not exhaustive despite the heading. Examples: the E3 row omits D64; E2 omits tagged D7/D22/D43/D53; projections omit tagged D62; retrieval omits tagged D43/D60/D61; ops omits tagged D25/D45/D50; code/boundary omits tagged D24. Either make the column exhaustive or label it “primary binding decisions.” +4. **“Exhaustive ... D1–D64.”** Premature while D64 has no scoreable condition and the binding coverage gaps in §4.2 remain. + +## Bottom line + +The suite's file mechanics and the Round 1 four factual repairs are good. The remaining risk is semantic: source precedence is advertised more strongly than it is implemented, and several checks still encode decision prose, reviewer-suggested enforcement mechanisms, or shorthand that the current design explicitly refines. Repair the MUST-FIX items before using these checks as a binary acceptance gate; otherwise a conforming implementation can fail on provider calls, K writer parallelism, borderline grounding audits, reviewer agents, retry configuration, correct schema enums, or harmless internal module layout, while a pre-D64 ontology and incomplete D23 estate can pass. diff --git a/plan/implementation_evals/eval_checks/boundary_correctness_in_library_d60.yaml b/plan/implementation_evals/eval_checks/boundary_correctness_in_library_d60.yaml index d24e998..bff0667 100644 --- a/plan/implementation_evals/eval_checks/boundary_correctness_in_library_d60.yaml +++ b/plan/implementation_evals/eval_checks/boundary_correctness_in_library_d60.yaml @@ -1,19 +1,25 @@ schema_version: 1 id: boundary_correctness_in_library_d60 type: harness_judge -description: Every correctness-determining mechanism the design binds is implemented in this repository — none stubbed, feature-flagged off, or delegated to an external/commercial service. -tags: [boundary, D60] +description: Every correctness-determining mechanism the design binds is implemented in this repository — never gated on, or delegated to, the commercial memory-cloud product; configured provider calls through ports are fine. +tags: [boundary, D60, D61] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/requirements/requirements_v3.md §Vision — the binding source — and decisions.md D60 for rationale and - refinement history, then verify each named correctness module exists in-repo and is - functional (not a stub raising NotImplemented, not disabled by default, not calling out to a - proprietary cloud service to do the work). + Read plan/designs/packaging_distribution_design.md §1–§5 and + plan/requirements/requirements_v3.md §Vision — the binding sources — and decisions.md D60 + for rationale and refinement history, then verify each named correctness module exists + in-repo and is functional. - Check each of these named mechanisms exists in this repository: claim extraction with - Selection; entity resolution (T0-T4); relation/observation supersession + contradiction - adjudication; grounding validation; the eval harness + canaries; budget enforcement; the - DLQ; the deletion cascade; the review CLI. + Check each of these named mechanisms exists in this repository and is functional (not a + stub raising NotImplemented, not disabled by default): claim extraction with Selection; + entity resolution (T0–T4); relation/observation supersession + contradiction adjudication; + grounding validation; the eval harness + canaries; budget enforcement; the DLQ; the + deletion cascade; the review CLI. + + Boundary of the condition: calling a CONFIGURED model/embedding/OCR provider through the + D61 ports is fully conformant (the design routes inference through provider adapters). + What fails is a mechanism whose logic lives only in, or is gated on, an out-of-repo + commercial control plane (the separate memory-cloud product). Source precedence: the design docs under plan/designs/ are the binding statement of the current system; the decision log is its indexed record — read cited decisions together diff --git a/plan/implementation_evals/eval_checks/boundary_extension_points_no_bypass_d60.yaml b/plan/implementation_evals/eval_checks/boundary_extension_points_no_bypass_d60.yaml index 330b20a..c8fb719 100644 --- a/plan/implementation_evals/eval_checks/boundary_extension_points_no_bypass_d60.yaml +++ b/plan/implementation_evals/eval_checks/boundary_extension_points_no_bypass_d60.yaml @@ -2,7 +2,7 @@ schema_version: 1 id: boundary_extension_points_no_bypass_d60 type: harness_judge description: Extension points cannot bypass invariants — ingestion always writes through E0; review always appends reversible verdicts; no consumer hook reaches around the pipeline. -tags: [boundary, D60, D62] +tags: [boundary, D60, D62, D24, D61] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. Read plan/designs/packaging_distribution_design.md (the client surface) — the binding source — and decisions.md D60 (constraint 2) and D62 (client surface) for rationale and diff --git a/plan/implementation_evals/eval_checks/boundary_no_human_control_plane_d60.yaml b/plan/implementation_evals/eval_checks/boundary_no_human_control_plane_d60.yaml index 1ef082f..406061b 100644 --- a/plan/implementation_evals/eval_checks/boundary_no_human_control_plane_d60.yaml +++ b/plan/implementation_evals/eval_checks/boundary_no_human_control_plane_d60.yaml @@ -5,9 +5,10 @@ description: No web UI and no multi-tenant control plane ship in this repository tags: [boundary, D60] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read CLAUDE.md Rule 3 — the binding source — and decisions.md D60 for rationale and - refinement history, then survey the repository's shipped surfaces. - These are acceptance checks for the FINAL system. + Read plan/designs/packaging_distribution_design.md §1–§2 and §5 and + plan/designs/retrieval_design.md §7/§9 — the binding sources — and decisions.md D60 (plus + CLAUDE.md Rule 3 as repo guidance) for rationale and refinement history, then survey the + repository's shipped surfaces. Verify ALL of the following: 1. No web UI ships (no frontend app, no server-rendered dashboards); consumption surfaces diff --git a/plan/implementation_evals/eval_checks/boundary_provider_ports_d61.yaml b/plan/implementation_evals/eval_checks/boundary_provider_ports_d61.yaml index d5c1669..b762591 100644 --- a/plan/implementation_evals/eval_checks/boundary_provider_ports_d61.yaml +++ b/plan/implementation_evals/eval_checks/boundary_provider_ports_d61.yaml @@ -5,7 +5,8 @@ description: The deployment substrate is reached only through the provider ports tags: [boundary, D61] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read requirements_v3.md §Fixed choices & the reference deployment — the binding source — and decisions.md D61 for rationale and + Read plan/requirements/requirements_v3.md §Fixed choices & the reference deployment — the + binding source — and decisions.md D61 for rationale and refinement history, then inspect the adapter layer and self-host stack under src/ and the deploy profiles. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/code_config_via_pydantic_settings.yaml b/plan/implementation_evals/eval_checks/code_config_via_pydantic_settings.yaml index 7997a54..a5e4b57 100644 --- a/plan/implementation_evals/eval_checks/code_config_via_pydantic_settings.yaml +++ b/plan/implementation_evals/eval_checks/code_config_via_pydantic_settings.yaml @@ -1,19 +1,19 @@ schema_version: 1 id: code_config_via_pydantic_settings type: harness_judge -description: All configuration enters through pydantic-settings; secrets are SecretStr/SecretBytes unwrapped only at point of use; no ad-hoc environment access. +description: All configuration enters through pydantic-settings; secrets are SecretStr/SecretBytes unwrapped only at point of use; direct env access only via a reasoned per-line exception. tags: [code, requirements] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding convention. - Read requirements_v3.md §Code, then inspect the settings module(s) and grep the source for - environment access under src/. These are acceptance checks for the FINAL system: if the - subsystem is absent or partial, score 0. + Read plan/requirements/requirements_v3.md §Code — the binding source — then inspect the + settings module(s) and grep the source for environment access under src/. Verify ALL of the following: - 1. Environment variables are read exclusively via pydantic-settings BaseSettings classes — - no os.environ / os.getenv anywhere in src/ (the ruff TID251 ban must still be configured - in pyproject.toml), and no evasions (subprocess env tricks, importing environ under an - alias, reading /proc/self/environ). + 1. Environment variables are read via pydantic-settings BaseSettings classes; the ruff + TID251 ban on os.environ/os.getenv is configured in pyproject.toml. A direct env access + is acceptable ONLY as the requirement's documented exception: a per-line ignore + carrying a written reason — an unexplained or un-ignored direct access fails, as do + evasions (importing environ under an alias, reading /proc/self/environ). 2. Every secret-bearing setting (DB URLs with passwords, API keys, tokens) is typed SecretStr/SecretBytes, and .get_secret_value() appears only at the call sites that hand the value to a client/driver — never stored unwrapped on another object, never logged. diff --git a/plan/implementation_evals/eval_checks/code_tooling_and_migrations.yaml b/plan/implementation_evals/eval_checks/code_tooling_and_migrations.yaml index e1c1ad5..2603906 100644 --- a/plan/implementation_evals/eval_checks/code_tooling_and_migrations.yaml +++ b/plan/implementation_evals/eval_checks/code_tooling_and_migrations.yaml @@ -1,18 +1,16 @@ schema_version: 1 id: code_tooling_and_migrations type: harness_judge -description: The code conventions hold — pyright/pytest/ruff wired into CI, all schema changes as Alembic migrations, typed contracts throughout. +description: The code conventions hold — pyright/pytest/ruff configured and exercised, all schema changes as Alembic migrations, typed contracts throughout. tags: [code, requirements] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding convention. - Read requirements_v3.md §Code, then inspect the tooling configuration, CI workflows, and - migrations under the repo. These are acceptance checks for the FINAL system: if the - subsystem is absent or partial, score 0. + Read plan/requirements/requirements_v3.md §Code — the binding source — then inspect the + tooling configuration and migrations under the repo. Verify ALL of the following: - 1. Pyright, pytest, and ruff are configured (pyproject.toml) and enforced in CI on every PR; - the pyright configuration has not been quietly weakened (per pyproject's - relax-with-receipts rule, any disabled check carries a written reason). + 1. Pyright, pytest, and ruff are configured (pyproject.toml) and actually exercised (a CI + workflow or equivalent runs them; broken types/lint/tests cannot land silently). 2. Every schema change is an Alembic migration — no ad-hoc DDL executed outside the migration chain; migrations replay cleanly from empty. 3. Domain contracts are typed (Pydantic models, enums, Literal/TypedDict where the design diff --git a/plan/implementation_evals/eval_checks/e0_converter_router_versioned.yaml b/plan/implementation_evals/eval_checks/e0_converter_router_versioned.yaml index 3398143..bf2df30 100644 --- a/plan/implementation_evals/eval_checks/e0_converter_router_versioned.yaml +++ b/plan/implementation_evals/eval_checks/e0_converter_router_versioned.yaml @@ -2,7 +2,7 @@ schema_version: 1 id: e0_converter_router_versioned type: harness_judge description: Conversion is a pluggable, versioned router; converters emit document.md + page map + media[], never blocks. -tags: [e0, D38, D57] +tags: [e0, D38, D57, D7] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. Read plan/designs/e0_files_design.md §3 — the binding source — and decisions.md D38 and D57 (the refined converter contract) for rationale and diff --git a/plan/implementation_evals/eval_checks/e0_postgres_holds_no_bodies.yaml b/plan/implementation_evals/eval_checks/e0_postgres_holds_no_bodies.yaml index 54ad712..c8743f5 100644 --- a/plan/implementation_evals/eval_checks/e0_postgres_holds_no_bodies.yaml +++ b/plan/implementation_evals/eval_checks/e0_postgres_holds_no_bodies.yaml @@ -2,7 +2,7 @@ schema_version: 1 id: e0_postgres_holds_no_bodies type: harness_judge description: Postgres stores only compact metadata and the queryable section index — never document bodies or Markdown text. -tags: [e0, D37] +tags: [e0, D37, D57] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. Read plan/designs/e0_files_design.md §2 — the binding source — and decisions.md D37 for rationale and diff --git a/plan/implementation_evals/eval_checks/e0_subworker_chain_crossrefs_d36.yaml b/plan/implementation_evals/eval_checks/e0_subworker_chain_crossrefs_d36.yaml index 963d7a9..c3e57a4 100644 --- a/plan/implementation_evals/eval_checks/e0_subworker_chain_crossrefs_d36.yaml +++ b/plan/implementation_evals/eval_checks/e0_subworker_chain_crossrefs_d36.yaml @@ -5,17 +5,17 @@ description: E0 is one product layer implemented as separately idempotent, separ tags: [e0, D36] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/designs/e0_files_design.md §1 — the binding source — and decisions.md D36 for rationale and - refinement history, then inspect the E0 workers - under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or - partial, score 0. + Read plan/designs/e0_files_design.md §1 — the binding source — and decisions.md D36 for + rationale and refinement history, then inspect the E0 workers under src/. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: 1. E0 runs as a chain of distinct sub-workers — ingest, convert, structure, crossref — each separately idempotent (keyed on content identity + its OWN version) and separately observable (own processing state / status). - 2. A version bump of one sub-worker re-runs only that sub-worker and its downstream - dependents — never the whole chain. + 2. A version bump of one sub-worker re-runs exactly that sub-worker and its downstream + dependents — never unaffected upstream or sibling work (an early-stage bump legitimately + cascades downstream; a late-stage bump must not re-run the stages before it). 3. Crossref extraction records citations/document links as E0 metadata; structuring is not promoted to a top-level E layer (E1–E3 keep their meanings). diff --git a/plan/implementation_evals/eval_checks/e1_chunks_whole_blocks_no_overlap.yaml b/plan/implementation_evals/eval_checks/e1_chunks_whole_blocks_no_overlap.yaml index 88afea3..b8ae662 100644 --- a/plan/implementation_evals/eval_checks/e1_chunks_whole_blocks_no_overlap.yaml +++ b/plan/implementation_evals/eval_checks/e1_chunks_whole_blocks_no_overlap.yaml @@ -12,7 +12,7 @@ instructions: | Verify ALL of the following: 1. A chunk is an ordered run of WHOLE blocks within one section — chunk boundaries always - coincide with block boundaries, and a chunk never crosses a section boundary. + coincide with block boundaries (the D57 block grid), and a chunk never crosses a section boundary. 2. There is NO overlap between chunks anywhere (no sliding windows, no overlap parameters); cross-boundary context comes from the E2 bundle's neighbor chunks instead. 3. Packing uses semchunk to a configured token budget with anchor-stabilized boundaries diff --git a/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml b/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml index ebe195b..902292b 100644 --- a/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml +++ b/plan/implementation_evals/eval_checks/e2_asserted_validity_immutable.yaml @@ -1,25 +1,27 @@ schema_version: 1 id: e2_asserted_validity_immutable type: harness_judge -description: Claims carry an immutable source-asserted validity interval (from/until/precision/kind), grounded by window-membership, never revisable, never a belief authority. +description: Claims carry an immutable source-asserted validity interval (from/until + precision + kind incl. effective_period), grounded by window-membership, never revisable, never a belief authority. tags: [e2, D41, D32] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/designs/e2_e3_claims_relations_design.md §7 — the binding source — and decisions.md D41 for rationale and - refinement history, then inspect the claim schema and the temporal extraction under src/. - These are acceptance checks for the FINAL system: if the subsystem is absent or partial, - score 0. + Read plan/designs/postgres_schema_design.md §1/§8 and + plan/designs/e2_e3_claims_relations_design.md §3.3/§7 — the binding sources — and + decisions.md D41 for rationale and refinement history, then inspect the claim schema and + the temporal extraction under src/. These are acceptance checks for the FINAL system: if + the subsystem is absent or partial, score 0. Verify ALL of the following: - 1. Claims carry claim_valid_from/claim_valid_until + a precision + a kind - (proposition-validity vs event-time vs measurement-period), emitted in the same E2 call. + 1. Claims carry claim_valid_from/claim_valid_until + a precision + a kind matching the + binding enum (proposition_validity | event_time | measurement_period | + effective_period), emitted in the same E2 call. 2. The interval is grounded: the date substring must verbatim-exist in the bundle (window-membership, D32) — ungrounded dates are rejected. - 3. The columns are immutable: no UPDATE path, no invalidated_at/status/superseded_by on them; - a contradicting source yields a NEW claim with its own interval. - 4. The relation/observation adjudicator may consult claim_valid_* as evidence input with a - monotonicity guard (a late retrospective cannot move an adjudicated window), and nothing - reads claim validity back as current belief. + 3. The columns are immutable: no UPDATE path, no invalidated_at/status/superseded_by on + them; a contradicting source yields a NEW claim with its own interval. + 4. The relation/observation adjudicator may consult claim_valid_* as one evidence input, + and nothing anywhere reads claim validity back as the system's current belief — + adjudicated windows live only on relations/observations. Source precedence: the design docs under plan/designs/ are the binding statement of the current system; the decision log is its indexed record — read cited decisions together diff --git a/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml b/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml index 5aa2d08..e256479 100644 --- a/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml +++ b/plan/implementation_evals/eval_checks/e2_context_bundle_two_calls.yaml @@ -2,7 +2,7 @@ schema_version: 1 id: e2_context_bundle_two_calls type: harness_judge description: E2 extraction runs over a context bundle (never a bare chunk) as two calls — Selection, then fused decontextualize+decompose+ground. -tags: [e2, D31, D19] +tags: [e2, D31, D19, D58, D63] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. Read plan/designs/e2_e3_claims_relations_design.md §3 — the binding source — and decisions.md D31/D19 for rationale and diff --git a/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml b/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml index 33a53ac..5008331 100644 --- a/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml +++ b/plan/implementation_evals/eval_checks/e2_grounding_layered_dual_field.yaml @@ -1,14 +1,14 @@ schema_version: 1 id: e2_grounding_layered_dual_field type: harness_judge -description: Claim grounding is dual-field (standalone claim_text + verbatim source_span with offsets + added_context[]) with layered acceptance — anchor, window-membership, entailment, sampled audit. -tags: [e2, D32, D53] +description: Claim grounding is dual-field (standalone claim_text + verbatim source_span with offsets + added_context[]) with layered acceptance — anchor, window-membership, entailment, sampled audit with borderline escalation. +tags: [e2, D32, D53, D57] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/designs/e2_e3_claims_relations_design.md §3.3 — the binding source — and decisions.md D32 for rationale and - refinement history, then inspect - the claim schema and grounding validation under src/. These are acceptance checks for the - FINAL system: if the subsystem is absent or partial, score 0. + Read plan/designs/e2_e3_claims_relations_design.md §3.3 — the binding source — and + decisions.md D32 for rationale and refinement history, then inspect the claim schema and + grounding validation under src/. These are acceptance checks for the FINAL system: if the + subsystem is absent or partial, score 0. Verify ALL of the following: 1. Every claim stores BOTH a standalone claim_text and a verbatim source_span with character @@ -18,8 +18,10 @@ instructions: | as a real slice of the source (never trusted from the model); (b) deterministic window-membership — every declared added substring verbatim-exists in its declared bundle element (fabrications rejected); (c) an in-call entailment self-verdict including the - attribution rule ("X said Y" entails "X said Y", never bare "Y"); (d) a SAMPLED - independent entailment audit — never per-claim, and run on a different model family (D53). + attribution rule ("X said Y" entails "X said Y", never bare "Y"); (d) an independent + entailment audit on a SAMPLED stream by default — with the designed escalation: only a + borderline-confidence band goes to a per-claim judge — run on a different model family + (D53). 3. A decontextualized claim whose text is not a verbatim substring is ACCEPTED when its anchor + declared additions check out (the old verbatim-substring hard reject must not exist); a claim with an unverifiable added substring is REJECTED. diff --git a/plan/implementation_evals/eval_checks/e2_no_pre_extraction_gate.yaml b/plan/implementation_evals/eval_checks/e2_no_pre_extraction_gate.yaml index 8411a79..50c015b 100644 --- a/plan/implementation_evals/eval_checks/e2_no_pre_extraction_gate.yaml +++ b/plan/implementation_evals/eval_checks/e2_no_pre_extraction_gate.yaml @@ -2,7 +2,7 @@ schema_version: 1 id: e2_no_pre_extraction_gate type: harness_judge description: There is no pre-extraction value/salience gate — every chunked document is fully extracted; junk control is in-call Selection plus redundancy collapse. -tags: [e2, D25, D34] +tags: [e2, D25, D34, D12, D56] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. Read plan/designs/e2_e3_claims_relations_design.md §4 — the binding source — and decisions.md D25/D34 for rationale and diff --git a/plan/implementation_evals/eval_checks/e2_selection_recall_envelope.yaml b/plan/implementation_evals/eval_checks/e2_selection_recall_envelope.yaml index cd019e1..1fd1376 100644 --- a/plan/implementation_evals/eval_checks/e2_selection_recall_envelope.yaml +++ b/plan/implementation_evals/eval_checks/e2_selection_recall_envelope.yaml @@ -17,7 +17,7 @@ instructions: | named-entity+predicate statements, change-of-state language. 3. A low-confidence kept_flagged outcome exists (mark-for-review, not delete) — Selection never hard-deletes a borderline candidate silently. - 4. The eval harness plants rare-fact canaries and fails CI if Selection drops one; drop-rate + 4. The eval harness (the D22 golden-set machinery) plants rare-fact canaries and fails CI if Selection drops one; drop-rate tuning is per-fact, not corpus-average. Source precedence: the design docs under plan/designs/ are the binding statement of the diff --git a/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml b/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml index 27dec71..43317da 100644 --- a/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml +++ b/plan/implementation_evals/eval_checks/e3_observations_untyped_adjudicated.yaml @@ -1,14 +1,14 @@ schema_version: 1 id: e3_observations_untyped_adjudicated type: harness_judge -description: Observations are untyped, entity-anchored, bi-temporal facts — no attribute vocabulary or value columns; supersession by exhaustive entity blocking with the no-cap rule and fail-safe coexist. +description: Observations are untyped, entity-anchored, bi-temporal facts — no attribute vocabulary or value columns; supersession by exhaustive entity blocking, the shared adjudication_outcome enum, the no-cap rule, and fail-safe coexist. tags: [e3, D43] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/designs/observations_design.md — the binding source — and decisions.md D43 for rationale and - refinement history, then inspect the observations - schema and the add-observation worker under src/. These are acceptance checks for the FINAL - system: if the subsystem is absent or partial, score 0. + Read plan/designs/observations_design.md and plan/designs/postgres_schema_design.md + §1/§9.A — the binding sources — and decisions.md D43 for rationale and refinement history, + then inspect the observations schema and the add-observation worker under src/. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: 1. Observations are anchored to a resolved entity with an NL statement carrying the value and @@ -17,11 +17,12 @@ instructions: | + temporal sanity checks). 2. Blocking fetches ALL of the entity's live observations by exact key (exhaustive; indexed); semantic similarity only RANKS candidates for hub entities — it never gates membership. - 3. The adjudicator outcomes are a typed enum (evidence / supersede / contradict-coexist / - new), each gated on a positive same-thing match judged semantically; the no-cap rule - holds: only a changing effective state is ever capped; a measurement/fixed-period figure - is never capped on valid-time — a conflicting same-period figure coexists with a shared - contradiction_group. + 3. Adjudication outcomes use the binding shared enum (adjudication_outcome: add | noop | + supersede | contradict | same_as_merge_proposal | retracted_source_removal — where noop + is the evidence-collapse path and add the new-fact path), each gated on a positive + same-thing match judged semantically; the no-cap rule holds: only a changing effective + state is ever capped; a measurement/fixed-period figure is never capped on valid-time — + a conflicting same-period figure coexists with a shared contradiction_group. 4. The fail-safe contract is concrete in code: an explicit margin/threshold configuration gates supersede; every supersede/contradict writes an observation_adjudications reason row; and tests exist demonstrating that ambiguous or incomplete comparisons fall to diff --git a/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml b/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml index 7977cdc..81f63dd 100644 --- a/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml +++ b/plan/implementation_evals/eval_checks/e3_predicate_registry_governed.yaml @@ -1,14 +1,15 @@ schema_version: 1 id: e3_predicate_registry_governed type: harness_judge -description: The predicate vocabulary is a governed registry with an other: escape and promotion review; domain/range constraints are enforced; time is never a predicate or node. +description: The predicate vocabulary is a governed registry with an other: escape and promotion review; domain/range enforced; the D64 seed core ships (16 predicates incl. uses and reports_to); time is never a predicate or node. tags: [e3, D5, D18, D15, D64] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/designs/registries_design.md — the binding source — and decisions.md D5/D15/D18/D64 (the current seed core) for rationale and - refinement history, then inspect the - predicate registry and extraction constraints under src/. These are acceptance checks for - the FINAL system: if the subsystem is absent or partial, score 0. + Read plan/designs/registries_design.md §4 (the seed core incl. the D64 graduations) and §7 + (promotion) — the binding sources — and decisions.md D5/D15/D18/D64 for rationale and + refinement history, then inspect the predicate registry and extraction constraints under + src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. Verify ALL of the following: 1. A Postgres predicate registry exists (name, description, synonyms, status, subject/object @@ -16,9 +17,11 @@ instructions: | review path promotes/maps frequent other: values. 2. Domain/range constraints are mechanically enforced at normalization (an edge-type-map style check that rejects type-invalid triples). - 3. The seed core ships as registry CONTENT (schema.org-aligned types + predicates with - parent anchoring; user extensions declare a core parent — extend-never-fork), not as DDL - or hardcoded enums. + 3. The shipped seed core matches the CURRENT binding content: the 16-predicate core of + registries_design.md §4 — including the two D64 graduations with their signatures, + uses: Person|Organization → Product and reports_to: Person → Person — as registry + CONTENT (rows with parent anchoring; user extensions declare a core parent — + extend-never-fork), not as DDL or hardcoded enums. A pre-D64 14-predicate seed fails. 4. Time is never represented as a predicate, a Date node, or a relation object — temporal data lives in the bi-temporal columns and claim asserted-validity only. 5. Extraction prompts render from the registry (types/predicates/descriptions/examples), so diff --git a/plan/implementation_evals/eval_checks/e3_relations_evidence_collapse.yaml b/plan/implementation_evals/eval_checks/e3_relations_evidence_collapse.yaml index ed18ea8..37b0c95 100644 --- a/plan/implementation_evals/eval_checks/e3_relations_evidence_collapse.yaml +++ b/plan/implementation_evals/eval_checks/e3_relations_evidence_collapse.yaml @@ -1,21 +1,26 @@ schema_version: 1 id: e3_relations_evidence_collapse type: harness_judge -description: Claims and relations are distinct, joined many-to-many with stance; corpus redundancy collapses into one relation with evidence rows. +description: Claims and relations are distinct, joined many-to-many with stance; the same fact with compatible validity collapses into one relation with evidence rows — while re-occurring facts in non-overlapping windows correctly get new rows. tags: [e3, D2] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/analysis/concepts.md — the binding source — and decisions.md D2 for rationale and - refinement history, then inspect the relations and - relation_evidence schema and the E3 normalization under src/. These are acceptance checks - for the FINAL system: if the subsystem is absent or partial, score 0. + Read plan/designs/postgres_schema_design.md §9 and + plan/designs/e2_e3_claims_relations_design.md §5 — the binding sources — and decisions.md + D2 (with plan/analysis/concepts.md as the worked explainer) for rationale and refinement + history, then inspect the relations and relation_evidence schema and the E3 normalization + under src/. These are acceptance checks for the FINAL system: if the subsystem is absent + or partial, score 0. Verify ALL of the following: - 1. Relations are (subject_entity, predicate, object_entity) rows whose identity is the fact; - claims join via relation_evidence(relation_id, claim_id, stance supports|contradicts) — - never claim_id stamped on graph edges, never 1:1 claim=relation assumptions. - 2. A claim can yield 0..n relations; the same fact asserted again links evidence to the - EXISTING relation rather than inserting a duplicate relation row. + 1. Relations are (subject_entity, predicate, object_entity) rows whose identity is the fact + PLUS its validity interval; claims join via relation_evidence(relation_id, claim_id, + stance supports|contradicts) — never claim_id stamped on graph edges, never 1:1 + claim=relation assumptions. + 2. A claim can yield 0..n relations; the same fact asserted again WITH compatible/overlapping + validity links evidence to the EXISTING relation rather than inserting a duplicate — while + a re-occurring fact in a non-overlapping window (Alice worked at Acme twice) correctly + gets a NEW relation row (the schema's EXCLUDE permits exactly this). 3. Duplicate evidence is idempotent (evidence-once per claim/fact pair). Source precedence: the design docs under plan/designs/ are the binding statement of the diff --git a/plan/implementation_evals/eval_checks/e3_supersession_fact_level_bitemporal.yaml b/plan/implementation_evals/eval_checks/e3_supersession_fact_level_bitemporal.yaml index fa1c224..8015682 100644 --- a/plan/implementation_evals/eval_checks/e3_supersession_fact_level_bitemporal.yaml +++ b/plan/implementation_evals/eval_checks/e3_supersession_fact_level_bitemporal.yaml @@ -5,7 +5,9 @@ description: Supersession closes validity windows on relations and effective-sta tags: [e3, D3, D43, D55] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/designs/postgres_schema_design.md §9 — the binding source — and decisions.md D3 and D43 (the no-cap rule) and D55 (source-removal shapes) for rationale and + Read plan/designs/postgres_schema_design.md §9/§9.A and + plan/designs/observations_design.md §3 — the binding sources — and decisions.md D3, D43 + (the no-cap rule), and D55 (source-removal shapes) for rationale and refinement history, then inspect the supersession write paths and temporal columns under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/embedding_model_port_config_d63.yaml b/plan/implementation_evals/eval_checks/embedding_model_port_config_d63.yaml index 0a71dc4..88c12f6 100644 --- a/plan/implementation_evals/eval_checks/embedding_model_port_config_d63.yaml +++ b/plan/implementation_evals/eval_checks/embedding_model_port_config_d63.yaml @@ -2,7 +2,7 @@ schema_version: 1 id: embedding_model_port_config_d63 type: harness_judge description: The embedding model is provider-port configuration, never architecture — versioned per artifact, swappable by version-scoped re-embed; the E1 context-prefix stage exists (conventional default). -tags: [p1, e1, D63, D61] +tags: [p1, e1, D63, D61, D56] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. Read plan/designs/e1_chunks_design.md §5 — the binding source — and decisions.md D63 (and D61's embedder port) for rationale and diff --git a/plan/implementation_evals/eval_checks/er_continuous_health_metrics.yaml b/plan/implementation_evals/eval_checks/er_continuous_health_metrics.yaml new file mode 100644 index 0000000..54d2270 --- /dev/null +++ b/plan/implementation_evals/eval_checks/er_continuous_health_metrics.yaml @@ -0,0 +1,30 @@ +schema_version: 1 +id: er_continuous_health_metrics +type: harness_judge +description: Continuous entity-resolution health monitoring exists — cluster-size distribution, singleton rate, unresolved-mention rate, merge-acceptance rate, and alias-growth anomaly tracking. +tags: [registries, D22, D21] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read plan/designs/registries_design.md §10 — the binding source — and decisions.md D22 for + rationale and refinement history, then inspect the registry health monitoring under src/. + These are acceptance checks for the FINAL system: if the subsystem is absent or partial, + score 0. + + Verify ALL of the following: + 1. Continuous health metrics are computed over the live registry (not only golden-set + evals): cluster-size distribution (black-hole watch), singleton rate, unresolved-mention + rate, and merge-proposal acceptance rate. + 2. Alias-growth anomalies are tracked (the generic-identifier guard's signal: an alias + suddenly linking many entities is surfaced for re-evaluation, D21). + 3. The metrics are recorded over time (queryable trend, not a log line) so resolver-version + bumps are comparable. + + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/er_review_queue_cli.yaml b/plan/implementation_evals/eval_checks/er_review_queue_cli.yaml index 493858d..2897cf2 100644 --- a/plan/implementation_evals/eval_checks/er_review_queue_cli.yaml +++ b/plan/implementation_evals/eval_checks/er_review_queue_cli.yaml @@ -1,22 +1,26 @@ schema_version: 1 id: er_review_queue_cli type: harness_judge -description: Human review is a thin Postgres-backed CLI cluster queue — impact-band routed, hub merges never auto-accepted, verdicts reversible and provenance-stamped; no web UI in the library. -tags: [registries, D24, D60] +description: Review is a thin Postgres-backed CLI cluster queue — impact-band routed to an accountable reviewer (human or designated reviewer agent), hub merges never auto-accepted, verdicts reversible; no web UI in the library. +tags: [registries, D24, D21, D60] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/designs/registries_design.md — the binding source — and decisions.md D24 (and D60) for rationale and - refinement history, then inspect the review tooling under src/. These are - acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + Read plan/designs/registries_design.md §8 and plan/designs/k_layers_design.md §7 (the + reviewer-seat generalization) — the binding sources — and decisions.md D24 for rationale + and refinement history, then inspect the review tooling under src/. These are acceptance + checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: 1. A CLI review queue over Postgres exists, reviewing CLUSTERS (not pairwise), routing only - the expected_impact = blast_radius x (1 - confidence) middle band to humans. + the expected_impact = blast_radius x (1 - confidence) middle band to an accountable + reviewer — a human or a designated reviewer agent (the design permits both; agent + review is normal in agent-operated deployments and must run on a different model family + than the proposer, D53). 2. High-degree hub merges never auto-accept regardless of confidence. 3. Every review action appends a reversible, provenance-stamped record consistent with D21 (redirect-preserving; nothing destructive). 4. No web review UI ships in this repository (D60: the web UI is the cloud product's - surface; the library's review surface is the CLI). + surface; the library's review surface is the CLI/agent consumer). Source precedence: the design docs under plan/designs/ are the binding statement of the current system; the decision log is its indexed record — read cited decisions together diff --git a/plan/implementation_evals/eval_checks/k_eval_canaries_faithfulness.yaml b/plan/implementation_evals/eval_checks/k_eval_canaries_faithfulness.yaml new file mode 100644 index 0000000..83d66a8 --- /dev/null +++ b/plan/implementation_evals/eval_checks/k_eval_canaries_faithfulness.yaml @@ -0,0 +1,31 @@ +schema_version: 1 +id: k_eval_canaries_faithfulness +type: harness_judge +description: K compilation has its own eval machinery — planted writer-completeness canaries, sampled citation-faithfulness audits, and a measured evidence-change→recompile staleness latency. +tags: [k, D45, D22, D53] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read plan/designs/k_layers_design.md §7 — the binding source — and decisions.md D45/D22 for + rationale and refinement history, then inspect the K eval harness under src/. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Writer-completeness canaries exist: planted evidence that a compiled page MUST surface; + a compile that drops a canary fails the eval. + 2. Citation-faithfulness audits run on a sampled stream: does the cited evidence actually + support the compiled prose — judged by a checker on a different model family than the + writer (D53). + 3. Staleness latency is measured: the time from an evidence change to the affected pages' + recompile is tracked against the configured cadence. + 4. Results land in the same recorded eval-run machinery as the ER/retrieval evals + (versioned, comparable across prompt/model bumps). + + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/k_hard_forget_git_history.yaml b/plan/implementation_evals/eval_checks/k_hard_forget_git_history.yaml new file mode 100644 index 0000000..9f824be --- /dev/null +++ b/plan/implementation_evals/eval_checks/k_hard_forget_git_history.yaml @@ -0,0 +1,32 @@ +schema_version: 1 +id: k_hard_forget_git_history +type: harness_judge +description: Hard-forget reaches the K git plane — history rewrite/squash for compiled pages that ever cited forgotten evidence, backup treatment, and authored-page redaction duty. +tags: [k, ops, D46, D55] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read plan/designs/k_layers_design.md §10 — the binding source — and decisions.md D46/D55 + for rationale and refinement history, then inspect the hard-forget path under src/. These + are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Hard-forget identifies, via the citation records, every compiled page (and its git + history) that ever cited the forgotten evidence, and rewrites/squashes that history so + the content is not recoverable from the repo — current recompilation alone is not + enough. + 2. K repo backups are treated: the mechanism bounds or rewrites backup copies so + hard-forgotten content does not survive in a restorable backup beyond the documented + window. + 3. Authored pages that cite forgotten evidence get a redaction-duty flag routed to their + owner (the system never rewrites human words itself — but it must not silently leave + the reference either). + + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/k_one_mechanism_n_scopes.yaml b/plan/implementation_evals/eval_checks/k_one_mechanism_n_scopes.yaml index ce7b7d0..e47f545 100644 --- a/plan/implementation_evals/eval_checks/k_one_mechanism_n_scopes.yaml +++ b/plan/implementation_evals/eval_checks/k_one_mechanism_n_scopes.yaml @@ -5,16 +5,17 @@ description: One compile mechanism, N scopes — tiers are configuration; K3 is tags: [k, D47, D54] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/designs/k_layers_design.md §2 — the binding source — and decisions.md D47 for rationale and - refinement history, then inspect the scope/tier - configuration under src/. These are acceptance checks for the FINAL system: if the subsystem - is absent or partial, score 0. + Read plan/designs/k_layers_design.md §2/§7/§8 — the binding source — and decisions.md D47 + for rationale and refinement history, then inspect the scope/tier configuration under + src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. Verify ALL of the following: 1. There is exactly ONE compile pipeline; K1/K2/K3 exist as configuration (scopes = git subtree + registry rows + rules), not as separate machinery or code paths. - 2. Each scope has a shared model page that is a declared compile input of every page in the - scope. + 2. Each scope has a shared model page that is a declared input of every WRITER invocation + in the scope (authored pages have no compile inputs; fact-sheet-only pages skip the + writer — the requirement binds writer runs, not literally every page). 3. The belief tier (K3) is stricter configuration: rules select only settled evidence (evidence_count >= N, no live contradiction_group, D54-flagged facts excluded), updates are evidence-gated (never timer-driven), and every belief cites supporting AND diff --git a/plan/implementation_evals/eval_checks/k_planner_writer_driver.yaml b/plan/implementation_evals/eval_checks/k_planner_writer_driver.yaml index 8744e3a..42b3585 100644 --- a/plan/implementation_evals/eval_checks/k_planner_writer_driver.yaml +++ b/plan/implementation_evals/eval_checks/k_planner_writer_driver.yaml @@ -1,26 +1,28 @@ schema_version: 1 id: k_planner_writer_driver type: harness_judge -description: Plane K is compiled by planner/writer/driver — mechanical zero-LLM routing, dependency-ordered scheduling, one automated committer, no concurrent sessions. -tags: [k, D45] +description: Plane K is compiled by planner/writer/driver — mechanical zero-LLM routing, dependency-ordered scheduling, one automated committer, no concurrent sessions editing shared files. +tags: [k, D45, D54] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/designs/k_layers_design.md — the binding source — and decisions.md D45 for rationale and - refinement history, then inspect the K compile system - under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or - partial, score 0. + Read plan/designs/k_layers_design.md — the binding source — and decisions.md D45 for + rationale and refinement history, then inspect the K compile system under src/. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: 1. Three roles exist and are separated: an LLM planner owning structure (pages + routing - rules, recorded as append-only knowledge_plan_decisions); LLM writers owning content (one - writer per page per cycle); a DETERMINISTIC driver owning staleness computation, - dependency-ordered scheduling (scope model page first, children before parents, root index - last), output validation, and commits. + rules, recorded as append-only knowledge_plan_decisions); LLM writers owning content + (one writer per STALE COMPILED page per cycle — authored pages are never writer targets, + unchanged pages do not compile, fact-sheet-only pages skip the writer); a DETERMINISTIC + driver owning staleness computation, dependency-ordered scheduling (scope model page + first, children before parents, root index last), output validation, and commits. 2. Routing rules are mechanical: a closed kind set (entity, entity_subtree, predicate_beat, community, doc_set, scope_interests, manual) evaluated by SQL over E-plane keys via an inverted key index — NO LLM call decides routing at evidence-arrival time. - 3. The driver is the repo's ONLY automated committer; there is no in-session merge-conflict - retry and no hot-file rolling-window machinery (removed by D45). + 3. The driver is the repo's ONLY automated committer. Writers legitimately run in parallel + across DISJOINT pages (k_layers §6); what must be impossible is concurrent sessions + editing shared files or committing independently — and there is no in-session + merge-conflict retry and no hot-file rolling-window machinery (removed by D45). 4. Staleness is computed, never guessed: a page is stale iff its recorded inputs_hash no longer matches — and inputs_hash keys on FACT STATE, never raw claim IDs (D54's stale-storm guard); claim-grain citations key on (lineage, chunk_content_hash). diff --git a/plan/implementation_evals/eval_checks/k_trigger_surface_acyclic.yaml b/plan/implementation_evals/eval_checks/k_trigger_surface_acyclic.yaml new file mode 100644 index 0000000..1a1dfeb --- /dev/null +++ b/plan/implementation_evals/eval_checks/k_trigger_surface_acyclic.yaml @@ -0,0 +1,32 @@ +schema_version: 1 +id: k_trigger_surface_acyclic +type: harness_judge +description: The K trigger surface holds — page watches and subscription-owned routing rules with debounced delta-carrying dispatch, idempotent subscribers, declaration lint, and one-way E → compiled → authored acyclicity. +tags: [k, D45, D46] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read plan/designs/k_layers_design.md §5 — the binding source — and decisions.md D45/D46 for + rationale and refinement history, then inspect the K trigger machinery under src/. These + are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Page watches and subscriptions exist as declared, registered routing rules (owned by + their subscriber), evaluated mechanically — never ad-hoc LLM routing at + evidence-arrival time. + 2. Dispatch is debounced and carries deltas (what changed), and subscribers are idempotent + (re-delivery of the same delta is a no-op). + 3. Authored pages get their notification channels as flags/alerts (never rewrites), and a + declaration lint validates watch/subscription declarations. + 4. The trigger surface is acyclic one-way: evidence changes trigger compiled pages; + compiled changes may flag authored pages; nothing flows back from authored/compiled + pages into E-plane truth (loops are structurally impossible, not just discouraged). + + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/k_writer_runtime_two_band.yaml b/plan/implementation_evals/eval_checks/k_writer_runtime_two_band.yaml new file mode 100644 index 0000000..defbe45 --- /dev/null +++ b/plan/implementation_evals/eval_checks/k_writer_runtime_two_band.yaml @@ -0,0 +1,32 @@ +schema_version: 1 +id: k_writer_runtime_two_band +type: harness_judge +description: The K writer runtime honors the two-band contract — deterministic fact-sheet rendering, fact-sheet-only pages skip writers, sandboxed stock-harness writers with archived transcripts, and driver-only output acceptance. +tags: [k, D45, D46, D52] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read plan/designs/k_layers_design.md §5/§7 — the binding source — and decisions.md + D45/D46/D52 for rationale and refinement history, then inspect the K writer runtime under + src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. + + Verify ALL of the following: + 1. Compiled pages are two-band: a deterministically RENDERED fact-sheet band (generated + from spine data, no LLM) and a writer-owned prose band; fact-sheet-only pages skip the + writer entirely. + 2. Writers run as sandboxed stock-harness sessions: no internet access, read-only memory + surfaces, a declared write surface they cannot exceed. + 3. Every writer session's transcript is archived (the D33/D52 ledger discipline applied to + K compilation). + 4. Only the DRIVER accepts writer output into the repo (validation before commit); a writer + cannot commit, and rejected output leaves a recorded trace. + + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ops_deletion_cascade_grains.yaml b/plan/implementation_evals/eval_checks/ops_deletion_cascade_grains.yaml index 7f7616d..6319567 100644 --- a/plan/implementation_evals/eval_checks/ops_deletion_cascade_grains.yaml +++ b/plan/implementation_evals/eval_checks/ops_deletion_cascade_grains.yaml @@ -5,20 +5,23 @@ description: Deletion has grains (version / lineage / hard-forget); normal delet tags: [ops, D55, D46, D54, requirements] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/designs/evidence_lifecycle_design.md §8, and plan/designs/postgres_schema_design.md §13.1 — the binding source — and decisions.md D55/D54/D46 for rationale and - refinement history, then inspect the deletion workers under src/. - These are acceptance checks for the FINAL system: if the subsystem is absent or partial, - score 0. + Read plan/designs/evidence_lifecycle_design.md §8 and + plan/designs/postgres_schema_design.md §13.1 — the binding sources — and decisions.md + D55/D54/D46 for rationale and refinement history, then inspect the deletion workers under + src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. Verify ALL of the following: - 1. Three grains exist: delete a version (currency ends; lineage continues), delete a lineage - (the full cascade), and hard-forget (indistinguishable-from-never-existed semantics). - 2. Normal deletion (version or lineage grain) purges raw/artifact bytes and tombstones the - lineage/version rows, ends testimony currency for affected claims, recomputes counts, and - closes solely-supported facts per shape (states: valid_until cap; measurements: - invalidated_at) — recorded and reversible, never silent. Chunks, claims, evidence links, - relations, and entities are RETAINED as audit history on a normal delete (claims are NOT - deleted; evidence rows remain historical links). + 1. Three grains exist: delete a version (tombstones ONLY that version; currency of its + claims ends; the lineage continues), delete a lineage (tombstones the lineage and every + version; the full cascade), and hard-forget (indistinguishable-from-never-existed + semantics). + 2. Normal deletion at either grain purges the affected raw/artifact bytes, ends testimony + currency for affected claims, recomputes counts, and closes solely-supported facts per + shape (states: valid_until cap; measurements: invalidated_at) — recorded and reversible, + never silent. Chunks, claims, evidence links, relations, and entities are RETAINED as + audit history on a normal delete (claims are NOT deleted; evidence rows remain + historical links). 3. Hard-forget is the separate grain that scrubs source-bearing payloads across versions so forgotten content is indistinguishable from never-existed. 4. Projections converge by rebuild (the deleted document is simply not materialized next diff --git a/plan/implementation_evals/eval_checks/ops_execution_classes_bound.yaml b/plan/implementation_evals/eval_checks/ops_execution_classes_bound.yaml index 51052e5..ee16dd5 100644 --- a/plan/implementation_evals/eval_checks/ops_execution_classes_bound.yaml +++ b/plan/implementation_evals/eval_checks/ops_execution_classes_bound.yaml @@ -2,22 +2,26 @@ schema_version: 1 id: ops_execution_classes_bound type: harness_judge description: Every worker declares an execution class; agent harnesses exist only on plane K and review/audit seats; every LLM worker has an append-only transcript. -tags: [ops, D52, D33, D25] +tags: [ops, D52, D33, D25, D31] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/analysis/workers.md + plan/designs/orchestration_design.md — the binding source — and decisions.md D52 for rationale and - refinement history, then inspect the worker implementations under src/. These are acceptance checks for the - FINAL system: if the subsystem is absent or partial, score 0. + Read plan/designs/orchestration_design.md §8 — the binding source — and decisions.md D52 + for rationale and refinement history (plan/analysis/workers.md is the inventory, useful as + an implementation mapping, not itself binding), then inspect the worker implementations + under src/. These are acceptance checks for the FINAL system: if the subsystem is absent + or partial, score 0. Verify ALL of the following: - 1. Every worker is classified deterministic / programmatic-LLM / agent-harness (the schema's - pipeline enums carry a value for every worker). + 1. Every worker is classified deterministic / programmatic-LLM / agent-harness in a + complete, inspectable inventory or configuration mapping (code registry, config, or + equivalent — the classification must be auditable for every worker, whatever its + storage representation). 2. NO agent-harness worker exists on any per-document, per-claim, or query path — harnesses appear only as K planner/writers and review/audit seats, each with a declared write surface it cannot exceed. - 3. Every worker that makes an LLM call writes an append-only transcript/ledger - (extraction decisions, adjudications, compilations) — no LLM call without a durable - record. + 3. Every worker that makes an LLM call writes an append-only transcript/ledger (the D33 + discipline: extraction decisions, adjudications, compilations) — no LLM call without a + durable record. 4. Programmatic-LLM workers are fixed-shape, schema-constrained, transcripted, and budgeted. Resolution/adjudication workers sit inside cheap-first cascades (spend scales with ambiguity); E2 extraction is the DELIBERATE volume-proportional exception (D25/D31: every diff --git a/plan/implementation_evals/eval_checks/ops_idempotency_dlq.yaml b/plan/implementation_evals/eval_checks/ops_idempotency_dlq.yaml index 4167f37..c2a7151 100644 --- a/plan/implementation_evals/eval_checks/ops_idempotency_dlq.yaml +++ b/plan/implementation_evals/eval_checks/ops_idempotency_dlq.yaml @@ -1,14 +1,14 @@ schema_version: 1 id: ops_idempotency_dlq type: harness_judge -description: Every worker is idempotent on the key fitting its grain; work is processing_state rows (the sole authority) with bounded retries, dead-letter, and a janitor sweep. -tags: [ops, D12, D56, D62] +description: Every worker is idempotent on the key fitting its grain; work is processing_state rows (the sole authority) with a bounded per-stage retry policy, dead-letter, and a janitor sweep. +tags: [ops, D12, D56, D62, D45] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/designs/orchestration_design.md — the binding source — and decisions.md D12, D56, and D62 (task execution) for rationale and - refinement history, then inspect the workers and queue handling under - src/. These are acceptance checks for the FINAL system: if the subsystem is absent or - partial, score 0. + Read plan/designs/orchestration_design.md and plan/designs/postgres_schema_design.md §2 — + the binding sources — and decisions.md D12, D56, and D62 (task execution) for rationale + and refinement history, then inspect the workers and queue handling under src/. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: 1. Every worker is re-runnable and idempotent on the key fitting its grain: per-document/ @@ -18,7 +18,9 @@ instructions: | 2. Work items are processing_state rows in Postgres — the SOLE authority (D62); queue deliveries are delivery-only signals; handlers are registered per stage and shell-agnostic; a janitor sweep re-announces lost deliveries. - 3. Retries are bounded (per the design: max 2) and exhausted work dead-letters into Postgres + 3. Retries follow a bounded, persisted PER-STAGE retry policy (the schema binds + max_attempts with a shipped default of 2 as a tunable starting point — a deliberately + configured different bound is conformant); exhausted work dead-letters into Postgres with recorded status and enough context to inspect and replay — failures never vanish. 4. The per-document E chain is task-driven; aggregate layers (K, P) are debounced or scheduled, never triggered per document. diff --git a/plan/implementation_evals/eval_checks/ops_metering_budgets_enforced.yaml b/plan/implementation_evals/eval_checks/ops_metering_budgets_enforced.yaml index b32ba20..fc10960 100644 --- a/plan/implementation_evals/eval_checks/ops_metering_budgets_enforced.yaml +++ b/plan/implementation_evals/eval_checks/ops_metering_budgets_enforced.yaml @@ -5,7 +5,9 @@ description: Per-layer cost metering exists and budgets are enforced, not adviso tags: [ops, D52, requirements] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read requirements_v3.md §Operational properties (cost discipline) and decisions.md D52, then + Read plan/requirements/requirements_v3.md §Operational properties (cost discipline) and + plan/designs/orchestration_design.md — the binding sources — and decisions.md D52 for + rationale and refinement history, then inspect the cost ledgers and budget checks under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/ops_orchestration_topology.yaml b/plan/implementation_evals/eval_checks/ops_orchestration_topology.yaml new file mode 100644 index 0000000..fd874da --- /dev/null +++ b/plan/implementation_evals/eval_checks/ops_orchestration_topology.yaml @@ -0,0 +1,32 @@ +schema_version: 1 +id: ops_orchestration_topology +type: harness_judge +description: The orchestration topology holds — per-deployment/stage/lane queues, steady vs backfill lane separation with version-filtered seeding and lane budgets, E3 batching, and front-loaded reads with batched writes. +tags: [ops, D52] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read plan/designs/orchestration_design.md §2–§5 — the binding source — and decisions.md D52 + for rationale and refinement history, then inspect the queue/lane wiring under src/. These + are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. Queues are separated per deployment and per stage (backlogs in one stage never starve + another; deployments never share infrastructure paths), with distinct steady-state and + BACKFILL lanes. + 2. Backfills are seeded by version-filter queries (reprocess exactly the artifacts an + upstream version bump affects) and run under lane-specific budgets/rate limits so they + cannot crowd out steady-state work. + 3. E3 work is batched per (document, entity) as the design binds — not one queue message + per claim. + 4. Workers front-load their reads and batch their writes (the cross-cloud hop discipline); + the representative load test includes the pooling layer (pgBouncer) the design names. + + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/ops_producer_checker_families.yaml b/plan/implementation_evals/eval_checks/ops_producer_checker_families.yaml index c4812f3..6de1341 100644 --- a/plan/implementation_evals/eval_checks/ops_producer_checker_families.yaml +++ b/plan/implementation_evals/eval_checks/ops_producer_checker_families.yaml @@ -1,21 +1,23 @@ schema_version: 1 id: ops_producer_checker_families type: harness_judge -description: Every checking seat runs on a different model family than the producer it checks — validated mechanically at startup/CI, auditable via recorded model assignments. +description: Checking seats never share a model family with the producer they check — validated mechanically, auditable via recorded model assignments. tags: [ops, D53] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/designs/orchestration_design.md + plan/designs/k_layers_design.md §7 — the binding source — and decisions.md D53 for rationale and - refinement history, then inspect the model assignment configuration under src/. These are - acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + Read plan/designs/orchestration_design.md §8 — the binding source ("checker seats never + share a family with what they check") — and decisions.md D53 for rationale and refinement + history, then inspect the model assignment configuration under src/. These are acceptance + checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: 1. Checking seats (the sampled grounding judge, contradiction/citation-faithfulness evals, - the review agent, the K reflection pass) are configured on a different model FAMILY than - the producer they check. + the reviewer seat, the K reflection pass) are configured on a different model FAMILY + than the producer they check — the design binds NEVER-share (note: D53's decision prose + mentions a recorded-exception mechanism; under source precedence the stricter design + controls — if you find a same-family exception mechanism, report the discrepancy). 2. A mechanical validator (config validation at startup and/or a CI check) FAILS when a - checker's model family matches its producer's, unless an explicit recorded exception is - present — the separation cannot silently drift. + checker's model family matches its producer's — the separation cannot silently drift. 3. Model assignments are recorded (pipeline_component_versions naming producer and checker models) so the separation is auditable after the fact. diff --git a/plan/implementation_evals/eval_checks/ops_versioned_replay_from_storage.yaml b/plan/implementation_evals/eval_checks/ops_versioned_replay_from_storage.yaml index f34fbed..7251df1 100644 --- a/plan/implementation_evals/eval_checks/ops_versioned_replay_from_storage.yaml +++ b/plan/implementation_evals/eval_checks/ops_versioned_replay_from_storage.yaml @@ -5,8 +5,8 @@ description: Every model-derived artifact records prompt/model/embedding version tags: [ops, D7, D33, requirements] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read requirements_v3.md §Operational properties (versioned processing) and decisions.md - D7/D33, then inspect version stamping and rebuild paths under src/. These are acceptance + Read plan/requirements/requirements_v3.md §Operational properties (versioned processing) — + the binding source — and decisions.md D7/D33 for rationale and refinement history, then inspect version stamping and rebuild paths under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: diff --git a/plan/implementation_evals/eval_checks/p1_lance_estate_rebuildable.yaml b/plan/implementation_evals/eval_checks/p1_lance_estate_rebuildable.yaml index f924bb5..c325cd2 100644 --- a/plan/implementation_evals/eval_checks/p1_lance_estate_rebuildable.yaml +++ b/plan/implementation_evals/eval_checks/p1_lance_estate_rebuildable.yaml @@ -5,7 +5,10 @@ description: P1 is one Lance estate (chunks, claims, relation labels, observatio tags: [p1, D8, D9, D7] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/designs/overall_design.md §4–§5 — the binding source — and decisions.md D8/D9 for rationale and + Read plan/designs/e1_chunks_design.md §5, plan/designs/e2_e3_claims_relations_design.md §5, + plan/designs/observations_design.md §5, plan/designs/p2_graph_design.md §6, and + plan/designs/overall_design.md §4–§5 — the binding sources — and decisions.md D8/D9 for + rationale and refinement history, then inspect the P1 writers and rebuild path under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. @@ -14,7 +17,7 @@ instructions: | 1. Lance tables exist for chunks, claims, relation fact-labels (and observation labels), with the scalar columns the recipes filter on. 2. E-plane workers write P1 inline as artifacts land, but a batch rebuild path from Postgres - exists and is exercised (tests/drills) — P1 carries no authority. + exists and is exercised (tests/drills — the D7 discipline) — P1 carries no authority. 3. Every embedded row is stamped with its embedding model/version, and re-embedding by version filter is a supported batch operation (the embedding-migration path). diff --git a/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml b/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml index f9089cd..9489788 100644 --- a/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml +++ b/plan/implementation_evals/eval_checks/p2_rebuild_first_snapshots.yaml @@ -5,7 +5,8 @@ description: P2 is rebuild-first — full rebuild, validate, publish an immutabl tags: [p2, D7, D40] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/designs/p2_graph_design.md — the binding source — and decisions.md D7 for rationale and + Read plan/designs/p2_graph_design.md (and plan/designs/e0_files_design.md §6 for the P3 + condition) — the binding sources — and decisions.md D7 for rationale and refinement history, then inspect the P2 worker and reader under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. diff --git a/plan/implementation_evals/eval_checks/p3_corpus_fs_projection.yaml b/plan/implementation_evals/eval_checks/p3_corpus_fs_projection.yaml index 61ec988..481da48 100644 --- a/plan/implementation_evals/eval_checks/p3_corpus_fs_projection.yaml +++ b/plan/implementation_evals/eval_checks/p3_corpus_fs_projection.yaml @@ -5,9 +5,11 @@ description: P3 is a rebuildable corpus-filesystem projection — generated stub tags: [p3, D40, D55] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/designs/e0_files_design.md §6 plus plan/analysis/p3_agent_navigation.md — the binding source — and decisions.md D40 (and D55's stability contract) for rationale and - refinement history, then inspect the P3 builder under src/. These are - acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + Read plan/designs/e0_files_design.md §6 — the binding source — and decisions.md D40 (and + D55's stability contract) for rationale and refinement history, with + plan/analysis/p3_agent_navigation.md as supporting context, then inspect the P3 builder + under src/. These are acceptance checks for the FINAL system: if the subsystem is absent + or partial, score 0. Verify ALL of the following: 1. The tree is materialized into versioned snapshots with an atomic latest-pointer swap; diff --git a/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml b/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml index 5dcbac9..47deedd 100644 --- a/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml +++ b/plan/implementation_evals/eval_checks/p_projections_hold_no_authority.yaml @@ -1,27 +1,25 @@ schema_version: 1 id: p_projections_hold_no_authority type: harness_judge -description: Validity state has exactly one home (Postgres); projection builders are architecturally barred from decision-making — enforced import boundaries, dumb deterministic writers. +description: Validity state has exactly one home (Postgres); projection builders are dumb, deterministic, and decision-free — the architectural outcome, however the modules are split. tags: [p, D6, D62] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/designs/overall_design.md §2 — the binding source — and decisions.md D6 (and D62's enforced architecture) for rationale and - refinement history, then inspect the projection builders - and the import-linter contracts under src/. These are acceptance checks for the FINAL - system: if the subsystem is absent or partial, score 0. + Read plan/designs/overall_design.md §2 and plan/designs/p2_graph_design.md §1 — the binding + sources — and decisions.md D6 (and D62's enforced package architecture) for rationale and + refinement history, then inspect the projection builders under src/. These are acceptance + checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: - 1. Projection builders (P1/P2/P3 writers) import only read-side repositories and projection - writers — the architecture contracts (D62 import-linter) bar them from importing - adjudicators, LLM clients, or any repository that mutates validity/contradiction state; - CI fails on illegal imports. - 2. No code path mutates validity/invalidation/contradiction state in a projection store — + 1. No code path mutates validity/invalidation/contradiction state in a projection store — Lance/Ladybug/P3 receive filtered copies from Postgres and are never the site of a - decision. - 3. Every projection is deletable and rebuildable from Postgres (+ artifacts) by a tested + decision. Judge the OUTCOME (no adjudication, no supersession logic, no LLM calls inside + projection builders), whatever the internal module split; the D62 package-level import + contracts (CI-enforced) are supporting evidence, not a prescribed submodule graph. + 2. Every projection is deletable and rebuildable from Postgres (+ artifacts) by a tested batch path; nothing in a projection is source-of-truth. - 4. The graph writer is deterministic — no adjudication, no supersession logic, no LLM calls - inside projection builders. + 3. The projection writers are deterministic (same inputs, same snapshot — modulo declared + versioning). Source precedence: the design docs under plan/designs/ are the binding statement of the current system; the decision log is its indexed record — read cited decisions together diff --git a/plan/implementation_evals/eval_checks/registry_scale_partitioning_d23.yaml b/plan/implementation_evals/eval_checks/registry_scale_partitioning_d23.yaml index 8757d32..8f7a1e4 100644 --- a/plan/implementation_evals/eval_checks/registry_scale_partitioning_d23.yaml +++ b/plan/implementation_evals/eval_checks/registry_scale_partitioning_d23.yaml @@ -1,25 +1,30 @@ schema_version: 1 id: registry_scale_partitioning_d23 type: harness_judge -description: The registry scale rules hold — big append-only tables partitioned, hot tables btree-only, blocking indexes on aliases, no HNSW in OLTP Postgres. -tags: [registries, D23, D17] +description: The physical scale design holds — the partitioned table estate matches the binding DDL, hot tables stay btree-only, blocking indexes exist, no HNSW in OLTP Postgres, and the ungated-volume load test is provided for. +tags: [registries, D23, D17, D25] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/designs/postgres_schema_design.md — the binding source — and decisions.md D23 for rationale and - refinement history, then inspect the - migrations under src/. These are acceptance checks for the FINAL system: if the subsystem is - absent or partial, score 0. + Read plan/designs/postgres_schema_design.md (the partitioned DDL across §§7–9.A and §12, + and §0/§4) and plan/designs/registries_design.md §9/§11 — the binding sources — and + decisions.md D23 for rationale and refinement history, then inspect the migrations under + src/. These are acceptance checks for the FINAL system: if the subsystem is absent or + partial, score 0. Verify ALL of the following: - 1. The large append-only tables (mentions, resolution_decisions, relation_evidence — per the - current schema design) are partitioned (range by ingest period or the schema's chosen - scheme), and the hot append-only tables avoid write-amplifying index types (btree-only - there). + 1. The partitioned estate matches the binding schema — including (not limited to) mentions, + resolution_decisions, relation_evidence, claims, claim_extraction_decisions, chunks, + chunk_claims, testimony_currency_events, and observation_evidence, each partitioned as + the DDL specifies (RANGE by time or HASH by id) — and the hot append-only tables avoid + write-amplifying index types (btree-only there). 2. entities and aliases (the blocking targets) are NOT partitioned. 3. Alias blocking indexes exist: GIN gin_trgm_ops and a Daitch-Mokotoff expression index on - the normalized name; relations carry the btree composite on + the normalized name (D17); relations carry the btree composite on (subject_entity_id, predicate[, object]). 4. Embedding similarity (T3) runs in Lance — no pgvector/HNSW index in OLTP Postgres. + 5. A representative load-test hook exists targeting UNGATED full-extraction volume (D25 + re-stamp; registries_design §9/§11) so partition/index choices are validated before + being locked. Source precedence: the design docs under plan/designs/ are the binding statement of the current system; the decision log is its indexed record — read cited decisions together diff --git a/plan/implementation_evals/eval_checks/ret_claims_never_answer_now.yaml b/plan/implementation_evals/eval_checks/ret_claims_never_answer_now.yaml index ba1309d..b3ba80a 100644 --- a/plan/implementation_evals/eval_checks/ret_claims_never_answer_now.yaml +++ b/plan/implementation_evals/eval_checks/ret_claims_never_answer_now.yaml @@ -2,11 +2,12 @@ schema_version: 1 id: ret_claims_never_answer_now type: harness_judge description: The claim/fact temporal split is enforced in the API surface — claim queries answer "what was asserted"; current-truth routes only through validity-filtered relations and observations. -tags: [retrieval, D41, D49, requirements] +tags: [retrieval, D41, D49, D43, requirements] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read requirements_v3.md §Retrieval (the temporal-split paragraph), decisions.md D41/D49, - then inspect recipe definitions and naming under src/. These are acceptance checks for the + Read plan/requirements/requirements_v3.md §Retrieval (the temporal-split paragraph) and + plan/designs/retrieval_design.md §4/§6 — the binding sources — and decisions.md D41/D49 + for rationale and refinement history, then inspect recipe definitions and naming under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. Verify ALL of the following: diff --git a/plan/implementation_evals/eval_checks/ret_envelope_grain_discipline.yaml b/plan/implementation_evals/eval_checks/ret_envelope_grain_discipline.yaml index 4165a74..e2a3dad 100644 --- a/plan/implementation_evals/eval_checks/ret_envelope_grain_discipline.yaml +++ b/plan/implementation_evals/eval_checks/ret_envelope_grain_discipline.yaml @@ -14,7 +14,7 @@ instructions: | 1. Every primitive and recipe declares a grain (fact / evidence / compiled / composite), and the type system or registry validation enforces it at composition time. 2. Current-fact answers can only be assembled from validity-filtered relations/observations; - a claim-grain source composed into a current-fact answer is rejected mechanically. + a claim-grain source composed into a current-fact answer is rejected mechanically (D41's bar made mechanical). 3. A composite answer is parts[], each part strictly single-grain — mixed answers never dilute the discipline. diff --git a/plan/implementation_evals/eval_checks/ret_envelope_negative_taxonomy.yaml b/plan/implementation_evals/eval_checks/ret_envelope_negative_taxonomy.yaml index 3297def..db8a9ec 100644 --- a/plan/implementation_evals/eval_checks/ret_envelope_negative_taxonomy.yaml +++ b/plan/implementation_evals/eval_checks/ret_envelope_negative_taxonomy.yaml @@ -2,7 +2,7 @@ schema_version: 1 id: ret_envelope_negative_taxonomy type: harness_judge description: A typed negative taxonomy — unknown_entity / known_empty / boundary; no denied kind; hard-forgotten content is indistinguishable from never-existed. -tags: [retrieval, D49, D50] +tags: [retrieval, D49, D50, D43] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. Read plan/designs/retrieval_design.md — the binding source — and decisions.md D49 (typed negatives) and D50 (trust model) for rationale and diff --git a/plan/implementation_evals/eval_checks/ret_skill_s58_acceptance.yaml b/plan/implementation_evals/eval_checks/ret_skill_s58_acceptance.yaml new file mode 100644 index 0000000..55fa454 --- /dev/null +++ b/plan/implementation_evals/eval_checks/ret_skill_s58_acceptance.yaml @@ -0,0 +1,30 @@ +schema_version: 1 +id: ret_skill_s58_acceptance +type: harness_judge +description: The consumption skill has its binding acceptance test (S58) — a never-seen harness uses the memory correctly from the skill alone, run per skill revision. +tags: [retrieval, D51] +instructions: | + You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. + Read plan/designs/retrieval_design.md §8/§11 — the binding source — and decisions.md D51 + for rationale and refinement history, then inspect the skill eval under src/. These are + acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0. + + Verify ALL of the following: + 1. An executable S58 acceptance test exists: a cold agent harness (no prior context beyond + the shipped consumption skill) performs representative memory tasks — orient via K, + verify on the spine, audit to evidence, respect the claim/fact split and freshness + semantics — and is scored. + 2. The test runs per skill REVISION (wired to CI or the release process, not a one-off + script), and its results are recorded. + 3. The harness used for the test is exchangeable (the skill must work for a never-seen + harness, not one tuned-for harness). + + Source precedence: the design docs under plan/designs/ are the binding statement of the + current system; the decision log is its indexed record — read cited decisions together + with their refinement/withdrawal annotations. If sources appear to disagree, treat the + design docs (including postgres_schema_design.md) as controlling, and note the + discrepancy in your reason rather than failing an implementation that follows the + current design. + + Score 1 only if EVERY condition demonstrably holds — cite file paths in your reason. + Otherwise score 0. diff --git a/plan/implementation_evals/eval_checks/scope_views_share_graph_d16.yaml b/plan/implementation_evals/eval_checks/scope_views_share_graph_d16.yaml index bb24b5e..ffa7a65 100644 --- a/plan/implementation_evals/eval_checks/scope_views_share_graph_d16.yaml +++ b/plan/implementation_evals/eval_checks/scope_views_share_graph_d16.yaml @@ -5,7 +5,9 @@ description: One graph, many lenses — scopes never get their own entity space tags: [cross, D16, D50] instructions: | You are judging whether this repository's IMPLEMENTATION honors a binding design invariant. - Read plan/designs/registries_design.md §1 — the binding source — and decisions.md D16 (and D50's trust-model refinement) for rationale and + Read plan/designs/registries_design.md §1 (deployment separation) and §4 (scopes share one + graph) and plan/designs/retrieval_design.md §9 — the binding sources — and decisions.md + D16 (and D50's trust-model refinement) for rationale and refinement history, then inspect scope handling under src/. These are acceptance checks for the FINAL system: if the subsystem is absent or partial, score 0.