From ae7bf20f3413be8b55c6ccfb5ff6bba44cb97127 Mon Sep 17 00:00:00 2001 From: Levi Figueira Date: Sun, 9 Aug 2026 11:23:38 +0100 Subject: [PATCH 1/2] docs: record the production repair ceremony receipts TASK-004 executed 2026-08-09 with the operator: alias-orphans and journal-duplicates applied exit-0 first try, lifecycle-statuses first run, doctor parity clear across 27 projects, scanner and list surfaces in exact agreement (233/26/11). Checkboxes flipped; backups, manifests, disposition rationale, and acceptance evidence recorded in receipts/ceremony.md. Claude-Session: https://claude.ai/code/session_011wW8VzMJvENoeSY6QxWYkK --- .../receipts/ceremony.md | 41 +++++++++++++++++++ .../TASK-004-production-repair-ceremony.md | 22 +++++----- 2 files changed, 52 insertions(+), 11 deletions(-) create mode 100644 docs/changes/20260807-state-dedupe/receipts/ceremony.md diff --git a/docs/changes/20260807-state-dedupe/receipts/ceremony.md b/docs/changes/20260807-state-dedupe/receipts/ceremony.md new file mode 100644 index 00000000..f700820c --- /dev/null +++ b/docs/changes/20260807-state-dedupe/receipts/ceremony.md @@ -0,0 +1,41 @@ +# Production Repair Ceremony — Receipts + +Executed 2026-08-09 (UTC 03:28–04:06), operator present, against `~/.local/share/loaf/loaf.sqlite`, using a binary built from main at `34761008` (the landed squash of PR #159). Every apply was rehearsed in preview with the identical invocation before running. + +## Backups and rollback manifests (retained in `~/.local/share/loaf/backups/`) + +| Step | Backup | Rollback manifest | +|---|---|---| +| Pre-surgery | `loaf-20260809-032843-120664000.sqlite` (verified) | — | +| alias-orphans apply | `loaf-20260809-033934-282482000.sqlite` | `alias-orphan-rollback-20260809T033935Z.json` | +| journal-duplicates apply | `loaf-20260809-040450-901042000.sqlite` | `journal-duplicate-rollback-20260809T040459Z.json` | +| lifecycle-statuses apply | `loaf-20260809-040538-031233000.sqlite` | `lifecycle-status-rollback-20260809T040538Z.json` | + +## alias-orphans + +Preview matched the review-time rehearsal exactly: 191 orphans — 168 retire (proven), 23 unproven, 1 dead alias, 150 orphaned sources, 1 named disposition. First apply exited 0 with post-apply verification passing. + +Operator dispositions (23 rows): + +- 3 June-13 archived task orphans with no title twin — retired (content preserved in git history and the manifest). +- 14 spark orphans across 7 titles that already had an aliased June-24 survivor — retired; realiasing would have resurfaced duplicates. +- 3 spark pairs with no aliased survivor — June-24 member realiased (`release-post-merge-guardrail-inverts-conventional-commits`, `session-lifecycle-states`, `startup-context-from-previous-wrap-up`), June-13 member retired. + +Named disposition: `report:7644bb23d2664de93b6cb6a5` archived as moot with a `status_normalized` event recording the unrecoverable evidence and the SPEC-047 rationale. + +## journal-duplicates + +Preview: 1,349 June-13-window rows, 1,156 June-24-window rows, 866 proven pairs, 614 unproven rows in 153 ambiguous groups. Multiplicity analysis: 119 groups equal across windows, 34 groups with fewer June-24 copies (the markdown's later truth). Operator policy: retire every June-13 member of a cross-window triple — 324 `--retire` dispositions generated from the migration's own classification. + +Apply retired 1,190 rows, exit 0. Post-apply: zero pairs, zero unproven; the June-13 window settled at its 159 genuinely-unique rows. A first flags-file generation bug (`--retire None` × 324) was caught by the rehearse-in-preview discipline before any invocation reached apply. + +## lifecycle-statuses (first run ever) + +14 entities rewritten, 2 events rewritten, zero mappable legacy statuses remaining. 53 out-of-vocabulary free-text statuses (e.g. `raw`, `absorbed`, `captured`, `active`, `unknown`) surfaced as warnings and deliberately left untouched — TASK-408 territory. + +## Acceptance + +- `loaf state doctor`: alias-parity clear — 27 projects, 189 table checks, raw == alias-reachable everywhere, `multi_alias=0`, `dangling_aliases=0`, exit 0. +- Housekeeping scanner equals canonical list output exactly: tasks 233, specs 26, reports 11 (formerly 299/38/14), ideas 57, sparks 62, brainstorms 4. +- `loaf task list --status done --json` returns the truthful count (zero — all done tasks were archived pre-ceremony). +- Zero `(entry_type, scope, message)` journal twins remain across the two import windows. diff --git a/docs/changes/20260807-state-dedupe/tasks/TASK-004-production-repair-ceremony.md b/docs/changes/20260807-state-dedupe/tasks/TASK-004-production-repair-ceremony.md index 26fce8c9..546229ec 100644 --- a/docs/changes/20260807-state-dedupe/tasks/TASK-004-production-repair-ceremony.md +++ b/docs/changes/20260807-state-dedupe/tasks/TASK-004-production-repair-ceremony.md @@ -35,17 +35,17 @@ npm run build # ceremony runs the binary built from this branch — no LOAF_DB ## Steps -- [ ] `loaf state backup` and record the backup ID (Recovery Tier: local rollback) -- [ ] `loaf state migrate alias-orphans` (preview): read per-project classification for all projects; record counts -- [ ] Disposition the unproven rows explicitly — expected: 23 rows (rehearsed on a production copy during review). 3 task orphans without title twins (`--retire` or `--realias` per row), and 20 sparks forming 10 both-orphan message pairs: one copy per import instant, neither holding an alias, no surviving twin to bind to. Each pair takes one `--realias` (the member that lives on) and one `--retire`; which member survives is ceremony judgment — the June-24-survives convention from entity twins is the sensible default -- [ ] Rehearse the exact apply invocation as a preview first: `loaf state migrate alias-orphans --retire … --realias …` (dispositions are accepted in preview and reflected in its totals) — the rehearsed and applied invocations must be identical -- [ ] `loaf state migrate alias-orphans --apply --retire … --realias …`; record the manifest path; first run must exit 0 with post-apply verification passing and a truthful non-zero `orphaned_sources` figure -- [ ] `loaf state migrate journal-duplicates` (preview): read pair counts and ambiguous matches; disposition ambiguities via `--retire`; then `--apply`; record the manifest path -- [ ] `loaf state doctor`: alias-parity section green — raw == reachable for every project and table, zero dead aliases -- [ ] Confirm the broken-evidence report is archived with its moot-rationale event -- [ ] `loaf state migrate lifecycle-statuses` preview, then `--apply`; record OOV statuses it could not map, if any -- [ ] Demonstrate count agreement: `loaf housekeeping` totals equal list-command counts for all seven aliased tables; `loaf task list --status done --json` returns exactly the done rows that exist -- [ ] Journal the ceremony: `decision(state)` with counts and dispositions; `discover(state)` for anything the preview revealed about other projects +- [x] `loaf state backup` and record the backup ID (Recovery Tier: local rollback) +- [x] `loaf state migrate alias-orphans` (preview): read per-project classification for all projects; record counts +- [x] Disposition the unproven rows explicitly — expected: 23 rows (rehearsed on a production copy during review). 3 task orphans without title twins (`--retire` or `--realias` per row), and 20 sparks forming 10 both-orphan message pairs: one copy per import instant, neither holding an alias, no surviving twin to bind to. Each pair takes one `--realias` (the member that lives on) and one `--retire`; which member survives is ceremony judgment — the June-24-survives convention from entity twins is the sensible default +- [x] Rehearse the exact apply invocation as a preview first: `loaf state migrate alias-orphans --retire … --realias …` (dispositions are accepted in preview and reflected in its totals) — the rehearsed and applied invocations must be identical +- [x] `loaf state migrate alias-orphans --apply --retire … --realias …`; record the manifest path; first run must exit 0 with post-apply verification passing and a truthful non-zero `orphaned_sources` figure +- [x] `loaf state migrate journal-duplicates` (preview): read pair counts and ambiguous matches; disposition ambiguities via `--retire`; then `--apply`; record the manifest path +- [x] `loaf state doctor`: alias-parity section green — raw == reachable for every project and table, zero dead aliases +- [x] Confirm the broken-evidence report is archived with its moot-rationale event +- [x] `loaf state migrate lifecycle-statuses` preview, then `--apply`; record OOV statuses it could not map, if any +- [x] Demonstrate count agreement: `loaf housekeeping` totals equal list-command counts for all seven aliased tables; `loaf task list --status done --json` returns exactly the done rows that exist +- [x] Journal the ceremony: `decision(state)` with counts and dispositions; `discover(state)` for anything the preview revealed about other projects ## Verification From 9f5eaeb0401f5d7587c5a039bd72d5e7865c47b3 Mon Sep 17 00:00:00 2001 From: Levi Figueira Date: Sun, 9 Aug 2026 12:31:52 +0100 Subject: [PATCH 2/2] docs: reflect the state-dedupe arc into the strategic record ADR-028 records that entity identity lives in the alias registry and derived IDs are mint-once opaque keys. ARCHITECTURE.md extends the identity section down to the entity level with the alias-parity invariant, and gains a Repair Migrations section codifying the preview/backup/manifest/apply/verify/rollback pattern with its four proven rules. STRATEGY.md gains two principles: truthful state as a product invariant, and review-lens convergence as the stop signal. Claude-Session: https://claude.ai/code/session_011wW8VzMJvENoeSY6QxWYkK --- docs/ARCHITECTURE.md | 17 ++++ docs/STRATEGY.md | 4 + ...ADR-028-registry-identity-mint-once-ids.md | 83 +++++++++++++++++++ 3 files changed, 104 insertions(+) create mode 100644 docs/decisions/ADR-028-registry-identity-mint-once-ids.md diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index aea30f20..82f8b43b 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -56,6 +56,10 @@ Skills call `loaf`, hooks enforce through `loaf`, and users see one command surf Loaf stores operational state in one global SQLite database at `$XDG_DATA_HOME/loaf/loaf.sqlite`, partitioned by project ID. New project IDs are generated and stored in SQLite; they are not derived from checkout path or friendly name. The `projects` row carries the friendly display name and current path, while `project_paths` records path mappings so a checkout can move without changing identity. Legacy path-hash IDs remain only as an adoption key for migrated pre-stable-identity data. +Entity identity follows the same discipline one level down (ADR-028). Derived entity IDs are mint-once opaque keys: computed at first creation, never recomputed for resolution. The aliases table — with the schema's only content-meaningful unique constraint, `UNIQUE (project_id, namespace, alias)` — is the identity registry, and the markdown importer resolves through it before deriving an ID for anything. Unaliased kinds resolve by natural key: journal entries by (entry type, scope, message) for markdown-origin rows, sources by project and path. Sparks whose message normalizes to an empty slug receive a deterministic content-hash alias so no row is born unreachable. + +The standing invariant is alias parity: for every project and every aliased entity table, raw row counts equal alias-reachable counts, with zero dead aliases. `loaf state doctor` checks it on demand (read-only, error severity without invalidating the database, naming `loaf state migrate alias-orphans` as the repair). The June-24 identity fork — a project rekey silently invalidating every derived ID, repaired by the state-dedupe Change — is the incident this invariant exists to catch on day one instead of week six. + ### Recovery Tiers and Restore Safety Recovery has three named tiers: `local_rollback` snapshots remain in the same data home for local corruption rollback, project-scoped replay is the ordinary rollback mechanism for later migrations, and `external_disaster_copy` is an operator-selected non-temporary external destination for a point-in-time copy. An explicit destination is resolved through symlinks and rejected when it is absolute-but-volatile; the path check does not prove that the destination is physically remote or durable, so `device_loss_protected` remains false. Backup and verification results include SQLite validity, journal retrieval readiness, search parity, project evidence, checksum, and the latest canonical journal watermark. @@ -64,6 +68,19 @@ Recovery has three named tiers: `local_rollback` snapshots remain in the same da Live activation is therefore a quiesced operator procedure: stop or terminate every harness, Loaf process, background writer, and process that might retain an open database connection; verify the backup and isolated rehearsal; retain a preserve-current backup; while quiesced move the old main database and any matching `-wal` and `-shm` sidecars together into durable quarantine; install the verified copy with mode `0600`; start current Loaf; run `loaf state doctor`, `loaf state status`, and a known journal retrieval check; and, on failure, quiesce again and activate the preserve-current copy. Sidecars from different database files must never be mixed. +### Repair Migrations + +Data surgery on the live database rides one sanctioned pattern, proven across three instances (`lifecycle-statuses`, `alias-orphans`, `journal-duplicates`): preview on a temporary copy → mandatory backup → fsynced JSON rollback manifest (file and parent directory, before COMMIT) → apply in one transaction → post-apply verification → `--rollback ` restoring every deleted row. Registered under `loaf state migrate`; a second apply is provably a no-op. + +Four rules the third instance made explicit: + +- **Classification iterates to a fixed point** inside the shared classifier, because proof predicates can be sensitive to what the run itself retires. Preview and apply must report the same result set; a repair that reports failure on a correct first apply is a defect (reproduced on a production copy during review, pre-merge). +- **Unproven rows refuse by default.** The migration never guesses; explicit per-row operator dispositions (`--retire`, `--realias`) are accepted in preview so the exact apply invocation is rehearsable, recorded verbatim in the manifest, and conflicting dispositions are a parse error. +- **Reference residue is swept from one shared enumeration.** The polymorphic entity-reference tables (events, relationships, entity_tags, bundle_members, backend_mappings, exports, artifact_bodies with its FTS mirror, aliases) are enumerated once and consumed by every retirement path, so repairs cannot drift from the schema or from each other. +- **FTS mirrors are derived data.** Rollback re-derives index state from restored content rows rather than restoring captured index bytes, and delete paths tolerate a desynced mirror instead of aborting (an unindexed FTS5 external-content delete raises SQLITE_CORRUPT — the tolerance probe exists because a pre-existing desync once made a repair unrunnable). + +The operational gate is rehearsal on a disposable production copy: `LOAF_DB` and `XDG_DATA_HOME` redirected to a sandbox, first apply must exit 0, second must no-op, and the acceptance queries must hold before the same invocation touches the real database. The state-dedupe ceremony (2026-08-09, receipts in the Change folder) ran exactly as rehearsed, including catching a generated-flags bug in preview that never reached apply. + ### Targets | Target | Output | Agents | Skills | Hooks | Runtime Plugin | diff --git a/docs/STRATEGY.md b/docs/STRATEGY.md index 8e4421ae..d37cdb28 100644 --- a/docs/STRATEGY.md +++ b/docs/STRATEGY.md @@ -34,6 +34,10 @@ An early `loaf doctor` silently skipped legacy-layout states -- a real `./AGENTS ADR-020 applies that lesson to the reverse migration: doctor now classifies root-file canonicality, retired `.agents/AGENTS.md` presence, and Claude compatibility-link drift independently. Plain diagnosis stays read-only; `--fix` asks before each repair, preserves legacy content and backups for accepted repairs, and requires `--force` to accept all repairs non-interactively. +**Truthful state is a product invariant.** Two read surfaces disagreeing about how much work exists is worse than an error: a done-task query that returned zero rows while 66 done tasks existed read as "no work," and agents inherited the blindness — the librarian had to bypass the CLI and read SQLite directly to produce an honest report. The June-24 identity fork sat invisible for six weeks because the list surfaces and the housekeeping scanner used different predicates and nothing compared them. The lesson generalizes: wherever two surfaces can drift, a detection diagnostic must exist (doctor's alias parity), and repair tooling must never guess — refuse-by-default with explicit operator dispositions is what made retiring 1,363 rows from the production database an act of judgment rather than faith. + +**Review lenses converge, and convergence is the stop signal.** The state-dedupe repair went through eight adversarial passes across two independent models, and each pass surfaced a different defect class: unsafe deletion proofs, then post-commit failure semantics, then cross-component seams, then documentation honesty. Finding *class* trajectory — not finding count — told us when review was done. Two shapes mattered: single-concern review briefs succeeded where a five-mandate mega-brief died, and for destructive tooling the highest-value verdict was a reviewer *reproducing* a defect on a disposable production copy before merge. Review depth is priced by irreversibility: data surgery earned eight passes; the docs commit that followed earned one. + **Personality is decorative, mechanics are durable.** An attempt to decouple agent personality (Warden/Fellowship lore) from agent mechanics through a swappable souls catalog was fully implemented, reviewed twice, and then pivoted in flight before merge: after months of real use the lore had never landed, and review flagged a prose-driven soul file as a brittle prompt-injection surface enforceable only by advice. What shipped keeps the profile neutralization and the skill prose audit; the catalog, CLI, install, and startup-restoration layers were removed. The implication for both personas: Loaf's value is the *framework* -- mechanical hooks, structured pipeline, profile boundaries, knowledge layer -- not a personality layer. Adding identity through prompt content is incompatible with hardening agents: it costs complexity, adds attack surface, and delivers value only when the user already buys into the metaphor. The lore concept survives in the archived record as an account of what was tried, with a possible future home in a Pi-based harness where it could be load-bearing rather than decorative. diff --git a/docs/decisions/ADR-028-registry-identity-mint-once-ids.md b/docs/decisions/ADR-028-registry-identity-mint-once-ids.md new file mode 100644 index 00000000..ab88c036 --- /dev/null +++ b/docs/decisions/ADR-028-registry-identity-mint-once-ids.md @@ -0,0 +1,83 @@ +# ADR-028: Entity identity lives in the alias registry; derived IDs are mint-once + +Decision Date: 2026-08-09 +Status: Accepted + +## Context + +Entity primary keys in the global SQLite database were derived — +`sha256(kind, project_id, alias)` — and the markdown importer resolved +identity by recomputing that derivation. Both halves assumed every input +to the hash was immutable. `project_id` was not: migration 3 rekeyed +projects from legacy path-hash IDs to opaque `proj_` IDs, rewriting the +`project_id` column but necessarily leaving the derived `id` columns +untouched. The next markdown import (2026-06-24) recomputed every ID +under the new salt, found no match, inserted a full second copy of every +artifact, and re-pointed each alias to the new twin via +`ON CONFLICT(project_id, namespace, alias) DO UPDATE SET entity_id`. + +The June-13 originals became alias-orphans: invisible to every list +surface (all list queries INNER JOIN through `aliases`) yet counted by +the housekeeping scanner (raw `WHERE project_id = ?`). The same fork +duplicated ~1,020 journal entries, which carry no aliases at all. The +damage was invisible for six weeks and was discovered by accident during +a housekeeping pass (journal `finding(state)`, 2026-08-07). + +The entity tables had no unique constraint other than the derived +primary key. The only content-meaningful uniqueness in the schema was +`aliases UNIQUE (project_id, namespace, alias)` — the registry was +already the de-facto identity authority; the importer just didn't +consult it. + +## Decision + +1. **Derived entity IDs are mint-once opaque keys.** An ID is computed + at first creation and never recomputed for resolution. No code path + may locate an existing row by re-deriving its ID; the one exception + is the alias-orphans repair migration's twin proof, which recomputes + against *historical* salts precisely to identify rows damaged before + this decision. +2. **Identity resolution goes through the registry.** Importers and any + other writer that might encounter an existing entity resolve + `(project_id, namespace, alias)` against the aliases table first and + reuse the registered entity's ID. Unaliased kinds resolve by natural + key (journal entries: entry type, scope, message for markdown-origin + rows; sources: project and path). +3. **Divergence is detected, not assumed away.** `loaf state doctor` + carries an alias-parity diagnostic: for every project and aliased + entity table, raw row counts must equal alias-reachable counts, with + zero dead aliases. Damage reports as an error diagnostic naming the + repair; it does not invalidate the database. + +## Alternatives rejected + +- **Re-deriving entity IDs after a rekey** (rewriting `id` columns + across every table and reference in one transaction). Rejected: large + blast radius insuring against a scenario the registry-first importer + already neutralizes. Recorded as permanently foreclosed in the + state-dedupe Change's Cut list. +- **Schema-level content uniqueness on entity tables.** Rejected: the + aliases registry is the identity authority; a second constraint + surface would drift from it. + +## Consequences + +- A project rekey, storage-home merge, or import replay can no longer + fork the identity space; at worst it mints alias-safe duplicates in + one documented spark corner case (verbatim-duplicated lines under + rekey re-import), which stays alias-reachable and repairable. +- Repair of pre-decision damage ships as audited migrations + (`alias-orphans`, `journal-duplicates`) on the repair-migration + pattern (see ARCHITECTURE.md, Repair Migrations). +- Executed 2026-08-09 against the production database: 191 alias + orphans and 1,190 duplicated journal rows retired; scanner and list + surfaces agree exactly (233/26/11); doctor parity clear across 27 + projects. + +## Evidence + +- PR #159 (squash `34761008`) and PR #158 (the captured brief) +- `docs/changes/20260807-state-dedupe/` — shape.md Decisions 2, 5, 8; + receipts/ceremony.md +- Journal: `decision(identity)` 2026-08-08, `decision(state)` and + `wrap(state-dedupe)` 2026-08-09