Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,10 @@ Skills call `loaf`, hooks enforce through `loaf`, and users see one command surf

Loaf stores operational state in one global SQLite database at `$XDG_DATA_HOME/loaf/loaf.sqlite`, partitioned by project ID. New project IDs are generated and stored in SQLite; they are not derived from checkout path or friendly name. The `projects` row carries the friendly display name and current path, while `project_paths` records path mappings so a checkout can move without changing identity. Legacy path-hash IDs remain only as an adoption key for migrated pre-stable-identity data.

Entity identity follows the same discipline one level down (ADR-028). Derived entity IDs are mint-once opaque keys: computed at first creation, never recomputed for resolution. The aliases table — with the schema's only content-meaningful unique constraint, `UNIQUE (project_id, namespace, alias)` — is the identity registry, and the markdown importer resolves through it before deriving an ID for anything. Unaliased kinds resolve by natural key: journal entries by (entry type, scope, message) for markdown-origin rows, sources by project and path. Sparks whose message normalizes to an empty slug receive a deterministic content-hash alias so no row is born unreachable.

The standing invariant is alias parity: for every project and every aliased entity table, raw row counts equal alias-reachable counts, with zero dead aliases. `loaf state doctor` checks it on demand (read-only, error severity without invalidating the database, naming `loaf state migrate alias-orphans` as the repair). The June-24 identity fork — a project rekey silently invalidating every derived ID, repaired by the state-dedupe Change — is the incident this invariant exists to catch on day one instead of week six.

### Recovery Tiers and Restore Safety

Recovery has three named tiers: `local_rollback` snapshots remain in the same data home for local corruption rollback, project-scoped replay is the ordinary rollback mechanism for later migrations, and `external_disaster_copy` is an operator-selected non-temporary external destination for a point-in-time copy. An explicit destination is resolved through symlinks and rejected when it is absolute-but-volatile; the path check does not prove that the destination is physically remote or durable, so `device_loss_protected` remains false. Backup and verification results include SQLite validity, journal retrieval readiness, search parity, project evidence, checksum, and the latest canonical journal watermark.
Expand All @@ -64,6 +68,19 @@ Recovery has three named tiers: `local_rollback` snapshots remain in the same da

Live activation is therefore a quiesced operator procedure: stop or terminate every harness, Loaf process, background writer, and process that might retain an open database connection; verify the backup and isolated rehearsal; retain a preserve-current backup; while quiesced move the old main database and any matching `-wal` and `-shm` sidecars together into durable quarantine; install the verified copy with mode `0600`; start current Loaf; run `loaf state doctor`, `loaf state status`, and a known journal retrieval check; and, on failure, quiesce again and activate the preserve-current copy. Sidecars from different database files must never be mixed.

### Repair Migrations

Data surgery on the live database rides one sanctioned pattern, proven across three instances (`lifecycle-statuses`, `alias-orphans`, `journal-duplicates`): preview on a temporary copy → mandatory backup → fsynced JSON rollback manifest (file and parent directory, before COMMIT) → apply in one transaction → post-apply verification → `--rollback <manifest>` restoring every deleted row. Registered under `loaf state migrate`; a second apply is provably a no-op.

Four rules the third instance made explicit:

- **Classification iterates to a fixed point** inside the shared classifier, because proof predicates can be sensitive to what the run itself retires. Preview and apply must report the same result set; a repair that reports failure on a correct first apply is a defect (reproduced on a production copy during review, pre-merge).
- **Unproven rows refuse by default.** The migration never guesses; explicit per-row operator dispositions (`--retire`, `--realias`) are accepted in preview so the exact apply invocation is rehearsable, recorded verbatim in the manifest, and conflicting dispositions are a parse error.
- **Reference residue is swept from one shared enumeration.** The polymorphic entity-reference tables (events, relationships, entity_tags, bundle_members, backend_mappings, exports, artifact_bodies with its FTS mirror, aliases) are enumerated once and consumed by every retirement path, so repairs cannot drift from the schema or from each other.
- **FTS mirrors are derived data.** Rollback re-derives index state from restored content rows rather than restoring captured index bytes, and delete paths tolerate a desynced mirror instead of aborting (an unindexed FTS5 external-content delete raises SQLITE_CORRUPT — the tolerance probe exists because a pre-existing desync once made a repair unrunnable).

The operational gate is rehearsal on a disposable production copy: `LOAF_DB` and `XDG_DATA_HOME` redirected to a sandbox, first apply must exit 0, second must no-op, and the acceptance queries must hold before the same invocation touches the real database. The state-dedupe ceremony (2026-08-09, receipts in the Change folder) ran exactly as rehearsed, including catching a generated-flags bug in preview that never reached apply.

### Targets

| Target | Output | Agents | Skills | Hooks | Runtime Plugin |
Expand Down
4 changes: 4 additions & 0 deletions docs/STRATEGY.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ An early `loaf doctor` silently skipped legacy-layout states -- a real `./AGENTS

ADR-020 applies that lesson to the reverse migration: doctor now classifies root-file canonicality, retired `.agents/AGENTS.md` presence, and Claude compatibility-link drift independently. Plain diagnosis stays read-only; `--fix` asks before each repair, preserves legacy content and backups for accepted repairs, and requires `--force` to accept all repairs non-interactively.

**Truthful state is a product invariant.** Two read surfaces disagreeing about how much work exists is worse than an error: a done-task query that returned zero rows while 66 done tasks existed read as "no work," and agents inherited the blindness — the librarian had to bypass the CLI and read SQLite directly to produce an honest report. The June-24 identity fork sat invisible for six weeks because the list surfaces and the housekeeping scanner used different predicates and nothing compared them. The lesson generalizes: wherever two surfaces can drift, a detection diagnostic must exist (doctor's alias parity), and repair tooling must never guess — refuse-by-default with explicit operator dispositions is what made retiring 1,363 rows from the production database an act of judgment rather than faith.

**Review lenses converge, and convergence is the stop signal.** The state-dedupe repair went through eight adversarial passes across two independent models, and each pass surfaced a different defect class: unsafe deletion proofs, then post-commit failure semantics, then cross-component seams, then documentation honesty. Finding *class* trajectory — not finding count — told us when review was done. Two shapes mattered: single-concern review briefs succeeded where a five-mandate mega-brief died, and for destructive tooling the highest-value verdict was a reviewer *reproducing* a defect on a disposable production copy before merge. Review depth is priced by irreversibility: data surgery earned eight passes; the docs commit that followed earned one.

**Personality is decorative, mechanics are durable.** An attempt to decouple agent personality (Warden/Fellowship lore) from agent mechanics through a swappable souls catalog was fully implemented, reviewed twice, and then pivoted in flight before merge: after months of real use the lore had never landed, and review flagged a prose-driven soul file as a brittle prompt-injection surface enforceable only by advice. What shipped keeps the profile neutralization and the skill prose audit; the catalog, CLI, install, and startup-restoration layers were removed.

The implication for both personas: Loaf's value is the *framework* -- mechanical hooks, structured pipeline, profile boundaries, knowledge layer -- not a personality layer. Adding identity through prompt content is incompatible with hardening agents: it costs complexity, adds attack surface, and delivers value only when the user already buys into the metaphor. The lore concept survives in the archived record as an account of what was tried, with a possible future home in a Pi-based harness where it could be load-bearing rather than decorative.
Expand Down
41 changes: 41 additions & 0 deletions docs/changes/20260807-state-dedupe/receipts/ceremony.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Production Repair Ceremony — Receipts

Executed 2026-08-09 (UTC 03:28–04:06), operator present, against `~/.local/share/loaf/loaf.sqlite`, using a binary built from main at `34761008` (the landed squash of PR #159). Every apply was rehearsed in preview with the identical invocation before running.

## Backups and rollback manifests (retained in `~/.local/share/loaf/backups/`)

| Step | Backup | Rollback manifest |
|---|---|---|
| Pre-surgery | `loaf-20260809-032843-120664000.sqlite` (verified) | — |
| alias-orphans apply | `loaf-20260809-033934-282482000.sqlite` | `alias-orphan-rollback-20260809T033935Z.json` |
| journal-duplicates apply | `loaf-20260809-040450-901042000.sqlite` | `journal-duplicate-rollback-20260809T040459Z.json` |
| lifecycle-statuses apply | `loaf-20260809-040538-031233000.sqlite` | `lifecycle-status-rollback-20260809T040538Z.json` |

## alias-orphans

Preview matched the review-time rehearsal exactly: 191 orphans — 168 retire (proven), 23 unproven, 1 dead alias, 150 orphaned sources, 1 named disposition. First apply exited 0 with post-apply verification passing.

Operator dispositions (23 rows):

- 3 June-13 archived task orphans with no title twin — retired (content preserved in git history and the manifest).
- 14 spark orphans across 7 titles that already had an aliased June-24 survivor — retired; realiasing would have resurfaced duplicates.
- 3 spark pairs with no aliased survivor — June-24 member realiased (`release-post-merge-guardrail-inverts-conventional-commits`, `session-lifecycle-states`, `startup-context-from-previous-wrap-up`), June-13 member retired.

Named disposition: `report:7644bb23d2664de93b6cb6a5` archived as moot with a `status_normalized` event recording the unrecoverable evidence and the SPEC-047 rationale.

## journal-duplicates

Preview: 1,349 June-13-window rows, 1,156 June-24-window rows, 866 proven pairs, 614 unproven rows in 153 ambiguous groups. Multiplicity analysis: 119 groups equal across windows, 34 groups with fewer June-24 copies (the markdown's later truth). Operator policy: retire every June-13 member of a cross-window triple — 324 `--retire` dispositions generated from the migration's own classification.

Apply retired 1,190 rows, exit 0. Post-apply: zero pairs, zero unproven; the June-13 window settled at its 159 genuinely-unique rows. A first flags-file generation bug (`--retire None` × 324) was caught by the rehearse-in-preview discipline before any invocation reached apply.

## lifecycle-statuses (first run ever)

14 entities rewritten, 2 events rewritten, zero mappable legacy statuses remaining. 53 out-of-vocabulary free-text statuses (e.g. `raw`, `absorbed`, `captured`, `active`, `unknown`) surfaced as warnings and deliberately left untouched — TASK-408 territory.

## Acceptance

- `loaf state doctor`: alias-parity clear — 27 projects, 189 table checks, raw == alias-reachable everywhere, `multi_alias=0`, `dangling_aliases=0`, exit 0.
- Housekeeping scanner equals canonical list output exactly: tasks 233, specs 26, reports 11 (formerly 299/38/14), ideas 57, sparks 62, brainstorms 4.
- `loaf task list --status done --json` returns the truthful count (zero — all done tasks were archived pre-ceremony).
- Zero `(entry_type, scope, message)` journal twins remain across the two import windows.
Original file line number Diff line number Diff line change
Expand Up @@ -35,17 +35,17 @@ npm run build # ceremony runs the binary built from this branch — no LOAF_DB

## Steps

- [ ] `loaf state backup` and record the backup ID (Recovery Tier: local rollback)
- [ ] `loaf state migrate alias-orphans` (preview): read per-project classification for all projects; record counts
- [ ] Disposition the unproven rows explicitly — expected: 23 rows (rehearsed on a production copy during review). 3 task orphans without title twins (`--retire` or `--realias` per row), and 20 sparks forming 10 both-orphan message pairs: one copy per import instant, neither holding an alias, no surviving twin to bind to. Each pair takes one `--realias` (the member that lives on) and one `--retire`; which member survives is ceremony judgment — the June-24-survives convention from entity twins is the sensible default
- [ ] Rehearse the exact apply invocation as a preview first: `loaf state migrate alias-orphans --retire … --realias …` (dispositions are accepted in preview and reflected in its totals) — the rehearsed and applied invocations must be identical
- [ ] `loaf state migrate alias-orphans --apply --retire … --realias …`; record the manifest path; first run must exit 0 with post-apply verification passing and a truthful non-zero `orphaned_sources` figure
- [ ] `loaf state migrate journal-duplicates` (preview): read pair counts and ambiguous matches; disposition ambiguities via `--retire`; then `--apply`; record the manifest path
- [ ] `loaf state doctor`: alias-parity section green — raw == reachable for every project and table, zero dead aliases
- [ ] Confirm the broken-evidence report is archived with its moot-rationale event
- [ ] `loaf state migrate lifecycle-statuses` preview, then `--apply`; record OOV statuses it could not map, if any
- [ ] Demonstrate count agreement: `loaf housekeeping` totals equal list-command counts for all seven aliased tables; `loaf task list --status done --json` returns exactly the done rows that exist
- [ ] Journal the ceremony: `decision(state)` with counts and dispositions; `discover(state)` for anything the preview revealed about other projects
- [x] `loaf state backup` and record the backup ID (Recovery Tier: local rollback)
- [x] `loaf state migrate alias-orphans` (preview): read per-project classification for all projects; record counts
- [x] Disposition the unproven rows explicitly — expected: 23 rows (rehearsed on a production copy during review). 3 task orphans without title twins (`--retire` or `--realias` per row), and 20 sparks forming 10 both-orphan message pairs: one copy per import instant, neither holding an alias, no surviving twin to bind to. Each pair takes one `--realias` (the member that lives on) and one `--retire`; which member survives is ceremony judgment — the June-24-survives convention from entity twins is the sensible default
- [x] Rehearse the exact apply invocation as a preview first: `loaf state migrate alias-orphans --retire … --realias …` (dispositions are accepted in preview and reflected in its totals) — the rehearsed and applied invocations must be identical
- [x] `loaf state migrate alias-orphans --apply --retire … --realias …`; record the manifest path; first run must exit 0 with post-apply verification passing and a truthful non-zero `orphaned_sources` figure
- [x] `loaf state migrate journal-duplicates` (preview): read pair counts and ambiguous matches; disposition ambiguities via `--retire`; then `--apply`; record the manifest path
- [x] `loaf state doctor`: alias-parity section green — raw == reachable for every project and table, zero dead aliases
- [x] Confirm the broken-evidence report is archived with its moot-rationale event
- [x] `loaf state migrate lifecycle-statuses` preview, then `--apply`; record OOV statuses it could not map, if any
- [x] Demonstrate count agreement: `loaf housekeeping` totals equal list-command counts for all seven aliased tables; `loaf task list --status done --json` returns exactly the done rows that exist
- [x] Journal the ceremony: `decision(state)` with counts and dispositions; `discover(state)` for anything the preview revealed about other projects

## Verification

Expand Down
Loading
Loading