feat: autonomous 3-2-1 backup system (centralized brains, strongest secret scanner, hard-fail)

.github/workflows/backup-system-tests.yml

@@ -0,0 +1,18 @@
+name: backup-system tests
+
+on:
+  push:
+    paths: ["backup-system/**"]
+  pull_request:
+    paths: ["backup-system/**"]
+
+jobs:
+  gate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install deps
+        run: sudo apt-get update && sudo apt-get install -y jq rsync
+      - name: Run gate (bash -n + behavior tests)
+        working-directory: backup-system
+        run: bash tests/run.sh

README.md

@@ -229,6 +229,20 @@ CKIS runs 5 crons that maintain the system without manual effort:
 
 ---
 
+## Autonomous Backup (3-2-1)
+
+A knowledge base with no backup is a liability. CKIS ships a **self-hosting, declarative backup system** — see [`backup-system/`](backup-system/).
+
+- **3-2-1**: local + private GitHub remotes (off-site) + external-drive git bundles (off-device).
+- **Declarative**: one `ckis-manifest.json` classifies every path (`track` / `regenerable` / `secret` / `sensitive` / `snapshot`). Adding a repo or tool is one entry.
+- **Three rings of autonomy**: push on session end, a daily `systemd --user` safety net (self-heals missing remotes, aggregates, runs physical), and a one-line health status in your session banner.
+- **Plug-and-play restore**: `ckis-restore.sh` rebuilds everything on a fresh machine/OS; only secrets are re-provisioned by hand (never in backup).
+- **Zero heavy deps** (`bash`, `git`, `jq`, `rsync`, `flock`) and a pure-bash test suite (`bash backup-system/tests/run.sh`).
+
+**Start here:** [`backup-system/README.md`](backup-system/README.md).
+
+---
+
 ## From Execution Plan to Agentic OS — The Evolution
 
 CKIS v1.0 (April 2026) started as a simple answer to one question: *how do you stop a second brain from becoming 822 empty files?*

backup-system/.gitignore

@@ -0,0 +1,3 @@
+ckis-manifest.json
+apparatus/
+*.log

backup-system/README.md

@@ -0,0 +1,98 @@
+# Autonomous Backup System
+
+A small, dependency-light, **self-hosting** backup system for a knowledge base / dotfiles / multi-repo setup. It gives you the **3-2-1 rule** (3 copies, 2 media, 1 off-site), runs itself, and is **plug-and-play** across machines and OSes.
+
+It was built to protect a multi-layer "second brain" (an Obsidian vault + an engineering knowledge base + per-project memory + the machine-level agent config), but the engine is generic: **you describe what to back up in one JSON manifest, and the scripts do the rest.**
+
+> Zero hard dependencies beyond `bash`, `git`, `jq`, `rsync`, `flock` (coreutils). No `yq`, no `bats`, no `gitleaks`, no cloud SDKs — so it restores on a bare machine.
+
+---
+
+## Why
+
+Most "backups" are a static cron job that rots the moment your setup changes. This one is **declarative and self-hosting**: the rules live in a versioned `ckis-manifest.json` that is itself backed up, and discovery is registry-driven, so adding a repo or a tool is **one manifest entry** — never a script edit.
+
+## How it works — three rings of autonomy
+
+1. **Ring 1 — push on event (fast path).** A git hook or session-stop hook calls `ckis-push.sh <repo>`: stage-all → commit-if-dirty → push with retry. Detached, lock-guarded, never blocks you. 100 new files = 1 commit = 1 push. **It fails hard** if a commit is blocked — never reports a stalled backup as success.
+2. **Ring 2 — reconciling safety net (the real-time floor).** A `systemd --user` timer runs `ckis-backup-all.sh` on a short interval (default **15 min**, set in the manifest). Being a *reconcile* rather than an event hook makes it **tool-agnostic** — it captures changes no matter which editor/agent made them, with no `inotify` dependency. It auto-creates missing private remotes, pushes drift, **centralizes every project's memory subdir** (any tool, any visibility) into one private repo, exports the curated config apparatus, sweeps every `.git/config` for embedded credentials, runs a throttled deep secret-audit, and does the physical backup if a drive is mounted. **It exits non-zero on any real failure.**
+3. **Ring 3 — passive visibility.** `ckis-backup-doctor.sh --oneline` prints a one-line health status (`BACKUP ✅ all pushed · physical 2d`) for a shell prompt or session banner. A blocked backup shows **🔴 FAILED** (a real problem), distinct from a benign **⚠** drift.
+
+## Layers it backs up
+
+| Class | Meaning | Destination |
+|---|---|---|
+| `track` | source of truth (text/config/scripts) | private git remote |
+| `regenerable` | derivable from tracked inputs | **excluded**, with a recorded rebuild command |
+| `secret` | credentials/tokens/keys | **never** a remote — encrypted physical disk only |
+| `sensitive` | private now, encrypt later | private remote (flagged for a future git-crypt repo) |
+| `snapshot` | costly binary | physical disk / Git LFS, not the main repo |
+
+## Install
+
+```bash
+git clone https://github.com/YOUR_USER/your-infra-repo.git ~/infra
+cd ~/infra
+cp ckis-manifest.example.json ckis-manifest.json
+$EDITOR ckis-manifest.json            # set github_owner + your targets
+./install.sh                          # symlinks bin/ -> ~/bin, systemd timer, secret-scan hooks
+```
+
+Wire Ring 1 into wherever a "work session" ends (a git `post-commit` hook, an editor hook, or an agent's stop hook):
+```bash
+~/bin/ckis-push "$HOME/path/to/repo" &
+```
+
+## Usage
+
+```bash
+ckis-backup-doctor            # health report (🔴 FAILED = real block, ⚠ = benign drift)
+ckis-backup-all               # force a full run (push drift, centralize, audit, physical)
+ckis-secret-audit             # scan every repo's working tree + .git/config for real secrets
+ckis-backup-physical          # physical backup to a mounted drive (auto-detected)
+```
+
+## Disaster restore (plug-and-play)
+
+On a brand-new machine or OS:
+```bash
+gh auth login                                            # or set up git credentials
+git clone https://github.com/YOUR_USER/your-infra-repo.git ~/infra
+cd ~/infra && ./bin/ckis-restore.sh
+```
+It clones every target to its manifest path, restores the config apparatus, and installs the runtime. **The only manual step is re-providing secrets** — by design, they are never in the backup.
+
+## Mobile (iOS + Android)
+
+Every target is just a git repo, so any mobile git client works. For Obsidian vaults, the **obsidian-git** plugin (`isDesktopOnly:false`) syncs each vault on iOS and Android with a fine-grained PAT — which also sidesteps Obsidian Sync's one-vault limit.
+
+## Design notes & hard-won lessons
+
+- **`$HOME` may itself be a git repo.** A directory without its own `.git` then resolves to the `$HOME` repo, and `git add -A` will try to stage your entire home directory. Every script guards with `is_repo_root` and refuses to operate on a parent repo. *(This one cost an 11-minute CPU hang to find.)*
+- **FAT/exFAT can't store `:` in filenames.** On FAT drives the per-file mirror is skipped; **git bundles** (single files holding full history + all filenames) are the authoritative physical copy.
+- **Heavy/third-party content is `regenerable`, not backed up** — e.g. a skill that bundles a 1 GB headless browser is reinstalled, not committed.
+- **Secrets never leave the machine.** A dependency-free `pre-commit` scanner blocks accidental commits — but it detects **real key material**, not mentions: tokens are gated by Shannon **entropy** (so a `ghp_xxxx…` placeholder or a doc that quotes one passes), PEM keys require an actual base64 body next to the marker, and it also covers **`.git/config` remote-URL credentials** and `class=secret` filenames. A `.ckis-secret-allow` file (or an inline `ckis-allow-secret` marker) sanctions docs/tests that legitimately quote a pattern.
+- **Silent success is the worst failure.** A backup that reports green while doing nothing is more dangerous than a crash. Marker-matching secret scanners false-positive on notes that *document* security work and can wedge a knowledge base's backup for hours; every failure path here exits non-zero and the banner shows 🔴, and `ckis-secret-audit` re-checks the whole system independently of any exit code.
+
+## Layout
+
+```
+bin/      ckis-push, ckis-backup-all, ckis-backup-doctor, ckis-backup-physical,
+          brains-sync, ckis-secret-audit, ckis-apparatus-export, ckis-restore
+lib/      common.sh        (logging, flock, retry, entropy, failure markers,
+                            agent-agnostic brain discovery, git helpers)
+hooks/    pre-commit-secret-scan.sh
+systemd/  ckis-backup.{service,timer}
+tests/    pure-bash test harness (run: bash tests/run.sh) — 12 suites
+install.sh · ckis-manifest.example.json
+```
+
+## Tests
+
+```bash
+bash tests/run.sh     # bash -n + (optional) shellcheck + behavior tests -> GATE: ✅ GREEN
+```
+
+## License
+
+See `LICENSE` in the repository root.

backup-system/bin/brains-sync.sh

@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+# brains-sync.sh
+# CENTRALIZE every per-project .brain/ into one private aggregator workdir,
+# regardless of which coding agent created the project (Claude Code, Codex,
+# Gemini, OpenCode, ...) or the project's own visibility. The CLIs are just
+# projects — there is no CLI-specific or public/private branching here.
+#
+# Discovery is agent-agnostic: the UNION of Dev Brain's registry and a bounded
+# filesystem scan of brain_roots (ckis::brain_repos). Regenerable/redundant
+# content (graph/, .brain-backup-*) is excluded so the aggregate stays high-value.
+set -uo pipefail
+HERE="$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)"
+source "$HERE/../lib/common.sh"
+ckis::init
+
+WORKDIR="${CKIS_AGG_WORKDIR:-$(ckis::expand "$(ckis::manifest '.discovery.aggregate_workdir')")}"
+case "$WORKDIR" in ''|null) ckis::warn "no aggregate_workdir configured — skipping brains-sync"; exit 0 ;; esac
+mkdir -p "$WORKDIR"
+
+EXC=()
+while IFS= read -r e; do [ -n "$e" ] && EXC+=(--exclude "$e"); done \
+  < <(ckis::manifest '.discovery.aggregate_exclude[]?' 2>/dev/null)
+
+synced=0
+while IFS=$'\t' read -r slug repo; do
+  [ -n "$repo" ] || continue
+  brain="$repo/.brain"
+  [ -d "$brain" ] || continue
+  mkdir -p "$WORKDIR/$slug"
+  if rsync -a --delete "${EXC[@]}" "$brain"/ "$WORKDIR/$slug/.brain"/ 2>>"$CKIS_LOG_FILE"; then
+    ckis::info "$slug: .brain centralized"; synced=$((synced+1))
+  else
+    ckis::err "$slug: rsync failed"
+  fi
+done < <(ckis::brain_repos)
+
+ckis::info "aggregator: $synced .brain dir(s) centralized into $WORKDIR"
+exit 0

backup-system/bin/ckis-apparatus-export.sh

@@ -0,0 +1,50 @@
+#!/usr/bin/env bash
+# ckis-apparatus-export.sh [dest-dir]
+# Curated allowlist export of the L0 machine apparatus (~/.claude) into the
+# infra repo's apparatus/ dir. NEVER a raw mirror; deny[] always wins so secrets
+# (.credentials.json, tokens, caches, session blobs) are physically excluded.
+set -uo pipefail
+HERE="$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)"
+source "$HERE/../lib/common.sh"
+ckis::init
+
+SRC="${CKIS_APPARATUS_SRC:-$(ckis::expand "$(ckis::manifest '.apparatus.source')")}"
+DEST="${1:-$CKIS_INFRA_ROOT/$(ckis::manifest '.apparatus.dest')}"
+[ -d "$SRC" ] || { ckis::warn "apparatus source missing, skip: $SRC"; exit 0; }
+mkdir -p "$DEST"
+
+# deny list -> rsync --exclude args (applied to every copy, defense in depth)
+EXC=()
+while IFS= read -r d; do [ -n "$d" ] && EXC+=(--exclude "$d"); done \
+  < <(ckis::manifest '.apparatus.deny[]')
+
+copied=0
+# 1. allowlisted top-level entries
+while IFS= read -r item; do
+  [ -n "$item" ] || continue
+  if [ -e "$SRC/$item" ]; then
+    rsync -a --delete "${EXC[@]}" "$SRC/$item" "$DEST/" 2>>"$CKIS_LOG_FILE" \
+      && { ckis::info "apparatus: exported $item"; copied=$((copied+1)); }
+  fi
+done < <(ckis::manifest '.apparatus.allow[]')
+
+# 2. auto-memory: projects/*/memory (re-included despite projects/ being denied)
+mg="$(ckis::manifest '.apparatus.memory_glob // empty')"
+if [ -n "$mg" ]; then
+  shopt -s nullglob
+  for md in "$SRC"/$mg; do
+    [ -d "$md" ] || continue
+    rel="${md#$SRC/}"
+    mkdir -p "$DEST/$(dirname "$rel")"
+    rsync -a --delete "$md/" "$DEST/$rel/" 2>>"$CKIS_LOG_FILE" \
+      && { ckis::info "apparatus: exported memory $rel"; copied=$((copied+1)); }
+  done
+  shopt -u nullglob
+fi
+
+# 3. hard safety net: never let a credential file survive in the export
+find "$DEST" -name '.credentials.json' -o -name '*.pem' -o -name 'id_rsa*' 2>/dev/null \
+  | while IFS= read -r leak; do rm -f "$leak"; ckis::warn "apparatus: purged stray secret $leak"; done
+
+ckis::info "apparatus export: $copied item(s) -> $DEST"
+exit 0

backup-system/bin/ckis-backup-all.sh

@@ -0,0 +1,142 @@
+#!/usr/bin/env bash
+# ckis-backup-all.sh — Ring 2: the daily safety-net orchestrator.
+# Registry+manifest driven, self-healing, degrades gracefully when a piece is
+# absent. Catches drift from interrupted sessions, edits outside Claude, hook
+# failures and brand-new projects. Idempotent; global lock.
+set -uo pipefail
+HERE="$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)"
+source "$HERE/../lib/common.sh"
+ckis::init
+
+_have_gh() { command -v gh >/dev/null 2>&1 && gh auth status >/dev/null 2>&1; }
+
+_run() {
+  ckis::info "═══ backup-all start ═══"
+  local failures=0
+  local scan="$HERE/../hooks/pre-commit-secret-scan.sh"
+
+  # 1. Export L0 apparatus into the infra repo (so the next push captures it).
+  local app_src; app_src="$(ckis::manifest '.apparatus.source // empty')"
+  if [ -n "$app_src" ] && [ -d "$(ckis::expand "$app_src")" ]; then
+    bash "$HERE/ckis-apparatus-export.sh" || ckis::warn "apparatus export had issues"
+  fi
+
+  # 2. Every target: self-heal missing remote, then push drift.
+  while IFS=$'\t' read -r slug path remote class kind; do
+    [ -d "$path" ] || { ckis::warn "target missing on disk: $slug ($path)"; continue; }
+    ckis::is_repo "$path" || { ckis::warn "target not a git repo: $slug"; continue; }
+    if ! ckis::repo_has_remote "$path" && [ -n "$remote" ] && [ "$remote" != "null" ]; then
+      if _have_gh; then
+        ckis::info "$slug: no remote — auto-creating private $remote"
+        gh repo create "$remote" --private --source="$path" --remote=origin >>"$CKIS_LOG_FILE" 2>&1 \
+          || ckis::err "$slug: auto-create failed"
+      else
+        ckis::warn "$slug: no remote and gh unavailable — committing locally only"
+      fi
+    fi
+    bash "$HERE/ckis-push.sh" "$path" || { ckis::warn "$slug: push reported an error"; failures=$((failures+1)); }
+  done < <(ckis::targets)
+
+  # 3. Centralize every project .brain/ (any agent, any visibility), then push.
+  local reg; reg="$(ckis::expand "$(ckis::manifest '.discovery.registry // empty')")"
+  bash "$HERE/brains-sync.sh" || ckis::warn "brains-sync had issues"
+  local agg_dir agg_remote
+  agg_dir="$(ckis::expand "$(ckis::manifest '.discovery.aggregate_workdir // empty')")"
+  agg_remote="$(ckis::manifest '.discovery.brain_aggregator // empty')"
+  if [ -n "$agg_dir" ] && [ "$agg_dir" != "null" ] && [ -d "$agg_dir" ]; then
+    # Ensure the aggregator has its OWN .git (a dir without one would resolve to
+    # a parent repo such as $HOME and add the whole home tree).
+    if ! ckis::is_repo_root "$agg_dir"; then
+      git -C "$agg_dir" init -q
+      git -C "$agg_dir" symbolic-ref HEAD refs/heads/master 2>/dev/null || true
+      ckis::info "aggregator: initialized own git repo at $agg_dir"
+    fi
+    # The aggregator holds private brains too, so it MUST be secret-scanned on
+    # commit just like every other backup target.
+    if [ ! -e "$agg_dir/.git/hooks/pre-commit" ]; then
+      ln -sf "$HERE/../hooks/pre-commit-secret-scan.sh" "$agg_dir/.git/hooks/pre-commit" 2>/dev/null || true
+    fi
+    if ! ckis::repo_has_remote "$agg_dir" && [ -n "$agg_remote" ] && _have_gh; then
+      if ! gh repo view "$agg_remote" >/dev/null 2>&1; then
+        gh repo create "$agg_remote" --private >>"$CKIS_LOG_FILE" 2>&1 || ckis::warn "aggregator: repo auto-create failed"
+      fi
+      git -C "$agg_dir" remote add origin "https://github.com/$agg_remote.git" 2>/dev/null || true
+    fi
+    bash "$HERE/ckis-push.sh" "$agg_dir" || { ckis::warn "aggregator push had issues"; failures=$((failures+1)); }
+  fi
+
+  # 3.5 Self-healing wiring + secret blind-spot sweep across every discovered
+  #     brain repo (any agent). Ensures new projects auto-get secret-scan
+  #     protection, and catches credentials embedded in any repo's .git/config
+  #     (the staged-content hook structurally cannot see those).
+  # .gitcfg marker namespace is re-evaluated every run (set if a token is found,
+  # cleared when fixed) — kept separate from push markers, which a successful
+  # push clears even though the embedded token would still be there.
+  _sweep_cfg() {  # slug repo
+    if bash "$scan" --git-config "$2" >/dev/null 2>&1; then
+      ckis::clear_fail "$1.gitcfg"
+    else
+      ckis::err "$1: embedded credential in .git/config"
+      ckis::mark_fail "$1.gitcfg" "embedded credential in .git/config"
+      failures=$((failures+1))
+    fi
+  }
+  while IFS=$'\t' read -r slug path remote class kind; do
+    [ -d "$path" ] || continue
+    _sweep_cfg "$slug" "$path"
+  done < <(ckis::targets)
+  while IFS=$'\t' read -r slug repo; do
+    [ -d "$repo" ] || continue
+    if [ ! -e "$repo/.git/hooks/pre-commit" ]; then
+      ln -sf "$scan" "$repo/.git/hooks/pre-commit" 2>/dev/null \
+        && ckis::info "$slug: secret-scan pre-commit auto-installed"
+    fi
+    _sweep_cfg "$slug" "$repo"
+  done < <(ckis::brain_repos)
+
+  # 3.6 Deep secret audit (full content scan of every repo), throttled to ~daily.
+  #     The universal net for repos whose own pre-commit we don't control (CLIs
+  #     with a build/supply-chain hook, etc.): a secret committed anywhere is
+  #     detected within a day and surfaces as a hard failure, even without our
+  #     pre-commit installed there. Cheap to amortize over a day.
+  local audit_marker="$CKIS_LOG_DIR/last-audit" now last_audit=0
+  now="$(date +%s)"
+  [ -f "$audit_marker" ] && last_audit="$(cat "$audit_marker" 2>/dev/null || echo 0)"
+  case "$last_audit" in ''|*[!0-9]*) last_audit=0 ;; esac
+  if [ "$(( now - last_audit ))" -ge 86400 ]; then
+    if bash "$HERE/ckis-secret-audit.sh" >>"$CKIS_LOG_FILE" 2>&1; then
+      ckis::clear_fail "secret-audit"; printf '%s' "$now" >"$audit_marker"
+    else
+      ckis::err "deep secret audit FOUND secret material — see log"
+      ckis::mark_fail "secret-audit" "deep audit found secret material"
+      failures=$((failures+1))
+    fi
+  fi
+
+  # 4. Physical backup if a drive is mounted (best-effort).
+  if bash "$HERE/ckis-backup-physical.sh" >>"$CKIS_LOG_FILE" 2>&1; then
+    ckis::info "physical backup ran"
+  else
+    ckis::info "physical backup skipped (no drive mounted)"
+  fi
+
+  # 5. Health summary.
+  ckis::info "health: $(bash "$HERE/ckis-backup-doctor.sh" --oneline)"
+
+  # 6. HARD-FAIL signal. Exit non-zero if anything failed this run OR a prior
+  #    unresolved failure marker persists, so systemd shows failure (not the old
+  #    silent "success") and the SessionStart banner screams. This is the fix for
+  #    the 24h-silent-outage: a blocked backup can no longer masquerade as green.
+  local lingering; lingering="$(ckis::list_fails | tr '\n' ' ')"
+  if [ "$failures" -gt 0 ] || [ -n "${lingering// /}" ]; then
+    ckis::err "═══ backup-all done WITH FAILURES (this run: $failures · unresolved: ${lingering:-none}) ═══"
+    return 1
+  fi
+  ckis::info "═══ backup-all done ═══"
+  return 0
+}
+
+ckis::with_lock "backup-all" _run
+rc=$?
+[ "$rc" = "75" ] && { ckis::info "backup-all already running, skipped"; exit 0; }
+exit "$rc"