From d1407180829ad79cd04c397d1cc7c8d2ce110c36 Mon Sep 17 00:00:00 2001 From: Eduardo Borjas Date: Fri, 19 Jun 2026 23:39:04 -0600 Subject: [PATCH] =?UTF-8?q?feat:=20autonomous=20context=20maintenance=20?= =?UTF-8?q?=E2=80=94=20reflux=20engine=20+=20qmd=20auto-reindex=20(v2.4.0)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reflux engine (ckis-reflux-all): keeps the canonical heart docs (_MEMORY/ _ACTIVE-PROJECTS/_INTERESTS/_PROFILE) fresh from accumulated session activity via a headless model call. Propose-only by default โ€” drafts an update+diff into a quarantine queue, never overwrites a live doc unless a human runs --apply. Structural guards (created immutable, no heading dropped, no shrink, line cap, secret-scan) reject any unsafe proposal. Optional systemd --user timer, cadence-throttled. qmd auto-reindex: backup reconcile refreshes the local BM25 search index each run (optional search_index manifest block); best-effort, never fails a backup. Both are optional declarative manifest blocks. 14-test test_reflux.sh (stub model, no network); full backup-system suite green. ๐Ÿค– Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 --- CHANGELOG.md | 14 + README.md | 11 + backup-system/bin/ckis-backup-all.sh | 19 ++ backup-system/bin/ckis-reflux-all.sh | 320 ++++++++++++++++++++++ backup-system/ckis-manifest.example.json | 31 ++- backup-system/install.sh | 16 ++ backup-system/systemd/ckis-backup.service | 0 backup-system/systemd/ckis-backup.timer | 0 backup-system/systemd/ckis-reflux.service | 16 ++ backup-system/systemd/ckis-reflux.timer | 16 ++ backup-system/tests/test_reflux.sh | 159 +++++++++++ 11 files changed, 600 insertions(+), 2 deletions(-) create mode 100755 backup-system/bin/ckis-reflux-all.sh mode change 100644 => 100755 backup-system/systemd/ckis-backup.service mode change 100644 => 100755 backup-system/systemd/ckis-backup.timer create mode 100755 backup-system/systemd/ckis-reflux.service create mode 100755 backup-system/systemd/ckis-reflux.timer create mode 100644 backup-system/tests/test_reflux.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index add9053..48d753d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,20 @@ Format: `[vX.Y.Z] YYYY-MM-DD โ€” Description` --- +## v2.4.0 โ€” Autonomous Context Maintenance: reflux engine + qmd auto-reindex (2026-06-19) + +### New +- **Reflux engine** (`backup-system/bin/ckis-reflux-all.sh`) โ€” autonomous context-maintenance that keeps the canonical "heart docs" (`_MEMORY` / `_ACTIVE-PROJECTS` / `_INTERESTS` / `_PROFILE`) fresh from accumulated session activity via a headless model call (`claude -p` or any equivalent). **Propose-only by default**: drafts an update + diff into a quarantine queue and never overwrites a live doc unless a human runs `ckis-reflux-all --apply ` (or sets a doc's `auto_apply=true`, off by default). Every proposal passes structural guards โ€” frontmatter `created` immutable, no section heading dropped, no shrink below `min_retain_pct`, line cap, secret-scan โ€” or it's rejected, never queued. Optional `systemd --user` timer runs it daily, cadence-throttled (per-doc + skip-if-pending) so it's never noisy; pending proposals surface in the session banner. Declarative `reflux` block in the manifest; remove it to disable. +- **qmd auto-reindex** โ€” the backup reconcile now refreshes the local qmd BM25 search index on every run (optional `search_index` manifest block), so `qmd search` stays fresh for edits made outside any agent (Obsidian desktop/mobile, direct edits). Best-effort: a stale index never fails a backup. + +### Changed +- `backup-system/install.sh` wires the optional reflux timer alongside the backup timer (only when a `reflux` block exists). `ckis-backup-all.sh` gains the best-effort search-index refresh step. `ckis-manifest.example.json` documents both new blocks. README: new "Autonomous Context Maintenance" section. + +### Tests +- `backup-system/tests/test_reflux.sh` โ€” 14 tests, no model call (stub), proving the structural guards reject every unsafe proposal, the live doc is never touched on propose, and `--apply` re-checks guards. Full suite green. + +--- + ## v2.3.25 โ€” Vault-Wide Routing-Table Audit + qmd Local Search (2026-06-18) ### New diff --git a/README.md b/README.md index d443001..fcc690f 100644 --- a/README.md +++ b/README.md @@ -247,6 +247,17 @@ A knowledge base with no backup is a liability. CKIS ships a **self-hosting, dec --- +## Autonomous Context Maintenance (optional) + +The same `backup-system/` ships two optional, declarative autonomy layers โ€” add one manifest block, no extra code: + +- **Search-index auto-refresh** (`search_index` block) โ€” if you use [qmd](https://github.com/tobi/qmd) for local vault search, the backup reconcile re-indexes on every run (and a session-end hook does it instantly), so `qmd search` stays fresh for edits made in Obsidian, mobile, or any agent. Best-effort: a stale index never fails a backup. +- **Reflux engine** (`reflux` block + `ckis-reflux-all`) โ€” keeps your canonical "heart docs" (`_MEMORY` / `_ACTIVE-PROJECTS` / `_INTERESTS` / `_PROFILE`) fresh from accumulated session activity via a headless model call (e.g. `claude -p`). **Propose-only by default**: it drafts an update + diff into a quarantine queue and *never* overwrites a live doc until you run `ckis-reflux-all --apply `. Every proposal passes structural guards (frontmatter `created` immutable, no section heading dropped, no shrink below a threshold, line cap, secret-scan) or it's rejected. A `systemd --user` timer runs it daily, cadence-throttled so it's never noisy; pending proposals surface in your session banner. This closes the gap where a vault grows for months while its canonical state files quietly go stale. + +Both are off until you add the block; remove the block to disable. See [`backup-system/ckis-manifest.example.json`](backup-system/ckis-manifest.example.json). + +--- + ## From Execution Plan to Agentic OS โ€” The Evolution CKIS v1.0 (April 2026) started as a simple answer to one question: *how do you stop a second brain from becoming 822 empty files?* diff --git a/backup-system/bin/ckis-backup-all.sh b/backup-system/bin/ckis-backup-all.sh index ce78456..992ea6b 100755 --- a/backup-system/bin/ckis-backup-all.sh +++ b/backup-system/bin/ckis-backup-all.sh @@ -120,6 +120,25 @@ _run() { ckis::info "physical backup skipped (no drive mounted)" fi + # 4.5 Refresh the local search index (best-effort, agent-agnostic floor). + # OPTIONAL (manifest .search_index). The qmd BM25 index is regenerable and + # lives outside any repo, so it is never a backup TARGET โ€” but it must stay + # fresh for edits made outside any coding agent (Obsidian desktop/mobile, + # direct edits). A stale index is NOT a backup failure: this NEVER touches + # $failures. Skipped cleanly if absent. + local idx_cmd idx_bin + idx_cmd="$(ckis::manifest '.search_index.refresh_cmd // empty')" + idx_bin="$(ckis::manifest '.search_index.binary // empty')" + if [ -n "$idx_cmd" ] && [ -n "$idx_bin" ] && command -v "$idx_bin" >/dev/null 2>&1; then + if $idx_cmd >>"$CKIS_LOG_FILE" 2>&1; then + ckis::info "search index refreshed ($idx_cmd)" + else + ckis::warn "search index refresh had issues (non-fatal): $idx_cmd" + fi + else + ckis::info "search index refresh skipped (not configured or binary absent)" + fi + # 5. Health summary. ckis::info "health: $(bash "$HERE/ckis-backup-doctor.sh" --oneline)" diff --git a/backup-system/bin/ckis-reflux-all.sh b/backup-system/bin/ckis-reflux-all.sh new file mode 100755 index 0000000..2793ccd --- /dev/null +++ b/backup-system/bin/ckis-reflux-all.sh @@ -0,0 +1,320 @@ +#!/usr/bin/env bash +# ckis-reflux-all.sh โ€” autonomous context-maintenance (reflux) engine. +# +# Keeps the vault's canonical "heart docs" (_MEMORY/_ACTIVE-PROJECTS/_INTERESTS/ +# _PROFILE) fresh from accumulated session activity, so a vault can grow for +# months without its canonical state files quietly going stale. Configured by the +# `reflux` block in ckis-manifest.json; remove that block to disable. +# +# SAFETY MODEL โ€” propose-before-apply: +# * Default run PROPOSES only. It writes a proposed rewrite + unified diff into +# the quarantine queue (manifest .reflux.queue_subdir) and NEVER overwrites a +# live heart doc. +# * A proposal is accepted into the queue ONLY if it passes ALL structural +# guards: frontmatter `created` immutable ยท no `#` section heading dropped ยท +# line count >= min_retain_pct of current ยท <= max_lines ยท secret-scan clean. +# A proposal failing any guard is rejected and logged, never queued. +# * Applying to the live file is a separate, deliberate act: `--apply ` +# (a human promotes a reviewed proposal) OR a doc's manifest auto_apply=true +# (kept false until trust is established). +# +# Usage: +# ckis-reflux-all propose for every due doc (recurring/self-digest) +# ckis-reflux-all --doc memory propose for one doc +# ckis-reflux-all --digest FILE use a provided digest (Stage-3 backfill via workers) +# ckis-reflux-all --apply QUEUEF re-check guards on QUEUEF's proposal, apply to live +# +# Idempotent; global lock. Never exits the parent shell. Best-effort model call. +set -uo pipefail +HERE="$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)" +source "$HERE/../lib/common.sh" +ckis::init + +SCAN="$HERE/../hooks/pre-commit-secret-scan.sh" +TODAY="$(date +%Y-%m-%d)" + +# โ”€โ”€ helpers โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +_vault() { ckis::expand "$(ckis::manifest '.reflux.vault // empty')"; } + +# frontmatter `created:` value (empty if none) โ€” used by the immutability guard. +_created_of() { awk -F': *' '/^created:/{print $2; exit}' "$1" 2>/dev/null | tr -d ' \r'; } + +# All heading lines (#..######) of a file, normalized โ€” used by the no-drop guard. +_headings_of() { grep -E '^#{1,6} ' "$1" 2>/dev/null | sed 's/[[:space:]]*$//'; } + +# Strip a leading/trailing ```markdown / ``` fence the model may add despite +# instructions, so the proposal is the raw file content. +_defence() { + awk 'NR==1 && /^```/{next} {buf[NR]=$0} END{ + last=NR; if(buf[last] ~ /^```[[:space:]]*$/) last-- + for(i=1;i<=last;i++) if(i in buf) print buf[i] + }' "$1" +} + +# Build a bounded "recent activity" digest from session logs + Dev Brain index + +# vault git log. Used when no --digest is supplied (the recurring path). Capped. +_build_digest() { + local vault="$1" days logs db + days="$(ckis::manifest '.reflux.sources.recent_days // 21')" + logs="$vault/$(ckis::manifest '.reflux.sources.session_logs // "01-daily/logs"')" + db="$(ckis::expand "$(ckis::manifest '.reflux.sources.dev_brain_sessions // empty')")" + + echo "## Vault commits (last ${days}d, non-auto)" + git -C "$vault" log --since="${days} days ago" --format='- %ad %s' --date=short 2>/dev/null \ + | grep -v 'ckis-backup: auto' | head -40 + + echo "" + echo "## Recent session-log summaries" + if [ -d "$logs" ]; then + # newest few daily logs, their tail (where the session summary lives) + find "$logs" -maxdepth 1 -name '*.md' -mtime "-${days}" 2>/dev/null \ + | sort | tail -8 | while read -r f; do + echo "### $(basename "$f")" + tail -25 "$f" 2>/dev/null + done + fi + + echo "" + echo "## Dev Brain sessions (recent)" + [ -f "$db" ] && tail -25 "$db" 2>/dev/null +} + +# Structural guards. Args: live proposal max_lines min_pct +# Returns 0 if ALL pass; otherwise prints the failing guard(s) and returns 1. +_guards() { + local live="$1" prop="$2" max="$3" pct="$4" fail=0 + local lc pc need + + # 1. frontmatter created immutable + if [ "$(_created_of "$live")" != "$(_created_of "$prop")" ]; then + ckis::warn "guard FAIL created-immutable: '$(_created_of "$live")' -> '$(_created_of "$prop")'"; fail=1 + fi + + # 2. no section heading dropped โ€” multiset comparison (NOT sort -u), so dropping + # one of two identical headings is still caught. `diff` reports a '<' line for + # any live-heading instance not matched by a proposal instance. + local dropped + dropped="$(diff <(_headings_of "$live" | sort) <(_headings_of "$prop" | sort) 2>/dev/null | grep -E '^< ' || true)" + if [ -n "$dropped" ]; then + ckis::warn "guard FAIL heading-dropped: $(printf '%s' "$dropped" | tr '\n' '|')"; fail=1 + fi + + # 3. line count not below min_retain_pct of live + lc="$(wc -l < "$live")"; pc="$(wc -l < "$prop")" + need=$(( lc * pct / 100 )) + if [ "$pc" -lt "$need" ]; then + ckis::warn "guard FAIL shrink: proposal ${pc} lines < ${pct}% of ${lc} (=${need})"; fail=1 + fi + + # 4. <= max_lines (if a cap is set) + if [ -n "$max" ] && [ "$max" != "null" ] && [ "$pc" -gt "$max" ]; then + ckis::warn "guard FAIL max-lines: proposal ${pc} > cap ${max}"; fail=1 + fi + + # 5. secret-scan clean โ€” FAIL-CLOSED: if the scanner is missing, reject rather + # than silently skip a security guard (a moved/renamed hook must not disable + # secret detection on the propose OR apply path). + if [ -f "$SCAN" ]; then + if ! bash "$SCAN" --files "$prop" >/dev/null 2>&1; then + ckis::warn "guard FAIL secret-scan: proposal contains secret material"; fail=1 + fi + else + ckis::warn "guard FAIL secret-scan: scanner missing at $SCAN (fail-closed)"; fail=1 + fi + + return $fail +} + +# Produce a proposal for one doc via the model. Args: live max_lines digestfile out +# Returns 0 and writes the de-fenced proposal to $out on success. +_propose() { + local live="$1" max="$2" digest="$3" out="$4" + local model raw prompt + model="$(ckis::manifest '.reflux.model_cmd // "claude -p"')" + + prompt="You are updating a canonical state file in a personal knowledge vault. +Output ONLY the complete updated markdown file content โ€” no preamble, no code fences, no commentary. + +HARD RULES: +- Preserve the YAML frontmatter exactly, EXCEPT set 'modified:' to ${TODAY}. NEVER change 'created:'. +- Keep EVERY existing section heading (every line starting with #). You may revise content under a heading but must not remove the heading itself. +- Stay at or under ${max:-150} lines total. Be concise; prefer updating existing bullets over adding new sections. +- Update state to reflect the RECENT ACTIVITY below. Do NOT invent facts it doesn't support. When unsure, keep the existing line. +- Never include secrets, API keys, or tokens. + +=== CURRENT FILE === +$(cat "$live") + +=== RECENT ACTIVITY (newest first) === +$(cat "$digest") + +=== END. Output the full updated file now (frontmatter first, no fences): ===" + + # Model call is best-effort; capture to a temp, then de-fence into $out. + # stdin MUST come from /dev/null: `claude -p` reads stdin, and this function + # runs inside a `while read` loop fed by process substitution โ€” without this + # redirect the model swallows the loop's input stream and only the first doc + # gets processed. + local tmp; tmp="$(mktemp)" + if ! $model "$prompt" "$tmp" 2>>"$CKIS_LOG_FILE"; then + rm -f "$tmp"; return 1 + fi + [ -s "$tmp" ] || { rm -f "$tmp"; return 1; } + _defence "$tmp" >"$out" + rm -f "$tmp" + [ -s "$out" ] +} + +# โ”€โ”€ modes โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +# Apply a reviewed queued proposal to its live target (re-running guards first). +_apply() { + local qfile="$1" + [ -f "$qfile" ] || { ckis::err "apply: queue file not found: $qfile"; return 1; } + local slug; slug="$(awk -F': *' '/^reflux-slug:/{print $2; exit}' "$qfile" | tr -d ' \r')" + local vault; vault="$(_vault)" + local rel max; rel="$(ckis::manifest ".reflux.docs[] | select(.slug==\"$slug\") | .path")" + max="$(ckis::manifest ".reflux.docs[] | select(.slug==\"$slug\") | .max_lines")" + [ -n "$rel" ] || { ckis::err "apply: unknown slug '$slug' in $qfile"; return 1; } + local live="$vault/$rel" pct; pct="$(ckis::manifest '.reflux.min_retain_pct // 70')" + + # The queue file stores the proposal body after a '---PROPOSAL---' marker. + local body; body="$(mktemp)" + awk 'f{print} /^---PROPOSAL---$/{f=1}' "$qfile" >"$body" + + if ! _guards "$live" "$body" "$max" "$pct"; then + ckis::err "apply ABORTED: proposal failed guards (live untouched): $qfile" + rm -f "$body"; return 1 + fi + cp "$body" "$live" + rm -f "$body" + ckis::info "applied reflux proposal โ†’ $rel ($slug)" +} + +# Propose for one doc. Args: slug rel max digestfile +_propose_doc() { + local slug="$1" rel="$2" max="$3" digest="$4" + local vault; vault="$(_vault)" + local live="$vault/$rel" + local pct; pct="$(ckis::manifest '.reflux.min_retain_pct // 70')" + local qdir; qdir="$vault/$(ckis::manifest '.reflux.queue_subdir // "00-inbox/_REFLUX-QUEUE"')" + [ -f "$live" ] || { ckis::warn "reflux: live doc missing, skip: $rel"; return 0; } + mkdir -p "$qdir" + + local prop; prop="$(mktemp)" + if ! _propose "$live" "$max" "$digest" "$prop"; then + ckis::warn "reflux: model produced no proposal for $slug (skip)"; rm -f "$prop"; return 1 + fi + + if ! _guards "$live" "$prop" "$max" "$pct"; then + ckis::err "reflux: proposal REJECTED by guards, not queued: $slug" + ckis::mark_fail "reflux-$slug" "proposal failed structural guards" + rm -f "$prop"; return 1 + fi + ckis::clear_fail "reflux-$slug" + + local ts qfile; ts="$(date +%Y%m%d-%H%M%S)"; qfile="$qdir/${slug}-${ts}.md" + { + echo "---" + echo "type: reflux-proposal" + echo "reflux-slug: $slug" + echo "target: $rel" + echo "generated: $(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo "status: pending-review" + echo "---" + echo "" + echo "# Reflux proposal โ€” \`$rel\`" + echo "" + echo "> PROPOSE-ONLY. The live file is unchanged. Review the diff below; to apply:" + echo "> \`ckis-reflux-all --apply \"$qfile\"\` (re-checks guards, then overwrites the live file)." + echo "" + echo "## Diff (live โ†’ proposed)" + echo '```diff' + diff -u "$live" "$prop" | sed '1,2d' || true + echo '```' + echo "" + echo "---PROPOSAL---" + cat "$prop" + } >"$qfile" + rm -f "$prop" + ckis::info "reflux proposal queued (live untouched): $qfile" + printf '%s\n' "$qfile" +} + +# Is a doc "due" for a proposal? (used by --due-only, the autonomous timer mode.) +# Returns 0 = due, 1 = skip. Skips when a pending proposal already exists (don't +# pile up) OR the live doc was refreshed within its cadence window (still fresh). +_due() { + local slug="$1" rel="$2" cadence="$3" + local vault qdir; vault="$(_vault)" + qdir="$vault/$(ckis::manifest '.reflux.queue_subdir // "00-inbox/_REFLUX-QUEUE"')" + # a pending proposal already awaits review โ†’ skip + ls "$qdir/${slug}-"*.md >/dev/null 2>&1 && return 1 + # live modified within cadence window โ†’ still fresh, skip + local days=1; [ "$cadence" = "weekly" ] && days=7 + local m mod_e now + m="$(awk -F': *' '/^modified:/{print $2; exit}' "$vault/$rel" 2>/dev/null | tr -d ' \r')" + [ -n "$m" ] || return 0 + mod_e="$(date -d "$m" +%s 2>/dev/null)" || return 0 + now="$(date +%s)" + [ "$(( (now - mod_e) / 86400 ))" -lt "$days" ] && return 1 + return 0 +} + +_run() { + local only_doc="" digest_override="" apply_file="" due_only=0 + while [ $# -gt 0 ]; do + case "$1" in + --doc) only_doc="$2"; shift 2;; + --digest) digest_override="$2"; shift 2;; + --apply) apply_file="$2"; shift 2;; + --due-only) due_only=1; shift;; + *) ckis::warn "unknown arg: $1"; shift;; + esac + done + + if [ -n "$apply_file" ]; then _apply "$apply_file"; return $?; fi + + ckis::info "โ•โ•โ• reflux start (propose-only) โ•โ•โ•" + local vault; vault="$(_vault)" + [ -d "$vault" ] || { ckis::err "reflux: vault not found: $vault"; return 1; } + + # Build (or reuse) the digest once for this run. + local digest cleanup_digest=0 + if [ -n "$digest_override" ]; then + digest="$digest_override" + [ -f "$digest" ] || { ckis::err "reflux: --digest file not found: $digest"; return 1; } + else + digest="$(mktemp)"; cleanup_digest=1 + _build_digest "$vault" >"$digest" + fi + + local proposed=0 rejected=0 skipped=0 + while IFS=$'\t' read -r slug rel max auto cadence; do + [ -n "$only_doc" ] && [ "$slug" != "$only_doc" ] && continue + if [ "$due_only" = 1 ] && ! _due "$slug" "$rel" "$cadence"; then + skipped=$((skipped+1)); continue + fi + if _propose_doc "$slug" "$rel" "$max" "$digest" >/dev/null; then + proposed=$((proposed+1)) + else + rejected=$((rejected+1)) + fi + done < <(ckis::manifest '.reflux.docs[] | [.slug, .path, (.max_lines|tostring), (.auto_apply|tostring), .cadence] | @tsv') + + [ "$cleanup_digest" = 1 ] && rm -f "$digest" + ckis::info "โ•โ•โ• reflux done โ€” proposed:$proposed rejected:$rejected skipped:$skipped (all live docs untouched) โ•โ•โ•" + + # A rejected proposal is a quality signal, not a backup failure: surface via a + # marker (doctor can show it) but do NOT exit non-zero on its own here. + return 0 +} + +# Only run when executed directly; sourcing (e.g. tests) gets the functions only. +if [ "${BASH_SOURCE[0]}" = "$0" ]; then + ckis::with_lock "reflux-all" _run "$@" + rc=$? + [ "$rc" = "75" ] && { ckis::info "reflux-all already running, skipped"; exit 0; } + exit "$rc" +fi diff --git a/backup-system/ckis-manifest.example.json b/backup-system/ckis-manifest.example.json index 06447f9..bb1a40f 100644 --- a/backup-system/ckis-manifest.example.json +++ b/backup-system/ckis-manifest.example.json @@ -22,9 +22,36 @@ }, "schedule": { - "_doc": "Cadence of the reconciling safety-net timer (the agent-agnostic real-time floor; no inotify dependency). reconcile_interval is templated into the systemd unit by install.sh.", + "_doc": "Cadence of the reconciling safety-net timer (the agent-agnostic real-time floor; no inotify dependency). reconcile_interval is templated into the systemd unit by install.sh. reflux_* drives the optional context-maintenance timer.", "reconcile_interval": "15min", - "boot_delay": "3min" + "boot_delay": "3min", + "reflux_interval": "1d", + "reflux_boot_delay": "10min" + }, + + "search_index": { + "_doc": "OPTIONAL. Local BM25 search index (qmd, https://github.com/tobi/qmd) over the vault. NOT a backup target โ€” the index is regenerable and lives outside any repo. This is the agent-agnostic refresh floor: re-index on every reconcile so search stays fresh even for edits made outside any coding agent (Obsidian desktop/mobile, direct edits). Best-effort and incremental: a stale index is NOT a backup failure, so this step never flips the run red. Skipped cleanly when the binary is absent. refresh_cmd is run via word-splitting, so it MUST be a plain space-separated command with NO shell metacharacters; binary is the first token, checked on PATH before running. Remove this block to disable.", + "refresh_cmd": "qmd update", + "binary": "qmd" + }, + + "reflux": { + "_doc": "OPTIONAL. Autonomous context-maintenance engine (ckis-reflux-all). Keeps your vault's canonical 'heart docs' fresh from accumulated session activity via a headless model call (model_cmd). PROPOSE-ONLY by default: it writes a proposal + diff into queue_subdir (a quarantine) and NEVER overwrites a live doc unless a human runs `ckis-reflux-all --apply ` OR a doc's auto_apply=true (keep false until trust is established โ€” structural guards cannot detect content 'hollowing', so a human diff review is the real backstop). Every proposal must pass the structural guards (frontmatter `created` immutable; no section heading dropped; line count >= min_retain_pct of current; <= max_lines; secret-scan clean) or it is rejected, never queued. model_cmd is any headless CLI that reads a prompt arg and prints the updated markdown (e.g. `claude -p`). Remove this block to disable the reflux engine + timer.", + "vault": "~/Documents/Notes", + "queue_subdir": "00-inbox/_REFLUX-QUEUE", + "model_cmd": "claude -p", + "min_retain_pct": 70, + "sources": { + "session_logs": "01-daily/logs", + "recent_days": 21 + }, + "_auto_apply_warning": "DO NOT set any doc's auto_apply=true without first adding a SEMANTIC guard. The structural guards cannot detect content 'hollowing' (a proposal that keeps all headings + line count but guts the substance). Propose-only + human diff review is the real backstop.", + "docs": [ + { "slug": "memory", "path": "00-inbox/_MEMORY.md", "max_lines": 150, "cadence": "daily", "auto_apply": false }, + { "slug": "active-projects", "path": "00-inbox/_ACTIVE-PROJECTS.md", "max_lines": 120, "cadence": "daily", "auto_apply": false }, + { "slug": "interests", "path": "00-inbox/_INTERESTS.md", "max_lines": 100, "cadence": "weekly", "auto_apply": false }, + { "slug": "profile", "path": "00-inbox/_PROFILE.md", "max_lines": 120, "cadence": "weekly", "auto_apply": false } + ] }, "secret_scan": { diff --git a/backup-system/install.sh b/backup-system/install.sh index 2cb2ff1..e21c5b9 100755 --- a/backup-system/install.sh +++ b/backup-system/install.sh @@ -41,6 +41,22 @@ if [ "${CKIS_NO_SYSTEMD:-0}" != "1" ] && command -v systemctl >/dev/null 2>&1; t else ckis::warn "could not enable systemd timer (headless? run: systemctl --user enable --now ckis-backup.timer)" fi + + # 2b. reflux timer (OPTIONAL autonomous context maintenance; only if configured) + if [ -f "$HERE/systemd/ckis-reflux.timer" ] && [ "$(ckis::manifest '.reflux // empty')" != "" ]; then + R_INTERVAL="$(ckis::manifest '.schedule.reflux_interval // "1d"')" + R_BOOTDELAY="$(ckis::manifest '.schedule.reflux_boot_delay // "10min"')" + sed "s|@BIN@|$BIN_DIR|g" "$HERE/systemd/ckis-reflux.service" >"$SYSTEMD_DIR/ckis-reflux.service" + sed -e "s|@INTERVAL@|$R_INTERVAL|g" -e "s|@BOOTDELAY@|$R_BOOTDELAY|g" \ + "$HERE/systemd/ckis-reflux.timer" >"$SYSTEMD_DIR/ckis-reflux.timer" + systemctl --user daemon-reload 2>/dev/null || true + if systemctl --user enable --now ckis-reflux.timer 2>/dev/null; then + systemctl --user restart ckis-reflux.timer 2>/dev/null || true + ckis::info "reflux timer enabled (propose-only, every $R_INTERVAL)" + else + ckis::warn "could not enable reflux timer (headless? run: systemctl --user enable --now ckis-reflux.timer)" + fi + fi fi # 3. secret-scan pre-commit into each target repo (don't clobber existing hooks) diff --git a/backup-system/systemd/ckis-backup.service b/backup-system/systemd/ckis-backup.service old mode 100644 new mode 100755 diff --git a/backup-system/systemd/ckis-backup.timer b/backup-system/systemd/ckis-backup.timer old mode 100644 new mode 100755 diff --git a/backup-system/systemd/ckis-reflux.service b/backup-system/systemd/ckis-reflux.service new file mode 100755 index 0000000..be774fd --- /dev/null +++ b/backup-system/systemd/ckis-reflux.service @@ -0,0 +1,16 @@ +[Unit] +Description=CKIS autonomous context reflux (heart-doc maintenance, propose-only) +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +# claude lives in ~/.local/bin and the engine in ~/bin โ€” a systemd --user unit +# starts with a minimal PATH, so set one that finds both (the model call needs it). +Environment=PATH=%h/.local/bin:%h/bin:/usr/local/bin:/usr/bin:/bin +# --due-only honors per-doc cadence + skips docs with a pending proposal, so a +# daily fire is not noisy. Propose-only: never overwrites a live heart doc. +ExecStart=@BIN@/ckis-reflux-all --due-only +TimeoutStartSec=900 +Nice=15 +IOSchedulingClass=idle diff --git a/backup-system/systemd/ckis-reflux.timer b/backup-system/systemd/ckis-reflux.timer new file mode 100755 index 0000000..ae4213a --- /dev/null +++ b/backup-system/systemd/ckis-reflux.timer @@ -0,0 +1,16 @@ +[Unit] +Description=CKIS autonomous context reflux โ€” heart-doc freshness +Documentation=https://github.com/aedneth/ckis + +[Timer] +# Cadence templated from ckis-manifest.json (.schedule.reflux_*) by install.sh. +# Propose-only + per-doc cadence throttle (--due-only) means a daily fire just +# tops up the review queue when a doc is genuinely due. Persistent catches missed +# runs after sleep/suspend; jitter avoids thundering herds. +OnBootSec=@BOOTDELAY@ +OnUnitActiveSec=@INTERVAL@ +Persistent=true +RandomizedDelaySec=300 + +[Install] +WantedBy=timers.target diff --git a/backup-system/tests/test_reflux.sh b/backup-system/tests/test_reflux.sh new file mode 100644 index 0000000..9785b2d --- /dev/null +++ b/backup-system/tests/test_reflux.sh @@ -0,0 +1,159 @@ +#!/usr/bin/env bash +# Contract for bin/ckis-reflux-all.sh โ€” the autonomous context-maintenance engine. +# The load-bearing guarantee under test: the structural guards reject any unsafe +# proposal, and the propose path NEVER touches the live heart doc. No claude -p +# is invoked (a stub model is used) so the test is fast and deterministic. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "$HERE/.." && pwd)" +source "$HERE/assert.sh" + +SB="$(mktemp -d)"; trap 'rm -rf "$SB"' EXIT +export CKIS_LOG_DIR="$SB/state"; mkdir -p "$CKIS_LOG_DIR" + +# Source the engine for its functions only (sourcing guard prevents _run). +export CKIS_MANIFEST="$ROOT/ckis-manifest.json" # real manifest for SCAN path etc. +source "$ROOT/bin/ckis-reflux-all.sh" + +# โ”€โ”€ a synthetic "live" heart doc โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ +LIVE="$SB/_MEMORY.md" +cat >"$LIVE" <<'EOF' +--- +type: capture +created: 2026-04-07 +modified: 2026-05-22 +tags: [memory] +status: active +--- + +# Live Business State + +## Project Alpha +- Stage: launched +- Client: Acme Corp + +## Blockers +- none + +## Focus +- ship the reflux engine +EOF +LIVE_LINES=$(wc -l < "$LIVE") + +# helper: clone live, run a mutator, return the proposal path +_mk() { local p="$SB/prop.md"; cp "$LIVE" "$p"; "$@" "$p"; printf '%s' "$p"; } + +# โ”€โ”€ Guard tests (max_lines=150, min_pct=70) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + +# 1. a faithful proposal (created same, all headings, modified bumped) โ†’ PASS +good="$SB/good.md"; cp "$LIVE" "$good"; sed -i 's/^modified:.*/modified: 2026-06-19/' "$good" +assert_ok "good proposal passes all guards" _guards "$LIVE" "$good" 150 70 + +# 2. changed `created` โ†’ REJECT +badc="$SB/badc.md"; cp "$good" "$badc"; sed -i 's/^created:.*/created: 2026-01-01/' "$badc" +assert_fail "changed created is rejected" _guards "$LIVE" "$badc" 150 70 + +# 3. dropped a `##` heading โ†’ REJECT +badh="$SB/badh.md"; grep -v '^## Blockers' "$good" >"$badh" +assert_fail "dropped section heading is rejected" _guards "$LIVE" "$badh" 150 70 + +# 4. shrunk below 70% of live โ†’ REJECT +bads="$SB/bads.md"; head -4 "$good" >"$bads" # only frontmatter-ish, way under 70% +assert_fail "shrink below 70% is rejected" _guards "$LIVE" "$bads" 150 70 + +# 5. over max_lines (cap=10) โ†’ REJECT (good doc is >10 lines) +assert_fail "over max_lines is rejected" _guards "$LIVE" "$good" 10 70 + +# 6. contains a real secret โ†’ REJECT +badk="$SB/badk.md"; cp "$good" "$badk" +printf '\n- token: ghp_%s\n' "aB3dE6gH9jK2mN5pQ8rS1tU4vW7xY0zA3bC6" >>"$badk" # ckis-allow-secret (test fixture) +assert_fail "secret material is rejected" _guards "$LIVE" "$badk" 150 70 + +# 7. duplicate-heading drop (multiset, not set) โ†’ REJECT +# live with two identical headings; proposal keeps only one instance. +dlive="$SB/dlive.md"; { cat "$good"; printf '\n## Notes\n- a\n\n## Notes\n- b\n'; } >"$dlive" +ddrop="$SB/ddrop.md"; { cat "$good"; printf '\n## Notes\n- a\n'; } >"$ddrop" # one '## Notes' dropped +assert_fail "dropping one of two identical headings is rejected" _guards "$dlive" "$ddrop" 150 70 + +# 8. secret scanner MISSING โ†’ FAIL-CLOSED (reject, never silently skip) +_SCAN_REAL="$SCAN"; SCAN="$SB/nonexistent-scanner.sh" +assert_fail "missing secret scanner fails closed" _guards "$LIVE" "$good" 150 70 +SCAN="$_SCAN_REAL" + +# โ”€โ”€ Propose path: quarantine + live untouched (stub model, no claude -p) โ”€โ”€โ”€โ”€โ”€โ”€ +# Fixture manifest: vault=$SB, model_cmd=a stub that emits a faithful proposal. +STUB="$SB/stub-model.sh" +cat >"$STUB" </dev/null || true # _build_digest tolerates non-repo, but be safe + +cat >"$SB/m.json" <"$digest" +CKIS_MANIFEST="$SB/m.json" _propose_doc "memory" "00-inbox/_MEMORY.md" 150 "$digest" >/dev/null +after="$(md5sum "$LIVE_VAULT" | awk '{print $1}')" + +assert_eq "live heart doc UNTOUCHED by propose" "$before" "$after" +qf="$(ls "$SB/00-inbox/_REFLUX-QUEUE/"memory-*.md 2>/dev/null | head -1)" +assert_file "proposal queued to quarantine" "$qf" +assert_contains "queue file carries the PROPOSAL marker" "$(cat "$qf" 2>/dev/null)" "---PROPOSAL---" +assert_contains "queue file shows a diff" "$(cat "$qf" 2>/dev/null)" "modified: 2026-06-19" + +# โ”€โ”€ Regression: the model call must NOT eat the doc-loop's stdin โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ +# `claude -p` reads stdin; the propose loop is fed by process substitution. If the +# model call doesn't redirect stdin from /dev/null it swallows the loop's stream +# and only the FIRST doc is processed. Reproduce with a stub that CONSUMES stdin. +STUB2="$SB/stub-eats-stdin.sh" +cat >"$STUB2" </dev/null 2>&1 # consume all stdin, like claude -p does +sed 's/^modified:.*/modified: 2026-06-19/' "$SB/00-inbox/\$REFLUX_DOC" +EOF +chmod +x "$STUB2" +cp "$LIVE" "$SB/00-inbox/_A.md"; cp "$LIVE" "$SB/00-inbox/_B.md" +# the stub needs to know which doc; simplest: emit a generic valid update of LIVE +cat >"$STUB2" </dev/null 2>&1 # consume all stdin, exactly like claude -p +sed 's/^modified:.*/modified: 2026-06-19/' "$LIVE" +EOF +chmod +x "$STUB2" +cat >"$SB/m2.json" </dev/null 2>&1 ) +n_q2=$(ls "$SB/00-inbox/_REFLUX-QUEUE2/" 2>/dev/null | wc -l | tr -d ' ') +assert_eq "stdin-eating model still processes ALL docs (not just 1)" "2" "$n_q2" + +# โ”€โ”€ Apply path: promotes the reviewed proposal to live (guards re-checked) โ”€โ”€โ”€โ”€ +CKIS_MANIFEST="$SB/m.json" _apply "$qf" >/dev/null 2>&1 +applied="$(grep -c '^modified: 2026-06-19' "$LIVE_VAULT")" +assert_eq "apply promotes proposal to live" "1" "$applied" + +assert_summary