v0.8.0: Smriti daemon — cross-agent capture, auto-start on macOS + Linux#76
Open
ashu17706 wants to merge 25 commits into
Open
v0.8.0: Smriti daemon — cross-agent capture, auto-start on macOS + Linux#76ashu17706 wants to merge 25 commits into
ashu17706 wants to merge 25 commits into
Conversation
- --whole flag for `smriti ingest file`: stores .md as single message (no paragraph splitting); warns without flag - `smriti projects <id>`: rich inspection report — sessions, messages, agents, tags, decisions, recent sessions - `smriti tags`: global or --project-scoped tag usage counts; --available mirrors category tree - `smriti status --project <id>`: scopes all stats (agents, categories) to a single project - 29 new tests in test/recall.test.ts covering all retrieval paths (full-doc, tags, project reports, multi-filter)
QMD submodule is now a clean upstream fork (d58fedf, v2.1.0+) — no Smriti-specific code lives there. Future upstream syncs are conflict-free. - src/memory.ts: moved from qmd/src/memory.ts; imports updated to ../qmd/src/store.js and ../qmd/src/llm.js; uses QMD's Database type - src/ollama.ts: moved from qmd/src/ollama.ts; self-contained, no changes - src/qmd.ts: re-exports now come from ./memory and ./ollama - qmd submodule: bumped to d58fedf (upstream v2.1.0+34 commits of fixes) Upstream picks up: security dep bumps, db-transaction-type fix, embedding overflow hardening, sqlite-vec actionable errors, GGUF magic error fix, Windows home fallback, status device probe opt-in, and more.
Issue #62 — Knowledge Density Scoring: - Add density_score REAL column to smriti_session_meta (migration) - computeDensityScore(): composite 0-1 score from tool calls (25%), file writes (25%), git ops (20%), decision tags (15%), errors (10%), token volume (5%) - Hook into storeSession() so every ingest auto-computes and persists score - Blend density into recallMemories() ranking: final = score*0.8 + density*0.2 - smriti enrich --density: backfill scores for all existing sessions - smriti show <id> extension: --density flag shows bar-chart breakdown Issue #63 — smriti digest: - New src/digest.ts: generateDigest() aggregates sidecar signals for a time window, groups by project, surfaces top tools/errors/costs - formatDigest() and formatDensityBreakdown() added to format.ts - smriti digest [--days N] [--project id] [--synthesize] [--model name] generates work summary; --synthesize calls Ollama for narrative
…import (#57) Phase 1: DB init via QMD SDK - New src/store.ts: cycle-free QMDStore singleton (setQmdStore/getQmdStore) - db.ts: remove initializeQmdStore() (duplicate of SDK's initializeDatabase) initSmriti() now calls SDK createStore() and is async - closeDb() delegates to closeQmdStore() Phase 2: Kill ../qmd/src/llm.js deep import in memory.ts - Replace getDefaultLlamaCpp() with getQmdStore().internal.llm - Replace insertEmbedding() call with getQmdStore().internal.insertEmbedding() - formatQueryForEmbedding/formatDocForEmbedding moved to ../qmd/src/store.js import (they are re-exported there; no longer touching llm.ts internals) Downstream: index.ts awaits initSmriti(); test/team.test.ts uses beforeAll for async init
Default-on quality mode in recallMemories(): - Calls store.internal.expandQuery() to generate lex/vec/hyde query variants - Runs FTS + vec search for each variant with 0.7 weight - Fuses all ranked lists via RRF (original queries at 1.0 weight) - Reranks top deduped candidates with store.internal.rerank() (60/40 blend) - --fast flag skips both steps for low-latency lookups Also fixes team-segmented.test.ts beforeAll to use async initSmriti().
Adds smriti_session_queries table + smriti_queries_fts virtual table. expandQuery() generates search aliases per session (lex/vec/hyde variants). searchFiltered() merges alias matches as 'query_alias' source results. storeSession() auto-enriches new ingests non-blocking (fire-and-forget). --dry-run shows generated aliases without writing; --project scopes batch.
Multi-angle recall (expandQuery + rerank default-on) feeds top-N sessions to ollamaAsk() which returns a grounded answer with [N] citations. --no-synthesize returns ranked sources only; --json returns structured output. Graceful fallback to sources when Ollama is unavailable.
smriti recall "query" --project X --wide searches all projects (bypassing project filter) and rerankss with intent "relevant to X project context" so the cross-encoder scores cross-project results against local needs. Results from other projects get project badge in output via session meta lookup. --wide without --project is equivalent to global unfiltered recall.
Recalls all sessions about a topic, sorts chronologically, and synthesizes an evolution narrative via Ollama showing decisions, reversals, refinements. --since <date> filters to recent history; --no-synthesize returns timeline only. --json returns structured timeline array. Graceful "not enough history" when < 2 sessions.
ollamaCheckConflicts() sends all top-N results in one batch to Ollama and parses CONFLICT [i] vs [j]: description responses. --check-conflicts on smriti recall flags contradictory pairs in output. --json includes conflicts array. No behavior change without the flag. SMRITI_CONFLICT_THRESHOLD env configures sensitivity (default 0.7).
Dual-write on ingest: each session written to ~/.cache/smriti/sessions/<id>.md. initSmriti() registers smriti-sessions collection when dir exists. storeSession() writes markdown + fires background store.update(). recall() uses store.search() when smriti-sessions has docs; falls back to recallMemories() otherwise (backward compat). rerank=false when --fast.
k-means over session embeddings, Ollama cluster naming, smriti clusters command, enrich --clusters, and recall --cluster <name> filter.
…#2) #1: parseFrontmatter now parses tags arrays into string[]; syncTeamKnowledge restores all tags from meta.tags with isValidCategory guard; falls back to scalar meta.category for old exports. #2: new src/team/config.ts with readConfig/writeConfig/mergeCategories/ exportCustomCategories; share writes custom categories to config.json (v2); sync reads config.json and upserts categories before scanning files; SyncResult gains categoriesImported; smriti config show/add-category/ sync-categories CLI added.
Pulls 49 upstream commits via fast-forward merge. Key changes touching
search behavior:
- Fix hybrid RRF weighting by query type (#004714a) — expansion-derived
lists no longer steal original-query 2x weight when inserted first
- CJK FTS support (#d045a8b) — Han/Hiragana/Katakana/Hangul queries
now searchable via char-level spacing of CJK runs in documents_fts
(one-time migration on first qmd query after upgrade; Smriti's
memory_fts is unaffected)
- Embed collection filter honored (#5b9f472)
- HTTP MCP rerank control (#e36ab96)
- Forward candidateLimit through search APIs (#3b7e065)
- Preserve docids across case-only renames (#dff6513)
- macOS Metal cleanup abort mitigation (#60c75cb)
Risk audit: all 11 QMD APIs Smriti imports (createStore, QMDStore,
hashContent, chunkDocumentByTokens, reciprocalRankFusion,
formatQueryForEmbedding, formatDocForEmbedding, RankedResult,
insertEmbedding, initializeMemoryTables, Database) verified backward
compatible. insertEmbedding gained an optional 7th param totalChunks
(partial-embedding pending state), unused by Smriti.
Also restore test scoping ("bun test --cwd ./test") so Smriti's test
runner doesn't pick up QMD's own test/ files — two new upstream tests
(cli-lazy-llm-import, local-config) hardcode cwd-relative paths and
would otherwise fail when discovered from the parent repo. Same fix
pattern as cef23f2 from the March 2026 sync.
Full plan and verification at qmd/docs/UPSTREAM_MERGE_PLAN.md.
First piece of the v0.8.0 daemon work (#72). Intentionally narrow: just the single-instance guard, IPC socket bind, and signal handlers. No watcher, no debounce queue, no ingest wiring — those are separate modules / commits. Implementation notes (from pre-impl smoke tests against Bun 1.3.6): - Single-instance is enforced via DAEMON_PID_FILE + kill(pid, 0) liveness probe, not Unix-socket bind contention. Bun's net.listen() silently succeeds on duplicate binds and steals connections from the original server — verified with a reproducer. PID-file pattern is the same one QMD uses for `qmd mcp --daemon`. - IPC socket is bound separately for the Claude Stop hook poke, with cleanup of any stale socket file from a previous crash. - SIGTERM/SIGINT install a graceful shutdown that closes the server, removes the socket file, removes the PID file, and exits with conventional 128+signo status for supervisor visibility. - detectRunningDaemon() handles three stale states: missing PID file (returns null), garbage PID file (cleans + null), dead PID via ESRCH (cleans + null). Live PID returns the PID; EPERM also returns the PID (process exists but is foreign — don't start alongside). 10 unit tests cover detectRunningDaemon() across the three stale states plus the live case, and startDaemon() across the happy path, contention path, stale-PID-recovery path, idempotent shutdown, and the onPoke wire. PRD also gains a new "Three pre-impl smoke-test findings" section documenting why chokidar was dropped, why socket-bind isn't the single-instance mechanism, and why ingest() will open a fresh DB handle per debounce flush. Refs #71, #72.
Second module of the daemon (#72). Wraps Node's fs.watch so the queue can subscribe to "anything happened under this root" with a single callback shape, regardless of OS-specific backend differences. - macOS: fs.watch(root, { recursive: true }, cb). Native FSEvents delivers a single watcher per root. - Linux: inotify doesn't implement `recursive`, so we walk the tree at startup and watch each directory. New directories are picked up on the fly by re-watching when we see a `rename` event whose target is a directory. - Windows: same code path as macOS (ReadDirectoryChangesW supports recursive natively). Event paths are normalized to absolute. Null filenames (some FS backends emit them under load) are filtered out. Errors on individual watchers are silently dropped rather than crashing the parent — losing one subdirectory is better than losing the daemon. 7 tests cover: non-existent root rejection, direct-child file creation, deep-subdirectory creation (recursion), content change, absolute-path normalization, close()-stops-events, and the watchedCount() topology assertion (1 on macOS/Windows native, N on Linux). Refs #71, #72.
Third module of the daemon (#72). Coalesces bursts of "this project changed" signals into a single onFlush per project per quiet window. - schedule(projectId) resets the timer for that project. Repeated calls inside the window collapse to one firing — this is what makes a busy agent session not trigger 200 ingests as it writes JSONL. - flush(projectId) is the synchronous hook-poke path: fire onFlush immediately, cancel any pending debounce for that project. - Errors thrown by onFlush are caught and logged via the optional log callback rather than rejecting the timer's microtask. The caller (typically the daemon entry point) decides how to surface ingest errors. - close() cancels everything pending; subsequent schedule() calls become no-ops. Matches the lifecycle of the daemon process itself. Timers are unref'd so they don't keep Node alive on their own — process lifetime is owned by the IPC server, not by pending debounce timers. 9 tests cover: basic schedule/wait, coalescing across rapid schedules, per-project independence, immediate flush(), flush() with no pending timer, close() preventing pending fires, close() blocking subsequent schedules, error isolation from onFlush, and the isPending() inspector. Refs #71, #72.
Fourth module of the daemon (#72). Pure helpers that turn an FS path into the agent name responsible for it, and produce the default list of (agent, root) pairs the daemon should watch. For v0.8.0 the routing is intentionally coarse — by agent, not by project. A change anywhere under ~/.claude/projects/ schedules a single "ingest all of claude" flush, debounced. ingest() is already incremental at the session level, so unchanged sessions cost almost nothing per flush. A per-project resolution layer can replace this without changing the daemon's structure. getDefaultAgentRoots() filters by existsSync so we don't crash trying to watch a Codex or Cline install that isn't on this machine. Copilot is included only when COPILOT_STORAGE_DIR is set, since its location varies by OS and isn't auto-detected here. resolveAgentForPath() uses a strict prefix-with-separator check to avoid the classic ".claude/projects" matching ".claude/projects- archive/" bug. 6 tests cover the four match cases (exact root, child path, no match, sibling-prefix non-match) plus multi-root dispatch and the empty-root handling. Refs #71, #72.
Fifth and final module of the daemon core (#72). Powers `smriti daemon stop` and `smriti daemon status` without going through the IPC socket. The deliberate choice not to go through the socket: lifecycle commands need to work even when the daemon is wedged in a way that makes it unresponsive on the socket. Working through the PID file + signals is the most robust way to inspect and shut down a process. - getDaemonStatus() reads the PID, probes liveness via the existing detectRunningDaemon() helper, and includes a startedAt timestamp derived from the PID file's birthtime (falls back to ctime on filesystems that don't track birth). The PID-file races (file disappears between detect and stat) report as not-running rather than crashing. - stopDaemon() sends SIGTERM and polls for the PID file to disappear (the daemon's signal handler is responsible for unlinking it as part of graceful shutdown). Three result states: stopped, not- running, timeout. Callers — typically `smriti daemon stop` — decide how to escalate on timeout (could SIGKILL, could surface to the user). 6 tests cover both functions across no-daemon, stale-PID, and live cases. The timeout-path test temporarily swaps out the harness's SIGTERM handler so receiving the signal during the test doesn't kill the test runner. With this commit, #72 has all five core daemon modules: server, watcher, queue, handlers, client. Wiring them into a top-level daemon entry point and the CLI happens in subsequent commits. Refs #71, #72.
Top-level wiring for the daemon (#72). Connects watcher → resolveAgent → queue.schedule, plus hook poke → queue.flush("claude"), plus the default onFlush that opens a fresh SQLite handle, calls ingest(), and closes the handle (per smoke-test finding 3). Dependency-injection-friendly: tests pass a mock flushAgent so they can verify the wiring without invoking real ingest() against the user's real DB. Production callers (the CLI) accept defaults and get the real ingest path. A few intentional choices: - One log function flows through every module. Defaults to console.error so foreground daemon output goes to stderr; in production the LaunchAgent/systemd unit redirects stderr to DAEMON_LOG_FILE. Tests pass () => {} to silence. - "No agent roots found" is a soft warning, not a fatal error. The daemon still runs (the hook poke still works for Claude if Claude later writes session files). Avoids the case where installing on a fresh machine fails because no agents have written logs yet. - defaultFlushAgent catches and logs both DB-open errors and ingest errors. One bad flush should not crash the daemon — the next FS event will retry. - shutdown() is idempotent and closes watchers and queue before the server. This guarantees no FS event arrives at a torn-down queue (which would be a no-op but log a misleading "closed" warning). 6 integration tests cover: single-flush via watcher, coalescing across rapid writes, hook poke wired to claude flush, multi-root routing, error isolation from flushAgent, idempotent shutdown. With this commit, #72 is structurally complete. Next step is the CLI wiring (#74) so `smriti daemon` actually invokes runDaemon(). Refs #71, #72.
Implements #73. Generates the platform-appropriate service file, registers it with the system supervisor, and exposes inverse operations for uninstall. macOS: - Writes ~/Library/LaunchAgents/dev.zero8.smriti.plist - Registers via `launchctl bootstrap gui/<uid> <plist>` (modern) - Falls back to `launchctl load -w` on older macOS where bootstrap isn't available - Treats EEXIST / "already loaded" as success, not failure — that's the idempotent re-install case - Uninstall calls `launchctl bootout`, falls back to `launchctl unload`, then removes the plist Linux: - Writes ~/.config/systemd/user/smriti.service - Registers via `systemctl --user daemon-reload && systemctl --user enable --now smriti` - Service includes Restart=on-failure + RestartSec=5 so a crashed daemon comes back automatically, plus Nice=10 + IOSchedulingClass= idle so background indexing doesn't fight foreground work - Uninstall calls `systemctl --user disable --now`, removes the unit file, then daemon-reload to flush systemd's view Pure template generators (generatePlist, generateSystemdUnit) are exported for unit testing without spawning real launchctl/systemctl. Real-world interaction goes through a RunCmd abstraction that the default install path implements with Bun.spawn — tests inject a recording runner instead, so they can assert which commands would have been called without actually registering anything with the host's service manager. 13 tests cover the plist + systemd-unit generators (incl. XML escaping and ExecStart quoting), the install happy paths (macOS bootstrap + load-fallback, Linux daemon-reload + enable), the idempotent re-install path, the EEXIST-as-success case, error propagation from systemctl, and both uninstall paths. Manual integration testing (real launchctl bootstrap on this machine) will happen during the release-readiness work in #75. Refs #71, #73.
Implements #74. Adds the user-facing entry points for the daemon that #72 and #73 built. Six subcommands: smriti daemon Run in foreground (debugging, systemd target) smriti daemon install LaunchAgent (macOS) / systemd unit (Linux) smriti daemon uninstall Reverse install smriti daemon status PID, uptime, watched agents smriti daemon stop SIGTERM the running daemon smriti daemon logs tail -F the daemon log file Dispatch happens BEFORE initSmriti() because the foreground daemon opens its own DB handle per ingest flush rather than sharing one. Sharing a long-lived connection across many ingest calls was ruled out by pre-impl smoke test 3 (Bun segfault at ~6.8 GB peak RSS). Each subcommand uses lazy imports — the daemon module graph isn't loaded for unrelated commands like `smriti search`. Keeps the hot path cold-start unchanged. `smriti daemon status` formats the uptime in the largest-fitting unit (seconds / minutes / hours / days) so the most common state ("running for 2 days") reads naturally without grep. Logs follow tail -F semantics so the command keeps working across log rotation, which both LaunchAgents and systemd will do over time. HELP text gains a "Daemon options" block alongside Ingest, Search, Recall, etc. Manual verification: $ smriti daemon status daemon: not running PID file: /Users/zero8/.cache/smriti/daemon.pid $ smriti daemon banana Unknown daemon subcommand: banana Usage: smriti daemon [install|uninstall|status|stop|logs] smriti daemon (run in foreground) Refs #71, #74.
- package.json: 0.6.0 → 0.8.0. (v0.7.0 was tagged in git without a matching package.json bump; we skip past it directly to 0.8.0 since the daemon is the headline change.) - CLAUDE.md quick-reference gains a "Daemon (v0.8+)" block covering all six subcommands, plus the recommended Stop-hook template that pokes the socket when the daemon is running and falls back to lockf when it isn't. Refs #71, #75.
ashu17706
changed the title
29 tasks
Two reference docs to make the v0.8.0 tag a five-minute event: - docs/internal/release-flow.md captures the four-phase release process (feature branch → staging on hardware → checklist → tag). Intended to be reused for every future release, not just v0.8.0. Includes the upgrade-restart gap that will become v0.8.1, the "what lives where" table, and the explicit list of things we do not do (no CI release pipeline, no RC channels, no release branches kept alive past tag). - docs/internal/release-notes-v0.8.0.md is the canonical body for the GitHub release. Written in the "what an engineer would tell a colleague about" voice that we agreed releases should land in. Pulls together the headline (cross-agent capture), the three design constraints that shaped it (smoke-test findings), the recommended Stop-hook update, what's deferred to 0.8.1, and the postmortem provenance that got us here. Issue #75 now contains the real-hardware acceptance checklist that gates tagging — once those boxes are green, the commands in release-flow.md execute the tag. Refs #71, #75.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #71, #72, #73, #74. Sets up #75 for release.
The headline feature for v0.8.0: a long-running `smriti daemon` that captures sessions from every agent in the background. Open Cursor, Codex, or Claude — your sessions land in Smriti's index without typing a command.
What's in this PR
Daemon core (`src/daemon/` — five modules)
Service installer (`src/daemon/install.ts`)
CLI (`src/index.ts`)
```
smriti daemon # Run in foreground (debugging, systemd target)
smriti daemon install [--force] # Install LaunchAgent / systemd unit, register
smriti daemon uninstall # Reverse install
smriti daemon status # PID, uptime, watched agents
smriti daemon stop # SIGTERM the running daemon
smriti daemon logs # tail -F the daemon log
```
Docs
Version
`package.json`: 0.6.0 → 0.8.0.
Design constraints baked in (from pre-impl smoke tests)
All three are documented in `docs/internal/daemon-prd.md`.
Test results
```
bun test test/daemon-*.test.ts
57 pass / 0 fail / 121 expect() calls — Ran 57 tests across 7 files in 4.11s
```
Full suite still passes (the 2 pre-existing QMD-submodule path failures are unrelated).
What's left for v0.8.0 (after this PR merges)
Why one big PR for the whole daemon
Each daemon module is small (50–250 LOC each), but they're tightly coupled — the watcher's events feed the queue, the queue invokes ingest, the hook poke routes to the queue, the lifecycle commands inspect the PID file the server writes. Reviewing them in isolation loses the most important property of the change (that they compose into a working daemon). Committing them as a sequence of small, individually-tested commits on one branch gives reviewers both the local view (commit-by-commit) and the global view (this PR description).
Commit history (9 commits)
```
chore(release): bump to v0.8.0; document daemon commands in CLAUDE.md
feat(cli): wire smriti daemon subcommands
feat(daemon): LaunchAgent + systemd-user installer (macOS + Linux)
feat(daemon): runDaemon() entry point wiring all five modules
feat(daemon): lifecycle client for stop / status
feat(daemon): agent-root routing helpers
feat(daemon): per-project debounce queue
feat(daemon): recursive watcher with macOS-native + Linux walk-and-watch
feat(daemon): scaffold server with PID-file single-instance + IPC socket
```
Refs #71, #72, #73, #74, #75.