Upstream a BMQ3-style read cache: MCP tool latency is product-defining for multi-agent use

[groksrc]

Problem

At power-user scale (~4,000+ notes), our MCP tool latency is severe enough that it forced a user to build a parallel tooling layer rather than use BM's tools directly: reads take 3–7s and search ~12s through the standard path. For multi-agent setups (several assistants hitting the same knowledge base concurrently), that latency makes BM effectively unusable — performance was the most repeated complaint in the user interview that surfaced this.

Prior art: BMQ3

The user runs BMQ3, a caching proxy in front of BM's SQLite, and has offered it as input for upstreaming. Architecture worth studying:

• HTTP engine with a thin stdio wrapper forwarding JSON-RPC; 9 tools (bm_read, bm_search, bm_grep, bm_query, bm_ls, bm_write, bm_edit, bm_delete, bm_health)
• RAM caches for entities / observations / relations / frontmatter / permalinks, invalidated on DB+WAL mtime — cheap and correct enough in practice
• Imports BM internals directly (FastEmbedEmbeddingProvider, apply_edit_operation, generate_permalink) so read/write semantics match BM's
• Writes return in ~25ms with the DB upsert backgrounded
• Explicitly does not replace graph traversal (build_context) or schema operations

Measured results: read 5ms cold / ~0.001ms cached (vs. 3–7s); search 95ms (vs. 12s); grep 0.05ms.

Proposal (v1.0)

Evaluate upstreaming the caching approach into BM itself — an in-process read cache over the SQLite index with mtime-based invalidation, so every MCP tool benefits without a sidecar proxy. Target: sub-100ms reads/search at the few-thousand-note scale.

Open questions:

• In-process cache in the API layer vs. a long-lived daemon (cloud mode already has a server; local stdio MCP is where cold-start + query cost bites)
• Invalidation: DB+WAL mtime is proven by BMQ3; interaction with sync writes needs care
• Which surfaces gain most: read_note, search_notes, list_directory first; build_context stays on the graph path
• Memory budget at 10k+ notes

Related

• Normalize the CLI / MCP tool argument surface for v1.0 #977 — the interface-design half of the same interview feedback (file-path-style addressing and grep-style search align with model training; BMQ3 implements both). This issue is the performance half.
• If we don't address this, the failure mode is more power users forking or building parallel layers.

[claude]

Claude finished @groksrc's task in 2m 14s —— View job

---

Issue Triage

• Gather context and read the issue
• Check for duplicate issues — no duplicates found
• Classify issue type, priority, component, complexity
• Apply labels (enhancement, v1.0)
• Provide triage analysis

---

Analysis

Type: Enhancement / Feature Request — this is a performance infrastructure proposal, not a bug. The system works correctly but is too slow at power-user scale for multi-agent use cases.

Priority: High
The latency numbers (reads 3–7s, search 12s) are bad enough that at least one power user built an entire parallel tooling layer (BMQ3) rather than use BM directly. The issue explicitly ties this to multi-agent workloads where several assistants hit the same knowledge base concurrently — the emerging primary use case. For that use case, the current performance makes BM effectively unusable. Keeping a user building workarounds when they want to contribute the solution upstream is a strong signal.

Component: MCP + Database layer
The bottleneck is the full round-trip for every MCP tool call through the SQLite index. The BMQ3 prior art localizes the fix to an in-process read cache in the API layer rather than rewriting the DB schema or sync engine. Primary tools impacted: read_note, search_notes, list_directory, recent_activity.

Complexity: Complex
This requires architectural decisions that will have broad impact:

• Where does the cache live? In-process in the API server (best for the long-lived cloud/server mode; stdio MCP restarts each session) vs. a persistent daemon approach
• Invalidation strategy: BMQ3's DB+WAL mtime approach is proven and cheap to implement; need to validate it handles concurrent sync writes correctly
• Memory budget at 10k+ notes needs modeling (entities + observations + relations + frontmatter + permalinks all cached)
• build_context / graph traversal should stay on the live path for now (as BMQ3 also chose)

Relationship to #977
Both issues come from the same user interview / QA session and should be evaluated together for v1.0. #977 addresses the API surface normalization (naming divergences); #980 addresses the performance floor. Both are prerequisites for multi-agent workloads at scale.

No duplicates found. Issue #686 (progressive note loading) is related — it addresses token cost per read — but the problems are distinct (L0/L1/L2 reduces payload size; this issue targets query latency). They're complementary and could eventually combine (cache hot at L0, fetch L1/L2 on demand).

---

Recommended Next Steps

1. Accept BMQ3 contribution for study: The author has offered the code as upstream input. Review the invalidation logic (DB+WAL mtime check) against the sync coordinator's write path to confirm it's safe.
2. Prototype in-process cache in API layer: Start with search_notes and read_note — highest-latency, highest-frequency tools. A simple dict keyed on permalink with mtime-based invalidation is the minimum viable version.
3. Benchmark before/after at 1k/5k/10k notes: The BMQ3 numbers (read 5ms cold, search 95ms) should be the target bar for the first iteration.
4. Defer build_context caching: graph traversal has different invalidation semantics and is less latency-sensitive than direct reads; keep it on the live path for v1.0.

Labels applied: enhancement, v1.0

[phernandez]

@groksrc I think the thing to do for this is to pull the note_content layer from the cloud into the basic-memory app. I have some notes for that.