feat(memory): ask the vault a question in plain words

docs/design-notes.md

@@ -132,14 +132,19 @@ Moving it makes every caller correct at once: the agent tool, the archivist,
 `stack memory search` from a terminal, and whatever asks next. Leaving it
 means the next consumer re-learns this the way the agent did.
 
-**What has to be decided when it moves.** The rewrite needs an LLM, so a
-search command that has never called a model would start to, and that changes
-its latency and its failure modes. Options are a flag (`--natural`), inferring
-it from the trailing `?` the way the archivist does, or keeping the rewrite as
-a lib function that callers opt into. The archivist also needs the keywords
-back, not just the regex, because it shows `Searched for: ...` so a family can
-see when a bad rewrite hid results; that visibility is worth keeping and the
-return shape has to carry it.
+**What was decided when it moved.** The rewrite lives in `memory.lib` and
+takes the caller's LLM, so memory owns the hop without owning a client. On the
+CLI it sits behind `stack memory search --nl`, which reaches the model in the
+bot-runner the same way `stack memory wiki` does. Opt-in rather than inferred:
+this surface's default query language is a regex, in which `?` is a quantifier,
+so inferring from a trailing `?` would turn `Zelt?` into a surprise model call,
+and it is on an agent's hot path where the default has to stay cheap. Chat keeps
+inferring, because there a rewrite per message is affordable and one character
+is a rule a family can learn. Two guards keep the cost honest: a query with no
+whitespace never leaves the host, and every failure (no AI, container down,
+model silent) falls back to searching the query literally. Both surfaces print
+`Searched for: ...`, because a rewrite that picked the wrong words has to look
+different from an empty vault.
 
 Both current callers should keep working unchanged through the move. That is
 the test.

stacklets/agent/runtime/memory_tool.py

@@ -64,11 +64,17 @@ async def execute(
         person: str | None = None,
         tag: str | None = None,
     ) -> str:
+        # `--nl` is what makes the parameter description above true. The
+        # CLI's default query language is a regex, so a question sent
+        # without it asks for those exact words, adjacent, and matches
+        # nothing. The CLI skips the model itself on a single word, so
+        # passing this always costs nothing on keyword lookups.
         args = [
             "stack",
             "memory",
             "search",
             query,
+            "--nl",
             "--limit",
             str(limit or 5),
         ]

stacklets/docs/bot/pipeline.py

@@ -28,8 +28,10 @@
 import asyncio
 import json
 import re
+import sys
 from dataclasses import dataclass, field
 from datetime import date
+from pathlib import Path
 from typing import TYPE_CHECKING, Any
 
 import aiohttp
@@ -57,6 +59,18 @@
     ModelCapabilities,
 )
 
+# Sibling stacklets resolve through `stacklets/` on sys.path: in the
+# bot-runner container that directory is the read-only mount, locally it
+# is the source tree. archivist.py wires the same path for its own
+# imports, but it imports this module first, so the path goes on here
+# rather than relying on who loads whom.
+_STACKLETS_DIR = Path(__file__).resolve().parents[2]
+if str(_STACKLETS_DIR) not in sys.path:
+    sys.path.insert(0, str(_STACKLETS_DIR))
+# The recall prompt lives with the vault it searches. See
+# `memory.lib`'s query-rewrite section for why it moved there.
+from memory.lib import rewrite_query as memory_rewrite_query  # noqa: E402
+
 # Back-compat alias — the docs pipeline shipped with `ImageAttachment`
 # before the framework introduced `LLMImage`. The fields are identical
 # so callers stay duck-typed.
@@ -791,35 +805,19 @@ async def rewrite_query(
     ) -> list[str]:
         """Extract search keywords from a natural-language question.
 
-        Used by the archivist when a message ends with `?`. The LLM
-        reads the family's topic + doctype ontology and produces 2-4
-        keywords that would literally appear in a matching document --
-        translation and synonym expansion happen here, so the regex
-        walker downstream stays dumb.
-
-        Best-effort: any LLM transport failure or parse error returns
-        an empty list, which the caller treats as "no rewrite, search
-        the question verbatim." Synonym selection is the LLM's job;
-        we don't second-guess it here, but we do strip empties and
-        cap the list length so a chatty model can't blow the regex up.
+        Used by the archivist when a message ends with `?`. The prompt,
+        the parsing, and the best-effort contract live in
+        `memory.lib.rewrite_query`, because memory owns the vault those
+        keywords are aimed at and every other caller of that vault needs
+        the same hop. What the archivist contributes is the LLM it
+        already built, which is all memory was missing.
         """
-        prompt = _build_rewrite_prompt(question, ontology_section, lang)
-        try:
-            raw = await self._llm.complete("recall", prompt, json_mode=True)
-        except (LLMUnavailableError, LLMModelNotFoundError, LLMTimeoutError) as e:
-            logger.warning("[recall] LLM unavailable for rewrite: {}", e)
-            return []
-        keywords = _parse_rewrite_response(raw)
-        if not keywords:
-            # Surface the raw payload so an empty keyword list is
-            # debuggable: an off-shape JSON response is a prompt or
-            # model issue, not a transport one, and we can't fix it
-            # blind.
-            logger.warning(
-                "[recall] rewrite parse produced no keywords; raw={!r}",
-                (raw or "")[:200],
-            )
-        return keywords
+        return await memory_rewrite_query(
+            question,
+            llm=self._llm,
+            ontology_section=ontology_section,
+            language=lang,
+        )
 
     async def synthesize_answer(
         self,
@@ -1218,75 +1216,6 @@ def _build_reformat_prompt(ocr_text: str) -> str:
 ---"""
 
 
-# ── Query rewrite ────────────────────────────────────────────────────────
-#
-# The recall-mode entry point. When a family member asks a question
-# (anything ending in `?`), the archivist asks the LLM to extract 2-4
-# keywords that would literally appear in a matching document. The
-# regex walker then OR-alternates them into a single search pattern.
-# Two wins: (1) "When did Bart get vaccinated?" becomes a search for
-# Impfung/MMR/Auffrischung, which actually hits the German vaccination
-# record; (2) the ontology block primes synonym + translation knowledge
-# without us having to ship a thesaurus.
-
-def _build_rewrite_prompt(
-    question: str, ontology_section: str, lang: str,
-) -> str:
-    """Recall-mode prompt: question → JSON list of search keywords."""
-    return f"""You extract search keywords from a question, so a regex walker can look up family documents.
-
-The family classifies their documents under these topics and forms:
-
-{ontology_section}
-
-Language hint: {lang}.
-- If the question is in German, prefer German keywords.
-- If it is in English, prefer English keywords.
-- For an ambiguous or generic question, include the most likely topic name plus one or two synonyms in the document language.
-
-Question: {question}
-
-Reply with a JSON object: {{"keywords": ["word1", "word2", "word3"]}}.
-2 to 4 keywords. Each keyword is a literal word that would appear in
-the document (a noun, a name, a topic). No phrases, no quotes, no
-prose around the JSON. Output ONLY the JSON object."""
-
-
-def _parse_rewrite_response(raw: str) -> list[str]:
-    """Pull a keyword list out of the rewrite LLM's response.
-
-    Accepts the requested object form `{"keywords": [...]}` and the
-    bare-array fallback that some smaller models produce when they
-    forget the wrapper. Empty list on any parse failure -- the caller
-    treats empty as "no rewrite, search the question verbatim." Caps
-    at 6 entries so a chatty model can't blow up the alternation
-    regex; trims whitespace and drops empties so a stray `""` doesn't
-    poison the join.
-    """
-    try:
-        data = json.loads(raw)
-    except (ValueError, TypeError):
-        return []
-
-    if isinstance(data, dict):
-        v = data.get("keywords")
-        candidates = v if isinstance(v, list) else []
-    elif isinstance(data, list):
-        candidates = data
-    else:
-        candidates = []
-
-    cleaned: list[str] = []
-    for x in candidates:
-        if isinstance(x, (str, int, float)):
-            s = str(x).strip()
-            if s:
-                cleaned.append(s)
-        if len(cleaned) >= 6:
-            break
-    return cleaned
-
-
 # ── Answer synthesis ─────────────────────────────────────────────────────
 
 def _format_evidence_block(evidence: list[dict]) -> str:

stacklets/docs/bot/recall.py

@@ -23,16 +23,29 @@
 Everything is best-effort: an unreachable LLM, a malformed response,
 or an empty keyword list falls back to the literal query. Recall is
 a quality-of-life feature, never a gate.
+
+Chat infers the trigger; the CLI does not. `stack memory search`
+reaches the same rewrite behind an explicit `--nl`, and that
+difference is deliberate on both sides. Here, every message is
+already being read for a family, and one character is a rule a
+family can learn. There, the default query language is a regex, in
+which `?` is a quantifier, so inferring from it would turn `Zelt?`
+into a surprise model call. Do not "fix" either one into the other.
 """
 
 from __future__ import annotations
 
-import re
 from typing import Optional, Tuple
 
 from loguru import logger
 
 from pipeline import Classifier
+# Memory owns the vault, so it owns the rewrite and the regex the
+# walker reads. This module keeps the chat-side half: when to spend an
+# LLM call, and how Paperless wants the same keywords rendered. The
+# import above puts `stacklets/` on sys.path, which is what makes the
+# sibling stacklet reachable here.
+from memory.lib import keywords_to_regex
 
 
 async def resolve_search_query(
@@ -102,10 +115,9 @@ async def resolve_search_query(
         return query, query, []
     logger.info("[recall] keywords: {}", keywords)
 
-    # Memory side: regex alternation, re.escape each keyword so a
-    # chatty LLM that returns "C++" or "Lisa's" can't blow up the
-    # compile step.
-    memory_regex = "|".join(re.escape(k) for k in keywords)
+    # Memory side: regex alternation, escaped per keyword. Memory
+    # renders this because memory is what has to read it back.
+    memory_regex = keywords_to_regex(keywords)
     # Paperless side: Whoosh OR alternation, bare tokens. The
     # PaperlessAPI wraps this in `_to_search_query` which adds the
     # prefix wildcard on each bare term -- so "fish OR price" becomes

stacklets/memory/bot/cli/rewrite.py

@@ -0,0 +1,60 @@
+"""memory rewrite <question> — the keyword hop, run where the model lives.
+
+`stack memory search --nl` needs a model, and the host `./stack` is
+stdlib-only by design. So the host asks this, inside the bot-runner,
+for the one thing it cannot work out itself: which words would
+literally appear in a document that answers the question. The host
+still does the searching.
+
+Output is one keyword per line on stdout, which is the smallest thing
+that survives a `docker exec` round trip without a parser. Exit 0 with
+keywords, exit 1 with none. Exit 1 is not an error: the host reads it
+as "no rewrite available" and searches the question literally, the
+same fallback the archivist has always had.
+
+The ontology comes from the live vault (`MEMORY_VAULT_DIR`), because
+the keywords have to be in the vocabulary this family files under. A
+`--vault` override on the host does not change that: the taxonomy is
+the family's, not the directory's.
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[3]))
+from memory.lib import get_ontology, rewrite_query  # noqa: E402
+
+from stack.ai.client import LLM  # noqa: E402
+
+HELP = "Turn a natural-language question into vault search keywords"
+
+
+async def run(llm: LLM, argv: list[str]) -> int:
+    """Entry point the dispatcher calls with the shared LLM client."""
+    if not argv or not argv[0].strip():
+        print("usage: rewrite <question>", file=sys.stderr)
+        return 2
+
+    question = argv[0]
+    vault = Path(os.environ.get("MEMORY_VAULT_DIR", "/data/memory/vault"))
+    language = os.environ.get("LANGUAGE", "en")
+
+    # `get_ontology` falls back to the shipped seed when the vault has
+    # no ontology.toml of its own, so a fresh install still gets sane
+    # topic names rather than an empty prompt section.
+    ontology = get_ontology(vault if vault.exists() else None)
+
+    keywords = await rewrite_query(
+        question,
+        llm=llm,
+        ontology_section=ontology.classifier_prompt_section(language),
+        language=language,
+    )
+    if not keywords:
+        return 1
+
+    print("\n".join(keywords))
+    return 0

stacklets/memory/bot/cli_entrypoint.py

@@ -10,6 +10,13 @@
 re-use the bot-runner as their tools runtime.
 
 Commands:
+    rewrite <question>
+        Print the search keywords a question should be looked up by,
+        one per line. `stack memory search --nl` is the caller: it
+        does the searching on the host and only comes here for the
+        words. Exit 1 means no keywords, which the host treats as
+        "search it literally" rather than as a failure.
+
     wiki [--home] [--member <slug>]... [--topic <slug>]... [--dry-run]
         Regenerate the family wiki's entry pages. Apply by default;
         `--dry-run` previews to stdout. Bare invocation regenerates
@@ -30,10 +37,11 @@
 
 from stack.ai.client import LLM, LLMUnavailableError
 
-from cli import wiki
+from cli import rewrite, wiki
 
 
 _HANDLERS = {
+    "rewrite": rewrite.run,
     "wiki": wiki.run,
 }
 

stacklets/memory/cli/_common.py

@@ -60,3 +60,50 @@ def dispatch(command: str, *argv: str) -> dict:
     if rc != 0:
         sys.exit(rc)
     return {"ok": True}
+
+
+def dispatch_capture(command: str, *argv: str,
+                     timeout: int = 60) -> tuple[int, str, str]:
+    """The same hop, for a caller that wants the output as a value.
+
+    `dispatch` is right when the container's output *is* the result:
+    it streams to the terminal and exits with the container's status.
+    `stack memory search --nl` is the other shape. It asks the
+    container for keywords and then does its own work with them, so it
+    needs them back, and it must stay alive when the hop is
+    unavailable.
+
+    Returns `(returncode, stdout, reason)`. A stopped container, a host
+    with no docker, and a model that timed out all come back as a
+    nonzero code, because to the caller they mean the same thing: no
+    answer from here, carry on without one.
+
+    `reason` is the first line of the container's stderr, and it is
+    first line rather than all of it on purpose. An entry point that
+    does not know the command prints its whole usage text, and a
+    version-skewed host (updated code, bot-runner not restarted yet)
+    would dump that over a family's search results. One line stays
+    diagnostic without ever becoming a wall.
+    """
+    if not _bot_runner_running():
+        return 1, "", f"{BOT_RUNNER_CONTAINER} is not running"
+
+    cmd = [
+        "docker", "exec", "-i",
+        BOT_RUNNER_CONTAINER,
+        "python", ENTRYPOINT_PATH, command, *argv,
+    ]
+    try:
+        result = subprocess.run(
+            cmd, capture_output=True, text=True, timeout=timeout,
+        )
+    except FileNotFoundError:
+        return 1, "", "docker CLI not found on this host"
+    except subprocess.TimeoutExpired:
+        return 1, "", f"{command} timed out after {timeout}s"
+
+    first_line = next(
+        (ln.strip() for ln in (result.stderr or "").splitlines() if ln.strip()),
+        "",
+    )
+    return result.returncode, result.stdout, first_line