Skip to content

notes_fts is never populated in production: notes have never been searchable #85

Description

@dmwyatt

notes_fts is an external-content FTS5 table over documents (v001 schema) with no production write path: upsert_documents never inserts into it or rebuilds it, and there are no triggers. transcript_fts and panels_fts are both maintained explicitly at insert time (db/transcripts.rs, db/panels.rs); notes is the one source that was never wired up.

Empirical check against the frozen benchmark snapshot (872 docs, 69 with non-empty notes), current main @ a55bec8:

grep "budget" --in notes   -> 0 meetings   (transcripts: 69)
grep "meeting" --in notes  -> 0 meetings   (transcripts: 642)
grep "review" --in notes   -> 0 meetings   (transcripts: 498)

Every notes query returns zero. --in notes is a documented search target and grep's count is an honesty contract (#65), so this silently under-reports; the notes arm of hybrid retrieval likewise contributes nothing.

Why tests never caught it: the test fixtures run INSERT INTO notes_fts(notes_fts) VALUES('rebuild') themselves, so every db-layer test exercises a freshly rebuilt index that production never has. The fix should remove that fixture step so tests go through the real maintenance path.

Fix shape: same as titles in PR #81, a rebuild in upsert_documents (or explicit row maintenance like transcripts/panels) plus a migration that backfills existing databases. Note the retrieval side effect: once notes actually contribute FTS candidates, grep counts and hybrid fusion change, so this wants the same benchmark before/after look as #75.

Found during the #75 implementation (PR #81), which deliberately left it untouched to avoid scope creep and benchmark drift.

Metadata

Metadata

Assignees

No one assigned

    Labels

    prio: p1High priority, address soonsize: mMedium, a few hours to a daystatus: readyReady to be worked ontype: bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions