#75: Index titles in FTS5 so title matching is word-based and bm25-scored

[dmwyatt]

Benchmark gate: passed

Before/after results are in the comment below (93-query golden set, fts/hybrid/rerank modes, frozen snapshot). Zero per-query regressions in any mode; hybrid and rerank metrics are per-query identical to baseline, and fts gains one exact-term query (first relevant hit from outside the top 10 to rank 1) with no losses.

What changed

The titles tier of search_meetings matched title LIKE '%query%': the whole query as one contiguous substring, contributing an unscored binary title_hit. The final ORDER BY m.title_hit DESC, ... then put any title hit above every scored content match. Both properties were the bug: whole-query-substring semantics miss reordered or non-adjacent words, and the unscored binary tier floods ranking when a common word matches many recurring titles.

This indexes titles in FTS5 (titles_fts, external-content over documents(title), mirroring notes_fts) and replaces the LIKE arm with a titles_fts MATCH scored by bm25, routed through the same sanitize_fts_query as the other sources. The title_hit tier is removed; documents are ordered by their best (lowest) bm25 across all sources, then recency.

title_hit was only ever a ranking alias inside that one SQL query; it never left search_meetings. Grep and search re-derive the "title match" card independently from the evidence layer (token-based title_matches), so no user-visible indicator depends on it. It is deleted, not surfaced.

Semantics change (intended)

Title matching is now word-based (implicit AND, any order, word-boundary tokens), consistent with notes, transcripts, and panels.

Reordered and non-adjacent words now match:

title:  "Budget review"
query:  review budget          before: no match     after: match

title:  "Daily Standup - Integrations"
query:  integrations standup   before: no match     after: match

Substring-inside-a-word no longer matches:

title:  "Quarterly Planning"
query:  art                    before: match        after: no match

Grep counts and result ordering shift accordingly. This also feeds hybrid search, where the FTS list seeds keyword_ids and keyword_total.

Ranking change and rationale

The binary title tier is gone from ORDER BY. bm25 values are not calibrated across FTS tables, but the query already mixes notes/transcripts/panels bm25 via MIN(score); titles joining that same pool is the consistent choice. The post-rerank title boost in query/adjust.rs is untouched: it operates on candidate titles regardless of how candidates were retrieved.

Migration and backfill

New migration v014_titles_fts.sql creates titles_fts and backfills it from existing documents rows (INSERT INTO titles_fts(titles_fts) VALUES('rebuild')), so databases synced before this migration work without a re-sync. upsert_documents rebuilds the index after writing titles so it stays current on future syncs. Verified end-to-end against a copy of a real 875-document database: the migration applied (with the automatic pre-migration backup), PRAGMA user_version advanced to 14, all 875 titles were backfilled, and a reordered two-word query that the old substring semantics would miss returned the expected meeting.

Tests

• New db-layer regression tests: reordered words, non-adjacent words across a separator, and the substring-inside-a-word case.
• New migration tests: titles_fts searchable on a fresh database, and backfill from a row inserted before v014 is applied.
• Updated the former test_search_meetings_title_match_ranks_first (which pinned the removed title-first tier) to assert the pooled bm25 semantics: all three "kubernetes" documents are returned, order decided by bm25 rather than a title tier.
• Bumped the schema-version assertions (13 to 14) in the migration and connection tests, and added titles_fts to the hand-written integration test schema and the fixture FTS rebuilds.

Scope boundaries kept: grans show <title-substring> selection and the --meeting filter keep their substring semantics (they are selectors, not search).

Closes #75

https://claude.ai/code/session_01TmhUwR8kAqCjZFvf3vFJXQ

[claude]

Claude finished @dmwyatt's task in 3m 18s —— View job

---

Reviewing PR #75 fix/75-titles-fts

• Read PR diff and changed files
• Check CLAUDE.md guidelines (migration steps followed: new v014_titles_fts.sql, registered in migrations(), total bumped to 14, tests/common/mod.rs fixture schema updated, migration tests added)
• Review migration SQL and schema changes
• Review db/meetings.rs ranking/query changes
• Review sync.rs / test fixtures updates
• Post findings

Review notes

I read the diff against origin/main, the new v014_titles_fts.sql migration, the reworked search_meetings UNION/ranking query in src/db/meetings.rs, the upsert_documents rebuild hook in src/db/sync.rs, and the migration/fixture/doc updates. Specifically checked for:

• Leftover references to the removed title_hit column (none — grep'd the whole tree).
• titles_fts JOIN correctness (titles_fts.rowid = d.rowid, matching the existing notes_fts/transcript_fts pattern exactly).
• Consumers of search_meetings (hybrid.rs, grep.rs, search.rs, benchmark/retriever.rs) — none depend on the removed title-tier ordering; hybrid.rs only uses doc IDs/counts.
• Migration bookkeeping: version bump (13→14) is consistent across connection.rs, migrations/mod.rs, and tests/common/mod.rs; new migration tests cover both a fresh DB and backfill from a pre-v014 row.
• DROP TABLE IF EXISTS + rebuild pattern mirrors the existing notes_fts migration (v001_initial_schema.sql) exactly, so it's consistent with established precedent in this codebase.
• Docs (README.md, SPEC.md, docs/database-schema.md) accurately reflect the new word-based/BM25 semantics.

No bugs or code-quality issues met the confidence bar for this review. The change is a clean, well-mirrored extension of the existing FTS5 pattern with solid test coverage (reordered words, non-adjacent words, substring-inside-word rejection, migration backfill).

Code review: no issues found.

[claude]

Claude finished @dmwyatt's task in 56s —— View job

---

Security Review

• Gather context (diff, changed files)
• Review FTS5 query construction / SQL injection risk
• Review migration SQL for injection/path issues
• Review auth token handling (unrelated files touched?)
• Review dependency changes
• Post findings

Security review: no issues found in this PR.

Notes from the review (informational, not blocking):

• The new titles_fts MATCH ? arm in search_meetings (src/db/meetings.rs) binds the sanitized query as a parameter rather than interpolating it into the SQL string, consistent with the existing notes_fts/transcript_fts/panels_fts arms. No SQL injection.
• v014_titles_fts.sql and the upsert_documents rebuild statement (src/db/sync.rs) contain no user-controlled input — they're static DDL/DML.
• No unsafe blocks, no new dependencies (Cargo.toml untouched), no auth-token or file-path handling touched by this diff.

[dmwyatt]

Search quality benchmark: candidate vs baseline

Benchmarked against the frozen 872-doc snapshot with the v2 golden set (93 queries, k=10). Baseline is origin/main a55bec8; candidate is this branch at 1489ea1 (snapshot copy migrated to v14, titles_fts backfilled with 872 rows). One controlled variable: notes_fts is empty in production databases (#85), identically so on both sides, so notes contribute nothing to either run.

Aggregate deltas (candidate minus baseline)

mode	hit-rate@10	recall@10	MRR@10	avg latency
fts	0.1720 → 0.1828 (+0.0108)	0.0934 → 0.0987 (+0.0054)	0.1392 → 0.1499 (+0.0108)	6.9 → 0.9 ms
hybrid	0.8602 → 0.8602 (±0)	0.7593 → 0.7593 (±0)	0.7220 → 0.7220 (±0)	130.3 → 113.3 ms
rerank-jina	0.9355 → 0.9355 (±0)	0.8054 → 0.8054 (±0)	0.8135 → 0.8135 (±0)	334.2 → 300.2 ms

Per-query wins / losses / ties

mode	wins	losses	ties
fts	1	0	92
hybrid	0	0	93
rerank-jina	0	0	93

The single fts win is an exact-term query that went from no relevant result in the top 10 to a relevant result at rank 1 (recall 0.00 → 0.50). Hybrid and rerank-jina are metric-identical to baseline on every query.

Stratum-level aggregates (baseline → candidate)

fts

stratum	n	hit-rate	recall	MRR
exact-term	17	94.1% → 100.0%	51.1% → 54.0%	0.761 → 0.820
mixed	38	0.0% → 0.0%	0.0% → 0.0%	0.000 → 0.000
paraphrase	38	0.0% → 0.0%	0.0% → 0.0%	0.000 → 0.000

hybrid and rerank-jina: all strata unchanged (exact-term 100.0% hit both sides; mixed and paraphrase identical to baseline in every metric).

Verdict

Net gain: the fts titles arm improves on exact-term queries with zero regressions in any mode, the end-to-end hybrid and rerank pipelines are quality-neutral, and fts latency drops from 6.9 ms to 0.9 ms avg (bm25 MATCH replacing the LIKE scan).

https://claude.ai/code/session_01TmhUwR8kAqCjZFvf3vFJXQ