add support for fts in mm grep using bm25

[nwaughachukwuma]

Summary

BM25 (Best Match 25) is the standard ranking function used by full-text search (fts) engines to score and rank documents by relevance to a query. This PR allows using fts with bm25 when --semantic/-s flag is specified and falls back to a %LIKE% query.

  flowchart LR                                                                                          
      A["search_chunks_fts()"] --> B{"FTS5?"}                                                   
      B -- yes --> C["search_chunks_bm25"]                                                              
      B -- no --> F["LIKE fallback"]                                                                    
      C --> D{"query ≥ 3 chars?"}                                                                       
      D -- no --> E["return []"]                                                                
      D -- yes --> G["build filters + phrase"]                                                          
      G --> I["FTS5 MATCH + bm25&#40;&#41;<br/>ORDER BY bm25 LIMIT ?"]                                  
      I --> J{"non-empty?"}                                                                             
      J -- yes --> L["return BM25 rows"]                                                                
      J -- no --> F                                                                                     
      F --> N["LIKE %q% scan<br/>→ return rows"]                                                        

Loading

[gemini-code-assist]

Code Review

This pull request introduces FTS5 full-text search capabilities over the chunks table using a trigram tokenizer and BM25 ranking, including automatic trigger-based synchronization and idempotent backfilling of legacy data. The reviewer feedback highlights two key improvements: first, avoiding a redundant and expensive fallback to a LIKE scan when FTS5 is available but returns no results; second, using the more robust pragma module_list instead of pragma compile_options to detect FTS5 support and prevent false negatives on certain SQLite builds.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.