From eabf58bbbdfcc65dba1f6a1c355faeccf19a7679 Mon Sep 17 00:00:00 2001 From: claudespice Date: Sat, 1 Aug 2026 03:25:36 -0700 Subject: [PATCH] docs(search): document English stemming for the built-in full-text engine MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit spiceai/spiceai#12220 switches the built-in Tantivy full-text index from the default tokenizer to `en_stem` (English Snowball stemming, with positions retained). Indexed and query terms are both stemmed, so `running` now matches `run`/`runs`. vNext only — the change is not in v2.1.2. --- website/docs/features/search/full-text.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/website/docs/features/search/full-text.md b/website/docs/features/search/full-text.md index ed05bbb4f..97fbd3282 100644 --- a/website/docs/features/search/full-text.md +++ b/website/docs/features/search/full-text.md @@ -26,6 +26,14 @@ Spice supports two full-text search engines: When no engine is specified, Tantivy is used automatically. +### Text Analysis + +The built-in Tantivy engine indexes text with Tantivy's `en_stem` tokenizer: terms are lowercased and reduced to their English (Snowball) stem, with token positions retained so phrase queries keep working. Query terms are analyzed the same way, so a search for `running` also matches documents containing `run` and `runs`. + +Stemming is always on for the built-in engine and has no configuration parameter. It is English-only — text in other languages is still tokenized and lowercased, but not stemmed. + +The local warm index used with `engine: elasticsearch` is a Tantivy index and analyzes text the same way. Searches [served directly by Elasticsearch](#warm-tier) — multi-column datasets, a warm index that could not be built, or an `on_zero_results: use_source` fallback — are analyzed by Elasticsearch's own analyzer for that index instead. + ## Enabling Full-Text Search To enable full-text search, configure your dataset columns within your dataset definition as follows: