Skip to content

docs(search): document English stemming for the built-in full-text engine - #2032

Merged
lukekim merged 1 commit into
trunkfrom
docs/fts-english-stemming
Aug 1, 2026
Merged

docs(search): document English stemming for the built-in full-text engine#2032
lukekim merged 1 commit into
trunkfrom
docs/fts-english-stemming

Conversation

@claudespice

Copy link
Copy Markdown
Collaborator

Summary

spiceai/spiceai#12220 changes how the built-in full-text engine analyzes text: the Tantivy index now uses the en_stem tokenizer (English Snowball stemming, lowercasing, WithFreqsAndPositions so phrase queries keep working) instead of Tantivy's default TEXT options. Tantivy's QueryParser is constructed with the index's tokenizer manager, so query terms are stemmed the same way — running now matches documents containing run and runs.

This is user-visible matching behavior with no configuration knob, and the Full-text Search page said nothing about analysis at all. Adds a short Text Analysis section under Engines covering:

  • what the built-in engine does (stem + lowercase, positions retained), and that it applies at query time too;
  • that it is always on, has no parameter, and is English-only;
  • the Elasticsearch nuance — the local warm index is a Tantivy index and analyzes the same way, while searches served directly by Elasticsearch (multi-column datasets, a warm index that could not be built, or an on_zero_results: use_source fallback) use Elasticsearch's own analyzer.

Source PRs

Doc pages changed (1)

  • website/docs/features/search/full-text.md — vNext only

Scoped to vNext: en_stem is not present at v2.1.2 (git grep en_stem v2.1.2 -- crates/search/ → no hits), so no versioned snapshot needs it.

Test plan

  • cd website && npm run build passes (the added #warm-tier link resolves to the #### Warm tier heading on the same page)
  • Versioned-docs propagation checked — addition, vNext only
  • Files updated: 1

…gine

spiceai/spiceai#12220 switches the built-in Tantivy full-text index from
the default tokenizer to `en_stem` (English Snowball stemming, with
positions retained). Indexed and query terms are both stemmed, so
`running` now matches `run`/`runs`.

vNext only — the change is not in v2.1.2.
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

✅ Pull with Spice Passed

Passing checks:

  • ✅ Title meets minimum length requirement (10 characters)
  • ✅ Has at least one of the required labels: area/blog, area/docs, area/cookbook, dependencies
  • ✅ No banned labels detected
  • ✅ Has at least one assignee: claudespice

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

🚀 deployed to https://96ad88d9.spiceai-org-website.pages.dev

@claudespice

Copy link
Copy Markdown
Collaborator Author

@copilot review

@lukekim
lukekim merged commit 51de6f3 into trunk Aug 1, 2026
8 of 15 checks passed
@lukekim
lukekim deleted the docs/fts-english-stemming branch August 1, 2026 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants