Add CaseFoldingFilter with flat lookup table - #16371
Conversation
|
This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution! |
I think we've gone thru this before with the ASCIIFoldingFilter, and the most efficient way is to just tableize basic ascii (first 128) and do a switch statement for the remainder. You dont need to build explicit table, just return toLowerCase() for value < 128. its already using two-stage tables behind the scenes. Otherwise you blow away too much cache with the big lookup table. It might look good in a microbenchmark but not in an overall indexer run. Plus the data is pretty sparse (most of the codepoints dont have case), using a big lookup table is overkill. |
Add a token filter that applies Unicode simple case folding to normalize text for case-insensitive matching, without requiring the ICU analysis plugin.
Performs simple case folding for ~1200 BMP codepoints, which differs from Character.toLowerCase() such as: Greek final sigma (ς→σ), micro sign to Greek mu (µ→μ) and others. For ASCII text, folding is identical to lowercasing. Supplementary codepoints (> U+FFFF) fall back to Character.toLowerCase().
Uses a flat char[65536] lookup table (128 KB) lazily initialized on first use (once per JVM), built from ~1200 {codepoint, folded} pairs (which get discarded after initialization) generated from ICU4J via ./gradlew :lucene:core:generateFoldTable
Inspired by #14389.
Benchmark
AMD EPYC (c5a.2xlarge), JDK 25, 256 tokens x 8 chars
Baseline is Character.toLowerCase()