Skip to content

Add CaseFoldingFilter with flat lookup table - #16371

Open
costin wants to merge 4 commits into
apache:mainfrom
costin:lucene/case-folding-table
Open

Add CaseFoldingFilter with flat lookup table#16371
costin wants to merge 4 commits into
apache:mainfrom
costin:lucene/case-folding-table

Conversation

@costin

@costin costin commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Add a token filter that applies Unicode simple case folding to normalize text for case-insensitive matching, without requiring the ICU analysis plugin.

Performs simple case folding for ~1200 BMP codepoints, which differs from Character.toLowerCase() such as: Greek final sigma (ς→σ), micro sign to Greek mu (µ→μ) and others. For ASCII text, folding is identical to lowercasing. Supplementary codepoints (> U+FFFF) fall back to Character.toLowerCase().

Uses a flat char[65536] lookup table (128 KB) lazily initialized on first use (once per JVM), built from ~1200 {codepoint, folded} pairs (which get discarded after initialization) generated from ICU4J via ./gradlew :lucene:core:generateFoldTable

Inspired by #14389.

Benchmark

AMD EPYC (c5a.2xlarge), JDK 25, 256 tokens x 8 chars
Baseline is Character.toLowerCase()

Distribution Baseline (ops/ms) FoldTable (ops/ms) Speedup
english 0.0739 0.0774 1.05x
german 0.0699 0.0882 1.26x
turkish 0.0565 0.1044 1.85x
russian 0.0137 0.1676 12.23x
greek 0.0136 0.1649 12.14x
armenian 0.0138 0.1673 12.13x
japanese 0.0180 0.1653 9.16x
cherokee 0.0101 0.1676 16.62x
mixed 0.0178 0.1046 5.86x

@github-actions

Copy link
Copy Markdown
Contributor

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

@github-actions github-actions Bot added the Stale label Jul 23, 2026
@rmuir

rmuir commented Jul 23, 2026

Copy link
Copy Markdown
Member

Uses a flat char[65536] lookup table (128 KB) lazily initialized on first use (once per JVM), built from ~1200 {codepoint, folded} pairs (which get discarded after initialization) generated from ICU4J via ./gradlew :lucene:core:generateFoldTable

I think we've gone thru this before with the ASCIIFoldingFilter, and the most efficient way is to just tableize basic ascii (first 128) and do a switch statement for the remainder. You dont need to build explicit table, just return toLowerCase() for value < 128. its already using two-stage tables behind the scenes.

Otherwise you blow away too much cache with the big lookup table. It might look good in a microbenchmark but not in an overall indexer run. Plus the data is pretty sparse (most of the codepoints dont have case), using a big lookup table is overkill.

@github-actions github-actions Bot removed the Stale label Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants