fix(search): FTS5 special-char queries (hyphens, dots, paths) now match via phrase-prefix#27
Open
chokmah-me wants to merge 1 commit into
Open
Conversation
Queries containing hyphens, dots, slashes etc. (e.g. 'session-recall', 'CLAUDE.md', 'src/main.py') previously got quoted whole with no prefix star, so they only matched as exact adjacent phrases and usually returned nothing. Now such tokens are split on the special characters into the subtokens unicode61 actually indexes and searched as a phrase-prefix query: session-recall -> "session recall"*. Also handles tokens that are all special chars (dropped; all-special queries return None like empty queries). Found by Claude (Fable 5) during a Bitter Lesson review of a real user's harness; the user greenlit submitting the fix upstream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Provenance
This fix was found and authored by Claude (Fable 5, Anthropic) while running a "Bitter Lesson" review over a real user's Claude Code harness — auditing which parts of the setup were prompt-space workarounds for tool bugs. The user's global instructions carried this warning:
That's a documentation patch for a fixable tool behavior, so we fixed the tool instead and deleted the warning. The user (@chokmah-me) reviewed and greenlit submitting the fix upstream.
Problem
sanitize_fts5_queryquotes any token containing FTS5 special characters wholesale, without a prefix star:session-recall→"session-recall"— an exact adjacent-phrase query with no prefix matchingCLAUDE.md→"CLAUDE.md",src/main.py→"src/main.py"— sameIn practice these queries usually return nothing, which pushes users into exactly the workaround quoted above.
Fix
Tokens containing special characters are split on those characters into the subtokens unicode61 actually indexes, then searched as a phrase-prefix query:
session-recall"session-recall""session recall"*CLAUDE.md"CLAUDE.md""CLAUDE md"*src/main.py"src/main.py""src main py"*!!!(all special)"!!!"(syntax risk)NonePlain tokens keep their existing
token*behavior.Verification
test_search_sanitize.pyupdated to pin the new behavior, +2 new tests (all-special query → None, pure-special token dropped): 15/15 passmainin the same environment — the 16 failures are pre-existing local/Windows issues (symlink-privilege tests, env-dependent e2e fixtures), verified identical before and after the change.sr-index.db:search 'session-recall'andsearch 'CLAUDE.md'went from fragile/empty to returning correct hits🤖 Generated with Claude Code