perf(query): pyarrow.compute filters, index-backed prefix scans, shared DB + pragmas - #193
Closed
nwaughachukwuma wants to merge 1 commit into
Closed
perf(query): pyarrow.compute filters, index-backed prefix scans, shared DB + pragmas#193nwaughachukwuma wants to merge 1 commit into
nwaughachukwuma wants to merge 1 commit into
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
nwaughachukwuma
force-pushed
the
claude/core-perf-optimization-s230yb
branch
from
August 12, 2026 10:30
9ce1d7e to
0d2f4ca
Compare
…ed DB + pragmas Stacked on the bm25/FTS work (#180); the range predicate lands inside _chunk_filter_sql so both the bm25 path and the LIKE fallback become index-selective per directory instead of post-filtering the global table. - Context.filter and find --sort/--depth via vectorized pyarrow.compute instead of an in-memory SQLite round-trip (62µs vs 6.8ms per filter at 500 files; Arrow schema/types now preserved). - uri LIKE 'prefix%' → index-backed range predicates (plan flips full-table SCAN → index SEARCH, asserted in tests) across the chunk filter, mm sql, semantic search, and prune_missing; mm sql reuses fetched rows instead of a third table scan. - New chunks(content_hash, mode, extraction_id, chunk_idx) and extractions(content_hash) indexes: the per-file has_text_chunks probe was scanning every orphan chunk in the global DB. - shared_db() at every call site (one schema bootstrap per process, was 4-6 per grep -s); busy_timeout/cache_size/mmap_size/temp_store pragmas; chunks_vec existence probe memoized per connection. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012GnYsrLTmdJphKkm7yjWDV
nwaughachukwuma
force-pushed
the
claude/perf-query-plans-s230yb
branch
from
August 12, 2026 10:38
5411164 to
48c92e5
Compare
nwaughachukwuma
changed the base branch from
claude/core-perf-optimization-s230yb
to
feat/hybrid-retrieval
August 12, 2026 10:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The uri-prefix range predicate lands inside
_chunk_filter_sql, so the bm25 path and the LIKE fallback both become index-selective per directory instead of post-filtering results from the global table.Context.filter/find --sort/--depth: vectorizedpyarrow.computeinstead of round-tripping the Arrow table through an in-memory SQLite DB (which also degraded timestamps to ints — schema is now preserved).uri LIKE 'prefix%'→ index-backed range predicates: thefiles.uriPK index is never used by LIKE, so every prefix query scanned the entire global DB. Applied to the chunk filter,mm sql, semantic search, andprune_missing;mm sqlalso reuses fetched rows instead of a third table scan.chunks(content_hash, mode, extraction_id, chunk_idx),extractions(content_hash)— the per-filehas_text_chunksprobe was scanning every orphan chunk in the global DB.shared_db()everywhere (one schema bootstrap per process; a singlegrep -sdid it 4-6x) +busy_timeout(writers previously failed immediately under concurrency), 64 MB page cache, mmap, in-memory temp store;chunks_vecprobe memoized.Speed gains
Context.filter(500 files)has_text_chunksprobe (per cat'd file)Validation
Full suite passes (1,197 passed; the 2
TestRecordingFilefailures are pre-existing environment failures on the base). Query-plan invariants are unit-tested (TestQueryPlans: SEARCH-not-SCAN assertion + range/LIKE semantic equivalence); filter has an A/B benchmark against the still-shippingquery_arrow_table.🤖 Generated with Claude Code
https://claude.ai/code/session_012GnYsrLTmdJphKkm7yjWDV