Summary
This is an umbrella tracking issue for vector-path performance work. It was prompted by a source-level comparison against NornicDB (a Go graph+vector DB with a Metal/CUDA/Vulkan acceleration path), and the comparison mostly confirms LodeDB's current direction rather than redirecting it.
The headline conclusion: we should not move search-time graph traversal or candidate generation onto the GPU. NornicDB keeps graph traversal and graph mutation on the CPU, puts only dense embedding/scoring work on the GPU, and its GPU exact re-scorer currently falls back to CPU because subset re-scoring can regress latency. Its basic CUDA top-k path also copies all scores back to host before selection, which is the "GPU math is fast, host plumbing is slow" trap. LodeDB already sits on the better side of that tradeoff: a resident fp16 copy, tiled matmul, a streaming device top-k that never materializes the full batch x corpus score matrix, a CPU kernel that stays the source of truth and single-query path, and O(changed) persistence.
So the useful takeaways are not "more GPU search". They are build-time work, an opt-in approximate mode for very large CPU corpora, a generation-keyed result cache, sharper filtered-scan routing, and a benchmark sweep to set the thresholds. Each is broken out below with concrete hooks and a priority. The intent is that the higher-priority items become their own issues that link back here.
What the comparison confirmed about LodeDB (verified against source)
- Resident fp16 + tiled matmul + streaming device top-k, CPU kernel is source of truth and single-query path: confirmed in
src/lodedb/engine/gpu_turbovec.py (_tiled_top_k_torch, the per-tile topk merge) and the routing in src/lodedb/engine/runtime_policy.py.
- Filtered/allowlist scan masks non-allowed rows to
-inf so top-k stays k rather than widening to the corpus: confirmed in gpu_turbovec.py (search_batch) and core.py (stable_ids_for_chunks).
- Routing fails closed under
required for eligible batches, single queries stay on CPU without raising, and dep/memory failures fall back visibly via telemetry: confirmed in runtime_policy.py (gpu_direct_turbovec_should_use) and core.py fallback fields (gpu_stage_one_status, gpu_fallback_reason).
- MPS off by default because NEON beat MPS at every batch size on the measured M1: confirmed in
runtime_policy.py and docs/benchmarks.md.
- Base
.tvim + .tvd deltas with checksum/fingerprint validation and O(changed) persistence: confirmed in src/lodedb/engine/turbovec_delta_store.py.
One correction to the comparison: the "A10 crossover around batch 50" figure is from a now-obsolete benchmark. After the 2026-06-13 torch.topk swap the resident scan beats the CPU kernel at every batch >= 2 (see the current benchmarks/direct_gpu_sweep numbers and the note in runtime_policy.py). The remaining CPU-favored cases are single queries (batch 1, bypassed by design) and small filtered subsets, which motivates item 1 below rather than a batch-size threshold.
Current state confirmed by reading the tree: there is no query result cache, no IVF / k-means / coarse-cluster mode, no GPU admission threshold beyond the hardcoded minimum batch of 2, and BM25 is a query-time ranker only (no build-time seeding or locality ordering). So items 1 through 5 are all genuinely new.
Work items
1. Filtered / selectivity-aware routing thresholds
The allowlist scan already maps filtered chunk ids to stable ids and masks the rest to -inf, but admission is governed only by batch size (minimum 2). A tiny filtered subset inside an otherwise-eligible batch still goes to the GPU, where CPU NEON over a few hundred to a few thousand candidates would win because orchestration dominates.
Add admission knobs alongside the existing LODEDB_GPU_DIRECT_* env vars: gpu_allowlist_min_rows (route below this to CPU even in a batch), and consider gpu_max_copy_back_bytes. Large allowlists and repeated identical-filter multi-query workloads keep the resident path. Hooks: runtime_policy.py (gpu_direct_turbovec_should_use), core.py (_try_query_resident_direct_batch). Cheap and directly actionable. Priority: high.
2. Generation-keyed query result cache
NornicDB caches Search() results by query + options with LRU and a 5m TTL, invalidated on index mutation. We have nothing equivalent. This does nothing for one-off RAG queries but helps agent loops, repeated memory recall, benchmark harnesses, and UI retries.
Keep it conservative and local. Cache only when the query carries text, include is small, and no raw payload is requested. Key on index generation + mode + top_k + filter signature + route profile + model/provider/task + query-text-or-embedding hash. Invalidate by generation, which we already track in the commit manifest, so invalidation is free and correct by construction. Separately, evaluate a small query-embedding cache: for single-query RAG the embedding model is the dominant cost (see issue #27 motivation), so caching the encode of a repeated query string may help more than caching results. Priority: high.
3. Benchmark: resident-policy sweep
Foundational, since it sets the thresholds for item 1 and the go/no-go for items 4 and 5. Sweep CPU vs CUDA vs MPS over corpus size, dim, bit width, batch size, top_k, filter selectivity, and allowlist size, and report the admission crossovers. Extend benchmarks/direct_gpu_sweep. Priority: high.
4. Native storage as the real query-latency lever
Scoring is already native and, for late interaction, measured at roughly 1% of query time (see PR #38 / issue #25). The dominant costs on the Python-bound path are engine bookkeeping, load/cold-start, and (for multi-vector) per-document matrix assembly. The largest durable wins are therefore in storage and the core, not in more GPU scoring:
This item is mostly a pointer that links the existing core/storage issues into the performance picture so they are weighed against the GPU-side work. Priority: high (via the linked issues).
5. GPU-assisted TurboVec build/encode (experiment)
NornicDB's most copyable GPU idea is build-time: it uses the GPU only to compute candidate neighborhoods during HNSW construction and keeps graph mutation on the CPU. We have no HNSW graph, but build does real work: rotation, quantization to 2/4-bit codes, scale computation, reconstruction, and resident upload, all CPU-only in the vendored Rust core today.
Prototype batching embeddings on the GPU, applying the TurboVec rotation/projection, computing scales and codes, and feeding encoded rows through the existing add_encoded/delta APIs. Target metric is build seconds and resident-upload time, not query latency, and .tvim/.tvd semantics stay unchanged. Note the architectural cost: encode lives in Rust, so a GPU encode path means crossing into Python/cupy during build. Bench first to confirm the build-time win justifies that. Priority: medium, experimental.
6. Opt-in approximate IVF / coarse-cluster mode
NornicDB has an IVF-HNSW candidate generator (k-means routes to nearest clusters, then per-cluster search). The analogue for us is explicitly approximate (mode="approx_ivf" or similar): route to the N nearest centroids, then run the exact TurboVec scan only inside selected clusters. This is a single-query escape hatch for corpora where exact full scan exceeds the latency budget. Exact stays the default; the mode reports recall/latency telemetry clearly. Gate on item 3 demonstrating a real corpus where exact full scan crosses the budget, since this adds a recall cliff and meaningful complexity to an engine whose pitch is exact + small. Priority: medium, gated.
7. Build ordering: lexical / metadata / cluster locality
NornicDB's BM25-seeded insertion cut a 1M build from 27 to 10 minutes (2.7x) by inserting lexically diverse high-IDF seeds first (256 terms x 8 docs = 2,048 seeds), reusing the seeds for k-means++ init. Important caveat: that win is specific to HNSW graph construction. LodeDB's exact scan recall does not depend on row order, so the headline speedup does not transfer. Row order can still matter for page-cache locality, filtered-allowlist density, .tvd compaction, and any future IVF/ANN mode. Therefore this is contingent on item 6: only worth a benchmark flag (sort/group rows by cluster or metadata before building the compact index, measure scan throughput, filtered-scan throughput, and compaction) once an approximate mode exists. Priority: low, contingent on item 6.
8. Docs: CPU control plane vs GPU dense-scoring plane
Add a short architecture note stating why LodeDB does not run GPU HNSW traversal and why single queries stay on CPU: CPU handles branching, filters, metadata, lexical/graph control flow, and single-query exact scans; the GPU handles dense resident batch scoring only when batch and filter shape amortize orchestration. This preempts the recurring "why not GPU HNSW?" question. Hook: docs/benchmarks.md or a new docs/architecture note. Priority: low, cheap.
What we deliberately will not do
- Copy NornicDB's CUDA top-k shape that copies all scores to host before selection. Our streaming device top-k is the right side of that tradeoff.
- Make float32 GPU residency the primary serving format. The compact 2/4-bit CPU index plus an optional fp16 resident copy is a better memory story (1M x 1024 float32 is roughly 4 GB of VRAM).
- Move HNSW traversal or candidate-graph walks to the GPU.
- Default MPS on. NEON was faster than MPS at every batch size on the hardware we measured.
Already shipped (not in scope here)
Summary
This is an umbrella tracking issue for vector-path performance work. It was prompted by a source-level comparison against NornicDB (a Go graph+vector DB with a Metal/CUDA/Vulkan acceleration path), and the comparison mostly confirms LodeDB's current direction rather than redirecting it.
The headline conclusion: we should not move search-time graph traversal or candidate generation onto the GPU. NornicDB keeps graph traversal and graph mutation on the CPU, puts only dense embedding/scoring work on the GPU, and its GPU exact re-scorer currently falls back to CPU because subset re-scoring can regress latency. Its basic CUDA top-k path also copies all scores back to host before selection, which is the "GPU math is fast, host plumbing is slow" trap. LodeDB already sits on the better side of that tradeoff: a resident fp16 copy, tiled matmul, a streaming device top-k that never materializes the full batch x corpus score matrix, a CPU kernel that stays the source of truth and single-query path, and O(changed) persistence.
So the useful takeaways are not "more GPU search". They are build-time work, an opt-in approximate mode for very large CPU corpora, a generation-keyed result cache, sharper filtered-scan routing, and a benchmark sweep to set the thresholds. Each is broken out below with concrete hooks and a priority. The intent is that the higher-priority items become their own issues that link back here.
What the comparison confirmed about LodeDB (verified against source)
src/lodedb/engine/gpu_turbovec.py(_tiled_top_k_torch, the per-tile topk merge) and the routing insrc/lodedb/engine/runtime_policy.py.-infso top-k stays k rather than widening to the corpus: confirmed ingpu_turbovec.py(search_batch) andcore.py(stable_ids_for_chunks).requiredfor eligible batches, single queries stay on CPU without raising, and dep/memory failures fall back visibly via telemetry: confirmed inruntime_policy.py(gpu_direct_turbovec_should_use) andcore.pyfallback fields (gpu_stage_one_status,gpu_fallback_reason).runtime_policy.pyanddocs/benchmarks.md..tvim+.tvddeltas with checksum/fingerprint validation and O(changed) persistence: confirmed insrc/lodedb/engine/turbovec_delta_store.py.One correction to the comparison: the "A10 crossover around batch 50" figure is from a now-obsolete benchmark. After the 2026-06-13
torch.topkswap the resident scan beats the CPU kernel at every batch >= 2 (see the currentbenchmarks/direct_gpu_sweepnumbers and the note inruntime_policy.py). The remaining CPU-favored cases are single queries (batch 1, bypassed by design) and small filtered subsets, which motivates item 1 below rather than a batch-size threshold.Current state confirmed by reading the tree: there is no query result cache, no IVF / k-means / coarse-cluster mode, no GPU admission threshold beyond the hardcoded minimum batch of 2, and BM25 is a query-time ranker only (no build-time seeding or locality ordering). So items 1 through 5 are all genuinely new.
Work items
1. Filtered / selectivity-aware routing thresholds
The allowlist scan already maps filtered chunk ids to stable ids and masks the rest to
-inf, but admission is governed only by batch size (minimum 2). A tiny filtered subset inside an otherwise-eligible batch still goes to the GPU, where CPU NEON over a few hundred to a few thousand candidates would win because orchestration dominates.Add admission knobs alongside the existing
LODEDB_GPU_DIRECT_*env vars:gpu_allowlist_min_rows(route below this to CPU even in a batch), and considergpu_max_copy_back_bytes. Large allowlists and repeated identical-filter multi-query workloads keep the resident path. Hooks:runtime_policy.py(gpu_direct_turbovec_should_use),core.py(_try_query_resident_direct_batch). Cheap and directly actionable. Priority: high.2. Generation-keyed query result cache
NornicDB caches
Search()results by query + options with LRU and a 5m TTL, invalidated on index mutation. We have nothing equivalent. This does nothing for one-off RAG queries but helps agent loops, repeated memory recall, benchmark harnesses, and UI retries.Keep it conservative and local. Cache only when the query carries text,
includeis small, and no raw payload is requested. Key on index generation + mode + top_k + filter signature + route profile + model/provider/task + query-text-or-embedding hash. Invalidate by generation, which we already track in the commit manifest, so invalidation is free and correct by construction. Separately, evaluate a small query-embedding cache: for single-query RAG the embedding model is the dominant cost (see issue #27 motivation), so caching the encode of a repeated query string may help more than caching results. Priority: high.3. Benchmark: resident-policy sweep
Foundational, since it sets the thresholds for item 1 and the go/no-go for items 4 and 5. Sweep CPU vs CUDA vs MPS over corpus size, dim, bit width, batch size, top_k, filter selectivity, and allowlist size, and report the admission crossovers. Extend
benchmarks/direct_gpu_sweep. Priority: high.4. Native storage as the real query-latency lever
Scoring is already native and, for late interaction, measured at roughly 1% of query time (see PR #38 / issue #25). The dominant costs on the Python-bound path are engine bookkeeping, load/cold-start, and (for multi-vector) per-document matrix assembly. The largest durable wins are therefore in storage and the core, not in more GPU scoring:
lodedb-coreconsolidation (issue Consolidate the engine into a native lodedb-core crate with thin language bindings #27) collapses per-operation PyO3 crossings and moves BM25/RRF, filter evaluation, and delta/WAL bookkeeping into Rust.BufReader/read_exactwith no mmap. An mmap-backed resident load cuts cold-start and large-index open latency.This item is mostly a pointer that links the existing core/storage issues into the performance picture so they are weighed against the GPU-side work. Priority: high (via the linked issues).
5. GPU-assisted TurboVec build/encode (experiment)
NornicDB's most copyable GPU idea is build-time: it uses the GPU only to compute candidate neighborhoods during HNSW construction and keeps graph mutation on the CPU. We have no HNSW graph, but build does real work: rotation, quantization to 2/4-bit codes, scale computation, reconstruction, and resident upload, all CPU-only in the vendored Rust core today.
Prototype batching embeddings on the GPU, applying the TurboVec rotation/projection, computing scales and codes, and feeding encoded rows through the existing
add_encoded/delta APIs. Target metric is build seconds and resident-upload time, not query latency, and.tvim/.tvdsemantics stay unchanged. Note the architectural cost: encode lives in Rust, so a GPU encode path means crossing into Python/cupy during build. Bench first to confirm the build-time win justifies that. Priority: medium, experimental.6. Opt-in approximate IVF / coarse-cluster mode
NornicDB has an IVF-HNSW candidate generator (k-means routes to nearest clusters, then per-cluster search). The analogue for us is explicitly approximate (
mode="approx_ivf"or similar): route to the N nearest centroids, then run the exact TurboVec scan only inside selected clusters. This is a single-query escape hatch for corpora where exact full scan exceeds the latency budget. Exact stays the default; the mode reports recall/latency telemetry clearly. Gate on item 3 demonstrating a real corpus where exact full scan crosses the budget, since this adds a recall cliff and meaningful complexity to an engine whose pitch is exact + small. Priority: medium, gated.7. Build ordering: lexical / metadata / cluster locality
NornicDB's BM25-seeded insertion cut a 1M build from 27 to 10 minutes (2.7x) by inserting lexically diverse high-IDF seeds first (256 terms x 8 docs = 2,048 seeds), reusing the seeds for k-means++ init. Important caveat: that win is specific to HNSW graph construction. LodeDB's exact scan recall does not depend on row order, so the headline speedup does not transfer. Row order can still matter for page-cache locality, filtered-allowlist density,
.tvdcompaction, and any future IVF/ANN mode. Therefore this is contingent on item 6: only worth a benchmark flag (sort/group rows by cluster or metadata before building the compact index, measure scan throughput, filtered-scan throughput, and compaction) once an approximate mode exists. Priority: low, contingent on item 6.8. Docs: CPU control plane vs GPU dense-scoring plane
Add a short architecture note stating why LodeDB does not run GPU HNSW traversal and why single queries stay on CPU: CPU handles branching, filters, metadata, lexical/graph control flow, and single-query exact scans; the GPU handles dense resident batch scoring only when batch and filter shape amortize orchestration. This preempts the recurring "why not GPU HNSW?" question. Hook:
docs/benchmarks.mdor a newdocs/architecturenote. Priority: low, cheap.What we deliberately will not do
Already shipped (not in scope here)
requiredrouting.