Late-interaction: memory-mapped binary patch store

[Davidobot]

Summary

Late-interaction retrieval (LodeLateInteractionIndex, #25) stores each document's patch matrix as base64 in the per-row text sidecar. The resident exact scan decodes every document into one in-memory float32/float16 matrix on build; corpora over resident_max_bytes fall back to a streaming path that re-reads and base64-decodes documents from disk on every query. Both the build cost and the streaming cost are dominated by base64 decode and full materialization.

A raw-binary, memory-mapped patch store would remove both: store the patch matrices as a contiguous binary file (plus a small offset/dtype index) and mmap it, so:

• the resident matrix is the mmap itself (or a cheap view), with no base64 decode on build;
• corpora past the RAM budget are served from the OS page cache via the mmap instead of re-decoding per query, so the over-budget path is fast (page-cache bound) rather than disk-decode bound;
• on-disk size drops by the ~33% base64 overhead.

Sketch

• A binary sidecar (e.g. patches.libin) holding all patch matrices concatenated at the index's storage precision (float32 / float16 / int8 + per-vector scales), with a parallel offset/count index keyed by document id. Keep it outside the engine's *.json scan (as the existing .meta config does).
• Build the resident matrix as an mmap view; for int8/float16 keep the GEMM upcast chunked as today.
• Maintain it incrementally alongside the existing resident delta: append new documents to the binary file, tombstone removed ones, compact periodically (mirrors the in-memory delta/compaction already in place).
• Keep the current base64-sidecar path as a fallback for read-only handles or where mmap is unavailable.

Out of scope

• Changing the MaxSim kernel or the engine's vector storage.
• The candidate-pruning / ANN direction for very large corpora (separate concern).

Follow-up to #25.

[Davidobot]

Prior art / context

Main track today uses no mmap. The vendored TurboVec loads .tvim/.tv snapshots with BufReader + read_exact into in-memory Vecs (packed codes + scales); the Python engine has no mmap. Persistence is a binary TurboVec snapshot plus O(changed) delta journals (.tvd/.jsd/…) and WAL — a delta-append design, not a memory-mapped one.

This was researched in the efficient-embeddings lineage. That repo implemented a binary columnar snapshot (.eeisnap) served via mmap + mmap.madvise(MADV_WILLNEED), with an append-update-log and periodic compaction; the research evidence records it as applied/useful (binary_snapshot_advice.json: method=mmap.madvise, advice=willneed, status=applied). Notably, that append-log + compaction shape is the same one the late-interaction resident cache already uses in memory (#38).

But the productionized form there is the delta-journaled binary snapshot, not the mmap columnar one. The .eevec/.eerank/.eeisnap cascade (including the mmap columnar snapshot) was removed in favor of the binary .tvim TurboVec snapshot + delta-persistence — which is exactly what LodeDB ships on the main track.

Implications for this issue:

• The madvise(WILLNEED) + binary columnar snapshot + append-log/compaction pattern is validated prior art and a good basis for the patch-store here.
• Prefer aligning with the existing delta-journal persistence model (binary base + O(changed) deltas) over a bespoke standalone format, so the patch store gets crash-atomicity and cheap incremental persistence for free and matches the rest of the engine.

[Davidobot]

Related non-blocking item from the PR #38 review loop: the streaming (over-budget) path currently enumerates the full (document_id, patch_count) list before scoring, so document enumeration is O(n_docs) in metadata even though patch decode/scoring is chunk-bounded. A native/mmap patch store should also expose paginated document enumeration so very large over-budget scans are bounded in total memory, not just in patch-decode memory.