feat(zipdaf): mmap zero-copy reads for dense STORE'd components (0.9.0) - #13
Merged
Conversation
ZipDaf dense reads of native little-endian Float64 / signed Int32 components are now memory-mapped (zero-copy ALTREP) when the store is opened read-only, matching files_daf(), h5df(), and DataAxesFormats.jl's ZipDaf. The archive writer already 8-byte-aligns every entry's data region (compute_alignment_padding), so mmap fires for any dense read; the alignment check is a belt-and-suspenders guard that also covers Julia-written stores. New C++ helper dafr_mmap_zip_stored_offset(xptr, key) returns the file byte offset + length of a STORE'd entry (or NULL if absent/compressed); R's .zip_mmap_dense() gates on it and opens an independent read-only mmap of the .daf.zip via mmap_real/mmap_int, so the ALTREP owns its own MmapRegion independent of the store/daf lifetime. Only read-only opens mmap: a writable store keeps pending appends in an in-memory overlay not present in the file mmap, so those offsets would be out of bounds - writable-mode reads fall back to eager. Compressed/packed components and non-native dtypes also fall back. Opt out with options(dafr.mmap = FALSE). On a 160 MB dense matrix a lazy get drops from ~280 ms to sub-millisecond and a full read reaches the files_daf mmap ceiling (~44 ms), closing the last measured perf-parity gap. No on-disk format change; Julia interop preserved. Tests in test-zip-daf-mmap.R (8). Claude-Session: https://claude.ai/code/session_01Av1oZxwSmuUXT4kDjnqjE9
The 7 mmap tests all build a zip_daf, which errors on Windows where MmapZipStore is not compiled (src/mmap_zip_store_win_stubs.cpp). Guard each with skip_if_no_mmap_zip(), matching test-zip-daf.R and the other mmap-zip-store test files. Claude-Session: https://claude.ai/code/session_01Av1oZxwSmuUXT4kDjnqjE9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ZipDaf dense reads of native little-endian Float64 / signed Int32 components are now memory-mapped (zero-copy ALTREP) when the store is opened read-only, matching
files_daf(),h5df(), andDataAxesFormats.jl'sZipDaf. This closes the last measured perf-parity gap with DAF.jl - all three backends now mmap large dense reads.Result
160 MB dense Float64 matrix (warm cache,
empty_cacheper iter, median of 11;dev/bench/mmap-bench.R):getget_sumZipDaf now matches FilesDaf and beats H5df on lazy reads (no hdf5r object open).
What changed
dafr_mmap_zip_stored_offset(xptr, key)->c(offset, nbytes)for a STORE'd entry (viastored_view,p - file_mmap_base()), orNULLif the key is absent or compressed. Windows stub added..zip_mmap_dense()gates like.h5_mmap_dense:dafr.mmapon,n > 0,mode == "r", eltype in {Float64, signed Int32},nbytes == n*eltsize, element-aligned offset. Opens an independent read-only mmap of the.daf.zipviammap_real/mmap_int, so the ALTREP owns its ownMmapRegion(lifetime independent of the store/daf, same model as H5df). Wired into.zip_get_vector_dense+.zip_get_matrix_denseafter the packed/String early-returns.Design notes
compute_alignment_padding, a port of upstream DAF.jl), so mmap fires reliably for both dafr- and Julia-written stores; the alignment check is belt-and-suspenders.packed=TRUEis per-component blosc nesting caught by.files_is_packedbefore the dense path, not archive DEFLATE. So the eager cost was astore_get_bytes+.decode_densedouble-copy, which mmap skips.mode == "r": a writable store keeps pending appends in an in-memory overlay not present in the file mmap, so those offsets would be out of bounds. Writable-mode reads, compressed/packed components, and non-native dtypes all fall back to the eager reader. Opt out entirely withoptions(dafr.mmap = FALSE).Testing
tests/testthat/test-zip-daf-mmap.R(8 tests): dense Float64 matrix/vector + Int32 vector are mmap-backed and correct with dimnames/names;dafr.mmaptoggle below the content-addressed cache; sparse, Int64, and writable-mode reads fall back to eager but stay correct.rcmdcheck --as-cran0-error (checking compiled code ... OK; the 2 warnings are the localcheckbashisms/qpdftool-absence ones).https://claude.ai/code/session_01Av1oZxwSmuUXT4kDjnqjE9