Skip to content

feat(zipdaf): mmap zero-copy reads for dense STORE'd components (0.9.0) - #13

Merged
aviezerl merged 2 commits into
mainfrom
zipdaf-mmap
Jul 7, 2026
Merged

feat(zipdaf): mmap zero-copy reads for dense STORE'd components (0.9.0)#13
aviezerl merged 2 commits into
mainfrom
zipdaf-mmap

Conversation

@aviezerl

@aviezerl aviezerl commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Summary

ZipDaf dense reads of native little-endian Float64 / signed Int32 components are now memory-mapped (zero-copy ALTREP) when the store is opened read-only, matching files_daf(), h5df(), and DataAxesFormats.jl's ZipDaf. This closes the last measured perf-parity gap with DAF.jl - all three backends now mmap large dense reads.

Result

160 MB dense Float64 matrix (warm cache, empty_cache per iter, median of 11; dev/bench/mmap-bench.R):

backend lazy get full get_sum
files (mmap ceiling) 0.68 ms 44.2 ms
zip (was eager 281 ms / 440 ms) 0.73 ms (1.1x) 44.2 ms (1.0x)
h5df (mmap) 2.13 ms 45.9 ms

ZipDaf now matches FilesDaf and beats H5df on lazy reads (no hdf5r object open).

What changed

  • New registered C++ helper dafr_mmap_zip_stored_offset(xptr, key) -> c(offset, nbytes) for a STORE'd entry (via stored_view, p - file_mmap_base()), or NULL if the key is absent or compressed. Windows stub added.
  • .zip_mmap_dense() gates like .h5_mmap_dense: dafr.mmap on, n > 0, mode == "r", eltype in {Float64, signed Int32}, nbytes == n*eltsize, element-aligned offset. Opens an independent read-only mmap of the .daf.zip via mmap_real/mmap_int, so the ALTREP owns its own MmapRegion (lifetime independent of the store/daf, same model as H5df). Wired into .zip_get_vector_dense + .zip_get_matrix_dense after the packed/String early-returns.
  • No on-disk format change; Julia interop preserved.

Design notes

  • The archive writer already 8-byte-aligns every entry's data region (compute_alignment_padding, a port of upstream DAF.jl), so mmap fires reliably for both dafr- and Julia-written stores; the alignment check is belt-and-suspenders.
  • ZipDaf archive entries are always STORED (method 0); packed=TRUE is per-component blosc nesting caught by .files_is_packed before the dense path, not archive DEFLATE. So the eager cost was a store_get_bytes + .decode_dense double-copy, which mmap skips.
  • Gate on mode == "r": a writable store keeps pending appends in an in-memory overlay not present in the file mmap, so those offsets would be out of bounds. Writable-mode reads, compressed/packed components, and non-native dtypes all fall back to the eager reader. Opt out entirely with options(dafr.mmap = FALSE).

Testing

  • New tests/testthat/test-zip-daf-mmap.R (8 tests): dense Float64 matrix/vector + Int32 vector are mmap-backed and correct with dimnames/names; dafr.mmap toggle below the content-addressed cache; sparse, Int64, and writable-mode reads fall back to eager but stay correct.
  • Full suite 0-fail (only expected c-blosc skips); rcmdcheck --as-cran 0-error (checking compiled code ... OK; the 2 warnings are the local checkbashisms/qpdf tool-absence ones).

https://claude.ai/code/session_01Av1oZxwSmuUXT4kDjnqjE9

aviezerl added 2 commits July 7, 2026 06:29
ZipDaf dense reads of native little-endian Float64 / signed Int32
components are now memory-mapped (zero-copy ALTREP) when the store is
opened read-only, matching files_daf(), h5df(), and DataAxesFormats.jl's
ZipDaf. The archive writer already 8-byte-aligns every entry's data
region (compute_alignment_padding), so mmap fires for any dense read;
the alignment check is a belt-and-suspenders guard that also covers
Julia-written stores.

New C++ helper dafr_mmap_zip_stored_offset(xptr, key) returns the file
byte offset + length of a STORE'd entry (or NULL if absent/compressed);
R's .zip_mmap_dense() gates on it and opens an independent read-only mmap
of the .daf.zip via mmap_real/mmap_int, so the ALTREP owns its own
MmapRegion independent of the store/daf lifetime.

Only read-only opens mmap: a writable store keeps pending appends in an
in-memory overlay not present in the file mmap, so those offsets would be
out of bounds - writable-mode reads fall back to eager. Compressed/packed
components and non-native dtypes also fall back. Opt out with
options(dafr.mmap = FALSE).

On a 160 MB dense matrix a lazy get drops from ~280 ms to sub-millisecond
and a full read reaches the files_daf mmap ceiling (~44 ms), closing the
last measured perf-parity gap. No on-disk format change; Julia interop
preserved. Tests in test-zip-daf-mmap.R (8).

Claude-Session: https://claude.ai/code/session_01Av1oZxwSmuUXT4kDjnqjE9
The 7 mmap tests all build a zip_daf, which errors on Windows where
MmapZipStore is not compiled (src/mmap_zip_store_win_stubs.cpp). Guard
each with skip_if_no_mmap_zip(), matching test-zip-daf.R and the other
mmap-zip-store test files.

Claude-Session: https://claude.ai/code/session_01Av1oZxwSmuUXT4kDjnqjE9
@aviezerl
aviezerl merged commit 9cad6bc into main Jul 7, 2026
6 checks passed
@aviezerl
aviezerl deleted the zipdaf-mmap branch July 7, 2026 05:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant