Conversation
Closes the last perf-parity gap with DataAxesFormats.jl: its H5df mmaps contiguous dense datasets (HDF5.readmmap / MappedData) while dafr eager- decoded them, leaving dafr ~3x slower on a full read of a large dense matrix and ~40x slower on a lazy/sliced read. FilesDaf already mmaps; this brings H5df to the same ALTREP fast path. - MmapRegion::open_readonly gains a byte `offset` (whole file still mapped from 0, so page-aligned; data()/nbytes() account for the offset). Exposed through mmap_real/mmap_int/mmap_lgl's new `offset` argument. - .h5_mmap_dense gates like Julia's ismmappable: native LE Float64 / signed Int32, contiguous (hdf5r $get_offset() errors on chunked, doubling as the contiguity check), uncompressed (storage_size == n*elt), element-aligned, non-empty. Everything else falls back to the eager reader. - HDF5 naturally element-aligns datasets >= its small-data-block size (2048 B); smaller ones are packed unaligned and read eagerly (cheap anyway). No on-disk format change, so Julia interop is preserved. - Opt out with options(dafr.mmap = FALSE), same switch as FilesDaf. Measured (160 MB dense Float64 matrix): full read 133ms -> 46ms (parity with the FilesDaf mmap ceiling), lazy read 103ms -> 2.4ms. Full suite: 0 failures / 6472 pass. rcmdcheck --as-cran: 0 errors (only the pre-existing tool-absence warnings). Claude-Session: https://claude.ai/code/session_01Av1oZxwSmuUXT4kDjnqjE9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the last perf-parity gap with
DataAxesFormats.jl: itsH5dfmemory-maps contiguous dense datasets (HDF5.readmmap/MappedData), while dafr eager-decoded them - leaving dafr ~3x slower on a full read of a large dense matrix and ~40x slower on a lazy/sliced read.files_daf()already mmaps; this bringsh5df()to the same ALTREP fast path.What changed
MmapRegion::open_readonlygains a byteoffset- the whole file is still mapped from 0 (so the mapping stays page-aligned);data()/nbytes()account for the offset. Exposed via a newoffsetarg onmmap_real()/mmap_int()/mmap_lgl(). Zero changes to the ALTREP layer itself..h5_mmap_densegates exactly like Julia'sismmappable: native little-endian Float64 / signed Int32, contiguous (hdf5r's$get_offset()errors on chunked/compact, which doubles as the contiguity check), uncompressed (storage_size == n * eltsize), element-aligned, non-empty. Anything else falls back to the eager reader.set_alignment, but HDF5 naturally element-aligns datasets ≥ its small-data-block size (2048 B); smaller datasets pack unaligned and read eagerly (cheap anyway). So dafr-written large dense datasets mmap; foreign/unaligned ones safely fall back. Julia interop is preserved (offset-based readers are unaffected by alignment padding).options(dafr.mmap = FALSE), the same switchfiles_daf()uses.Measured (160 MB dense Float64 matrix)
get+sum)get)Verification
rcmdcheck --as-cran: 0 errors; the only warnings are the pre-existingcheckbashisms/qpdftool-absence artifacts.test-h5df-mmap.R(mmap fires for large dense Float64/Int32 matrix + vectors, correct values/dimnames, eager fallback below threshold + for sparse, option toggle), plus offset cases intest-altrep-mmap.R.ZipDaf mmap (the other single-file backend) is intentionally left for a follow-up.
https://claude.ai/code/session_01Av1oZxwSmuUXT4kDjnqjE9