Skip to content

feat(h5df): mmap zero-copy reads for large dense components (0.8.0) - #12

Merged
aviezerl merged 1 commit into
mainfrom
h5df-mmap
Jul 6, 2026
Merged

feat(h5df): mmap zero-copy reads for large dense components (0.8.0)#12
aviezerl merged 1 commit into
mainfrom
h5df-mmap

Conversation

@aviezerl

@aviezerl aviezerl commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Closes the last perf-parity gap with DataAxesFormats.jl: its H5df memory-maps contiguous dense datasets (HDF5.readmmap / MappedData), while dafr eager-decoded them - leaving dafr ~3x slower on a full read of a large dense matrix and ~40x slower on a lazy/sliced read. files_daf() already mmaps; this brings h5df() to the same ALTREP fast path.

What changed

  • MmapRegion::open_readonly gains a byte offset - the whole file is still mapped from 0 (so the mapping stays page-aligned); data()/nbytes() account for the offset. Exposed via a new offset arg on mmap_real() / mmap_int() / mmap_lgl(). Zero changes to the ALTREP layer itself.
  • .h5_mmap_dense gates exactly like Julia's ismmappable: native little-endian Float64 / signed Int32, contiguous (hdf5r's $get_offset() errors on chunked/compact, which doubles as the contiguity check), uncompressed (storage_size == n * eltsize), element-aligned, non-empty. Anything else falls back to the eager reader.
  • No on-disk format change. hdf5r exposes no set_alignment, but HDF5 naturally element-aligns datasets ≥ its small-data-block size (2048 B); smaller datasets pack unaligned and read eagerly (cheap anyway). So dafr-written large dense datasets mmap; foreign/unaligned ones safely fall back. Julia interop is preserved (offset-based readers are unaffected by alignment padding).
  • Opt out with options(dafr.mmap = FALSE), the same switch files_daf() uses.

Measured (160 MB dense Float64 matrix)

before after
full read (get+sum) 133 ms 46 ms (parity with the FilesDaf mmap ceiling)
lazy read (get) 103 ms 2.4 ms

Verification

  • Full test suite: 0 failures / 6472 pass (141 environmental skips).
  • rcmdcheck --as-cran: 0 errors; the only warnings are the pre-existing checkbashisms/qpdf tool-absence artifacts.
  • New tests: test-h5df-mmap.R (mmap fires for large dense Float64/Int32 matrix + vectors, correct values/dimnames, eager fallback below threshold + for sparse, option toggle), plus offset cases in test-altrep-mmap.R.

ZipDaf mmap (the other single-file backend) is intentionally left for a follow-up.

https://claude.ai/code/session_01Av1oZxwSmuUXT4kDjnqjE9

Closes the last perf-parity gap with DataAxesFormats.jl: its H5df mmaps
contiguous dense datasets (HDF5.readmmap / MappedData) while dafr eager-
decoded them, leaving dafr ~3x slower on a full read of a large dense
matrix and ~40x slower on a lazy/sliced read. FilesDaf already mmaps; this
brings H5df to the same ALTREP fast path.

- MmapRegion::open_readonly gains a byte `offset` (whole file still mapped
  from 0, so page-aligned; data()/nbytes() account for the offset). Exposed
  through mmap_real/mmap_int/mmap_lgl's new `offset` argument.
- .h5_mmap_dense gates like Julia's ismmappable: native LE Float64 / signed
  Int32, contiguous (hdf5r $get_offset() errors on chunked, doubling as the
  contiguity check), uncompressed (storage_size == n*elt), element-aligned,
  non-empty. Everything else falls back to the eager reader.
- HDF5 naturally element-aligns datasets >= its small-data-block size
  (2048 B); smaller ones are packed unaligned and read eagerly (cheap
  anyway). No on-disk format change, so Julia interop is preserved.
- Opt out with options(dafr.mmap = FALSE), same switch as FilesDaf.

Measured (160 MB dense Float64 matrix): full read 133ms -> 46ms (parity
with the FilesDaf mmap ceiling), lazy read 103ms -> 2.4ms.

Full suite: 0 failures / 6472 pass. rcmdcheck --as-cran: 0 errors (only
the pre-existing tool-absence warnings).

Claude-Session: https://claude.ai/code/session_01Av1oZxwSmuUXT4kDjnqjE9
@aviezerl
aviezerl merged commit 21ebf48 into main Jul 6, 2026
6 checks passed
@aviezerl
aviezerl deleted the h5df-mmap branch July 6, 2026 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant