feat: H5df backend (0.7.0) - #11
Merged
Merged
Conversation
Single-file .h5df HDF5 store, interoperable with DataAxesFormats.jl H5df. Full-parity scope (read+write+delete+overwrite+simple reorder) with compression/grouped-stores/mmap/crash-safe-reorder/sparse-string-write deferred as named Phase-2 ceilings. Claude-Session: https://claude.ai/code/session_013CEMsVoeh6FpsysEWowYw2
Placeholder-free plan with hdf5r idioms verified by probing hdf5r 1.3.10: chunk_dims=NULL flat datasets, scalar-space writes, matrix-orientation no-op transpose, group-vs-dataset sparsity, empty-vlen read guard, link_delete. 8 tasks: scaffold, scalars, axes, vectors, matrices, relayout/reorder/guards, adversarial+Julia interop, packaging. Claude-Session: https://claude.ai/code/session_013CEMsVoeh6FpsysEWowYw2
Adds axes/<axis> HDF5 dataset storage with eager creation of vectors/<axis> and every matrices/<a>/<b> pairing (incl. self) on add_axis, matching FilesDaf/ZipDaf layout so a Julia reader sees the expected group structure. delete_axis cascades: axes/<axis>, vectors/<axis>, matrices/<axis>, and matrices/<other>/<axis> for every other axis, then clears the in-memory axis cache and bumps the axis version counter (matches the FilesDaf/MemoryDaf/ZarrDaf convention documented in cache.R; the reference snippet omitted it). Also adds .h5_safe_read, a guard around hdf5r's crash when reading an empty vlen-string dataset, needed here for the empty-axis case and reused by later vector/matrix tasks.
Dense matrices write/read directly via hdf5r (no transpose): hdf5r reverses dims on disk, matching Julia's column-major convention. Sparse as CSC colptr/rowval/nzval group (1-based), bool-all-true omits nzval. Claude-Session: https://claude.ai/code/session_013CEMsVoeh6FpsysEWowYw2
- Bump to 0.7.0 with NEWS entry for the H5df backend. - Merge H5df/H5dfReadOnly class docs into the h5df topic (@Rdname): a separate H5df.Rd would collide case-insensitively with h5df.Rd and fail R CMD check's portable-file-name check. @inheritParams DafReader documents the inherited constructor args. - Update the stale test-complete.R assertion that expected the old "not supported yet" stub; .h5df now dispatches to h5df(). - rcmdcheck --as-cran: 0 errors, 0 warnings (bar the local checkbashisms/ qpdf tool-absence artifacts that do not occur on CI). Claude-Session: https://claude.ai/code/session_013CEMsVoeh6FpsysEWowYw2
- Apply .h5_coerce_int64 to the dense matrix read too (integer64 keeps a dim; the prior comment claiming otherwise was wrong), so a read-modify- rewrite of a Julia Int64 matrix no longer downgrades it to Int32. - Assert matrix values after reorder_axes and the delete_axis subgroup cascade (both previously ran unverified). - Broaden Julia->R interop to string scalar/vector and sparse vector. - hdf5r cannot read the HDF5 bitfield type Julia uses for Bool; surface a clear error instead of hdf5r's cryptic conversion failure, document the caveat in NEWS, and assert it in the interop test. (R->Julia bool works.) Claude-Session: https://claude.ai/code/session_013CEMsVoeh6FpsysEWowYw2
Design spec + implementation plan stay in this branch's history (commits d5fa4cf, bb30fad); main tracks only package files, matching the ZipDaf (PR #9) convention. Claude-Session: https://claude.ai/code/session_013CEMsVoeh6FpsysEWowYw2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the
H5dfstorage backend - a whole Daf store in one.h5dfHDF5 file, interoperable withDataAxesFormats.jl'sH5df. This is the last of the two deferred single-file backends (ZipDaf shipped in 0.6.0).h5df(path, mode)+open_daf()dispatch on*.h5df. Built onhdf5r(already a Suggests dep).format_*parity with the other leaf backends: scalars, axes, dense + sparse (CSC) vectors and matrices, string components, delete/overwrite,relayout, and a simplifiedreorder. Not append-only (unlike ZipDaf).dafUInt8[1,0]marker dataset;scalars/axes/vectors/matricesgroups; sparsity is group-vs-dataset (colptr/rowval/nzval, 1-based); no HDF5 attributes.H5dfis not gated off Windows (unlike the POSIX-onlyMmapZipStorebehind ZipDaf).Interop
Bidirectional interop verified live against DataAxesFormats.jl 0.3.0 (
test-h5df-julia-compat.R): R->Julia and Julia->R for scalars, axes, Int32/Float64 vectors, dense-matrix orientation, and sparse matrices; Julia->R also for string scalar/vector and sparse vector.One documented limitation:
hdf5rcannot read the HDF5 bitfield type Julia uses forBool, so reading a Julia-writtenBoolcomponent in R raises a clear, actionable error (writingBoolfrom R to a store Julia reads works fine). Noted in NEWS.Deferred (Phase 2, YAGNI)
Compressed/packed writing, grouped
.h5dfs#/groupstores, mmap zero-copy reads, and crash-safe reorder - each additive without reworking the v1 layout.Test plan
R-CMD-checkgreen across all 5 OSesaltrep-sanitygreentest-h5df.R(64 assertions),test-h5df-adversarial.R(4),test-h5df-julia-compat.R(9, julia-gated). Localrcmdcheck --as-cran: 0 errors, 0 warnings (bar the local checkbashisms/qpdf tool-absence artifacts).https://claude.ai/code/session_013CEMsVoeh6FpsysEWowYw2