Background
The current multithreaded filter path has a split policy model:
- FAPL owns
h5::threads{N} / h5::backpressure{N} and the worker pool.
- DAPL owns
h5::high_throughput and a per-dataset pipeline opt-in.
pool_pipeline_t still uses std::queue/future scheduling and per-chunk heap buffers, while the repo also contains H5Qall.hpp Vyukov queues and ring/arena machinery.
This makes the mental model harder than it needs to be for full async I/O: HDF5 calls must remain serialized per file, while filter work can run in parallel.
Goal
Introduce a file-level pipeline context/policy surface so users can express multithreaded read/write filter-chain behavior at FAPL scope, with DAPL acting as a per-dataset policy view rather than a separate execution universe.
Target user shape:
h5::fapl_t fapl = h5::pipeline::multithreaded{
h5::threads{8},
h5::arena{512_MiB},
h5::backpressure{64}
};
Scope
- Add the public FAPL-level pipeline policy syntax.
- Preserve existing
h5::threads{N} / h5::backpressure{N} behavior.
- Keep
h5::high_throughput working as the dataset opt-in / compatibility path.
- Centralize the policy so future Vyukov queue + arena scheduling can hang from the same file-level context.
- Add focused tests for syntax, policy resolution, and compatibility.
Non-goals
- Do not rewrite the whole filter engine in one pass.
- Do not expose low-level Vyukov queue topology as the default API.
- Do not make concurrent HDF5 C-API calls from worker threads.
Acceptance Criteria
h5::pipeline::multithreaded{h5::threads{N}, h5::arena{...}, h5::backpressure{M}} compiles as a FAPL policy.
- Existing
h5::threads{N} tests continue to pass.
- The policy preserves the invariant: HDF5 calls stay serialized; CPU filter work remains the parallelizable portion.
- Existing high-throughput read/write tests remain green.
Background
The current multithreaded filter path has a split policy model:
h5::threads{N}/h5::backpressure{N}and the worker pool.h5::high_throughputand a per-dataset pipeline opt-in.pool_pipeline_tstill usesstd::queue/futurescheduling and per-chunk heap buffers, while the repo also containsH5Qall.hppVyukov queues and ring/arena machinery.This makes the mental model harder than it needs to be for full async I/O: HDF5 calls must remain serialized per file, while filter work can run in parallel.
Goal
Introduce a file-level pipeline context/policy surface so users can express multithreaded read/write filter-chain behavior at FAPL scope, with DAPL acting as a per-dataset policy view rather than a separate execution universe.
Target user shape:
Scope
h5::threads{N}/h5::backpressure{N}behavior.h5::high_throughputworking as the dataset opt-in / compatibility path.Non-goals
Acceptance Criteria
h5::pipeline::multithreaded{h5::threads{N}, h5::arena{...}, h5::backpressure{M}}compiles as a FAPL policy.h5::threads{N}tests continue to pass.