Skip to content

refactor, unify filter pipeline scheduling around FAPL context #287

@steven-varga

Description

@steven-varga

Background

The current multithreaded filter path has a split policy model:

  • FAPL owns h5::threads{N} / h5::backpressure{N} and the worker pool.
  • DAPL owns h5::high_throughput and a per-dataset pipeline opt-in.
  • pool_pipeline_t still uses std::queue/future scheduling and per-chunk heap buffers, while the repo also contains H5Qall.hpp Vyukov queues and ring/arena machinery.

This makes the mental model harder than it needs to be for full async I/O: HDF5 calls must remain serialized per file, while filter work can run in parallel.

Goal

Introduce a file-level pipeline context/policy surface so users can express multithreaded read/write filter-chain behavior at FAPL scope, with DAPL acting as a per-dataset policy view rather than a separate execution universe.

Target user shape:

h5::fapl_t fapl = h5::pipeline::multithreaded{
    h5::threads{8},
    h5::arena{512_MiB},
    h5::backpressure{64}
};

Scope

  • Add the public FAPL-level pipeline policy syntax.
  • Preserve existing h5::threads{N} / h5::backpressure{N} behavior.
  • Keep h5::high_throughput working as the dataset opt-in / compatibility path.
  • Centralize the policy so future Vyukov queue + arena scheduling can hang from the same file-level context.
  • Add focused tests for syntax, policy resolution, and compatibility.

Non-goals

  • Do not rewrite the whole filter engine in one pass.
  • Do not expose low-level Vyukov queue topology as the default API.
  • Do not make concurrent HDF5 C-API calls from worker threads.

Acceptance Criteria

  • h5::pipeline::multithreaded{h5::threads{N}, h5::arena{...}, h5::backpressure{M}} compiles as a FAPL policy.
  • Existing h5::threads{N} tests continue to pass.
  • The policy preserves the invariant: HDF5 calls stay serialized; CPU filter work remains the parallelizable portion.
  • Existing high-throughput read/write tests remain green.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions