Coalescing ManifestStore.get_many

[TomNicholas]

Implements the chunk-coalescing idea from #947: teach ManifestStore to satisfy a batch of chunk requests with fewer, larger reads by deriving an "effective shard index" from the manifests at read time.

ManifestStore.get_many (overriding the new cross-key Store.get_many) resolves the requested chunk keys to (source file, byte range), groups requests by source file, coalesces each group into runs, and serves each run with one ranged read that is sliced back into per-chunk buffers. No file-format assumptions, no spec changes. Keys that aren't plain manifest-backed chunks (metadata, inlined, missing) fall back to per-key get.

Coalescing uses the object_store / async-tiff two-knob model: references merge into one read when their gap is <= coalesce_max_gap_bytes and the merged read stays <= coalesce_max_bytes.

Why coalesce_max_gap_bytes defaults to 0

Benchmarking a 2D map-tile query (5×5 chunks of air_temperature) against a remote Met Office file on Azure, versus the status quo of one get per chunk:

config	reads	fetched	over-read	median	vs baseline
no coalescing (per-chunk get) — status quo	25	621 KiB	1.00×	273 ms	1.00×
gap=0 (merge only adjacent)	5	621 KiB	1.00×	97 ms	2.8× faster
gap=1 MiB, cap=8 MiB	1	1964 KiB	3.16×	456 ms	1.7× slower
gap=1 MiB, cap=256 KiB	5	621 KiB	1.00×	99 ms	2.8× faster
gap=256 KiB, cap=8 MiB	5	621 KiB	1.00×	96 ms	2.8× faster

A 2D spatial box is contiguous along one axis but strided along the other (row-major chunk layout), so a large gap bridges the row stride and pulls in the chunks between rows — reading ~3× the needed bytes and running slower than no coalescing. Merging only adjacent references is a pure win: zero over-read, ~2.8× faster (collapsing 25 requests into 5). Hence the default gap of 0; coalesce_max_bytes (8 MiB) just bounds the size of any single read. A non-zero gap is available for callers who know their access is contiguous.

Depends on the Store.get_many API and its use in BatchedCodecPipeline in zarr-python (zarr-developers/zarr-python#4112, zarr-developers/zarr-python#4113); until those are released this is a dormant override, tested in isolation.

xref #947

Draft — feedback welcome on the approach and the config surface.

[codecov]

Codecov Report

❌ Patch coverage is 98.80952% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 89.93%. Comparing base (80edcb4) to head (7761438).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
virtualizarr/manifests/store.py	98.80%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1033      +/-   ##
==========================================
- Coverage   90.07%   89.93%   -0.14%     
==========================================
  Files          36       36              
  Lines        2206     2275      +69     
==========================================
+ Hits         1987     2046      +59     
- Misses        219      229      +10     

Files with missing lines	Coverage Δ
virtualizarr/manifests/store.py	92.92% <98.80%> (+2.72%)	⬆️

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[betolink]

This is really cool work @TomNicholas, I like that this will be handled at the chunk manifest level since you have a map of the byte ranges. I'm curious on why this should be a Zarr responsibility.

[TomNicholas]

I'm curious on why this should be a Zarr responsibility.

As opposed to what?

For context there is a rationale here: #947 (comment)

[maxrjones]

I'm curious on why this should be a Zarr responsibility.

As opposed to what?

For context there is a rationale here: #947 (comment)

Probably as opposed to a lower level, e.g., https://developmentseed.org/obstore/latest/api/get/#obstore.get_ranges has a coalesce argument or a wrapper store for coalescing could be added to obspec_utils - https://obspec-utils.readthedocs.io/en/latest/.

It does seem this is a more general problem than ManifestStore, which mostly exists to handle indirection.

[TomNicholas]

See #947 (comment) for why I don't think that can work - I want to coalesce requests over multiple independent keys, which means I need to be able to request multiple zarr keys at once.

ManifestStore, which mostly exists to handle indirection.

ManifestStore (and IcechunkStore, which would implement the same algorithm in rust) is the only place that knows both about the zarr keys the users wants and also about the storage layout (via the manifests).